AI 的挑战与未来：对齐、可靠性与经济影响

好的，今天我有幸采访到伊利亚·苏茨克维，他是 OpenAI 的联合创始人兼首席科学家。伊利亚，欢迎来到月球协会。

Okay, today I have the pleasure of interviewing Ilya Sutskever, who is the co-founder and chief scientist of OpenAI. Ilya, welcome to the Lunar Society.

谢谢，很高兴来到这里。

Thank you, happy to be here.

第一个问题，不许谦虚。很多科学家会在自己的领域取得重大突破，但能在整个职业生涯中做出多个独立定义领域的突破的科学家要少得多。区别是什么？你和其他研究者有什么不同？为什么你能在一个领域取得多个突破？

First question, and no humility allowed. There are many scientists who will make a big breakthrough in their field. There are far fewer scientists who will make multiple independent breakthroughs that define their field throughout their career. What is the difference? What distinguishes you from other researchers? Why have you been able to find multiple breakthroughs in a field?

嗯，谢谢你的美言。这个问题很难回答。我的意思是，我非常努力。我倾尽所有，到目前为止这很有效。我想就这么简单。

Well, thank you for the kind words. It's hard to answer that question. I mean, I try really hard. I gave it everything I got, and that worked so far. I think that's all there is to it.

为什么 GPT 没有被更多非法使用？为什么没有更多外国政府用它来传播宣传或诈骗老太太之类的？

What's the explanation for why there aren't more illicit uses of GPT? Why aren't more foreign governments using it to spread propaganda or scam grandmothers or something?

我的意思是，也许他们还没有大规模这么做，但如果现在就有一些正在发生，我也不会感到惊讶。当然，我想他们会拿一些开源模型尝试用于这个目的。我预计未来他们会对此感兴趣。技术上可行，只是他们还没想够，或者还没有用他们的技术大规模实施。或者也许正在发生，但你不知道。

I mean, maybe they haven't really gotten to do it a lot, but it also wouldn't surprise me if some of it was going on right now. Certainly, I imagine they'd be taking some of the open source models and trying to use them for that purpose. I would expect this would be something they'd be interested in in the future. It's technically possible; they just haven't thought about it enough or haven't done it at scale using their technology. Or maybe it's happening but you don't know it.

如果正在发生，你能追踪到吗？

Would you be able to track it if it was happening?

我认为大规模追踪是可能的，是的。我的意思是，这需要特殊操作，但可行。

I think large-scale tracking is possible, yes. I mean, this requires a special operation, but it is possible.

现在有一个窗口期，AI 的经济价值非常高，比如说相当于飞机的规模，但我们还没达到 AGI。这个窗口有多大？

Now there's some window in which AI is very economically valuable, on the scale of airplanes let's say, but we haven't reached AGI yet. How big is that window?

我认为这个窗口，很难给你一个精确的答案，但肯定是一个好几年的窗口。这也是一个定义问题，因为 AI 在成为 AGI 之前，价值会逐年增加，我是指数级增长。所以从某种意义上说，尤其是事后看来，可能感觉只有一两年，因为那两年比之前所有年份都大。但我要说，去年 AI 已经产生了相当多的经济价值，明年会更大，之后越来越大。所以我认为这会是一个好几年的时间段，而且从现在到 AGI 基本上都是如此。

I think this window, it's hard to give you a precise answer, but it's definitely going to be a good multi-year window. It's also a question of definition because AI before it becomes AGI is going to be increasingly more valuable year after year, I'd say in an exponential way. So in some sense, it may feel like, especially in hindsight, it may feel like there was only one year or two years because those two years were larger than the previous years. But I would say that already last year there has been a fair amount of economic value produced by AI, and next year it's going to be larger and larger after that. So I think it's going to be a good multi-year chunk of time, but that's going to be true from now to AGI pretty much.

好的，我很好奇，如果现在有家初创公司用你们的模型。到某个时候，如果你有了 AGI，世界上就只有一家企业了，对吧？就是 OpenAI。他们还有多少窗口期？有没有企业能真正生产出 AGI 无法生产的东西？

Okay, well, I'm curious if there's a startup that's using your models right now. At some point, if you have AGI, there's only one business in the world, right? It's OpenAI. How much window do they have? Do any businesses have where they're actually producing something that AGI can't produce?

是的，我的意思是，这和问多久能到 AGI 是同一个问题。我觉得很难回答。我犹豫着不给你一个数字，也是因为圣埃尔斯维尔效应，那些乐观的、正在研究这项技术的人往往会低估达到目标所需的时间。但我让自己脚踏实地的方法是特别想想自动驾驶汽车。有一个类比：如果你看一辆特斯拉，看它的自动驾驶行为，它看起来什么都能做，但很明显在可靠性方面还有很长的路要走。我们的模型可能也处于类似的位置，看起来什么都能做，但同时我们需要做更多的工作，直到真正解决所有问题，让它变得非常好、非常可靠、稳健且行为良好。

Yeah, well, I mean, it's the same question as asking how long until AGI. I think it's a hard question to answer. I hesitate to give you a number also because there is the St. Elsewhere effect, where people who are optimistic, people who are working on the technology, tend to underestimate the time it takes to get there. But I think the way I ground myself is by thinking about the self-driving car in particular. There is an analogy where if you look at a Tesla and if you look at the self-driving behavior of it, it looks like it does everything alright, but it's also clear that there is still a long way to go in terms of reliability. And we might be in a similar place with respect to our models, where it also looks like we can do everything, and at the same time we'll need to do some more work until we really iron out all the issues and make it really good and really reliable and robust and well-behaved.

到 2030 年，AI 占 GDP 的百分比是多少？

By 2030, what percent of GDP is AI?

哦天哪，这个问题很难回答。非常难回答。给我一个上下限吧。问题是我的误差范围是对数尺度的，所以我可以想象一个巨大的百分比，也可以想象一个令人失望的小百分比。

Oh gosh, hard to answer that question. Very hard to answer that question. Give me an over under. The problem is that my error bars are in log scale, so I could imagine a huge percentage, I could imagine a disappointingly small percentage at the same time.

好的，那我们假设一个反事实，它只占很小百分比。假设到了 2030 年，这些大语言模型并没有创造太多经济价值，尽管你认为这不太可能。你现在最好的解释是什么，为什么会出现这种情况？

Okay, so let's take the counterfactual where it is a small percentage. Let's say it's 2030 and not that much economic value has been created by these LLMs, as unlikely as you think this might be. What would be your best explanation right now why something like this might happen?

我最好的解释？我真的认为这不太可能，这是评论的前提。但如果我接受你问题的前提，嗯，为什么在现实世界的影响方面令人失望？我的答案是可靠性。如果最终你希望它们可靠，但它们却不可靠，或者可靠性比我们预期的更难。我真的不认为会这样，但如果我必须选一个，你告诉我‘嘿，为什么没成功？’，那就是可靠性。你仍然需要检查答案并仔细核对所有内容，这确实会抑制这些系统所能产生的经济价值。它们在技术上会成熟，只是它们是否足够可靠的问题。

My best explanation? I really don't think that's a likely possibility, so that's the preface to the comment. But if I were to take the premise of your question, well, why were things disappointing in terms of the real-world impact? And my answer would be reliability. If somehow it ends up being the case that you really want them to be reliable and they ended up not being reliable, or if reliability turns out to be harder than we expect. I really don't think that will be the case, but if I had to pick one and you tell me 'hey, why didn't things work out?', it would be reliability. That you still have to look over the answers and double-check everything, and that just really puts a damper on the economic value that can be produced by those systems. They'll be technologically mature; it's just a question of whether they'll be reliable enough.

是的，从某种意义上说，不可靠就意味着技术上不成熟，你明白我的意思。

Yeah, in some sense not reliable means not technologically mature, if you see what I mean.

是的，有道理。

Yeah, fair enough.

生成模型之后是什么？之前你在研究强化学习。这基本上就是全部了吗？这是一个能让我们达到 AGI 的范式，还是之后还有别的东西？

What's after generative models? Right, so before you were working on reinforcement learning. Is this basically it? Is this a paradigm that gets us to AGI, or is there something after this?

我的意思是，我认为这个范式会走得非常非常远，我不会低估它。我认为这个确切的范式很可能不会是 AGI 的最终形态。我的意思是，我不太愿意确切地说下一个范式是什么，但我认为它可能会整合过去所有不同的想法。

I mean, I think this paradigm is going to go really, really far, and I would not underestimate it. I think it's quite likely that this exact paradigm is not going to be the AGI form factor. I mean, I hesitate to say precisely what the next paradigm will be, but I think it will probably involve integration of all the different ideas that came in the past.

你指的是某个具体的想法吗？

Is there some specific one you're referring to?

我的意思是，很难具体说明。所以你可以说下一个词预测只能帮助匹配人类表现，也许不能超越它。要超越人类表现需要什么？所以我挑战下一个词预测无法超越人类表现的说法。

I mean, it's hard to be specific. So you could argue that the next-token prediction can only help match human performance, and maybe not surpass it. What would it take to surpass human performance? So I challenge the claim that next-token prediction cannot surpass human performance.

表面上，如果你只是学习模仿来预测人们的行为，那就意味着你只能复制别人。但这里有一个反驳观点，说明情况可能并非如此。如果你的神经网络足够聪明，你只需问它，一个具有深刻洞察力和远见能力的人会怎么做？也许这样的人并不存在，但神经网络很有可能推断出这样的人应该如何行事。你明白我的意思吗？

Like on the surface, if you just learn to imitate to predict what people do, it means that you can only copy people. But here is a counterargument for why that might not be quite so. If your neural net is smart enough, you just ask it like, what would a person with great insight and visionary capability do? Maybe such a person doesn't exist, but there's a pretty good chance that the neural net will be able to extrapolate how such a person should behave. Do you see what I mean?

是的，但如果不是从普通人的数据中，我们如何获得关于那个人会做什么的洞察呢？因为如果你仔细想想，很好地预测下一个词意味着什么？实际上，这比看起来要深刻得多。很好地预测下一个词意味着你理解了产生那个词背后的现实。这不仅仅是统计学——好吧，它是统计学，但统计学是什么？为了理解统计数据、压缩它们，你需要理解世界是如何产生这些统计数据的。然后你说，好吧，我有所有这些人的数据。是什么造就了他们的行为？他们有思想、情感和想法，他们以某种方式行事。所有这些都可以从下一个词预测中推导出来。我认为这应该有可能，虽然不是无限的，但在相当程度上，你可以说，如果你有一个具有这样那样特征的人，你能猜出他会怎么做吗？这样的人并不存在，但因为你非常擅长预测下一个词，你仍然应该能够猜出那个人会做什么——这个假设的、拥有远超我们心智能力的虚构人物。

Yes, although where would we get the sort of insight about what that person would do if not from the data of regular people? Because if you think about it, what does it mean to predict the next token well enough? What does it mean actually? It's a much deeper question than it seems. Predicting the next token well means that you understand the underlying reality that led to the creation of that token. It's not statistics—well, it is statistics, but what is statistics? In order to understand the statistics, to compress them, you need to understand what is it about the world that creates those statistics. And so then you say, okay, well, I have all those people. What is it about people that creates their behaviors? Well, they have thoughts and feelings and ideas, and they do things in certain ways. All of those would be deduced from next-token prediction. And I'd argue that this should make it possible, not indefinitely but to a pretty decent degree, to say, can you guess what you would do if you took a person with this characteristic and that characteristic? Such a person doesn't exist, but because you're so good at predicting the next token, you should still be able to guess what that person would do—this hypothetical imaginary person with far greater mental ability than the rest of us.

当我们对这些模型进行强化学习时，多久之后强化学习的大部分数据会来自人工智能而非人类？

When we're doing reinforcement learning on these models, how long before most of the data for the reinforcement learning is coming from AI and not humans?

我的意思是，强化学习的大部分数据已经来自人工智能了。是的，人类被用来训练奖励函数，但奖励函数在与模型交互时是自动的，强化学习过程中产生的所有数据都是由人工智能创建的。所以如果你看看当前的技术范式，由于 ChatGPT 而受到广泛关注的基于人类反馈的强化学习（RLHF）——确实有人类反馈。人类反馈被用来训练奖励函数，然后奖励函数被用来创建训练模型的数据。

I mean, already most of the data for reinforcement learning is coming from AI. Yeah, well, it's like the humans are being used to train the reward function, but then the reward function in its interaction with the model is automatic, and all the data that's generated during the process of reinforcement learning is created by AI. So if you look at the current technique paradigm, which has been getting significant attention because of ChatGPT, reinforcement learning from human feedback—so there is human feedback. The human feedback is being used to train the reward function, and then the reward function is being used to create the data which trains the model.

有没有可能完全将人类从循环中移除，让它以某种 AlphaGo 的方式自我改进？

And is there any hope of just removing the human from the loop and having it improve itself in some sort of AlphaGo way?

是的，当然。我觉得在某种意义上，我们的计划很大程度上就是这样。你真正想要的是教导人工智能的人类教师与人工智能合作。你可以想象这样一个世界：人类教师做 1%的工作，人工智能做 99%的工作。你不希望它是 100%的人工智能，但你希望它是一种人机协作，来教导下一个机器。

Yeah, definitely. I mean, I feel like in some sense, our hopes for our plan very much so. The thing you really want is for the human teachers that teach the AI to collaborate with an AI. You might want to think about it as being in a world where the human teachers do 1% of the work and the AI do 99% of the work. You don't want it to be 100% AI, but you do want it to be a human-machine collaboration which teaches the next machine.

目前，我有机会试用这些模型。它们似乎不擅长多步推理，而且正在变得更好，但要真正突破这个障碍需要什么？

Currently, I've had a chance to play around with these models. They seem bad at multi-step reasoning, and they are getting better, but what does it take to really surpass that barrier?

我认为专门的训练会让我们达到目标。基础模型的更多改进也会让我们达到目标。但根本上，我也不觉得它们在多步推理上那么差。我实际上认为它们不擅长内心的多步推理，但它们不被允许大声思考。但当它们被允许大声思考时，它们表现得相当好。我预计随着更好的模型和专门的训练，这一点会显著改善。

I think dedicated training will get us there. More improvements of the base models will get us there. But fundamentally, I also don't feel like they're that bad at multi-step reasoning. I actually think that they are bad at mental multi-step reasoning but they're not allowed to think out loud. But when they are allowed to think out loud, they're quite good. And I expect this to improve significantly both with better models and with special training.

互联网上的推理词元会用完吗？它们足够多吗？

Are we running out of reasoning tokens on the internet? Are there enough of them?

关于这个问题，有说法认为确实在某个时候我们会用尽训练这些模型所需的词元。是的，我认为这一天会到来。到那时，我们需要有其他训练模型的方法，其他有效提升能力、优化行为的方法，确保它们在没有更多数据的情况下也能完全按照我们的意愿行事。不过，我们还没有用尽数据。还有更多。我想说数据情况仍然相当好；还有很多路要走。但到了某个时候，是的，数据会用完。

So for context on this question, there are claims that indeed at some point we will run out of tokens in general to train these models. And yeah, I think this will happen one day. By the time that happens, we need to have other ways of training models, other ways of productively improving their capabilities and sharpening their behavior, making sure they're doing exactly precisely what we want without more data. Well, we haven't run out of data yet. There's more. I would say the data situation is still quite good; there's still lots to go. But at some point, yeah, data will run out.

最有价值的数据来源是什么？是 Reddit、Twitter、书籍吗？你愿意用许多其他类型的词元来交换什么？

What is the most valuable source of data? Is it Reddit, Twitter, books? What would you trade many other tokens of other varieties for?

一般来说，你希望得到谈论更聪明事物的词元，更有趣的词元。所以你提到的所有来源都是有价值的。好吧，也许 Twitter 不算。但我们需要转向多模态来获得更多词元吗？还是我们仍有足够的文本词元？

Generally speaking, you'd like tokens which are speaking about smarter things, tokens which are more interesting. So all the sources you mentioned are valuable. Okay, so maybe not Twitter. But do we need to go multimodal to get more tokens, or do we still have enough text tokens left?

我认为仅靠文本仍然可以走得很远，但转向多模态似乎是一个非常有前景的方向。

I think that you can still go very far in text only, but going multimodal seems like a very fruitful direction.

如果你方便谈这个，我们还没有抓取词元的地方是哪里？

If you're comfortable talking about this, where is the place where we haven't scraped the tokens yet?

显然，我不能为我们回答这个问题，但我相信对每个人来说，这个问题的答案都不同。

Obviously, I can't answer that question for us, but I'm sure that for everyone there's a different answer to that question.

不靠规模或数据，仅靠算法改进，我们能获得多少个数量级的提升？很难回答，但有很多还是很少？

How many orders of magnitude improvement can we get just not from scale or not from data but just from algorithmic improvement? Hard to answer, but is there some a lot or some a little?

我的意思是，只有一种方法可以知道。

I mean, it's only one way to find out.

让我听听你对这些不同研究方向的快速看法。检索式 Transformer——以某种方式将数据存储在模型外部并检索——似乎很有前景。你认为这是一条前进的道路吗？

Let me get your quick-fire opinions about these different research directions. Retrieval Transformers—just somehow storing the data outside of the model itself and retrieving it somehow—seems promising. But do you see that as a path forward?

我认为它看起来很有前景。

I think it seems promising.

机器人技术——OpenAI 放弃它是正确的步骤吗？

Robotics—was it the right step for OpenAI to leave that behind?

是的，当时继续研究机器人技术确实不太可能，因为数据太少。那时，如果你想研究机器人，你需要成为一家机器人公司。你需要有一大群人致力于制造和维护机器人。即使如此，如果你只有 100 个机器人，那已经是一个巨大的运营，但你不会得到那么多数据。所以在一个大部分进步来自算力和数据结合的世界里——我们一直处于这种状态，是算力和数据的结合推动了进步——从机器人技术中获取数据没有路径。所以当时做出了停止研究的决定。

Yeah, it was like back then it really wasn't possible to continue working in robotics because there was so little data. Back then, if you wanted to work on robotics, you needed to become a robotics company. You needed to really have a giant group of people working on building robots and maintaining them. And even then, if you're only going to have 100 robots, it's a giant operation already, but you're not going to get that much data. So in a world where most of the progress comes from the combination of compute and data—that's where we've been, where it was the combination of compute and data that drove the progress—there was no path to data from robotics. So back in the day, that decision was made to stop working on it.

在机器人领域，以前没有前进的道路。现在有了吗？

In robotics there was no path forward. Is there one now?

所以我认为现在有可能开辟一条前进的道路，但需要真正致力于机器人技术。你必须说，‘我要制造成千上万、甚至数十万台机器人，并以某种方式从它们那里收集数据，找到一条渐进的道路，让机器人做稍微更有用的事情，然后获得的数据用于训练模型，让它们做稍微更有用的事情。’你可以想象这种渐进改进的路径：你制造更多机器人，它们做更多事情，你收集更多数据，等等。但你真的需要致力于这条道路。如果你说‘我想让机器人成为现实’，这就是你需要做的。我相信有些公司正在考虑这样做，但我认为你需要真正热爱机器人，并愿意解决所有物理和后勤问题。这与软件完全不同。所以我认为，只要有足够的动力，今天就可以在机器人领域取得进展。

So I'd say that now it is possible to create a path forward, but one needs to really commit to the task of robotics. You really need to say, 'I'm going to build many thousands, tens of thousands, hundreds of thousands of robots and somehow collect data from them and find a gradual path where the robots are doing something slightly more useful, and then the data that is obtained and used to train the models, they do something slightly more useful.' So you could imagine this kind of gradual path of improvement: you build more robots, they do more things, you collect more data, and so on. But you really need to be committed to this path. If you say 'I want to make robotics happen,' that's what you need to do. I believe that there are companies who are thinking about doing exactly that, but I think that you need to really love robots and need to be really willing to solve all the physical and logistical problems of dealing with them. It's not the same as software at all. So I think one could make progress in robotics today with enough motivation.

有什么想法你很兴奋想尝试，但因为当前硬件运行不佳而无法实现？

What ideas are you excited to try but you can't because they don't work well on current hardware?

我不认为当前硬件是限制。好吗？我认为事实并非如此。明白吗？所以任何你想尝试的东西，你都可以直接启动。当然，你可能会说，‘嗯，我希望当前硬件更便宜，或者内存更高，处理器带宽更大，等等。’但总的来说，硬件根本不是限制。

I don't think current hardware is a limitation. Okay? I think it's just not the case. Got it? So anything you want to try, you can just spin it up. Or I mean, of course, like this, the thing you might say, 'Well, I wish current hardware was cheaper, or maybe it had higher memory, processor bandwidth, let's say.' But by and large, hardware is just not a limitation.

我们来谈谈对齐。你认为我们会有对齐的数学定义吗？

Let's talk about alignment. Do you think we'll ever have a mathematical definition of alignment?

数学定义，我认为不太可能。嗯。我确实认为我们会有多个定义，从不同方面看待对齐，我认为这就是我们获得所需保证的方式。我的意思是，你可以观察行为，在各种测试中观察行为，在各种对抗性压力情境中观察行为，你可以从内部观察神经网络如何运作。我认为你必须同时考虑所有这些因素。

Mathematical definition, I think is unlikely. Uh-huh. Like, I do think that we will instead have multiple definitions that look at alignment from different aspects, and I think that this is how we will get the assurance that we want. By which I mean, you can look at the behavior, you can look at the behavior in various test, in various adversarial stress situations, you can look at how the neural net operates from the inside. I think you have to look at all several of these factors at the same time.

在发布模型之前，你需要有多确定？是 100%，95%？

And how sure do you have to be before you release a model? Is it 100%, 95%?

嗯，这取决于模型的能力。模型能力越强，你需要越自信。好吗？所以假设它几乎是 AGI。AGI 在哪里？嗯，取决于你的 AGI 能做什么。请记住，AGI 是一个模糊的术语。就像普通的大学生就是 AGI，对吧？是的。但你明白我的意思吗？AGI 的含义有很大的变异性，所以根据你设定的标准，你需要或多或少地自信。

Well, it depends how capable the model is. The more capable the model is, the more confident you need to be. Okay? So just say it's something that's almost AGI. Where is AGI? Well, depends what your AGI can do. Keep in mind, AGI is an ambiguous term. Like your average college undergrad is an AGI, right? It's, yeah. But you see what I mean? There's significant variability in terms of what is meant by AGI, and so depending on where you put this mark, you need to be more or less confident.

你之前提到了几条通往对齐的路径。你认为目前哪一条最有前景？

You mentioned a few of the paths towards alignment earlier. What is the one you think is most promising at this point?

我认为这将是一个组合。我真的认为你不会只想用一种方法。我认为人们想要组合多种方法，我们花费大量算力进行对抗性测试，以发现你想要教授的行为与模型表现出的行为之间的任何不匹配。我们使用另一个神经网络从内部观察神经网络，以了解其内部运作。我认为所有这些方法都是必要的。每种方法都降低了未对齐的概率，并且你还希望处于一个对齐程度比模型能力增长更快的世界。我想说，目前我们对模型的理解仍然相当初级。我们取得了一些进展，但还有更多进展可能。因此，我预计最终真正成功的是，当我们有一个被充分理解的小型神经网络，被赋予研究一个未被理解的大型神经网络行为的任务，以进行验证。

I think that it will be a combination. I really think that you will not want to have just one approach. I think people want to have a combination of approaches where we spend a lot of compute adversarially to find any mismatch between the behavior that you wanted to teach and the behavior that it exhibits. We look inside into the neural net using another neural net to understand how it operates on the inside. I think all of them will be necessary. Every approach like this reduces the probability of misalignment, and you also want to be in a world where your degree of alignment keeps increasing faster than the capability of the models. I would say that right now our understanding of our models is still quite rudimentary. We made some progress, but much more progress is possible. And so I would expect that ultimately the thing that will really succeed is when we will have a small neural net that is well understood that's given the task to study the behavior of a large neural net that is not understood, to verify.

到什么时候大部分研究将由 AI 完成？我的意思是，今天当你使用 Copilot 时，对吧？你如何划分？

By what point is most of the research being done by AI? I mean, so today when you use Copilot, right? How do you divide it up?

所以我预计在某个时候，你问你的，你知道，ChatGPT 的后代，你说，‘嘿，我在想这个和那个，你能建议一些我应该尝试的有成果的想法吗？’然后你实际上会得到有成果的想法，对吧？我认为这将使你能够解决以前能解决的问题。明白吗？但它只是告诉人类，给他们想法，更快的东西。它本身并没有与世界互动。我的意思是，你可以用各种方式划分，但我认为瓶颈在于好的想法、好的见解，而这正是神经网络可以帮忙的。

So I expect at some point you ask your, you know, descendant of ChatGPT, you say, 'Hey, I'm thinking about this and this, can you suggest fruitful ideas I should try?' And you would actually get fruitful ideas, right? And I think that will make it possible for you to solve problems you could solve before. Got it? But it's somehow just telling the humans, giving them ideas, a faster something. It's not itself interacting with the world. I mean, you could slice it in a variety of ways, but I think the bottleneck there is good ideas, good insights, and that's something which the neural net could help with.

如果你能设计一个十亿美元的奖项，奖励某种对齐研究结果或产品，那么为这个十亿美元奖项设定的具体标准是什么？对这样一个奖项有意义的东西。

If you could design a billion-dollar prize for some sort of alignment research result or product, what is like the concrete criterion set for that billion-dollar prize? Something that makes sense for such a prize.

你问这个问题很有趣。我实际上也在想这个确切的问题。我还没有想出确切的标准。也许是一些有益的东西，也许是一个奖项，我们可以说两年后、三年后或五年后，回顾起来说，‘那是主要成果。’所以与其说有一个奖项委员会立即决定，我们等五年然后追溯颁发。但目前还没有我们能识别的具体东西，比如‘你解决了这个特定问题，你取得了很大进展。’我认为很大进展，是的。我不会说这就是全部。

It's funny you asked this. I was actually thinking about this exact question. I haven't come up with an exact criteria yet. Maybe something that would be the benefit, maybe a prize where we could say that two years later, or three years later, or five years later, we'll look back and say, 'That was the main result.' So rather than say that there is a prize committee that decides right away, we wait for five years and then award it retroactively. But there's no concrete thing we can identify yet, like 'You solve this particular problem and you made a lot of progress.' I think a lot of progress, yes. I wouldn't say that this would be the full thing.

你认为端到端训练是越来越大模型的正确架构，还是我们需要更好的方式将事物连接在一起？

Do you think end-to-end training is the right architecture for bigger and bigger models, or do we need better ways of just connecting things together?

我认为端到端训练非常有前景。我认为将事物连接在一起非常有前景。一切都有前景。

I think end-to-end training is very promising. I think connecting things together is very promising. Everything is promising.

OpenAI 预计 2024 年收入为 10 亿美元。这很可能是正确的，但我很好奇：当谈到一种新的通用技术时，你如何估计它会带来多大的意外之财？比如，为什么是那个特定的数字？

OpenAI is projecting revenues of a billion dollars in 2024. That might very well be correct, but I'm just curious: when you're talking about a new general-purpose technology, how do you estimate how big a windfall it will be? Like, why that particular number?

我的意思是，你看看当前，你看看，你知道，我们已经有一个产品相当一段时间了，从两年前的 GPT-3 时代通过 API，我们看到了它的增长。我们也看到了对 DALL-E 的反应如何增长，所以你看到了对 ChatGPT 的反应。我认为所有这些都为我们提供了信息，使我们能够对 2024 年做出相对合理的推断。也许这是一个答案。就像，你需要有数据；你不能凭空想出这些东西，否则你的误差范围会偏离，你的误差范围在每个方向都会是 100 倍。我的意思是，但大多数……

I mean, you look at the current, you look at the, you know, we've already had a product for quite a while now, from the GPT-3 days from two years ago through the API, and we've seen how it grew. We've seen how the response to DALL-E has grown as well, and so you see how the response to ChatGPT is. And I think all of this gives us information that allows us to make a relatively sensible extrapolation to 2024. Maybe that would be one answer. Like, you need to have data; you can't come up with those things out of thin air, because otherwise your error bars will be off by, your error bars are going to be like 100x in each direction. I mean, but most...

指数增长不会一直持续，尤其是当规模越来越大时，对吧？那么在这种情况下，你怎么判断呢？我是说，你会押注它们会失败吗？和你聊过之后不会了。我们来谈谈 AI 之后的未来是什么样子。像你这样的人——我猜你每周工作 80 小时，朝着某个你痴迷的伟大目标努力——在一个你基本上住在 AI 养老院的世界里，你会满足吗？或者 AI 到来后你具体会做什么？

Exponentials don't stay exponential, especially when they get into bigger and bigger quantities, right? So how do you determine in this case that... I mean, would you bet against them? Not after talking with you. Let's talk about what a post-AI future looks like. So are people like you—I'm guessing you're working like 80-hour weeks towards some grand goal that you're really obsessed with—are you going to be satisfied in a world where you're basically living in an AI retirement home? Or what are you concretely doing after AI comes?

我认为 AI 到来后我会做什么，或者人们会做什么，这是一个非常棘手的问题。人们从哪里找到意义？但我认为这是 AI 可以帮助我们的。我想象的一件事是，我们都能变得更加开悟，因为我们会与一个 AGI 互动，它帮助我们更正确地看待世界，通过互动让我们内心变得更好。想象一下与历史上最好的冥想老师交谈。我认为那会很有帮助。但我也认为，因为世界会变化很大，人们很难准确理解正在发生什么以及如何真正做出贡献。我认为有些人会选择成为半 AI，以真正扩展他们的心智和理解，从而能够解决社会将面临的最困难的问题。

I think the question of what I'll be doing or what people will be doing after AI comes is a very tricky question. Where will people find meaning? But I think that's something that AI could help us with. One thing I imagine is that we'll all be able to become more enlightened because we'll interact with an AGI that will help us see the world more correctly, become better on the inside as a result of interaction. Imagine talking to the best meditation teacher in history. I think that will be a helpful thing. But I also think that because the world will change a lot, it will be very hard for people to understand what is happening precisely and how to really contribute. One thing that I think some people will choose to do is to become part AI in order to really expand their minds and understanding, to really be able to solve the hardest problems that society will face.

你会成为半 AI 吗？

Are you going to become part AI?

这非常诱人。确实诱人，是的。

It is very tempting. It is tempting, yeah.

你怎么看？3000 年还会有实体人类吗？我怎么知道 3000 年会发生什么？那会是什么样子？地球上还有人类走来走去吗？或者你们有没有具体想过你们希望这个世界在 3000 年变成什么样？

What do you think? Will there be physically embodied humans in 3000? How do I know what's going to happen in 3000? What does it look like? Are there still humans walking around on Earth? Or have you guys thought concretely about what you actually want this world to look like in 3000?

嗯，让我描述一下我认为这个问题不太对的地方。它暗示我们可以决定我们希望世界变成什么样。我不认为这个图景是正确的。我认为变化是唯一不变的，所以当然即使在 AGI 建成之后，世界也不会静止。世界会继续变化、演化，经历各种转型。我真的不知道——我认为没有人知道 3000 年世界会是什么样子。但我确实希望有很多人类的后代能够过上幸福、充实的生活，他们可以自由地做自己想做的事，或者他们自己解决自己的问题。有一件事我不希望——一个我觉得非常无趣的世界——就是我们建造了这个强大的工具，然后政府说：‘好吧，AGI 说社会应该这样运行，现在我们应该这样运行社会。’我更喜欢一个世界，人们仍然可以自由地犯自己的错误，承受后果，并逐渐在道德上进化，通过自己的力量前进，而 AGI 更像是提供一个基本的安全网。

Well, let me describe to you what I think is not quite right about the question. It implies that we get to decide how we want the world to look like. I don't think that picture is correct. I think change is the only constant, and so of course even after AGI is built, it doesn't mean that the world will be static. The world will continue to change, evolve, and go through all kinds of transformations. I really have no—I don't think anyone has any idea of how the world will look like in 3000. But I do hope that there will be a lot of descendants of human beings who will live happy, fulfilled lives where they are free to do as they wish, or they are the ones solving their own problems. One thing I would not want—one world which I would find very unexciting—is one where we build this powerful tool and then the government says, 'Okay, the AGI said that society should be run in such a way, and now we should run society in such a way.' I'd much rather have a world where people are still free to make their own mistakes, suffer their consequences, and gradually evolve morally and progress forward on their own through their own strength, with the AGI providing more like a base safety net.

你花多少时间思考这类事情，而不是只做研究？

How much time do you spend thinking about these kinds of things versus just doing the research?

我确实经常思考这些事情。是的，我认为这些都是非常有趣的问题。

I do think about those things a fair bit. Yeah, I think those are very interesting questions.

我们今天拥有的能力在哪些方面超出了你在 2015 年的预期？在哪些方面仍然没有达到你当时的预期？

In what ways have the capabilities we have today surpassed where you expected them to be in 2015? And in what ways are they still not where you would have expected them to be by this point?

2015 年，我的想法更多的是……我只是不想押注深度学习会失败。我想对深度学习下最大的赌注。我不知道具体怎么实现，但它会自己搞定的。

In 2015, my thinking was a lot more... I just don't want to bet against deep learning. I want to make the biggest possible bet on deep learning. I don't know how, but it will figure it out.

有没有什么具体方面超出了你的预期或低于你的预期？比如你在 2015 年做的某个具体预测已经实现了？

Is there any specific way in which it's been more than you expected or less than you expected? Like some concrete prediction you had in 2015 that's been anointed?

不幸的是，我不记得我在 2015 年做的具体预测了。但我确实认为，总的来说，在 2015 年，我只是想对深度学习下最大的赌注。我没有具体想法七年内事情会发展到什么程度。在 2015 年，我确实和人们在 2016 年、也许 2017 年打赌，说事情会发展得很远。但具体细节……它既让我惊讶，我也在做这些激进的预测，但我想也许我内心只有 50%相信它们。

Unfortunately, I don't remember concrete predictions I made in 2015. But I definitely think that overall in 2015, I just wanted to make the biggest bet possible on deep learning. I didn't have a specific idea of how far things would go in seven years. In 2015, I did have all these bets with people in 2016, maybe 2017, that things would go really far. But specifics... it's both the case that it surprised me and I was making these aggressive predictions, but I think maybe I believed them only 50% on the inside.

你现在相信什么，甚至 OpenAI 的大多数人都会觉得牵强？

What do you believe now that even most people at OpenAI would find far-fetched?

我认为在这一点上，因为我们交流很多，OpenAI 的人对我的想法有很好的了解。所以是的，我们在 OpenAI 达到了一个点，我认为我们在所有这些问题上看法一致。

I think that at this point, because we communicate a lot, OpenAI people have a pretty good sense of what I think. So yeah, we reached the point at OpenAI where I think we see eye to eye on all these questions.

谷歌有它的定制 TPU 硬件，它有来自所有用户的数据，比如 Gmail 等等。这是否让它在训练更大更好的模型方面比你们有优势？

Google has its custom TPU hardware, it has all this data from all its users, Gmail, and so on. Does it give it an advantage in terms of training bigger models and better models than you?

我认为当 TPU 刚出来时，我真的很印象深刻，心想：‘哇，这太棒了。’但那是因为我当时不太了解硬件。实际情况是，TPU 和 GPU 几乎是同一回事。它们非常非常相似。GPU 芯片稍大一点，TPU 芯片稍小一点，可能更便宜一点，但然后他们生产的 GPU 比 TPU 多，所以我认为 GPU 可能反而更便宜。但根本上，你有一个大处理器和大量内存，两者之间存在瓶颈连接。TPU 和 GPU 都在试图解决的问题是，将浮点数从内存移动到处理器所需的时间内，你可以在处理器上执行几百次浮点运算，这意味着你必须进行某种批处理。在这个意义上，两种架构是一样的。所以我觉得硬件，在某种意义上，硬件唯一重要的是每浮点运算的成本，整体系统成本。没有太大区别。实际上，我不知道 TPU 的成本是多少，但我怀疑可能 GPU 更贵，因为它们的数量更少。

I think when first the TPU came out, I was really impressed and thought, 'Wow, this is amazing.' But that's because I didn't quite understand hardware back then. What really turned out to be the case is that TPUs and GPUs are almost the same thing. They are very, very similar. A GPU chip is a little bit bigger, a TPU chip is a little bit smaller, it may be a little bit cheaper, but then they make more GPUs than TPUs, so I think the GPUs might be cheaper after all. But fundamentally, you have a big processor and a lot of memory, and there is a bottleneck link between those two. The problem that both the TPU and the GPU are trying to solve is that by the time it takes to move one floating point from memory to the processor, you can do several hundred floating point operations on the processor, which means you have to do some kind of batch processing. In this sense, both architectures are the same. So I really feel like hardware, in some sense, the only thing that matters about hardware is cost per flop, overall system cost. There isn't that much difference. Actually, I don't know how much the TPU costs, but I would suspect that probably GPUs are more expensive because there are fewer of them.

当你做工作时，有多少时间花在配置正确的初始化、确保训练运行顺利和找到正确的超参数上？又有多少时间只是提出全新的想法？

When you're doing your work, how much of the time is spent configuring the right initializations, making sure the training run goes well, and getting the right hyperparameters? And how much is it just coming up with whole new ideas?

我会说这是两者的结合。但提出全新的想法实际上只是工作中很小的一部分。当然，提出……

I would say it's a combination. But coming up with whole new ideas is actually a modest part of the work. Certainly coming up with...

新想法很重要，但我认为更重要的是理解结果、理解现有想法、理解正在发生的事情。因为通常你有一个非常复杂的系统，运行它，然后得到一些难以理解的行为。理解结果、弄清楚下一步该做什么实验——很多时间都花在这上面。理解可能哪里出错了，是什么导致神经网络产生了意想不到的结果。我们确实也花很多时间想新点子，但我不太喜欢这种说法。不是说它不对，但我认为主要活动其实是理解。

New ideas are important, but I think even more important is to understand the results, to understand the existing ideas, to understand what's going on. Because normally you have this very complicated system, you run it, and you get some behavior which is hard to understand. Understanding the results, figuring out what the next experiment to run—a lot of the time is spent on that. Understanding what could be wrong, what could have caused the neural net to produce a result which was not expected. I'd say a lot of time we also spend coming up with new ideas, but I don't like this framing as much. It's not that it's false, but I think the main activity is actually understanding.

你觉得两者有什么区别？

What do you see as the difference between the two?

至少在我看来，当你说‘想出新点子’时，我会想‘哦，如果它这样那样会怎样’，而理解更像是‘这整个东西是什么？真正的基本现象是什么？潜在的影响是什么？为什么我们这样做而不是那样做？’当然，这跟‘想点子’很接近，但我认为真正的行动在于理解。

At least in my mind, when you say 'come up with new ideas,' I think 'oh, what if it did such and such,' whereas understanding is more like 'what is this whole thing? What are the real underlying phenomena going on? What are the underlying effects? Why are we doing things this way and not another way?' And of course this is very adjacent to what can be described as coming up with ideas, but I think the understanding part is where the real action takes place.

这能描述你的整个职业生涯吗？比如回想一下 ImageNet 之类的，那更多是一个新想法还是更多是理解？

Does that describe your entire career? Like if you think back on ImageNet or something, was that more a new idea or was that more understanding?

哦，那绝对是理解。那是对非常古老的事物的一种新理解。

Oh, it was definitely understanding. It was a new understanding of very old things.

在 Azure 上训练的经历如何？

What has the experience of training on Azure been like?

用 Azure？太棒了。我的意思是，微软一直是我们非常好的合作伙伴，他们确实帮助 Azure 达到了非常适合机器学习的状态。我们对此非常满意。

Using Azure? Fantastic. I mean, Microsoft has been a very, very good partner for us, and they've really helped take Azure and bring it to a point where it's really good for ML. We're super happy with it.

整个 AI 生态系统对台湾可能发生的事情有多脆弱？比如说台湾发生了海啸之类的。AI 整体上会怎么样？

How vulnerable is the whole AI ecosystem to something that might happen in Taiwan? So let's say there's a tsunami in Taiwan or something. What would happen to AI in general?

这绝对会是一个重大挫折。可能相当于几年内没人能获得更多算力。但我预计算力会涌现出来。例如，我相信英特尔有几代前的晶圆厂。这意味着如果英特尔愿意，他们可以生产类似四年前 GPU 的东西。所以，虽然不是最好的。我其实不确定关于英特尔的说法是否正确，但我确实知道台湾以外有晶圆厂。它们只是没那么好，但你仍然可以用它们，并且仍然可以走得很远。这只是一个挫折。

It's definitely going to be a significant setback. It might be something equivalent to no one being able to get more compute for a few years. But I expect compute will spring up. For example, I believe that Intel has fabs from a few generations ago. So that means if Intel wanted to, they could produce something GPU-like from four years ago. So yeah, it's not the best. I'm actually not sure if my statement about Intel is correct, but I do know that there are fabs outside of Taiwan. They're just not as good, but you can still use them and still go very far with them. It's just a setback.

嗯，随着这些模型越来越大，推理可能会变得成本过高。

Well, inference could become cost prohibitive as these models get bigger and bigger.

我对这个问题有不同的看法。并不是说推理会变得成本过高。更好的模型的推理确实会更贵，但算不算过高？这取决于它有多有用。如果它更有用，那么即使贵，也不算过高。打个比方：假设你想和律师谈谈，你有案子或需要建议。你完全愿意每小时花 500 美元，对吧？所以如果你的神经网络能给你非常可靠的法律建议，你会说‘我愿意花 400 美元买这个建议’。突然之间，推理就变得非常不贵了。问题是：神经网络能否以这个成本给出足够好的答案？能。而且你会有价格歧视——不同模型用于不同用例。今天已经是这样了。在我们的产品 API 中，我们提供多种不同规模的神经网络，不同客户根据他们的用例使用不同的神经网络。如果有人能用一个小模型微调得到满意的结果，他们就会用那个。但如果有人想做更复杂、更有趣的事情，他们会用最大的模型。

I have a different way of looking at this question. It's not that inference will become cost prohibitive. Inference of better models will indeed become more expensive, but is it prohibitive? Well, it depends on how useful it is. If it is more useful, then even if it is expensive, it is not prohibitive. To give you an analogy: suppose you want to talk to a lawyer, you have some case or need some advice. You are perfectly happy to spend $500 an hour, right? So if your neural net could give you really reliable legal advice, you'd say 'I'm happy to spend $400 for that advice.' And suddenly inference becomes very much non-prohibitive. The question is: can a neural net produce an answer good enough at this cost? Yes. And you'll just have price discrimination—different models for different use cases. It's already the case today. On our product, the API, we serve multiple neural nets of different sizes, and different customers use different neural nets depending on their use case. If someone can take a small model and fine-tune it and get something satisfactory, they'll use that. But if someone wants to do something more complicated and interesting, they'll use the biggest model.

你怎么防止这些模型变成商品，不同公司互相压价，直到基本上就是 GPU 运行的成本？

How do you prevent these models from just becoming commodities, where these different companies just undercut each other's prices until it's basically the cost of the GPU run?

我认为毫无疑问有一种力量在试图制造这种情况。答案是你要不断进步。你要不断改进模型。你要不断提出新想法，让我们的模型更好、更可靠、更值得信赖，这样你才能相信它们的答案。所有这些。

I think there is without question a force trying to create that. And the answer is you've got to keep on making progress. You've got to keep improving the models. You've got to keep on coming up with new ideas and making our models better, more reliable, more trustworthy, so you can trust their answers. All those things.

但假设现在是 2025 年，2024 年的模型以成本价提供，而且仍然相当不错。如果仅仅一年前的模型甚至更好，人们为什么要用 2025 年的新模型？

But let's say it's 2025 and the model from 2024 is being offered at cost, and it's still pretty good. Why would people use a new one from 2025 if the one from just a year older is even better?

有几个答案。对于某些用例，这可能是真的。2025 年会有新模型驱动更有趣的用例。还有一个推理成本的问题。你可以研究如何以更低的成本提供相同的模型，所以不同公司会以不同的成本提供相同的模型。我还可以想象一定程度的专业化，一些公司可能试图在某个领域专业化，并比其他公司更强。我认为这在一定程度上可能是对商品化的回应。

There are several answers there. For some use cases, that may be true. There will be a new model from 2025 which will be driving the more interesting use cases. There's also going to be a question of inference cost. You can do research to serve the same model at less cost, so different companies will serve the same model at different costs. I can also imagine some degree of specialization, where some companies may try to specialize in some area and be stronger in that area compared to other companies. And I think that may be a response to commoditization to some degree.

随着时间的推移，这些不同公司的研究方向是趋同还是分化？它们是在做类似的事情，还是分支到不同的领域？

Over time, do these different companies' research directions converge or diverge? Are they doing similar things over time, or are they branching off into different areas?

我会说在短期内，看起来是趋同的。我预计这会是趋同-分化-趋同的行为。短期工作有很多趋同。长期会有一些分化，但一旦长期工作开始结果，我认为会再次趋同。

I'd say in the near term, it looks like there is convergence. I expect this going to be a convergence-divergence-convergence behavior. There is a lot of convergence on the near-term work. There's going to be some divergence on the longer term, but once the longer-term work starts to fruit, I think there will be convergence again.

明白了。当其中一个找到了最有前景的领域，大家就都往那里去？

Got it. When one of them finds the most promising area, everybody just goes there?

没错。现在显然发表得少了，所以这个有前景的方向被重新发现需要更长时间，但我是这么想象的。我认为会是趋同、分化、趋同。

That's right. Now there is obviously less publishing now, so it will take longer before this promising direction gets rediscovered, but that's how I imagine it. I think it's going to be convergence, divergence, convergence.

我们一开始谈过一点，但随着外国政府了解到这些模型有多强大，你担心间谍或某种攻击来获取你的权重，或者以某种方式滥用这些模型并了解它们吗？

We talked about this a little bit at the beginning, but as foreign governments learn about how capable these models are, are you worried about spies or some sort of attack to get your weights, or somehow abuse these models and learn about them?

是的，这绝对是绝对不能忽视的事情。我们尽最大努力防范。但这将是每个构建这些东西的人都会面临的问题。

Yeah, it's definitely something that you absolutely can't discount. And it's something that we guard against to the best of our ability. But it's going to be a problem for everyone who's building this.

你怎么防止权重泄露？我的意思是，你有非常优秀的安全人员。有多少人？

How do you prevent your weights from leaking? I mean, you have really good security people. How many people?

我不能透露细节，但我们有一个强大的安全团队，我们非常重视这件事。

I can't go into details, but we have a strong security team and we take it very seriously.

如果他们想直接窃取权重，有多少人能做到？

If they wanted to just stage into the weights, how many people could do that?

我能说的是，安全团队做得非常好，所以我真的不担心权重泄露。

What I can say is that the security people have done a really good job, so I'm really not worried about the weights being leaked.

你期望这些模型在如此规模下会出现哪些涌现特性？会有全新出现的东西吗？

What kinds of emerging properties are you expecting from these models at this scale? Is there something that just comes about de novo?

我确信会有新特性出现，真正令人惊讶的新特性。我不会感到意外。我真正兴奋、或者说希望看到的是可靠性和可控性。我认为这将是非常重要的一类涌现特性。如果有了可靠性和可控性，就能解决很多问题。可靠性意味着你可以信任模型的输出；可控性意味着你可以控制它。我们拭目以待，但如果这些涌现特性真的存在，那就太棒了。

I'm sure things will come. I'm sure really new surprising properties will come up. I would not be surprised. The thing I'm really excited about, or the thing I'd like to see, is reliability and controllability. I think that will be a very important class of emergent properties. If you have reliability and controllability, that helps you solve a lot of problems. Reliability means you can trust the model's output; controllability means you can control it. We'll see, but it'll be very cool if those emergent properties did exist.

有没有办法提前预测，比如在这个参数量下会发生什么，在那个参数量下会发生什么？

Is there some way you can predict it in advance, like what will happen at this parameter count, what will happen at that?

我认为对特定能力做出一些预测是可能的，尽管这绝对不简单，而且至少在今天，你无法以非常精细的方式做到。但我认为在这方面做得更好非常重要，任何有兴趣并有研究想法的人都可以做出有价值的贡献。

I think it's possible to make some predictions about specific capabilities, though it's definitely not simple and you can't do it in a super fine-grained way, at least today. But I think getting better at that is really important, and anyone who is interested and has research ideas on how to do that can make a valuable contribution.

你有多认真对待缩放定律？如果有论文说需要增加这么多数量级才能获得所有推理能力，你会认真对待吗，还是认为它会在某个点失效？

How seriously do you take the scaling laws? If there's a paper that says you need this many orders of magnitude more to get all the reasoning out, do you take that seriously, or do you think it breaks down at some point?

缩放定律告诉你的是下一个词预测准确率的变化。将下一个词预测准确率与推理能力联系起来是一个完全不同的挑战。我确实相信存在联系，但这种联系很复杂。我们可能会发现其他方法可以在单位努力下获得更多推理能力，比如一些特殊的推理词元。我认为它们会有帮助。

The scaling law tells you what happens to your next-token prediction accuracy. There is a whole separate challenge of linking next-token prediction accuracy to reasoning capability. I do believe there is a link, but this link is complicated. We may find that there are other things that can give us more reasoning per unit effort, like for example some special reasoning tokens. I think they can be helpful.

你是在考虑雇佣人类为你生成词元，还是全部来自已经存在的数据？

Is this something you're considering, just hiring humans to generate tokens for you, or is it all going to come from stuff that already exists out there?

我认为依靠人类来教我们的模型做事，特别是确保它们行为良好、不产生虚假信息，是非常明智的做法。

I think relying on people to teach our models to do things, especially to make sure they are well-behaved and don't produce false things, is an extremely sensible thing to do.

我们恰好同时拥有所需的数据、Transformer 和这些 GPU，这难道不奇怪吗？你觉得所有这些事情同时发生很奇怪，还是你不这么认为？

Isn't it odd that we have the data we need at exactly the same time as we have the Transformer, at the exact same time that we have these GPUs? Is it odd to you that all these things happened at the same time, or do you not see it that way?

这是一个有趣的情况。我会说这很奇怪，但在某种程度上又不那么奇怪。原因如下：数据、GPU 和 Transformer 存在的驱动力是相互交织的。数据的存在是因为计算机变得更好更便宜；晶体管越来越小，每个人拥有个人电脑变得经济可行。一旦每个人都有个人电脑，你就想连接到网络，于是有了互联网。有了互联网，突然就出现了大量数据。GPU 也在同时改进，因为晶体管越来越小，人们需要找事情做；游戏成了其中一件事。然后某个时刻，英伟达说：‘等等，我们把它变成通用 GPU，也许有人会发现它有用。’结果它对神经网络很好。如果游戏不存在，GPU 可能会晚 5 年或 10 年出现。但很难想象没有游戏的世界。可能存在一个反事实的世界，GPU 比数据晚 5 年或早 5 年出现，那样事情可能就不会像现在这样准备就绪。但这就是图景：所有这些维度的进步都是紧密交织的。这不是巧合；你不能挑选哪些维度会进步。

It is an interesting situation. I will say that it is odd, but it is less odd on some level. Here is why: the driving force behind the fact that the data exists, the GPUs exist, and the Transformer exists is all interconnected. Data exists because computers became better and cheaper; we got smaller transistors, and it became economical for every person to have a personal computer. Once everyone has a personal computer, you want to connect to the network, you get the internet. Once you have the internet, you suddenly have data appearing in great quantities. GPUs were improving concurrently because of smaller transistors, and you're looking for things to do with them; gaming turned out to be a thing. Then at some point, Nvidia said, 'Wait a second, let's turn it into a general-purpose GPU; maybe someone will find it useful.' Turns out it's good for neural nets. It could have been that the GPU arrived 5 or 10 years later if gaming wasn't a thing. But it's hard to imagine a world without gaming. There could be a counterfactual world where GPUs arrived 5 years after the data or 5 years before, in which case things might not have been as ready to go. But that's the picture: all this progress in all these dimensions is very intertwined. It's not a coincidence; you don't get to pick and choose which dimensions improve.

这种进步有多不可避免？如果你和杰弗里·辛顿以及其他几位先驱从未出生，深度学习革命还会在同一时间发生吗？会延迟多久？

How inevitable is this kind of progress? If you and Geoffrey Hinton and a few other pioneers were never born, does the deep learning revolution happen around the same time? How much does it delay?

我认为可能会有一些延迟，大概一年左右。很难说。我不太想给出更长的回答，因为 GPU 会不断改进。在某个时刻，我无法想象有人不会发现它。如果没有人做，计算机会越来越快、越来越好，训练神经网络会变得更容易，因为你有更大的 GPU，所以需要的工程努力更少。你不需要过多优化代码。当 ImageNet 出现时，它很大且很难使用。现在想象一下，你等几年，它变得很容易下载，人们可以随便摆弄。所以我猜测最多几年。不过我不太想给出更长的回答；你无法重演世界，你不知道。

I think maybe there would have been some delay, maybe like a year delay. It's really hard to tell. I hesitate to give a longer answer because GPUs would keep on improving. At some point, I cannot see how someone would not have discovered it. If no one had done it, computers keep getting faster and better, it becomes easier to train neural nets because you have bigger GPUs, so it takes less engineering effort. You don't need to optimize your code as much. When ImageNet came out, it was huge and very difficult to use. Now imagine you wait a few years and it becomes very easy to download and people can just tinker. So I would imagine a modest number of years maximum. I hesitate to give a longer answer though; you can't rerun the world, you don't know.

让我们回到对齐问题。作为一个深刻理解这些模型的人，你直觉上认为对齐会有多难？

Let's go back to alignment for a second. As somebody who deeply understands these models, what is your intuition of how hard alignment will be?

在当前能力水平下，我认为我们有一套相当好的思路来对齐它们。但我不会低估对齐比我们更聪明的模型的难度，这些模型能够歪曲自己的意图。我认为这是一个需要大量思考和研究的领域。这是学术研究人员可以做出非常有意义贡献的一个领域。

With the current level of capabilities, I think we have a pretty good set of ideas of how to align them. But I would not underestimate the difficulty of alignment of models that are actually smarter than us, of models that are capable of misrepresenting their intentions. I think it's something to think about a lot and to research. This is one area where academic researchers can make very meaningful contributions.

你认为学术界会提出关于实际能力的重要见解，还是说在这一点上只会是公司？

Do you think academia will come up with important insights about actual capabilities, or is that going to be just the companies at this point?

公司会实现这些能力。我认为学术研究也很有可能提出这些见解。

The companies will realize the capabilities. I think it's very possible for academic research to come up with those insights as well.

只是出于某种原因，这种情况似乎不太常发生。但我不认为学术界有什么根本性的问题。并不是说学术界做不到。我想可能他们只是没有在思考正确的问题，因为也许在公司内部更容易看清需要做什么。

It just doesn't seem to happen that much for some reason. But I don't think there's anything fundamental about academia. It's not like academia can't. I think maybe they're just not thinking about the right problems or something, because maybe it's just easier to see what needs to be done inside these companies.

我明白了。但有可能有人会意识到，‘是的，我完全——比如，我为什么要排除这种可能性？’你明白我的意思吗？这些语言模型通过哪些具体步骤开始真正影响原子世界，而不仅仅是比特世界？

I see. But there's a possibility that somebody could just realize, 'Yeah, I totally—like, why would I possibly rule this out?' You see what I mean? What are the concrete steps by which these language models start actually impacting the world of atoms and not just the world of bits?

嗯，你看，我不认为比特世界和原子世界之间有明确的界限。假设神经网络告诉你，‘嘿，这是你应该做的事情，它会改善你的生活，但你需要以某种方式重新布置你的公寓。’然后你去重新布置了你的公寓。神经网络是否影响了原子世界？是的。

Well, you see, I don't think there is a clean distinction between the world of bits and the world of atoms. Suppose the neural net tells you that, 'Hey, here is something that you should do and it's going to improve your life, but you need to rearrange your apartment in a certain way.' Then you go and you rearrange your apartment as a result. Did the neural net impact the world of atoms? Yes.

有道理。有道理。你认为要达到超级智能，还需要几个像 Transformer 一样重要的额外突破，还是说我们基本上已经在某本书里得到了洞见，只需要实现并连接它们？

Fair enough. Fair enough. Do you think it'll take a couple of additional breakthroughs as important as the Transformer to get to superintelligence, or do you think we basically got the insights in the books somewhere and we just need to implement them and connect them?

所以我不太认为这两种情况之间有这么大的区别，让我解释一下原因。我认为过去取得进展的方式之一是，我们理解到某物一直拥有一个理想的属性，但你没有意识到。那么这是一个突破吗？你可以说是的。它是书本上某物的实现吗？也是的。所以我的感觉是，其中一些很可能会发生，但事后看来，它不会感觉像是一个突破。每个人都会说，‘哦，当然，某某东西能起作用是显而易见的。’你看，对于 Transformer，它被提出来作为一个重大的具体进展，是因为它是那种几乎对任何人都不明显的东西。所以人们可以说，‘是的，这不是他们知道的事情。’但如果一个进展来自像——让我们考虑深度学习最根本的进展：一个用反向传播训练的大神经网络可以做很多事情。新颖性在哪里？不在神经网络，也不在反向传播。但不知何故，它绝对是一个巨大的概念性突破，因为在很长一段时间里，人们就是没有看到这一点。但现在每个人都看到了，每个人都会说，‘当然，这完全显而易见，大神经网络——每个人都知道它们能做到。’

So I don't really see such a big distinction between those two cases, and let me explain why. I think one of the ways in which progress has taken place in the past is that we've understood that something had a desirable property all along, but you didn't realize. So is that a breakthrough? You can say yes it is. Is it an implementation of something on the books? Also yes. So my feeling is that a few of those are quite likely to happen, but that in hindsight it will not feel like a breakthrough. Everybody is going to say, 'Oh well, of course, it's totally obvious that such and such thing can work.' You see, with the Transformer, the reason it's being brought up as a big specific advance is because it's the kind of thing that was not obvious for almost anyone. So people can say, 'Yeah, it's not something which they knew about.' But if an advance comes from something like—let's consider the most fundamental advance of deep learning: that a big neural network trained with backpropagation can do a lot of things. Where's the novelty? It's not in the neural network, it's not in the backpropagation. But then somehow it is most definitely a giant conceptual breakthrough, because for the longest time people just didn't see that. But now everyone sees it, everyone's going to say, 'Well of course, it's totally obvious, big neural networks—everyone knows that they can do it.'

你对你前导师的前向-前向算法有什么看法？

What is your opinion of your former advisor's forward-forward algorithm?

我认为这是一种尝试在没有反向传播的情况下训练神经网络，我认为如果你有动力去理解大脑是如何学习其连接的，这会特别有趣。原因是，据我所知，神经科学家们确实相信大脑无法实现反向传播，因为突触中的信号只朝一个方向移动。所以如果你有神经科学的动机，你想说，‘好吧，我如何能提出一种在不进行反向传播的情况下近似反向传播良好特性的方法？’这就是前向-前向算法试图做的。但如果你只是想工程化一个好的系统，没有理由不使用反向传播。它真的是唯一的算法。

I think that it's an attempt to train a neural network without backpropagation, and I think that this is especially interesting if you are motivated to try to understand how the brain might be learning its connections. The reason for that is that, as far as I know, neuroscientists are really convinced that the brain cannot implement backpropagation because the signals in the synapses are only moving in one direction. So if you have a neuroscience motivation and you want to say, 'Okay, how can I come up with something that tries to approximate the good properties of backpropagation without doing backpropagation?' That's what the forward-forward algorithm is trying to do. But if you are trying to just engineer a good system, there is no reason to not use backpropagation. It's the only algorithm, really.

我在不同场合听你谈到过这种需要——比如把人类作为 AGI 存在的现有例子，对吧？那么你在什么时候会不再那么认真地对待这个比喻，觉得没有必要在研究上追求它？因为它对你来说作为一种存在案例很重要。比如，在什么时候我不再关心人类作为智能的存在案例，或者作为你在模型中追求智能时想要遵循的模型例子？

I've heard you in different contexts talk about the need—like using humans as the existing example case that AGI exists, right? So at what point do you take the metaphor less seriously and feel don't feel the need to pursue it in terms of research? Because it is important to you as a sort of existence case. Like, at what point do I stop caring about humans as an existence case of intelligence, or as an example of the model you want to follow in terms of pursuing intelligence in models?

我明白了。我的意思是，我认为受到人类的启发是好的。我认为受到大脑的启发是好的。我认为正确地受到人类和大脑的启发是一门艺术，因为很容易抓住人类或大脑的非本质特性。我认为许多试图从人类和大脑中寻找灵感的研究人员常常变得有点具体——人们会有点，‘好吧，那么我们应该遵循哪种认知科学模型？’同时，考虑神经网络本身的概念，人工神经元的概念。这也是受大脑启发的，但结果证明它非常富有成效。那么你如何做到这一点？人类行为的哪些本质让你说，‘这向我们证明了这是可能的’？哪些是非本质的？不，实际上这就像某种更基本事物的涌现现象，我们只需要专注于把我们的基础做对。我会说，一个人可以而且应该谨慎地受到人类智能的启发。

I see. I mean, I think it's good to be inspired by humans. I think it's good to be inspired by the brain. I think there is an art to being inspired by humans and the brain correctly, because it's very easy to latch on to a non-essential quality of humans or of the brain. And I think many people whose research is trying to be inspired by humans and by the brain often get a little bit specific—people get a little bit, 'Okay, so what cognitive science model should we follow?' At the same time, consider the idea of the neural network itself, the idea of the artificial neuron. This too is inspired by the brain, but it turned out to be extremely fruitful. So how do you do this? What behaviors of human beings are essential that you say, 'This is something that proves to us that it's possible'? What is inessential? No, actually this is like some emergent phenomenon of something more basic, and we just need to focus on getting our own basics right. I would say that one can and should be inspired by human intelligence with care.

最后一个问题：为什么在你的案例中，率先进入深度学习革命与仍然是顶级研究人员之一之间有如此强的相关性？你会认为这两件事不会有那么大的相关性，但为什么会有这种相关性？

Final question: why is there, in your case, such a strong correlation between being first to the deep learning revolution and still being one of the top researchers? You would think that the two things wouldn't be that correlated, but why is that correlation?

我不认为这些事情是高度相关的，确实。我觉得在我的案例中——我的意思是，老实说，这个问题很难回答。你知道，我只是不断非常努力地尝试，结果到目前为止已经足够了。

I don't think those things are super correlated, indeed. I feel like in my case—I mean, honestly, it's hard to answer the question. You know, I just kept trying really hard, and it turned out to have sufficed thus far.

明白了。所以是毅力？

Got it. So it's perseverance?

我认为这是一个必要但不充分的条件。就像，你知道，很多事情需要结合起来才能真正弄明白一些事情。你需要真正全力以赴，还需要有正确的看待事物的方式。很难给这个问题一个真正有意义的答案。

I think it's a necessary but not a sufficient condition. Like, you know, many things need to come together in order to really figure something out. You need to really go for it, and also need to have the right way of looking at things. It's hard to give a really meaningful answer to this question.

好了。Ilya，这真是一次愉快的经历。非常感谢你来到学会。感谢你带我们到办公室。谢谢。

Alright. Ilya, it has been a true pleasure. Thank you so much for coming out to the Society. I appreciate you bringing us to the offices. Thank you.

是的，我真的很享受。非常感谢。

Yeah, I really enjoyed it. Thank you very much.

感谢收听。下次见。干杯。

Listening. I'll see you next time. Cheers.