AI 数据危机：神话还是现实？

真正的负面结果是人们不会再相信他们看到的任何东西。即使没有 AI，这种情况也会发生，但 AI 绝对是这个过程的催化剂。我们拥有客观事实记录，这是一个相对较新的现象。人类进化时，我们只能信任亲近的人，这就是我们相信事物的方式。准备好了吗，Zico？我超级兴奋，老兄。我期待这一期很久了，非常感谢你今天来参加。

The real negative outcome is that people are not going to believe anything that they see. It didn't even need AI to get there, but AI is absolutely an accelerant for this process. It is a relatively new phenomenon that we have a record of objective fact in the world. Humans evolved in an environment where all we could do was trust our close associates. That's how we believed things. Ready to go, Zico? I am so excited for this dude. I've been looking forward to this one for a while, so thank you so much for joining me today.

太好了，谢谢。很高兴来到这里。

Great, thanks. Wonderful to be here.

我们要讨论一些相当深刻的话题。在深入之前，你能用 60 秒介绍一下你为什么擅长讨论这些话题，以及你目前的角色吗？

Now we're going to discuss some pretty meaty topics. Before we dive in, can you give me the 60-second context on why you're so well-versed to discuss them and your roles today?

我似乎在这里收集工作。我有多个不同的角色。但首先，我是一名教授，也是卡内基梅隆大学机器学习系的主任。我在这里大约 12 年了。机器学习系真的很独特，因为这是一个专门研究机器学习的系。我最近开始担任主任，所以每天都能沉浸在机器学习的业务和思想中。另外，我最近加入了 OpenAI 的董事会，就在几周前，这也非常令人兴奋。

So I seem to be collecting jobs here. I have a number of different roles. But I am first and foremost a professor and the head of the machine learning department at Carnegie Mellon. I've been here for about 12 years. The machine learning department is really kind of unique because it's a whole department just for machine learning. I've been heading it up as of quite recently, so I get to immerse myself in the business and the thought of machine learning all day every day. Also, I am recently on the board of OpenAI, which I joined a couple weeks ago, and it's been extremely exciting as well.

现在我想从一些基础和机制开始。当我们看支撑当前 AI 系统的基本技术时，你能帮我理解当前 AI 系统背后的基本技术是什么吗？

Now I want to start with some foundations and mechanics. When we look at the basic techniques that underpin current AI systems, can you help me understand what are the basic techniques today behind current AI systems?

我们以 LLM 为例来谈 AI，但要知道 AI 的范围远不止于此。LLM 非常神奇。它们最基本的工作方式是：从互联网上获取大量数据，训练一个模型。我知道这是个口语化的说法，但基本上你构建了一大组数学方程，这些方程学会预测给定序列中的下一个词。比如，如果你看到‘the quick brown fox’作为起始短语，它会预测出‘jumped’。这是一个常用短语，用来包含英语中的所有字母。明确地说，我们训练一个大模型来预测互联网上的词。然后当真正与 AI 系统对话时，我们只是用这个模型来预测回复中的下一个词。坦率地说，这能奏效有点荒谬。我认为人们经常用这种机制来贬低它们。我知道有人说：‘哦，AI 只是预测词，仅此而已，所以它不可能有智能。’我认为这显然是错的。我认为神奇的是这个科学事实：当你构建这样一个模型，训练它预测词，然后让它自由地一个接一个预测词并把它们串联起来，这个过程产生的结果是有智能的。我真的相信这些系统是有智能的。毫无疑问。我想说，这个事实——你可以训练词预测器，它们能产生智能、连贯的长篇回复——是过去 10 年、20 年甚至更长时间里最显著的科学发现之一。我认为这作为科学发现常常没有得到应有的重视，因为它确实是一个科学发现。

Let's talk about AI as LLMs, but with the context that AI is a much broader topic than this. LLMs are amazing. The way they work at the most basic level is that you take a lot of data from the internet, you train a model. I know that's a very colloquial term, but basically you build a great big set of mathematical equations that will learn to predict the words in the sequence given to them. So if you see 'the quick brown fox' as your starting phrase, it will predict the word 'jumped'. This is a common phrase we use to use every letter in the English language in a single phrase. To be clear, we train a big model on predicting words on the internet. Then when it comes time to actually speak with an AI system, all we do is use that model to predict what's the next word in a response. To put it bluntly, it's a little bit absurd that this works. I think people often use this mechanism as a way to dismiss them. I know people say, 'Oh well, AI is just predicting words, that's all it's doing, therefore it can't be intelligent.' I think that's demonstrably wrong. What I think is amazing is the scientific fact that when you build a model like this, train it to predict words, and then just turn it loose to predict words one after another and chain them together, what comes out of that process is intelligent. I really believe these systems are intelligent. Definitely. I would say that this fact—that you can train word predictors and they produce intelligent, coherent, long-form responses—is one of the most notable, if not the most notable, scientific discoveries of the past 10, 20 years, maybe much longer. I really think this is not often given its due as a scientific discovery, because it is a scientific discovery.

我能直接问一下吗：你提到数据输入这个要素非常必要。很多人认为我们已经耗尽了现有的数据资源。我们将需要合成数据来真正补充现有数据，或者我们需要创造新的形式，比如转录 YouTube 视频，那大概有 1500 亿小时之类的。你认为我们在多大程度上已经耗尽了可用的数据资源，并且正面临数据短缺危机？

Can I just dive in and ask you: you mentioned the element of the data input being so necessary. Everyone or a lot of people think that we've plundered the resources of data that we have already. We will need synthetic data to really supplement the data that we already have, or we need to create new forms, be it the transcription of YouTube videos which is like 150 billion hours or whatever that is. To what extent do you think it's true that we've plundered the data resources that we have available and we are running into a data shortage crisis?

我认为这个问题有两个截然相反的答案，就像很多问题一样。因为你说得对：想法是，由于这些模型是为了预测互联网上的文本而构建的，如果文本用完了，它们就会停滞，达到极限。我认为这实际上并不正确，有几个原因。从这些模型的趋势来看，有两种方式可能对也可能错。确实，很多容易获取的数据，互联网上最高质量的数据，已经被这些模型消耗了。没有另一个维基百科了。高质量的好文本只有那么多。另一方面，首先，我们只谈文本——公开可用的文本。如果开始谈内部可用的文本，从非常直接的角度看，我们远未用尽所有可用数据。公开模型训练的数据量大约在 30TB 左右。30TB 的文本数据——听起来很多，但实际上是很小的数据量。如果你装几个微型 SD 卡，它真的可以放在手掌里。有更多可用的数据我们目前没有用来构建这些模型。我在想多模态数据、视频数据、音频数据——我们有大量的可用数据。仅仅几十 TB 并不是那些索引互联网的大公司存储的数据量。数据比这多得多，我们远未接近利用整个储备。至于我们能否很好地利用这些数据——因为文本数据在某种意义上是最浓缩的形式——很多数据不是文本数据。这还有待观察，但我们远未达到这些模型可用数据的极限。可以说，我们无法处理它是因为我们没有足够的算力，但我们远未达到数据极限。

I think there are two diametrically opposed answers to this question, as with many questions. Because you're exactly right: the thought is because these models are built to predict text on the internet, if you run out of text, then they would plateau, reaching a limit. I don't think this is actually true for several reasons. From a raw standpoint of trending these models, there are two ways this is sort of maybe true, maybe false. It is true that a lot of the easily available data, the highest quality data on the internet, has been consumed by these models. There is not another Wikipedia. There's only so much really high-quality good text available. On the flip side, first of all, we're only talking about text—publicly available text. If you start talking about internally available text, from a very straightforward standpoint, we have not gotten close to using all the data that's available. Public models are trained on the order of 30 terabytes of data or something. 30 terabytes of text data—this sounds like a lot, but it's a tiny amount of data. If you load up a few micro SD cards, it will literally fit in the palm of your hand. There is so much more data available that we are not using right now to build these models. I'm thinking about multimodal data, video data, audio data—we have massive amounts available. Just a few tens of terabytes is not the amount of data these large companies that index the internet are storing. There is so much more data than this, and we have not really come close to tapping that whole reserve. Now, whether or not we can use that data well—because text data is in some sense the most distilled form—a lot of this is not textual data. That remains to be seen, but we are nowhere close to hitting the limits of available data in these models. Arguably, we're unable to process it because we don't have enough compute, but we're nowhere close to data limits.

从其他角度来看，使用这些新形式的多模态数据面临哪些挑战？

In other senses, what are the challenges of using these new forms of multimodal data?

嗯，我认为最大的挑战就是算力。拿视频数据来说，想想视频文件和文本文件的大小差异。如果我们转录这个播客，可能只有几千字节。但如果提取视频数据，大概会有 6.5GB 左右。没错，相差好几个数量级。可以说，取决于你的观点，也许真正有价值的信息并不全在我的语音和视频里。也许我身后的东西很有价值，但暂且忽略这一点，你可以说那里没有那么多可用内容。但我认为，如果谈到我们作为人类所使用的数据类型，视觉数据、时空数据对我们的智能概念至关重要。这对我们与世界互动以及思考自身智能的方式非常重要。所以我无法想象更多模态的数据——无论是视频、音频、其他时间序列还是其他感官信号——没有价值。有海量数据可用，但我们还没有弄清楚如何正确利用它们，要么是因为算力限制——你必须处理所有数据，这需要算力——要么是因为我们在跨模态传输和泛化方面的限制。但我认为它一定有用途。

Well, I think the biggest challenge is simply compute. If you have something like video data, just think about the size of a video file versus a text file. So if we transcribe this podcast, it would be a few kilobytes. If you take the video dump from it, it'll be on the order of, I don't know, about six and a half gigabytes. Exactly right. So we're many tens of thousands of magnitudes of difference. Arguably, depending on your opinion, maybe the entirety of the actual valuable information is not in the audio of my voice and the video. Maybe the stuff behind me is real valuable, but ignoring that for a moment, you could argue that there's not as much usable content there. But I think if we talk about what we think of as humans and what kind of data humans use, I would argue that visual data, spatiotemporal data, is hugely important to our conception of intelligence. This is hugely important to the way we interact with the world and the way we think about our own intelligence. So I can't fathom that there is not value to many more modalities of data, be it video, audio, other time series, and other sensory signals. There are massive amounts of data available, and I think we have not yet figured out how to properly leverage those due to either limitations of compute—you have to process all that data and it does take compute—or limitations in how we transfer and generalize across these modalities. But I think there has to be a use for it.

如果我们推到逻辑极端，假设我们已经耗尽了数据储备，而你提到即使那样，我们也不会看到模型性能的停滞。为什么？因为人们通常会这么认为。

If we took it to a logical extreme and said we had plundered the reserves of data, and you mentioned that even if we had, we would not see a plateauing in performance of models. Why is that? Because one would assume so.

这里有几种不同的观点。一个是，我们似乎仍然处于这样一个世界：即使使用相同的数据，增加模型大小也能获得更好的性能。显然，更大模型的真正价值在于它们能吸收更多数据，但同样正确的是，如果你只用一个固定的数据集多次训练，使用更大的模型通常会效果更好。所以我认为我们还没有达到那个平台期。另一件事，可能与你关于合成数据的观点有关，是我不认为有人会争辩说当前模型从提供给它们的数据中提取了最大可能的信息。一个简单的例子：如果你训练一个分类器来对一堆图像中的猫和狗进行分类，你会得到一定水平的性能。如果你在完全相同的图像上训练一个生成模型，从该生成模型生成更多合成数据，然后在这些合成数据上训练，你并不会好很多，但确实会好一点。这太神奇了。这意味着我们当前的算法还没有从现有数据中最大化地提取信息。我们可以对当前数据应用更多的演绎、推理和其他过程来提供更多价值。随着模型变得更大更好，它们可以说可以通过合成数据或不同的机制自己做到这一点。

There are a few different notions here. One is the fact that we still seem to be in a world where you can increase model size and get better performance even with the same data. Obviously, the real value of bigger models is they can ingest more data, but it is also true that if you just take a fixed dataset and run over it multiple times, if you use a bigger model, it will often work better. So I think we have not really reached the plateau there. The other thing, which is maybe related to your point on synthetic data, is that I don't think anyone would argue that current models extract the maximum information possible from the data presented to them. A simple example: if you train a classifier to classify images of cats versus dogs on a bunch of images, you get a certain level of performance. If you train a generative model on those exact same images, generate more synthetic data from that generative model, and then train on that synthetic data, you don't do much better but you do a little bit better. That's just wild. What that means is our current algorithms are not yet maximally extracting the information from data we have. There are way more deductions and inferences and other processes that we can apply to our current data to provide more value. As models get bigger and better, they can arguably do this themselves through synthetic data or different mechanisms.

当我们考虑从价值提取的角度优化现有数据时，还可以做些什么来从现有数据中获取更多价值？

When we think about optimizing the data that we have in terms of value extraction, what could be done further to get further value from the data that we have?

老实说，我不太确定。我认为这是目前研究中的一个重大开放问题：我们如何从现有数据中提取最大信息量？但正如我所说，我认为我们甚至还没有接近提取所有可用数据。所以当我审视这个领域时，我们知道我们远未从所有可用数据的闭包中提取最大信息，也远未处理所有可用数据，那么认为这会导致模型性能停滞的想法，与我们所看到的现实并不相符。

I don't really know, to be honest. I think this is a major open question right now in research: how do we extract the maximal information content from the data that we have? But again, as I said, I don't think we're close to even extracting all the data that's available. So when I look at this landscape, and we know that we aren't close to extracting the maximal information from the closure set of all the data available, and we haven't come close to processing all the data available to us, the idea that somehow this is a recipe for models plateauing in performance just doesn't jive with the reality of what we see.

好了，我们就像在课堂上一样。我们已经在数据不是瓶颈这一点上画了个大叉。很好。那模型呢？老实说，这个节目的乐趣之一就是我可以复述其他聪明人说过的话并加以检验。每个人都跟我谈到转向许多更小模型的世界，这些模型可能更高效。我们在多大程度上同意这一点？这是对的吗？我们应该如何解读？

Okay, so we're in the classroom together. We've got a big cross on that: data not the bottleneck. Good. What about models? Honestly, again, the joys of this show is I can kind of just regurgitate statements that other smart people have said and test them. Everyone talks to me about moving to this world of many smaller models which may be more efficient. To what extent do we agree with that? Is that right? How should we interpret that?

老实说，我对此不太确定。我认为我们还没有达到一个平衡点，能够清楚了解模型大小的稳态是什么，适用于什么应用以及如何使用——是作为通用系统还是用于非常特定的目的。这一切目前仍在探索中。我要说的是，我日常工作中经常使用这些模型。我几乎只使用我能获得的最大模型，因为它效果更好。当我没有重复做某个特定任务时，当我想要那种通用性时，我会使用更大的模型。小语言模型的概念——我认为这在未来很有可能。它是在我们达到通用性之后才出现的。一旦我们做某件事足够多，意识到有一个小任务我们想重复很多次，也许以前我们会为此训练一个定制的机器学习模型。但想法是，一旦你有一个重复多次的常规任务，而一个小模型可以完成它，那么专门为那个任务定制一个小模型可能就变得有价值了。但老实说，我认为整个动态还有待观察。我们只是不知道最终会使用哪种模型的平衡点在哪里。

I sort of don't really know here, to be honest. I think we have not yet reached an equilibrium point where we have a good sense of what the steady state of model size is for what application and how it's being used—whether it's a general-purpose system or for a very specific reason. This is all still being figured out right now. What I will say is that I use these models very regularly for my daily work. I work almost exclusively with the largest models available to me because it just works better. When I don't have a given task that I'm doing over and over, when I want to have that generality, I want to work with the larger models available. The notion of small language models—I think this might be very much a possibility in the future. It kind of comes after we reach this point of generality. Once we've done something enough and we realize there is still a small task we want to do many times, maybe before we would have used a custom-trained machine learning model for this. But the idea is that once you have a routine task that you're repeating again and again enough times, and a small model can do it, it probably does become valuable to specialize a small model for that task only. But to be honest, I think this whole dynamic is still to be played out. We just don't know what the equilibrium point is going to be of what kind of models are being used.

前几天我请来了 Cohere 的 Aiden，他说越来越难看到模型有可见的进步了。

I had Aiden on from Cohere here the other day, and he said that it is harder and harder to see visible gains in models.

考虑到模型如今惊人的性能和知识，以前你可以随便从街上拉个人，他们比模型聪明，但现在模型变得如此聪明，越来越难区分了。几乎到了那种 92%对 94%的程度。我认为这实际上更多与我们的基准测试以及人们通常使用这些模型的方式有关，而不是模型本身。如果你看看模型面临的一些最困难的问题，我们仍然看到更大模型和不同技术带来的进步。部分问题在于这些模型是自身成功的受害者。人们日常频繁使用它们，有一套常问的问题，比如写传记或历史。在许多预设问题上，模型已经做得很好，所以新模型并没有明显更好。例如，如果要求写卡内基梅隆大学的历史，Llama 8B 就能做得很好。也许最大的闭源模型会好一点，但那些不是相关的问题。我最常用模型的领域是编程和转录讲座。在这些任务上，我绝对没有看到收益停滞。最新模型明显比之前的迭代更好，让我的生活更轻松，让我在与它们交互时能上升到更高的抽象层次。所以这种看法更多与人们对模型能做什么的想象力有限有关，但这会随着时间演变。

Given now the incredible performance and knowledge of them, before you used to be able to take anyone off the street and they'd be smarter than the models, but now the models have got so smart it's harder and harder to distinguish. It's almost getting to that kind of 92% versus 94%. I think that actually has much more to do with our benchmarks and the way people typically use these models than the models themselves. If you look at some of the hardest problems models face, we are still seeing gains from larger models and different techniques. Part of the problem is that these models are victims of their own success. People use them regularly in daily life and have a suite of questions they ask, like writing a biography or history. On many pre-formatted questions, models already do well, so newer models don't do notably better. For example, if I ask to write a history of Carnegie Mellon University, Llama 8B can do that fine. Maybe the largest closed-source models are a bit better, but those aren't the relevant questions. The domain I use models for most is coding and transcribing lectures. On those tasks, I am absolutely not seeing plateauing gains. The latest models are notably better than previous iterations and make my life easier, allowing me to move to higher levels of abstraction when I interact with them. So this perception has more to do with people's limited imagination of what they can do with these models, but that will evolve over time.

我在看模型生态系统时纠结的一点是模型的商品化。一年前、18 个月前，它还很昂贵困难，参与者很少。现在这么多。商品化和成本降低——鉴于这是多年来我们看到的最快商品化的技术之一，你预计模型格局会如何发展？

One thing I struggle with when looking at model ecosystems is the commoditization of models. A year ago, 18 months ago, it was so expensive and hard, with few players. Now there are so many. The commoditization and reduction in cost—how do you expect this model landscape to play out, given it's one of the fastest commoditizing technologies we've seen in years?

在最近开源模型的发布和闭源模型的持续进步之间，它发展得非常快。早期也有很多开源模型泛滥，彼此之间并不更好，只是公司为了证明他们也能做到而训练。这并不清楚是否有价值。如果有很好的开源模型，为什么还要从头训练自己的语言模型？这会继续吗？也许吧，也许不。我认为很可能会出现整合，但我不太确定会如何发展。

It's been evolving so quickly between recent releases of open-source models and continued progress in closed-source models. There was also a proliferation early on of many open-source models that were not better than each other, just training for companies to demonstrate they could do it too. It's not clear that's valuable. Why would you want to train your own language model from scratch if there are very good open-source ones? Will that continue? Maybe, maybe not. I think there will most likely be consolidation, but I'm not quite sure how it will play out.

你认为那些生存下来并获胜的模型公司会做出什么决策？你提到了操作系统的泛滥，只有少数存活下来。那些生存并繁荣的模型公司会做出什么决策？

What decisions do you think the model companies that survive and win will make? You mentioned the proliferation of operating systems and only a few survived. What decisions will the surviving and thriving model companies make?

现在很多公司在考虑训练自己的模型，好像这是默认做法。但这在未来将不再经济可行，所以不会再发生了。

There are a lot of companies thinking about training their own models now, as if it's the default. But this won't be economically viable in the future, so it won't happen anymore.

我们提到了数据和模型。第三个支柱是算力。人们说我们已经到了收益递减的阶段——更多算力并没有带来相应的性能提升。我们真的到了瓶颈，还是还有更多空间？

We've mentioned data and models. The third pillar is compute. People are saying we've reached diminishing returns—more compute isn't leading to aligned performance. Have we really reached a bottleneck, or is there more room to run?

我不太确定说我们在算力方面达到平台期的理由是什么。我看到的大多数缩放定律都表明你可以继续扩展。只是更贵了。你可以说仅仅 Scaling 可能不是实现更好结果的最有效方式，我认为这很可能是真的。有其他更好的方法可以在不增加算力的情况下实现同样的改进，但算力似乎仍然是一个主要因素，并且仍然能改进事物。所以这更多是关于模型在推理和训练时成本的经济权衡。这些正成为更实际的担忧，而不是 Scaling 的实际限制。

I'm not really sure what the rationale is for saying we've plateaued in the compute sense. Most scaling laws I've seen suggest you can keep going. It's more expensive. You could argue that just scaling may not be the most efficient way to achieve better results, and I think that's very likely true. There are other better ways to achieve the same level of improvement than compute, but compute still seems to be a major factor and still improves things. So it's more a calculus about the monetary tradeoffs of how much models cost at inference and training. These are becoming more practical concerns than the actual limits of scaling.

你认为公司在多大程度上是在追求 AGI 和超级智能，而不是制造出色的产品并利用 AI 赚钱？

To what extent do you think corporations are chasing AGI and superintelligence versus making amazing products and leveraging AI to make money?

这两者并不互斥。我将 AGI 定义为一个在一年期项目中功能上等同于亲密合作者的系统。一个你像重视亲密合作者——我的学生或同事——一样重视的东西，持续一年。虚拟的也可以；具身 AI 需要更长时间。根据这个定义，AGI 可以自动化我工作中一年内所有功能性事务，除了较软的情感特质。这是一个很高的标准。我对何时会发生非常不确定。

Those are not mutually exclusive. I define AGI as a system that acts functionally equivalent to a close collaborator over the course of a year-long project. Something you would value as much as a close collaborator—a student or colleague—over a year. It's fine if it's virtual; embodied AI will take longer. Under that definition, AGI could automate everything I functionally do in my job over a year, besides softer emotional qualities. That's a high bar. I am massively uncertain as to when that will happen.

至于何时会发生，但我经历的一个重大转变是，我认为这很可能在我有生之年发生。我一直认为 AGI 的答案是‘不会在我有生之年’，而现在我给出的时间框架是 4 到 50 年之间，这确实反映了我巨大的不确定性。我个人不认为它会很快到来，但考虑到进展速度和我在演变中看到的东西，我很难忽视它。我们必须非常认真地对待这种可能性。

When this will happen, but a massive shift I've undergone is I think this will probably happen in my lifetime. I think the answer to AGI has always been 'not in my lifetime,' and the time frame I give right now is I think this is between 4 and 50 years or something like that, which really captures my massive uncertainty. I don't personally think it'll be on the low side, but I have a hard time dismissing it also given the rate of progress and the things I see evolving. We have to take that possibility very seriously.

这不就是所有新技术引入社会时的常态吗？有一个渐进的曲线，就业替代，社会动荡，这是技术的自然周期。

Is that not the same with all new technology introductions to society? There is a gradual curve, employment displacement, societal upheaval, and that is a natural cycle with technology.

我其实也同意我们会适应它。我不想淡化可能需要的转型程度，但我也认为，在这个新世界中，赢家将是那些生存、繁荣并占据主导地位的公司。我现在说的不是 AI 公司，而是其他公司。那些人们担心会解雇所有员工的公司，因为他们可以用 AI 替代——最成功的公司不会是那些解雇所有员工、让 AI 做完全相同事情的公司。他们会理解什么在变化，以及人们现在在引导这些系统、提供总体指导和框架方面最能做什么。生存下来的公司将是最善于利用其劳动力来充分利用这项新技术的公司。

I actually also agree with you that we will adapt to it. I don't want to downplay the extent of transformation that might be necessary here, but I also think that the winners in this new world will be the companies that survive and thrive and become dominant. Not talking about the AI companies for now, but the rest of the companies. The ones that people worry will fire all their workers because they can have an AI replace them—the ones that succeed best will not be the ones that fire all their workers to have an AI do the exact same thing. They'll be the ones that understand what's changing and what people can best do now in terms of steering these systems, providing overall guidance and framework about where we want to go with all this intelligence. The companies that survive will be the ones that best leverage their workforce to make the best use of this new technology.

你认为当前的模型提供商是否为消费者提供了特别好的入门途径，让他们知道如何最好地利用这些技术？

Do you think the current providers of models in particular give a particularly good on-ramp to consumers for how to leverage their technologies best?

这其实是一个非常微妙的问题。我们是否有能被劳动力最大化利用的 AI 产品？目前答案是否定的。显然，人们能用这些工具做什么和他们实际在做什么之间存在差距。对于大型企业来说，一个很大的担忧其实是他们数据的可移动性或可转移性。他们希望一切都在本地。他们非常不愿意让任何东西在他们的数据上进行训练。

This is actually a very nuanced question. Do we have AI products that are able to be maximally used by workforces? The answer right now is no. Clearly there is a gap between what people could use these things for and what they're using them for. For large enterprises, a big concern is actually the mobility or transferability of their data. They want everything on-prem. There's a big unwillingness to have anything trained on their data.

你认为我们会看到 AI 让大型企业从云端回归本地部署吗？

Do you think we will see AI bring back a movement from large enterprises away from the cloud back to on-prem?

我觉得这有点意思，因为企业都很乐意把数据放在云端。他们都用云服务存储数据，但说到‘用这个训练？不，不行。’我认为这很大程度上源于对这个过程如何运作的误解。而且，坦率地说，我认为这与一个事实有关：如果你考虑把所有内部数据都倒进一个大语言模型，这是站不住脚的。你做不到，原因有很多。最明显的是数据有访问权限——不是每个人都能访问所有数据。而语言模型的默认模式是，如果你在某个数据上训练过，你很可能能把它从系统中提取出来，如果你足够想的话。所以这与传统数据中的访问控制不兼容。这些就是担忧。但明确地说，有非常简单的解决方法。这很可能就是为什么基于 RAG 的系统如此普遍，并且即使微调可用，它们可能仍会存在。RAG，对于没听过这个词的人来说，是检索增强生成。它基本上意味着你出去获取你能访问的、有权限的、与你的问题相关的数据，全部注入到模型的上下文中，然后基于这些数据回答问题。所以这些基于 RAG 的技术会继续流行，正是因为它们尊重正常的数据访问流程。但我感觉很多犹豫实际上源于对这些模型如何工作的根本误解。人们认为，如果你让 ChatGPT 回答关于你任何数据的问题，这些数据就会以某种方式被训练并合并到模型中——无论是 API 调用、基于 RAG 的调用还是其他什么。这根本不是真的。这些模型不是这样工作的。这些模型是在一个非常大的数据集合上一次性训练的。如果你使用 API 访问之类的东西，你的数据不会被训练；模型不会在那上面重新训练。即使它被训练了，那也不是一回事。模型能回答你的问题并不意味着它在训练你的数据。这实际上只是非常简单的误解，我认为很多人很难克服。我在和公司交谈时仍然看到这些误解。所以从某种意义上说，我们的营销做得不好，因为人们并不真正理解，在某些用例中，这并不比一开始就把数据放在云端更冒险，而他们通常都这么做了。他们都转向了云端。所以我认为这会随着时间自然发生。

I find this kind of interesting in a way, because enterprises are all very happy to put their data in the cloud. They all use cloud services to store their data, but then 'train on this? No, can't do that.' I think a lot of it comes from a misunderstanding about how this process works. Also, frankly speaking, I think it has to do with the fact that if you think about the model of just taking all your internal data and dumping it into a large language model, this is not tenable. You can't do this for a number of reasons. The most obvious one being the data has access rights—not everyone gets access to all the data. And the default mode of language models is that if you train on some data, you can probably get it back out of the system if you want to enough. So this doesn't work with the access controls people have in traditional data. These are the concerns. Now, to be clear, there are very easy ways around this. This is probably why RAG-based systems are so common and probably will remain even with the advent of fine-tuning availability. RAG, for those who haven't heard the term, is retrieval augmented generation. It basically means you go out and fetch the data you can access that you have access rights to, relevant to your question, inject it all into the context of the model, and answer the question based on this data. So these RAG-based techniques will remain popular precisely because they respect normal data access procedures. But I feel like a lot of this hesitancy actually comes from a fundamental misunderstanding of how these models work. People think that if you have ChatGPT answer a question about any of your data, that data is somehow being trained upon and merged into the model—whether it's an API call, a RAG-based call, or anything else. It's just not true. That's not how these models work. These models are trained once on a very large collection of data. If you use things like API access, your data is not going to be trained on; the model will not be retrained on that. And even if it was, that is not the same thing. The fact that the model can answer your question does not mean the model is training on it. This is honestly just very simple levels of misunderstanding that I think a lot of people have a very hard time getting over. I still see these misconceptions when I talk with companies. So it's not like we've done a good job of marketing in some sense, because people don't really understand that in certain use cases, this is no riskier than just having your data in the cloud to begin with, which all of them typically do. They've all moved that way. So I think this will just happen naturally with the progression of time.

你认为人们对 AI 还有哪些最大的误解？太多了。我的意思是，我不认为人们真的知道，但还有哪些特别令人沮丧的？

What do you think of the other biggest misconceptions that people have towards AI? There are so many. I mean, I didn't think people actually know, but what are others which are really frustrating?

说实话，最让我沮丧的是，有些人对我们是否一定会很快实现 AGI，或者反过来，认为由于某些原因，我们绝对不可能用当前模型实现 AGI，表现出极大的确定性。这确实开始让我有点恼火。我个人，即使是 AI 寒冬怀疑论的产物，看到这些模型中发生的事情，也感到惊叹。那些一直敲响警钟、说‘看，这就要来了’的人，在很多情况下在我看来被证明是正确的。我根据看到的证据更新了我的后验信念。所以，很多人对 AGI 的哲学中最让我恼火的是，可观察到的证据几乎没有改变他们的信念一丝一毫。他们对实现通用 AI 需要什么有特定的信念，或者也许 AI 是……

The thing that frustrates me most, honestly speaking, is the degree of certainty that some people have about whether we will definitely get there very soon, or even more on the flip side, that there's absolutely no way that we will ever achieve AGI with these current models because of XYZ. This does actually start to irk me a little bit. I personally, even as a product of the AI winter skepticism, see what's happening in these models and I am amazed by it. The people who have been ringing this bell for a while, saying 'look, this is coming,' have in many cases in my view been proven right. I've updated my posterior beliefs based on the evidence I've seen. So what irks me the most about a lot of people's philosophy of AGI is how little observable evidence has changed their beliefs one iota. They had certain beliefs about what it would take to get to general AI, or maybe that AI was...

我能问你吗，我其实很担心 misinformation、deepfakes、恶意网络攻击的制造。我觉得我们没花足够时间讨论这个。当你思考现实中的实际危险时，你最担心什么？如果这些是我担心的，我广泛从事 AI 安全领域，所以我有很多担忧，而且有不同层次的担忧。我可以谈谈目前对我来说最严重的担忧。

Can I ask you, a big concern for me actually is misinformation, deepfakes, the creation of malicious cyber attacks. I don't think we spend enough time talking about this. When you think about the reality underlying practical dangers, what most concerns you? If those are some that concern me, I work very heavily in the field of AI safety broadly, so I have a huge number of concerns here, and sort of different tiers of concerns. I can get into what's the highest concern for me right now.

在回答之前，我想先深入谈谈你的观点，因为这是相关话题。这些不是我最大的担忧，但让我们谈谈 misinformation、deepfakes、虚假内容，以及一般用这些工具传播各种 misinformation。这当然是个大问题，我对此深感担忧。但最终结果不会是人们开始相信他们看到的 misinformation。真正的负面结果是人们不再相信他们看到的任何东西。所以可以说，我们已经在这条路上走了很远，人们基本上不相信任何不符合他们现有信念的阅读或看到的东西。这甚至不需要 AI 就能达到；AI 绝对是这个过程的加速器。不过我想说的是，这不是一个新现象。这实际上是人类进化中的常态。我们拥有客观事实的记录是一个相对较新的现象。视频之类的东西在 100 多年前并不存在。人类在只能信任亲密伙伴的环境中进化。那是我们相信事物的方式。从某种意义上说，我们不再拥有客观真理的记录是悲剧性的。我也对此感到困扰，但从另一个角度看，也许我们只是回到了过去的世界，在那里我们只能信任亲密伙伴关于世界的信念。

Before I answer that, I'm actually going to delve into your points since it's on topic. These are not my biggest concern, but let's talk about misinformation, deepfakes, fake content, just in general using these tools to proliferate different kinds of misinformation. This is a massive concern, of course, and I am deeply worried about this. But the net result of this outcome is not going to be that people start to believe everything they see in misinformation. The real negative outcome is that people are not going to believe anything that they see anymore. So arguably, we are already well along this path where people basically don't believe anything that they read or see that doesn't already conform to their current beliefs. It didn't even need AI to get there; AI is absolutely an accelerant for this process. What I will say, though, and this is a point I like to make, is that this is not a new phenomenon. This is actually the human condition as we evolved. It is a relatively new phenomenon that we have a record of objective fact in the world. Things like video didn't exist more than 100 years ago. Humans evolved in an environment where all we could do was trust our close associates. That's how we believed things. In some ways, we see it as tragic that we are no longer in a world with a record of objective truth. I am also troubled by this, but in another sense, maybe we're just getting back to the world we used to live in, where all we could do was trust our close associates about what we believe about the world.

但如果我们只信任身边认识多年的人，而且是在亲眼见到他们时，甚至不是他们发来的东西，这难道不会导致人类知识的进步减少吗？

Does that not lead to a reduction in the advancement of human knowledge though, if we only trust the people around us who we've known for years, when we see them in person, not even when they send us something?

显然存在巨大的负面外部性，但我们在视频出现之前就已经很好地发展了知识。我们仍然取得了科学进步。所以会有一些群体认为某些科学知识有价值，并推动它们，即使历史上一直存在大量不重视这些科学进步或对其性质有不同看法的人群。我们已经处于这个世界。这就是我们生活的世界。我认为这绝对是一个加速器，也是一种遗憾，让我们更倾向于无法拥有客观现实的那一派，但人类，可以说我们的自然状态，就是不在客观现实的性质上达成一致。我不想显得太消极，因为我认为我们仍然会取得进步。

Obviously there are massive negative externalities, but we did evolve knowledge very well at a time before video existed. We still made scientific progress. So there will be groups that decide that certain bodies of scientific knowledge are valuable and they will advance those, even in light of large other portions of the population that have existed throughout all of history too, that don't value those scientific advances or think differently about the nature of scientific advances. We are already in this world. This is the world we already live in. I think it is definitely an accelerant and a shame that this puts us more toward the camp of failing to have an objective reality, but humans, arguably our natural state, is to not agree on the nature of objective reality. I don't want to come across as too negative about this, because I think we will still absolutely make progress.

我觉得你挺乐观的。但对我来说，这就是为什么你会看到现有媒体品牌的价值在上升，因为人们信任它们生产的内容。所以你会相信《纽约时报》的推文，但某个随机账号的图片，你不知道。而且你可以说会有很多人不相信《纽约时报》说的任何话。顺便说一句，已经有这样的群体了。有很多国家的人不会相信《纽约时报》发表的任何东西。所以我们在某种程度上已经处于那种状态。我认为我们将需要更多地依赖群体，以及他们相关的信念结构。但这在某种程度上是人类的条件。不想太哲学化，但这就是我们一直以来的状态。视频只是一个短暂的插曲，让我们以为有 100 年的客观证据，但很快就不再是了。

I think you're being quite positive. But to me, this is why you see the increasing value of existing media brands, because people place validity and trust in the content they produce. So you will trust the New York Times tweet where it shows something, but you know some random account with a picture, you don't know. And you could argue that there's going to be a whole group of people that don't believe anything the New York Times says. There is already that group, by the way. There's plenty of countries where they would not believe anything published in the New York Times. So we're already there to a certain extent. I think we will need to rely arguably more on groups, but also their associated belief structures about this. But this is the human condition to a certain extent. Not to get too philosophical, but this is how we've always had to be. Video is a short blip where we think there's some objective evidence for 100 years of our history, and that's going to be no longer true pretty soon.

当你思考 AI 安全时，平台本身应该成为正义的仲裁者吗？Twitter、Facebook、Reddit。它们来决定什么内容不允许？我认为有些东西不应该在社交媒体上分享，而且其他人也都同意这一点。显然有些内容完全被视为非法，不能发布到社交媒体上；每个人都同意这一点。大家也同意，虽然不是所有人，但很多人认为，如果你想在社交媒体上表达自己，不应该要求你符合某些意识形态和观点。所以显然有一个中间地带。你必须把握好分寸，适应实际情况。我认为在很多方面，AI 在 misinformation 之类的事情上，并没有发明 misinformation。在 AI 出现之前很久就有 misinformation 和宣传。你可以说它是加速器，就像它加速我们拥有的一切一样。但它没有发明这些东西。我的希望是，我们现有的许多社会、经济和政府结构能够继续提供它们为当前审核方式提供的指导，即使在 AI 世界中也是如此。

When you think about AI safety, should the platforms themselves be the arbiters of justice, of what's right and what's not? Twitter, Facebook, Reddit. Are they the ones to say no, this is not allowed content? There are some things I believe that should not be shared on social media, and by the way, everyone else agrees with this too. There's obviously content that is outright considered illegal that you cannot post to social media; everyone agrees on this. Everyone also agrees, well not everyone, but a lot of people, that in general there should not be a requirement to conform to certain ideologies and opinions if you want to express yourself on social media. So there's obviously a middle ground. You have to tow the line here and adapt to the reality of the situation on the ground. I think in many ways, AI when it comes to things like misinformation, it did not invent misinformation. There was misinformation and propaganda long before AI. You can argue it's an accelerant, like it's an accelerant for everything that we have. But it did not invent these things. My hope is that a lot of our existing social, economic, and governmental structures can continue to provide the same guidance they have provided for our current take on moderation, even in an AI world.

作为一个政府组织，今天你该如何应对？你本应制定法规、政策，却要面对 RAG、FLOPs、Transformer 架构，所有这些庞大的技术词汇和架构信息？他们完全不知道这意味着什么。我的问题是：我们的政府结构是否能够有效监管 AI？

How do you respond as a government organization today where you are supposed to set regulation, set policy, and you are dealing with RAG, FLOPs, Transformer architecture, all these bulky technical words and architecture and information? They have no idea what it means. My question is: are our governments structurally set up to regulate AI effectively?

关于这一点，我有两种看法。

On this point, I hold two views.

我直说吧。监管不是我的直接领域，所以这不是我研究的内容。但我确实想回到一些 AI 安全的话题上。我从未描述过我最担心的是什么。我认为，像许多新技术一样，监管和政府提供框架以确保新技术造福世界绝对有其作用。这在某种程度上正是我们组建政府的原因。在这个大框架下，我认为绝对有必要更好地理解如何以及在何处将 AI 作为一种技术进行监管。然而，我也认为，这些监管有时演变的许多细节可能有些误导或没有抓住重点。当我读到它们时，它们会在几个月内过时，因为它们处理问题的方式并不符合这些系统实际开发的性质。我认为监管 AI 的下游用途要容易得多。例如，在 misinformation 方面，我们已经有了处理诽谤等问题的法律。在许多情况下，由于 AI 充当了加速器，现有法律只需稍作调整以应对 AI 产生的速度和数量，就足以监管许多有害用例。但与此同时，我认为这也不够。当然，对于技术，尤其是像这样强大的技术，我们必须考虑如何监管它们。我不知道那会是什么样子。这非常困难，因为它变化得太快了。

I'll say this straightforwardly. Regulation is not directly my wheelhouse, so it's not what I work on. But I do want to get back to some AI safety points. I never described my biggest fear. I think that like a lot of new technologies, there's absolutely a role for regulation and for governments to provide frameworks for ensuring that new technologies benefit the world. This is why we form governments to a certain extent. Under that umbrella, I believe there is absolutely the need to better understand how and where we can regulate AI as a technology. However, I also think that a lot of the details about how those regulations sometimes evolve can be a bit misguided or miss the point. When I read them, they're going to become dated in a matter of months because they approach the problem in a way that doesn't match the nature of how these systems are really developed in practice. I think it is much easier to regulate the downstream uses of AI. For example, when it comes to misinformation, we already have laws that deal with libel and things like that. In many cases, because AI is acting as an accelerator, existing laws with slight tweaking to deal with the velocity and volume that AI can produce can suffice to regulate many harmful use cases. But at the same time, I don't think that's sufficient either. Of course, there are going to be ways in which technologies, especially ones as powerful as this, we have to think about how we can regulate them. I don't know what that looks like. It's extremely hard because it changes incredibly rapidly.

说到模型的安全保管，我想讨论一下你心中的安全担忧层级。我提到了我的。你会如何分类你的？

Speaking of the safekeeping of models, I want to discuss the hierarchy of safety concerns you have. I mentioned mine. How would you categorize yours?

当然。目前我在 AI 安全方面最大的担忧，我认为会导致许多负面下游效应，就是我们现有的 AI 模型无法可靠地遵循规范。这些模型被调优以遵循指令。作为开发者，你可以给它们一些指令，但如果用户输入了某些内容，它们可能会转而遵循那些指令。我们都见过这种情况。它有很多名字：提示注入、越狱等。核心问题是，我们很难强制执行关于模型能产生什么内容的规则。我们常说模型被训练成不做某些事。一个常见的例子是偷车接线。大多数商业模型，如果你问如何偷车接线，它们会说做不到。但通过多种手段很容易操纵这些模型，让它们相信真的应该告诉你如何偷车接线，因为你把自己锁在车外了，情况紧急。这与我们习惯的程序行为非常不同。我们习惯计算机程序只做被告知的事，不多不少。这些模型并不总是按指令行事；有时做得太多或太少。我们非常不习惯像看待这些模型那样看待计算机软件。老实说，我并不在乎模型是否告诉我如何偷车接线。互联网上有说明，它们并没有透露什么敏感信息。然而，当我们开始将这些模型集成到更大的系统中，当我们开始拥有解析互联网并执行任务的智能体时，如果它们运行模型解析不可信的第三方数据，这些数据基本上可以控制那些模型。从网络安全的角度来看，这就像这些模型都有一个我们知道的缓冲区溢出漏洞，而且我们不知道如何修补。我们还没有找到用模型修复这个问题的方法。我们正在取得进展，但这确实是一个令人担忧的问题。在聊天机器人这样的领域，负面影响可能不那么令人担忧，但当你开始拥有更复杂的 LLM 系统时，这就变得非常令人担忧了。我把这个担忧放在首位的原因是，我认为我们需要解决这个问题，否则所有其他下游担忧都会变得更糟。例如，人们经常谈论生物风险或网络攻击风险。我特别担心网络风险。我认为在很多情况下，这些模型基本上已经解决了这个问题；它们已经能够分析和查找代码中的漏洞。这非常令人担忧。我们通常认为解决这个问题的方法是，我们会发布某些模型，并说‘不要利用你的模型能力对某些基础设施发起明显的网络攻击’。但我们无法让它们遵循这个指令。能够访问模型的人，甚至有时是闭源模型，都可以越狱这些东西并获取访问权限。我们在解决越狱问题、确保模型遵循规范方面取得了巨大进展，但在解决这个问题之前，很难说所有其他危险的事情都得到了控制。

Sure. The biggest concern I have right now in AI safety, which I think leads to a lot of negative downstream effects, is that the AI models we have are not able to reliably follow specifications. These models are tuned to follow instructions. You can give them some instructions as a developer, but then if a user types something, they can follow those instructions instead. We've all seen this. It goes by many names: prompt injection, jailbreaking, etc. The core point is we have a very hard time enforcing rules about what these models can produce. Often we say models are trained just to not do things. A common example is hotwiring a car. Most commercial models, if you ask how to hotwire a car, they'll say they can't do it. But it's very easy through a number of means to manipulate these models and convince them that they really should tell you how to hotwire a car, because you've locked yourself out and it's an emergency. This is very different from how we're used to programs acting. We are used to computer programs doing what they're told, nothing more and nothing less. These models don't always do what they're told; sometimes they do too much or too little. We are very unused to thinking about computer software like these models. To be honest, I don't really care if models tell me how to hotwire a car. There are instructions on the internet; they're not revealing anything sensitive. However, as we start to integrate these models into larger systems, as we start to have agents that parse the internet and do things, if they're running their model parsing untrusted third-party data, that data can essentially gain control of those models. From a cybersecurity standpoint, it's like these models have a buffer overflow in all of them that we know about and don't know how to patch. We don't know how to fix this yet with models. We are making progress, but this is a real concern. The negative effects in a domain like a chatbot might not be that concerning, but as you start having much more complex LLM systems, this becomes much more concerning. The reason I place this concern first is that I think this fact is something we need to figure out, or all the other downstream concerns get much worse. For example, often people talk about risks like bio risks or cyber attack risks. I'm very concerned about cyber risks in particular. I think this is essentially already solved in many cases by these models; they can already solve and analyze code to find vulnerabilities. This is extremely concerning. The way we think about fixing this normally is we would have certain models that we release and say, 'Don't use your model ability to create obvious cyber attacks against certain infrastructure.' But we can't make them follow that instruction. Someone with access to a model, or even sometimes to a closed-source model, can jailbreak these things and get access. We are making immense progress in solving this problem of preventing jailbreaks and making sure models follow a spec, but until we solve this problem, it's very hard to say all the other dangerous things are under control.

AI 可能展现的下游效应或危险能力变得非常令人担忧。这是对模型所有其他不良行为的倍增效应，这就是为什么我现在如此担心。

Downstream effects that AI could demonstrate, or dangerous capabilities, become much more concerning. This is a multiplier effect on everything else bad these models can do, which is why I'm so concerned about it right now.

没错。随之而来的被加剧的要素是恐怖袭击或更糟。所以这是个很好的引子：如果越狱和操纵模型是攻击向量，那么收益是什么？我们能做什么？在这里，我们试图评估模型的核心有害能力。人们对此思考了很多：化学武器、生物武器、网络攻击。就我个人而言，我认为网络攻击比生物威胁等更清晰、更现实的威胁。同时，我不想忽视任何这些担忧。比我研究得多的人非常担心，所以我想以应有的尊重对待这个问题，因为这些都是巨大的问题。

Exactly. The subsequent elements that become heightened are terrorist attacks or worse. So that's a good lead-in: if jailbreaks and manipulation of models are the attack vector, what is the payoff? What are the things we can do? Here, we're trying to assess the core harmful capabilities of models. People have thought a lot about this: chemical weapons, biological weapons, cyber attacks. Personally, I think cyber attacks are a much more clear and present threat than bio threats, for example. At the same time, I don't want to dismiss any of these concerns. People who have looked at this much more than I am very concerned, so I want to treat this with the respect it deserves because these are massive problems.

AI 模型有很多潜在危害。有些主要与规模相关，比如你提到的错误信息，但有些是这些模型可能实现的能力，它们会大大降低做坏事的门槛——比如制造一个零日漏洞，让全球一半的软件瘫痪。担忧不在于它们最初能自主做到，而是它们将所需技能门槛降得如此之低，以至于大量恶意行为者都能掌握这些能力。生物风险、化学风险或其他事情也是如此。如果我们开始让每个人都能制造有害物品，这些担忧就必须被认真对待，视为真实的可能性。

There are a lot of potential harms of AI models. Some are associated primarily with scale, like misinformation you mentioned, but some are capabilities we think these models might enable, where they would lower the bar so much for bad things—like creating a zero-day exploit that takes down software over half the world. The concern is not that they can do this autonomously initially, but they can lower the bar so far in the skill level required that it puts these capabilities in the hands of a huge number of bad actors. The same is true for biological risk, chemical risk, or other things. These concerns have to be taken seriously as genuine possibilities if we start putting into everyone's hands the ability to create harmful artifacts.

Scale 的 Alex Wang 在节目中说过一句精彩的话：我们现在拥有一种技术，其潜在危险性和影响力比核武器更大。我的问题是：如果真是这样，或者部分如此，有没有任何世界它应该被开放？两个问题：一是 AI 像核武器一样危险，二是这对某些模型的开放发布意味着什么？

Alex Wang at Scale said a brilliant line on the show: essentially we have a technology now that is more potentially dangerous and impactful than nuclear weapons. My question to you is: if that is the case, or even partially the case, is there any world in which it should be open? Two issues: one is AI as dangerous as nuclear weapons, and what does this imply about the open release of certain models?

我讲两点。我认为核武器类比并不好，因为核武器只有一个目的：摧毁东西。也许更好的类比是整个核技术，因为它既能制造核武器，也能为大量人口提供无二氧化碳的电力。很多人押注核能作为无碳能源的方式。但特别与核武器的类比经常被夸大，正是因为 AI 有很多好的用途。核武器可以说只做一件事，而且不被认为是好的用途。所以这是一种非常不同的技术。现在说你的第二点：开放模型辩论，这在 AI 安全讨论中经常出现。我大体上是开源模型的粉丝。开源发布模型——我实际上说开放权重，因为通常这些并不是传统意义上的开源；它们更像是闭源可执行文件，你可以在自己的电脑上运行。开放权重模型推进了我研究这些系统的能力；它们是学术界及其他领域研究的主要工具，并且正在成为整个 AI 生态系统的关键部分。第二：如果你看看当前最好的模型，比如 GPT-4、Claude 3.5、Gemini，我不会对拥有一个能力相当的开放源码模型感到紧张，就灾难性影响而言，因为这些模型本身并不那么危险；我们对它们有很好的掌控。我们可能已经达到了那个点，因为 Llama 3 405B 已经很接近了——我不认为它完全达到那个水平，但正在接近。这次发布还没有造成灾难性事件，而且可以说不会，因为这些模型还有一段路要走。然而，这是一个很大的然而，我认为会有那么一个时刻，当某种能力达到某个点时，我们应该在把这些东西交给任何人随意使用之前三思。我确实认为，在我们当前发展可及的范围内，存在某些能力水平，如果我问是否应该把这些给每个人，不仅使用而且随意调整和专业化，我会说会有让我感到不舒服的点。例如，一个模型能够分析任何代码库、二进制可执行文件、网站或 JavaScript，并立即找到漏洞来瘫痪大部分互联网或软件——如果这种能力被展示出来，我会很难说‘当然，发布吧’。我们都知道修补软件比发现漏洞难得多。是的，存在双重用途；你可以用它做好事，但风险很大。

I'll make two points. I think the nuclear weapon analogy is not a great one because nuclear weapons have one purpose: to destroy things. Maybe a better analogy is nuclear technology as a whole, because it can create nuclear weapons but also provide non-CO2-emitting power to a huge number of people. Many people are betting on nuclear as the way to create carbon-free energy. But the analogy to nuclear weapons in particular is often overstated precisely because AI has many good uses. Nuclear weapons arguably do one thing, and it's not considered a good use. So this is a very different kind of technology. Let me get to your second point: the open model debate, which is frequently played out in discussions on AI safety. I'm a fan of open-source models in a general sense. Open-source release of models—I really say open-weight because often these are not actually open-source in the traditional way; they're more like closed-source executables that you can run on your own computer. Open-weight models have advanced my ability to study these systems; they've been the primary tool for research in academia and beyond, and they're becoming a critical part of the overall AI ecosystem. Number two: if you look at current best models like GPT-4, Claude 3.5, Gemini, I would not be all that nervous about having an open-source model as capable as these in terms of catastrophic effects, because these models aren't by themselves that dangerous; we have a good handle on them. We're arguably already there because Llama 3 405 billion is pretty close—I don't think it's quite at that level yet, but it's getting there. This release has not yet caused some catastrophic event, and arguably it won't, because these models still have a ways to go. However, and this is the big however, I think there will come a time when a certain capability reaches the point that should give us pause when it comes to just turning these things over to whoever wants to use them however they want. I do think there are certain levels of capabilities within eyesight of our current development that, if I were to ask whether we should give this to everyone not just to use but to tune and specialize however they want, I would say there will be a point where I get uncomfortable with that. For example, a model that could analyze any codebase, binary executable, website, or JavaScript and immediately find a vulnerability to take down a large portion of the internet or software—if this capability were demonstrated, I would have a very hard time saying, 'Of course, let's just release it.' We all know that patching software is much harder than finding exploits. Yes, there is dual use; you can use it for good, but the risks are significant.

要更好地保障软件安全，但这需要时间，很难。我不认为我们应该立即仓促地发布一个能够发现世界上任何代码漏洞的模型。我不希望这样的模型以开放权重的方式发布，让任何人都能使用。目前的情况让我感到些许安慰：至少现在，闭源模型总是在同等能力的开放权重模型发布之前一段时间就推出了。我认为这其实是件好事，因为我的希望是——我们也是偶然走到这一步的，本来不一定会这样。我知道有些公司正在推动从一开始就开源比以往任何时候都更强大的模型，这让我有点紧张。但目前我们还没到那个地步；我们处于一个最强大的模型、某种能力的首次发布通常来自闭源模型的世界。我认为这是好事。它给了我们一些时间来逐步了解和理解这些模型的能力，在一个更可控的环境中，这样我们就能达到一定程度的安心——也许不是完全安心，但至少是某种程度的安心——来说，是的，发布一个类似的开源模型可能没问题。我真诚地希望，如果某个模型真的展示了能瞬间为任何可执行代码或编译代码创建漏洞利用的能力，并且我们首先在闭源模型中看到了这一点，我们会认真考虑是否真的想发布一个同等能力的开放权重模型，让任何人都能使用。

To secure software better, but that takes time, it's hard. I don't think we should just immediately snap to release a model that could find a vulnerability in literally any code that's out there in the world. I wouldn't want that to be released open weight for anyone to use. Now, I take some solace in the current situation we find ourselves in, which basically, at least for now, there's a constant stream of closed models that are released some time before an equivalently capable open weight model, right? And I think this is actually a very good thing because my hope would be that—and we sort of found ourselves here by accident, it didn't have to be like this. I know some companies are pushing to open source more powerful models than we have ever had before right at the outset, and that makes me a little nervous. But right now we're not in that world; we're in a world where the most capable models, the first releases of a certain capability, typically come from closed source models. I think this is a good thing. I think it gives us some time to essentially come to terms and understand the capabilities of these models in a more controlled environment, such that we can reach a level of comfort—maybe not full comfort, but at least a level of comfort—to say, yes, it's probably okay if we release a similar model open source. And my sincere hope would be that if one of these models does really demonstrate the ability to create an exploit for any executable code or compiled code or anything else instantly, and we see that in the closed source model first, we would think a little bit about whether we really want to release an equivalent model open weight and just for anyone to use.

关于 AI 安全，我有没有什么没问到但应该问的问题？

Is there anything I have not asked on AI safety that I should have asked?

关于 AI 安全，我被问到的最大的问题之一是——因为到目前为止我向你介绍的其实是一套非常实用的建议和非常务实的观点，对吧？我的意思是，我谈到了防止越狱，谈到了在保障这些模型安全方面取得进展，谈到了当前开源模型和闭源模型发布之间的相互作用。我确实认为，虽然这不是我的领域，但那些关于智能体式 AGI 系统开始故意对人类造成伤害的更离谱的场景——这些通常就是所谓的 rogue AI，它决定要消灭人类并开始计划如何做到这一点。在我看来，坦率地说，这些是遥远的、有点科幻的场景——这些往往是我们讨论 AI 安全时的争论焦点。关于这一点，我想说两件事。第一，我认为绝大多数 AI 安全不应该围绕这些话题；绝大多数应该关注我们让系统更安全的非常实际的关切，就像我目前和你讨论的那些。当前系统中已经存在巨大的安全考量和风险，即使在稍微更强大的系统中也肯定存在，无论 AGI 的时间框架如何，当然也无论 rogue 智能 AI 系统的时间框架如何。我也不想完全否定这一点。我的说法是：我很高兴人们在思考这些问题。我很高兴人们在思考能力的影响，甚至是我认为遥远的场景。这些都是值得思考的好事，就像 AI 系统更直接的危害，比如错误信息、滥用这些东西一样。

One of the biggest questions I am asked about AI safety—because what I laid out to you so far was honestly a pretty practical set of recommendations and a pretty pragmatic view on the field, right? I mean, I'm talking about preventing jailbreaks, I'm talking about making progress when it comes to securing these models, I'm talking about the interplay between the current release of open source models and closed source models. I do think that while it is not my area, the more far-fetched scenarios about sort of agentic AGI systems that start intentionally acting harmful against humans—these are often the so-called rogue AI that decides it wants to wipe out humanity and goes about planning how to do this. What seemed to me, and I'll be honest here, far-flung, kind of sci-fi-ish scenarios—these are often the debates we have when it comes to AI safety. I want to say two things about this. The first is that I think the vast majority of AI safety should not be about these topics; the vast majority should be about quite practical concerns we have on making systems safer, like the kind I've talked with you about so far. There are already massive safety considerations and risks that are present in current systems and would certainly be present even in slightly more capable systems, irrespective of the time frames associated with AGI and certainly the time frames associated with a rogue intelligent AI system. I also don't want to dismiss this entirely. The way I would put it is: I am glad people are thinking about these problems. I'm glad people are thinking about the effects of capabilities and even what I consider far-flung scenarios. They are good things to think about, as are much more immediate harms of AI systems like misinformation, like misuse of these things.

你认为哪个遥远的场景最值得思考？因为大多数人只会想到机器人抢走工作、杀死人类，最终是抢走我们的工作。我认为抢走工作比杀死人类更紧迫。

What far-flung scenario do you think is most good to think about? Because most people just go robots killing jobs, killing humans, ultimately post-killing our jobs. I think killing jobs is much more immediate of a concern than killing humans.

我经常举的一个例子，试图把这两方面结合起来——一方面是 AI 接管世界、杀死我们所有人，另一方面是更持怀疑态度的学术界人士会说：我现在看到一条通往这样一个世界的道路，在几年后，我们开始将 AI 模型集成到越来越多的软件中，我们开始越来越多地构建它，我们让这些东西在行动上更自主一些，我们开始自然地——因为软件为我们做所有事情——我们开始自然地将它注入到我们所有的软件中，包括处理关键基础设施的软件，比如控制电网的东西，对吧？然后突然之间，你有了这些智能体，它们在控制电网等方面扮演着积极的角色。这导致了，即使在我看来，大规模关联故障的可能性，这些故障可能导致电力中断，而且我们无法轻易恢复，影响到国家的大部分地区。在这个世界里，我认为老实说——如果我们走错了路，这绝对不是不可想象的。在这个电力被切断的世界里，我们可以争论并决定这是系统的一个 bug，我们一开始就不应该在这里安装 LLM，或者我们可以争论这实际上是 rogue AI 接管并决定切断电源以便杀死全人类。但谁在乎呢？电还是断了。这对国家来说仍然是一场灾难。所以我们必须有一个计划来思考如何应对这类事件。这是一个我经常提到的例子。在某种程度上，无论 AI 是故意以邪恶的方式行事并欺骗人类，还是这是系统的一个 bug 和缺陷，最终效果在某些情况下是一样的。因此，我们迫切需要建立结构来防止这些事情成为可能。

An example I often use here to try to bring a little bit of these two sides—the sort of AI taking over the world, killing us all, and the more skeptical-minded academic folks will say: I see a path right now to a world in which, in a few years from now, we start integrating AI models into more and more of our software, we start building it up more and more, we sort of make these things a little bit more autonomous in their actions, we start naturally—because software does everything for us—we start naturally infusing this into all software we have, including software that handles things like critical infrastructure, stuff that controls the power grid, things like this, right? And now all of a sudden you have these agents that are sort of taking an active role in doing things like controlling power grids. This leads to the possibility of, even in my view, sort of massive correlated failures that could do things like bring down power electricity in a way that we can't restore it easily for a large portion of the country. Now in this world, and I think it's honestly not—again, if we go down the wrong path, this is definitely not that impossible to imagine. Now in this world where the power has been shut off, you know, we can debate and decide whether this was a bug in the system and we should never have installed LLMs here in the first place, or we could debate whether this was actually the rogue AI taking over and deciding to shut off the power so it could kill all humanity. But who cares? The power is still off. This is still a catastrophic event for the country. And so we have to have a plan for how to think about events like this happening. This is an example I come to too often. To a certain extent, it doesn't matter whether the AI is intentionally doing something in an evil fashion while deceiving humans or whether this is a bug and a flaw in the system; the end effects are the same in some cases. And so we need to desperately put in structures in place that prevent these things from being possible.

哦，我们很庆幸他们有 CrowdStrike。嗯，确实。所以，我们现在都非常熟悉关联故障的弊端，对吧？想象一下，如果所有运行电网的系统也是如此，这并非不可信。问题在于，这些系统，因为我们并不真正理解——我的意思是，我们并不理解它们，我们不了解这些东西内部如何运作——可能的关联故障、可能的攻击向量，所有这些我们都不了解。正因为如此，我们需要非常谨慎地考虑如何部署这些系统，如何考虑安全问题，尤其是当涉及到……

Oh, we just appreciate that they had CrowdStrike. Well, exactly. So yeah, we are all very familiar right now with the downsides of correlated failure, right? And imagine if that was also true of all the systems that were operating the power grid right now, which is not impossible to believe. And the problem is that these systems, because we don't understand really—I mean, we don't understand them right, we do not understand how these things work internally—the possible correlated failures, the possible attack vectors, all these sorts of things we don't understand it. And because of this, we need to think very carefully about how we deploy these systems, how we consider safety concerns, especially when it comes to...

像关键基础设施这类问题，我认为是极其紧迫的担忧。还有生物风险——虽然我在这方面工作较少——但这些都是潜在的紧迫问题。你不需要相信有超级智能的邪恶机器人，才能把这些视为紧迫问题。AI 安全是当下的担忧，我们都需要正视它现在就是担忧，并立即开始解决问题。令人震惊的是——不知道你是否还记得银行密码验证，比如‘我的声音就是我的密码’——我真的希望你现在不是这样，因为 Eleven Labs 已经能用我的声音做很棒的事了。这太疯狂了。我认为我们现有的系统已经颠覆了大量我们建立起来的体系，而且随着 AI 技术的不断演进，它们还会被进一步颠覆。这些都是我们必须正视的现实问题。

Things like critical infrastructure that I think are extremely pressing concerns. And yes, things like bio-risk again—I work much less on these—but these are potentially pressing concerns. And you don't have to believe in superintelligent evil robots in order to have these as pressing concerns. AI safety is a concern right now, and we all need to come to grips with the fact that it's a concern right now and start solving the problems right now. The astonishing thing is—I don't know if you remember banking by password tests, like 'my voice is my password'—I really hope yours isn't right now because Eleven Labs is doing pretty great things with my voice. It's really wild. And I think our current systems, the things that we have already, upend a massive amount of the systems we built in place, and they will continue to be upended more and more from evolving AI technology. These are real concerns that we have to come to terms with.

在快速问答之前，最后一个问题：你对我们将要进入的未来感到乐观吗？你希望你的孩子更多地与 LLM 和模型对话，而不是与人对话吗？

Final one for you before we do a quick fire: are you optimistic about this future we're moving into, and do you want your children to speak more to LLMs and models than they do to humans?

我会把自己归类为 AI 方面的乐观主义者。我已经很喜欢这些工具了。我对这些工具能带来的潜力感到兴奋，甚至包括 AGI。我在这里用‘工具’这个词并非贬义。希望 AGI 是一种工具，对吧？希望 AGI 是一个我们仍然可以部署来实现我们目标的系统。我忍不住对这些事情感到兴奋。这是我们整个领域长期工作的结晶，正在以直接有利于我所做事情的方式结出果实。所以我想要这些工具。也许这说到了关键点：我想要开发和改进这些工具的安全性，因为我想使用它们。并不是说我们有某种道德义务必须开发这些工具——也许确实有，或者我们必须开发 AI 和 AGI，也许这是真的——但那不是激励我开发它们的动力。我开发它们是因为我想使用它们。而要让它们达到那个点，它们必须是安全的。这是一个必要条件。这就是为什么我致力于构建和提升 AI 系统的安全性。

I would classify myself as an optimist when it comes to AI. I already enjoy these tools. I'm excited about the potential things we can do with these tools, yes, even up to AGI. And I use the word 'tool' here not pejoratively. Hopefully AGI is a tool, right? Hopefully AGI is a system that we still deploy to our ends, to achieve our ends. And I can't help but be excited about these things. This is the culmination of a lot of the work that we as the field have been doing, and it's coming to fruition in a way that is directly beneficial for a lot of the things that I do. So I want to have these tools. And maybe this gets to the clear point here: I want to develop and improve safety of these tools because I want to use them. It's not that we have some moral imperative that we have to develop these tools—I mean, maybe there is, or we have to develop AI and AGI, maybe that's true—but that's not what motivates me to develop them. I want to develop them because I want to use them. And to be able to have them to reach that point, they have to be safe. It's a necessary condition. That's why I work on building and improving the safety of AI systems.

Z，我可以和你聊一整天。但我想进入快速问答环节。我说一个简短的陈述，你给我你的即时想法。可以吗？

Z, I could talk to you all day. I do want to move into a quick fire. I say a short statement, you give me your immediate thoughts. Does that sound okay?

好的。每个问题我有多长时间？

Okay. How long do I have for each one?

60 秒。

60 seconds.

好的。

Okay.

你曾经对模型有什么看法，后来改变了想法？

What did you believe about models that you later changed your mind on?

在我职业生涯的大部分时间里，我都认为模型架构非常重要，通过设计巧妙复杂的架构和内部子模块，就能通往更好的 AI 系统。但现在我基本不再这么认为了。我认为模型本身不重要，架构也不重要，这同样适用于 Transformer。我觉得只要花足够的时间，任何东西都可以替代它们。所以我认为，在很大程度上，我们在很多 AI 工作中已经处于后架构时代了。

For a lot of my career, I was thinking that model architectures really mattered, and by having clever complex architectures and submodules inside architectures, that would be the route to better AI systems. For the most part, I don't believe this as much anymore. I think basically models don't matter, architectures don't matter, and that applies to Transformers too. I think anything could kind of work in their stead if we just spend enough time on it. So I think that to a large extent, we're kind of post-architecture in a lot of our AI work.

真是个突破性的观点。你曾经对数据有什么看法，后来改变了想法？

What a breakthrough snap that is. What did you believe about data that you later changed your mind on?

恰恰相反，我曾经认为数据必须经过高度筛选才有价值，数据的价值本质上来自非常手动化的标注和人力密集型的整理。当前 AI 的一个巨大惊人洞见是，我们可以在很大程度上直接从互联网上抓取现有数据，基于这些数据训练模型，然后得到惊人的结果。不是说筛选没有价值——当然有这方面的因素——但在很大程度上，这更像是旧范式的无监督学习，这绝对令人难以置信。

Kind of on the contrary, I thought that data had to be highly curated to be valuable, and the value in data came essentially from very manual labeling and human-intensive curation. The big amazing insight of current AI is that we can, to a large extent, just suck up data that exists out there on the internet, train models based upon that, and get amazing things to come out of it. Not to say there's no value in curation—of course there are elements of this—but to a very large extent, this is kind of the old paradigm of unsupervised learning, and that's absolutely incredible.

加入 OpenAI 董事会是什么流程？是 Sam 直接打电话给你说‘嘿，喜欢你的白板，有兴趣加入我们董事会吗’？

How does joining the OpenAI board work? Does Sam just call you up and go, 'Hey, love the whiteboard, fancy coming on our board'?

实际上，在我开始担任系主任的前一天，我收到了董事会主席 Brett 的邮件，只是说：‘嘿，你想谈谈可能加入 OpenAI 董事会吗？’所以我想，反正我已经在经历一次重大的职业转变了，为什么不加倍下注，同时做两件事呢？但基本上，我开始和他以及董事会其他成员进行了一些对话。我对有机会向董事会提供我在 AI 和 AI 安全方面的观点感到非常兴奋，事情就这样开始了。

Actually, the day before I started as department head, I got an email from Brett, the chair of the board, just saying, 'Hey, do you want to talk about maybe joining the OpenAI board?' So I figured, you know, I was already embarking on one massive career change, so why not double down and just do two at the same time? But basically, I started having some conversations with him and the rest of the board. I got very excited about the potential to provide my perspectives on AI and AI safety to the board, and things went from there.

角色和职责是什么？他们是不是规定了一年四次董事会会议，中间还有饼干和咖啡？

What are the roles and responsibilities? Do they set them out like four board meetings a year and biscuits and coffee in between?

是的，一年有四次董事会会议。但我认为我被邀请加入董事会是因为我是 AI 和 AI 安全方面的专家，我很高兴能向其他董事会成员提供我关于 AI 的观点和专业知识。

There are four board meetings a year, yes. But I think I'm being brought on the board as an expert in AI and AI safety, and I am excited to provide my perspective and expertise specifically on AI to the rest of the board.

你相信‘中国在 AI 进展上落后美国两年’这个说法吗？

Do you believe the statement that China is two years behind the US in terms of AI progression?

这里绝对有一些国家之间争夺 AI 主导地位的竞赛因素。但我实际上会采取不同的立场，我认为有些事情，比如 AI 安全，非常需要全世界共同努力来制定标准，帮助改善每个人的未来。因为是的，某些事情可以由单个国家完成——能力可能由单个国家推进得更快——但安全本质上是全球性的。所以我们需要共同努力来构建安全的 AI 系统。

There are absolutely some elements here of a race between different countries for AI dominance. But what I will actually take a different stance on this and say that I think there are certain things, like for example AI safety, where we very much need to work as a world to help set standards and help better the future of everyone here. Because yes, certain things can be done by countries—capabilities can maybe advance more by countries—but safety is something that is inherently global. So we need to work together to build safe AI systems.

你被问到的最常见但你认为不应该被问的问题是什么？

What is the most common question you are asked that you don't think you should be asked?

最常见的问题与过分强调架构有关，那些过分强调 AI 系统中架构的问题。比如认为 Transformer 是让所有 AI 成为可能的关键。我经常被问到‘Transformer 之后是什么？’之类的问题。而现实是，正如我之前所说，这也许是个很好的金句，我们可以说已经处于后架构阶段了。架构并不重要。我们现在可以用很多不同的架构来做同样的事情。我希望我能把对话引导到另一个方向，让我们不是根据模型的特定结构来考虑它们——因为就能力而言，结构有点无关紧要——而是更多地根据输入的数据和下游产生的能力来思考这些模型。

The most common one has to do with things like putting an overemphasis on the architecture questions, questions that put an overemphasis on the architectures involved in AI systems. So this notion that somehow the Transformer was the thing that made all AI possible. I'm often asked questions like, 'What comes after the Transformer?' and things like that. And the reality is, as I said before, which probably makes for a good sound bite, we are arguably in a post-architecture phase. They don't really matter. We could do what we're currently doing with a whole lot of architectures right now. And I hope that I can steer the conversation to more of one where we consider these models not in terms of their particular structure—because it's somewhat irrelevant when it comes to capabilities—and we think about these models more in terms of the data that goes into them and the capabilities they produce downstream.

我这番漫无边际的谈话……我真的很享受。我很高兴你的日程安排没有太紧，不然我就惨了。非常感谢你如此出色。

My meandering conversation... I've so enjoyed this. I'm so glad you didn't have too much time with the schedule, otherwise I would have been screwed. But thank you so much for being so brilliant.

太好了。非常感谢你的邀请。嗯，我希望我不期待你不得不把这一切剪辑在一起，把我散乱的思绪整理成听起来连贯的东西，但你知道，我相信你会做得很好。

Great. Well, thank you very much for inviting me. And yeah, I hope I don't look forward to you having to edit all this together to sort of form my rambling thoughts into something that sounds coherent, but you know, I'm sure you'll do a great job with this.