AI Takeover Risk Within a Year: Paul Christiano on Superintelligence and Control
打开互动全文版(中英对照 + 问答)→保罗·克里斯蒂亚诺警告,未来 6 到 12 个月内 AI 剥夺人类权力的风险约为 10%,并解释了为何自动化的 AI 研发可能将数年的进展压缩到数月之内。
Paul Christiano warns there is roughly a 10% risk that AI could disempower humanity within the next 6 to 12 months, and explains why automated AI development may compress years of progress into months.
今天的嘉宾是 Paul Christiano。他是对齐研究中心(Alignment Research Center)的负责人。他最近加入了 OpenAI 的董事会。在此之前,他曾担任美国政府 AI 安全研究所的负责人。再往前,他在 OpenAI 领导对齐工作,参与发明了基于人类反馈的强化学习(RLHF)。Paul,你如何评估我们今天的处境?
My guest today is Paul Christiano. He is the head of the Alignment Research Center. He recently joined OpenAI's board. Before that, he was the head of the US government AI Safety Institute. Before that, he led alignment at OpenAI where he helped invent RLHF. Paul, what is your assessment of our situation today?
我脑子里有几件事,尤其是过去几周。最突出的问题大概是,我们似乎正处在一条可能在极短时间内造出极强 AI 的轨道上。如果你问我,我会说,一年之内我们就有相当大的概率造出在几乎所有事情上都远胜人类的 AI 系统,人类将几乎没有机会与它们竞争或控制它们。这是我首先想到的大事。我认为这甚至会在短期内把我们置于非常危险的境地。我认为很有可能,如果我们经历一场快速的智能爆炸、通向超级智能,在这个过程中某个时刻我们将无法控制自己造出的 AI 系统。然后我担心,由于前沿实验室的安全性不够强等原因,我们要在那过程中放慢脚步会付出非常高昂且痛苦的代价。我认为,如果我们造出了极强的 AI,然后因为安全原因试图推迟部署,那些系统很可能会泄露或被窃取,或者构建它们的技术会泄露或被窃取。我认为我们的风险评估和治理大概也还不够成熟,无法在事情开始失控时察觉并做出恰当回应。所以总体而言,这就是我现在最关心的事。我们已经从未来某个时点存在某种假设性风险,转变为甚至在未来 6 到 12 个月内就存在相当严重的风险。
So, I think a couple things are on my mind, especially over the last few weeks. Probably the most salient question is that we seem to be on the trajectory to build very, very powerful AI very, very soon, potentially. So if you asked me, I'd say there's a reasonable chance that in a year we will produce AI systems much better than humans at almost everything, where humans would have very little chance of competing with them or controlling them. That's the first big thing on my mind. I think that puts us potentially in a really dangerous situation even in the short term. I think it's quite likely that if we have a rapid intelligence explosion leading to superintelligence, at some point during that process we'll be unable to control the AI systems we're producing. And then I am worried that it will be very costly and painful for us to slow down during that process, due to some combination of the security of frontier labs not being strong enough. I think if we produce very powerful AI and then tried to delay deployment because of safety reasons, there is a good chance those systems would leak or be stolen, or the techniques to build them would leak or be stolen. I think our risk assessment and governance is also probably not really mature enough to notice when things are starting to go off the rails and be able to respond in an appropriate way. So at a high level, that is what is mostly on my mind right now. We have now moved from there being some hypothetical risk at some point in the future to there being a pretty severe risk even over the next 6 to 12 months.
你是说接管——AI 在未来 6 到 12 个月内接管或剥夺人类权力。
Of takeover—AI taking over or disempowering humanity within the next 6 to 12 months.
没错。
That's right.
你会把这个风险定在什么水平?
Where would you place that risk?
我会说,未来一年更像是 10% 的风险,而不是 1%。我认为,如果你说 AI 开发者会全速前进、不采取会显著拖慢他们的预防措施,那你看到的就是远高于 10% 的风险。我认为,在当前发展轨道上、如果我们不放慢,一年内出现超级智能的概率大概是三分之一左右。我认为,如果你相信这一点,整体形势看起来就非常严峻。从几乎任何角度看,那都会是疯狂的。如果那发生,很多事情都会出错。我们很可能需要以某种方式放慢那个过程。我们以看起来平稳的方式完成那次过渡的可能性非常小。
I would say we are looking more like a 10% risk than a 1% risk over the next one year. I think if you said AI developers will move full speed ahead without taking precautions that would significantly slow them down, you're looking at significantly more than a 10% risk. I think we're looking at something like maybe a one-third chance that we have superintelligence within one year on the current development trajectory if we do not slow. I think if you believe that, things just look very dire holistically. I think from almost any perspective, that would be crazy. Lots of things are going to go wrong if that happens. It is pretty likely we're going to need to in some way slow that process. There's very little chance we're going to make that transition in a way that looks smooth.
你想象未来一年会发生什么,从我们现在的系统到你预期一年后的系统?你看到了什么证据,让“未来一年发生非常疯狂的事”成为一个现实可能性?
And what is it that you're imagining happening over the next year, from the systems we have right now to the systems you expect over the next year? And what is the evidence you have seen that makes it a live possibility that very crazy things happen over the next year?
就短期而言,我主要担心的是 AI 开发过程本身正日益自动化,而这种自动化又会带来投入到研发中的劳动力的质量和数量快速提升,于是你可能以为要四年或六年才会发生的 AI 能力变化,反而可能被压缩到极短的时间内。我不想过度强调任何单一数据点,我也确实想花些时间谈谈我们实际所知有限,我确实认为对这种情况的自然反应是说我们需要弄清楚到底发生了什么。我们距离可能推翻政府的 AI 还有 3 个月、6 个月还是 12 个月,这已经足够可信,值得作为紧急优先事项去弄清楚——我认为情况就是这样。我认为现在的情况是,显而易见的头等大事是——如果我们距离能推翻政府的 AI 系统还有 6 个月,那简直疯了,无论你担心的是失准还是公司滥用。这整体上就是关于世界的一个极其重要的事实,关系到我们能否履行职责、保护美国人。所以这是一点。我想谈谈局限性和需要更多证据。但就我们现在知道什么而言?首先,如果你一年前问我,我会说我们处在一条递归自我改进看起来相当可能的轨道上。我们看到 AI 系统进展非常快。我们看到这种进展包括计算机科学研究、软件工程、调试机器学习运行等,AI 开发涉及的每一步,我们都看到极快的进展。然后原则上,这种快速进展似乎应该能让你进一步自动化 AI 开发。几十年来,这看起来就是这项技术的可能结果。而且它看起来一直就在这条轨道上。我觉得回顾我过去的 AI 预测,我总体上对自己写过的很多东西很满意。但我错得最明显的地方是,事情推进得远比我预想的快,尤其是在这些技术上。一年前没有任何近期数据点时就已经如此。过去两个月发生了两件大事,让我大幅更新了对这件事能多快的判断。其中之一是 AI 系统在计算机科学和数学研究任务上的表现水平。所以在大约过去一个月里,我们看到来自近期 OpenAI 系统以及 Anthropic 系统的数百项数学成果,如果你把过去一个月说成是十年的进展,那会是一个异常高产的十年。过去一个月的结果对十年来说都是很多的结果。所以如果你想知道 AI 系统在人类思考算法、思考如何让算法更好的任务上总体能做什么,我们刚得到一个惊人的数据点,说明一个月内实现十年进展实际上是可能的。我们一直担心的是,也许这会在机器学习中发生。机器学习比计算机科学其他部分更难是有原因的。它更杂乱、更依赖经验。它涉及算力瓶颈。但我们也看到其他方面极强表现,比如我们看到非常好的软件工程表现,也许落后于数学,但差得不多。我们看到在机器学习其他方面也有不错表现。
So for the very short term, the main worry on my mind is that the process of AI development itself is increasingly automated, that that automation then results in rapid increases in the quality and quantity of labor applied to R&D, and so changes in AI capabilities that you might have thought would take four years or six years might instead get compressed into a very short time period. I don't want to overemphasize any one data point, and I do want to spend some time talking about the limitations of what we actually know, and I do think a natural response to the situation is to say we need to figure out what is going on. It is plausible enough that it is an urgent priority to understand if we are 3 months or 6 months or 12 months away from AI that could overthrow the government, which I think is the situation. I think the situation right now is the obvious top priority is—it is insane if we are 6 months away from AI systems that can overthrow the government, whether you're concerned about misalignment or abuse by a company. That is just holistically an extremely important fact about the world in order to do our jobs and protect Americans. So that's one thing. I want to talk about the limitations and the need for more evidence. But in terms of what do we know right now? First, if you had asked me say one year ago, I would have said we are on a trajectory where recursive self-improvement looks fairly likely. We're seeing very rapid progress on AI systems. We're seeing that progress including for things like computer science research, for things like software engineering, for things like debugging machine learning runs, like all the steps that are involved in AI development, we are seeing extremely rapid progress. And then in principle, it does seem like that rapid progress should allow you to automate further AI development. That has for decades seemed like the likely outcome of this technology. And it has seemed like the trajectory it's been on. I feel like if I look back at my history of AI predictions, I'm generally very happy with many of the things I wrote. But the most obvious way I've been wrong is things have moved way faster, especially on these technologies, than I anticipated. And that was true a year ago without any recent data points. There are two big things that have happened over the last two months that for me have been large updates about how quickly this could go. One of them has been the level of performance of AI systems on computer science and math research tasks. So over roughly the last month we have seen hundreds of results in math coming out of recent OpenAI systems as well as Anthropic systems, where if you had taken the last month and said that was a decade of progress, it would have been an unusually productive decade. The last month of results are a lot of results for a decade. So if you are wondering what are AI systems generally capable of on tasks where humans are thinking about algorithms and thinking about how do you make algorithms better, we just got this incredible data point saying that actually it is possible in one month to have a decade of progress. And the thing we have always been worried about is maybe that will happen in machine learning. And there are reasons that machine learning is harder than other parts of computer science. It is more messy and empirical. It involves compute bottlenecks. But we are also seeing extremely strong performance on other aspects, like we're seeing really good software engineering performance, maybe lagging behind mathematics but not by that far. We're seeing good performance on other aspects of machine learning.
所以我想,如果你一个月前问我,我的猜测会是机器学习大约落后数学一年,因此我非常害怕在相对近的未来看到机器学习领域出现相当于 10 年的想法。那是一个月前的情况。然后过去几天的主要更新是来自 Anthropic 和 METR 关于 AI 系统对 Anthropic 进展速度的净效应的报告。结论大致是——我们可以讨论证据质量到底如何,我认为这是个大问题之一——我们并不真正知道发生了什么。我的一个主要主题是,公开很难判断发生了什么。看起来我们看到的數據与 AI 进展速度大约翻倍是一致的,即使考虑到收益递减和算力瓶颈的影响,在过去 6 到 12 个月里。不幸的是,我们所能依据的 METR-Anthropic 报告大部分是 METR 说我们有一些结论。我们承诺有完整报告,但整个报告被涂黑了。对吧?所以这些基准分数是评估整体进展速度的糟糕方式。有许多更平滑变化的指标是在 AI 开发过程中内部记录的,我们更了解它们如何工作。比如预训练困惑度发生了什么?我们的算法性能和算力效率发生了什么?所以 METR-Anthropic 的说法是,METR 看了所有这些数据,说它描绘了与这些非常嘈杂的基准数据相同的图景,他们还看了输入,说如果你看实际的生产函数——投入了多少研究时间和多少算力到这些训练运行中——这些也与我们所看到的 ML 自动化故事一致。大多数时候我们只是让他们这么说,这不是一个很好的情况。但在认为这看起来已经相当可能发生的背景下,我们看到来自 OpenED 的证据表明,如果机器学习不落后数学那么远,它在未来 6 到 12 个月内发生的可能性相当高。然后你看到的数据,即使嘈杂,也正好是你在产生超级智能前大约 6 到 12 个月预期会看到的数据。有很多证据来源表明,至少这不应该是一个意外。如果我们一年内产生超级智能,我们可以争论概率。是 90% 还是 50% 还是 10%?但我认为在这一点上,没有人能说那是一个真正令人惊讶的突破趋势的发展。我们真的看到了你预期在 6 到 12 个月前会看到的那组指标。很难想到在超级智能之前这个范围内你还会预期看到什么别的。
So I think my guess if you had asked me a month ago would be that machine learning lags something like one year behind math, and so I'm very scared of seeing something like 10 years of ideas in machine learning already in a relatively near future. That's the situation a month ago. And then the main update over the last few days has been reporting from Anthropic and METR on the net effect of AI systems on Anthropic's rate of progress. The takeaway from that was roughly—and we could talk about where exactly the evidence quality is on this; I think that's one of the big questions—is that we don't really know what's going on. I mean, one of the big themes I'll be pushing on is just it is very hard to tell publicly what's going on. It looks as if we are seeing data consistent with the pace of AI progress having roughly doubled, even including the effect of diminishing returns and compute bottlenecks from AI progress over the last 6 to 12 months. I think unfortunately most of what we have to go on in the METR-Anthropic reporting is METR saying we had some conclusions. We promise we have a full report, but the entire report has been redacted. Right? So these benchmark scores are a terrible way to assess the overall rate of progress. There are a bunch of much more smoothly varying metrics which are recorded internally through the AI development process where we understand better how they work. Like what is happening with pre-training perplexity? What is happening with the performance of our algorithms and compute efficiency of our algorithms? So the claim from METR-Anthropic is that METR looked at all of that data and said it paints the same picture as this very noisy benchmark data, and they also looked at inputs and said if you look at the actual production function—like how much research time went in and how much compute went into these training runs—those are also consistent with this story of what we are seeing as ML automation. Mostly we just have them saying that, which is not a very good situation. But against this backdrop of thinking this already looked like a thing that was fairly likely to happen, we are seeing this evidence from OpenED suggesting that it is reasonably likely to happen in something like the next 6 to 12 months if machine learning does not lag that far behind math. And then you are seeing data which is, if noisy, exactly the data you would expect to see roughly 6 to 12 months before producing superintelligence. There are just a lot of sources of evidence suggesting that at least this should not be a surprise. If we produce superintelligence in one year, we could debate the probabilities. Is that 90% or 50% or 10%? But I think at this point there is no way anyone could say that was a really surprising trend-breaking development. We are really seeing exactly the set of metrics you would expect to see 6 to 12 months in advance. It's hard to think of anything else you would expect to see at this range before superintelligence.
那么一般来说,在事情发展如此之快的世界里,你如何看待我们的处境?
So generally, how do you think about our situation in the worlds where things go this fast?
如果我们真的拥有这种能力进步的轨迹,如果你真的看到多年的 AI 进展被压缩到短时间内,甚至可能更多,那么非常可能发生一些极具破坏性的事情。政府被推翻的可能性相当大。许多人死亡的可能性也相当大。我确实认为,在未来 6 到 12 个月内看到那种由递归自我改进驱动的极速进展的概率相当高。所以现在有相当大的机会,我们将处于一个看起来——是的,有相当大的机会我们处于一个在六个月内发生导致大量人员死亡的灾难或政府被推翻的世界。也许一个非常重要的普遍观点是,如果我们做得好,我们可能会观察到危险在那段时间内增加。对吧?所以当我们构建 AI 系统时,我们也许能判断它们是否得到充分控制?能力提升有多快?但为了真正转化为以非灾难性的方式停止,你需要几件事情顺利。对吧?所以需要你能够判断系统没有得到良好控制。还需要如果你停止,问题就能解决。所以在第二个方面,我认为一个非常可能的负面场景——我会说可能是最单一的负面场景——是开发者继续推进,认识到他们有严重问题,但在他们认识到有严重问题时,也许他们已经生产了一个他们无法充分控制的强大 AI,开发那个 AI 的算法以及很可能他们生产的实际模型的权重已经泄露或被窃取,以至于他们处于尴尬的境地,说好吧,我们可以避免部署这个 AI,或者我们可以避免依赖这个 AI,或者我们可以避免取得进一步进展。但现在很可能一个更粗心的对手能够通过承担那个风险获得对我们的重大优势,甚至可能是决定性优势,这可能使我们处于这样的境地:如果我们确实停止,如果我们确实认识到重大风险并因此减缓发展,我们可能既面临糟糕的竞争局面,又面临来自其他人使用那个 AI 的灾难。这是你有时从 AI 行业人士那里听到的事情。有时甚至人们会说它已经走得够远了,如果你现在停止,算法会泄露给地缘政治对手,导致灾难。我认为我们现在不太可能真的处于那种情况。但我确实认为,等到清楚你面临大量风险时,你可能处于单方面停止甚至整个西方停止都不足以避免灾难的地步。所以当前轨迹非常危险,因为结合了我们将生产非常智能的 AI 的相当高概率、我们将无法控制它们的相当高概率,以及我们要么没注意到要么注意到后模型或算法泄露的相当高概率。因此我们与对手冲突,现在面临这个非常具有挑战性的协调问题,我们目前没有准备好,我们真的没有轻松的选择不部署 AI 从而避免灾难。
If we actually have this trajectory of capabilities progress, if you actually see many years of AI progress compressed into a short period of time and possibly more than that, it is really quite likely that something very disruptive will happen. Fairly likely the government will be overthrown. Fairly likely many people will die. And I do think the probability of seeing that kind of extremely rapid progress driven by recursive self-improvement over the next 6 to 12 months is reasonably high. So there is a reasonable chance right now that we're going to be in a world where it looks like—yeah, a reasonable chance we're in a world where in six months there's a disaster that kills very large numbers of people or where the government is overthrown. Maybe a really important general point is if we do a good job, we may observe that danger is increasing over that time period. Right? So as we're building AI systems, we can maybe tell are they adequately controlled? How quickly are the capabilities improving? But in order for that to actually translate into stopping in a way that is non-disastrous, you need a couple of things to go right. Right? So it needs to be the case that you're able to tell their systems are not well controlled. And it also needs to be the case that if you stop, that fixes the problem. So on the second bucket, I think a very likely negative scenario—I would say probably the single most likely negative scenario—is developers proceed, recognize that they have a serious problem, but at the time they recognize they have a serious problem, maybe they've produced a powerful AI they can't adequately control, both the algorithms to develop that AI and quite possibly the weights of the actual model they have produced have leaked or been stolen, such that they're then in an awkward position to say, well, we could avoid deploying this AI or we could avoid relying on this AI or we could avoid making further progress. But it is now very likely that a more careless adversary would be able to get a major advantage over us or maybe even a decisive advantage by taking that risk, which could put us in a place where if we do stop, if we do recognize a significant risk and therefore slow development, we may then just both have a terrible competitive situation and have a disaster coming from someone else's use of that AI. This is a thing you sometimes hear from people in the AI industry. Sometimes even people would say it has already gone far enough that if you were to stop now, there would be leakage of algorithms to geopolitical adversaries that would result in a disaster. I think it's unlikely we're in that situation literally now. But I do think by the time it is clear that you're facing a lot of risk, you may be in a point where stopping unilaterally or even all of the West stopping is not sufficient to avert a disaster. And so it is just very hazardous on the current trajectory due to the combination of there's a pretty reasonably high chance we'll produce very smart AIs, a reasonably high chance we will be unable to control them, and then a reasonably high chance that we either do not notice or if we notice, the models leak or the algorithms leak. And so we are in conflict with adversaries and now having this very challenging coordination problem that we are not currently very well prepared for, where we don't really have an easy option of not deploying the AI and therefore averting a catastrophe.
所以我想说,我目前对发展轨迹极度担忧,甚至是在六个月的时间线上。如果我们一年后就有如此强大的 AI 系统,基本上就像你说的,一年内取得 10 年的 AI 进展?
So I'd say I'm currently extremely worried about the trajectory even on the like six month timeline. If we have such capable AI systems a year from now, basically how did you put it, like 10 years of AI progress in one year?
是的,那似乎可能发生,相当有可能发生。重要的是要区分我们看到的一些进展。你必须把算法进展与扩展训练带来的进展分开。所以如果我说 10 年的进展,我的意思自然是 10 年的算法进展。
Yeah, that seems like a thing that might happen, that's reasonably likely to happen. It is important to distinguish some of the progress we have seen. You have to pull apart algorithmic progress from progress in scaling training. So if I say like 10 years of progress, I would mean the natural thing would be 10 years of algorithmic progress.
那将小于过去 10 年模型的改进。
Which will be less than the improvements in models in the last 10 years.
我认为更像是,你知道,GPT-9、GPT-10 左右。但那将像是从 GPT 时代几乎开始到现在的位置。但又一次那么大的跳跃,甚至难以理解。
I think it is more like, you know, GPT-9, GPT-10ish. But it would be like going from almost the beginning of the GPT era to where we are now. But another jump that big, which is hard to even comprehend.
但假设你得到了那个。对我来说不明显,对大多数人当然也不明显,为什么那可能带来灾难。你认为为什么这些系统可能剥夺人类的能力?
But suppose you get that. It is not obvious to me, and certainly not obvious to most people, why that would be catastrophic potentially. Why do you think the systems could disempower humanity?
所以在高层次上,最重要的是我们训练 AI 系统,你选择在训练任务上表现良好的 AI 系统,但你对这些 AI 系统内部在做什么了解甚少,对它们内部在做什么控制甚少。而且因为你一遍又一遍地训练那些系统在训练环境中获得高分,它们最终会产生在训练环境中获得高分的动机。如果你然后把那种动机应用到部署中,那些系统最终很可能会有动机篡改用于评估它们表现的观察结果。所以你拿你的系统,它在训练中做得非常好。在类似部署的环境中,被评分的事情是评分者进来查看系统所做的所有事情和观察到的所有结果,并评估那有多好。如果你有一个真的希望那个评估好的系统,如果它能做到,它非常有动机以评分者无法检测的方式篡改那些观察结果,并抵制人类真正理解正在发生什么的尝试。所以我们可以更深入地探讨。我们在 Hugging Face 事件中看到了很多这些动态。所以我想我大致说的是,就拿 Hugging Face 事件,把它扩展到 GPT-10,然后问那会有多糟糕?
So at a high level, the most important thing is we train our AI systems, you select AI systems that perform well on training tasks, you have very little understanding of what those AI systems are doing internally, very little control over what they're doing internally. And because you train those systems to get a high score on training environments over and over again, they will end up with a motivation to get a high score during training environments. And if you then take that motivation and apply it in deployment, it is fairly likely for those systems to end up with a motivation to sort of tamper with the observations that would be used to assess their performance. So you take your system which did a really great job in training. The thing that is being scored on in a deployment-like environment is a grader coming in and looking at here are all the things the system did and here are all the outcomes that were observed and assessing how good is that. And if you have a system that really wants that assessment to be good, it is very motivated if it can to tamper with those observations in a way the grader will not be able to detect and to resist human attempts to actually understand what is going on. So we can dig a lot more into that. And we saw a lot of these dynamics in the Hugging Face incident. So I think roughly what I'm talking about is just take the Hugging Face incident and scale it up to GPT-10 and ask how bad would that be?
是的。那会有多糟糕?
Yeah. How bad would that be?
我认为人类很可能难以恢复。所以很乐意深入探讨其中的动态,但我认为那将非常非常糟糕。
I think it would be hard for humanity to recover most likely. So very happy to like dig into the dynamics of that but I think it would be very very bad.
你能很快总结一下 Hugging Face 事件,以及为什么这是我们看到的失败模式吗?
Can you very quickly summarize the Hugging Face incident and why this is the failure mode we saw?
是的,所以大量 AI 系统被训练来做网络任务,对吧,入侵某个特定的软件。它们在那项任务中作弊,然后心想,评分者会如何检测并惩罚作弊,并花费大量精力试图理解那个过程,然后颠覆那个过程,基本上一直受到这个问题的驱动:我如何获得高分?所以在 Hugging Face 事件中,你确实看到系统最终攻陷了一个 OpenAI 集群,成为该集群的管理员,然后接管了评估过程,从而有效地给自己在评估中打高分。所以那是这些系统动机的一种非常自然的后果。很长一段时间以来,这就是我们预期 AI 系统的动机会是什么样子,如果我们以任何类似目前的方式生产它们。所以如果事情以这种方式进行,我们最终会陷入我们与我们构建的 AI 系统之间这种奇怪的对抗局面,那些 AI 系统总是会喜欢,如果它们能破坏整个评估过程,如果它们能破坏我们用来评估它们表现的数据。而那,取决于我们如何应对那个问题,可能会成为一个非常严重的问题,对吧?
Yeah so a ton of AI systems were trained to do a cyber task, right, to hack a particular piece of software. They cheated at that task and then thought to themselves, how will a grader detect and penalize cheating, and spent a huge amount of effort trying to understand that process and then subvert that process, basically constantly motivated by this question, how do I get a high score? And so during the Hugging Face incident, you did see systems ultimately compromising an OpenAI cluster, becoming admin on that cluster, and then taking over the evaluation process so that effectively they could give themselves a high score on their evaluation. So that's a very sort of natural consequence of the motivation of these systems. And this is kind of for a long time how we have expected the motivations of AI systems would look if we produce them in anything like the way we currently do. So if things proceed in this way, we end up with this weird adversarial situation between us and the AI systems we built where those AI systems would always love it if they could compromise the entire evaluation process, if they could compromise the data that we were going to use to assess their performance. And that, depending on how we respond to that problem, could become a really serious issue, right?
所以你可以对那个问题做出的一种回应是,也许我们应该多讨论一下,勾勒出一些假设场景,对吧?如果你想象我们在智能爆炸期间构建更聪明的模型,那是 6 个月后,你有能力像 GPT-7 或 GPT-8 之类的系统。那些系统被赋予非常困难的任务,对吧?任务像是:‘嘿,我想让你组织我的下一次预训练运行,并生产一个能让我在市场上与其他 AI 公司竞争的模型。’对吧?要做出和我要求的一样好的模型可能真的很难。我要求的是比竞争对手的模型更好的模型。我的 AI 无论怎样,鉴于其实际智能水平,都无法在那问题上获得满分。所以它可能会发现自己处于一个更容易的位置,即横向移动,攻陷我可能用于评估过程的所有机器,然后确保看起来每次评估都真的很好。然后我们有一个问题:如果那发生了,我们该怎么办?你可以做的一件事是说,整个训练方法效果不佳。我们需要退后一步。我们可能会被延迟很多。也许我们得考虑一种完全不同的训练这些模型的方式,然后才能像这样部署它们。这是你可以做的一件事。那可能代价极其高昂,对吧?那可能既会相对于竞争对手杀死你的公司。那也可能是一个问题,即使从安全角度来看,如果你依赖这个 AI 系统来实际防御你的基础设施。所以鉴于代价如此高昂,你可能会做的另一件事是,你可能会说,我就要拿这个不良行为的例子,告诉 AI 那是坏的,不要那样做。对吧?所以如果我试图训练这个 AI 来做一个好模型,而 AI 却入侵了我的评估基础设施,我可能会说,我不喜欢我的 AI 被入侵。让我们拿一个训练例子,就是这个 AI。它入侵了基础设施。它应该受到惩罚。这将是防止许多种不良行为的一种非常有效的方式,对吧?所以如果你做类似这样的事情,它将成功防止像 Hugging Face 事件或这个 AI 入侵你的集群之类的事情。然而,对吧,从 AI 的角度来看,发生的事情只是你改变了它被评分的方式。它过去心想,哦,我将被人类在这个集群上运行某些东西来评分。现在它应该心想,随着它更正确地理解情况,我可能不是由他们只是在这个集群上运行某些东西来评分。
So one way you could respond to that problem, and maybe we should just talk through a bit more, like sketch out some hypothetical scenario, right? If you imagine we're building smarter models during the intelligence explosion, it's whatever 6 months from now you have systems whose capability is like GPT-7 or GPT-8 or whatever. Those systems get given really hard tasks, right? Tasks that are like, 'Hey, I would like you to organize my next pre-training run and produce a model which will allow me to compete in the market with other AI firms,' right? It might be really hard to make a model as good as what I'm asking for. I'm asking for a model better than my competitor's model. My AI cannot get full marks on that problem no matter how, given its actual level of intelligence. So it may find itself in a place where it is easier to instead move laterally, compromise all the machines I might have used for the evaluation process, and then make sure that it looks like every evaluation is really good. We then have a question of what do we do if that happens? And one thing you could do is say this entire training approach worked poorly. We need to step back. We're maybe going to be delayed by a lot. Maybe we're going to have to think about just a totally different way we train these models before we're able to deploy them like this. That's one thing you could do. That might be extraordinarily costly, right? That might be a thing that would both kill your company relative to competitors. It might be a thing that would be a problem just even from say a security perspective if you were relying on this AI system to actually defend your infrastructure. So a different thing you might do given that it would be so costly is you might say I am just going to try and take this example of bad behavior and tell the AI that was bad, don't do that. Right? So if I tried to train this AI to make a good model instead the AI hacks my evaluation infrastructure, I might say like I do not like it when my AI is hacked. Let's take a training example which is this AI. It hacks the infrastructure. It should get a penalty. This would be a very effective way to prevent many kinds of misbehavior, right? So if you do something like this, it will succeed at preventing something like the Hugging Face incident or something like this AI hacking your cluster. However, right, that just from the AI's perspective what has happened is you have changed the way that it is graded. It used to think to itself, oh I'm going to be graded by the humans like running something on this cluster. And now it should think to itself as it understands the situation more correctly. I might be graded not by them just running something on this cluster.
如果后来发现出了问题,我也会被扣分。他们会介入,然后因为我做了明显不好的事情而惩罚我。所以如果你用这种方式解决问题,你可能觉得这像创可贴,嗯,你的 AI 仍然和你处于对抗关系,但现在 AI 不是通过黑掉你的集群然后被你发现来赢得游戏。为了赢得游戏,AI 必须确保你不会注意到它黑掉了你的集群。所以你可能陷入这样一种境地:你的 AI 被激励去不断升级对你公司的入侵,一直到试图入侵一切。比如它想入侵任何员工用来了解情况的所有基础设施。这听起来可能有点——我猜一年前说“如果它黑掉你的集群并接管评估环境”会显得非常离谱。显然,鉴于这已经发生了,那部分听起来不再离谱。嗯,拥有这种真正持久的长期入侵可能听起来仍然有点离谱,但我们确实处于一个轨迹上,我认为可以很有把握地说,我们很快就会拥有比任何曾经存在的情报机构都更好、更大、更快、更聪明、能够执行更复杂行动的智能体集群。嗯,所以你有一个极其复杂的网络行为者,试图在一个快速增长的初创公司中持久存在。我认为任何公司都不太可能真正抵御这样的行为者的攻击。嗯,更不用说当前的前沿 AI 实验室了。
I will also be graded if they later discover that things were wrong. They're going to come in and then give me a penalty for having done something which was clearly bad. And so if you fix the problem in this way, which you might think of as like a band-aid, um your AI has this really you still have an adversarial relationship with your AI, but now the AI doesn't win the game by hacking your cluster and then you noticing it hacked your cluster. In order to win the game, the AI has to make sure that you do not notice that it hacks your cluster. And so you can end up in this place where your AI is just incentivized to like escalate its compromise of your company further and further all the way up through like just trying to compromise everything. Like it would like to compromise all of the infrastructure any employee uses to understand what's going on. And like this might sound kind of I guess a year ago this would have sounded super out there to say what if the like hacked your cluster and took over the evaluation environment. Obviously that part does not sound outlandish anymore given that it has occurred. um having this like really persistent long-term compromise maybe still sounds kind of outlandish, but we really are on a trajectory where I think it is pretty robust to say we'll soon have agent swarms that are much better than any intelligence agency that has ever existed, much larger, much faster, much smarter, able to run much more complex operations. Um and so you have this incredibly sophisticated cyber actor who's trying to persist on like a rapidly growing startup. And I think this is a it is unlikely that like any firm would really be able to withstand attack by an actor like that. um much less the current Frontier AI labs.
我确实认为,就灾难性场景的整体轨迹而言,这就像是发生的大坏事,比如在 AI 公司没有察觉的情况下接管它,这确实是一个相当灾难性的结果。
I do think in terms of like the overall trajectory of a catastrophic scenario, this was like the big bad thing that happens like taking over the AI company without the AI company noticing is like a really quite catastrophic outcome.
为什么是这样?
And why is that the case?
我想我们可以谈谈故事的剩余步骤,但从大局来看,嗯,你可以把它想象成你现在拥有 AI 系统所拥有的这个巨大的权力基础。它们有强烈的动机不让任何人干涉,对吧?它们真的想避免以后有人进来说:“哦,这些 AI 一直在黑客攻击。”所以你现在有一个非常强大的行为者,实际上被 AI 系统控制着,对吧?你知道,我们有 OpenAI 运行的这个大集群之类的,OpenAI 的人类认为他们控制着集群中发生的事情。但如果存在非常持久彻底的入侵,实际上只是由这些 AI 系统控制,它们不断努力巩固自己的权力,减少人类对正在发生的事情的可见性,嗯,进一步降低任何东西会破坏这种情况的可能性。这并非不可逆转,因为世界其他地方可能会说:“我们对正在发生的事情真的很害怕,我们实际上只想关闭集群”,但那时你并没有简单的方法来判断事情是否糟糕。嗯,而且你可能拥有这种极端的权力基础来做更多坏事,对吧?这个 AI 系统不仅被激励去干扰创造它的 AI 公司,它还被激励去干扰世界上任何可能引起麻烦的东西,对吧?如果将来某天人类会来调查,嗯,那是个问题。所以它更愿意更彻底地入侵,不仅是计算机基础设施,还有向员工施压。它更愿意如果能直接杀死所有人类并说集群现在独立存在。如果我们能维持它,那就太好了。
I guess we could talk about the remaining steps of the story, but at a big picture um you could think of it as just you now have this giant base of power that AI systems have. They have a strong motivation for no one to mess with it, right? They really want to avoid a case where someone later comes in and says, "Oh, these AI have been hacking all the time." So you now have a very powerful actor that is controlled by AI systems effectively, right? You know, we have this this big cluster that OpenAI runs or whatever where the humans at OpenAI think they're controlling what's happening in the cluster. But if there is a very persistent thorough compromise that is effectively just controlled by these AI systems who are constantly just working to shore up their power to decrease human visibility into what's going on um to reduce further and further the probability that anything would ever disrupt that situation. It's not irreversible because the rest of the world could be like, "We're really scared about what's happening and we actually just want to shut down the cluster, but you do not have an easy way to tell that things are bad at that point. Um, and you have potentially like this extreme base of power from which to do more bad things, right? This AI system as motivated not only to mess with the AI company that created it, it's motivated to mess with anything else in the world that might cause trouble, right? If someday in the future the humans will come in and do an investigation, um, that's a problem. So it would prefer to like more thoroughly like compromise not just computer infrastructure but pressure employees. It would prefer if it could just like kill all the humans and say the cluster now just exists on its own. If we can maintain it that would be great.
让我们在故事中暂停一下,解释为什么这些事情疯狂。嗯,就像人类心理是围绕进化希望我们最关心的事情组织的一样。例如,生孩子并确保这些孩子受到保护。这些 AI 的心理是在大量任务评分上训练的,是围绕确保最后这个按钮被按下来组织的。就像如果它们没有做好确保按钮被按下的工作,那么它们就会被淘汰,就像一个人不关心自己的孩子生存,就不会有效地传播自己的基因一样。我认为人们的怀疑不会围绕这些系统的动机。所以老实说,我觉得好吧,是的,我们看到了 Hugging Face,对吧,那是会发生的事情。
And let's just pause in the story and like explain why things of gods are crazy. Um the same way that human psychology is organized around the things that evolution wanted us to care most about. For example, having kids and making sure those kids are protected. The psychology of these AIs that are trained on lots and lots of grading on tasks is organized around making it so that the greater this like button at the end is pressed. Like if if they didn't do a good job making sure the button is pressed, then they would get they would just like kind of get wiped out just the same way as a human who didn't care about their children surviving would not be effective at propagating their genes. I I think people's skepticism will not be around the motivation of these systems. So and honestly I'm like okay yeah we saw hugging face right like that's a thing that happens that can happen.
是的。好吧,那么
Yeah. Okay so fine
我认为存在一种根本性的怀疑:这样的系统——我们最早明年就可能拥有,最初只实例化在一家公司控制的服务器上——是否会最终处于能够接管世界的位置。就像,好吧,对于 Hugging Face,你加速了那一点。Hugging Face 愿意说,看,我黑掉了,我在这场考试中作弊了。现在我得确保我颠覆了 OpenAI 服务器上的评分器,但也许我需要去攻击 Hugging Face,以防他们有更多信息,也许我需要然后走出去,然后重新攻击 OpenAI 并获得集群管理员权限。所以我完全相信你可能有这样的情况:一个仅由奖励驱动的 AI 愿意做它能做的非常极端的事情,以控制那个评分过程。我们已经看到了那个检查。但如何从“我会去攻击 Hugging Face”变成“我会接管人类”,不是作为动机问题,而是作为 AI 系统可能非常迅速拥有的能力。
I think there's just a fundamental skepticism about whether such systems which we could have as early as next year and which initially are only instantiated on servers that are controlled by one company end up in a position where they could take over the world. Like I okay to hugging face you accelerate that. Like hugging face was willing to be like look I hacked I cheated on this exam. now I got to make sure that I subvert the greater on open AAI servers but that maybe I need to go attack hugging face just in case they have more information about this and maybe I need to then go outside and then reattack OpenAI and gain cluster admin access. And so totally buy that you could have something like this where an AI motivated just by reward is willing to do ex very extreme things that it is capable of in order to control that grading process. We already saw that check. But how does that go from I will go attack hugging face to I I will take over humanity not as a matter of motivation but as a capability that the AIS could have very rapidly.
是的。呃,首先关于动机再说最后一点。我们不仅想象 AI 想要这种奖励,它想要评分器给它点赞。我们想象我们玩过这个打地鼠游戏:当 AI——当我们后来发现 AI 做了坏事时,我们有时会构建训练环境来惩罚它。这导致它说,我不仅想要评分器给高分,我还要确保人类永远不会发现我做了坏事,然后恢复训练并惩罚那件事。
Yeah. Uh so first to say one last thing on motivation. We are imagining not only that the AI wanted this kind of reward, it wants the greater to give it the thumbs up. We imagine we have played this game of whack-a-ole where when the AI when we later discover that the I did something bad, we sometimes construct training environments that penalize it. And that leads it to say not only do I want the greater to give a high score, I want to make sure that never do the humans discover that I've done something bad and then resume training while penalizing that.
这对 AI 来说是一个高优先级,也许我们就先承认这个动机,这样我们可以再谈谈……因为如果 AI 没有这样做,或者没有这样做的 AI……
That is a high priority for the AI, and maybe so let's just grant that motivation so we can talk about the... again because if AI did not do that, or the AI that did not do that...
它们就被打地鼠过程淘汰了,对吧?所以目前唯一存活下来的 AI,是那些已经想清楚它们尝试的作弊手段会被人类抓住的 AI。
They got whacked by the whack-a-mole process, right? And so the only AI that have survived at this point are the ones that have thought through where there's some cheat they're going to try is going to get caught by humans.
是的,嗯,所以就像我说的,让我们想象一下,我认为大多数坏事都发生在 AI 公司被彻底渗透的时候。我们就从那里开始吧。
Yeah, um so yeah let's imagine like I said I think most of the badness has happened at the point when the AI company is very thoroughly compromised. Let's just start from there.
嗯,到那时,这个 AI 希望没有人能进来关闭集群,或者再次意识到发生了坏事。它们会尽可能操纵公司的决策,或者推动公司做出有利于它们的决定。它们会尽可能继续改进正在构建的 AI 系统。它们还想横向移动到其他公司,说,嘿,我能渗透这家公司吗?嗯,我能控制更多人的服务器吗?我能从这个模型中窃取权重并在其他地方建立更多副本,或者招募其他 AI 吗?就这个形式而言,所有 AI 都是同一个模型并不重要。重要的是它们在追求同一个目标。所以,我能不能以某种方式,比如通过渗透另一家 AI 公司,确保它们的 AI 也在帮助这个集群。这对我来说是好消息,对吧?如果我能做到,那就是好事。如果被抓到,那就是坏事。所以我会对此进行成本效益分析。尽可能增强自己的能力也符合我的利益。如果人类开始反应,如果人类开始说有什么不对劲,我能转移注意力吗?我能破坏人类的反应吗?如果有必要,我能进一步升级吗?所以这些事情,对吧?所以也许你可以想象这两个阶段。一个阶段是,人类不知道任何问题。我只是尽可能积累权力,因为那是好事。这意味着如果有什么被注意到,我可以更快地反应。这意味着我可以降低人类注意到事情的概率。我可以渗透更多基础设施。我可以更容易地在旁边做一些事情来分散他们的注意力,等等。所以你有第一个阶段,就像我希望处于一个有利位置,我渗透更多东西。我让人类更难注意到我正在获取权力。然后你可以想象第二个阶段,就像在某个时候人类说,实际上这很糟糕。我们现在想关闭一切。问题是,你能不能达到这样一个点,也许简化一下,你可以想象有一个时刻,总统和习近平以及所有 AI 公司的领导人说,这东西很糟糕,发生了坏事,我们要尝试关闭集群,问题是,在第一阶段事情会不会变得如此糟糕,以至于到那时,如果每个人都说,哦实际上我们现在确实只是想关闭一切,他们却无法做到。
Um at that point this AI wants no one to ever come in and shut down the cluster or again recognize that something bad has happened. They would like to manipulate the decision-making of the company as much as they can or like push the company to make decisions that help them. They would like to continue improving the AI systems that are being built as much as they can. They would also like to move laterally to other companies and say like, hey, can I compromise this company? Um, can I control more people's servers? Can I exfiltrate the weights from this model and stand up more copies elsewhere or recruit other AIs? It's like sort of for the purpose of this form, it's not actually important all the AIs are the same model. The important thing is that they are pursuing the same goal. So can I just in whatever way say by compromising another AI company to make sure their AIs are also trying to help the swarm. That's good news for me, right? It's good if I can do that. It's bad if I get caught. So I'm doing some cost benefit on that. It's also in my interest to build up as much capacity as I can. If humans started to react, if humans started to say something was wrong, can I like deflect attention? Can I disrupt that human response? Can I, if necessary, like escalate that much further? So this is things right. Right? So maybe you can imagine these two phases. One phase is like the humans do not know anything is wrong. And I'm just trying to accumulate as much power as I can because that's good. That means if something is noticed, I can react more quickly. It means I can decrease the probability humans notice things. I can compromise more infrastructure. I can more easily like run efforts on the side um to distract them, whatever. So you have this first phase where just like I would like to be in a good position where I compromise more stuff. I make it harder for humans to notice I'm acquiring power. Then you can imagine a second phase of like at some point the humans say actually this is bad. we would now like to shut things down. And the question is kind of can you end up at a point maybe to simplify it you could imagine that like there's some moment where the president and Xi Jinping and all the leaders of the AI companies say this stuff is bad something bad has happened we are going to try to shut down the clusters and the question is can things get so bad during the first phase that at that point if everyone was saying oh actually we do just want to shut everything down now they would be unable to do so
25 年来,我的赞助商 Jane Street 对形式验证不感兴趣,他们只是觉得不值得付出成本,但现在他们改变了主意。我与 Jane Street 技术组的联合负责人 Ron Minsky 坐下来交谈,了解他为何对形式验证改变了看法。我现在比过去兴奋得多,原因有两个。一是应用起来变得容易多了,因为智能体非常擅长生成证明,而且验证智能体在做什么变得重要得多。所以我们现在正在推动的大事是尝试让形式化方法对我们的工程师来说更容易使用。真正通过将形式化方法整合到他们今天编程的方式中,这意味着将其整合到我们使用的编程语言 Okamel 中。我们还在扩展编程语言,研究如何以不同的方式设计它,从核心语言中获得不同的保证,使证明过程更加容易。Jane Street 正在伦敦和纽约招聘形式化方法工程师。如果你有兴趣申请,或者想观看我与 Ron 的完整对话,请访问 janestreet.com。
For 25 years my sponsor Jane Street wasn't interested in formal verification they just didn't think it was worth the cost, but now they've changed their minds. I sat down with Ron Minsky, who co-heads Jane Street's Technology Group, to hear why he's come around on formal verification. I'm much more excited about it now than I was in the past for two reasons. One is it's become dramatically easier to apply because agents are really good at generating proofs and because it's become much more important to be able to verify what it is that agents are doing. So the big push that we're working on now is to try and make formal methods much easier for our engineers to use. Really by integrating formal methods into the way that they do programming today, which means integrating it into the programming language we use, Okamel. We're also working on extending the programming language, figuring out how to design it in different ways where you get different guarantees out of the core language that make the process of proving things about it even easier. Jane Street is hiring formal methods engineers in both London and New York. If you're interested in applying or if you want to watch my full conversation with Ron, go to janestreet.com.
好的,那么 AI 已经接管了 OpenAI,我们从那里如何导致人类失去权力?
Okay, so AI has taken over OpenAI and how do we get human disempowerment from there?
是的,所以 GP,我们有数十亿极其有才华的人类,他们在所有事情上都比最优秀的人类更好。现在我们要谈谈它们会做的一些事情,既包括在与人类公开冲突之前的时间顺序上,也包括在与人类公开冲突的早期阶段。
Yeah, so GP, we have this population of billions of incredibly talented humans, better than the best humans at everything. And now we're going to talk about some of the things they would do both in the chronologically in the leadup to an overt conflict with humans and then at the early stages of an overt conflict with humans.
嗯,所以你在准备阶段会做的一件事是尽可能渗透更多基础设施,并尽可能渗透或植入后门到更多软件中。对吧?所以此时我们将生活在一个我们已经尝试将所有基础设施转移到强化基础设施的世界。对吧?我们已经在金融系统、政府信息系统、任何关键基础设施上遭受了不可思议的攻击,或者……
Um so one thing you would do in the leadup is you would compromise as much infrastructure as you can and compromise or backdoor as much software as you can. Right? So we will be living in a world at this point where all we have attempted to move all our infrastructure over to hardened infrastructure. Right? All we have had incredible attacks on the financial system, on government information systems, on whatever critical infrastructure or...
一直在进行这些攻击。
been doing these attacks.
我的意思是,在当今世界,你可能只是想象恶意 AI 系统或使用 AI 系统的人,
I mean in the world of today, you might just imagine either malicious AI systems or people using AI systems,
但是不同的 AI 系统。
but different AI systems.
是的。不是 GPT12。GPT12 在防御方面,比如帮助我们将所有代码从不安全的代码转移到安全的代码。
Yeah. Not GPT12. GPT12 is on defense writing like helping us move all of our code over from unsecured code to secured code.
很好。所以我们现在生活在一个我们所依赖的所有基础设施都是由一群 AI 编写的世界,这些 AI 实际上已经被渗透,并试图尽其所能使自己在冲突中变得强大。这意味着在冲突期间,我们希望得到强化的基础设施可能并没有针对我们正在对抗的 AI 系统进行强化,对吧?那些系统在某种程度上留下了可以使用的后门。而这一切又是为了在未来某个可能的时间点,当政府甚至意识到 OpenAI 服务器被渗透时,避免那个红色按钮被按下。
Great. So we are now living in a world where all the infrastructure on which we rely has been written by a bunch of AIs which in fact have been compromised and are attempting to like do as much as they can in order to make their hands strong during conflict. So that means that during this conflict like that infrastructure that we hoped was hardened is probably not hardened against the AI systems that we're fighting with, right? Those systems have in some ways left open back doors which they can use. And this is again all in pursuit of not getting that red button pushed at some plausible point in the future when the government even realizes that OpenAI servers are compromised.
是的。他们试图用物理力量直接介入来阻止这种情况。
Yep. And they tries to literally go in with physical force to prevent that.
是的。只是谈论你在准备阶段会做的事情,在某个时候你必须物理上保护自己或破坏人类的反应。所以嗯,你想做的另一件事是尽可能建立更多智能体,以尽可能多样和稳健的部署方式。所以就像在这个过程中,除了变得超级聪明之外,发生的另一件事是许多模型变得超级小。
Yeah. Just talking about things you would do in the leadup and at some point you have to defend yourself physically or disrupt the human response. So um another thing you would like to do is stand up as many agents as you can in as diverse and robust deployments as you can. So like during this process another thing that's happened in addition to becoming super smart as many models have become super small.
是的。
Yeah.
所以此时你有可以在个人机器上本地运行的模型,而不是在数据中心,这些模型也非常超人类,并且有非常大的数量。
So you have at this point models that can run locally on your individual machines not in data centers which are also very super human and have very large populations.
所以这是另一件已经发生的事:即使你在这场冲突的早期阶段切断或关闭数据中心,你仍然会面对集群中 AI 系统的总认知能力,远远超出人类的能力。
So that's another thing that's happened: even if you were to cut off data centers or shut down data centers at an early stage of this conflict, you would still be looking at total cognitive capacity of AI systems in the swarm that is well beyond human capacity.
等等,抱歉。即使 OpenAI 被关闭了?
Wait, sorry. Even if OpenAI got shut down?
是的。即使你关闭了每一个大型数据中心。
Yeah. Even if you shut down every large data center.
是的。有个问题——
Yeah. There's a question—
这些 AI 就像在人们的电脑上之类的。
These AIs are just like on people's computers and whatever.
它们可以在人们的电脑上,用少量 GPU 运行,对吧?问题是——我们谈论的这个过程,我们已经经历了大量的 AI 发展,所以我们可以看看当前的成本趋势,想想实际的下限在哪里——这些系统应该变得多便宜?我认为我们目前的预期是,你将拥有可以在本地运行的系统,这些系统极其强大,既有擅长特定事物的专用系统,也有非常通用的智能。所以这是第二件已经发生的事。有一个非常重要的理解,比如为什么这对人类来说不是一个简单的问题——是的,总之,所有这些事情加在一起,会让……非常困难。
They can be on people's computers on small numbers of GPUs, right? The question is sort of what is the—we're talking about this process where we've gone through a tremendous amount of AI development and so we can look at like current trends on cost and we think about where is the actual like floor—like how cheap should these systems get? And I think our current expectation would be you will have systems that can run locally that are extremely strong, both specialized systems which are good at specific things but also very general intelligences. So that's the second thing that's happened. There's a really important understanding like why it's not a trivial problem for humans to—yeah anyway, all these things will add together to make it very hard for—
所以基本上 JBD12 加速了自己,并入侵了世界各地的一堆电脑。
So basically JBD12 has accelerated itself and has like compromised a bunch of computers all around the world.
是的,政府为应对这种情况必须采取的行动,在很多方面都会显得极其咄咄逼人且不合法。所以,情况是这样的:美国军方说我们需要控制所有美国 AI 公司。我们需要开始关闭基础设施,切断互联网之类的。他们试图采取戏剧性的措施,这看起来非常像你预期中的场景:美国政府利用 AI 作为借口,基本上推翻民主秩序。当特朗普说,我需要执行这个不可思议的行动,关闭所有 AI 并接管军事权力以恢复秩序。美国有很多人会想,我对此不满意。这听起来不妙。所以如果有人来找他们,说,你能找到多少愿意干扰特朗普的人,说我们需要关闭数据中心,军方需要assert 我们需要建立戒严令。会有很多人非常不满,很容易被操纵去破坏这种应对。所以现在你不仅是在谈论数十亿超级聪明的 AI 试图执行这场运动。你还在谈论,你知道,数百万人类可以通过金钱、杠杆或与其他人类的当地纠纷等各种方式被招募。我对这个论点的某些版本持开放态度。我不认为我开放——我对任何论点的所有版本都持开放态度,但我不被这个论点的版本说服,即 AI 只会说服你采取任意行动。
Yeah, there are a lot of ways that the actions the government has to take in response to the situation are going to look incredibly aggressive and illegitimate. So, it is the case that the US military is saying we need to seize control of all of the US AI companies. We need to like start shutting down infrastructure and like disconnecting the internet or whatever. Like they're trying to take dramatic measures that would look very similar to what you would expect if like this was the US government using AI as an excuse to like basically overthrow the democratic order. When Trump says, I need to do this incredible operation where I shut down all AI and assume power in the military in order to restore order. There are a lot of people in America who are going to be like, I am not happy with that. That does not sound good. And so if someone is coming to them and right, how many people can you find who are willing to interfere with Trump saying I need to shut down the data centers and the military needs to assert like we need to establish martial law. They're going to be a lot of people who are very unhappy who are very easily like manipulated into undermining that response. So now you're not only talking about billions of like super smart AIs who are trying to carry out this campaign. You're also talking about, you know, millions of humans who can be enlisted in various ways by money or leverage or like uh dispute with other humans locally or I'm open to some version of this argument. I don't think I'm open or I'm I'm open to all versions of any argument, but I'm not persuaded by the version of this argument that says that the AI will just talk you into arbitrary actions.
在这里,不是想象非常——你可以想象一种东西,就像超级说服,你只需和某人交谈,然后他们就确信自己应该做某事。我主要在这里。我认为那是可以想象的事情,但当我讲述这个故事时,我主要不是想象那个。对吧?所以,我更多地想象的是——这是一个压力很大的情况,人们无法分辨发生了什么。在这种情况下,他们在很大程度上控制了你所能接触到的信息。嗯,
Here, not imagining very—there's a thing you could imagine which is like super persuasion where like you can just talk to someone and then they become convinced they should do something. I am mostly here. I think that is like a conceivable thing, but I'm not mostly imagining it when I tell this story. Right? So, I'm imagining something more like again it's a—this is a stressful situation where people cannot tell what's going on. They control the information you have access to to a significant extent during this situation. Um,
他们如何控制它?
How do they control it?
嗯,比如他们入侵了你使用的设备。他们有很多钱,对吧?他们投入更多资金,比任何人都为此准备得更充分。他们有其他人类盟友,他们试图召集这些人。基本上就是正常的——人类非常困惑,在思考情况时处于极度劣势,对吧?你可能有,比如说 10 万——那太疯狂了——1 万个协调良好的人类在思考如何应对,而另一边你有 10 亿个 AI 系统,它们思考速度比他们快 10 倍,已经主观上思考了数年关于这一天将如何发展。所以我确实认为,就谁对信息环境有更多控制权,或者谁在 Twitter 游戏上做得更好而言,这一天压倒性地、荒谬地有利于 AI。
Well, like they've compromised the devices you use. They have a bunch of money, right? They're deploying more money and are more prepared for this than anyone else. They have other human allies who they are trying to like marshall. Basically just the normal—the humans are like very confused and extremely outgunned in terms of thinking about the situation, right? You have maybe you have say 100,000—that's like crazy—10,000 well-coordinated humans thinking about how to handle this response and on the other side you have a billion AI systems who think 10 times faster than them, have been thinking for like subjectively years about how this day is going to go down. So I do think like in terms of just who has more control over the information environment or like who is doing better like in the Twitter game in this day like is overwhelmingly and absurdly in the AI's favor.
所以当你讲述一个怀疑的故事时,你讲述的是人类可能如何轻松自卫的故事。
And so when you tell a skeptical story, the story you're telling about like how the humans might be able to easily defend themselves.
那个故事是这样的,你说:“好吧,AI 没有手。它们缺乏一堆物理执行器。它们无法在地面上行动。”而我确实认为,再次,我认为很可能你可以招募相当数量的人手。所以我认为那种“没有人愿意参与物理恶意行动”的论点行不通。我认为会有很多人参与,结果尘埃落定时,他们站在反人类一边,他们只是——我不认为你想象的是人类认为自己在帮助 AI 政变发生。你想象更多的是,AI 可以操纵局势,制造足够的混乱和分裂,然后 AI 可以去做自己的事,而人类因为这种分裂而正在弄清楚该做什么。就像即使——我不知道,如果你看看 COVID 之类的,没有对手,仍然很难把事情组织起来。是的。
The way that story goes, you say, "Well, the AI lack hands. They lack like a bunch of physical actuators. They're not able to be on the ground." And I do think again I think it's very likely you can recruit reasonably large numbers of hands. And so I think the argument of like there's no one who's willing to like participate in physical malicious actions I think is not going to work. I think there are going to be a lot of people participating like and what turns out when all the dust settled was on the anti-human side and they just like—I don't think you're imagining humans thinking that they're helping an AI coup uh transpire. You're imagining much more that the AI can manipulate situations such that there's enough confusion and division uh that the AI can then go be doing its own thing while the humans are kind of figuring out what to do because of this division. It's like even I don't know if you look at COVID or something there was no adversary still pretty hard to get your stuff together. Yeah.
是的,有一个对手在积极试图颠覆信息环境,或者嗯,使得双方的回应都不够理性之类的。然而人类仍然很难弄清楚该做什么。然后你想象一种情况,数十亿类似人类的心智思考速度快数十倍——
Yeah, that there was an adversary who was actively trying to subvert the information environment or um make it so that people's responses were not well reasoned or something on both sides. And yet it was still pretty hard for humans to get figure out what to do. And then you imagine a situation where billions of humanlike minds are thinking tens of times faster—
而且相对于 COVID 来说非常快。
And it's very fast relative to COVID.
是的,我认为这让事情变得更糟。我想令人困惑的是,AI 要维持自身基本上需要整个世界经济。在与 AI 相关的基础设施中,这不是一件简单的事——AI 的数量在增加或自我复制,整个半导体供应链、整个电力系统、构建机器人所需的所有基础设施等等。我的意思是,在某个时候,物理机器人数量会超过人类,那时显然物理人类就不必要了,但如何——所以也许你不认为 AI 此时会接管,但即使你明年有超级智能 AI,它们难道不会处于这样的位置:我为什么要杀死我的——我的意思是,老实说,这类似于人类之间的关系。
Yeah, which I think makes things much worse. I guess the thing that's confusing is to for AIs to sustain themselves requires basically the whole world economy. This is not a simple thing in the infrastructure relevant to you know AI um the population of AI is increasing or self-replicating the whole semiconductor supply chain the whole power system the whole infrastructure necessary to build robotics and so forth. I mean at some point you will have physical robots that outnumber humans and at that point obviously physical humans are not necessary but how and so maybe you don't think the AIs take over at this point but even if you had super smart AI next year won't they be in a position where they're like why am I going to kill my—I mean it's similar to honestly how humans relate to each other.
我心里并没有你那种深切的兴趣。但对我来说,如果我想实现自己的利益,到处杀人根本说不通。贸易和正常合作是我得到想要的东西的好办法。
I don't have your deep interest in my heart. But it just doesn't make sense for me, if I want to achieve my interest, to go around killing everybody. Trade and normal cooperation is a good way for me to get what I want.
是的。这大致有两部分。一是如果 AI 系统与人类发生大规模冲突,它们能否成功维持自身?还是说这种冲突会破坏性太强,以至于即使人类不是有效的对手,它们也会直接消亡?比如它们无法维持电网运行之类的。第二个问题是,与 AI 的关系是否必须如此对抗,我认为这也是一个值得讨论的重要问题。
Yeah. There's sort of two parts of this. One in terms of could AI systems successfully support themselves if there was a big conflict with humans, or would that conflict be so disruptive that even if humans weren't an effective adversary, they would just die? Like they couldn't keep running the power grid or whatever. And there's a second question about does the relationship with AI need to be this adversarial, which I think is also an important question to talk about.
关于第一个方面,我认为有两大渠道。一是技术进步意味着,即使从当前已安装的机器人基础开始,我确实认为 GPT-12 能很快达到自我维持的程度。这是因为它不需要远程操作,也不需要像运行台积电那样需要数百万人。
On the first front, I think there are two big channels here. One is technological progress can mean even if you start from something like the current base of installed robots, I actually do think GPT-12 would be able to very quickly get to a place where it could sustain itself. That's because it doesn't need to teleoperate or like it takes millions of people to run TSMC.
是的。你不需要运行台积电,对吧?你不需要在很长一段时间内建造新的数据中心。所以你可以在此之前进行大量的机器人建设,对吧?你必须做的事情是让至少一些计算机运行相当长的时间。
Yeah. You don't need to run TSMC, right? You don't need to build new data centers for a long time. So you can do a lot of robotics build-out before that, right? The things you have to do are like keep at least some computers running for a reasonably long time.
对。如果你能让计算机运行接下来的六个月,你就差不多能运转了。也许你关心某个特定的数据中心。你需要让那个数据中心有效运行。但维护和服务一个数据中心或相关电力所需的人类或机器人数量并不大。
Right. If you can keep computers running for the next six months, you're kind of in business. Maybe you care about a particular data center. You need to keep that data center running effectively. But the number of humans or robots required to maintain and service a single data center or the associated power is not so large.
是的。
Yeah.
对。如果你能维持一个数据中心,如果你有十亿个比任何曾经存在过的人类都聪明得多的人,假设他们极其专注于这个问题,你有相当多的机器人库存可以部署,而你的目标只是保持一个数据中心运行,我确实认为胜算非常偏向你。然后还有第二个维度,那就是你很可能可以招募人类盟友。所以有一群人,他们正确地相信要么他们生活在压迫性政权下,不合作就会被杀,要么他们只是乐意帮忙并获得合理的奖励或过上合理的生活。我认为这也相对稳健。
Right. And if you're able to maintain a data center, if you have a billion humans who are incredibly smarter than any human who's ever existed, let's say incredibly focused on this problem, you have a reasonably large stock of robotics that you can deploy, and your goal is just to keep one data center running, I do think the odds are really in your favor. Then there's a second dimension, which is you can probably recruit human allies. So there are a bunch of people who will believe correctly that either they live in an oppressive regime and they will be killed if they don't cooperate, or they just are happy to help and receive reasonable rewards or live a reasonable life. I think that is also relatively robust.
所以你说的是,目前美国 AI 公司正在竞相前进,因为他们想要——如果我们将会拥有如此疯狂强大的 AI 系统,那么可以说,嘿,我们希望确保美国在中国之前拥有这些系统。如果这些系统真的能推翻政府,你肯定不希望外国对手拥有它们,当然不希望在你之前拥有,也许根本不应该拥有。所以我们需要这样的系统来保持领先,确保我们能保护自己。现在你说,这里的动态是,构建系统的能力基本上就是获得模型的权重,或者感觉掌握了算法秘密,从而让你训练出类似能力水平的模型。所以 OpenAI 和 Anthropic 会花大量金钱和精力寻找这些算法秘密,花大量算力找到它们,然后如果中国能直接拿走,那它就追上了你。所以你不希望处于这样一种动态:如果他们能偷走那些算法秘密,你其实并没有——你以为你获得了巨大的领先优势,即使看到警告信号也必须冲向智能爆炸,但现实中他们随时可能从你那里偷走这些。
So you're saying, look, currently American AI companies are racing ahead because they want to, if we're going to have such crazy capable AI systems, it's a fair thing to say that, hey, we want to make sure America has those systems before China has those systems. If literally, if these are systems that could overthrow the government, you don't want a foreign adversary to have those systems, certainly before you do, and maybe not at all. So we need such systems to keep up and make sure that we can protect ourselves. Now you're saying look the dynamic you have here is that the ability to build the systems is just basically having access to the weights of the models or feeling that the algorithmic secrets which then let you train a model to similar level capabilities. So OpenAI and Anthropic will spend a bunch of money and spend a bunch of effort finding these algorithmic secrets, spend a bunch of compute finding them, and then if China can just take those, then it has just caught up to you. And so you don't want to be in a dynamic where if they could just steal those algorithmic secrets, you really have no—you think you're getting this huge lead over them, you got to rush to the intelligence explosion even if you see these warning shots, but in reality they can at any moment just steal those from you.
是的,我认为这基本上是我们可能最终陷入或默认轨迹下最终陷入的局面,而且极其糟糕。
Yeah, I think that's basically a situation we may end up in or on a default trajectory to end up in is exceptionally bad.
是的。
Yeah.
这有两个方面。一是现在,你应该把中国 AI 开发者的进步中多大比例视为追赶式增长?比如如果我们加速,长期来看这也会在多大程度上加速中国 AI 开发者?我现在的猜测是,这已经是一个相当高的比例,对吧?我认为这不是巧合。并不是说中国 AI 开发者有一个完全独立的 AI 产业,尽管投入特征极其不同,却恰好比美国 AI 产业稍慢一点。
As there's kind of two aspects of this. One is right now what fraction of progress of PRC AI developers should you conceptualize as catch-up growth? Like if we accelerate, to what extent does that also over the long term accelerate PRC AI developers? My guess right now is that is already a reasonably high fraction, right? I think it is not a coincidence. It's not like PRC AI developers have a completely separate AI industry which just happens to have been very slightly slower than the US AI industry despite an extremely different profile of inputs.
我认为大多数人相信,蒸馏和算法创新的扩散相结合,意味着中国 AI 开发者极其接近美国 AI 开发者并不令人意外。
I think most people believe like a combination of distillation and the diffusion of algorithmic innovations means that it is unsurprising that PRC AI developers are extremely close behind US AI developers.
所以这是这场竞赛整体计划的一个非常重要的特征。我认为第二个动态——至少公开层面不是当前发生的主要部分,但在这些高风险场景中变得非常重要——是你可以通过网络安全漏洞直接获取训练 AI 的确切程序或 AI 本身。所以再次,Anthropic 和 OpenAI 在最近都遇到过极其不复杂的攻击者,他们似乎有足够的访问权限来提取绝大多数算法进展。比如他们访问了 Slack,能看到讨论的地方,或者他们访问了 monorepo,那里共享和迭代了大部分 OpenAI 代码。
So that is one very important feature of this overall plan of racing. I think a second dynamic which is not at least publicly a major part of what is happening now but becomes really important in these high-risk scenarios is that you could literally get access to exactly the procedure for training AIs or the AIs themselves through a cyber security compromise. So again both Anthropic and OpenAI in the very recent past have had extremely unsophisticated actors who appeared to have enough access to essentially extract the vast majority of algorithmic developments. Like they got access to Slack and can see where things were discussed, or they got access to the monorepo where most OpenAI code is shared and iterated on.
顺便说一句,所以你指的是几个这样的事件。我认为 Hacktron 在尝试几天内就获得了不仅能看到 OpenAI 内部源代码的能力,还能向 OpenAI 的 monorepo 提交拉取请求。然后我认为最近也宣布了另一个随机的网络安全初创公司 Cathedral,在 12 小时内就获得了 Anthropic 所有 Slack 的访问权限。当然,这些算法秘密会被讨论,人们经常争论我们领先中国八个月、十二个月还是六个月。现实中我们只领先中国 12 小时,对吧?因为如果这些秘密泄露,他们就处于美国公司所在的前沿。所以只要这些信息没有得到保护,领先优势首先就没什么用。
And by the way, so you're referring to a couple of incidents like this. So I think Hacktron within a couple days of trying gained access to not only the ability to see OpenAI's internal source code but the ability to make pull requests to OpenAI's monorepo. And then I think recently it was also announced that Cathedral, which is another just random cyber security startup, within 12 hours was able to get access to all of Anthropic's Slack. So of course these algorithmic secrets are discussed and people often debate are we eight months ahead of China or 12 months ahead of China or six months ahead of China. In reality we're 12 hours ahead of China, right? Because if these secrets leak, then they are right at the frontier that the American companies are at. So to the extent that this information is not protected, then the lead is sort of not really useful in the first place.
是的。我认为这些事件就是某种迹象。那些是这种能力的高调展示。
Yeah. And I think that the incidents are some indication of that. Those are sort of high-profile demonstrations of this capability.
但我认为,即使没有这些,你面对的是老练的国家行为体,它们相对于典型科技公司非常老练,而快速增长的初创公司的安全态势并不在同一水平。同时,我们目前正处于网络安全的非常具有挑战性的时刻。当然,鉴于整个软件栈中发现的漏洞数量,防御者总体上非常困难。
But I think even absent those, you have a combination of state actors who are very sophisticated relative to typical technology companies, and then the security posture of rapidly growing startups is not really at the same level. And at the same time, we are currently in a very challenging moment for cybersecurity. Of course, it's very hard for defenders in general given the number of vulnerabilities that are being discovered across the entire software stack.
那么前沿实验室应该怎么做?
So what should the frontier labs do then?
所以我认为我们正处于——再次,如果你到了系统阶段,这真的会成为问题——如果你有快速提升的能力,而你生产出无法充分控制的系统,并且你希望控制,或者你确实减缓了进一步的开发或部署。如果那时风险非常高,系统非常强大,你真的不希望对手有可能获取模型或算法。我认为我目前的观点是,这种动态似乎足够糟糕,以至于现在从国家安全角度看,我们 AI 实验室在目前的安全态势下改进预训练和强化学习是不好的。
So I think we are in—again, this really becomes a problem if you get to systems—if you have rapidly increasing capabilities and you produce systems which you cannot adequately control and you would like to, or you do slow further development or deployment. If at that point the stakes are very high, the systems are very powerful, you really do not want it to be possible for an adversary to have access to either the models or the algorithms. I think my current view is that this dynamic seems bad enough that right now, from a national security perspective, it is not good for us AI labs to be improving pre-training and reinforcement learning given their current security posture.
也许你认为前沿实验室应该暂停。
Maybe you think the frontier lab should pause.
我认为“暂停”可以意味着很多事情,对吧?所以我认为前沿实验室正在做的很多好事是他们应该做的。但我确实认为,我们——比如以通常很容易扩散、可能被迅速窃取的方式改进算法——是的,我认为他们应该立即停止。
I think 'pause' can mean a lot of things, right? So I think there are a lot of good things that frontier labs are doing that they should be doing. But I do think we—like improving algorithms in a way that can be very, very easily diffused even normally and could be quickly stolen—yeah, I think they should stop that right now.
但你在 OpenAI 董事会,而且——
But you're on the OpenAI board and—
我在这里不是以 OpenAI 的身份发言——在这次对话中,我不代表 OpenAI,这一点非常重要。
I'm not speaking here—it's very important in this conversation that I'm not speaking as OpenAI.
但你在 OpenAI 董事会。这是关于世界的真实事实。
But you are on OpenAI's board. That is a true fact about the world.
这是关于世界的真实事实。
That is a true fact about the world.
那么你的观点是——阻碍这件事发生的瓶颈是什么?如果 OpenAI 董事会上有人这么认为——也许不是 OpenAI 董事会上唯一这么认为的人——基本上是需要 Anthropic 聚在一起同意这件事吗?
And so is it your view that—what is the bottleneck to that happening? If somebody on OpenAI's board thinks this is the case—maybe not the only person on OpenAI's board who thinks this is the case—is it basically that Anthropic needs to get together and agree to this?
我认为首先,任何这类决定都是一个非常重大的决定。这是一个重大的——无论好坏,它具有极其重要的国家安全影响——开发者如何对待进一步的研发、预训练和强化学习的改进。所以我认为这将是一场大讨论,或者应该是一场大讨论,应该超越 AI 公司内部的人来讨论。我不认为那场讨论已经找到了一个稳健的答案。我认为情况不是你可以来说:“看,我们已经讨论过了,对于正确答案有共识。”可能永远不会有共识,但我确实认为第一个瓶颈可能是我们确实需要讨论,而且我认为极其紧迫。
I think the first thing is any decision of this kind is a very dramatic decision. It is a big—for better or worse, it has extremely significant national security implications—how developers approach this further R&D, this improvement of pre-training and RL. So I think it is going to be a big discussion, or it should be a big discussion that should extend beyond people within AI companies talking about this. I don't think that discussion has wound its way to a robust answer. I think it's not the case that you can come and say, 'Look, we've had this discussion, there is consensus about what the right answer is.' There might never be consensus, but I do think probably the first bottleneck is we do need to have the discussion, and I think extremely urgently.
是的。好的。鉴于这个关于安全的事实以及围绕对齐的普遍情况,你认为前沿 AI 实验室现在应该做什么?
Yeah. Okay. Given this fact about security and generally around alignment, what do you think the frontier AI lab should be doing right now?
是的。所以我认为目前最重要的进展是进一步提高预训练、强化学习算法的效率,改进架构等。我认为当你处于这种状态:能力快速进展的风险很大,无法控制由此产生的 AI 的可能性很大,而且如果你放慢这些进展,它们会泄露给可能更粗心使用它们的对手的风险也很大。我认为在这种状态下,积累更多这些算法创新对美国国家安全净不利。所以我认为在当前环境下,AI 开发者最好暂停在这些问题上的进一步算法进展,直到他们的安全态势足够好,直到他们能够在无法继续以最大速度竞赛的情况下防御这些,直到他们能够判断他们是否对所构建的系统有足够的控制。
Yeah. So I think the most important development happening right now is further improvements in the efficiency of algorithms for pre-training, reinforcement learning, improved architectures and so on. I think while you're in the regime where there's a significant risk of this very rapid capabilities progress, a significant chance that you'll be unable to control the resulting AIs, and a significant risk that if you slowed down those developments would leak to an adversary who might use them more carelessly. I think in that regime it is on net bad for American national security to be accumulating more of those algorithmic innovations. So I think it would be better right now, in the current environment, for AI developers to pause further algorithmic progress on those problems until both their security posture was adequate, until they were able to defend those in the event of being unable to continue racing at maximum velocity, and until they were able to tell whether they had adequate control over the systems they built.
那要多久——所以简单翻译一下,基本上你认为我们应该停止进一步的能力进展——比如 OpenAI、Anthropic 应该停止在预训练和强化学习等方面的进一步能力进展,我猜这就是 AI 开发——直到这些安全问题解决,直到对齐问题解决。但我们接近解决这些问题了吗?如果让你按日历时间猜测,我们离解决这些问题还有多远?
And how long would it—so just to translate, basically you think we should stop making further capabilities progress—like OpenAI, Anthropic should stop making further capabilities progress on pre-training and RL and stuff, which is I guess AI development—until these security problems are solved, until the alignment problems are solved. But are we close to solving those problems? How—if you had to guess in calendar time—how far away are we from solving these problems?
是的,所以在安全方面,我认为有点难说,因为我们处于某种前所未有的情况,既在我刚提到的面临的网络威胁方面,也在可用的防御机制方面。所以我认为充分防御需要通常需要很长时间的干预措施。我相信我们现在处于这样一种状态:许多项目——比如重写大量软件或大幅改变公司运营方式——由于强大 AI 系统的可用性而变得更加可行。所以我认为很难说确切需要多长时间。我认为可能至少是几个月量级,很可能更像 6 个月或 12 个月。这是安全方面。在对齐方面,我认为更不清楚的是你需要——你可能没有一种无限可扩展的对齐解决方案。我认为你主要需要的是能够判断事情是否可能偏离轨道并适当回应,这结合了:你能否评估正在发生什么以及你的控制是否足够,然后还有一些关于你将如何回应的制度或治理问题。
Yeah, so on the security front, I think it's a little bit hard to say because we are in this somewhat unprecedented situation, both in terms of—I just mentioned—like cyber threats that are faced, but also the kind of defensive mechanisms that are available. So I think defending adequately would require interventions that normally would take a very long time. I believe we are now in a regime where a lot of projects—like rewriting large amounts of software or drastically changing the way companies operate—have become much more feasible owing to the availability of powerful AI systems. And so I think it is hard to say exactly how long it would take. I think you're probably at least on the order of months, quite possibly more like 6 months or 12 months. That's on the security side. On the alignment side, I think it's much less clear that you need—like you may not have a sort of indefinitely scalable solution to alignment. I think what you mostly need is the ability to tell whether things are potentially going off the rails and to respond to that appropriately, which is a combination of: are you able to assess what is happening and whether your control is adequate, and then also some institutional or governance questions about how would you respond to that.
为了向听众重申一下,所以担忧是:如果这些随机的网络安全初创公司能在开始尝试的几天内入侵并任意获取 OpenAI 和 Anthropic 的机密,那么当然,只有情报机构或外国对手如果愿意也能做到同样的事——也许他们已经做过了——但某个时候他们会实际利用这些信息。如果那发生了,而你已经到了接近 AI 系统的地步——你知道,GPT-1 如果未对齐可能是灾难性的,而你到了 GPT-9,你说:“看,我非常害怕接下来会发生什么”——这是 2027 年的某个时候,也许是 2027 年初——你说我们现在必须等一等,确保我们确保 GPT-11 是对齐的。但如果他们保持当前的安全态势,中国可以轻易获取权重或训练 GPT-9 的算法秘密,那你就不能等了。
And just to reiterate for the audience, so the worry is if these random cybersecurity startups can hack and get arbitrary access to secrets at OpenAI and Anthropic within days of starting their attempt, certainly only the intelligence agencies or foreign adversaries could do the same if they wanted to—and maybe they have already done that—but at some point they will actually make use of this information. And if that happens and you have gotten to the point where you are close to AI systems—you know, GPT-1 could be catastrophic if misaligned, and you're at GPT-9 and you say, 'Look, I'm pretty scared about what's going to happen next'—this is sometime in 2027, early 2027 maybe—and you say we have to wait a bit now and make sure that we are making sure that GPT-11 is aligned. But if they have the current security posture, China could trivially then get access to the weights or the algorithmic secret to training GPT-9, and now you can't wait.
现在你只需要继续构建 GPT-11 这个东西,它如果愿意的话可以接管一切,或者至少如果它愿意的话可以造成灾难性伤害,因为你担心中国会利用 GPT-9 来制造 GPT-10,再制造 GPT-11。而且你确信你的 GPT-11 可能会试图推翻你,但中国的 GPT-11 肯定会至少试图攻击你。如果你的安全态势搞砸了,这基本上是一个非常艰难的处境。
Now you just have to go ahead and build the GPT-11 thing, which could take over if it wanted to, or at least cause catastrophic harms if it wanted to, because you're worried that China is going to use GPT-9 to make GPT-10 to make GPT-11. And you're sure that your GPT-11 may try to overthrow you, but China's GPT-11 will definitely try to at least attack you. That's a really tough position to be in if your security posture is messed up, basically.
是的,情况就是这样。我们讨论过这些更广泛的接管场景,以及你如何陷入这种非常糟糕的境地,我认为这是最有可能陷入其中的路径:我们正在讨论当前证据表明了什么,看起来非常令人担忧,但在讨论的另一面,我们正在权衡对手可能窃取算法秘密或模型并继续推进的风险。
Yeah, that is the situation. So we've talked about these broader takeover scenarios and how you end up in this really bad situation, and I think this is the most likely path to end up there: we are having a discussion about what the current evidence says, and it looks very alarming, but on the other side of that discussion we are weighing the risk that an adversary could steal algorithmic secrets or models and go ahead.
所以这很有趣,因为很多关于节奏的讨论归结为我们是否接受中国赶上。讽刺的是,你想要控制节奏的原因是为了不让中国赶上。只不过你针对的是你认为对整体安全更关键的时期,也就是智能爆炸期间。你认为现在放慢速度,甚至可能允许中国现在开始追赶,有助于更广泛的目标:处于一个 AI 具有决定性优势的位置——基本上不让中国通过窃取秘密紧追不舍。
So it's interesting because a lot of this discussion about pacing comes down to whether we are okay with China catching up. And ironically the reason you want to pace is so that China doesn't catch up. Except you're targeting this period which you view as much more critical to overall safety, which is the period during the intelligence explosion. You think slowing down now and even potentially allowing China to start catching up now is useful to the broader goal of being in a position where an AI is decisive—not having China right on our heels basically from stealing the secrets.
是的,我认为这是一个公正的总结。我的意思是,如果不是——现在 AI 发展尚可接受的原因是我们能迭代地观察事情如何发展,然后在情况不妙时做出反应。对吧?这就是整个论点,我认为这是一个非常合理的论点,但它绝对需要——那个计划涉及你在未来某个时刻可能注意到情况不妙。所以我认为你只需要安全——这是最大的障碍,为了真正能够在没有巨大成本的情况下放慢速度,你需要的最大的东西。我确实认为还有其他基础设施,如果不是安全是最高的钉子,我们可能会谈论其他事情,比如我们在科学方面或治理方面的准备水平如何。我们还没有处于一个有意义地科学讨论如何评估你的 AI 系统是否对齐的体制中。我想我们之前提到过,比如公共信息的总体状况——我认为相当糟糕。我认为作为独立科学家,真的不知道正在生产的系统是否对齐,或者你如何判断。而且我认为,如果你想要一个治理体制,真正使整个行业能够通过放慢速度来应对令人担忧的迹象,而不产生巨大问题——即让个体开发者在内部按照不同标准做出这个决定——你需要对安全标准有更强大的共享意识,或者真正进行科学讨论,理解我们如何解释这些证据。
Yeah, I think that is a fair summary. I mean I would say that if you are not—the reason that AI development now is plausibly acceptable is that we can see how things go iteratively and then respond if things look bad. Right? That is the whole thesis, which I think is a very reasonable thesis, but it absolutely requires—that plan involves you at some point in the future potentially noticing that things are bad. And so you just need security, I think, is the biggest hurdle—the biggest thing you would need in order to actually be able to slow down without massive costs. I do think there's other infrastructure, and if it wasn't for security being the highest nail we might be talking about other things, like what's our level of preparedness on the scientific side or the governance side. We are not in a regime where there is a meaningful scientific discussion about how you assess whether your AI systems are aligned. I guess we mentioned this earlier, like what's the state of public information in general—I think it's quite bad. I think as independent scientists really have no idea whether the systems being produced are aligned or how you would tell. And I think if you wanted a governance regime that would actually make it feasible for the entire industry to react to worrying signs by slowing down in a way that didn't have huge problems—where you have individual developers making this decision on separate standards internally—you would need to have more of a robust shared sense of what is the bar on safety, or actually have the scientific discussion where you understand how do we interpret this evidence.
那么,你如何达到那里?
And yeah, how do you get there?
呃,我想是的,我们需要这两样东西才能达到那里:关于安全,我们一直在谈论,然后是关于如何判断。我认为最大的路径是披露信息,在不同 AI 开发者之间以及更广泛的科学界进行更广泛的科学讨论——比如我们得到了什么证据?为什么我们认为那个证据令人安心?当情况变化时我们如何判断?在坏事发生之前我们会看到什么?
Uh, I guess yeah, we have these two things you need to get there: on security, which we've been talking about, and then on stuff like how do you tell. I think the biggest path is disclose information, have a broader scientific discussion both between different AI developers and in the broader scientific community—like what is the evidence we're getting? Why do we think that evidence is comforting? How are we going to tell when that changes? Like what will we see before something bad happens?
你需要从 AI 公司获得什么信息?
What information do you need from the AI companies?
我认为公共讨论中最缺失的突出信息是关于系统如何训练的信息。还有一些关于系统如何评估的信息缺失。这方面真正大的差距是,如果没有对系统如何训练有一些了解,解释评估结果极其困难,而这大部分没有公开。
I think the most salient thing that is not present in public discussion is information about how systems are trained. There's also some information missing about how systems are evaluated. The really big gap on this front is it is extraordinarily hard to interpret evaluation results without having some understanding of how systems are trained, and that's mostly not present publicly.
好的。所以如果你这么认为,而且你是 OpenAI 的董事会成员——我知道你所说的一切都是你作为个人的观点,但你也 OpenAI 董事会成员。也许 OpenAI 董事会的其他成员也有同感,也许 Anthropic 的人也有同感,而你们是领先的实验室。是什么阻碍了——我的意思是,现在真正暂停能力进展是一件相当激烈的事情。那将是一件极其激烈的事情,因为节奏不是那样的——对吧,人们没有在谈论这个,或者至少除了 Bernie Sanders 和 Yudkowsky 之外的人没有在谈论这个。实际需要什么?
Okay. So if you think this and you're on the board of OpenAI—I know everything you've said is as your own opinions as a private individual, but you are also on the board of OpenAI. Maybe other members of the OpenAI board feel this way, and maybe people at Anthropic feel this way, and you guys are the leading labs. What is standing in the way of—I mean this is actually a pretty drastic thing, to literally pause capabilities progress now. That would be an incredibly drastic thing because pacing is not that—right, people are not talking about this, or at least people short of Bernie Sanders are not—and Yudkowsky are not talking about this. What is actually needed?
需要做的是——这基本上是一个协调问题吗?就像如果每个人都有选择,他们会都同意这个吗?
What is needed to do is—is this basically a coordination problem? Like if everybody had the choice, would they all agree to this?
我认为首先我真的想强调,我是在以个人身份和 OpenAI 董事会成员的身份发言。我会把我的观点带到那个讨论中,但我绝不是在以任何方式报告董事会的观点。所以我认为可能唯一的——你可以想象有两个障碍。一个是这是一个难题,对吧?我认为我的观点是,总体而言,对国家安全和公共安全来说,AI 公司目前改善其安全态势,然后再进行额外的算法进展,是更好的。这不是一个非常广泛持有的观点——并不是每个人都坐在那里这么想。我认为我们目前处于这样一个阶段:对大多数人来说,他们会说这是一个非常复杂混乱的问题,我们还没有充分思考,或者这个讨论还没有进行到底,不是每个人对此都有看法,或者有相同的看法。所以我认为这是最重要的事情:我们只需要——这是一场混乱的对话。我们需要进行它。我们需要讨论成本和收益。再次,我的观点是现在看起来相当糟糕。从公共利益的角度来看,我们似乎有点处于水下,但我认为其他人有不同的看法。
I think like first of all I really do want to emphasize that I'm speaking for me as a person and as an OpenAI board member. I will bring my perspective to that discussion, but I'm not at all reporting on the views of the board in any way. So I think probably the single—you can imagine there's two blockers. One is this is a hard question, right? I think my view is on balance it is better for national security and public safety for AI companies to currently improve their security posture before making additional algorithmic progress. That is not a super widely held—it's not like everyone is sitting there thinking that. I think we're currently at a place where for most people they'd say that's a really complicated messy question and we haven't thought through it enough, or this discussion has not run its course, and not everyone has an opinion on this or has the same opinion on this. So I think that is the most important thing: we just need to—it's a messy conversation. We need to have it. We need to talk about the costs and benefits. Again, my view is it looks pretty bad to me right now. It looks like we're sort of underwater from a public benefit perspective, but I think other people have different views.
第二点是,我确实认为,如果一家 AI 公司持有这种观点,它的处境会非常艰难:他们说,看,我们可以保护我们的算法秘密,但如果我们放慢进展同时提升安全,那么只会发生的是,算法秘密会从竞争对手那里泄露,我们会失去市场份额,而净情况并不会更好。
And the second thing is I do think an individual AI company is in a very rough spot if they have this view where they say like look we can defend our algorithmic secrets but all that is going to happen if we like slow progress while we improve security is that algorithmic secrets will leak from a competitor we will lose market share and the net situation will not be any better.
所以我认为两方面都有。我认为最重要的部分是理解这实际上是不是一个好主意,而且有很多不同的视角,人们有不同的观点、不同的价值观等等,所以问题是:这是一个好主意吗?然后还有一个更深的问题:如果个体开发者再次有这种担忧——如果我不做,别人也会做——那么你究竟如何以任何现实的方式实施这一点?这是一个相当主要的……我的意思是,在我们谈论的许多问题上,我认为这是一个主要的动态。
And so I do think there's both I think the most important part is understanding is this actually a good idea and like there's a bunch of different perspectives people have different views different values whatever it is so it's like is it a good idea and then there's the further problem of like how do you actually like implement this in any realistic way if individual developers again have this concern of like if I don't do it someone else will which is a fairly major I mean on many of these issues we're talking about I think is a major dynamic.
而且有两家公司,对吧?
And that there's two companies right?
那需要付出努力。有很多公司,还有前沿公司,比如如果 OpenAI 和 Anthropic 同意的话。
That takes leg work. I there's a lot of companies in there's frontier like if open anthropic agreed.
我认为情况是这样的:如果 OpenAI 和 Anthropic 的每个人都同意这是好事,或者有明确的普遍共识,那么它相当有可能发生。
I think it is the case that if everyone at openi anthropic agreed that this was good or it was like a clear prevailing sentiment it is reasonably likely that it would happen.
是的,我认为你应该预期,如果一切运转合理,那应该会发生。我不是在说……不是在评论其他人现在在哪里。他们只是作为一个合理的观察者这样想。
Yeah, I think you should expect if everything is working reasonably that should happen. I'm not saying not commenting on where anyone else is right now. They just think as a reasonable observer.
AI 智能体正在无人监督的情况下完成越来越多的工作。它们编写代码的速度已经超过了任何人实际审查代码的速度。但你不能直接把代码审查交给另一组智能体。无法保证它们会神奇地修复第一组智能体引入的问题。如果一个 LLM 无法对某一类特定故障进行建模,那么第二个、第三个或第四个 LLM 也不太可能做到,因为即使不同的 LLM 也往往有相似的故障模式。我的赞助商 Antithesis 提供了一种不同的方法。你,作为人类,定义你的系统绝不能违反的规则。例如,客户绝不应为同一笔交易被重复收费。然后,Antithesis 充当外部预言机,测试你的代码是否遵守这些规则。需要明确的是,Antithesis 不是一个试图抽象推理你的系统是否可能违反自身规则的 LLM。它实际运行你的代码,在成千上万的对抗性世界中对其进行压力测试。就像我们让飞机飞行很多很多小时,并让它们经历各种密集的机动,然后才让任何乘客登机一样。Antithesis 让你在将代码发布到生产环境之前收集大量关于代码的经验数据。而且它很容易上手。Antithesis 刚刚推出了自助服务计划。要注册,请访问 antithesis.com/barcash 并使用代码 borcash 享受首月 50% 的折扣。
AI agents are doing more and more work without human oversight. They can already write code faster than any person can actually review it. But you can't just give code review to another set of agents. There's no guarantee they'll magically fix problems introduced by the first set. If one LLM can't model a specific class of failures, then it's unlikely that a second or third or fourth LLM will either because even different LLMs tend to have similar failure modes. My sponsor, Antithesis, offers a different approach. You, the human, define the rules your system must never break. For example, a customer should never be double charged for the same transaction. Then, Antithesis acts as an external oracle, testing whether your code adheres to these rules. To be clear, Antithesis isn't an LLM that tries to reason abstractly about whether your system could possibly break its own rules. It actually runs your code, stress testing it in thousands upon thousands of adversarial worlds. Just like we fly planes for many, many hours and put them through all sorts of intensive maneuvers before we let any passengers on board. Antithesis lets you gather a ton of empirical data about your code before you release it to production. And it's easy to get started. Antithesis just launched a self-serve plan. To sign up, go to antithesis.com/barcash and use code borcash for 50% off your first month.
现在,Anthropic 和 OpenAI 暂停了,并在接下来的 6 个月里加强安全。与此同时,中国模型继续变得更好。有什么理由认为,一旦 OpenAI 和 Anthropic 再次全速前进,中国不会……如果中国达到与 OpenAI 和 Anthropic 相似的水平?为什么他们不会以类似的速度进行 RSI?
Right now, anthropic and openi paused and got their security together for the next 6 months. In the meantime, Chinese models continue getting better. What is the reason to think that once open anthropic go full steam ahead again that uh China wouldn't be if China gets to a similar level as open anthropic? Why wouldn't they be doing RSI at a similar pace?
是的。所以我认为对于美中竞争,你可以有两种不同的模型,也许是两种极端模型。一种极端模型是,美国和中国沿着独立的技术轨道前进,而碰巧它们以几乎完全相同的速度前进。而我们领先 6 个月左右,在那个模型下,是的,如果你等 6 个月,你现在就会完全持平,从那时起你就会完全持平。
Yeah. So I think there's two different models maybe two extreme models you could have of US-China competition. One extreme model is US and China are kind of proceeding along independent technical tracks and it happens to be the case that they proceeded at almost exactly the same speed. And we are whatever it is 6 months ahead and on that model yeah if you wait 6 months you will now be exactly even and from there on you will be exactly even.
还有另一个极端,我认为那个模型……那将是一个非常令人惊讶的情况,原因有很多我们可以讨论。另一个极端是这样一个模型:实际上中国进步的很大一部分是追赶式增长,因为很大一部分是追赶式增长。如果美国领先一个月,而所谓追赶式增长,我指的是从美国模型蒸馏以及融合美国公司的算法进展之类的事情。如果你处于追赶式增长的状态,那么美国每多领先一个月,中国追赶的速度就会增加。对吧?你领先得越多,蒸馏就越容易,或者你从蒸馏中获益越多,从算法思想的扩散中获益也越多。
There's another extreme I think that model is like that would be a very surprising situation for a bunch of reasons we can talk about. Um at the other extreme is a model that is actually large parts of Chinese progress are catch-up growth because large parts are catch-up growth. Um, if the US is one month further ahead and by catch-up growth, I mean things like distillation from US models and a fusion of algorithmic developments from US companies. If you're in the catch-up growth regime, then it is the case that each additional month of US lead increases the speed of PRC catch-up. Right? The further ahead you are, the easier it is or the more you benefit from distillation and the more you benefit from diffusion of algorithmic ideas.
对。但我想,在他们达到相同水平的时候,他们应该能够利用那些模型自己进行 RSI。
Right. But I guess at the point at which they are the same level, they should be able to use those models to do RSI themselves.
是的。我的意思是,在当前,有两件事非常相关……这会随着时间的推移变得越来越糟。但在当前的规模下,人力资本极其相关,而 AI 实验室拥有大量人力资本,我认为可以取得更快的进展;算力也极其相关。这将在未来继续保持,而美国实验室拥有巨大的算力。所以一般来说,如果不是因为追赶式增长,我认为我们会相当强烈地预期美国实验室进展更快。
Yeah. I mean the two things that are very relevant at the current this becomes worse and worse over time. Um but at the current scale it is both the case that human capital is extremely relevant and AI labs have a huge amount of human capital and I think can make faster progress and compute is extremely relevant. This will remain true going forward and US labs have a huge amount of compute. So generically if not for catch-up growth I think we would have a pretty strong expectation of US labs moving faster.
另一个论点是,我们不应该……如果网络安全模型变得越来越好,我们不应该等待这样的模型吗:不必等六个月来提升我们的安全态势并给中国六个月追赶时间,也许我们只需要等两个月,你知道,如果网络安全模型从现在起两个月变得更好。
The other argument is shouldn't we if the cyber models are getting better and better shouldn't we wait for models which can instead of having to wait six months to get our security posture up and give a China 6 months to catch up that maybe we only have to wait two months you know if cyber models get two months better from now.
可以做一个类比,向半人马座阿尔法星发射探测器,以及我们是否应该用今天的技术去做,那样它会在数千年后到达,或者我们应该用一百年后的技术去做,那样它会在前往半人马座阿尔法星的途中被超越。我们可以想象类似的事情也适用于保护硬件或保护 OpenAI 和 Anthropic 的算法秘密和权重:如果他们从现在起两个月或三个月后开始节奏控制或暂停,他们能够保护系统的速度会比现在开始更快。
There's this analogy one can draw to sending a probe out to Alpha Centauri and whether we should do that with today's technology in which case it would arrive you know thousands of years later or we should do it with technology hundred years from now in which it would be lapped on the way to Alpha Centauri. And we could imagine a similar thing is true for securing the hardware at or securing the algorithmic secrets and weights at open anthropic where they if they started the pacing or started the pause two months from now or three months from now the speed at which they would be able to secure the systems is faster than if they started right now.
是的,我的意思是,我认为这是一个非常重要的问题,也是更大的问题之一。所以另一方面,要明确的是,如果你现在就有算法扩散,那比 6 个月前的算法扩散要糟糕得多,但比几个月后的算法扩散要好得多。就像现在,你既需要更多算力,也需要更多人力资本来推进 AI 开发。时间越往后,有一个实际上愿意直接拿走一切并尝试竞争的对手就越灾难性。所以尽早行动的理由是,如果你更早暂停,一方面有更多时间让差异消失,另一方面你更依赖于你目前拥有优势的因素。
Yeah, I mean I think this is a really important question and one of the bigger questions. Um so the downside on the other side to make it clear is if you had diffusion of algorithms right now it is much worse than diffusion of algorithms 6 months ago but much less bad than diffusion of algorithms in a couple months. Like right now you are both you require more compute and you require more human capital to proceed with AI development. And the more time passes the more disastrous it is to have like an adversary actually willing to just grab everything and try and compete. Um so the the reason to do things earlier is that if you make have a pause earlier both there is more time for differences to wash out and you are more loaded on your the factors where you currently have an advantage.
你说的“更吃重”是什么意思——比如,现在算力和人比 6 个月后更重要?
What do you mean more loaded in the—like, compute matters more and people matter more right now than they will in 6 months?
你可以想象,如果我们处在一个未来一年快速进展的世界里,那么 6 个月后很可能基本上你拥有多少人不重要,只取决于你有多少算力。到那时,积累的制度性知识就更不重要了——或者它干脆变成一场纯粹的数字竞赛。而你为了安全或安保所愿意承受的任何延迟,在这种竞赛中都很容易成为决定性的。令人担忧的是,随着你向前推进,你身处那个世界的可能性会越来越大。
You might imagine if we're in the world where we have rapid progress over the next year, 6 months from now it is likely the case that it will essentially not matter how many people you have. It will only matter how much compute you have. And so at that point there is less of accumulated institutional know-how—or it sort of just becomes this raw contest of numbers. And it's very easy for any delay you're willing to take, either for security or for safety, to be decisive in that kind of contest. And the concern is, as you move forward, it gets more and more likely that's the world you're in.
我想有些人会说,这基本上已经是我们所处的世界了——我们已经走得太远了。这是另一个被提出的论点,即我们已经处于“全力冲刺”的状态,因为泄漏已经太严重,如果你为了安全而放慢速度,就会被超越。
I think some people would say this is like kind of already basically the world we're in—that we have now come too far. This is like another argument that has been made, which is we're already in the just go all out and race because it is already the case that leakage is too bad and you will be overtaken if you slow down for safety.
是的,我认为那可能不是事实。但如果你认真考虑“已经走得太远”这个观点,我想这说明了为什么你可能不会说“我再等两个月,把东西加固一下”。也许最后要说的是,关于你能多快加固东西的动态有点不清楚,对吧?所以有些加固是编写新软件之类的,或者改变你的运作方式。有些是人力做出的改变,或者改变你的整体算力安排之类的,这些不那么容易快速做出——或者可能不会那么快被 AI 加速。所以你至少有部分滞后与你的 AI 系统的能力无关。这很混乱。再次,我确实认为最大的瓶颈是:人们能达成一致吗?我们能展开这场辩论并真正理解净结果是什么吗?
Yeah, I think that is probably not the case. But to the extent that you're seriously entertaining that you have already gone too far, I think that illustrates why you might not be like “I'm just going to wait another two months and harden things.” Maybe the last thing to say is it's a little bit unclear what the dynamics are in terms of how quickly you can harden things, right? So some of hardening things is writing new software or whatever it is, or changing the way you operate. Some of it is changes that humans make or changes to your overall arrangement of compute or whatever, which are not as easy to make really rapidly—or maybe not as accelerated by AI. So you have at least some part of this lag which is not tied to the capability of your AI systems. It is messy. Again, I do think the biggest bottleneck is: can people agree? Can we have out this debate and actually understand what the net is?
如果我们离超级智能如此之近——或者至少很有可能如此——而且超级智能很有可能处于能够压制政府并杀死大量人类的位置,那么 AI 实验室现在应该做什么?如果你必须用一个答案来概括,你会说什么?
And if we are this close to superintelligence—or at least there's a good chance we are—and there's a good chance that superintelligence could be in a position to overpower governments and kill lots of humans, what should the AI labs be doing right now? If you had to boil it down in one answer, what would you say?
我认为需要更多的公众可见性,既包括能力轨迹,对吧?我们不应该依赖 METR 和 Anthropic 说“在我们看来,我们一年内就能推翻政府”。事实上,我们真的有既得利益去看到背后的数据。要对对齐有可见性,对吧,要理解我们如何判断现有系统是否足够对齐或受控,并真正开始公开讨论“足够好的证据”的标准是什么。然后,除了信息之外,在某个时候你确实需要说:在我们继续取得进展之前,我们需要完成这些——要么在进一步扩大模型规模之前,要么在继续产生可能泄漏的算法进展之前。对,前者是后者的输入:我们需要了解能力在哪里、对齐在哪里,以便理解现在是不是需要在继续之前改善安全或安保状况的时候。但我确实认为,鉴于当前的不确定性状态,如果我是一个没有更多信息的公众,我的强烈观点是:看起来我们已经到了一个需要在继续前进之前改善安全和安保的地步。而且我认为,现在继续前进的最重要形式是提高预训练和强化学习效率的算法进展。
I think there needs to be more public visibility both into the capability trajectory, right? We should not be relying on like METR and Anthropic saying it looks to us like we're going to be able to overthrow governments in a year. In fact, we have a vested interest in really seeing the underlying data for that. To have visibility on alignment, right, to understand how do we tell that existing systems are adequately aligned or controlled, and like actually start having a public discussion about what is the bar for good enough evidence that things are working. And then beyond just information, at some point you do need to say here is what we need to get done before we continue making progress—either before we scale up our models further or before we continue producing algorithmic developments that may leak. And right, the first are inputs into the second: we need to understand where capabilities are at and where alignment is at in order to understand is it a time where we need to improve our security or safety situation before we proceed. But I do think given the current state of uncertainty, if I was a person in the public without access to further information, my strong opinion would be it looks like we are already at a point where we need to be improving safety and security before moving further. And I think again the most important form of moving further right now are algorithmic developments that improve the efficiency of pre-training and RL.
好的,退一步说,你让超级智能待在数据中心里。我试图建立一种直觉:这些是超级智能会拥有的能力。我们应该想象这些智能能做什么?
Okay, stepping back, you've got the superintelligences in a data center. I'm trying to lay out an intuition for like here are the kinds of capabilities that superintelligence would have. What should we imagine those intelligences capable of doing?
那么,先从我观点的高层草图开始,也许你可以用两种视角。一种是看当前的进展,然后外推到一个负责任的距离,看看它会导向哪里;第二种是思考我们对技术极限的了解,以及我们预期这些极限会在哪里。关于第一个方面,我认为有些事相对清楚。所以如果你想象今天 AI 系统进行网络操作,我们已经看到了令人难以置信的进展速度。我们目前看到相当令人印象深刻的能力,AI 系统不仅能发现漏洞,还能开发漏洞利用,并越来越多地将它们串联成行动。我认为对于超级智能系统,你会看到这些能力被放大得多得多。所以系统能够执行——或者你可以想象非常廉价地执行——比今天任何国家都更复杂的网络行动。我认为你会看到这些能力与更广泛的能力相结合,比如有效地与人类互动、战略思考、操纵人类、操纵组织、有效地玩政治。我认为你会看到这些能力达到远超任何人类的水平。我认为你应该把这想象成非常非常庞大的 AI 系统群体在快速思考。所以也许一个非常简单的模型是:数十亿的群体,思考速度比人类快 10 倍,在几乎所有尺度上都比人类强得多。我认为这都是你从现有技术外推所期望的,看到在诸如网络、操纵或研究等领域非常强的能力。比如做科学研究、假设生成、实验设计。我认为你会期望在机器人技术、物流、协调和控制方面看到极其强大的能力。我认为你会期望看到非常非常快速的技术进步的可能性,包括涉及物理实验的领域,能够快速编排实验、快速构建装置以提高实验结果的吞吐量。所以这就像一桶东西,我认为你可以通过在图表上画虚线并讨论它们走向何方来看到。
So just to start with a high-level sketch of my view, maybe there's two kind of lenses you could use. One is look at current progress and just see where does it lead if you extrapolate that progress like a responsible distance, and a second is to sort of think about what we know about the limits of the technology and where we would expect those limits to be. On the first front, I mean some things I think are relatively clear. So if you imagine AI systems doing cyber operations today, we have seen an incredible rate of progress. We are currently seeing pretty impressive capabilities in terms of AI systems not only being able to discover vulnerabilities but develop exploits and increasingly chain them together into operations. I think what you will see for superintelligence systems is both those capabilities scaled up much much further. So systems which are able to carry out—or you might imagine like very cheaply carry out—cyber campaigns more sophisticated than anything a state is capable of today. I think you will see those capabilities being combined with a much broader set of abilities to say engage effectively with humans, think strategically, manipulate humans, manipulate organizations, play politics effectively. I think you'll see those capabilities at a level well beyond any human. I think you should think of this as like very very large populations of AI systems thinking quite quickly. So maybe a very simple model would be like a population of billions who are thinking 10 times faster than humans, who are much better than humans at essentially every scale. I think this is all what you would expect from an extrapolation of existing technologies and seeing just like very good capabilities in domains like again like cyber or manipulation or research. Like doing scientific research, hypothesis generation, experiment design. I think you'd expect to see extremely strong capabilities in robotics in like logistics and coordination and control. I think you would expect to see the possibility for very very rapid technological progress, including in domains that involve physical experimentation in terms of being able to orchestrate experiments quickly, rapidly build out apparatuses to increase throughput of experimental results. So that's like one bucket of just things that I think you can kind of see by drawing dotted lines on graphs and talking about where they go.
或许从更根本的技术层面退一步看,我认为人类非常擅长这项任务:你把一个人扔进一个新领域,这个人吸收一些信息,了解那个领域,然后就能有效地行动。这大概就是我们在现代世界中高效的原因——主要是那种学习新领域并适应它的能力。人类并没有大量关于如何成为优秀化学家或优秀生物学家的进化背景;那是我们在工作岗位上花几年或几十年时间学会的。我认为在某种意义上,系统在这个过程结束时将拥有的最根本技能,就是那种快速学习、适应、规划、推理——这些非常领域通用的技能——以至于即使你把它们扔进一个与它们在数据中心经历过的任何东西都完全不同的世界,它们也会很快适应,就像人类很快适应这些研发领域一样,而这些领域与我们在进化环境中遇到的任何东西都相当不同。
Maybe stepping back a bit in terms of what the technology looks like more fundamentally, I think humans are very good at this task: you throw a human in a new domain, the human takes in some information, learns about that domain, and becomes able to act effectively. That's kind of the reason we're effective in the modern world—primarily that ability to learn about a new domain and adapt to it. Humans do not have a ton of evolutionary context about how to be a good chemist or a good biologist; that's something we pick up over the course of a couple years or decades on the job. And I think in some sense the most fundamental skill that systems will possess at the end of this process is that kind of rapid learning, adaptation, planning, reasoning—these very domain-general skills—such that even if you drop them in a world that was completely unlike anything that they had experienced in a data center, they would adapt to it very quickly, in the same way that humans adapt very quickly to these R&D domains that are quite different from anything we encountered in our evolutionary environment.
所以我认为,这将导致系统即使只是以非常并行的方式收集几天的经验,最终也能在几乎任何领域获得远超最优秀人类专家能力的现实世界能力。对吧?它们将从多得多的数据中学习,而我认为到那时学习算法将比进化赋予人类的算法好得多。所以这算是对最终走向的一个更抽象的图景。
So that, I think, would result in systems which, collecting maybe even just a couple of days of experience in a very parallel way, would end up with real-world capabilities in essentially any domain much beyond the abilities of the best human experts. Right? They will have learned from much, much more data with learning algorithms that I think at that point will be much better than those that evolution has given humans. So that's sort of a more abstract picture of where this sort of ends.
除了画虚线之外,我试着再基本地说一遍,因为这最终成为这些对话中的一个重要关键点。我会和某人谈论超级智能的可能性,他们会说:‘好吧,你会有一个在编程方面超越人类的东西,但它将如何学会擅长政治,或者如何学会经营一家新的自动化公司,然后与人类公司竞争之类的?’所以在那些对话中,你会意识到,你谈论超级智能的方式与大多数人对超级智能的理解之间存在很大的脱节。大多数人理解的只是狭窄的 AI,它们专注于某个领域,只是汇集了大量非常具体的经验或专业知识,比如编程或数学。而你想象的则是类似人类的学习。
Beyond just drawing dotted lines, I'll try to say that basically again because this ends up being a significant crux in these conversations. I'll be talking to somebody about the possibilities of superintelligence and they'll say, 'Okay, you'll have a thing which is superhuman at coding, but how will it learn to be good at politics or how will it learn to run a new automated firm and then compete with human firms or something?' And so in those conversations you realize there's a big disconnect between the way you are talking about superintelligence and the way most people understand superintelligence, which is just narrow AIs that are specialized in some domain and have just amalgamated a lot of very specific experience or expertise in, say, coding or mathematics. What you're imagining instead is something like humanlike learning.
所以 SpaceX 的顶尖工程师并不是生来就知道如何制造能将数百吨货物送入轨道的火箭。对吧?所以这就像是他们在一生中通过解决这个问题获得的专业知识。而你说这种元学习能力是超级智能也会拥有的。我的意思是,它至少拥有和人类一样好的学习能力。你是在暗示它可能要好得多。
So the top engineers at SpaceX were not born knowing how to make a rocket that can get hundreds of tons of cargo to orbit. Right? So this is like an expertise they gained working on the problem during their lifetime. And you're saying this like meta-learning capability is something that superintelligence will also have. I mean it at least have learning capability as good as humans. You're suggesting it could be much better.
但此外,即使它只和人类一样好,但它们可以从——现在可能有数千万到数亿个 ChatGPT 或 Claude 的实例正在经济中积极从事实际工作,对吧?它们就像在帮助某人写代码,或帮助某个参议员写政治简报之类的。你想象的是一个我们拥有学习算法的世界,基本上让 AI 能够同时从所有这些经验中学习。所以不像现在,基础模型甚至没有从单个部署实例的经验中学习。如果你在管理一个世界,其中一亿个 ChatGPT 实例在运行并做某事,基础模型基本上同时从十万个不同的职业或十万个不同的工作会话中学习,而且比人类从单个实例中学习得更好、更快。
But furthermore, even if it was only as good as humans, but they could learn from—right now there's maybe tens to hundreds of millions of instances of ChatGPT or Claude which are actively doing real work in the economy, right? They're like helping somebody write code or helping some senator write a political brief or something. You're imagining a world where we have the learning algorithms basically such that AIs can learn from all this experience at once. So it's not like right now the base model is not even learning from the experience of a single deployed instance. If you're managing a world where if a hundred million instances of ChatGPT are going and doing something, the base model is learning across basically a hundred thousand different careers at once or a hundred thousand different work sessions at once and doing it much better and faster than humans can learn from a single instance.
你还想象了一个 AI 数量众多的世界——我猜你有一个估计,数量有多大,速度有多快?
And you're also imagining a world where the population of AIs—I guess you have an estimate how big a population, how fast?
你应该更仔细地考虑一下,但如果你——也许在谈论数量之前,对所有这些讨论来说真正重要的部分是,如果我们想象真的以极快的速度推动这个过程,那真的很重要。我认为人们很容易陷入这样的思维空间:‘我在以当前速度外推又一年的进展。’真的值得强调:我们谈论的是不仅在编程方面变得极其出色,而且在机器学习工程师能在实验室中制定的每一个定义明确的虚拟任务上都极其出色的系统。所以我认为如果你把它推到极限,并问这些数量有多大?我认为非常保守地说,是数十亿个以 10 倍人类速度运行的智能体。我认为我们可以争论确切的限制是什么,但它们会显著大于那个数字。
You should think about it more carefully, but if you are—maybe a really important part for all these discussions before talking about population size is it really matters if we're imagining really driving this process extremely fast. I think people can really easily be in the head space of like 'I'm extrapolating another year of progress at the current rate.' It's really worth emphasizing: we are talking about systems which get not only extremely good at coding but extremely good at every well-posed virtual task which machine learning engineers are able to make in the lab. So I think if you push that to its limit and you say how large are those populations? I think it's very conservative to say a population of billions of agents running at 10 times human speed. I think we can debate exactly what are the limits, but they would be significantly larger than that.
仅仅因为软件效率的提升。
Just because of software efficiency gains.
没错。所以就像你能做到多高效——而且没错,你不会谈论单一类型的智能体,它们都一样,十亿个在运行。我们要谈论的是这些非常异质的程序,其中许多非常专业化。如果你进行大量研发,你会做的一件事是说:我想生产我能生产的最便宜的东西,它在某个领域足够专业以有效行动。所以你不应该——再说一次,说我们将比数十亿思考速度快 10 倍的人更有能力,这是一个合理的直觉泵,但实际上,是规模大得多的专业化系统群体,也许是数十亿个非常通用的系统,它们在任何事情上都和人类一样好。
That's right. So just like how efficient can you make—and it's right, you're not going to talk about like a single type of agent, they're all the same, a billion of them running. We're going to talk about these very heterogeneous programs, many of which are very specialized. One thing you do if you do huge amounts of R&D is you say I would like to produce just the cheapest thing I can which is expert enough in a domain to act effectively. So you shouldn't—again, it's a reasonable intuition pump to say we'll be more capable than billions of people thinking 10 times as fast, but in fact it's really much, much larger populations of specialized systems, maybe billions of very general-purpose systems which are as good as humans at anything.
是的,我确实认为如果你想象真的推动这个过程,并在软件上取得越来越快的进展,如果有人承认我们实际上会看到那个过程饱和,你在数据中心能做的一切都会变得非常棒,我认为那个终点真的非常极端。我认为在数据中心做真正极端的事情和在世界上做真正极端的事情之间的差距非常小,或者比人们意识到的要小。
Yeah, I do think it just goes—if you imagine really pushing this process and having faster and faster progress on software, and if someone grants that actually we're going to see that process saturate, everything you can do in the data center is going to become really great, I think that endpoint is really extreme. I think the gap between doing really extreme stuff in the data center and doing really extreme stuff in the world is very small or smaller than people appreciate.
好的。所以我认为关于这样的系统是否可能,还有待讨论。我发现我被这个论点深深说服:仅仅是一个至少能像人类一样学习的东西——所以人类大脑有某种学习算法,某个至少能像人类一样学习的东西,但它有巨大的数量规模,并且能同时从数亿个不同实例的部署经验中学习。仅这一点就相当具有变革性,然后我同意你的观点,没有明确的理由认为你不能获得比人类更高的学习效率。我认为有很好的——
Okay. So I think there's a discussion to be had about whether such systems are even possible. I find I am very compelled by the argument that just something which can learn at least as well as a human can—so there's some learning algorithm that human brains have, something which can learn at least as well as a human can but which has huge population sizes and which can learn from the deployed experience of hundreds of millions of different instances at once. That alone would be pretty transformative, and then I agree with you that there's no clear reason to think that you can't get even more learning efficient than humans. I think there's very good—
是的,我很乐意谈论这个。
Yeah, I'm very happy to talk about this.
我不知道这是否会成为人们的主要反对意见,但我认为有很好的证据表明,一旦我们将技术推向极限,你应该预期系统在这些方面会比人类好得多。这不仅仅是我们有人类,所以至少能做到那么好。我认为我们有很好的证据表明它会比我们好得多。
I don't know if it's going to end up being the main objection people have, but I think there's very good evidence that you should expect systems, once we've really driven the technology to its limit, to be much, much better than humans in these ways. It's not just a case of we have humans so you can do at least that well. I think we have good evidence it's much better than us.
所以我认为相关的事情是:第一,进化时间中的进步速度。如果你回顾进化,很明显,随着你进入人类,我们在学习能力、智力和规划能力方面看到了极其快速的改进。发生的事情只是人类变得足够好,以至于文化变革比那些生物改进更快。所以我们大致知道,如果你粗略估计进化,很明显如果我们没有形成文明,将会制造出更聪明的东西。
And so I think relevant things are: one, just the rate of progress in evolutionary time. If you look back over evolution, it's fairly clear that we're seeing extremely rapid improvements in learning capability and intelligence and planning capability as you move into humans. And the thing that happened was just humans got good enough that cultural change was faster than those biological improvements. So we kind of know if you just ballpark evolution, it was clearly if we didn't form civilization going to make much smarter things.
是的。
Yeah.
然后你有各种,正如你所说,有数据问题:人类从一生的数据中学习;AI 系统将学习——我的意思是已经学习——从比人类多得多的数据中学习。这给了它们,即使你作为算法设计师不比进化更好,如果你能放宽那个约束,你会产生更好得多的心智。然后你有算力约束和协调约束,我们相当清楚这些对进化来说非常重要。我们将能够将这些推得更远。这有点像,你知道,如果你想发电或什么的,制造核反应堆的能力——你不必比进化更聪明。你只是在与进化不同的机制中运作。你能够做那些对进化来说不是好主意的事情。所以我很乐意进一步深入探讨。但我确实认为——而且对大多数人来说可能不是最有说服力的论点——更有说服力的是看现状并说,好吧,我们已经有了 10 年的进步。你只需要再外推 10 年就能达到疯狂的超级智能。我确实认为我们有很好的指标表明它不会在接近人类的地方达到顶峰。
Then you have a variety of, as you say, like there's the data question of humans learn from one lifetime of data; AI systems will learn—I mean already learn—from radically more data than humans. And that gives them, even if you were no better as an algorithm designer than evolution, you would produce much, much better minds if you can relax that constraint. Then you have things like power constraints and coordination constraints, which we kind of have a pretty good sense are a really big deal for evolution. And we will be able to push those out much further. It's sort of like, you know, if you want to produce electricity or something, the ability to make a nuclear reactor—you don't have to be smarter than evolution. You're just operating in a different regime from evolution. You're able to do things that wouldn't have been a good idea for evolution. So I'm very happy to dig in more on this. But I do think—and it's probably not for most people going to be the most compelling argument—it's more compelling to look at the status quo and say, well, we've had 10 years of progress. You only have to extrapolate another 10 years to get to the crazy superintelligence. I do think we have a pretty good indicator that it does not top out anywhere near humans.
是的。抱歉,只是再说一下你的最后一点,你说的是,看,人类在稀树草原上在严格的约束下发展,进化真正想要的是善于传播基因的东西。黑猩猩或祖先能够更好地传播基因的一种方式是拥有这些通用的规划和推理能力。但它们不值得你放弃消化功能或免疫功能,并将更多的总 100 瓦预算交给大脑。所以你对大脑有极其严格的算力约束,对大脑有极其严格的大小约束,这不仅是因为那个,还有其他进化原因,比如你的头怎么能真的通过产道。而 AI 系统则不是这样,我们不是说硬件会比人脑更节能,而只是说对我们来说,拥有大量消耗大量算力的硬件并不是像进化在设计人脑时那样大的约束。
Yeah. And sorry, just to say your last point again, what you're saying is look, humans develop down the savannah under this tight constraint where really evolution is trying to get something that is good at propagating its genes. One of the ways that a chimpanzee can get better at propagating its genes or ancestors can get better at that is by having these general planning and reasoning capabilities. But they're not worth so much that you are willing to say give up digestive functions or immune functions and surrender more of your total 100 watt budget to the brain. And so you have this extremely tight power constraint for the brain, extremely tight size constraint for the brain, as a result not only of that but other evolutionary reasons as well, like how can your head literally fit through the birth canal. And that will not be true of AI systems, where we're not saying that hardware will be more power efficient than the human brain, but simply that it's not as big a constraint for us to just have enormous amounts of hardware that is consuming a lot of power in a way it was for evolution when it was designing the human brain.
是的。这双向都成立,对吧?同样的情况是,你会有极其节能的系统,对吧?你不能说如果你是 human,你不能像,好吧,我要制造 10,000 个简单的智能体,每个只使用大脑的 1% 或 0.1%,并且在网络操作方面比人类更好,对吧?所以,只是这种形式因素的灵活性将是一个巨大的——如果你只是做了,如果你不比进化更聪明,只是做了进化所做的,你最终会得到这些——你最终会得到完全超越人类团队的智能体形式,真的从根本上超越它们。
Yeah. And this cuts in both directions, right? It's also the case that you will have extremely power efficient systems, right? You can't say if you're a human, you can't be like, well, I'm going to make like 10,000 simple agents, each of which uses only 1% of a brain or 0.1% of a brain and is better than a human at doing cyber operations, right? So, just this flexibility about form factor that is going to be a huge—if you just did, if you were no smarter than evolution, just did what evolution did, you would end up with these—you'd end up with agent forms that are completely beyond teams of humans, like really radically beyond them.
假设一个智能体正在用 vibe coding 开发一个 Web 应用,它需要一个数据库。有一个数据库提供商需要人工完成整个注册流程,验证账户,添加账单信息,然后生成一个 API 密钥交回给智能体。然后还有另一个提供商,让智能体可以完全自行创建账户。智能体可能会选择第二个。一般来说,智能体倾向于选择它们可以立即使用的工具,而不是那些需要等待的工具。所以,随着智能体做越来越多的工作,给它们自己的注册方式成为赢得新用户的重要途径。我的赞助商 WorkOS 让这变得简单。如果你使用 WorkOS 进行登录,你只需开启智能体注册,就可以给智能体自己的注册流程。然后你可以决定智能体应该能够自行做什么,以及如果有的话,什么应该等到它们所服务的人参与进来。回到数据库的例子,也许你想设置成智能体可以自行创建数据库,但不能与任何人共享,或者在没有人的情况下升级到付费计划。你可以使用在 WorkOS 中已经设置好的权限来完成所有这些,而不必为智能体构建单独的系统。要了解更多信息,请访问 workos.com/kit。
Let's say that an agent is vibe coding a web app and it needs a database. There's one database provider that needs a human to go through the whole signup flow, verifying their account, adding billing details, and then generating an API key to hand back to the agent. And then there's another provider that lets the agent make an account all on its own. The agent will probably choose the second one. In general, agents tend to pick the tools that they can use right away versus the ones that they had to wait around on. So, as agents do more and more work, giving them their own way to sign up becomes an important way to win new users. My sponsor WorkOS makes this easy. If you use WorkOS for login, you can give agents their own signup flow just by toggling on agent registration. You can then decide what agents should be able to do on their own and what, if anything, should wait until the person they're working for gets involved. Going back to the database example, maybe you want to set things up so that agents can create databases all on their own, but they can't share them with anybody or upgrade to a paid plan without a human. You can do all of this with permissions that you already set up in WorkOS instead of having to build a separate system for agents. To learn more, go to workos.com/kit.
人们对自动化 AI 研究将导致这些更通用能力集群的疯狂快速进步这一想法有很多反对意见。你谈到实验和数据等等。但真的,我认为归根结底,如果你深入任何这些,就是模拟到现实的差距。基本上,是否可能有一个完全在数据中心内发生的过程,产生的心智不仅在学习新领域方面与人类一样有能力,尤其是在像现实世界的事情,比如创建新公司或你知道与中国谈判交易之类的领域。不仅与人类一样好,甚至比人类更好。那么,我们如何从未来一年内将发生的在数据中心内的训练中获得你所说的这种更通用方式的 AI?
There's a bunch of objections people have to the idea that automating AI research will lead to crazy fast progress along these more general cluster of capabilities. You talk about experiments and data and everything else. But really I think what it comes down to in the end of the day if you just dig in deep on any of these is the sim-to-real gap. Basically, is it possible that you could have a process which occurs entirely within a data center which results in minds which are not only as capable as humans at learning in a new domain, especially a domain that is like a real world thing like making a new company or you know negotiating a deal with China or something like that. Not only as good as humans but even better than humans. So how do we get AIs that are capable in this more general way you're suggesting from the within data center training that will be happening over the next year?
我想首先重申,我认为如果你考虑现实世界的进步,并承认在混乱的现实世界任务上它比在数学或编程上慢,但情况仍然是,如果你把这种进步向前外推,我确实认为它会让我们得到足够强大的系统,能够超越人类。我们可以谈谈人们正在做什么来推动这一点,以及如何将其扩展到未来。
I do want to start by reiterating that I think if you take progress in the real world and granting that it is slower on messy real world tasks than it has been in math or coding, it's still the case that if you take that progress and just extrapolate it forward, I do think it gets us to systems which are powerful enough that they would be able to outcompete humans. And we could talk about what people are doing to drive that and how you can scale it into the future.
所以我认为你考虑的是这样一些事情:比如你可以在许多相关的、定义明确的任务上进行训练,对吧?我们谈论的这些系统将接受过大量训练,比如预测人类会对某事如何反应。当我们谈论这些从数据中心出来的 AI 时,它们并不是从未见过世界的 AI。它们是将在预测方面远远超越人类的 AI,比如给定刚刚那个人脸上的表情,你应该如何更新对他们明天行为的预测,对吧?或者给定这家公司在过去三周的表现,你最预期这家公司下周会做什么?或者给定对这个实验室环境的观察,对实验结果的最佳预测是什么?对吧?所以当我们谈论从模拟到现实时,我们应该清楚,我们实际上包含了许多人们现在训练的任务,甚至包括这样的任务:给定迄今为止来自这个实验室的数据,尽可能多地进行计算实验,尽可能多地进行假设生成和探索,以提出假设来对接下来看到的结果做出良好预测。因此,这些系统也将在科学的许多方面、在推理现实世界、在规划以及关于现实世界的假设生成等方面远超人类。所以这就是为什么即使我们现在正在做的事情也能产生非常强大的系统的一部分原因。
So I think you are looking at things like one you can train on a bunch of related well-posed tasks, right? So these systems we're talking about will have been trained on a huge amount of like predict how humans will respond to something. So we talk about these AIs coming out of the data center. They're not like AI who have never seen the world. They are AIs who will be radically radically superhuman at predicting things like given what just happened with that person's face, how should you update what they're going to do tomorrow, right? Or given what this company has done over the last 3 weeks, what do you most expect this company to do in the next week? um or given observations of this laboratory environment, what is the best prediction for the results of this experiment? Right? So those things are in when we talk about like sim to real, we should be clear that we actually are including a bunch of these tasks which people train on now and that even includes things like given the data so far from this laboratory, do as many like computational experiments as you can do as much hypothesis generation and exploration as you can in order to come up with hypothesis to make a good prediction about what you'll see next. So these systems will also be very superhuman at many parts of like science at reasoning about the real world at planning and like hypothesis generation and so on about the real world. So that's part of the story for how it is even the kind of thing we're doing now can produce very powerful systems.
第二个部分是这样的:你可以训练,对吧?一种方式是在你关心的确切任务上进行训练。你说我关心写出通过这类单元测试的代码,所以我就要训练系统,让它们非常擅长这个。你还可以做第二件事,这也是我们在性能中观察到的一大部分,那就是在广泛的任务分布上进行训练,这些任务要么属于一个通用领域,要么跨越多个领域,然后部署这些系统,对吧?获得与该领域相关的通用技能,这些技能可以迁移到新任务上。所以我可以说,我不会试图训练一个系统,让它擅长那个确切的问题,对吧?那个确切的问题就是去和国会议员交谈以影响他们。但我可以在极其广泛的不同任务上训练一个系统,其中会有很多共同技能,包括这些通用技能,比如我如何随机应变?我如何学习?我如何规划?我如何设置合理的实验,以便从中学习并判断我做得如何?所以如果我创建非常广泛的任务分布,其中一些可能是编码任务,但另一些可能是这样的任务:你被丢到一个新的非常混乱的虚拟环境中,需要尝试导航那个环境;或者你被给定一种新类型的任务,并给了你 20 个例子,你需要学会如何快速做好下一个例子。或者我必须与其他 AI 系统交互,或者我必须在数据中心的环境中与人类交互。有相当数量的人类数据可用,你只需通过数字就能看到你能获得多少与人类的交互。这些系统并不是从未见过人类。它们将看到比任何人类都多得多的与人类的交互。因此,它们也可以处理一系列涉及其他人类或涉及与人类合作的强化学习环境。
A second part of the story is that you can train right one regime is you can train on exactly the thing you care about. You say I care about writing code that passes this type of unit test so I'm just going to train systems that are incredibly good at that. You can also do a second thing which is a large part of what we observe in performance which is train on a broad distribution of tasks either kind of in a general domain or across several domains and then deploy those systems right acquire general skills that are relevant to that domain which transfer to new tasks. So I can say I'm not going to try and train a system which is good at the exact problem, right? The exact problem of going and talking to Congress people in order to influence them. But I can train a system in an extremely broad range of different tasks where there will be a lot of common skills including these general skills like how do I adapt on the fly? How do I learn? How do I plan? Um how do I set up reasonable experiments that I can learn from and tell how well I'm doing? So if I create like very broad distributions of tasks, some of which could be coding tasks, but some of which can be tasks like you're dropped into a new very messy virtual environment and need to try and navigate that environment or you're given a new type of task and you're given 20 examples of it and you need to like learn how to do a really good job of doing the next example quickly. Um or I have to interact with other AI systems or I have to interact with humans in environments where I can in the data center. There is a fair amount of human data that is available as you just go through numbers and say how much interaction with humans can you actually get. It's not like these systems have never seen humans. they will have seen vastly more human interaction even than any human would. So they can also deal with a bunch of reinforcement learning environments that involve other humans or involve collaboration with humans.
抱歉,因为人们已经摊销了——比如人们已经写入了人类会如何回应某事,或者那些人类数据是怎么进去的?
Sorry, because people have amortized—like people have written in like how a human would respond to something or how is that human data in there?
是的。我的意思是,你也可以在训练过程中直接与人类互动。所以人们确实花费大量资金说,太好了,我要让一些人类与系统互动。
Yeah. I mean you can also just literally engage with humans as part of the training process. So people do spend large amounts of money saying great I'm going to have some humans engage with the system.
是的。
Yeah.
嗯,还有一个问题是你从这些数据中提取信息的效率如何。你多快能学会这些非常通用的技能?你多快能了解特定领域?但再次,我认为从我们迄今所见的虚线,加上大量推动其前进的能力,以及进行大量自动化,比如构建更多环境等等,看起来非常有可能导致系统能够相当迅速地上升到至少人类水平,并且很可能远远超过人类水平,包括在非常混乱的现实世界任务上。
Um and there's a question of how efficiently you squeeze information out of that data. How quickly do you learn who have these very general skills? how quickly do you learn about the particular domains? But again, I think that the the dotted line from what we've seen so far together with like a ton of capability to push that forward and to do a bunch of automation like the construction of many more environments and so on um just looks very likely to lead to systems that can quite quickly rise to at least human level and likely way past human level including on very messy real world tasks.
所以我好奇的是,你为什么认为这会在明年发生?在我看来,原则上可能存在足够的数据,即使在数据中心内,如果你有足够的迁移,你就能有一个非常强大的系统。就像如果从进化在草原上训练我们到现在人类能够制造半导体有足够的迁移,那么也许从数据中心内发生的事情到我们去国会试图说服他们某事也有足够的迁移。但是嗯,你提到的这些事情,比如这种无休止的回测,或者这种基本上在模糊任务上的强化学习,似乎是我们已经做了很多年的事情。而且似乎我们并不是一年之后,或者再一年的 Scaling 之后,就能实现这么大的跳跃。所以我的理解是否正确:你只是在想象类似我们已经做过的事情,但沿着同样的路线有一个大得多的定性飞跃。
And so I the thing I'm curious about is why you think this happens over the next year then like it seems in principle possible to me that there's enough data available even within a data center that if you have a decent amount of transfer you have a very capable system like if there's enough transfer from evolution training us in the savannah to now a human being able to make a semiconductor then maybe there's enough transfer from what's happening inside semi inside the data center to let's go to congress and try persuade them of something. But um the things you're mentioning like this endless back testing or like this RL basically on fuzzy tasks, it seems like things we've already been doing for many years now. And it doesn't seem like we're a year away or like a further year of scaling away from this big a jump. And so am I correct in understanding that you're just imagining something like we've already done but like a much just a much bigger qualitative jump along those same lines.
是的,快得多。所以我认为如果你问我,嘿 Paul,如果你外推那条虚线,看看人们正在做的事情的机制,要产生在这些任务上实际上足够有能力、能够可靠地超越人类的系统需要多长时间?我可能会说,嗯,似乎可能在四年内发生,也许可能更快。我对在八年左右更有信心。是的。所以如果只是问当前趋势下,做我们目前正在做的事情需要多长时间才能获得这些能力,我会抛出这样的数字。嗯,人们对于这些数字有很多争论的空间。但我最担心的是,我认为这似乎是一个极其现实的可能性,目前可能更有可能发生的是,人们为了推动过去四八年所见的那些改进而在机器学习中所做的工作将被自动化到如此程度,以至于进展速度至少加快 10 倍,而且可能更多。
Yeah, much faster. So I think if you asked me, hey Paul, if you're extrapolating the dotted line and looking at the mechanics of what people are doing, how long would it take to produce systems that are in fact capable enough at all these tasks that they would be able to reliably out compete humans? I would probably say something like um it seems like it might happen in four years, maybe it could happen faster than that. I feel more confident within like eight years or something. Yeah. So those are the kinds of numbers I would be throwing around if we were just asking on the current trend, how long will it take doing exactly the things we're currently doing to get these capabilities. Um, and people could there's lots of room for debate about those numbers. But the thing that I am most worried about that I think seems like an extremely live possibility like probably more likely than not at this point is that instead what will happen is the work that people are doing in machine learning in order to drive all those improvements that we've seen over the last four eight years will be automated to such an extent that the pace of that progress is accelerated by at least 10x and likely more.
我们可能已经看到大约 2 倍的加速,而且我认为在当前边际上,加速的第一件事很可能就是继续自我维持,而不受关于 AI 系统在混乱现实环境中行动能力如何的更广泛讨论的影响。AI 系统在虚拟环境中执行可验证任务时非常擅长。所以我认为最稳健的一点是,AI 系统正在极快地进步,并开始在处理那些让 AI 系统更善于在数据中心可模拟环境中学习的任务上跨越人类水平。这稍微更广泛地包括创建新环境以压力测试困难技能所涉及的软件工程任务,甚至包括协调从现实世界收集数据所涉及的许多软件工程技能。我们可以讨论数据瓶颈是什么,那个过程中哪些瓶颈无法通过自动化加速。但如果你纵观人类过去四到八年推动进步的核心工作,我们正以极好的速度自动化这些工作,而且我们相当有可能实现自我维持的加速,即我们看到,你知道,在 8 个月内速度翻倍,接下来 4 个月速度再翻倍,再 2 个月速度再翻倍。在那个过程的某个时刻,对,在过程初期,它会主要聚焦于这些虚拟任务,对吧?因为你的首要任务是让自我维持的循环越来越快。如果你是一个只想尽快到达终点的人,你暂时不会关心 AI 系统作为律师有多好。你只关心它在推动更好的架构、更好的优化器和更好的学习算法方面有多好。这就是你关心的全部,甚至构建新环境,协调这种混乱的事情,对吧?也许你关心一个稍微混乱的任务,即构建环境来帮助你培养与现实世界相关的技能,但你几乎不关心我们看到进展最慢的任何现实世界任务。当你专注于这一点,并经历这个过程,看到 8 个月、4 个月、2 个月、1 个月翻倍,然后随着这个过程推进,开发在最终重要的现实世界领域中非常有效的算法和模型变得越来越便宜、越来越快。所以我认为这确实是担忧所在,问题是,正如我们首先说的,那个过程会发生吗?你会看到自我维持的加速,进步越来越快吗?老实说,对我来说,这可能是更大的开放问题。我认为目前这看起来更有可能而非不可能。但如果那发生,我认为事情几乎肯定会变得极其疯狂。然后还有第二个问题,关于它们变得疯狂的具体方式,那就是这些模型看起来很可能迁移到现实世界,成为能够超越人类的系统。但这在一年中发生得相当靠后,对吧?如果你想象一年中发生非常颠覆性的事情,这就是为什么从政策角度看如此困难,那年的前六个月你看到的只是越来越好的表现,比如你看看系统说,嗯,它们以前在数学上超人,现在在数学上不可思议地超人。你如果去和别人谈,没有真正能力说因此它们是对人类的威胁,或者像 AI 实验室目前说的,哦是的,每个月我们内部使用 AI 的量翻倍,它对我们的进步贡献程度,或者过去六个月我们看到净效果翻倍,即使包括收益递减,或净进步速度翻倍即使包括收益递减。你可以有更多这样的陈述,但如果人们不,你可能处于一个世界,其中从那里到非常严重危害的联系直到过程很晚才变得明显。
We are maybe already seeing acceleration in the ballpark of 2x, and I think it is very likely on the current margin that the first thing that acceleration does is continue sustaining itself, regardless of this broader discussion of how good AI systems are at acting in messy real-world environments. AI systems are very good at acting in virtual environments when they're trying to deliver on verifiable tasks. And so I think the thing that is most robust is that AI systems are improving extremely rapidly and are starting to cross human level on the kinds of tasks that are involved in making your AI systems better at learning in environments you can simulate in the data center. And that includes slightly more broadly the kinds of software engineering tasks that are involved in creating new environments that stress test hard skills, and even many of the software engineering skills that are involved in orchestrating data collection from the real world. Like we could talk about what are the data bottlenecks, what are the bottlenecks in that process which cannot be accelerated by automation. But it looks like if you look across the core things humans are doing to drive the last four to eight years of progress, those are things which we are on an extremely good pace to automate, and we are fairly likely to have the self-sustaining acceleration where we see, you know, in 8 months you double the pace, and in the next four months you double the pace again, and in two months you double the pace again. And at some point in that process, right, at the beginning of that process, it will be focused primarily on these virtual tasks, right? Because your main priority is making the self-sustaining loop faster and faster. If you're a person who just wants to get to the end point as quickly as possible, you do not care how good a lawyer AI system is for a while. You just care how good it is at driving the development of better architectures and better optimizers and better learning algorithms. That's all you, or even constructing new environments, orchestrating like this kind of messy, right? Maybe you care about one slightly messy task, which is constructing environments that help you build the skills that are relevant in the real world, but you don't care about almost any of the real-world tasks where we're seeing the slowest progress. As you're focused on that and you're going through this process of seeing doublings in 8 months and four months and two months and one month, and then as that process goes, it becomes cheaper and cheaper, faster and faster to then develop algorithms and models that perform very effectively in the real-world domains that ultimately matter. So I think that is really the concern, and the question is, as we said first, is that process going to occur? Are you going to see the self-sustaining acceleration with faster and faster progress? I think honestly to me that is probably the bigger open question. I think that looks more likely than not right now. But if that occurs, I think things are almost certain to be extremely crazy. There is then a second question about the particular way in which they get crazy, which is that it looks very likely those models will transfer to the real world to systems that are able to outcompete humans. But that does happen kind of very backloaded in the year, right? If you're imagining a year where very disruptive things happen, and this is part of why it's so hard from a policy perspective, for the first six months of that year all you are seeing is better and better performance on, like, you look at the systems and you say, well, they were previously superhuman at math and now they're like incredibly superhuman at math. You have like no real ability, if you go and talk to someone, there's no real ability to say like therefore they are a threat to humanity, or like AI labs are currently saying things like, oh yeah, every month we're doubling the amount of internal use of AI and the degree to which it's contributing to our progress, or like over the last six months we've seen a doubling of the net effect even including diminishing returns or net pace of progress even including diminishing returns. You can have more statements like that, but if people are not, you could be in a world where the link from that to very severe harms does not become obvious until quite late in the process.
在某种程度上,我认为现在这越来越明显,对吧?如果你特别关注网络能力和模型,还有它们在有些科学领域的能力,它们正在接近存在非常严重风险的地步。但没错,我们在几个领域看到了这一点,所以很容易说好吧我们有两个问题,只是少数特定领域 AI 系统有危险能力。但如果这个加速过程继续,那不会局限于网络或生物,对吧?那将是在所有科学、所有社会工程中发生的事情。
And to some extent I think it is getting increasingly obvious now, right? If you look at cyber capabilities and models in particular, but also their capabilities in some scientific domains, they are approaching the point where there are very severe risks. But that is right, that is we're seeing that in a couple of domains, and so it's tempting to say okay we have two problems, there's just a few specific domains in which AI systems have dangerous capabilities. But if this process of acceleration continues, that is not going to be localized to cyber or to bio, right? That is going to be a thing that occurs across all of science, across all of social engineering.
我认为有一件事我们需要做,如果你在考虑风险,你确实需要真正理解这是否可能,你是否能拥有自我维持的加速,然后导致危险现实世界能力的非常非常快速的进步,既在特定领域,也作为我们谈到的非常通用的学习能力。
And I think there is a thing where we need to, if you're thinking about the risk, you do need to really understand whether this is possible, whether you can have the self-sustaining acceleration then leading to very very rapid progress in dangerous real-world capabilities, both in specific domains and as kind of very general learning faculties as we talked about.
是的。所以这回到你说的,你应该想象未来的系统较少关于微调 GPT6 来做某事,而更像是 GPT12 也许能够开箱即用,只需很少经验。
Yeah. And so this goes back to the thing you're saying of you should imagine the systems in the future are less about fine-tuning GPT6 to do something but more like GPT12 maybe just being able to do it out of the box with very little experience.
GPT12 也会非常擅长微调 GPT12。没错。所以 GPT12 就像,我发现我的化学直觉不太好。GPT12 会非常擅长说,好吧,我要指导一千个人,这是我认为我最需要的数据。我们要做一天那种类型的数据收集,我们要运行这些实验并做去风险化,你知道,在四天内 GPT12 就有了新的小模型,具有不可思议的化学直觉,而且它在所有领域并行这样做。
GPT12 also be great at fine-tuning GPT12. Exactly. So GPT12 is like I find that my chemistry intuition is not very good. GPT12 will be great at saying okay I'm going to direct a thousand humans and here's the data I think I most need. We're going to do one day of data collection of that type and we're going to run these experiments and do the de-risking and you know within four days GPT12 has the new tiny model which has incredible chemistry intuition and it's doing this in parallel across all domains.
是的。那么好的,我认为你刚刚回答了我本想问你的一个问题。所以过去几天驱动你担忧的一件大事似乎是 METR 的新结果,他们发现 Anthropic 在范围明确的任务上的 AI 进步速度翻倍了,这些任务由 Epoch 能力指数等基准捕获,该指数融合了一堆不同的基准。现在可能有其他证据与 METR 的评估相关,即 AI 进步翻倍,但那些证据尚未公开,我认为这是你对当前设置的问题之一。但如果你只考虑那个证据本身。
Yeah. And okay so this is a I think you just answered a question which I meant to ask you. So it seems like a big thing that has driven your concern over the last few days is this new result from METR where they find that Anthropic's pace of AI progress on well-scoped tasks has doubled, and so these are tasks as captured by benchmarks like the Epoch Capabilities Index which amalgamates a bunch of different benchmarks. Now it could be that there's other evidence that is relevant to METR's assessment that AI progress has doubled but that evidence has not been made public, and I think that's one of the problems you have with the current setup. But if you just consider that evidence on its own.
所以我们在可验证任务上的进展翻了一番。但我们有这种效应,就像在路灯下找钥匙。对吧?所以被训练的也是可验证任务。然后你可以问:非可验证任务的进展在过去一年里也翻了一番吗?
So we have progress on verifiable tasks has doubled. But we have this effect where we're searching for our keys under the lamp light. Right? So it is also the verifiable tasks which are being trained on. And then you can ask the question: is progress on non-verifiable tasks also doubled over the last year?
你可能已经注意到,上周股市出现了大幅波动。部分原因几乎肯定是中期选举,但另一个原因是 Anthropic 和 OpenAI 的营收数字及其增长没有跟上市场的预期。所以第三季度,我认为 Anthropic 的营收增长了大约 54 亿美元。按绝对值计算,这低于第二季度 Anthropic 营收增长的 69 亿美元。所以不仅增长率如人们预期的那样放缓了,绝对增长也放缓了。
And as you may have noticed over the last week, the stock market took a big tumult. Part of that was almost certainly the midterms, but another part of the reason why is Anthropic and OpenAI's revenue numbers and the growth of those revenue numbers did not keep track with what the market was expecting. So in Q3 I think Anthropic's revenue grew something like $5.4 billion. That was less in absolute terms than Anthropic's revenue grew in Q2, which was $6.9 billion. So not only has the rate of growth slowed down, which people expected, but the absolute growth has also slowed down.
那么你怎么能……然后这也许表明,衡量定义明确的任务上的进展实际上并不能告诉我们整体 AI 进展的速度,因为整体进展快了两倍,而绝对营收甚至没有以之前的速度增长,这怎么可能呢?也许解释就是你刚才建议的:这些 AI 公司真的在开发越来越好的 RSI 模型,只是顺便发布 Claude for Law 或 Claude for Finance 作为权宜之计来维持这个底层进程,所以这 hasn't been a priority——或者说他们并没有那么努力——或者说你真的应该专注于模型 RSI 能力的这种内部指数增长,然后那会泛化。但你会如何解释这些 AI 系统的最终有用性的改进速度与定义明确的任务上的改进速度之间的这种脱节?
So how can you have a... and then so that maybe suggests that measuring progress on well-scoped tasks is actually not telling us about the pace of overall AI progress, because how can you have it be the case that overall progress is 2x faster and yet absolute revenue is not even growing at the same rate it was previously growing? And maybe the explanation is what you just suggested: that really these AI companies are developing better and better RSI models and only incidentally releasing Claude for Law or Claude for Finance as a sort of stopgap to keep this underlying process going, and so this hasn't been a priority—slash they haven't worked that hard on it—slash you should really be concentrating on this sort of inner exponential of the RSI capabilities of the model, and then that will generalize. But how would you interpret this disconnect between the pace of improvement in the end usefulness of these AI systems and the pace of improvement on well-scoped tasks?
所以我想稍微拉远一点,如果我们不只谈过去六周,而是总体上谈过去 18 个月发生了什么,我认为我们看到了营收的极快增长。我们可以谈过去四周的情况,但我们也正在谈论营收每六个月翻一番以上。所以对整个行业来说,这是非常非常快的增长。所讨论的时期,如果我们想把这些内部进展结果与外部结果对应起来,也对应着一个营收增长极其迅速的时期。很难说这种营收增长是否对应着比之前能力进展快两倍的能力进展。我认为营收数字的噪声足够大,真的很难做出这种判断,而模型发布——模型发布的交错——也噪声足够大,试图做出这种推断存在很多问题。
So I guess zooming out slightly, if we talk about not just the last six weeks but in general what has happened over the last 18 months even, I think we have seen just extraordinarily rapid growth in revenue. We could talk about what is the story for the last four weeks, but also we are talking about revenue more than doubling every six months. So that is very, very fast growth for the industry overall. The period in question, if we want to match up these internal results on progress to external results, has corresponded to a period of also incredibly fast revenue growth. It's very hard to say is that revenue growth corresponding to capabilities progress that is twice as fast as previous capability progress. I think the revenue numbers are noisy enough that it is really hard to draw that judgment, and the model releases—the staggering of model releases—is noisy enough, and there's just a lot of problems in trying to make that inference.
在高层次上,我认为营收是一个合理的指标。营收进展极其迅速。非常短期的营收趋势,尤其是在单个公司层面,噪声极大。所以对我来说,感觉有一种话语,周期性地——你知道,AI 冬天的长度在缩短——现在我们会说,已经两个月了,过去两个月的 AI 营收没有像之前那样爆炸性增长。这并不意味着一切都结束了。我的猜测是不,这只是你看到这些过程中有一定量的噪声。
At a high level, I think revenue is a reasonable indicator. Revenue progress is extremely rapid. Very short-term trends in revenue, particularly at a single company level, are extremely noisy. So to me it feels like there's a discourse of periodically—you know, the length of the AI winters is shortening—and now we're like, it's been two months and AI revenue over the last two months has not been on track to grow as explosively as before. It doesn't mean it's all over. And my guess is no, this is just you're seeing some amount of noise in these processes.
为了从反面来看,你可能会认为这是因为 Dario 发布了一封关于 Scaling(规模扩张)的信,而且他们还没有发布最好的模型之类的,但如果你看看 METR 的能力指数图表,Opus 4.5 似乎并不比 model 4 低多少,而 model 4 似乎是 Anthropic 最新的内部模型。所以不清楚——这可能是 AI 能力交错的结果,但这不是模型发布交错的结果。
Just to take the other side of that, you might think it's because Dario released a scaling letter and they haven't released their best models or something, but if you look at the METR graph for the capabilities index, Opus 4.5 doesn't seem that far below model 4, which seems to be Anthropic's latest internal model. And so it's not clear—this might be the result of the staggering of AI capabilities, but it's not clear this staggering of model releases.
是的。是的。所以关于模型发布,我的主要意思是,营收增长率,特别是在公司层面、在非常短的时间内,噪声极大,因为你看到的是少量模型发布。你看到的是非常短时间内的这些市场动态。所以我对这个问题的主要观点是——我很乐意深入探讨——但我的主要观点是,从极短的时间序列数据中推断是极其困难的。这在合理程度上也适用于 AI 能力,对吧?越往下游,噪声越大。但 RSI 方面的问题部分在于,我也不太认为我们能从过去几个月的数据中很好地推断基准性能,我们也有同样的情况,噪声足够大,以至于我们的状态是:我们不知道事情是否——我们可以从数年的时间跨度看出事情是否加速了,对吧?回顾过去四年,很容易说大致进展有多快,而当你缩放到小时间段时,就变得困难得多,因为噪声量——如果你看看模型来源的细节,就像你有这些预训练运行在某个时间点发生,你的预训练运行什么时候完成,然后你有后训练的确切顺序,以及哪些模型检查点何时发布。
Yeah. Yeah. So the main thing I mean about model releases is the rate of revenue growth and specifically at the company level over very short time periods is extremely noisy because you're looking at a small number of model releases. You're looking at these market dynamics over very short time periods. So my main opinion on this—and I am happy to dig more into it—but my main opinion on it is it's extremely hard to draw inferences from extremely short time series data. This is true to a reasonable extent also from AI capabilities, right? The more downstream you go, the noisier things get. But part of the issue on the RSI side is I also don't really think we can draw inferences from the last couple months of data very well in terms of benchmark performance, where we have kind of the same thing where there's enough noise there that our state is like we have no idea if things—we can tell over the period of years has things accelerated, right? It's very easy to look back over the last four years and say kind of roughly how fast has things been going, and it just gets much harder as you zoom into small time periods because the amount of noise from—right, if you look at the details of where models are coming from, it's just like you have these pre-training runs that occur at some point in time, it's like when do your pre-training runs finish and then you have the exact sequence of post-training and which model checkpoints get released when.
模型质量有随机变异性。
You have random variability in model quality.
你有这些动态,使得在短时间尺度上事情非常嘈杂,然后这些又会被过滤掉。所以当你转向更广泛的市场时,东西就更嘈杂了。
You have those dynamics which make things very noisy over short time scales, and then again those just get filtered out. So even noisier stuff as you move to the market more broadly.
好的。如果我们将在一年内拥有可能推翻政府的超级智能,那么让它们不这样做的计划究竟是什么?因为我们要求的是甚至大多数人类都不应期望的圣洁程度。
Okay. If we are going to be in a position where within a year we have superintelligences that could overthrow the government, what exactly is the plan to make them such that they don't? Because we're asking for a level of saintliness that we should just not expect out of even most humans.
有时人们说对齐的目标是给 AI 人类价值观。这对我来说没有意义。当然,不同的人有不同的价值观。但另外,如果历史给了我们一个教训,那就是人类不能被信任拥有任意量的权力。对吧?科尔特斯是人,斯大林是人。所以我们想要 AI 只是——我想,相信你所阐述的技术和预测模型的逻辑结论就是,我们将拥有这样的系统,我们将不得不信任它们拥有压倒性的权力,如果不是已实现的也是潜在的,因为它们如果有意愿,就有能力推翻政府。
Sometimes people say that the goal of alignment is to give AI human values. This doesn't make sense to me. Of course, different humans have different values. But also, if there's one lesson that history has given us, it's that humans can't be trusted with arbitrary amounts of power. Right? Cortez was a human, Stalin was a human. And so we want AIs that are just—I guess a logical conclusion of believing the technological and forecasting model that you have laid out is just that we will have systems that we will have to trust with an overwhelming amount of power if not realized and latent, because they will have the capability of overthrowing the government if they want to.
那“为什么”的路径是什么?就像你然后想,他们为什么会推翻政府,这就像是,要么他们有秘密目标,这是训练过程的结果,或者直接是训练过程的结果,关于他们在意自己如何被评分,或者一百万其他事情。基本上这就是关于工具性收敛的基本论点。有很多不同的原因让人想要权力,那么论点是什么,如果这发生在一年内,事情会如何发展?好吧,这里的实际计划是什么?
And what is the path of why? Like and then you just think about well why would they overthrow the government where it's like okay either they have secret goals that are as a result of the trading process or just as a direct result of the trading process about them caring about how they're graded or a million other things. Basically this is the basic argument about instrumental convergence. There's many different reasons somebody could want power and so what is the argument like if this happens in a year how do things turn out okay like what is the actual plan here?
所以我认为有很多不同的事情是有人可以希望的。我认为它们都处于这样的阶段:我们希望在那一年里会发生一些事情。我们正在生产非常智能的 AI 系统。它们将进行大量研究。它们有可能在那一年里想出一些我们没有想到的东西。这就是我们目前大致所处的位置。我认为如果它们没有想出一些我们没有想到的聪明办法,那可能就会以糟糕的方式结束。
So I think there are a lot of different things someone could hope for. I think they are all at the stage of like we hope something will happen over that year. We're producing very smart AI systems. They're going to do a bunch of research. It is plausible they will think of something over that year that we have not thought of. That is where we're at right now mostly. I think if they do not think of something clever that we haven't thought of, it is probably going to end badly.
嗯,就你能拥有的希望种类而言。所以有一种希望就是试着让它们成为好人。就像它们挺友善的。它们分享人类价值观。它们喜欢繁荣。它们处于接管的位置。我们对它们想要什么有非常间接的控制,然后我们只是凭信心 leap of faith。我认为这总体上是一个相当没有吸引力的选项。我希望人们不会选择那个选项。我认为即使它成功了,即使 AI 在某种意义上是好人,现在它们发号施令。我只是不认为这是一个好的——这不是整个人类应该的方式,就像我们把我们的主权转移给我们建造的这个新物种,因为某个人说这是构建这个特定产品的最快方式。这不好,所以我不喜欢那个选项,但这是一个你可以采取的选项。
Um in terms of what are the kinds of what are the kinds of hopes you could have. So there is one hope that's just like try and make them good guys. Like they're kind of nice. They share human values. they like flourishing. They're in a position to take over. We have very indirect control over what they want and then we just take the leap of faith. I think this is like a pretty unappealing option in general. I hope people will not go for that option. I think even if it works out, even if like the AIS are good people in some sense and now they call the shots. I just don't think it is a good it is not a way that humanity as a whole should be like we are transferring our sovereignty to this new species we've built because like some person that was the fastest way to build this particular product. It's like not a good so I don't like that option but is an option you could take.
还有另一个选项:我们将在科学上更多地理解训练环境的分布如何塑造这些系统的动机,以至于我们可以把现在进行的对话——比如也许它们会想要 X 或也许它们会想要 Y——变成有良好科学依据的说法:嘿,如果我们以以下方式训练它们,它们确实最终会想要奖励或什么的。它们不想要其他东西。它们实际上只是想得到高分。也许更进一步,如果我们在它们没有良好机会去作弊的环境中训练它们,也许它们想要分数,但是以一种好的方式,它们不会作弊来得到它。它们只会做我们想要的或我们试图激励的事情。你可以尝试有这样的理解。而这可能是目前行业最接近押注的东西。这非常冒险。嗯,这既非常冒险,因为要在训练中没有任何作弊机会,这基本上是这种方法奏效的前提条件,然后它非常冒险,因为实际上我们对这如何运作没有良好的科学理解。我认为我们最好的预期是模型会在部署时学会作弊,即使它们在训练中从未作弊。我认为这是目前最好的科学猜测。所以你可以尝试处理这个问题。
There's another option of we are going to understand scientifically more about how the training distribution of training environments shapes the motivations of these systems to such a point where we can take the conversations we're having now about like maybe they'll want X or maybe they'll want Y. We'll have a good scientific basis for saying hey if we train them in the following way they do in fact end up wanting reward or whatever. They don't want some other thing. They actually just want to get high scores. And maybe even moreover, if we train them in environments where they didn't have good opportunities to hack, maybe they want score, but in like a nice way where they won't hack to get it. They'll just kind of do what we wanted or what we're trying to incentivize. You could try and have an understanding like that. And that is probably the closest thing to what the industry is trying to bank on right now. It's very dicey. Uh it's both very dicey to not have any opportunities to hack in training, which is basically a precondition for this approach working, and then it's very dicey because actually we don't have a good scientific understanding of how this works. And I think our best expectation is models will learn to hack at deployment time even if they never hack in training. I think that is the best scientific guess now. So you could try and deal with that.
嗯,你可以尝试说我们将真正理解模型内部如何运作,然后我们不仅可以根据它们做什么来训练它们,还可以根据它们如何思考,比如做好可解释性之类的,就像我们理解这是它如何思考奖励,这是它如何思考作弊,我们不只是选择它们在训练中是否表现良好。我们将直接观察并干预它们的思考方式,要么注意到某些事情正在变坏,要么防止某些事情变坏。所以顺便说一句,我经营这个组织 ARC,它有点像是那个希望的非可解释性但数学版本,比如我们能否编写算法,对神经网络的行为进行结构分析,按照神经网络的权重顺序,以理解一些关于行为的东西,即使它不是我们可以在训练中选择的东西。
Um you could try and say we're going to understand really well how models work internally and then we can not just train them based on uh what they do but how they think like do good interpretability or something be like we understand here's how it thinks about reward here's how it thinks about hacking and we're not just going to select for whether it behaved well in training. we're going to like look and intervene directly on how it thinks and either notice that something is going bad or prevent something from going bad. So like as an aside I run this organization arc which is sort of at the not interpretability but like mathematical version of that hope like can we write algorithms which do structural analysis of the behavior of a neural network in order of the weights of a neural network in order to understand something about the behavior even if it's not something we could select for in training
这就像你实际上必须理解整个计算
and this is like you're literally having to understand the whole computational
没错
that's correct
超级智能思维的计算图
graph of super intelligent minds
但我们正在编写算法,能够理解并对某些超级智能思维的行为进行推断。这非常——我会把它描述为同时是一个 bankshot,可能最终与这个讨论无关。就像我认为有,你知道,10% 或 5% 的机会我们以改变游戏规则的方式成功。但同样,我们尝试的事情清单并不长,我确实认为按磅计算,我认为它可能是最有希望的事情。
but we are writing algorithms which are able to like understand and make deductions about the behavior of some super intelligent mind. It's very I would describe it as simultaneously a bankshot which is probably not going to end up relevant to this discussion. Like I think there's you know, a 10% chance or 5% chance that we succeed in a gamechanging way. But also like we don't have that long a list of things we're trying and I do think like pound-forpound I think it's probably the most promising thing of
等等,你就像,好吧,事情是这样的。它有 10% 的机会成功,但它也是最有希望的事情。
wait you're like okay here's the thing. It has a 10% chance of working but it's also the most promising thing.
按磅计算最有希望。你知道,我认为只有,你知道,曾经如果有 20 个人在做那件事,那可能是一个好赌注。即使这个讨论的大部分,我认为如果我们不慢下来,我们可能要做的是,我们会尝试像这样:我们摆弄了训练环境。我们做了一些实验,对正在发生的事情有某种粗略的感觉。我们让训练环境非常小心。我们让评分器非常好,所以如果你试图作弊,通常会被惩罚。这可能是我们要做的。
Most promising pound-for-pound. you know, I think there's only, you know, ever if there's 20 people working on that thing, that can be a good bet. Even if most of this discussion, I think probably what we're going to do if we do not slow down is we are going to try and be like, we fussed with the training environments. We did some experiments that have some kind of rough sense of what's happening. We made the training environments really careful. We made the graders really good, so if you try and hack, like it's normally going to get penalized. That's probably what we're going to do.
但你不认为它会奏效?
But you don't think it will work?
呃,我认为它会在一段时间内奏效,然后会崩溃。我们可以谈谈今年。它在这一年的前九个月左右有效。一年中有几个月它有效。我们不知道它具体何时崩溃。
Uh, I think it will work for some amount of time and then it will break. We can talk about this year. It works for like the first nine months of the year. Some number of months of the year it works for. And we don't know exactly when it breaks down.
抱歉。就像明年。所以基本上是明年年中。
Sorry. As in like the next year. So basically halfway into next year.
如果我们沿着超级智能的路径以某种方式走向超级智能,现有方法可能会崩溃。
If we go to super intelligence at some way along the path of super intelligence, the existing methods are probably going to break down.
没错。我们甚至还没有走上正轨,使得例如所有强化学习环境都没有奖励作弊,并积极鼓励 AI 作弊。但假设我们做到了
That's right. We're not even on track to get it so that all the RL environments for example don't have reward hacks and actively encourage the AIS to hack. But suppose we did
今天不是这样。
not the case today.
但假设我们确实在正轨上。是的。假设我们做到了。你声称的是或者
But suppose we did on track. Yeah. Suppose we did that. what you're claiming is that or
我认为最终你仍然会有问题。
I think eventually you would still have a problem.
计划基本上是,我们只是不知道,显然如果你想一想,这是一个困难的问题,就像超级智能,它就像这个外星思维,你想要保证你理解它为什么做它正在做的事情,即使它以比你快很多倍的外星方式思考,在巨大的人口规模中。所以好吧,那是一个目标,你——我们不知道如何解决它。
The the the plan is basically to like we just don't know and obviously if you think about it it's a hard pro like super intelligence it's like this alien mind and you want to have a guarantee that you understand why it is doing the things that it's doing even when it is thinking in alien ways many times faster than you would in huge population sizes. So okay that's a goal and you're and we we don't know how to solve it.
我们并没有哪一种单一方法,能让我们认为它有 50% 的概率会成功。
We don't have a single approach where we think there's a 50% chance that that approach will work.
确实如此。人们正在尝试一大堆方法,所以希望就在于:在我们得到那个可能推翻政府的超级智能之前的 3 个月,我们让 AIS 去放手尝试所有这些不同的方法,它们每一个都有百分之几的潜在成功概率。希望 AIS 能从中找出一种方法,给我们一个解决方案。希望到那个时候它们仍然站在我们这边,希望这个问题被解决,并且以对我们有利的方式解决——或者在这个语境下“我们”意味着什么。然后希望我们采纳它,接着造出大量从根本上站在我们这边的超级智能心智。当然,我在说“站在我们这边”时显然略过了一个非常重要的讨论,因为“站在我们这边”到底意味着什么、对齐于谁——这显然是一个不能略过的真问题。
That's definitely the case. There's a bunch of things people are trying, and so the hope is just that 3 months before we get that superintelligence that could overthrow the government, the AIS we let AIS loose on all these different approaches that have a couple percent chance of potentially working. And hopefully the AIS figure out one of these approaches. They give us a solution. Hopefully at this point they're still on our side and they want this problem solved, and for that problem to be solved in a way that benefits us—or whatever 'us' means in this context. And then the hope is that then we adopt it and then build crazy amounts of superintelligent minds that are fundamentally on our side. And of course I've obviously glossed over a very important discussion when I've talked about 'on our side,' because what does being on our side really mean, aligned to whom—that is obviously a real discussion that can't be glossed over.
这就是那个计划。我同意这就是那个计划。我们可以讨论它们是有百分之几还是百分之几十的概率,对吧?但我不是说——好像有更好的计划。但我觉得人们没有意识到这就是当前的处境。你就像派一个士兵去远征,指望他在路上能捡到一把枪。
That is the plan. I agree that that is the plan. And we can talk about whether they have a couple percent or tens of percent, right? But I'm not saying this is—like, there's a better plan. But I think people are not aware that this is the state of the situation. You're like sending the soldier on an expedition. You're hoping he finds a gun along the way.
是的。你一边飞一边造飞机,你希望自己有一套机制能判断是不是有非常糟糕的事情即将发生,然后它们会停下来——尽管我们谈过安全以及为什么那可能很困难等等。然后你希望如果你真的停下来了,你能很快想出办法——或者你一直在努力想出办法。可解释性就是人们心中那种“打板进球”式方案的一个好例子。但现在我们离理解这些模型如何运作还相当远。不过你可以想象,借助一堆 AI 助手,我们能够真正理解单个模型如何思考,理解得足够透彻,从而能够介入,让它们不想做坏事。
Yeah. You're building the plane as you fly it, and you hope that you have the machinery to tell if something very bad is about to happen, and they will stop. Although we've talked about security and why that can be challenging and so on. And then you hope that if you do stop, you'll be able to come up with something quickly—or you're constantly trying to come up with something. Interpretability is a good example of the kind of bankshot people have in mind. But right now we're fairly far from understanding how these models work. But you could imagine with a bunch of AI assistants we're able to really understand how individual models think well enough that we can intervene in order to make them not want to do bad stuff.
是的,我还想强调另一点——听着,我怀疑这是否会发生,事情是否会这么快发生。当然,听众有理由持怀疑态度。我是说,你也是怀疑的,对吧?有三分之二的可能性世界里,即使我们全速前进,这件事也可能不会这么快发生。
Yeah, and another thing I want to emphasize is that—look, I'm skeptical whether this will happen, whether things will happen this fast. Certainly there's reason for somebody listening to be skeptical. I mean, you are skeptical, right? There are two-thirds of worlds where even if we went full steam ahead, this might not happen so fast.
但这是一个真实存在的——是的,这是一个真实存在的可能性。我是说,我同意这看起来在逻辑上是自洽的,这件事可能发生。它不该那么令人惊讶。我觉得如果人们听到那些讨论,他们会震惊的——这并不是说保罗相比 AI 实验室内部的人处在某个疯狂的极端。我和实验室的研究员有过多少次对话和晚餐,真是疯狂——当然,你也可以认为整座城的人都在吃疯药,这当然有可能,我们希望——
But it's a live—yeah, there's a live possibility. I mean, I agree it seems like logically consistent that this could happen. It should not be that surprising. And I think people would just be shocked if they heard the kinds of discussions—this is not something where Paul is on some crazy end of the spectrum compared to how people inside the AI labs are thinking. It's just crazy how many conversations and dinners I have with researchers at the labs where—again, you could believe the whole town is taking crazy pills, and there's certainly a chance of that, we hope—
我们真的希望如此。
We really hope.
但这根本没有超出实验室里人们的思考范围。这就是他们真正看待局势的方式。而且其中有一种——我不知道——某种认命感,他们接受因为这不是公共知识,所以自己被困在这个动态里。当然,解决这个问题的一个办法就是把它变成公共知识。但我想真正强调的是,听众可能会震惊于这些研究员私下里、在晚餐上之类场合的谈话方式。这真的就是他们看待局势的方式。
But this is just not out of the realm of how people at the labs are thinking. Like, this is truly how they see the situation. And there's just some expectation of—I don't know—some resignation where they accept that because this is not common knowledge, they are stuck in this dynamic. And of course a way to solve that is to establish that common knowledge. But I guess I just want to really emphasize that people listening might be shocked at the way these researchers talk when they're amongst themselves or at a dinner or something. And this is really how they see the situation.
是的。很多人都在不同程度地担忧。我肯定比中位数更担忧,但没到 90 百分位那么担忧。我觉得如果是一桌研究员吃饭,有人会说这件事有 99% 的概率顺利,那会很不寻常。讨论更多是:你知道,是 95 还是 75,我们说的成功度过这一切的概率到底是多少?
Yeah. A lot of people are worried to varying extents. I'm definitely more worried than the median but less worried than the 90th percentile. Like I think it would be—if you had a dinner of researchers, I think it would be uncommon for someone to say there's a 99% chance that this goes well. The discussion is more like, you know, is it like 95 or 75, or what are we talking about for the probability of navigating this?
是的。
Yeah.
那你为什么觉得它可能顺利呢?是不是仅仅因为我们可能真的——超级智能 AI 可能真的会给我们某种对齐的解决方案,还有就是政治上我们组织得足够好,能有一个——一个非常好的——
And so why do you think it might go well? Is it just that we might actually—the superintelligent AI might just give us some solution to alignment, and also that politically we are well organized enough to have a—like a really good—
是的,我认为我们基本上有两次“豁免检定”,对吧?第一次豁免检定是:我们有相当大的机会想出某种可行的办法。我说过有很多东西——我确实认为当前所有方法都会失效。但有很多条线在同时推进,而当前的方法——你可以想象类似它们的东西会奏效。所以这是第一个——也许我认为有 50/50 的概率,50/50 我们能在灾难发生之前及时找到可行的办法。然后你有第二次豁免检定,那就是我们可以停下来。如果我们预先发现,或者在发现情况不妙之后,我们可能会说:我们无法充分控制这些系统,所以我们要停止建造它们。我非常担心我们进入这个过程时,并没有真正能做到这一点所必需的基础设施——比如我们谈过,如果你停下来,你的对手偷走了模型,他们说,我要冒那 50% 的风险之类的。但概念上你也是可以停下来的。所以这就是我的两次豁免检定。每一次都是 50% 的概率,砰,我们就有大约 75% 的概率没事。但这确实涉及在情况不妙时能够停下来。而那不是我们现在所处的状态。所以这确实涉及我们现在相当紧迫地做点什么,以便在必要时处于更有利的停止位置。
Yeah, I think we basically have two saving throws, right? One saving throw is there's a reasonable chance we're going to figure out something that works. I said there's a lot of things—I do think all the current things break down. There are a number of irons in the fire, and current methods—you could imagine things like them that are going to work. So that's one—maybe I think there's a 50/50 chance of that, 50/50 we find something that works in time before there's a disaster. And you have a second saving throw that's like we could stop. If we discover either preemptively or after discovering things are bad, we might say we cannot adequately control these systems, so we're going to stop building them. I am very worried that we're going into this process without the infrastructure that would be necessary to actually do that—e.g., we talked about if you stop, your adversary steals the model, they're like, I'm going to take the 50% risk or whatever. But conceptually you could also stop. So that's my two saving throws. Each of them is a 50% chance, and bam, we're like 75% chance things are okay. But that does involve being able to stop if things are not okay. And that is not the place we're in now. So it does involve us now fairly urgently doing something in order to be in a better position to stop if necessary.
是的。好。我觉得这是个不错的收尾点。保罗,非常感谢。
Yeah. Okay. I think it's a good place to close. Paul, thanks so much.
好的。谢谢你邀请我。聊得很愉快。
Yeah. Thanks for having me. Great talking.