设计得听起来对，而不是知道什么是对的

我们现在面临的情况是，模型根本不把真相放在首位。每个人都用过 ChatGPT，每个人都用过代码，它会告诉你你想听的，而不是你需要听的。智能显然很重要，也很强大。但不可靠的智能是有限的。如果你有一个神谕，你去问问题，它给你答案，有时对有时错，而你无法分辨，那只有两种可能的结果：人们要么全信，要么全不信。我认为这两种情况都在发生，但都不是我们想要的。

We are in a situation with models where truth really isn't even a first-order. Everyone's used ChatGPT, everyone's used code, and it will tell you what you want to hear rather than what you need to hear. Intelligence is important, obviously. It's powerful. Intelligence without reliability is limited. If you have this oracle that you go to and you ask questions and it gives you answers and sometimes they're right and sometimes they're wrong and you can't tell the difference. There's only two possible outcomes. People believe everything or people believe nothing and I think we're seeing both and neither is what you want.

Dan，欢迎来到播客。

Dan, welcome to the podcast.

谢谢邀请。

Thanks for having me.

好的，我们直接进入正题。“设计成听起来正确，而不是知道什么正确”。给我解释一下这个说法。

All right. We're going to hop straight into things. Designed to sound right, not to know what's right. Explain this a little for me.

设计成听起来正确，而不是知道什么正确。所以，今天的模型比如 LLM。人们常说，LLM 就是大型语言模型。也许值得回顾一下什么是语言模型，也就是它们变大之前的 LM。有趣的是，语言模型最初并不是为了今天这样的用途而发明的。如今，当人们遇到大型语言模型时，他们将其视为一种非常横向、可塑的智能。你可以问它任何问题，让它做任何事。它可以总结、翻译。这是一种令人惊叹的横向性、广度和可塑性。但最初语言模型是为了其他任务而发明的。语言模型最初是在语音识别等任务的背景下发明的。例如，在语音识别中，有人对着麦克风说话，音频中检测到了声音，但含义的唯一线索是：哪些词听起来像那样？通常会有很多不同的词听起来一样。所以，如果你对某人说的话运行所谓的声学模型，得到的结果是一系列听起来基本相同的句子。一个经典的例子是“hard to recognize speech”和“hard to wreck a nice beach”，我甚至需要夸张地发音才能说清楚。语言模型的任务就是遍历这些假设列表——这些假设如果被说出来，听起来会和你刚才说的一样——然后选出最可能的那一个。这意味着语言模型的任务是接受一个句子、一些词、一个文档、一段语言，并给它打分。这个分数是什么？它混合了以下因素：它是否合乎语法？比如，这是否符合该语言的词语组合方式？它是否切题？如果有已知的话题，它在语义上是否合理，还是胡言乱语？语言模型会将所有这些因素打包成一个简单的概率，帮助你区分胡言乱语和实际说的话。所以，语言模型的工作就是给句子打分。传统上，语言模型并不像今天这么大。它们更小，会查看单个片段，比如相邻的词。这个词常见吗？这个词对常见吗？这个词三元组常见吗？然后它们将这些组合成一个统计分数。就这样。人们并不期望它们体现智能，但确实期望它们打包大量关于语言的关联性：哪些词会跟在哪些词后面？这包括语法，比如限定词和名词，对吧？“the podcast”。还包括习语，比如“kick the blank”，人们会填上“oh”。

Designed to sound right, not to know what's right. So, today's models like an LLM. People say that, you know, LLM now, large language model. Maybe it's worth just going back and talking about what a language model is, just the LM before they got large. Interestingly, language models were not invented to be what they are today. So, today when people run into a large language model, they're thinking about these as this very horizontal plastic kind of intelligence. You can ask it about anything. You can ask it to do anything. It can summarize, it can translate. And this is sort of this amazing horizontality, breadth, and plasticity of intelligence. But originally language models were invented for something else. Language models were originally invented in the context of tasks like speech recognition where, for example, if you think about speech recognition and somebody is speaking into a microphone and there's audio and the sounds that they have produced are detectable in that sound signal, but the meaning your only clue is, well, what words sound like that? And often there are a bunch of different words that would sound like that. And so, if you run what's called an acoustic model over something that somebody said, what's going to come out is a whole list of different sentences that sound basically the same. A classic one is the hard to recognize speech versus hard to wreck a nice beach, which I have to hyper articulate to even make it clear what I'm saying. And the job of a language model was to go down this list of hypotheses that, if spoken, would sound like what you just said, and pick the one that was most probable. And that meant that the job of a language model was to take a sentence, some words, a document, a chunk of language, and assign it a score. What was that score? It was some mix of is it well-formed? Like, is this how you put words together in that language? Is it on topic? If there's a topic that you're aware of, is it semantically sensible or is it nonsense? And this language model would bundle all of these factors into just a simple probability that would help you tell the nonsense and gibberish from what the person actually said. So, a language model's job was to assign goodness scores to sentences. And classically, language models weren't ever large as they are today. They were smaller. They would look at individual pieces. They would look at adjacent words. Is this a common word? Is this pair of words common? Is this triple of words common? And they would put this together into a statistical score. And that was that. People didn't expect them to embody intelligence, but they did expect them to bundle together a lot of correlations about language. What words follow each other? And this includes things like syntax. You have a determiner and a noun, right? The podcast. It includes idioms like, you know, if you think kick the blank, people are going to fill in oh.

一个球。

A ball is there.

是的。嗯，谢谢。也许吧？我的大脑在踢球。

Yeah. Well, thank you. Maybe? That's my brain kicking the ball.

嗯，是的。所以这其实是个很好的点。到底是哪个？是“kick the ball”吗？在体育语境中，这说得通。是“kick the bucket”吗？在另一个语境中，这也有道理。一个代表对体育和世界的知识，比如你踢什么？球是人们经常踢的东西。而“kick the bucket”代表习语知识。这是关于英语语言的知识：有一些固定短语，“kick the bucket”与踢或桶无关。它已经被词汇化，表示不同的意思。你的语言知识将所有这些打包在一起。所以，语言模型旨在尽可能多地封装关于语言的关联性。语法如何运作？意义如何运作？习语如何运作？所有这些关联性共同体现了语言知识，以及随之而来的世界知识、流行文化知识。“The Empire Strikes blank”，对吧？如果你注意到“back”，你就知道一些关于《星球大战》的东西。所以，语言模型是统计收集大量关联性的工具，然后判断这些假设中哪些是好的，哪些看起来像是随便堆砌词语的“词语沙拉”。这就是它的目的。现在，随着我们不断 Scaling 这些模型，开始利用所有文本——维基百科上的所有内容、整个网络上的所有内容——并且重要的是，有几项发明非常关键，其中之一是使用神经表示来表示上下文中的词语，另一个是建模长距离依赖的能力。目前人们主要用 Transformer 来实现这一点。随着语言模型能够捕捉越来越多的关联性、越来越长的距离，以及它们为词语归纳出的越来越复杂的表示，它们捕捉“合乎语法”的想法逐渐退居次要地位，取而代之的是它们捕捉更深层次的通用知识表示。这就是今天的 LLM。然而，核心事实并没有改变：它们接受一个词序列（上下文），然后进行下一个词预测。实际上，它们进行的是下一个 token 预测，token 与词不同，在很多情况下更像是词块。当你向 LLM 提问时，它会从一个概率分布中逐个生成这些 token。例如，你说“kick the”，它知道什么关联性？它知道接下来可能应该是一个名词。

Well, yeah. So, this is actually a great point. Like, which is it? Is it kick the ball? Well, in a sports context, that makes sense. Is it kick the bucket in some other context, right? And, you know, in an idiom that might make sense. And one represents knowledge of sports, the world, like what do you kick? Well, ball is a thing people kick all the time. As kick the bucket represents idiomatic knowledge. That's a thing about knowing the English language is that there are these fixed phrases that kicking the bucket has nothing to do with kicking or buckets. And it's been lexicalized to mean something different. And your knowledge of language bundles all this together. And so, a language model is meant to kind of encapsulate as many correlations as you can get about language. How does the grammar work? How does meanings work? How do idioms work? All of these correlations together would embody knowledge of language and with it knowledge of the world, knowledge of pop culture. The Empire Strikes blank, right? Like if you notice a back there, you know something about Star Wars. So, language models were these tools for gathering a bunch of correlations statistically and then saying of these hypotheses like which of these are good, which of these look like we just threw a bunch of words together and got word salad. And that was sort of the purpose. Now, as we scaled these up and up and up and up and started tapping into all the text, you know, everything on Wikipedia, everything on the whole web, and importantly, there are a few inventions that were kind of very central, one of which was the use of neural representations for words in context and another was the ability to model long distance. The dominant way people do that now is with transformers, and as language models started to be able to capture more and more correlations, longer and longer distance with more and more sophisticated representations that they were inducing for words, the idea that they were capturing well-formedness started to take a backseat to the idea that they were capturing this deeper representation of general purpose knowledge. And that's the LLMs today. However, nothing has really changed the fact that at their core, they take a sequence of words, the context, and then they do next word prediction. In fact, they actually do next token prediction, which is tokens are not the same as words. They're more like word chunks in many cases, and that when you ask a question to an LLM, it begins generating these tokens one by one from a probability distribution where, for example, you say kick the and what correlations does it know? Well, it knows that what comes next, you know, maybe that should be a noun.

这不像是一个放动词的地方，而是一个放名词的地方。可能是“球”，也可能是“桶”，取决于上下文。我们刚才是在聊足球，还是在聊死亡？这些系统从根本上就是为了预测下一个词的概率分布，选出最合理的下一个 token。所以它们本质上是“合理性引擎”，而不是“真相引擎”。我们希望这些系统总能生成真实的事实，但它们的优化目标是生成与真相难以区分的输出。

This isn't like a time to put another verb. This is a place where nouns go. Maybe it should be a ball. Maybe it should be a bucket. It depends on the context. Were we just talking about soccer or were we talking about a death? And so, these systems are fundamentally designed at their core to predict the distribution of what comes next, to put the most plausible next token. So, these are fundamentally plausibility engines, right? And that's different from a truth engine. So, what you get is a system which we would love if these systems would always generate true facts. The thing we know is what they've been optimized for, which is to generate output that is indistinguishable from the truth.

所以所有这些相关性组合起来，都是为了向最终用户呈现真相。一个猜测越难以与真相区分，你就越相信它是真相。但它从未可验证地检查过那是否是真相，只是尽可能接近而已。

So, all of these correlations put together are kind of in the pursuit of presenting the truth to the end user. And the more indistinguishable a guess, the more you believe that that is the truth. But it's never verifiably checking whether that is the truth or not. It's just trying to get as close to that as possible.

是的，显然，作为人类，我们有谈论事物的机制。我们谈论真相，在模型中我们谈论幻觉，谈论欺骗之类的东西。我认为值得退一步，谈谈这些意味着什么。如果一个 LLM 在生成文本，对吧？需要指出的是，这是核心 LLM。人们在它之上构建结构。之后还有其他的训练阶段，这些都是当今技术故事的重要组成部分，我们可以深入探讨一些。但如果你只考虑 LLM 的核心部分，比如序列“伯克利的人口是”，然后下一个是什么？作为人类你会怎么填？告诉我。

Yeah, so obviously, as humans, we have mechanisms that we use to talk about things. We talk about truths. We talk about in a model, we talk about a hallucination. We talk about things like deception. And I think it's worth taking a step back and talking about what these things mean. So, if an LLM is generating some text, right? And again, it's worth pointing out that this is the core LLM. People build structures on top of them. There are other phases of training that follow this, and those are all kind of important part of the story of the technologies that are being built today, and we can get into some of those. But if you think about just the core piece of an LLM, and if you take a sequence like "the population of Berkeley is" and then what comes next, well, how would you fill that in as a human? Like, what comes next? You tell me.

我会去谷歌搜索，或者用 Perplexity。

I'd Google it. I'd use Perplexity.

所以，你说了人类应该说的话，就是“我其实不知道”。我在 UC Berkeley 工作，所以你知道不只是可能——你知道那里会是一个数字。伯克利的人口是——你知道那里是个数字。你知道不是 10，不是 100，也不是 1000 万。所以你对那里会是什么有很多相关性。大概是个六位数，但你知道的更多：你知道你没有这个信息。你不仅不确定答案，而且你清楚自己根本没有掌握这个信息。这就是元认知。人类在基本操作中高度元认知：你不断意识到自己知道什么、不知道什么，在与他人交谈时还有心智理论，知道对方知道什么、不知道什么，我们一直在建模这些。LLM 和基于它们的技术并不高度元认知。它们处于另一个极端：只是不断输出 token。真实 token 和不代表真实信息的 token 之间的区别——模型底层并不会访问数据库然后想“哦，我没有那个数字”。它只是抛出最佳猜测，如果它见过足够多的信息，最佳猜测往往是真的。但当人类说谎或编造信息时，这些都是元认知陈述。系统在核心形式上没有这些属性。它们只是输出 token，这些 token 代表信息，可能对也可能错。我认为在很多方面，把一切都视为幻觉更有用——许多是正确的，许多不是，对核心模型来说基本一样。人们正在认真思考的一件事是，如何构建具有真相属性的系统，因为底层语言模型并不具备这种属性，它们只有合理性优化。

So, you said exactly what a human should say, which is like "I don't actually know." So, like, I don't know that. And I work for UC Berkeley. So, you knew not just what possible. Like, you knew a number would go there. The population of Berkeley is — you know a number goes there. You know it's not 10. You know it's not 100. You know it's not 10 million. And so, you have a lot of correlations about what goes there. It's going to be some, you know, probably some six-digit number, but you know more than that. You know that you do not have this information. Not only are you not certain of the answer, you know you simply are not in possession of that information. This is metacognition. And people in their kind of basic operation are highly metacognitive. You're constantly aware of what knowledge you have, what you don't, theory of mind in terms of what you're talking to somebody, what they know, what they don't know, and we're constantly modeling these things. LLMs and technologies that have been built on them are not highly metacognitive. They are on the other extreme. They are just cranking out tokens, and the difference between tokens that are true and tokens that do not represent true information — this is not something that deep down in the model, it's accessing a database and thinking, "Oh, I don't have that population number." It's just throwing out its best guess and if it's seen that information enough, its best guess will often be true. But when a human tells a lie or makes up some information, these are all metacognitive statements. Systems don't have these properties in their core form. They just crank out tokens. Those tokens represent information. That information might be right, it might be wrong, and I think it's in many ways more useful to think of everything as a hallucination. Many of them are correct, many of them are not. It's mostly the same to the model at the core. And one of the things people are really thinking hard about is how do we build systems that have truth properties because the underlying language models do not come with that sort of a property. They come with plausibility optimization.

好的，跟我聊聊 Perplexity 吧。我记得大约两年前听过一期播客，采访创始人 Arvind。他们的创始理念是，通过 AI 模型输出的每个句子都必须可验证为真，并且要有来源支持。如果这不是 LLM 本身的功能，那它是怎么工作的？是在上层的架构层吗？具体是怎么实现的？

Okay, talk to me about Perplexity, for example. I remember listening to a podcast with the founder Arvind, I think it was like 2 years ago. And I think their founding thesis was that every sentence they output through their AI model has to be verifiably true and has to be backed up by sources. How does that work if it's not on the LLM itself? Is that at an infrastructure layer above it? Like, how does that actually work?

首先，我是伯克利计算机科学教授，也是 Scale Cognition 的 CTO 和联合创始人。Scale Cognition 的工作简而言之就是构建不说谎的模型，构建可以对其施加保证性控制的模型。我认为构建具有可信和可控属性的系统这一使命极其重要，这是我们领域最重要的任务之一。谈到 Scale Cognition 的模型，这就是它们所做的，也是它们的区别所在：你可以对它们会做什么做出保证，更重要的是，对它们永远不会做什么做出保证。所以我认为这个目标非常重要。而且，智能但不可靠的模型在社会学和经济上的风险非常高，对吧？如果我们思考智能，它是一个多方面的东西。最近人工智能系统中智能的爆发在这些方面并不均匀。比如横向性——一个系统能了解广泛的不同主题，并进行融合对话——这方面的进步已经突破天际，爆炸式增长。可靠性是智能的一个方面，但它没有跟上。你可以从语言模型最初作为给句子合理性打分的机器这一基本方式中看到这一点。这从根本上说不是一项可靠的技术，而是一项有噪声的技术。人们一直在试图改造可靠性，因为可靠性没有跟上智能的其他方面。我认为在未来几年，你会看到人们越来越认识到，没有可靠性的智能在应用上极其有限。

Yeah, I mean, first of all, I'm a professor at Berkeley in computer science and I'm also CTO and co-founder at Scale Cognition. And what we do at Scale Cognition is, in a nutshell, build models that don't lie. Build models where you can place guaranteed controls on them. And I find this mission in general of building systems that have trustable and controllable properties to be incredibly important. This is I think one of the most important tasks we have as a field. And if we talk about Scale Cognition models, this is what they do. This is what differentiates them. Is you can make guarantees about what they'll do and even more importantly guarantees about what they will never do. So, I think this goal is very important. And I think the risks sociologically, economically to models that are sort of smart but unreliable are very high. Right? So, if we think about intelligence, intelligence is a very multifaceted thing. And the recent explosion of intelligence in artificial intelligence systems has not been uniform across those facets. So, things like horizontality — one system that can know about a wide range of different topics and have conversations that blend those together — that advancement has been through the roof, like explosive growth. Reliability is a facet of intelligence and it has not kept pace. And you can sort of see this from the just the basic way where language models started as machines for placing scores over the plausibility of a sentence. Like this is not a fundamentally reliable technology. It is a fundamentally noisy technology. People have been trying to retrofit reliability because reliability has not kept up with the other facets of intelligence. And I think that what you're going to see going forward in the next few years is an increased realization that intelligence without reliability is extremely limited in its application.

你在哪里构建这种技术使其可靠？你实际上是怎么做的？给我详细解释一下。

Where are you building in this technology that makes it reliable? How are you actually doing this? Break that down for me.

是的，我也可以把这个问题放在更广泛的可靠性背景下来谈。我认为整个社区都意识到我们需要可靠性。假设你想构建一个系统。我们的模型正在为一些最大的公司的智能体系统和客户服务等提供支持。当你在进行银行交易、填写处方或预订航班时，你必须把事情做对。犯错的成本非常高。说“哦，我订了你的航班，但实际上没有”或“我给了你别人的处方”这类事情非常糟糕。所以可靠性非常重要。人们现在是如何处理这个问题的？实际上，有两种常见的反模式。一种是说“好吧，我有了这个 LLM，谁知道它会做什么？”有很多这样的例子，系统失控，谈论离题、不恰当的内容，提供虚假信息，编造退款政策——你能想到的各种失败模式。所以人们做的一件事是，他们试图把这个广泛、横向、灵活但难以控制的 LLM 压缩，使得 LLM 只能做决策，比如在对话树的哪个分支路由，或者下一步采取什么行动。这是控制系统的一种方法：限制它能采取的行动，基本上就是沿着老式的有限自动机走，就像 IVR 系统那样。

Yeah, I can also put this in the context of reliability more broadly. I think the whole community is realizing that we need reliability. So let's say you wanted to build a system. Our models are powering some of the biggest companies' agentic systems and things like customer service. And when you are doing a bank transaction or filling a prescription or booking a flight, you have to get things right. The cost of a mistake is really high. The cost of saying, 'Oh, I booked your flight, but actually I didn't,' or 'I gave you someone else's prescription.' These things are really bad. So reliability is so important. How do people deal with this today? Well, there are really two anti-patterns that you see. One is a pattern of saying, 'All right, I've got this LLM and who knows what it's going to do?' And there are all these examples out there of systems going rogue, talking about things that are off-topic, inappropriate, giving false information, making up refund policies—you name it, all kinds of failure modes. So one thing people have done is they've tried to take this broad, horizontal, flexible but hard-to-control LLM and squeezed it down so that all you can do with the LLM is make a decision between, say, which branch of this conversation tree do I route to, or what action am I going to take next? And this is one way to get control of a system: limit the actions it can take to basically walking along an old-style finite automaton, like an IVR system.

给我解释一下。那是什么？

Explain that for me. What is that?

有限自动机？这是人们常用的一种模式，在计算机科学中非常有影响力。它说存在一些状态，你可以通过所谓的“转移”在这些状态之间移动。例如，一个试图进行对话的有限自动机可能会说，首先你问候用户。这是第一个状态。然后你说，“我能帮你什么？”然后有一些转移。他们被允许说三件事中的一件。如果他们没有说其中一件，你就重复问题。这些都是经典的 IVR 系统，它们把对话设计成沿着树走。

A finite automaton? So this is a common pattern people have used and very influential in computer science where you say there are states and you can move between the states with what are called transitions. For example, a finite automaton trying to do a conversation might say, first you greet the user. That's the first state. Then you say, 'What can I help you with?' And there are transitions. They are allowed to say one of three things. If they don't say one of those things, you repeat the question. Those are classic IVR systems that designed conversations as walking down a tree.

那是不是有点像 if-then 语句？

So is that kind of like if-then statements?

你可以把它们想象成有点像那样，但更像是一个流程图，你可以移动。你经常看到这种情况。例如，如果你按照说明操作，就像“如果你的手机卡住了，先做这个，如果不行，再做这个，如果这个灯亮着，做这个，但如果那个灯亮着，做那个。”所以有那种 if-then 的感觉，但实际上只是一个分支树结构。如果你打过老式的客服电话，你会遇到“按 1 进入账单”之类的，然后你沿着树走。让 LLM 受控的一种方法是说，“它们真正能做的只是沿着一些非常僵硬的系统路由。”这意味着你不能做所有这些酷炫的事情，比如做上下文引用、回头改变主意，说“实际上，我这两个问题都有，而且它们相互影响。”在这种情况下，你把这个非常强大的 LLM 放在了最微小、最安全的决策上。你可以获得可靠性，但失去了那种上下文能力。这就像模型脑叶切除术。模型只被允许做最微小的事情。你经常看到的另一种反模式是：你有一个模型，但它不可靠。所以你引入第二个模型来检查第一个模型。借用关于正则表达式的笑话，现在你有两个问题了。不仅你有模型检查模型，而且在很多情况下，人们把这些构建成几十个模型的星座。不仅你有模型检查模型检查模型，这使得很难推理它们会做什么，你还必须等待。你等待第一个模型完成，然后第二个模型检查它，第三个模型检查那个，延迟变得非常高。这个想法的一个吸引人之处是，如果一个模型 90% 的时间是正确的，那么两个模型就更正确。但这在实践中并不是这样运作的。在实践中，往往会出现棘手的情况。棘手的情况会混淆所有模型，你会得到级联错误，而且很难推理。我会把这两件事都描述为试图在根本上基于 token 的不可靠模型上改造上层结构，试图让它们变成它们不是的东西。Scale Cognition 的假设是，改造可靠性不是正确的方法。相反，你应该直接在模型中架构这些东西。所以如果你看看我们的模型 APT-1，它有不同的架构。它不是逐个 token 生成，而是在信息片段和你可以对这些信息采取的行动的层面上运作。这意味着它是一个专家模型。它被设计为在一个温和的上下文中运作，一边是人在说话，说人类语言，做上下文引用、回头改变主意、提问。但另一边，有一堆 API 来帮助你解决这个银行问题。

You can think of them a little bit like that, but it's more like a flowchart where you can move. You see this a lot. For example, if you're following instructions, it's like, 'If your phone is stuck, first do this, and then if that doesn't work, do this, and then if this light is on, do this, but if that light is on, do that.' So there's that if-then feeling, but it's really just a branching tree structure. If you've called old-style customer support, you've gotten this like 'push one for billing' and you walk down the tree. One way to get LLMs under control is to say, 'All they really can do is route along some very rigid system.' That means you can't do all these cool things like make contextual references, go back and change your mind, and say, 'Actually, I have both of these problems, and they're interacting.' In this case, you've taken this really powerful LLM and put it in charge of the tiniest, safest little decisions. You can get reliability, but you lose that contextual power. This is like model lobotomy. The model's only allowed to do the tiniest little things. There's another anti-pattern you see a lot: you have a model, and it's unreliable. So you bring in a second model to check the first model. To steal the joke about regular expressions, now you have two problems. Not only do you have models checking models, and in many cases people build these out to constellations of dozens of models. Not only do you have models checking models checking models, where it becomes very hard to reason about what they're going to do, you also have to wait. You wait for the first model to finish, then the second model to check it, and the third model to check that, and the latency gets really high. One appealing thing about this idea is the thought that if one model is right 90% of the time, then two models are right even more. But that's not how these things tend to work in practice. What tends to work in practice is there are tricky cases. Tricky cases confuse all the models, and you get cascading errors, and it's very hard to reason about. I would characterize both these things as attempts to retrofit onto a fundamentally token-based unreliable model superstructures that try to make them into something they're not. The Scale Cognition hypothesis is that retrofitting reliability is not the way. Instead, you want to architect these things directly in the model. So if you look at our model APT-1, it has a different architecture. Instead of generating token by token, it operates at the level of pieces of information and actions you can take over those. That means it is a specialist model. It's designed to operate over a gentle context where you have a person on one side speaking human, saying things in context, referencing back, changing their mind, asking questions. But on the other side, there are a bunch of APIs to help you with this banking problem.

我们首先需要验证你的身份，然后才能进行转账，而且转账必须是从你的账户发起的。这涉及一系列按特定顺序执行的操作，一旦搞错，很多地方都可能出问题。你不想凭空捏造别人的账号，或者做出与用户要求不符的操作，也不想按照一个根本不是你的退款政策来行事——那只是有人在 2019 年 Reddit 上随口说的。所以我们的模型架构是围绕移动信息片段并对其执行操作来设计的。这样做的好处是，token 本身没有语义。一个 token 只是最终会组成账户余额的一串单词中的一小块。你把几个 token 拼在一起，就成了账户余额。在我们的系统中，因为我们不是逐 token 地组装信息，而是对已有信息做决策，这就成了一个更偏向元认知的系统。如果你不是逐字逐句地合成信息，那么当你必须从已批准的来源获取已有信息时，有一整类错误你就再也不会犯了。所以这是一种不同的模型架构。

We need to first authenticate you, then in order to do the transfer, it has to actually be from your account. There's this web of actions you take in certain sequences, and a lot of things you can do wrong if you mess that up. You don't want to hallucinate someone's account number or do something other than what they asked you to do, or act according to a refund policy that's not your refund policy—it's just something somebody was talking about in 2019 on Reddit. So our model is architected around moving around pieces of information and taking actions on them. What's nice about this is tokens don't have semantics. A token itself is just a piece of a sequence of words that will eventually come to be an account balance. You put a couple tokens together and it's an account balance. In our systems, because we're not assembling information token by token, we're making decisions about existing information, this starts to be a more meta-cognitive system. If you're not synthesizing information character by character, there's a whole class of mistakes you just can't make anymore when you're required to take information that already exists from approved sources. So it's a different architecture to the model.

好的，我想了解一件事。你们目前专门服务于客户体验和市场领域，对吗？

Okay, here's something I want to understand. You're specifically serving in customer experience and markets right now, correct?

是的。

Yep.

从我在市场上看到的情况来看，通常是某种大语言模型，然后事后才加装这些措施。而你们所做的，实际上是直接把可靠性融入到大语言模型本身，对吗？

From what I've seen out there in the market, it's usually some sort of LLM and then they retrofit these measures after the fact. What you're doing here is actually baking in that reliability to the LLM itself, correct?

是的，没错。所以你可以这样理解：首先，这个模型本身是一种语言和行动模型。它理解语言，也理解执行操作以及这些操作背后的语义。正如你所说，它能理解人以及人们在不同语境下说话的各种方式。当它执行操作时，业务逻辑、来源保证和信息完整性都内置于模型的操作和架构中，这意味着你不需要事后检查，也不需要引入第二个模型。这意味着我们的模型不仅更准确，而且速度可以快很多。事实证明，它们也可以小很多。

Yep, that's right. So you can think of this as, first of all, the model itself is sort of a language and action model. It understands language but also understands taking actions and the semantics behind those. And you can think of it exactly as what you said. It's going to understand people and all the ways people speak contextually. When it takes actions, the business logic, provenance guarantees, and information integrity are all baked into the operation and architecture of the model, which means you don't need to post-check it. You don't need to bring in a second model. That means our models are not only more accurate, they also can be a lot faster. And it turns out they can be a lot smaller, too.

这是否意味着，对于每一个签约客户，你们都必须专门为他们设计一个大语言模型？

Does that mean for every single company that you get on as a client, you have to specifically design an LLM for them?

不。

No.

那是怎么做到的呢？我在想，你说这是内置于上下文中的，对吧？一家银行和另一家银行做出的决策不同。我很好奇，当客户不同时，这些操作是如何实际注入到大语言模型中的。

How does that work? I'm wondering, like, you're saying this is baked in the context, right? One bank makes different decisions from another bank. I'm curious how those actions are actually infused into the LLM when they're different customers.

这是个好问题。首先，我们的模型和市面上大多数模型一样，有一个训练过程，当然不需要为每个用户从头开始训练。它训练在什么数据上呢？基本上是训练在一大堆模拟的智能体交互上。所以模型知道，在包含大量复杂 API 和有一定历史记录的人类用户的环境中，对话的模式是什么样的。它知道如何将人们的需求与操作联系起来，也知道如何将操作串联成序列。因为最终，如果你想帮人们把事情办好，你必须执行多个操作，按顺序执行，并且不能出错，这意味着你必须遵守所有相关的语义和业务逻辑。所以它在训练数据中已经见过各种各样的情况了。基本方法就是可验证数据生成。因此你不需要为每个用户专门训练，因为它已经见过各种各样的智能体上下文、各种大小和形状的 API、人们所处的各种情境，以及许多不同的领域。所以它开箱即用，已经知道如何做这些事。那么问题来了，正如你所说，一家银行有一套政策，另一家银行有另一套政策，银行业不同于旅游业或医疗保健业。那么区别在哪里呢？大语言模型中标准的控制界面是提示词，对吧？我们都熟悉为大语言模型编写提示词。我认为这实际上是大语言模型最令人印象深刻的地方——可塑性就在这里体现。你可以给它一个提示词，让它做几乎任何事情；你只需描述任务，它就开始运行。但这也是缺乏可靠性的根源。所以你写了一个提示词，系统开始执行。假设它没做对，你怎么办？你修改提示词。你的控制界面就是提示词。所以你加一些指令。好吧，也许还是不够可靠。那怎么办？你把指令移到末尾。哦，还是不行。好吧，你把它们改成全大写。还是不行。好吧，你在指令末尾加感叹号。加到第三个感叹号的时候，你意识到这可能不是正确的做法。所以我们的模型做的一件事是，它们还暴露了其他控制界面，让你能够阐明什么样的业务逻辑、策略等——这些是模型被训练去使用并且其结构也围绕它们构建的。我还想说一点，就是你放入提示词的内容与你得到的结果之间关系并不明确。提示词有点像一种请求。如果你思考一下技术，以及我们作为一个领域、一种文化构建可靠技术的方式——如果我去教一门计算机科学入门课，你会学到什么？CS 101 这类课程中最重要的教训是什么？关于我们如何能够构建和推理可靠的大型系统，最重要的教训就是模块化。这个理念是，你可以把复杂的东西分解成更小的部分，每个部分都有一个定义良好的契约来描述它的功能：输入是什么，输出是什么，它们之间的关系是什么。如果你提供这种形式的输入，就能保证得到这种输出。然后你可以独立地处理每个部分，因为关于它们如何交互有一个模块化的契约——它们会做什么是有保证的。

That's a great question. So, first of all, our models, like most models out there, have a training process and it does not have to be done from scratch for each user, of course. What is it trained on? It's basically trained on a bunch of simulated agentic interactions. So the model knows, in an environment of a whole bunch of complex APIs and a human with some history, the patterns of conversation. It knows how to connect what people want to actions. It knows how to chain actions together into sequences. Because ultimately, if you want to get things done for people, you have to take multiple actions, you have to take them in sequence, and you have to not mess up, which means you have to obey all the semantics and business logic associated with them. So it has already seen a wide range of these sorts of things in its training data. The basic approach here is one of verifiable data generation. So you don't have to train it specifically for every user because it's already seen a wide range of agentic contexts, a wide range of sizes and shapes of APIs, situations people are in, lots of different domains. So it already out of the box knows how to do these. The question then is, as you say, one bank has one policy and another bank has another policy, and banking is different from travel or healthcare. So what makes a difference? The standard control surface you have in an LLM is a prompt, right? We're all familiar with writing prompts for LLMs. I think this is actually the source of what's most impressive about LLMs—this is where their plasticity comes into play. You can give it a prompt to do kind of anything; you just describe the task and it's off to the races. But it's also the source of the lack of reliability. So you write your prompt and the system does its thing. Well, let's say it didn't do the right thing. What do you do? You change the prompt. Your control surface is the prompt. So you add some instructions. Okay, maybe it's still not reliable enough. So what do you do? You move those instructions to the end. Oh, it's still not working. Okay, you put them in all caps. All right, that's not working. Okay, you start putting exclamation marks at the end of the instruction. And somewhere around the third exclamation mark, you realize that this is maybe not the way. So one of the things that our models do is they also expose other control surfaces that let you articulate what sorts of business logic, policies, and things like that that they are trained to use and that they are structurally built around. I should say a little bit as well that this issue—what you put into the prompt has a sort of ill-defined connection to what you get out. A prompt is kind of a request. If you think about technology and the ways we've built reliable technologies as a field, as a culture—if I go and teach an intro CS class, what do you learn? What's the big lesson in a CS 101 kind of class? The big lesson about how we have been able to build and reason about large systems that are reliable is modularity. The idea that you can take something complex, break it into smaller pieces where each piece has a well-defined contract that describes what it does: what's its input, what's its output, what is the relationship between them. If you provide inputs of this form, you're guaranteed to get this out. And then you can take each of those pieces and work on them independently because there's a modular contract about how they interact—there's guarantees about what they will do.

当然，它们可能会有 bug，但当它们正常工作时，这些保证让你能够推理它们将如何按顺序工作。这种分解和模块化的思想是我们在计算机科学中构建大型可靠系统的最强大工具。

And you know, of course there can be bugs in them, but when they are working correctly, those guarantees let you reason about how they will work in sequence. And this idea of decomposition and modularity is the most powerful tool we have in computer science for building large, reliable systems.

解释一下。所以，我在两个维度上理解这一点，但我想在第三个维度上理解它。好的。一个非常庞大、复杂的系统由各个模块化组件组成，这些组件有定义的契约，从而产生可靠的结果。

Explain that right. So, I'm understanding this on two dimensions, but I want to understand it in the third dimension. Okay. Very large, complex system made up of individual modular components which have defined contracts which lead to reliable results.

是的，当然。

Yeah, sure.

完全同意。我困惑的是它们相互交互的顺序和优先级。所以，我们知道契约是正确的，但我们如何知道哪个先执行、哪个后执行？

Totally. What I'm confused about is the order and the priority with which they interact with each other. So, we know that the contracts are correct, but how do we know what goes first and what goes last?

是的，显然当你进行这种分解时，它并不简单，也不是免费的。思考大型系统架构的人会考虑：有哪些部分？我们应该如何将这个大的问题切分成小问题？很多编程、很多计算机科学实际上归结为将大的难题分解成更小、更易管理的问题。这种分而治之只有在完成后能够组装成一个整体时才有效。所以，例如，你可能有一个复杂的企业系统，其中一个概念是，在某个时候我们需要能够请求这些信息。所以，有一个数据库，它的工作是：你把信息放入数据库，你取出信息，并且事务性数据库遵循非常精确的语义。这就是一个部分。然后人们可以继续工作，改进数据库，使其更快更好。只要它们都按照数据库应有的方式工作，你就可以在它们之上构建。然后还有另一个部分，一个路由系统。你将大系统分解成小部分。这不仅管理了复杂性，还意味着它们可以独立开发。所以，它让你能够用更多的人并行处理不同的部分。这是计算机科学入门课中的一个重要教训。机器学习是如何工作的？我们拥有的最强大的工具是端到端优化。把数据放在一边，把动作放在另一边，进行强化学习，将一切反向传播，中间的一切似乎以越来越隐式的方式工作。这是一个强大的工具。你拿整个网络，训练它根据文档的整个前缀预测下一个词。沿着这条路走下去，就有了 LLM。但这些方法是矛盾的，对吧？这是这类技术中许多摇摆之一：在能够对某物行为做出保证与能够说“我实际上不想知道或关心里面发生了什么”之间的平衡。解决这个问题很难，因为事实证明有些东西你确实想要控制。比如你希望你的策略得到遵循。你希望这个 API 的语义得到尊重，并且你希望得到保证。但还有一大堆东西你不想涉足，比如试图以某种手动方式写下每个英文单词的含义。所以我认为，当你审视现有的技术时，它们没有正确的组合。较老的技术没有足够的上下文现实世界常识性知识。而如果你看 LLM，它们缺乏可靠性。你现在在领域里看到的是人们在弥补这种可靠性的缺失。我认为这是目前最重要的使命：以可信和可控的方式构建这些系统。这就是我们专注的方向。

Yeah, so there's obviously when you do this decomposition, it's not trivial and it's not for free. And what somebody thinking about the architecture of big systems is going to think about is: what are the pieces? How should we slice this big problem into small problems? A lot of coding, a lot of computer science really boils down to dividing big hard problems into smaller more manageable ones. And that divide and conquer only works if you can then assemble it when it's done into a whole. So, you might, for example, have a complex enterprise system and one concept is, well, at some point we're going to need to be able to ask for this information. So, there's a database whose job it is: you put information in the database, you get information out, and there's a very precise semantics that transactional databases obey. And then that's one piece. So, people can go off and they can work on databases, make them better and faster. And as long as they all work the way databases are supposed to work, you can build on top of them. And then over there there's another piece, a routing system. And you divide big systems into small pieces. And that not only manages the complexity, it also means that they can be worked on independently. So, it lets you use more people to work on different pieces in parallel. So, that's this great lesson from an intro CS class. How does machine learning work? Well, the most powerful tool we've had in machine learning has been end-to-end optimization. Put the data in one side, put the actions in the other side, do your reinforcement learning, propagate the heck out of everything all the way back, and everything in between seems to work in increasingly latent ways. And that's been a big powerful tool. You take all of the web and you train it to predict the next word on the basis of the whole prefix of the document. And down this road lies LLMs. But these things are at odds, right? And this is one of the many pendulums there are in this kind of technology: the balance between being able to make guarantees about what something does versus being able to say I actually don't want to know or care what's going on in there. And working this out is hard because it turns out there are some things that you do want to be able to control. Like you want your policy followed. You want the semantics of this API respected and you want that to be guaranteed. But then there's a whole bunch of stuff you don't want to get into the business of trying to write down what every word in English means in some manual way. So I think that when you look at the technologies that are available, they don't have the right mix of these things. Older technologies don't have enough of the contextual real-world common sense plastic knowledge. And if you look at LLMs, they don't have the reliability. And what you're seeing now in the field is people reconciling that lack of reliability. And I think that's the most important mission right now: to build these systems in ways which are trustable and controllable. So that's what we're focused on.

好的，我认为可信和可控的部分在商业企业层面完全合理，对吧？如果你与一个企业客户交谈，他们采用你的服务，你与他们合作。如果你显然在 99% 到 100% 的时间里得到正确的结果，那就有意义。你会得到更多合同，卖出更多服务，卖出更多产品。但除了商业性，除了金钱，为什么这很重要？为什么这些系统的正确性很重要？

Okay, I mean I think the trustable and controllable part makes complete business enterprise commercial sense, right? If you're speaking to an enterprise customer and they take you on, you're working with them. If you obviously get the right results 99 to 100% of the time, then that makes sense. You get more contracts, you sell more of your service, sell more of your product. But beyond commerciality, beyond money, why is it important? Why is it important these systems are correct?

是的，我同意你所说的一切。我同意，显然你查询到的银行账户余额应该是你的实际余额，而不是别人的或一个看似合理的数字，这很重要。这当然是对的，但我认为当一个人去像 ChatGPT 这样的系统提问时，他们能够信任返回的信息，这一点同样重要。总的来说，有一个问题：人们何时以及如何需要信任信息？他们如何判断什么是可信的？这有一段完整的历史，最终这是一个数字素养的问题。我认为人们构建对社会没有腐蚀性的技术非常重要。所以，如果你想了解某个话题，我们倒回 30 或 40 年前，你会去图书馆，查资料，拿几本书，借出来，然后阅读那些书。你仍然需要处理这些信息，综合它们，但你不太会担心：这是幻觉吗？这是编造的信息吗？因为为了呈现在你面前，有人研究了这本书，编辑编辑了这本书，然后图书馆决定购买它，所有这些步骤和社会结构在本质上对信息进行了合理的筛选。然后时间向前推进一点，你使用网络搜索。所以，如果你回到 2000 年代左右，一旦搜索引擎变得可靠，你输入一个关于那个话题的搜索，你会得到一堆网页。现在，这些网页不像图书馆的书那样经过高度筛选。有些甚至实际上不相关，因为搜索引擎并不完美。有些网站看起来有点可疑，也许你在阅读时处于一种试图判断的状态：这真的与我的查询相关吗？这个网站有权威性吗？它可信吗？所以现在，事情并没有全部为你筛选好。你必须自己决定相信什么。你是如何做出这些决定的？

Yeah, I mean I agree with everything you said. I agree it's obviously important that the bank account balance you get cited be your actual bank account balance and not someone else's or a plausible number or whatever. That's certainly true, but I don't think it's any less important that when a person goes to a system like ChatGPT and asks a question that they be able to trust the information that comes back. And in general, there's this question of when and how do people need to trust information? How do they decide what is trustworthy? And there's a whole history of this, and this is ultimately a question of digital literacy. I think it's very important that people build technologies that are not socially corrosive. So, if you wanted to learn about some topic, and we rewind 30, 40 years, you would go to the library, you'd look it up, you'd get a couple books, you'd check them out, and you'd read those books. And you would still have to process this information, synthesize it, but you'd be less worried about is this a hallucination? Is this made up information? Because in order to get in front of you, somebody researched this book, an editor edited the book, then the library had to decide to buy it, and there are all of these steps and social constructs that did a reasonable job of curating, essentially, the information. And then you go forward a little bit in time, and you go to web search. So, if you go to the 2000s or something like that, and once search engines were solid, and you type in a search on that topic, and you get back a bunch of web pages. Now, these aren't as highly curated as the books from the library. Some of them aren't even actually on topic because search engines aren't perfect. Some of these websites look a little bit suspicious and maybe you're reading something and you're in a mode of trying to decide: is this actually relevant to my query? Is this website authoritative? Is it trustworthy? And so now things haven't all been curated for you. You're having to make decisions about what to believe. How did you make these decisions?

所以这个网站有很多拼写错误，可能加载非常慢，而且图形设计看起来像是 90 年代的。你试图对信息的深层属性做出判断：这些信息可靠吗？可信吗？而一路上有很多迹象可以帮助你。软件工程中有一个概念，由 Kent Beck 提出，Martin Fowler 在其极具影响力的重构书籍中推广，那就是代码坏味道。我不知道你是否了解代码坏味道，但我认为这是一个绝妙的想法。代码可以正确或错误，没有理由说代码不能各方面都很漂亮但实际上是错的。然而，在实践中，很多情况下，架构系统深处可能存在问题，但表面上有一些迹象。例如，在软件工程中，这通常表现为函数的参数列表变得很长，或者其他表面上的所谓坏味道。我喜欢这个类比，因为它不是任何事情的证明，只是相关。再次强调，是相关。它只是与底层存在问题相关，因为存在产生这些问题的机制，而这些机制具有相关性。所以如果你访问一个网站，有拼写错误，并不意味着信息是错误的，但它是相关的。如果网站加载慢，也是相关的。你有了所有这些坏味道，所有这些信息坏味道，对吧？它们有助于建立信任。如果你使用机器翻译系统，输入一些你不懂的语言的文本，它输出你的语言。你现在能读了，但这真的是翻译吗？还是只是胡乱拼凑的词语？它可能意思相反，或者完全无关。你怎么知道？你不知道，但有坏味道。早期的翻译系统，如果翻译中仍然有很多源语言的词或片段，如果语序不像英语，感觉语序还卡在另一种语言里，这些就是翻译错误的坏味道。它们与数据不足或模型在该领域无法足够灵活地处理相关。所以即使你不懂另一种语言，很多时候也能判断出机器翻译很糟糕。而对于 LLM，系统的错误和不正确陈述没有坏味道。你问一个问题，得到一个答案，它很自信、流畅，风格和权威性与真实内容相同，你无法分辨。这很严重，因为如果你有一个神谕，你去问问题，它给出答案，有时对有时错，而你无法区分。只有两种可能的结果：人们相信一切，或者人们什么都不信。我认为我们正在看到这两种情况，而两者都不是你想要的。

So this website has got a bunch of spelling errors and maybe it loaded really slowly and the graphic design looks like it's straight from the '90s. And so you're trying to make a decision about this deep property of information: Is this information reliable? Is this trustworthy? And there's just a lot of signs that help you along the way. There's this idea in software engineering that was coined by Kent Beck and popularized by Martin Fowler in a very influential book on refactoring, which is this idea of code smells. I don't know if you know about code smells, but I think this is just a brilliant idea. So code can be right or wrong and there's no reason why code can't be like beautiful in every way but just actually wrong. However, it turns out in practice that in many cases there are things that can be deeply wrong with an architected system that have superficial tells. So for example, in software engineering this is often things like the function's argument list is getting really long or other surface what they call smells. And I love this analogy because it's not proof of anything. It's just correlated. Again, with correlations. It's just correlated with there being something wrong underneath because there are these mechanisms that give rise to these problems that have correlations in them. And so if you went to the website and there were typos, it doesn't mean the information is wrong, but it's correlated. If you go to the website and it's slow to load, it's correlated. And you had all these smells, all these sort of information smells, right? That help build trust. If you went to a machine translation system and let's say you go to a machine translation system and you type in some text in a language you don't speak and it pops out in your language. Well, you can read it now, but is this actually a translation or is this just a bunch of words thrown together? It could mean the opposite of what it's supposed to mean or be totally unrelated. How would you know? Well, you don't, but there are smells. So, early translation systems, if the translation still had a bunch of words or chunks that were from the source language, if it felt like the word order didn't feel like English, it maybe felt like the word order was still stuck in the other language, these would be sort of mistranslation smells. They're correlated with the fact that there's not enough data or this is an area where the model is not able to do things in a sufficiently flexible way. And so you could tell a bad machine translation without speaking the other language, at least a lot of the time. With LLMs, the mistakes and the incorrect claims of the system have no smells. You ask a question, you get an answer, it's confident, it's fluent, it's the same style and authoritativeness as something that's actually true, and you can't tell. This is a big deal because if you have this oracle that you go to and you ask questions and it gives you answers and sometimes they're right and sometimes they're wrong and you can't tell the difference. There's only two possible outcomes: people believe everything or people believe nothing. And I think we're seeing both and neither is what you want.

我问你这个问题，这些文章难道没有激励来随着时间的推移给你更多真相吗？比如，如果你去一个网站，一半时间它给你的答案毫无意义，你会说，“好吧，我去另一个网站。”我认为这是很多人抛弃 OpenAI，然后转向 Claude，然后 Claude 遇到自己的问题，人们又抛弃它转向其他东西的原因之一。但他们难道没有激励吗？他们的业务依赖于你能够信任它。所以他们当然应该构建你信任的系统。

Let me ask you this, are incentives not aligned for these articles to give you more truth over time, right? Like if you were going to an article and half the time they fed you back an answer that didn't make any sense, you'd be like, "Okay, I'm going to another article." I think it's a lot of the reason that people kind of ditched OpenAI, and then they went to Claude, and then Claude ran into its own problems, and then people are ditching that and moving into other things. But are they not incentivized? Like their business runs on you being able to trust it. So surely they should build systems that you trust.

一旦你在 AI 系统周围提到激励这个词，答案总是很复杂。所以他们有激励吗？如果你纯粹问这个问题，如果你有两个基于聊天的信息系统，一个比另一个准确得多，你有激励变得更准确吗？是的，当然，对吧？但还有其他激励。例如，我们早些时候讨论了 LLM 的核心是什么，但之后你还会做其他事情，比如后训练、对齐训练，我认为很多人看到的是像 RLHF（基于人类反馈的强化学习）这样的步骤，简而言之，有很多方法可以做到这一点。简而言之，你问人类，你更喜欢这个答案还是那个答案？有时你甚至能看到这个，你与系统交互，它问你，你更喜欢 A 还是 B，对吧？那会怎样？如果你说我喜欢 A，它就会更频繁地给出 A。

As soon as you say the word incentive around an AI system, the answer is always it's complicated. So are they incentivized? If you purely ask the question, if you have two chat-based information systems, and one is much more accurate than the other, are you incentivized to be more accurate? Yes, of course, right? But there are other incentives. So for example, we talked early on about what the core of an LLM is, but there are other things you do after that in terms of post-training, in terms of alignment training, something that I think a lot of people see because it shows up to them is steps like RLHF, reinforcement learning from human feedback, where in a nutshell, there's many ways to do this. In a nutshell, you ask humans, do you like this answer or this answer better? And sometimes you even see this, you're interacting with a system, and it asks you, do you like A or B better, right? And what does that do? Well, if you say I like A better, it's going to do A more often.

所以也可能注入偏见，对吧？

So could infuse bias as well, right?

绝对会。所以当强化学习成为机器学习中如此重要的工具时，听起来很 fancy，比如什么是机器学习？抱歉，什么是强化学习？强化学习实际上核心非常简单，有很多很多方法，以及一堆 brilliant 的创新，我要把它们完全卡通化。其核心是尝试事物，并在有效的事物上加倍投入。所以，如果你训练一个系统玩游戏，这实际上效果很好。比如，计算机现在在象棋和围棋等游戏中比人类强很多。系统如何变得如此优秀？嗯，一个强化学习系统，比如围棋，它如何学习下棋？它会玩一些游戏，甚至可能与自己玩，对吧？也许它随机下。它下得不是很好，但最终有人赢了，因为你知道游戏规则，你可以在最后识别谁赢了。程序的一个副本赢了，一个输了。所以，你应该做赢的程序所做的事，多做那些。少做其他事。如果你做得足够多，你最终会学会。这实际上是很多学习的基础。我们尝试事物，获得经验，不做导致糟糕体验的事，重复导致良好体验的事。所以强化学习非常强大，但结果是你学到的正是你优化的东西，这听起来不错，因为那是你想要的，对吧？但这是有问题的。例如，如果你去找人类，只问他们更喜欢什么，你会学会产生人类更喜欢的输出。

Absolutely. So when reinforcement learning has been such an important tool in machine learning, sounds fancy, like what is machine learning? Sorry, what is reinforcement learning? Reinforcement learning is actually something really simple at its core, and there are many, many ways, and a bunch of brilliant innovations that I'm going to totally turn into a cartoon. At its core, reinforcement learning is trying things and doubling down on the ones that worked. So, this actually works great if you're training a system to play a game. Like, if you take computers are now better than humans at games like chess and go and kind of by a lot. How do systems become so good? Well, a reinforcement learning system, say for go, how will it learn to play? Well, it'll play some games, maybe even against itself, right? Maybe it plays randomly. It's not very good at playing, but in the end someone wins because you know the rules of the game, and you can identify at the end who won. Well, one copy of the program won, and one lost. So, you should do whatever the winning program did, you should do more of that. You should do less of the other thing. And if you do this enough, you eventually learn. And this is sort of at the bottom of a lot of learning. We try things, we have experiences, we don't do the things that led to bad experiences, and we repeat the things that led to good experiences. So, reinforcement learning is very powerful, but what it turns out is you learn kind of exactly what you optimize, and that sounds good because that's what you want, right? But, it's problematic. And so, for example, if you go to humans and you just ask them what they like better, you'll learn to produce output that humans like better.

这就像回形针理论？

It's like the paperclip theory?

哦，是的。

Oh, yeah.

是的。

Yeah.

是的，就是这样。这是一个非常小、可能危害较小的回形针工厂，但同样的问题：你告诉一个吸尘器优化它吸了多少灰尘，但没有告诉它不能把灰尘倒回地上。没有理由认为它不会把地面弄脏、清理干净、再弄脏、再清理干净，从而累积奖励。我们觉得这些例子好笑，因为你会想：“哦，我只要指定得更清楚就行了。”但事实上，每当你构建一个奖励信号，它就会偏离真相。所以你真正想要的是像你希望构建的那样的系统。这就是我们的使命。你想要构建的系统是那些会告诉你真相的系统，你可以说你不希望平衡真相和用户满意度。你想说，在一切必须真实的前提下，再优化其他东西。你应该把它看作约束优化。你必须说实话。在真实的事情中，你当然有一些灵活性。

Yes, it is. This is a very tiny and maybe less harmful paperclip factory, but this same question: you tell a vacuum cleaner to optimize how much dirt it picks up and you don't tell it that it can't dump that dirt back on the ground. There's no reason to think it's not going to make a mess, clean it up, make a mess, clean it up and rack up reward. And we laugh at these because they're funny because you're like, 'Oh, well, I would just specify it better.' But really anytime you build a reward signal it diverges from the truth. And so what you really want is systems like the thing you would want to build. This is our mission. The thing you want to build is systems that will tell you the truth, and you can say you don't want to balance truth and user satisfaction. You want to say subject to the constraint that everything be true, now optimize some other stuff. You want to think of it as constrained optimization. You have to tell the truth. Amongst things that are true, of course, you have some flexibility.

你优化真相是因为你功利主义以及你如何看待世界吗？

Are you optimizing for truth because you're utilitarian and how you think about the world?

我认为目前我们与模型的关系中，真相甚至不是一个首要概念。

I think that right now we are in a situation with models where truth really isn't even a first order idea.

不，为了给观众描绘一个画面，每个人都用过 ChatGPT，每个人都用过代码，它会告诉你你想听的，而不是你需要听的。它会肯定你提出的商业模式，肯定这个，而不会真正挑战你。你必须明确要求它。所以所有这些模型实际上都被激励去让你开心，而不是提供真实的结果。

No, and to paint that as a picture for the audience, everyone's used ChatGPT, everyone's used code and it will tell you what you want to hear rather than what you need to hear. It will reaffirm your kind of business model that you're proposing, reaffirm this and it won't actually challenge you. You have to explicitly ask it. So all of these models are actually incentivized to make you happy rather than provide you a true result.

完全同意。有趣的是，其实很容易看出它们为什么会这样，因为时不时地，它们会向人类展示——无论是在训练时闭门进行，还是实时来自真实流量——它们会向人类展示“你喜欢 A 还是 B？”结果人们可能更喜欢 A。人们喜欢系统说“这是一个有洞察力的问题，切中了要害”。人们喜欢这类东西。人们不喜欢系统说“嗯，听着，我不想打击你，但你有点困惑。这是一个经典错误。让我试着澄清一下。”平均而言，人们不喜欢那样。现在，有些人可能更喜欢那样。有些人对谄媚过敏，但总的来说，反馈回系统的信号是“是的，要自信。要谄媚。告诉人们他们很棒。”人们喜欢被认可，对吧？

Totally. And it's interestingly it's actually pretty easy to see why they're like this because from time to time they're going to show a human either kind of behind closed doors when they train it or even live from real traffic. They're going to show a human something that says 'Do you like A or do you like B?' And it turns out people maybe like A better. Well, people kind of like it when the system tells you 'That's an insightful question and it cuts to the heart of the matter.' People like that kind of stuff. People don't like when it's like 'Well, look, I hate to break it to you but you're kind of confused. That's a classic mistake. Let me try to clear this up.' People on average don't like that. Now, some people may prefer that. There are people who are going to be allergic to sycophancy but broadly speaking the signal that comes back into the system is like 'Yeah, be confident. Be sycophantic. Tell people they're amazing.' People like to be validated, right?

而且你在主题演讲上也会告诉人们同样的事，对吧？你即将上台。是的，我让你做最新模型的主题演讲。你希望它上台说“不，实际上我们需要再考虑一下。你不知道自己在说什么”，还是“这是一个漂亮的答案，配上一张漂亮的图表，可能对也可能错，但看起来很棒”？

And what are you going to tell people on a keynote as well, right? You're about to take. Yeah, I'm putting you to your keynote of your latest model. You want it to come up and say 'No, actually we need to think about this more. You don't know what you're saying' or 'Here's a beautiful answer with a beautiful graph that may or may not be wrong but it looks great.'

而且人们显然都知道幻觉。我们不必过多谈论幻觉，但显然人们知道这些，并且不想要它们。如果你问“嘿，你希望系统只是告诉你虚假的事情吗？”人们会说“不，我不想要那样。”所以问题不在于人们不想要真相。而在于这是一个多属性的事情，只要诚实与其他因素进行权衡，就会有一个损失另一个的情况。这就是为什么我认为区分很重要——这不仅仅是功利主义意义上的——区分你能保证的东西和你只想优化的东西。

And as people are like, obviously people are aware of hallucinations. We don't have to talk too much about hallucinations but obviously people are aware of these and they don't want them. If you ask 'Hey, do you want the system to just tell you false things?' people say 'No, I don't want that.' And so the problem isn't so much that people don't want the truth. It's that it's a multi-attribute thing and as long as honesty is traded off against other factors, there's going to be some loss of one for the other. That's why I think it's important to separate, and this isn't just in a utilitarian sense, it's important to separate what you can guarantee from what you just would like to optimize.

我有一个技术问题。抱歉打断你。如果这实际上是在物理或软件层面上内置的呢？比如，这些大语言模型中让用户开心的激励来自哪里？如果这就是正在发生的结果，那它一定存在于某个地方。

I have a technical question. Sorry to interrupt you. What if it's actually baked in on like a physical or a software level? Like where does that incentive come to make the user happy in these LLMs? That it must exist somewhere if that's the result that's happening.

再说一次，每次我谈论时，都有很多技术、很多模型和很多公司，我说的任何话都会有点夸张，但一个相当好的核心体现方式是：你直接拿你的模型及其权重，它会生成各种输出，并带有相关的概率。当你训练这些模型时，你基本上是在说，“嘿，把这个输出的概率提高。”它是怎么做到的？它摆动这个参数，摆动那个参数。具体哪些参数朝哪个方向移动导致那个输出的概率上升——这就是反向传播所做的。基本上有一堆微积分告诉你，如果我摆动这个，那边会发生什么。所以当你说，“提高这个东西的概率，降低那个东西的概率”时，机器学习的数学就接管了。从机制上讲，这体现在系统中，当你说，“我更喜欢那个好答案，而不是‘对不起，你搞混了’。”你把两者都呈现给模型，给它们打分，然后提高一个，降低另一个。很多机器学习都归结为：这是一个答案，那是另一个答案，你按这种方式打分，调整你的分数。如果你需要提高某个东西的分数，你就看导致它的权重，然后提高它们。如果你需要降低分数，你就看相应的权重，然后降低它们。这不仅是大型语言模型或 Transformer 的核心机制，甚至不仅仅是神经网络中单个神经元的核心机制。这或多或少是所有机器学习进行的方式。如果你回到最早的加权函数表示，比如几十年前的 McCulloch-Pitts 感知机，你会指着某个东西说，“这是正例还是负例？”系统会说，“负例。”你会说，“不，不对，是正例。”你怎么让它变成正例？你确保如果它再次分类这个，它会说正例。怎么做？你朝增加这个分数方向稍微改变数字。你反复这样做，权重就体现了决策边界。我们看着它说，“最终，如果你得到足够多的相关性，我们称之为知识。”关于什么构成知识，有一整套哲学，但在通常用法中，所有这些权重一起就是系统的知识。它们只是不太声明式，所以很难真正理解它们在做什么。

So again, anytime I'm talking there are many technologies, many models, and many companies, and anything I say is going to be a bit of a caricature, but a pretty good core way that manifests is: you literally take your model and its weights, and it produces various outputs with probabilities associated with them. When you train these models, you're basically saying, 'Hey, take this output and increase its probability.' How does it do that? It wiggles this parameter, it wiggles that parameter. Exactly which parameters moved in which directions cause that output to go up in probability—that's what backpropagation does. There's basically a bunch of calculus that tells you if I wiggle this, what happens over there. So when you say, 'Raise the probability of this thing and lower the probability of that thing,' the math in machine learning takes over. Mechanically, this comes into the system when you say, 'I prefer that's a great answer to sorry, you're confused.' You present both to the model, you score them, and then you raise one and lower the other. A lot of machine learning boils down to this: here is an answer, here is another answer, you scored them this way, adjust your scores. If you need to raise the score of something, you look at whatever weights led to it and raise them. If you need to lower a score, you look at the appropriate weights and lower them. This is the core mechanism not just of large LLMs or Transformers, not even just single neurons in a neural net. This is more or less how all machine learning proceeds. If you go back to the earliest representations of weighted functions, like the McCulloch-Pitts Perceptron from decades ago, you would point at something and say, 'Is this a positive or a negative example?' The system would say, 'Negative.' You'd say, 'Nope, that's not right. Positive.' What would you do to make it positive? You make sure that if it ever classified this again, it would say positive. How? You change the numbers a little bit in the direction that increases the score of this one. You do this repeatedly, and the weights come to embody decision boundaries. We look at that and say, 'Ultimately, if you get enough correlations, we call that knowledge.' There's a whole philosophy on what makes something knowledge, but in common usage, all those weights together are the knowledge of the system. They're just not very declarative, so it can be hard to really understand what they're doing.

那么，责任的重担最终是否又落回到我们人类身上？比如，如果我们刚才做的是基于人类反馈的强化学习（RLHF），并且它基于我们喜欢的回答而不是不喜欢的，理论上，如果我们只想要一个对照组，并且整个论点总是追求真相而不是你喜欢的回答，那么我们难道不是更有机会微调出一个追求真相而非任何偏见的模型吗？

So, does the onus of responsibility actually end up coming back on us as humans? Like, if we're doing this RLHF just now and it's based on a response that we like rather than not like, theoretically, if we just want a control group and the entire thesis is always aim for the truth rather than a response that you like, do we not then have a better chance of fine-tuning a model that pursues truth rather than any bias?

是的，我的意思是，先抛开一个问题：在我看来，非常诚实的模型也需要架构上的改变，对吧？我们之前讨论过元认知。你知道不回答“伯克利的人口是多少”的一个很大原因是你意识到自己知道什么和不知道什么。这种信息的存在或缺失，至少不是显式计算的。所以这是必须攻克的一个难题。如果你去问人们想要什么，他们是人类，对吧？这取决于你如何措辞问题。有些人会说他们想要那个东西，但实际上他们想要另一个。你可以说责任在人类身上。然而，我确实认为提供这些技术、提供模型的公司有道德责任，不要腐蚀数字素养。这意味着什么？首先，我们实际上不知道如何应对那些说谎时没有迹象的系统。我们真的不知道答案。我认为答案部分在于构建不说谎的系统。但如果系统要向你提供信息，它们可以引用来源。这是一点。它们可以逐条声明地追溯来源，说，“我在这里得到这个，我如何知道这个。”但这实际上很难，有两个原因。第一，有很多方式出错，并不归结为未能引用来源，对吧？可能有非常微妙的混淆，信息不仅仅是一堆事实。但更现实的是，系统通常引用来源的方式要么是检索增强系统（RAG 系统），它抓取了一些来源并说，“我用了那些。”但无法保证信息在你生成答案时被保留。它只是意味着，“我在生成你的答案时有这些信息可用。”并不意味着你保留了信息或没有改变它。另一件事是，你可以事后尝试说，“好吧，让我去找一个支持这个的理由。”那也有问题。这些系统本质上不是以这种模式运作的：我有一个问题，让我去找一些来源，让我审查这些来源。好的，我审查了来源。我将以保证事实性的方式获取信息，并附带审计追踪呈现给用户。这根本不是这些系统的工作方式。我认为这实际上是这些系统应该工作的方式。我认为系统应该将信息流作为一等对象。信息从哪里来？到哪里去？有什么证明没有出错？我认为这种架构能产生在更深层次上真正诚实的系统。这基本上就是我们在 Scale Cognition 所做的。我认为你会越来越多地看到系统更努力地引用来源，但这是一个非常棘手的问题：把本质上分散在大量权重中的信息综合起来，那里真的没有来源。它把信息混在一起，合成新的 token 序列，然后事后试图说，“好吧，我能引用哪个网页来支持这个？”这有点混乱，而且结果并不可靠。

Yeah, I mean, setting aside the issue that in my opinion, models that are going to be very truthful are also going to need architectural changes, right? We talked earlier about metacognition. A big part of how you knew not to answer 'What is the population of Berkeley?' is you were aware of what you do and don't know. That presence or absence of information is not something, at least, that's explicitly computed. So that's one of the nuts that is going to have to be cracked. If you go to people and ask them what they want, these are humans, right? It's going to depend on how you phrase the question. Some people are going to say they want that one thing, but they actually want the other. You could say the responsibility's on humans. However, I do think that companies providing these technologies, providing models, do have ethical responsibilities to not be kind of corrosive to digital literacy. What does that mean? Well, first of all, we don't actually know the answer to what do you do about systems that have no tells when they lie. Like, we don't actually know the answer. I think the answer is, in part, build systems that don't lie. But if systems are going to give information to you, well, they could cite their sources. That's one. They could go to source kind of claim-by-claim and say, 'Here's where I got this. Here's how I know this.' But this turns out to actually be hard for two reasons. One, there are a lot of ways to be incorrect that do not boil down to failure to cite a source, right? There could be very subtle confusions, and information is not just a list of facts. But even more presently than that, the way systems generally cite their sources is either it was a retrieval-augmented system, a RAG system, that grabbed some sources and said, 'Well, I used those.' But there's no guarantee that the information was preserved as your answer was generated. It just meant, 'I had this available to attend to when I generated your answer.' It doesn't mean that you preserved the information or haven't changed it. The other thing is that you can post hoc try to say, 'Okay, let me go find a justification for this.' That's problematic, too. These systems are not inherently operating in the mode of: I have a question. Let me go find some sources. Let me vet those sources. Okay, I vetted the source. I'm going to take the information in a way that's guaranteed to preserve its factuality, present it to the user with an audit trace. This is just not how these systems work. I think this is actually how these systems should work. I think systems should have information flow as a first order object. Where did the information come from? Where is it going? What is the certificate that nothing went wrong? This is the architecture that I think gives rise to systems that can actually be truthful in a deeper way. This is sort of what we do at Scale Cognition. And I think increasingly you're going to see systems out there that do work harder to cite their sources, but it's just a very thorny problem to take something that is fundamentally synthesizing information that's scattered across a bunch of weights, where there really isn't a source. It's mushing that information together, synthesizing new token sequences, and then afterwards trying to be like, 'All right, well, what web page can I cite in support of this?' That's kind of messy, and it turns out to be not reliable.

如果你去搜索概览，点击某个做这种事情的系统的引用来源，你常常会发现答案并不在那里，或者在那里，但内容不同。我在想，从那些实际开发这些模型的前沿实验室的角度来看，对吧？我们在谈论这种对真理的追求。在我看来，他们所要做的就是让这些系统比当前的人类更可靠，对吧？不需要达到绝对真理。它只需要比市场部的 Karen 更可靠，对吧？显然她会犯错。显然她可能有自己的目的。显然她是从不同地方获取信息。其中一些可能过时了。其中一些可能被曲解了，或者计算可能出错了。他们只需要构建一个不是绝对正确，但比当前情况更好的系统。

If you go to a search overview and click on one of the cited sources for systems that do that, you will often find the answer is not there, or it's there, but it's different. I'm thinking about this in terms of the Frontier Labs actually developing these models, right? We're talking about this pursuit of truth. In my head, all they have to do is make these systems more reliable than humans currently are, right? Doesn't need to reach absolute truth. It just needs to be more reliable than Karen in marketing, right? Like obviously she gets things wrong. Obviously she may have an agenda. Obviously things like she's pulling information from different places. Some of that might be outdated. Some of them might be played wrong or calculations might be wrong. They just need to build a system that's not absolutely true, but better than what currently is.

是的，我认为这是一个非常好的观点。所以，你知道，智能很重要，显然它很强大。没有可靠性的智能是有限的。在 Scale Cognition，我们专注于超级可靠性。你刚才说的本质上就是这个意思。你需要系统更可靠。现在，我认为有很多情况，我们知道有很多情况，可靠性实际上是可以证明的。就像那是极限，那是上限。它通过合同保证 100%，因为它的结构方式，它实际上不可能改变这些信息。就像编译器可以保证，你知道，它不能保证你的代码没有 bug，但它可以保证它正在执行那段代码。这一点是，我们真的至少想超越人类，对吧？我认为这是我们至少应该努力做到的。我再说一遍，智能是多方面的，在知识广度方面，有些方面已经轻松地远远超过了人类。我们可以讨论为什么进展这么快，但当然，在 LLM 中，你去 ChatGPT，它肯定比我们任何人都了解更多的主题。可靠性落后了。所以，为了让可靠性赶上，我们需要新技术，对吧？为什么？因为如果你继续沿着当前的缩放定律做下去，你会继续在你擅长的事情上变得更好，而在你落后的事情上继续落后。并不是说可靠性不能提高。而是人们一直在做的事情在那里并不是特别有效。有一种想法是，如果你只是吸收越来越多的数据，你最终会学到一切。但样本复杂性是真实存在的。有些东西学得比其他东西快。所以，我喜欢做一个思想实验，假设你想学习法语。我现在住在蒙特利尔，所以这是一个非常相关的考虑。

Yeah, I think it's a really great point. So, you know, intelligence is important, obviously it's powerful. Intelligence without reliability is limited. At Scale Cognition we're focused on super reliability. And what you're talking about is essentially that. It's that you need systems to be more reliable. Now, I think there are many cases and we know there are many cases where the reliability can actually be provable. Like that's the limit. That's the upper bound. It's 100% guaranteed contractually because of how it's structured it can't actually have changed this information. In the same way that say a compiler can guarantee, you know, it can't guarantee your code doesn't have bugs, but it can guarantee that it's executing that code. This point that well, we really want to get above humans at least, right? I think that's the least we should be trying to do. I would say again, intelligence is multifaceted and some aspects in terms of breadth of knowledge have blown way past humans really easily. We can talk about why that went so fast, but certainly in an LLM, you go to ChatGPT, it certainly knows about more topics than any of us do. Reliability has lagged behind. So, the thing is in order to make the reliability catch up, we're going to need new techniques, right? Why? Because if you just keep doing the things that are on the scaling law you're on right now, you're going to keep getting good at what you're good at and you're going to keep lagging behind on what you're lagging behind on. It's not that reliability can't go up. It's that the things people have been doing are not particularly effective there. There's this idea that if you just take in more and more data, you're eventually just going to learn everything. But sample complexity is real. The fact that some things get learned faster than others. So, I like a thought experiment I like to give is let's say you wanted to learn just be French. I live in Montreal at the moment, so this is very pertinent consideration.

太好了。试着学法语。假设你不会说法语，你想学法语。你的方法就是读英文小说，一本接一本地读。偶尔有人说法语。要么有翻译，要么从上下文能明白。所以你边读边学一点点法语。当然，你需要读很多书才能有进步。但想法是，哦，继续读，继续读，继续读，你就会学会。答案可能是，在某种无限极限下，但这不是一种有效的方式。对吧？大多数书与此无关。当出现一些法语片段时，你会遇到很多像“bonjour”这样的词。你会遇到很多你已经知道的小常识。你慢慢掌握整个语言所需的时间将是天文数字。我觉得当人们说“我们只要继续扩大我们正在做的事情，一切都会改善”时，有点这种味道。显然，有些事情改善得快，有些事情改善得非常慢，而可靠性一直改善缓慢，考虑到架构，这并不奇怪。所以，我确实认为重要的是，当你扩大规模时，事情全面变好，但同样真实的是，有些事情一直落后。而且，我们扩大规模的能力也是有限的，对吧？感觉系统已经……抱歉，请继续。

Excellent. Try to learn French at the moment. Let's say you don't speak French, you want to learn French. And the way you're going to do this is just read novels written in English, book after book after book after book. And every now and then somebody says something in French. And it's either translated or clear from context. And so you learn little snippets of French as you go. And of course, you have to read a lot of books to get very far. But the idea is, oh, just keep reading, keep reading, keep reading, and you'll get there. And the answer is like maybe in some infinite limit, but this is not an efficient way. Right? Most books are not related to this. When some snippet of French shows up, you're going to get a lot of like bonjour. You're going to get a lot of little common things you already know. And the time it will take you to slowly gather the whole language, it's just going to be astronomical. And I feel like there's a little bit of this vibe when people say, "Well, we're just going to keep scaling up what we're already doing, and everything will improve." Well, clearly some things improve fast, and some things improve very slowly, and reliability has been improving slowly, and it's not surprising given the architecture. So, I do think it's important to like it's both true that when you scale things up, things get better across the board, but it's also true that some things have consistently lagged behind. And for that matter, we're also limited in our ability to scale up, right? Like It feels like systems went Sorry, go ahead.

不，你刚才说的是技术和规模之间的区别，对吧？长期以来我们一直在谈论缩放定律、更多的数据中心、更多的电力、更多的能源。我想问你，也是问观众，最近一次改进这些模型的重大技术转变或技术变革是什么？

No, what you're talking about is difference between technique and scale, right? And so long we've been talking about scaling laws and more data centers, more power, more energy. The question for you, just for the audience, what was the last kind of great technological shift or technique shift that improved these models?

呃，这个问题太好了。让我试着用一个故事来回答，稍微退一步。研究有超级周期，它们基本上看起来都一样。大致是这样的。研究人员、技术人员在构建系统。它们的性能有极限。我们想让事情变得更好。我们想让系统更准确。我们想让系统更智能，然后我们遇到了某种瓶颈。我们有系统，它们很复杂，人们很难取得进展。某个新想法出现了，它是变革性的。这种新技术，即使是在简单的、未优化的形态下，也超越了排行榜和基准测试，尽管更简单，却胜过所有这些系统，每个人都说：“好吧，让我们抛弃所有那些东西。让我们做这个新东西。”然后它们起作用了，他们优化了这个新东西，他们扩大了它的规模，它变得越来越好。然后它开始放缓，对吧？一开始你以为你在指数曲线上。结果你发现你是在 S 曲线的起点，事情变慢并趋于平稳。然后你再次遇到瓶颈。就像你用这种技术无法再进一步，所以你开始问：“差距在哪里？”然后你开始看到那些差距通常是一直存在的，而且通常有已知的解决方案。通常那些解决方案实际上是在过去发明的，但可能因为技术变化而不适用，所以需要重新发明，但随后你进入了一个阶段，用新的架构、新的技术来增强这个单一的想法，使其变得更好，然后也许某个新的创新出现，再次颠覆一切。我在职业生涯中见过好几次这种情况。

Uh such a great question. Let me try to answer with a story to kind of step back. So, research has super cycles, and they basically always look the same. It looks something like this. Researchers, technologists are building systems. They have a limit to how well they perform. We want to make things better. We want to make systems more accurate. We want to make systems smarter, and we've sort of hit a kind of wall. We've got systems, they're complicated, and people are having trouble making progress. Some new idea comes onto the board, and it's transformative. This new technique even in its sort of simple incarnation, unoptimized, is blowing past leaderboards and benchmarks, and outperforming all these systems despite being simpler, and everybody says, "All right, well, let's throw all that stuff out. Let's do this new thing." And they work, and they kind of optimize the new thing, and they scale up the new thing, and it's just getting better and better and better. And then it starts to slow down, right? At the beginning you think you're on an exponential curve. And it turns out you're at the beginning of an S curve and things slow down and they level off. And then you start to hit walls again. Like you just can't get any further with this technique and so you start saying, "Where are the gaps?" And then you start to see that those gaps were generally things that have been there all along and often they have known solutions. Often those solutions were actually invented in the past and maybe don't apply because the technology has changed so they need to be reinvented, but you then go into this phase of augmenting this singular idea with new architectures, with new technologies that make it better and then maybe some new innovation comes along and wipes the board again. And I've seen this several times in my career.

我是在 80 年代开始做自然语言处理的，当时人们从语言学家编写陈述性规则转向基于语料库的方法，这个方法我们沿用至今。那是一个巨大的进步。转向统计模型是巨大的进步。在分类任务中，支持向量机超越了所有方法，取代了朴素贝叶斯。巨大的进步。神经模型，巨大的进步。大规模 Transformer，巨大的进步。每一次都推动事情向前发展，但最终都会饱和。然后基于 Transformer 的大语言模型被喂入越来越多的网络数据，变得越来越好。每个人都说：“把其他东西都扔掉，只管 Scaling 这个。”一开始效果很好，但漏洞开始显现。例如，数据用完了。为什么 LLM 这么聪明？它只训练了大约一个月，数据来自网络，而网络是人类知识 30 年的积累，代表了上千年的知识。这是一次性事件。没有一百个其他网络拥有更高级的知识。所以你撞墙了。

So I started natural language processing in the '80s, when people switched from linguists writing declarative statements about language to corpus-based methods, which we still do today. That was a big gain. The move to statistical models was a big gain. In classification, support vector machines outperformed everything, replacing naive Bayes. Big gain. Neural models, big gain. Transformers at large scale, big gains. Each of these jumped things forward but eventually saturated. Then large language models based on Transformers were fed more and more of the web and just got better. Everyone said, 'Let's throw out everything else and just scale this up.' At first it worked well, but gaps began to show. For example, you run out of data. Why is an LLM so smart? It trained for about a month on the web, which is a collection of human knowledge built over 30 years of web pages, representing millennia of knowledge. That's a one-time event. There's no hundred other webs with more advanced knowledge. So you hit a wall.

那该怎么办？现在你看到了其他想法：推理、不同类型的强化学习。人类做出好决策有两种方式。一种是通过记忆：我碰过一次炉子，不会再碰了。语言让你从别人的错误中学习。那是基于记忆的决策。另一种是通过预测：你想，“如果我这样做，就会发生那样的事，也许我不该做。”你像国际象棋引擎一样向前推演。80 年代到 2010 年的 AI 专注于预测，搜索替代假设，尝试各种方法直到成功。LLM 擅长一次性生成答案，所以人们停止了搜索。但一旦撞墙，人们就说：“对于数学问题，也许我们应该尝试一堆方法，看看哪个能得出正确证明。”那就是推理，利用测试时算力让模型更聪明。它本质上就是搜索。这些智能的核心组件不断以新形式回归。但推理并非完美：测试时算力很昂贵，消耗 token。如果你尝试 10 个方法并选一个，你要花 10 倍的 token。

So what do you do? Now you see other ideas: reasoning, different kinds of reinforcement learning. In humans, there are two ways to make good decisions. One is by memory: I touched the stove once and won't do it again. Language lets you learn from others' mistakes. That's decisions from memory. Another way is by prediction: you think, 'If I do this, then that will happen, so maybe I shouldn't.' You play the game forward like a chess engine. AI from the '80s to 2010 focused on prediction, searching alternative hypotheses, trying things until they work. LLMs are good at generating a single-shot answer, so people stopped doing search. But as soon as it hits a wall, people say, 'For math problems, maybe we should try a bunch of things and see which result in correct proofs.' That's reasoning, using test-time compute to make the model smarter. It's basically search. These core components of intelligence keep coming back in new guises. But reasoning isn't all ideal: test-time compute is expensive, burning tokens. If you try 10 things and pick one, you spend 10 times as many tokens.

那么前沿实验室在数据中心、能源、GPU 上花费数十亿甚至数万亿美元。但一年半前，DeepSeek 推出了一个成本极低的模型，因为新技术让它更快更高效。这样的进步随时可能出现。如果我们知道这是可能的，为什么这些实验室现在还要在 Scaling 上花这么多钱，如果新技术可能把它抹掉？

So we have frontier labs spending billions or trillions on data centers, energy, GPUs. But a year and a half ago, DeepSeek came out with a model at a fraction of the cost because of new technology that made it faster and more efficient. Advances like that could come all the time. If we know this is possible, why are these labs spending so much on scaling right now, if it could be wiped out by new technology?

这是颠覆性技术与规模化技术之间永恒的故事。颠覆性技术往往是初创公司的领域，而大型、资本雄厚的公司总是特别擅长在已经奏效的事情上加倍下注。

This is the perennial story of disruptive technology, which tends to be the domain of startups, versus scaled technology. A large, highly capitalized company is always particularly well suited to doubling down on things that are already working.

Scaling 成功了，所以他们继续扩大规模。这就是为什么我认为传统上你会看到很多来自初创公司的颠覆，因为初创公司不适合在已经在大规模上奏效的事情上加倍下注。他们适合提出颠覆性的新想法。而且我确实认为，在具有不同属性的系统方面，将会出现很多颠覆。我们专注于具有非常不同可靠性特征的系统，更高效的系统。如果你还记得，几年前，因为 Scaling 取得了成功，在 1% 的网页、10% 的网页和 100% 的网页上训练之间存在巨大差异。参数不足和参数充足之间也存在巨大差异，随着你成长，你需要足够的参数来捕捉你的数据，因为那本质上就是数据存储的地方。而且你能抽象这些东西的程度是有限的。但因为 Scaling 如此成功，有一个阶段几乎就是越大本身越好。如果你有两个模型，一个更大，你会想，“哦，那它一定更好。” 但事实并非如此，对吧？它可能更大，但同时也更差。在其他条件相同的情况下，拥有更多的算力、更多的数据、更多的参数，这些事情非常有帮助。但重要的是要记住，规模本身并不是目标。它是在固定架构下向前推进的一种方式。首先，我认为每个人都承认我们看到了收益递减。简而言之，就是没有更多的千年数据可以训练了，对吧？你现在必须开始做其他事情了。所以这是人们认为会是指数曲线，但实际上是 S 曲线并且趋于平稳的事情。所以我认为这是 Scaling 的一个问题。而且这被 Scale 实际上很重要这一事实所混淆。比如智能，很多智能的组成部分在最小的规模下无法工作。你必须拥有足够的规模。所以这一切都变得复杂了，但你肯定看到现在人们在谈论，好吧，有没有一种方法可以在没有所有测试时算力的情况下做到这一点？就像一个推理模型，为了生成一个简短的响应，它会在给出响应之前自言自语数千个 token。很明显，如果你按 token 付费，如果你在出售 token，这是一个好主意，对吧？你现在可以卖出更多的 token。但它增加了延迟，增加了成本。如果你能在更小、更快的模型中获得相同的行为，人们会更喜欢那样。而你现在看到以前没有看到的是，人们开始说，好吧，什么模型能快速完成我需要的事情？什么模型能在我们可以提供的技术上完成我需要的事情？因为电力是有限的，GPU 是有限的。所以我认为人们已经摆脱了“越大越好”的观念，即使成本更高、耗时 10 倍也不更好。你会想，实际上，这些都不好。唯一重要的是输出的质量，然后其他一切，更快更好，更便宜更好。

Scale has worked, so they keep scaling. This is why I think traditionally you do see a lot of disruption from startups because startups are not suited to doubling down on things that are already working at the larger scales. They're suited towards coming up with new ideas that are disruptive. And I do think there is going to be a lot of disruption in terms of systems that have different properties. We're focused on systems that have very different reliability profiles, systems that are more efficient. If you remember, a few years back because there had been success from scaling up and there is a big difference between training on 1% of the web and 10% of the web and 100%. There's a big difference between having not enough parameters to enough parameters, as you grow, you need enough parameters to capture your data because that's essentially where the data is getting stored. And there's a limit to how much you can abstract that stuff. But because scaling was so successful, there was a phase where it was almost like bigger was better in and of itself. If you had two models and one was bigger, you're like, "Oh, well, it must be better." Well, it doesn't have to be better, right? It could be bigger and also worse. All else equal, having more compute, more data, more parameters, these things are very helpful. But it's important to remember that scale is not a goal in and of itself. It is a way to push forward under a fixed architecture. And first of all, I think everybody's acknowledged we're seeing diminishing returns. Which boils down really in a nutshell to there's no more millennia of data to train on, right? You've just got to start doing other things now. So that's something that people thought was going to be an exponential curve, but it was an S curve and it leveled off. So I think that's one issue with scale. And this is confounded by scale is actually important. Like intelligence, a lot of pieces of intelligence don't work at the smallest scales. You have to have enough scale. And so that's all made it complicated, but you definitely see people talking about now things like, okay, is there a way to do this without all the test time compute? Like a reasoning model that for a short response generated, you know, thousands upon thousands of tokens talking to itself before it gives that response. Well, it's clear that if you're paying by token, if you're selling tokens, this is a great idea, right? You can sell so many more tokens now. But it increases latency, it increases cost. And if you can get the same behavior in a smaller model, in a faster model, people are going to prefer that. And what you're seeing today that you didn't before was people starting to say, okay, what is a model that can do what I need fast? What is a model that can do what I need on technology that we can provision? Because there's a finite amount of power, there's a finite amount of GPUs. And so I think people have gotten away from bigger is better even if it costs more is no better and takes 10 times as long. You're like, actually, none of that is good. All that matters is the quality of the output, and then everything else, faster is better, cheaper is better.

是的。

Yeah.

而且更易部署更好。所以，我确实认为你会看到更多专业模型。我确实认为你会看到更多模型将其优势集中在需要的地方，而不是像，一个经典的例子是，我不知道你是否看过 Chipotle 的例子，那是一个点餐机器人，有人问，如何在 Python 中反转链表之类的？当然，模型可以回答。但它不应该。为什么我们要拿一个能做所有这些事情的模型，然后努力确保它不做那些事情？这里有些东西不对劲。最终，我认为这关乎可靠的智能。你需要系统做什么？你需要系统永远不做什么？什么会有正确的性能特征？你不希望用户等待 6 秒才能得到答案。你不希望系统跑偏到奇怪的方向，你也不希望不断与一个没有良好控制杠杆的系统搏斗。

And more deployable is better. So, I do think you're going to see more specialist models. I do think you're going to see more models that have focused their strengths where they need to be as opposed to, like, a classic example of this is I don't know if you saw the Chipotle example where it's like an ordering bot and somebody's like, how do you reverse a linked list in Python or something like that? And of course, the model can answer. But like, it shouldn't. And why would we take a model that can do all of these things and then work really hard to make sure it doesn't do those things? There's just some things that are off here. And ultimately, I think it's about reliable intelligence. And what do you need the system to do? What do you need the system to never do? And what is going to have the right performance profiles? You don't want users waiting 6 seconds for an answer. You don't want the system going off on weird tangents, and you don't want to be constantly wrestling a system that doesn't have any good control levers.

我想到的是路由。当你谈到这个时，不是拥有一个能做所有事情的广泛模型。我想了很多关于智能体框架的例子。我们现在看到很多人使用不同的框架，他们发送所有这些超特定的智能体或模型来执行非常特定的任务，然后将结果带回一个空间。这就是你想象的未来吗？我们有超特定的模型，它们能正确完成超特定的任务，然后我们有某种框架来编排这一切。

My mind goes to routing. When you're talking about this, instead of having one large breadth of a model that does everything. I think a lot about agent harnesses, for example. We're seeing a lot of people kind of use different ones just now, and they're sending all these hyper-specific agents or models to do very specific tasks and then bringing that back to one space. Is that kind of what you imagine for the future? We have hyper specific models that do hyper specific tasks that get that right and then we have some sort of harness that orchestrates all this.

是的，我认为有两个极端，但两个极端都不太奏效。如果你走向极端，模型完全广泛，即使你有一个如此庞大、广泛、在各方面都超级智能的模型，以至于可靠性终于赶上了人类，这对于可靠性是最重要的任务来说，将是一种极其浪费的方式。你希望架构的可靠性不会滞后。但一方面，你可以有那种庞大且不可控的模型。那不好。另一方面，你可以有那种缩小到非常小以适合在框架中做出控制决策的模型。它们现在如此超特定，以至于失去了魔力。我认为与足够广泛的智能互动带来的魔力在于，是的，你在询问你的账户余额或更改航班之类的事情，但你在世界知识的背景下这样做，你有你的理由，你希望系统能够意识到，哦，你在问这个，但你实际上指的是那个，你不能被困在流程图中，但也不能处于一个完全不可控、即兴发挥的系统中。你希望中间的系统能够出色地、智能且可靠地完成你需要它们做的事情。所以我认为你会看到不是超特定于决策树中一个决策的模型。

Yeah, I think there are two extremes which neither extreme really works. So if you go to the extreme where the model is completely broad and even if you had a model which was so big and so broad and so super intelligent in every way that the reliability was finally catching up to people, this would be an incredibly wasteful way to do a task where reliability is the most important thing. You want to have architectures where the reliability is not lagging behind. But on one side you can have models which are sort of large and uncontrollable. That's not good. On the other side you can have models that are again shrunken down so small to fit in like making control decisions in a harness. They're now so hyper specific that they're losing the magic. I think the magic that comes from interacting with a sufficiently broad intelligence is that yes you're asking about your account balance or changing a flight or something like that, but you're doing it in a context of world knowledge and you have your reasons and you want the system to be able to realize that oh you're asking about this but you really mean this and you can't be stuck in a flow chart but you also can't be in a system that's just totally uncontrollable and winging it. You want systems in the middle which are able to do a really good intelligent and reliable job of what you need them to do. And so I think you're going to see models which are not like hyper specialized to one decision in a decision tree.

我认为你会看到模型专门针对垂直对话等领域。你认为人类有什么特殊之处是我们永远无法复制的吗？我认为你谈到的对话中很大一部分是元认知和上下文，对吧？例如，当我做决定时，我大脑中有大量流动的上下文，我能够进行元认知或思考它并应用于情境。那是基于逻辑的，还是人类独有的？

I think what you're going to see models specialized to, you know, kinds of vertical conversations, things like that. Do you think there's anything special about humans that we're never going to be able to replicate here? I think a big part of the conversation that you're speaking about are two things that you speak about as meta-cognition and context, right? So, for example, when I'm making a decision, I have so much fluid context in my brain and I'm able to meta-cognate or maybe think about that and apply it to the situation. Is that logic-based or is that something that's uniquely human?

嗯，这取决于我们往哪个方向走，要么是神经科学问题：人类是如何做到的？比如在大脑中，一切是如何布局和运作的？神经科学已经取得了一些巨大进展，但也有大量未知。另一个方向更哲学化：人类是否有某种特质？是否有深层原因使我们无法构建一个模仿我们行为（包括元认知行为等）的系统？哲学家们对此争论不休。我认为关键是要指出，即使在功能主义观点下，人类的所有计算最终都是机械计算，原则上可以复制，但这与说“是的，当然，我们做的任何事都能被人工智能完成”是不同的。这与说“我们今天的技术稍微扩展一下就能做到”也不同。所以，我不认为“有些事机器永远做不到”是我的观点。我的观点更像是：人工智能在哪些方面持续落后，要赶上人类需要架构、技术或方法的更大改变，而不仅仅是继续做我们正在做的事。我们已经反复谈到一个方面：可靠性。元认知和心智理论，即我们不仅不断追踪自己知道什么，而且在与人交谈时，追踪对方知道什么、理解什么。我们追踪人们的目标。对话总是有社会层面，我们通常为了某个目的而交谈。而这些系统还做不到。它们没有元认知，没有丰富的心智理论。

Um, so depending on where we go with this, we're either getting to a neuroscience question. How do humans do this? Like mechanistically in our brains, how is everything laid out and how does it operate? And there've been some great advances in neuroscience and like there's also just a ton we don't know. The other way we could go to this is more philosophical. Like, is there something about humans? Are there like deep reasons why we could not build a system that mimics our behaviors, including the meta-cognitive behaviors and so on. And you know, philosophers argue about this. I think the important thing to point out is, even under a functionalist view where every computation that happens in humans is ultimately a mechanistic computation and can be replicated in principle, that is different. Saying that yes, of course, anything we do can be done by an artificial mind. That is different from saying the technology we have today, scaled up a little bit, will do it. So, I don't think in terms of, okay, here's a thing that a machine can never do. I feel like that's just not my outlook. My outlook is more like here are the places where artificial intelligence has lagged behind persistently and where to catch up with humans would really require more change in architecture or technology or approach than just keeping doing what we're doing. We already talked about one repeatedly which is reliability. Metacognition in general theory of mind, the fact that we are constantly tracking not just what we know, but when we're talking to somebody, what do they know? What do they understand? We're tracking people's goals. We are constantly there's a social level of conversation. We're usually using conversation for some purpose. And so these systems are not there yet. They are not metacognitive. They do not have a rich theory of mind and of other people.

而且它们无法在乎，对吧？你刚才提到的，人类的独特优势之一是感受同理心的能力以及它如何影响决策。如果它们无法在乎，如果纯粹是机械的，那现在这肯定是一个关键组成部分。

And they can't care, right? Something that's coming up in what you're just saying is that one of the unique advantages of human is the ability to feel empathy and how that influences decisions. And if they can't care, if it is purely mechanistic, then that's got to be a critical component now.

是的，我认为归根结底，我不会冒险说人工智能系统永远无法在乎。我看不出有什么理由不能。这与说我们今天的系统没有是截然不同的。

Yeah, I think it boils down to I would not go out on a limb to say something like an artificial system can never care. Like I don't see any reason it couldn't. That's a very different statement than the systems we have today.

是的。

Yeah.

没有真实的情绪，因为我认为与当今系统互动的一个难点是，我们谈到过人类犯错时，有时会意识到自己知识的不足，但人类犯错时通常会有一些迹象表明可能错了。而系统可以表现得自信、权威，却可能是错的。它们也可以说出从人类角度会出自同理心的话，却没有同理心。因此，我认为在看待这些时，非常重要的一点是，不能因为系统输出了 X，就推断人类输出 X 时的含义。如果人类这么说，可能出于完全不同的原因。这就涉及机制了。我发现这很有趣，作为工程师，你基本上是用简单部件构建复杂事物，这是综合活动。科学家则采用分析方法，把复杂事物分解来理解。我们开始问：人类如何拥有同理心？意识如何产生？然后看系统，问它有没有同理心？这很有趣，因为这是科学。这不是“同理心是什么，我们如何构建它”，那是工程问题。而是面对一个人造物，我们其实并不真正理解它，因为它有非常类人的行为，但我们知道这些行为在很多情况下源于非常不类人的原因。试图追根究底。所以，我们对这些模型进行神经科学研究，说明其中存在我们不理解的复杂性。因此很难断言 X 绝对没有发生。但很明显，系统可以在没有人类全部上下文和机制的情况下，闪现类人行为。

Do not have authentic emotions because I think one thing that's very hard about interacting with systems today, we talked a little bit about how when humans make mistakes, one, they're sometimes aware of their lack of knowledge, but when humans make mistakes, there's often these smells, these indications that like maybe this is incorrect. And systems can act confident, they can act authoritative, they can be wrong. They can also say things that from a human would have come from a place of empathy without having a place of empathy. And so, I think as we look at these things, it becomes very important to not say that if the system outputs X, I can conclude something from humans outputting X, right? If a human says this, it might be for a totally different reason. And now we get into the mechanism. And I actually find this very interesting as, you know, as an engineer, you go and you basically build complicated things out of simple pieces. That's what engineers do. It's a synthesis activity. A scientist takes an analytical approach. A scientist takes something complicated that they don't understand and they try to understand it by breaking it apart. And we start asking questions like, all right, well, how do humans have empathy? How do humans have consciousness or whatever? And then you go to a system, you're like, well, does it have empathy? This is interesting because this is science. This is not here's what empathy is, how do we build it? That would be an engineering question. This is like, here's an artifact that we actually don't really understand because it has some very human-like behaviors, but we know that those behaviors are arising for, in many cases very not human-like reasons. And trying to get to the bottom of that. So, the fact that we're doing neuroscience on these models tells us that there is complexity in there we don't understand. So, it's very hard to say definitively X isn't happening. But, it does seem very clear that systems can have flashes of human-like behavior without having all of the context and mechanisms that give rise to that behavior in humans.

非常有趣。我很好奇你个人层面。你身兼多职：CTO、研究员、教授。这需要很多精力，我想你有一个独特的优势，就是通过多个视角看待 AI 和 LLM：一是教学，二是研究最前沿，三是构建商业上可行的产品。你认为这些如何融合成你的观点？有没有哪个占主导？你能像……抱歉说得太直接，但像模型推理那样，比如 Perplexity 的模型委员会？你可以看到三个不同的人。它们不同吗？

Super interesting. I'm curious about you on an individual level. A lot of different hats, CTO, researcher, professor. Like that's a lot to focus on and I guess you got a pretty unique advantage in that you're looking at AI and LLMs through like a couple different lenses. One in terms of teaching it, one in terms of researching the absolute frontier of it, and then three, building something that's commercially viable to sell people. How do you think they all fuse together into the viewpoint that you have, and does any dominate more than the other? Are you able to kind of like look at that as a... Sorry to be on nose, but like a model reasoning, like what are the... What is it? A council of models that they have in Perplexity? You could see the three different people. Are they different...

我的大脑是专家混合模型吗？是的，你基本上是在问我脑子里是不是专家混合。我不知道，也许吧。但我确实认为这些不同的视角并不像你想象的那么不同。

Is my brain a mixture of experts? Yeah, so you're basically asking if I am a mixture of experts in my head. And I don't know, maybe. I do think though that these different perspectives aren't as different as you might think.

所以，我认为将它们联系在一起的是，有一系列重要的概念，这个领域现在正在研究：什么是智能，如何构建智能系统，智能系统的各个方面是什么？一个系统要有效，需要满足哪些下游需求？可能是在企业中有效，对吧？这是一个非常产品化的问题。需求是什么？社会学上的有效性，数字素养，比如系统需要做什么才能不与人们的工作方式产生不良互动。还有关于这些事情的沟通方面。当我们构建系统时，我们需要能够清楚地告诉人们它们是什么。甚至像这样简单的事情：这是一个并不总是正确的系统。当它犯错时，你可能无法察觉。这是大多数人都不知道的事情。所以，对我来说，这些都只是同一个问题的不同视角，那就是我们如何构建可以信任、可以控制的 AI 技术？这不仅仅是一个技术问题，但它确实是一个技术问题。这不仅仅是一个商业问题，但它确实是一个商业问题。这也不仅仅是一个社会问题，但它确实是一个社会问题。我认为，如果你不同时看到所有这些部分，就会导致不正确的解决方案。而这个问题太重要了，不能推出半生不熟的错误解决方案。

So, I think the thing that ties them all together is there is a set of important concepts that the field is just now working through about what is intelligence, how do we build intelligent systems, what are the facets of an intelligent system? What are the downstream needs out of a system for it to be effective? That could be effective in the enterprise, right? That's a very product-based kind of question. What are the requirements? Effective sociologically, digital literacy, like what does a system need to do to not have some bad interaction with how people work. And also this aspect of communication about these things. When we're building systems, we need to be able to tell people clearly what they are. Like even just simple things like this is a system which is not always right. And when it makes mistakes, you may not be able to tell. This is something that most people don't know. So, to me, these are all just lenses on the same question, which is how do we build AI technologies which can be trusted, which can be controlled? This is not just a technological question, but it is a technological question. It's not just a commercial question, but it is a commercial question. And it's not just a social question, but it is a social question. And I think if you don't have all of these pieces in view at once, that's going to result in solutions that aren't the right ones. And this is too important a problem to be sending out half-baked wrong solutions.

我的最后一个问题是我会问每一位嘉宾的问题。我问他们同样的事情。每个人理解的方式都不同。有些人谈论家庭，有些人谈论目标。人们谈论一切。但你可以按自己的方式理解。我鼓励你花 5 到 10 秒思考一下再回答。Dan，你为什么做你现在做的事？

My last question is a question that I ask every single one of my guests. I ask them the same thing. Everyone takes it in a different way. Some people talk about family, some people talk about purpose. People talk about everything. But you take it however you want it. I encourage you to take 5 to 10 seconds to sit on it before answering. Dan, why do you do what you do?

作为一个从事 AI 工作的人来回答这个问题很有趣，因为在我的职业生涯中发生了太多变化。如果你 20 年前问我为什么从事 AI 和语言工作，我会说我喜欢语言，我确实喜欢。我喜欢思考智能的核心组成部分是什么，我小时候读了很多科幻小说，人们对未来的看法显然各不相同，但我真正共鸣的是那些充满希望的。我们可能拥有能够帮助我们取得更多成就的系统。AI 的有趣之处在于，那时——而且你知道，这并不久远——在我职业生涯早期，AI 最大的问题是东西不够好用。这是一个非常令人兴奋的挑战，因为我们可以从一个问题到另一个问题。比如，让我们构建一个机器翻译系统。我的意思是，它们在《星际迷航》里有，他们有通用翻译器。它们在《银河系漫游指南》里也有。看起来我们应该能够构建这个。那很令人兴奋，我认为现在仍然令人兴奋。那是消除人与人之间的障碍，让所有人都能看到所有信息的方式。这太棒了。我喜欢这些东西。我会看着所有这些问题，心想，我做这些是为了构建帮助人们沟通、获取信息、管理信息的技术。这首先是一个非常困难的问题，但如果能成功，就非常令人兴奋。我也受到科学问题的激励。比如我们重建古代语言，做各种事情，对我来说，这些都是难以解决的迷人问题。我确实认为我的答案最近发生了变化，因为以前 AI 的问题基本上源于它们不好用的方式。今天 AI 的问题在很多情况下源于它们成功工作的方式。对吧？你有一个系统如此接近信息神谕，以至于当它系统地欺骗人们时，你就有了问题。人们失去了知道该相信什么的能力，因为我们正在夺走那些帮助他们理解信息的东西。我们鼓励将工作委托给系统。对吧？如果你说，好吧，我以前写这些内容，但我现在让语言模型来做。很容易产生一种态度，认为你不再需要成为作家了。但问题是，现在你必须成为编辑。对吧？东西回来了，现在你必须编辑它。编辑很难，而且是一种不同的技能。在这种情况下，我们构建的技术不能仅仅构建技术本身。你必须给人们时间、脚手架和产品可供性，这样他们才能用它来提升自己，而不仅仅是生成一堆垃圾。所以，当我今天环顾四周，我看到 AI 将如何融入文化和社会存在很多不确定性。我今天所做的一切，是因为我仍然看到 AI 能做的所有伟大事情，这些让我开始了这段旅程。但我现在也看到了这项技术的某些方面，这些方面以前尚未显现，现在却来得非常快。所以我今天所做的，我这样做的原因，实际上是确保在 AI 系统部署时，它们是你可以控制的系统。它们是你可以信任的系统。它们是你可以推理其行为的系统，因为我认为 AI 系统所有令人兴奋的潜力都需要这一点。没有这一点，我不想最终陷入一种情况，即我们拥有腐蚀性的技术，没有人能信任，这真的会把我们引向一条我们不想走的道路。

So, it's interesting to answer this question as somebody working on AI because so much has changed during the course of my career. If you had asked me 20 years ago why am I working on AI, on language I would have said you know I love language, which I do. I love thinking about what are the core components of intelligence and I read a lot of science fiction as a kid and there's a range obviously of how people see the future but I really resonated with the hopeful ones. The idea that we could have systems that could help us achieve more. And what's interesting about AI is because at that time and you know, this isn't that long ago, right? At that time earlier in my career, the biggest problem we had in AI is that things didn't work well enough. And that was this really exciting challenge because we could go from problem to problem. Like let's build a machine translation system. I mean, they have them in Star Trek. They got the universal translator. They have them in Hitchhiker's Guide. It seems like we should be able to build this. That was exciting and I think it's still exciting. That's the way you can remove barriers between people and make all information visible to everyone. This is amazing. I love this stuff. And I would look at all of these problems and think, you know, I'm doing what I'm doing to build technologies that help people communicate, help people access information, help people manage information. And that just seemed like first of all, it's a really hard problem, but really exciting if you can succeed. I was motivated by scientific questions too. Like we were reconstructing ancient languages and doing all kinds of things that to me felt like these fascinating problems that had resisted solution. I do think something has changed in my answer recently because the problems in AI before stemmed from basically the ways in which they didn't work. The problems in AI today stem in many cases from the ways in which they do work successfully. Right? That you have a system which is so close to an information oracle that when it systematically lies to people, you have a problem. You have people losing the ability to know what to believe because we're taking away things from people that helped them understand that information. We're encouraging delegation of work to a system. Right? If you say, okay, I used to write this content, but I'm going to have an LM do it. It's easy to have this attitude that you don't have to be a writer anymore. But the thing is, now you have to be an editor. Right? The thing comes back and now you have to edit it. And editing is hard and it's a different skill. And this is a case where we're building technologies that you can't just build the technology. You have to give people time and scaffolding and product affordances so that they can use this to make themselves better and not just to generate a bunch of slop. And so when I look around today I see a lot of uncertainty in how AI will roll out into culture, into society and what I do today is because I still see all the great things that AI can do that got me started on this journey. But I now also see aspects of this technology that were not yet in view and now are coming very fast. And so what I do today, the reason I do it, is really to make sure that as AI systems get deployed that they be systems that you can control. They be systems that you can trust. They be systems that you can reason about their behavior because I think all of the exciting potential of AI systems requires that. And without that I don't want to end up in a situation where we have corrosive technologies that no one can trust that are really just sending us down a path that is not the one we wanted to be on.

回答得很美。没有评论，没有问题，没有疑问。Dan，我想感谢你的时间。我想这是我最近录制的播客中最长的一期。所以首先，我感谢你抽出时间。第二，如果人们想了解更多关于你是谁、你做什么，他们如何联系你？

That's a beautiful answer. No comments, no questions, no queries. Dan, I want to thank you for your time. I think this is the longest podcast I've recorded in a while. So first I want to appreciate you taking the time. Two, if people want to learn more about who you are, what you do, how do they go about getting in contact?

嗯，正如你所说，我身兼多职。你可以在其他地方了解更多关于每一个的信息。我在学术方面，在加州大学伯克利分校的所有研究，都在网上。我鼓励人们了解一下 Scale Cognition，那也在网上。非常感谢你邀请我上节目。

Well, as you said, I wear lots of hats. You can find out more about each of them in another place. All of my research on the academic side, at UC Berkeley, that's all on the web. I encourage people to learn a little bit about Scale Cognition, and that's also on the web. And I really appreciate you having me on the show here.

这非常有趣。技术问题可能很容易，但这些关于我们为何做事的深层问题非常重要。我只想感谢你关注这些。

It was a lot of fun. Technical questions can be easy, but these deeper questions about why we do things are so important. I just want to say thank you for focusing on those.

我很感激。Dan，非常感谢你。

I appreciate it. Dan, thank you so much.

不客气。

My pleasure.