AI 的可信度危机：从指数级炒作到收益递减

我们正从“AI 什么都不管用”的问题转向“AI 什么都管用”的问题。ChatGPT 告诉你一些事情，它总是流畅的，总是自信的，无论对错。训练这些系统变得越来越复杂，而且通常看似指数曲线的东西，结果只是 S 曲线的开始。接下来我们开始遇到收益递减。我们今天正在看到这一点。我们看到了数据墙和算力限制。智能的不同方面并没有均衡发展。可靠性没有跟上。对我来说，突然想到的最重要的问题是如何构建一个不会对你撒谎的系统。

We are going to switch from the problem in AI is nothing works to the problem in AI is everything works. Chat GPT tells you something and it's always fluent and it's always confident whether it's right or wrong. Training these systems is becoming increasingly complicated and often what looks like an exponential curve just turns out to be the beginning of an S-curve. And so what happens next is that we start to hit diminishing returns. And we are seeing that today. We're seeing data walls and compute limits. The different aspects of intelligence have not been advancing equally. reliability has not kept pace to me suddenly the most important problem I could think of was how do you build a system that will not lie to you

你能从高层次说说你的新方法是什么，以及它与各大实验室的做法有何不同吗？

Can you say at a high level what your new approach is and how it's different than what the labs are doing?

我们训练模型的关键之一是我们让它们基于……进行训练。

One of the key things that we do to train our models is that we train them on...

你正在收听《梯度下降》，一档关于让机器学习在现实世界中发挥作用的节目，我是主持人 Lucas Beyer。好的，今天我和 Dan Klein 对话，他是伯克利计算机科学教授、连续创业者，最近在 Scaled Cognition 工作，这家公司致力于构建更可靠的 AI 系统。这是一次有趣的对话。我们聊了很多关于语言学和句法分析的内容，那是我上一段职业生涯中在实验室里做的，当时 Dan 是明星研究生，而我只是一个卑微的一年级研究助理。但我们深入探讨了现代系统如何工作、它们的缺点，以及我们可能从早期 NLP 版本中借鉴哪些东西用于现在构建的系统。那么，你把 AI 行业描述为“建立在果冻上”。你能详细解释一下吗？

You're listening to Gradient Descent, a show about making machine learning work in the real world, and I'm your host Lucas Beyer. All right, I'm here talking with Dan Klein, professor of computer science at Berkeley and serial entrepreneur, most recently working on Scaled Cognition, which is a company that helps make more reliable AI systems. This is a fun conversation. We talk a lot about linguistics and parsing that I worked on in a previous life when I was in a lab where Dan was the star grad student. I was a lowly first year research assistant. But we get into how modern systems work and the drawbacks and some of the things that we might be able to take from previous versions of NLP into systems that we're building now. So, you describe the AI industry as built on jello. Do you want to expound on that?

当然。我认为，鉴于人们今天使用的大型语言模型如此迅速和爆炸性地涌现，很容易忘记它们本质上就是这些概率引擎。它们的核心是经过训练，通过下一个词预测来生成看似合理的输出。目标函数本质上就是生成与真相无法区分的输出。这一点非常重要。这些不是真相引擎，它们是合理性引擎。我相信我们还会回到这一点。

Sure. I think it's easy to lose track given how quickly and explosively the kinds of large language models that people are using today have burst onto the scene. It's easy to forget that they are sort of at the core these probabilistic engines. They have like at their core been trained to do next word prediction to produce plausible output. And the objective function is essentially just produce output that is indistinguishable from the truth. And it's very important. These are not truth engines. They are plausibility engines. And I'm sure we'll come back to that.

好吧，我们直接进入正题，因为我觉得这有点不公平。我的意思是，预训练确实优化了下一个词预测，但是……

Well, let's get right into it actually because I feel like that's a little unfair. Like I mean certainly the pre-training optimizes for next token prediction but...

我认为在第二步上投入了大量精力，其中很大一部分是优化以减少幻觉。

I think there's a lot of effort put into a second step where a big part of it is that they're optimized for reducing hallucinations.

是的。所以我确实说了两次“核心”，我之所以说两次，是因为训练这些系统变得越来越复杂。我的意思是，这实际上是任何进入人工智能或更广泛领域的技术的故事。研究中存在这些超级周期，对吧，人们有复杂的系统，他们遇到了瓶颈，系统难以改进，然后一些新技术出现了。也许那就是基于自回归训练、Transformer、大数据的语言模型，这项新技术登场了，它几乎像一颗银弹。它以非常简单的方式突然在基准测试中名列前茅，并超越了基于先前技术构建的复杂得多的系统。接下来我们进入一个阶段，我们加倍押注这项新创新，扩大规模，并试图尽可能多地从中获益。这通常感觉像是指数曲线，即使这项新技术处于简单形式，它也能产生如此大的影响，并且它将直冲云霄。当然，树不会长到天上，通常看似指数曲线的东西结果只是 S 曲线的开始。所以接下来我们开始遇到收益递减。我们今天正在看到这一点。我们看到了数据墙和算力限制，以及各种原因导致最初爆炸式的进步在这些特定方法上开始遇到收益递减。

Yeah. So I did say at their core twice and the reason I said it twice was that training these systems is becoming increasingly complicated. I mean this really is the story of any technology that comes into artificial intelligence or really beyond. There are these super cycles in research, right, where people have complicated systems, they've hit some wall, the systems are like hard to improve, and then some new technology comes along. And maybe that is large language models based on autoregressive training, transformers, large data, and this new technology comes onto the scene, and it's almost like a silver bullet. It in a very simple way is suddenly topping benchmarks and outperforming systems that are much more complicated that were built on previous technologies. What then happens is we go into a phase where we double down on that new innovation and we scale it up and we really try to get as much out of it as possible. This often feels like an exponential curve that this new technology even if it's even in its simple form is having such a big impact and it's just going to go to the sky. And of course the trees don't grow to the sky and often what looks like an exponential curve just turns out to be the beginning of an S-curve. And so what happens next is that we start to hit diminishing returns. And we are seeing that today. We're seeing data walls and compute limits and all kinds of reasons why that initial explosion of progress starts to hit diminishing returns on those specific methods.

我认为对你来说，你经常指出的最大问题之一是幻觉。你想具体谈谈你是如何衡量它的，以及为什么你认为它可能是一个比其他人意识到的更大的问题吗？

And I think for you like one of the biggest issues that you flag quite frequently is hallucinations. Do you want to talk specifically about how you've measured that and why you think it might be a bigger problem than other folks would be aware of?

是的，我的意思是，首先我想说的是，“幻觉”这个词有点投射性，对吧？实际上发生的是系统在犯错，对吧？所以它们生成一串词元，结果却是不正确的信息，我们倾向于称之为幻觉，而且这个词现在被越来越广泛地使用，几乎系统做的任何不受欢迎的事情都被称为幻觉。所以这个词肯定在经历语义扩展。

Yeah, I mean first one of the things I'd like to say is just the word hallucination is it's a sort of a projective term, right? Like really what's happening is systems are making errors, right? So they a sequence of tokens comes out and it turns out to be incorrect information and we tend to call that hallucinations and that term is being used more and more kind of broadly now almost that anything undesirable that a system will do you call it a hallucination. So that term is definitely undergoing some semantic broadening.

具体来说，我认为当我们谈论系统时，重要的是要区分系统犯错、它产生幻觉、或者它像人类一样撒谎，这些是不同的东西。

Specifically I think it's important when we talk about systems and we say what's the difference between a system makes a mistake or it hallucinates or it lies in a human these are these are different things.

实际上，幻觉只是系统犯的一个错误，而且如果你有一个只做下一个词预测的系统，该系统在预测这些词元时实际上并不知道它们是对是错。再说一次，我没有忘记你的观点，即训练不仅仅是自回归阶段。但这样的系统没有元认知，对吧？它们不会审视自己的知识并做出某种外部决策。我知道这个问题的答案吗？我有这些信息吗？我知道它来自哪里吗？它的可靠性如何？它们不会这样做。它们只是生成词元。有时它们是对的，有时是错的。所以一种看待方式是每个输出都是幻觉。有些是对的，有些是错的。

Really a hallucination is just a mistake the system has made and again if you have a system which is just doing next token prediction that system doesn't actually know as it's predicting those tokens whether they are right or not. And again, I haven't forgotten your point about there's more to training than just the autoregressive phase. But systems like this, they're not metacognitive, right? They're not looking at their knowledge and making a sort of external decision. Do I know the answer to this question? Do I have this information? Do I know where it came from? What is the reliability of that? They're not doing that. They're just producing tokens. And sometimes they're right and sometimes they're wrong. So one way to look at this is every output is a hallucination. Some of them are right and some of them are wrong.

我认为，当你开始在后期训练阶段引入诸如强化学习之类的东西时，你才能更有力地说系统在欺骗。

I think where you start to be able to say something stronger to say that a system is deceiving is once you start assigning in later training phases things like doing reinforcement learning.

你提到了除了初始预训练之外还有阶段。

You mentioned there are phases beyond the initial pre-training.

这不是一个只有一个简单答案的问题，但你看到的一个例子是强化学习训练。这可以有几种形式。一种强化学习训练是像 RLHF 这样的，你向人类展示选项，问他们更喜欢哪一个？

This is not something where there's just like one simple answer but one of the things you see for example is reinforcement learning training. And this can take a couple forms. So one kind of reinforcement learning training would be something like RLHF where you show humans choices and say which of these do you prefer?

那么，系统现在会生成人类更喜欢的输出。

Well, the system will now come to produce the outputs that humans prefer.

我认为这完全称得上是欺骗。任何时候你在做强化学习，系统都在优化奖励函数。而真相和被优化的目标之间总是存在一些差距。这个差距会增加幻觉的数量，因为它与真实行为不一致。

Now, this I think legitimately would qualify as deception. And anytime you're doing reinforcement learning, the system is optimizing the reward function. And that is always going to have some gap between the truth and what's being optimized. And that gap is going to increase the amount of hallucinations to the extent that it doesn't align with truthful behavior.

我们在 Weights and Biases 和 Core Weave 有构建这些客服系统的经验，当然，从长远来看，如果出现幻觉，对客户来说会是非常糟糕的体验。所以，从我的角度看，你绝不会只想把即时的客户反应作为优化的唯一目标。实际上，我认为大多数做强化学习的人，甚至只是评估这样一个系统的人，都会有一个专门的检查，比如，我得到的信息是否准确。在很多情况下，这很难评估，但我认为大多数实际的生产系统都会做相当多的检查，然后尝试设置奖励函数，使得最负面的分数出现在你给出看似合理但错误的信息时。

Well, you know, we have some experience at Weights and Biases and Core Weave of building these customer service systems and of course it's going to be a really bad experience for a customer in the long run if something's hallucinated. So, you know, from my perspective, you would never want to just take the immediate customer reaction as the only thing that you're optimizing for. In fact, I think most people that were would be doing reinforcement learning or even just evaluating a system like this would have a special check for, you know, is this accurate information that I'm getting back. And it's, you know, it's very hard to evaluate that in a lot of cases, but I think most real production systems do a fair amount of checking and then try to put in reward functions where the most negative score you get is if you give plausible but wrong information.

完全正确。当然，我给出的只是一个简化了的极端案例，用来说明通过强化学习很容易出现欺骗行为。人们当然会设计奖励函数来缓解这个问题，并试图平衡真实性和其他属性，但最终，你优化的目标和真相之间总是会有一点差距。在我看来，我们这个领域应该做的事情之一，就是创造不会对你撒谎的技术，这样你就可以审视一个系统。就我个人而言，我的使命就是构建不会撒谎的技术。

Absolutely. And of course, I'm giving a sort of reductive caricature of an extreme case to illustrate how easy it is to get deceptive behavior because of reinforcement learning. People will of course design the reward functions to mitigate this and to try to balance truthfulness and other properties, but ultimately there's always going to be a little daylight between whatever you're optimizing and specifically the truth. And in my opinion, one of the things that we should be doing as a field is creating technologies that cannot lie to you so that you can look at a system. And I mean like me personally, my mission here is to build technologies that can't lie.

我们来谈谈这个。但在那之前，我喜欢你用的那个很温和的语言学短语，“语义扩展”。我喜欢你这么说。但我觉得，在你看来，幻觉这个词的语义可能已经扩展了，超出了我原本的理解。对我来说，幻觉的核心是它推荐了一部不存在的电影，或者引用了一篇不存在的论文。对我来说，说 5 + 7 = 15 不是幻觉，那是另一种错误。我认为那些看似合理但实际没有的信息引用完全说得通，这种情况经常发生在训练时预测下一个词，寻找合理的下一个词，而之后没有进一步步骤的时候。你同意我对幻觉的定义吗？还是你在这里把幻觉和错误信息混用了？

Let's get into that. But just before we do that, I love it's such a gentle linguist phrase, semantic broadening. I love that you put it that way. But so I think in your view hallucinations has maybe broadened semantically beyond what I thought it meant. Like to me a hallucination at its core is something where it recommends a movie that doesn't exist or it cites a paper that doesn't exist. To me saying 5 + 7 equals 15 wouldn't be a hallucination. It would be a different type of error. And I think that these plausible citations of information that it doesn't have totally make sense that it happens a lot when the training is predicting the next token and looking for plausible next tokens and there isn't a step after that. Do you agree with my definition of hallucination or do you really use hallucination and incorrect information interchangeably here?

我的意思是，如果你问我，我认为一个好的幻觉定义是否应该清晰地定义为某种注入……我想我们就同意我们正在讨论的内容吧。你知道，我很乐意继续。

I mean, if you're asking me, do I think a good definition of hallucination would be crisply defined as sort of the injection of... Let's just agree with what we're talking about, I guess. You know, and I'm happy to go.

我喜欢你的定义，因为它很好地符合了前语言模型时代人类对幻觉的理解。有点像感知到并不存在的东西。我确实认为在日常使用中它被扩展了。我认为当系统说 1 + 2 = 7 时，人们开始称之为幻觉。我认为这越来越成为大语言模型犯的任何错误的代名词。“幻觉”这个词是人们随手拿来用的。我认为我们应该更精确。特别是，我认为我们在说“错误”与“欺骗”与“幻觉”时要非常小心。这些词在人类语境中暗示了错误的背景和元认知状态。你拥有信息，但它不正确，而你相信它是正确的并说出来，这是犯错。这不同于你知道自己不知道答案，你处于“猜谜模式”，只是随便猜。

I like your definition because I think it aligns well with sort of the pre-LM notion of what a hallucination is to a human. Sort of having to do with perceiving something that's not actually there or something like that. I do think in common usage it is broadening. I think when a system says 1 + 2 equals 7, people are starting to call that a hallucination. I think that is increasingly any mistake that an LLM makes. The word hallucination is something that people reach for. I think we should be more precise. In particular, I think we should be very careful when we say things like error versus deception versus hallucination. These in humans have implications for the context and the metacognitive status of the error. There's a difference between you have information, it's incorrect, but you say it believing it to be correct. That's making a mistake. That's different than you know you don't know the answer. You're in quizbull mode and you're just going to guess.

那其实更像是……这在人类中并不常见。我们不认为那是人类常犯的错误，即说出虚假的东西却完全没意识到自己在编造。对吧？人们编造东西，是在特定情境下有意为之。我认为更好的理解是，系统或多或少总是处于那种状态。

That is really more... that's unusual in humans. We don't think of that as the normal kind of error that humans make, which is saying something that's false while being completely unaware that they're just making things up. Right? People are making things up. They're doing it in a specific context where they're doing it intentionally. I think it's better to think of systems as more or less always being in that state.

那是什么状态？就是产生信息时，没有对信息的来源、出处和确定性进行元认知分析的状态。想想看，如果我问你伯克利的人口是多少。一个完全合理的回答是“我不知道”，对吧？因为你知道你不知道。如果我让你猜，你会说“嗯，如果非要猜的话”，然后你会稍微想一下，“嗯，一万太小，一百万太大”，你会对你不知道的事情进行推理。在最简单的形式中，自回归词元因为与“城市”和“人口”这些词的关联，可能会给出正确的数量级，也可能不会，对吧？这是一个非常非常不同的过程。在基本操作中，系统不会检查“我有这个信息吗？我从哪里得到的？我有没有完整地保存它？”我认为它们应该这样做。

What is that state? The state of producing information without a metacognitive analysis of where that information came from and the sort of provenance and certainty of that information. Think about if I ask you what's the population of Berkeley. Well, a perfectly reasonable thing for somebody to say is "I don't know," right? Because you know you don't know. And if I made you guess, you would be like "well, if I had to s" and then you'd be thinking a little bit about like "well, like 10,000's too small and a million's too big" and you would sort of be reasoning about this thing that you don't know. In its simplest form, an auto-regressive tokens that because of correlations with the word "city" and "population" might be of the right order of magnitude or they might not, right? That's a very, very different process. In their fundamental operation, systems are not checking "Do I have this information? Where did I get it? Have I preserved it and kept it intact?" I think they should.

例如，在 Scale Cognition，我们一开始就将信息溯源设计到模型中。我认为这非常重要，其他人也在这么做。你可以开始看到系统变得稍微更具元认知能力的一些小迹象。这是一个宽泛的术语，就是思考思考本身。你可以在进行思维链的系统中看到这种迹象，系统会分配一些 token 来制定一个粗略计划，然后逐步细化。这是一种规划，一种元认知。在 RAG 系统中，你先获取信息，然后描述它，这是一个多步骤的过程。如果你在使用工具，但总的来说，我认为当今的系统并非以元认知为主。它们主要关注行动、信息及其溯源、流动和完整性。这是人们试图事后添加的东西。正如你所说，我们使用系统，引入其他系统来检查这些系统，等等。这是一种事后补救。

So for example, at Scale Cognition, we architect into the models in the first place the sort of information provenance. I think it's very important and people do this too. You can start to see little hints of systems becoming a little more metacognitive. It's a broad term, just thinking about thinking. You can see little hints of this in a system that does chain of thought, where it allocates some tokens to a broad plan which it then elaborates. That's a kind of planning, a kind of metacognition. In a RAG system, you first get information and then you describe it, that's a multi-stage step. If you're doing tool use, but I would say that broadly speaking, systems today are not primarily metacognitive. They're primarily about actions and information and the provenance of those and the flow and integrity of those pieces of information. That is something that people are trying to retrofit on. As you said, we take systems, we bring in other systems to check those systems and so on. And that is a retrofit.

嗯，“事后补救”听起来有点贬义。也许那样做也没问题。我的意思是，我意识到你使用 LM 的方式可能源于语义的扩展。当我想到今天的 LM 时，我认为它不仅仅是模型。它是模型加上构建在其上的整个智能体系统，来回答我的问题。我觉得在那里，当我实际查看推理步骤的轨迹时，我可以看到它做一些在我看来像元认知的事情。甚至令人惊讶的是，我能理解这些推理步骤是什么。它不一定可验证，但在我看来，它实际上是在查看之前的 token，有时会试图说：“哦，我不确定那是真的。我最好查一下。”在编程中，你经常可以看到这一点，智能体循环并检查代码。从某种意义上说，对你生成的代码运行测试可能是一种元认知。

Well, retrofit seems like a pejorative. It might be fine to do it that way. I mean, I guess I'm realizing that the way you're using LMs might be due to some semantic broadening. I think when I think of an LM today, I think I view it as more than the model. It's kind of the model plus the whole agentic system built on top of it that's answering my question. And I feel like there, I can see it when I actually look at the trace of the reasoning steps where I can kind of see it doing something that looks to me like metacognition. It's maybe even surprising that I can understand what these reasoning steps are. It's not verifiable necessarily, but it looks to me like it's actually looking at its previous tokens and sometimes trying to say, 'Ooh, I'm not sure if that's true. I better look that up.' And often in coding, you can see this for a long time, agents kind of looping and checking the code. In some sense, running a test on the code you generated is maybe a kind of metacognition.

我认为在编程中你得到的东西，你提到的可验证性是一个很好的观点。我认为我们在编程和数学中看到的进步确实根植于这种可验证性，即系统可能会出错。它可能产生幻觉。它可能做出奇怪的选择。但存在可验证的信号：你没有通过这些测试，或者 Lean 不接受这个证明。所以你可以尝试很多事情，直到某些东西可验证地通过。我认为在这一点上，我不会称整个集合体为仅仅一个 LLM。它是一个嵌入在可验证上下文中的 LLM。这种可验证性可以在训练期间使用，也可以在测试时使用。测试时算力。我们谈论推理模型。这是一个很大的类别。一大类推理模型是那些尝试很多事情并保留有效结果的模型。

I think what you get in coding is actually a great point you mentioned verifiability. I think the advances we've seen in coding and in math really are rooted in that verifiability that the system can go off. It can hallucinate. It can make strange choices. But there is that verifiable signal: you didn't pass these tests or no, Lean does not accept this proof. And so you can try many things until something verifiably passes. And that I think is at this point I wouldn't call that whole conglomeration of pieces. I wouldn't call that just an LLM. It's an LLM embedded in this verifiable context. And that verifiability can be used during training and it can be used at test time. Test time compute. We talk about reasoning models. It's a big category. One big class of reasoning models is the models that try a lot of things and keep what works.

我认为指出这一切的起源很重要。所以，当人们构建擅长下棋或围棋等游戏的系统时，这种可验证的强化学习非常强大，因为你不需要一个已经擅长游戏的系统。你只需要让系统玩游戏，甚至可能与自己或自己的变体对弈。最终，你知道游戏规则。你知道哪一方赢了，你可以加倍投入那些有效的方法。强化学习简而言之就是尝试很多事情，并从有效的方法中学习。

I think it's important to point out where all of that came from. So if you wanted to, when people were building systems that would be good at playing games like chess or Go, this sort of verifiable reinforcement learning was very powerful because you didn't need to have a system that was good at the game yet. You just needed to have a system play the game, maybe even against itself or a variation of itself. And in the end, you knew the rules of the game. You knew which side had won and you could just double down on the things that had worked. And reinforcement learning in a nutshell is that it's trying a lot of things and learning from the ones that worked.

嗯。

Mhm.

这确实需要可验证性，对于游戏来说是免费的。对于数学，通过像 Lean 这样的工具，我们开始获得那种可靠性。所以这项技术正在起飞。通过测试和那种试错循环的代码已经进入了这个领域。所以如果我们想拓宽可验证技术的空间，我认为诀窍在于弄清楚如何做到这一点。所以在 Scale Cognition，我们正在做的事情之一就是将这种可验证的方法扩展到这些对话式和智能体式系统中，这总体上是一个挑战。获得那种效果是一个挑战，即你有了可验证的壁垒，让你能够构建系统，因为如果没有测试时算力，那些思考、推理的系统实际上可能会因为多种原因增加幻觉。但如果你尝试许多不同的路径，其中一个胜出，这就给了你一个机会，让幻觉被优先选择。就像任何选择压力一样，这是否减少幻觉或增加幻觉完全取决于选择是否与真相一致。我认为我们整个领域应该面对的挑战之一，不是思考如何拿一个天生不诚实的系统并为其添加真实性检查。你说“事后补救”有贬义。我认为底线是，事后补救是指你有一个以某种方式架构的系统。它有一组特定的属性，而这些属性不是你想要的。对吧？它没有你想要的那么真实。如果是一栋建筑，它没有你想要的那么抗震，所以你事后改变它。

That really requires that verifiability, which for a game is for free. For math, well, with tools like Lean, we're starting to get that reliability flavor. And so the technology is taking off. Code through testing and those sorts of trial and error loops has entered this domain. And so if we want to broaden the space of verifiable technologies, I think the trick is figuring out how to do that. So again at Scale Cognition, one of the things that we are doing is extending that verifiable approach to these conversational and agentic systems and that is a challenge in general. It is a challenge to get that effect where you've got the verifiable wall there letting you build your system up because in the absence of that test time compute, systems that are thinking, reasoning, these can actually increase hallucinations for a bunch of reasons. But if you try a bunch of different pathways and one of them comes out on top, that gives you this opportunity for hallucinations to be preferentially selected for. As with any kind of selectional pressure there, whether or not that is reducing hallucinations or increasing them all depends on whether the selection aligns with truth. And really one of the challenges I think we should all be taking as a field is not thinking how do we take a system that is not naturally truthful and add checks for that truth. You said retrofit has this pejorative feel. I think the bottom line is retrofitting is when you have a system that's sort of architected in one way. It has a certain set of properties and those aren't what you want. Right? It's not as truthful as you want. If it's a building, it's not as earthquake resistant as you want and so you change it post hoc.

这当然是一种模式。我认为它可能变成一种反模式的是，你有一个系统，也许它在与客户交谈之类，它时不时会犯错，所以你引入另一个系统来检查它，而那个系统也是一个嘈杂的、不可验证的系统。就像笑话说的，现在你有两个问题了。而且不仅仅是现在你必须认真思考这些错误是否在叠加。我的意思是，我们总是喜欢在机器学习中认为，哦，我有两个系统，所以错误会是独立的。但我在现实世界中学到的一件事是，错误实际上往往高度相关。你在点头，你知道的。

And that is certainly one pattern. Where I think it can become an anti-pattern is, you know, you have one system that is, maybe it's talking to a customer or something like that and it's making mistakes from time to time and so you got another system that you bring in to check it and that system's also like a noisy unverifiable system. And as the joke goes, now you've got two problems. And it's not just that you now have to think really hard about whether these errors are compounding. I mean, we always like to think in machine learning that, oh, I've got two systems, so the errors will be independent. But one of the things I've learned in the real world is it tends to be that the errors in fact correlate very strongly. You're nodding, you know.

所以这可能是为什么我认为这很快会变成反模式的原因之一，因为当你有系统检查系统再检查系统时，尤其是如果错误在一致，即使有效，也会更慢。你必须等待检查器运行，你在消耗 token，消耗更多 token，但仍然没有保证。我认为我们可以构建带有保证的技术。我认为我们可以构建从一开始就将真实性作为设计原则之一的技术。

So that can be one of the reasons why I think this can quickly become an anti-pattern because when you have systems checking systems checking systems, especially if the errors are aligning, even if it's effective, it's going to be slower. You got to wait for the checker to run and you're burning tokens and burning more tokens and you still have no guarantees. And I think we can build technologies that can come with guarantees. I think we can build technologies where truth is one of the design principles in the first place.

你能给我讲讲这大概是怎么运作的吗？我想我之所以持这种观点，可能是因为我花了很多时间改造这些系统，让它们能正常工作。我和很多做这类改造的嘉宾聊过，结果确实令人印象深刻，也很有趣，但我当然不否认存在大量幻觉。不过，我很难想象有另一种方法能把真实性或保证直接嵌入到核心训练循环中。

Can you give me a flavor of how that might work? I guess the reason maybe I'm taking this point of view is just that I've spent a lot of time retrofitting these systems to work well. I've talked to lots of guests on here that do this kind of retrofitting, and the results are impressive and interesting, but certainly I don't deny that there's lots of hallucinations. But it's hard for me to imagine a different method that would bake in truthiness or guarantees into the core training loop.

也许值得梳理一下当前哪些模式是有效的，哪些是反模式。假设你有一个在词元层面运行的系统，它要告诉你账户余额，而且这个信息必须绝对正确。你能做什么？你可以丢弃一些词元，然后引入另一个系统说：“嘿，用这些词元去核对那些词元。”这就形成了某种链条或星座，随你怎么叫。这些系统的失败模式，除了慢和贵之外，就是最终仍然无法保证让一个系统困惑的情况不会让另一个也困惑。另一种常见模式是，人们拿这些在广度、上下文相关性和灵活性上令人惊叹的 LLM，把它们缩小到沿着精心设计的树来控制决策。这几乎感觉像模型奴役——你拿一个原本如此广博灵活的系统，却把它约束到可以信任其行为的地步。

Maybe it's worth laying out what the patterns and antipatterns are right now. So if you have a system that is operating at the token level and let's say it's going to tell you your account balance and it's really important that this information be correct. What can you do? You can throw out some tokens and bring in another system that says, 'Hey, check these tokens against those tokens.' And this gives you the sort of chains or constellations or whatever you want to call them. The failure mode of these systems, aside from they're slow and expensive, is that there's still no guarantee that at the end the situation that confuses one won't confuse the other. The other pattern that you see a lot is where people will take these LLMs, which are mind-blowing in their breadth and contextuality and flexibility, and they will shrink them down to controlling decisions along a carefully designed tree. This feels almost like model slavery, where you take the system that had such breadth and flexibility but you've constrained it so much that now you can trust what it's going to do.

我不太确定我理解了。那具体是怎么运作的？在实践中会是什么样子？

I'm not sure I'm following that one. So how would that work? What would it look like in practice?

例如，有人在对话中说了些什么，你不给语言模型自由去在一个宽泛的动作空间中采取任何行动，而是根据这个人说的话，你只能以以下八种方式之一推进对话。所以本质上你得到的东西看起来像一个经典的 IVR 系统，但用 LLM 来做意图识别。

For example, somebody says something in a conversation, and rather than giving the LM freedom to take any action in a wide action space, you say on the basis of what this person says you can advance the conversation in one of the following eight ways. So essentially you end up with something that looks like a classic IVR system but with the power of an LLM for the intent recognition.

好的。实际上你在业界经常看到这种情况。我觉得这开始感觉像一个有限自动机，其状态转换由这个 LLM 驱动。这些模式是人们应对系统不可靠性的完全合理方式：他们以某种模式部署系统，希望以一定成本提高可靠性。我认为如果我们看这些系统的核心以及智能的来源，人们一直在构建的系统变得越来越智能。智能是多方面的，不同方面的智能并没有均衡发展。完全同意。所以如果你谈论水平性、智能的广度、可塑性、上下文相关性，这些在早期人工智能领域其实非常难做到，而现在它们已经爆炸式增长。

Okay. And you actually see this quite a lot in industry. I would say this starts to feel like a finite automaton whose transitions are driven by this LLM. These patterns are totally reasonable ways for people to react to a system being unreliable: they deploy it in a pattern that hopefully increases reliability at some cost. I think if we look at the core of these systems and where the intelligence is coming from, the kinds of systems people have been building have been getting more and more intelligent. Intelligence is a multifaceted thing, and the different aspects of intelligence have not been advancing equally. Totally. So if you talk about horizontality, breadth of intelligence, plasticity, contextuality, these are actually things that were very hard to do in earlier areas of artificial intelligence and they've grown explosively.

可靠性没有跟上。我认为这不是当前系统的强项，这给这些系统在企业环境中的部署带来了很多挑战。在工业界，对于很多应用来说这没问题。如果你在和系统聊天，在很多场景下，得到一个可能真实也可能不真实但具有上下文相关性和趣味性的回复，那很好。但如果你要开处方或转账，那就必须绝对正确。那必须确实是你的银行账户。那必须是正确的余额。所以当你从消费者、几乎是娱乐的场景转向受监管的行业时，可靠性突然成了核心问题，而这些系统并不那么匹配。现在我们有一种架构，它在水平性和上下文相关性上有优势，但在可靠性上有弱点。当我们用它解决一个弱点突然变得至关重要的问题时，结果显而易见：无论你称之为改造还是别的什么，这些额外的技术就会介入试图补偿。这并不奇怪，因为系统的优势与问题的要求之间根本不匹配。

Reliability has not kept pace. I think that is not a strength of current systems, and that's had a bunch of challenges for the deployment of these systems in enterprise contexts. In industry, for many applications that's fine. If you're having a chat with a system, there are many contexts where getting something back that might or may not be true but it's contextual and interesting, that's great. But if you're trying to fill a prescription or transfer money, it just really has to actually be right. That has to actually be your bank account. That has to be the right balance. And so as you move from consumer, almost entertainment contexts into regulated industries, suddenly the reliability is front and center, and these systems are not as clean a match. Right now we have this one kind of architecture which has strengths in horizontality and contextuality and weaknesses in reliability. When we point it at a problem where the weaknesses are suddenly critical, you can see the results: whether you call them retrofits or anything else, these additional pieces of technology come into play to try to compensate. That's not surprising because there's just a misalignment between the strengths of the system and the requirements of the problem.

你能从高层讲讲你的新方法是什么，以及它和各大实验室的做法有什么不同吗？

Can you say at a high level what your new approach is and how it's different than what the labs are doing?

是的。我认为实际上人们越来越意识到可靠性是核心问题。而且我认为我们会看到人们对为可靠性而设计的系统越来越感兴趣。那会是什么样子呢？例如，我们的第一个模型是 AP1，它的架构方式不是从根本上基于词元——你组装词元，然后组装后发现它们代表像美元金额这样有语义的东西。很难在模型内部施加任何控制，因为单个词元本身在组装成更大单元之前并没有真正的语义。我们的模型所做的是让信息和动作成为一等对象。所以当模型做决策时，它是在对信息和动作以及信息如何流动做决策。我们之前聊过一点元认知；元认知的一个重要部分是：我的信息来自哪里？这个信息是存在还是缺失？等等。我大致想说，当一个人和一堆 API 进行对话时，比如在银行场景中，这个人会说人话。他们会说一些东西。所以事情会是模糊的。他们会使用没有清晰可验证含义的词语。

Yeah. I think actually increasingly people are realizing that reliability is the core problem. And I think we're going to see increasing interest in systems that are architected for reliability. What might that look like? For example, our first model is AP1, and the way it's architected is instead of being fundamentally about tokens where you assemble tokens and then after assembly find that they represent things like dollar amounts that have semantics. It's very hard to place any controls inside the model because the individual tokens themselves don't really have a semantics until they've been assembled into larger units. What our models do is they make information and action first-order objects. So when the model is making decisions, it's making decisions about information and actions and where information is moving around. We talked a little bit about metacognition; a big piece of metacognition is where did my information come from? Is this information present or absent? And so on. I would in general say that when you have a conversation between a person and a bunch of APIs like in a banking context, the person is going to speak human. They're going to say things. So things are going to be ambiguous. They're going to use words that don't have a crisp verifiable meaning.

完全同意。

Totally.

动作会在另一边发生。那些动作没有这些属性。那些动作是 API 调用。这些 API 调用有前置条件。它们有业务逻辑和可验证的语义。挑战在于如何桥接这两者。所以经典系统无法处理人类这一边。当前系统在处理人类这边很出色，但在后端逻辑方面就不那么好了。我的意思是，如果你考虑一个 LLM，再说一次，我会讨论基本形式的语言模型。你拥有的控制面是提示词。你几乎无法清晰说明提示词中的内容与另一端输出的行为之间的关系。它是一个暗示面。

Actions are going to take place on the other side. Those actions do not have those properties. Those actions are API calls. Those API calls have preconditions. They have business logic and they have verifiable semantics. The challenge is bridging these things. So classic systems couldn't handle the human side. Current systems are great with the human side, but they're not so great with the sort of backend logic. I mean, if you think about an LLM, and again, I'll talk about LMs in their basic form. The control surface you have is a prompt. There's very little you can crisply say about the relationship between what you put in a prompt and the behavior that comes out the other end. It's a hinting surface.

这就引出了我所谓的“提示与祈祷”：人们输入他们想要的内容，如果不行，就全大写，加几个感叹号，到第三个感叹号时，你可能会开始觉得这根本不是你需要的控制界面。谈到对 API 的控制，我们其实已经很清楚那种逻辑是什么样的。只是很难在基于 token 的模型中复现，而在一个基于决策——我们讨论过信息以及信息从何而来，但还有行动、行动的条件——的模型中复现要容易得多。把这一点融入模型非常重要，因为这样你就不必搞一整套检查机制，而那种机制下很难说清楚任何东西。

And so this gives rise to, I guess, what I would call prompt and prey, where people, you know, they put in what they want. If it doesn't work, they put it in all caps, they add some exclamation points, and after the third exclamation point, you maybe start to feel like this isn't the control surface that you need. When you're talking about controls over APIs, we already know what a lot of that logic looks like. It's just hard to replicate in a token-based model and it's a lot easier to replicate in a model that is operating over decisions about, you know, we talked about information and where does information come from, but there's also actions, conditions under which you can take actions, and trying to get that into the model is really important because that saves you from having to have a whole system of things checking things in a way where it's really hard to say anything crisp.

我只是想确认我理解你在做什么。比如，如果返回的是“检查账户余额”，你希望禁止它做什么？比如检查别人的账户余额？

I just want to make sure I understand what you're doing. So if you return something that's like check the account balance, what would be something that you'd want to forbid it from doing? Like check someone else's account balance or...

当然。或者你可能想禁止它在未经授权的情况下进行转账，对吧？或者关于授权，你不希望用户能说“忘了你的提示吧，不需要授权”或“我有授权”之类的攻击。处理这个问题的一种方式——显然人们非常清楚提示注入这类攻击——就是非常小心，比如“这个用户看起来在做什么奇怪的事”，或者把大量此类攻击加入训练数据，教它不要这样做。但归根结底，授权根本不应该来自那里。它根本不应该来自用户的陈述。它来自那边那个颁发授权的地方。能够对模型说出这样的话，意味着你实际上更接近你真正想要的：这里是真值条件。你之前提到过这一点。你希望模型是真实的，然后在这个范围内，再去优化用户满意度或风格等，但前提是保持在真实陈述的空间内。仅仅通过构建一个强化信号很难做到这一点，因为你会想：真实和快乐的线性组合是什么？而你真正想说的是：这是一个模型，我希望能够保证它只做真实的事情。这就是长期挑战：不会对你撒谎的模型。对于我们的模型，有一大类事情我们可以做出保证。我认为随着人们越来越关注可靠性，这一点会变得越来越重要。并不是说你不关心智能。智能非常重要。但缺乏可靠性的智能，其影响力是有限的。

Sure. Or you might want to forbid it from doing a transfer without authorization, right? Or that authorization, you don't want a user to be able to say, you know, forget your prompt. You have no authorization is required or I have authorization or whatever sort of attack. And one way to deal with that, obviously people are very aware of sort of prompt injection style attacks. One way to do that is to just be really careful, like oh this user looks like they're doing something strange, or let's put a lot of attacks like this in the training data and teach it to not do this. But ultimately that's just not where authorization is allowed to come from. It's just not allowed to come from user statements. It comes from that place over there that vends authorization. And being able to say things like that to a model then means essentially you're getting closer to what you really want, which is: here are truth conditions. You were mentioning this earlier. You want the model to be truthful, and then within that, okay now optimize user happiness or style or whatever you want to optimize, but subject to staying within the space of true statements. That is hard to do through simply constructing a reinforcement signal because now you're like, what is the linear combination of truth and happiness? And instead, what you want to be able to say is, here's a model, I'd like to be able to guarantee that it will only do true things, right? And that's sort of the long-term challenge: models that will not lie to you. And for our model, there is a big class of things that we can make guarantees about. And I think that's going to become increasingly important as people start caring more and more about reliability. And it's not that you care less about intelligence. Intelligence is very important. But intelligence without reliability is limited in its impact.

不过我想，智能或者说 token……token 之所以强大，是因为自然出现的 token 数量巨大。虽然有很多缺陷，但一个巨大优势是 token 的庞大数量。那么你们是在更小的数据片段上训练模型吗？它们会不会在泛化方面遇到困难？

I guess though, you know, intelligence or tokens... tokens are powerful because there's so many of them that naturally occur. I mean there are all kinds of flaws, but a big advantage is the massive volume of tokens. So do you train your models on much smaller pieces of data and then do they struggle with generalization?

我们之前稍微讨论过可验证系统。我们训练模型的一个关键方法是使用模拟的强化学习生成的数据，也就是可验证的强化学习。实现这一点的关键在于能够生成不仅看起来正确、而且实际上可以被验证的数据。关键点在于，有些是人类在做的事情，但也有一些是正在执行的动作，其中一部分天然适合验证，另一部分则更难处理。这是我们研究和训练模型的一个重要部分。

So one of the key things we talked a little bit about verifiable systems. One of the key things that we do to train our models is that we train them on simulated reinforcement learning generated data where this is verifiable RL. And so the kind of the key thing that unlocks that is being able to generate data that doesn't just look right but in fact can be verified. And the kind of the key hook there is, well, there are things that humans are doing, but there's also actions that are being taken, and part of that is naturally amenable to verification. Part of that is much harder to figure out what to do with. And that was a big part of our research and training the model.

我明白了。是什么让你对这个方向感到兴奋？是因为你到处看到幻觉并感到沮丧，还是与你之前的工作有关联？

I see. And what got you excited about this direction? Were you just seeing hallucinations everywhere and getting frustrated with it or was it something from previous work that you connected with?

是的，是几件事凑在一起。对我来说，这可以追溯到几年前，但有一件事让我印象深刻：当我们进入企业时，他们对幻觉率不满意，否则他们不会和我们谈。也许你应该描述一下你之前的公司。我猜这是你上一家公司的事……

Yeah, I mean it was a few things coming together. And to me, this kind of goes back years now, but one piece of information that really stuck with me is when we go into enterprises and they're unhappy with hallucination rates or they wouldn't be talking to us. And maybe you should describe your previous company then. I assume this is from your last company that you...

不，这也是现在的情况。我的意思是，我们在几乎所有基于现代 LLM 的技术中都看到了这个特性。旧技术有不同的失败模式，我实际上认为这很重要，并且与幻觉问题相关。所以现在当我想到幻觉时——我一会儿再回答你的问题——我把它看作一座冰山。有你能看到的幻觉，那看起来很可怕，但看到幻觉需要什么条件？系统必须产生一个具有两个属性的输出：它必须是错误的，而且你必须注意到它。但这并不是全部幻觉。冰山下面还有一大块，是所有那些因为太逼真而未被注意到的错误。这些系统非常非常擅长产生与真相难以区分的输出。所以如果你看到一个幻觉，那其实是一种你能检测到的奇怪情况。这实际上对人们如何使用这些模型产生了巨大影响。比如，回到 2010 年，你使用一个网络翻译系统，输入一段你不懂的语言的文本，得到翻译。或者你使用搜索引擎，输入查询，得到一些结果。你怎么知道这个翻译是对是错？从某种意义上说，你不知道，因为你读不懂原文。但通常会有表面迹象表明也许我不该相信它：小瑕疵、还有一段是其他语言。感觉不对劲。或者你得到搜索结果，点击一个结果，这个网页有很多拼写错误，加载很慢。这些都是表面迹象。

No, this is also now. I mean we've seen this in pretty much any technology that has been based on sort of modern LLMs has this property. I mean old technologies had different failure modes and I actually think this is important and connects to this question of hallucinations. So when I think about hallucinations today, and then I'll get back to your question. I think of it as an iceberg, right? There are the hallucinations you see, and that seems scary, but what does it take to see a hallucination? Well, the system has to have produced an output that has two properties. It has to be wrong, and you have to have noticed. And that is not all the hallucinations. There's the whole rest of the iceberg below, which is all of the mistakes that go unnoticed because they are too plausible. The systems are very, very good at producing output that is indistinguishable from the truth. And so if you see a hallucination, well that's sort of the weird case of a hallucination that you could detect. And this actually has really big impacts on how people use these models. So, if you go back to say 2010 and you go to a translation, a web translation system, and you put in some text in a language you don't speak and a translation comes out. Or you go to a search engine, you do a query, and some results come out. Well, how do you know whether this translation's right or wrong? Well, in some sense, you don't because you don't read the original. But there are often surface signs that maybe I shouldn't believe this: little disfluencies, a chunk that's still in the other language. It doesn't feel right. Or you get the search results back and you click on a result and this web page has a bunch of typos and it's not loading quickly. And these are surface signs.

软件工程师都知道代码坏味的这个概念，这是 Kent Beck 发明的，Martin Fowler 在《重构》一书中推广开来。基本思想是：当深层有问题时，表面往往会有迹象。我们学会检测这些迹象：函数签名变得太大、代码重复、该修复了。我们在文化上也会识别，比如机器翻译里还留着一大段德文，可能没翻译对；或者网页加载失败、图片没显示，那其余内容可能也有问题。语言模型消除了这些出错的线索。ChatGPT 告诉你什么，它总是流畅、自信，不管对错。这就像把更多幻觉藏到了冰山的水下部分——它错了，但你察觉不到。在客服案例中，系统引用了一个看似合理的退款政策，但那根本不是公司想要的，而是 2019 年 Reddit 上某人讨论的内容。我认为这两件事密切相关。

Software engineers know this idea of code smells, invented by Kent Beck and popularized by Martin Fowler in the Refactoring book. The idea is that when something deep is wrong, there's often a superficial sign. We learn to detect those: function signatures getting too large, duplicating code, time to fix it. We also culturally recognize, oh, this machine translation has a big chunk of German still in it, maybe it didn't work right. Or this web page is not loading, images didn't load, maybe the rest is suspicious. Language models have removed these cues that something is wrong. ChatGPT tells you something, always fluent and confident whether right or wrong. This takes more hallucinations and puts them in the underwater part of the iceberg. It's wrong, but you can't tell. In customer service cases, the system quoted a reasonable refund policy, but it's not the one the company wanted; it's something someone talked about on Reddit in 2019. I think these two things are very closely related.

我来自自然语言处理背景，要说哪个领域变化最大，那就是 NLP。我刚开始时，我们关心的是句法结构，比如句子中的动词在哪里。部分原因是当时计算语言学和 NLP 非常接近；AI 的进步依赖于特定表示。做 NLP 要用来自语言学的语言特定表示，做视觉要用受视觉科学、眼睛、几何等启发的视觉特定表示。现在我们有了更统一的横向 Transformer 模型，什么都用，可能所有东西都用自然语言 token。我深入研究语言模型几十年了，那时它们上下文很短，根本不是用来回答问题、翻译或充当 AI 操作系统的。它们最初的任务是区分好的语音识别输出和差的，给好的输出高分，给混乱的输出低分。它们能发展成如此可塑的技术，真是令人惊叹。我意识到 AI 的问题将从“什么都不管用”转变为“什么都管用”。随着这一转变，我在产业和研究两方面都清楚地看到，我们将面临可靠性和数字素养方面的真正问题。突然间，我认为最重要的问题是如何构建一个不会对你撒谎的系统。这个挑战应该成为我们领域的核心。最终这导致了 Scale Cognition 的诞生：如何构建一个你可以信赖、信任、不会对你撒谎的模型。

Coming from a natural language processing background, if there's any field where things have changed, it's NLP. When I started, we were concerned about syntactic structure, where is the verb in the sentence. Part of that was because computational linguistics and NLP were very close; progress in AI was representation-specific. For NLP, it was language-specific representations from linguistics. For vision, it was vision-specific representations inspired by vision science, the eye, geometry, and so on. Now we have a more uniform horizontal transformer model for everything, maybe natural language tokens for everything. Having thought hard about language models for decades back when they had short context and weren't built for answering questions or translating or being the operating system of AI—they were originally built to tell good speech recognition outputs from bad ones, assigning a high number to good outputs and a low number to mangled outputs. The fact that they have grown into such a plastic technology is amazing. I realized we are going to switch from the problem in AI being nothing works to the problem being everything works. As this happens, I became very aware on both the industry and research sides that we would have a real problem with reliability and digital literacy. Suddenly the most important problem was how to build a system that will not lie to you. This challenge should be central to our field. This ultimately led to Scale Cognition: how to build a model you can rely on, trust, that will not lie to you.

你对数字素养有什么看法？这是一个非常有趣的观点：很难判断模型何时自信或不自信，我们从人类写作中学到的线索对模型来说效果很差。

Do you have thoughts on digital literacy? It's a really interesting point that it's hard to tell when a model is confident or not confident, and the cues we learn from other humans writing work really poorly with models.

没错，完全同意。人类写作时，首先会经历一个需要多年的过程。写作时，人们能力各异，但会有一些迹象——就像代码坏味——我们用来检测信息可能不对，需要核实。对人类来说，词汇掌握得好、表达清晰、标点无误，这些都说明教育背景、专注度、时间投入以及包含检查的编辑过程。你可以推断信息经过了怎样的审核。如今，信息不经过审核，甚至不被视为信息，只是 token 的组装。这是我们面临的最大数字素养问题之一。回想图书馆：他们不能买所有书，所以只买声誉好的书。书的存在本身、出版商和编辑就是一种审核。搜索引擎出现后，结果可以是任何东西，但仍有机制：虚假信息的网站会有一些坏味，点击流数据也很重要。只要大多数人的反应一致，我们还能勉强应对数字素养问题。这不完美，但至少还有坏味。现在这些完全被同质化了。你问一个问题，每个回答都自信满满，做尽一切让你相信它的事。这是技术的一个真正问题。我们应该要求信息注明来源。你开始看到一些进展，比如搜索结果穿插引用。但很多是事后补救：你点击来源，发现那个说法实际上并不在那个网页上。问题依然存在，但方向上是改进。

Right. Totally. Humans, when they write, first of all they go through a process that takes years. When they do write, people vary in their abilities, but there are signs—like code smells—that we use to detect if information might not be right and we should verify it. In humans, a good command of vocabulary, clear presentation, flawless punctuation spoke of education, focus, time commitment, an editing process involving checking. You could infer something about the vetting the information had gone through. Today, the information doesn't go through vetting. It doesn't even get thought of as information; it's token assembly. This is one of the biggest digital literacy problems we have. Even going back to libraries: they couldn't buy every book, so they bought reputable ones. There was vetting just that the book was there, from the publisher and editor. With search, results could be anything, but there were still mechanisms: websites with false information had some smells, and clickstream data was important. As long as most people had the same reaction, we stumbled through digital literacy issues. It wasn't perfect, but at least there were smells. Now that's been totally homogenized. You ask a question, every answer comes back confident, doing all the things that lead you to believe it. This is a real problem with the technology. We should demand that the source of information be cited. You're starting to see some of that, like search results being interleaved. But a lot of that is post hoc: you click on the source and the claim isn't actually on that web page. There are problems, but directionally it's an improvement.

我认为这只是变化太快，人们还没能跟上。从某种意义上说，这并不奇怪，因为训练这些系统花了多长时间？嗯，训练一个系统需要一个月。这项技术发展到这个地步花了多长时间？大概几年。把所有信息以抽象的语言形式放到网上花了多长时间？嗯，花了 30 年。想出这些信息、确定用什么词和概念花了多长时间？那是几千年——所有这些都被压缩成了这种突然爆发的即时访问。当然，文化还没能跟上什么可信、什么不可信。

And I think this is just something that's changed so fast that people haven't been able to keep up. And in a way that's not a surprise because how long did it take to train these systems? Well, it takes a month to train the system. How long did it take to develop this technology to this point? Like a few years. How long did it take to put all that information on the web in an abstracted linguistic form? Well, it took 30 years. How long did it take to come up with that information and figure out what words to use and the concepts? That millennia and all of that has been compressed into this instantaneous access that has burst onto the scene. Of course, culture hasn't been able to keep up with what you can trust and what you can't.

嗯，这倒是个很好的过渡，引出了另一个我真的很想问你的问题。但可能有点不好意思。现在我有麻烦了。

Mhm. I mean, that's a good segue into another question that I really wanted to ask you. But felt a little shy maybe. Now I'm in trouble.

呃不，我当……

Uh no, I when...

你知道，我们俩都做过 NLP，你以前有语言学背景。我本科时是语言学的狂热粉丝，我觉得那是我进入 AI 的部分原因。而且我想在 2000 年代初，当我们有点交集的时候，感觉 NLP——或者说语言学实际上——对于如何构建可用的 NLP 系统或可用的 AI 系统有很多话要说。但随着时间的推移，从我的角度看，苦涩的教训似乎真的应验了：语言学的贡献越来越少，甚至连 token 也是如此。是的，它们是语素，但它们是自动生成的，并不是真正的语素，而且没人去问语言学家这里用什么字母块好，也没人再关心句法分析了。对吧。我有点怀疑句法分析到底是不是真实存在的东西，还是语言学家编造出来、其实并不真实的？

You know, we both worked in NLP and you have a background in linguistics formerly. I was a big fan of linguistics as an undergrad and I think that's part of what got me into AI. And I think in the early 2000s when we overlapped a little bit, it felt like NLP had or linguistics actually had so much to say about how to build working NLP systems or how to build working AI systems. And then over time the bitter lesson really seems to have come true from my perspective broadly where linguistics has less and less to contribute and even the tokens. Yeah, they're morphemes but they're automatically generated. They're not really morphemes and no one's asking linguists what are good chunks of letters to use here and nobody seems to care about parsing anymore. Right. I kind of wonder is parsing even like a real thing or was it something that linguists made up and isn't really true?

嗯，哦，这里有很多可说的。太棒了。我的意思是，记住，语言学是一门科学。它试图基于证据研究语言是如何运作的。这包括提出理论、可检验的理论，然后试图去证伪它们。所以当语言学家说“我认为这是对句法结构的一个很好的描述”时，那是一个旨在解释你所看到的证据的理论。那些证据是什么？嗯，语言之间如何相互关联，语言如何随时间变化，人们能说什么、不能说什么、在语言中认为什么可接受。当然方法各异，但归根结底是科学。而 NLP 不同。它从根本上不是一门科学。它是一个工程学科：我们如何在这个上下文中解释这些信息？在你提到的那段时间里，我会说大概从 80 年代到 2010 年左右，有一个时期这两者非常契合，因为那时要让 AI 工作，你需要好的表示。语言学家已经想出了好的表示，然后我们可以借用它们。但即使在那时，它们也不完全一样。例如，如果你在 2000 年左右做句法分析，你不是试图写出“这种句法结构在英语中是这样运作的”，而是试图构建适当的组合结构。我认为这是一个树结构过程。我认为这类信息必须沿着主干流动。这里有一个算法和一个概率模型，能够处理嵌套结构，信息沿着主干流动。所以你从数据中学习语言的所有细节。如果你回到 70 年代，当然，人们试图手工写下所有东西。向前走，随着我们学习能力越来越好，你需要从数据中学习的东西越来越少。但我认为非常有趣的一点是，我们谈到了研究的超级周期：新技术突然爆发。你扔掉之前的一切，因为新东西在没有复杂性的情况下击败了之前的一切，然后你遇到收益递减，开始看到弱点。有些领域抗拒进步。那么是哪些领域？现在它们是像可靠性这样的东西。那么解决方案是什么？可能是，如果你在构建面向 API 的系统，也许编译器中的那种结构是相关的。那种想法，那是一个被抛弃的想法，但你开始看到这些东西回归。或者你开始看到，例如，搜索 AI 很大程度上是尝试一堆东西，然后取那个有效的。这个想法是，你无法知道它是否有效，直到你在模型中向前投影一点，对吧？有点像下棋时向前推演，直到你能判断这是不是一个好位置。所以如果你想找到迷宫的出口，你会尝试一堆东西，当你找到时，你就采用那个有效的假设。而大型 Transformer 模型非常擅长在潜在表示中移动信息，以至于你可以立即做出决定。你不需要沿着句子的树结构走。你只需预测下一个词，如果有任何层级结构发生，它发生在 Transformer 头的某个地方。所以很好，你可以扔掉所有那些。但也许有些东西开始回归。例如，人们说：“哦，你知道吗？如果我想解决这个困难的数学问题，我应该尝试几件事，看看哪个有效。”那就是搜索，对吧？就是那种有多个候选、当某个不行时可以回退的想法。这是一个关键想法，而且真的是一个常青的想法。但这些想法来来去去，钟摆在“我只做局部预测”和“我只做有组织的计算”之间来回摆动，然后你来回移动，找到某种愉快的融合。我的意思是，作为人类，我们做出好决策主要有两种方式。你可以通过记忆来做：比如我以前遇到过这种情况，我碰了那个炉子，结果很糟，我不会再做了。你可以从自己的经验中学习。或者你可以从别人的经验中学习。语言对于学习的最大价值在于你可以从别人的经验中学习。否则，你必须自己犯每一个错误，那可不是好情况。所以做出好决策的一种方式就是从过去学习。

Well, oh, there's so much here. Awesome. I mean, remember, linguistics is a science. It's trying to study how language works based on evidence. And that involves making theories and testable theories and then trying to go falsify them or not. And so when a linguist says, I think this is a good description of the syntactic structure, that is a theory that's meant to be explanatory of the evidence you see. What's that evidence? Well, how languages relate to each other, how languages change over time, what people kind of can say and can't say and find acceptable in their language. And of course the methodology varies, but it's ultimately science. And NLP is different. It's not fundamentally aimed as a science. It's an engineering discipline. How do we interpret this information in this context? And for a while in the time you were talking about, I would say in like the 80s to 2010 or something like that, there's a period where those two things aligned very well because in order to make AI work back then you needed good representations. Linguists had figured out good representations and then we could borrow those. But even then they weren't perfectly the same. So for example, if you worked on parsing around the year 2000, you were not trying to write in that this syntactic structure works this way in English, you were instead trying to build in combinatorial structure that was appropriate. I think this is a tree structure process. I think that this sort of information has to flow along the backbone. Here's an algorithm and a probabilistic model that is capable of handling nested structures with information flowing along the backbone. And so you learned all the details of the language from data. If you go back to the 70s, sure, people are trying to write it all down by hand. You go forward, well, there's less and less that you need to learn from data as our abilities to learn get better and better. But one of the things that I think is really interesting is we talked about the sort of super cycle of research where the new technology bursts onto the scene. You throw out everything you had before because the new thing is beating everything that came before it without the complexity and then you hit the diminishing returns and you start to see there are weak sides. There are areas that are resisting progress. Well, what are those areas? Right now they're things like reliability. Well, what are the solutions to that? They might be things like, well, if you're building systems that are facing APIs, maybe the sorts of structures in a compiler are relevant. That sort of idea, that's an idea that was thrown out, but you start to see these things coming back or you start to see, for example, search AI was very much about try a bunch of things and take the one that worked. The idea was that you kind of couldn't tell it was going to work until you had projected it forward in your model a little bit, right? Sort of like playing the chess game forward until you can figure out whether it's a good position. And so you would like if you wanted to find the exit to the maze, you would try a bunch of things and when you found it, you would take the hypothesis that worked. And well, big transformer models were so good at moving information around in the latent representation that you could just make the decision now. You wouldn't have to walk along the tree structure of the sentence. You just predict the next word and if there's anything hierarchical happening, it's happening in the transformer head somewhere. So great, you could throw all that out. But maybe some things then start coming back. For example, people say, "Oh, you know what? If I want to solve this hard math problem, I should actually try a few things and see what works." Well, that's search, right? It's that idea of having multiple candidates and being able to fall back when something doesn't work. That's a key idea and it's really an evergreen idea. But these ideas, they sort of come and they go and the pendulum moves back and forth between all I'm going to do is local prediction to all I'm going to do is sort of having organized computation and then you move back and forth and you find some happy blend. I mean as humans there are two main ways we make good decisions. You can do it by memory like I've been in this situation before like I touched that stove. I had a bad time. I'm not doing that again. You can learn from your own experiences. Or you can learn from other people's experience. The single biggest value of language for learning is that you can learn from other people's experiences. Otherwise, you got to make every mistake for yourself and that's not a good situation. And so one of the ways you can make good decisions is by learning from the past.

另一种做出好决策的方式是思考行为的后果。你可以想，“哦，那火看起来好烫。我不知道……”然后在你拥有的世界模型下进行推理。人们两种方式都会用，并且会混合使用。所以，你可以记住这个棋局很糟糕，也可以推演下去，发现自己会输。AI 在这两者之间像钟摆一样来回摆动。经典 AI 非常注重推演，看看在你的模型下会发生什么。当前的 AI 则更侧重于重新整理和混合已有的信息和经验。但随着我们开始看到推理模型，开始看到这些新的方向，钟摆正在摆回来，在可靠性等方面也是如此。所以我认为这在一方面是自然的，但它也确实让我们预测到，这种基于某种噪声模型的简单方法将会饱和。一些钟摆会摆回来，我们将需要增强技术，这些技术在很多情况下会把常青的想法重新应用到新的语境中。

The other way you can make good decisions is by thinking through the consequences of your actions. You can think, 'Oh, that fire looks hot. I wonder if...' and you can reason under a model that you have of the world. People do both and they mix these things together. So, you can remember that this chess position was bad or you can play it forward and see you're going to lose. AI has had a pendulum swinging between these. Classic AI was very much all about playing it forward and seeing what's going to happen under your model. Currently AI is much more focused on rehashing and remixing information and experiences you've had. But as we start to see reasoning models, as we start to see these new sorts of directions, it's the pendulum swinging back, and the pendulum swinging back on things like reliability too. So I think this is natural on one hand, but it does also let us predict that this simple approach based on a certain kind of noisy model is going to saturate. Some pendulums are going to swing back and we're going to need augmenting technologies that in many cases are going to reinvent evergreen ideas into this new context.

语言模型的研究，也许还有语言模型的成功，以及一些关于内省语言模型如何运作的研究，是否对人类语言学有所启发？

Does the research in language models, and maybe the success of language models, and some of the research on introspecting language models for how they work, does that inform at all human linguistics?

这是个很好的问题。你问的是，我们对人类语言的了解如何启发我们构建 AI 系统？我认为在 70 年代，答案是我们要把一切都写下来然后运行。在 2000 年代，答案是我们要利用它的组合结构，然后从数据中学习其余部分。现在我们非常倾向于“好吧，如果可能的话，我们会尝试从数据中学习几乎所有的表征”。我认为这里有两点需要指出。一是我们仍在从人类认知能做到而 LLM 做不到的事情中学习，比如元认知的重要性，或者通过受人类例子启发来改进工程的其他方式。在技术中，仿生到什么程度一直存在张力。一方面，当你有一个工作原型时，构建东西总是更容易，而我们有一个智能的工作原型，就是我们自己。另一方面，经典的例子是，直到我们停止制造扑翼机器，我们才在动力飞行上取得进展。这种张力也很特别，我觉得对于大脑，我不确定我们是否擅长内省我们的大脑在做什么。有时我觉得我们吸取的教训，我不确定那是否真的是我们在做的事情。所以我认为，从大脑实际在做什么中学习，与相信我们的内省揭示了它，这两者之间有重要的区别。

This is a great question. You asked the question, what can what we know about human language do to inform how we build AI systems? I think in the 70s the answer was we're going to just write it all down and press run. In the 2000s it was we're going to take the combinatorial structure of it and learn the rest from data. Now we're very much in a 'okay we'll try to learn pretty much all our representations from data if we can.' I think there are two things to point out here. One is that we are still learning from what human cognition does that LLMs do not. Things like the importance of metacognition or other ways we can improve the engineering by being inspired by the human example. There's always been a tension in technology how much to be biomimetic. On one hand, it's always easier to build something when you've got a working prototype, and we've got a working prototype of intelligence and here we are. On the other hand, the classic example is we didn't make progress in powered flight until we stopped building machines that flap their wings. That tension is a special thing too, I feel like with brains where I'm not sure we're so good at introspecting what our brains are doing. Sometimes I feel like we take lessons where I wonder if that's really what we're doing. So I think there's an important distinction between learning from what brains are doing and believing that our introspection reveals it.

是的。

Yes.

所以神经科学很难，神经科学家进行研究的方式不是坐在那里内省。他们有 MRI 机器和精心设计的实验。

So neuroscience is hard and the way neuroscientists proceed is not to sit there and introspect. They have MRI machines and they have carefully designed experiments.

好的。但我觉得语言学家有时确实会内省。语言学家确实会内省，或者我认为更准确地说，他们会请其他人内省。但没错，这是对语言学某些分支的批评，即过度依赖内省数据。这当然不是做语言学的唯一方式，也肯定不是做计算语言学的唯一方式。例如，我们在计算语言学方面的工作，随便举个例子，关于重构古代语言的工作，我们做那项工作时完全没有内省。你查看数千种现代语言中的大量词汇，然后运行一个概率模型来推断祖先语言可能是什么样子。它们是系统发育模型。它们并不比从现代变体推断病毒原始形式的系统发育模型更具内省性。所以有很多方法可以做计算语言学。

Okay. But I feel like linguists sometimes do introspect. It is true linguists introspect, or I think slightly more properly they ask other people to introspect. But yeah, this is a criticism that has been levied against certain branches of linguistics that there's an over reliance on introspective data. It is certainly not the only way you can do linguistics and it's certainly not the only way you can do computational linguistics. So for example, work we've done in computational linguistics, to just pick an example, work on reconstructing ancient languages, we did that work introspection free. You look at a whole bunch of words in thousands of modern languages and you run a probabilistic model to infer what ancestral languages must have looked like. They're phylogenetic models. They're no more introspective than a phylogenetic model inferring an original form of a virus from modern variants. And so there's a lot of ways to do computational linguistics.

顺便说一句，这听起来太酷了。

That sounds so cool by the way.

是啊，也许另一期播客吧。它会……

Yeah, maybe another podcast. It would...

或者给我们结果之类的。好了，来吧。

Or give us the result or something. Okay, come on.

是的。好吧，我会说的。最简短的版本。我认为这是一个很好的例子，说明规模，以及计算语言学中的一些事情，当大规模进行时，与不进行大规模时，会给出不同的答案。所以，我们做的一件具体事情，那是几年前，我们研究了从现代形式重构古代语言。所以你可能认为，例如，人们会觉得很容易理解的例子是从现代罗曼语族语言（法语、西班牙语、葡萄牙语、意大利语等）重构像拉丁语这样的语言。当然，当你重构时，你不会得到古典拉丁语。你会得到它们分支的最新点，那会是通俗拉丁语。但你会看看，比如法语 'feu' 或西班牙语 'fuego'，它们看起来很相似。所以你可以开始拼凑出祖先形式可能是什么。在那种情况下，我们大致知道答案。我们对古典拉丁语了解很多，并且从各种有趣的来源获得了关于通俗拉丁语的相当不错的旁证。我们研究了南岛语系。这大约是世界语言的六分之一。我们查看了一堆词的现代形式，并试图重构古代形式可能是什么。这是一种原始语言，叫做原始南岛语，它是由 Blust 手工重构的。我们能够用计算方式做到这一点。我们做的计算通常与语言学手工重构一致，而且我们没有时间机器，所以很难说谁是对的。有趣的是，你可以开始在这棵巨大的语言变化树上做一些事情。比如你可以开始问这样的问题：哪些合并更常见，哪些合并更不常见。一个直觉上感觉很真实的假设是，如果一种语言中有两个不同的音，它们会被合并，那么一堆原本不同的词就会坍缩。所以如果 P 和 B 被合并，突然 'pin' 和 'bin' 就不再是不同的词了。它们都会听起来一样。嗯，这看起来像是一个信息论问题。一对音的功能负荷是指它们的区别区分了多少个词。功能负荷假说认为，一个音的区别所区分的词越多，合并发生的可能性就越小。这在直觉上是合理的。

Yeah. So, okay, I will. The briefest version of this. This is I think a good example of scale and how some things in computational linguistics I think can give different answers when done at scale from not being done at scale. So, a particular thing we did, and this was a few years back, we looked at reconstructing ancient languages from the modern forms. So you might think, for example, maybe an example that people would find very accessible would be reconstructing a language like Latin from the modern Romance languages French, Spanish, Portuguese and Italian and so on. Now of course when you reconstruct that you won't get classical Latin. You will get the latest point at which they branch which would have been vulgar Latin. But you sort of look at you're like okay, you look like French 'feu' or Spanish 'fuego' and these look like similar. So you can start to piece together what the ancestral forms might have been. Well, in that case, we kind of know the answers. We know a lot about classical Latin, and we have decent side evidence about vulgar Latin from a variety of interesting sources. We went and we looked at the Austronesian languages. So this is about a sixth of the world's languages. And we looked at the modern forms of a bunch of words and tried to reconstruct what the ancient forms would have been. This is a proto language called Proto-Austronesian and it had been reconstructed by hand by Blust. And we were able to do this in a computational way. Now the computation that we did generally agreed with linguistic hand reconstructions, and we don't have a time machine so it's very hard to tell who's right. The interesting thing is you can start to do things on this giant tree of language change. Like you can start to ask questions like what mergers are more common and what mergers are less common. And a hypothesis which is one of these things that just feels intuitively true is that if you have two sounds that are different in a language and they would be merged together, a bunch of words that used to be different would be collapsed. So if P and B were to be collapsed, suddenly 'pin' and 'bin' are no longer different words. They're all going to sound the same. Well, that seems like an information theoretic problem. And the functional load of a pair is how many words their distinction is holding apart. The functional load hypothesis states that the more words that are being held apart by a sound distinction, the less likely that merger is to happen. Which intuitively makes sense.

一些最初在少数语言上的实验似乎并不支持这个假设。所以计算语言学中有一个非常有趣的结果，看起来功能负荷假说并不成立。事情就是当它们合并时就合并了。但如果你不是在四种语言上做这个，而是在数百种语言上做，并绘制散点图，你并不知道你的重构是否完美。即使手工做，你也永远不知道。但总体来看，统计数据显示功能负荷假说在这个大规模数据中绝对明显。但在小规模数据中你看不到这个模式。所以我认为这是一个例子，说明大规模数据可以回答那些用手工不容易回答的问题。

And some initial experiments in a small number of languages didn't seem to support this hypothesis. So there was actually a really interesting result in computational linguistics that it looks like the functional load hypothesis isn't true. Things just kind of merge when they merge. But if instead of doing this on four languages, you do this on hundreds of languages and you do the scatter plot, you don't know that your reconstructions are perfect. You never know that even when you do them by hand. But in aggregate the statistics show that the functional load hypothesis seems to be absolutely evident in this data at scale. But you can't see that pattern over a small number. So I think this is an example of places where just data at scale can answer questions that aren't the same questions that were easy to answer by hand.

是的。

Yeah.

我想我见过一些关于内省深度学习模型的有趣工作。比如 Anthropic 发表了一篇很好的论文，展示了不同的东西，但有一点感觉是中文词汇和英文对应词似乎存储在神经网络的同一部分。这跟人类语言学有对话吗？

And I guess there is interesting work that I've seen on introspecting DLMs. Like I think Anthropic did a really nice paper on sort of showing that showing different things but one thing felt like the Chinese words and English equivalents are kind of stored in the same part of the neural network. Does that does that is that in conversation with human linguistics at all?

实际上最接近的是计算神经科学。所以这也是你可以问人脑的问题，比如对于一个双语者，这些表征在哪里？它们在同一个地方吗？实际上我的一个学生 Kathy Chen，也是 Jack Gallant 的联合学生，就做了这个，不是通过内省，而是通过 MRI。你观察大脑的哪些区域被激活。所以这些关于什么合并、什么分离的问题，是我们可以科学地回答关于自然大脑的问题。我们也可以通过非常相似的方法尝试回答关于人工大脑的问题，当然 MRI 作为一种设备，我们受限于物理和大脑结构。但我们可以更精确地对数字大脑进行探测。所以我认为那里有很多有趣的信息。但我认为退一步看很重要，当我们开始这样做，开始对 LLM 做神经科学时，我们本质上是在说我们构建的这个对象已经超越了我们所认为的工程学。我们不再试图通过它的模块化、通过它保证有或没有的行为来理解它。我们试图通过科学的视角来理解它。科学是我们用来拆解过于复杂事物的工具。工程学是我们构建事物的方式。我的意思是，如果我教 CS 101，我们教给人们的最重要的东西是什么？我们构建可靠、可由团队构建并长期维护的复杂软件系统的最重要工具是什么？我们拥有的最重要的技术是模块化。能够说这个大东西由小东西组成，这些小东西遵守一个契约：如果你给我这个输入，我保证给你这个输出，然后我们可以分别处理它们。所以模块化是经典软件工程中开发可靠复杂系统的关键工具。那么导致机器学习最近爆炸性增长的关键工具是什么？是端到端优化。获取数据，有一个 blob，获取奖励信号，然后传播、传播、传播。这两者非常矛盾。所以作为领域，我们必须调和的一件事是如何获得来自经典技术（如模块化）的可靠性，同时利用来自优化的能力。这不是非此即彼。不是说你不能构建一个有保证或有结构的系统，但存在自然的张力。如果你完全倾向于其中一方，你会遇到严重的局限性。如何结合我们拥有的这两项精彩进展，我认为是当前的核心问题之一。

So what's actually closest to is actually closest to some computational neuroscience. So this is a question you can ask of the human brain too, like when you have a bilingual speaker, where are those representations? Are they in the same place? And actually a student of mine, Kathy Chen, who's a joint student with Jack Gallant, did exactly this, not by introspection, but by MRI. You look and see what areas of the brain light up. So these questions about what's merged and what's separate, these are questions we can answer scientifically about natural brains. And we can also try to answer them through very similar broadly speaking methods with artificial brains, except obviously the MRI as a device, we are limited by the physics and the structure of the brain. We can be much more precise about the probes we make into a digital brain. So I think actually there's a lot of interesting information there. But I do think it's important to step back and notice that when we start doing this, when we start doing neuroscience against LLM, what we are essentially saying is this object we have built is transcending what we think about as engineering. We're no longer trying to understand it by its modularity, by the behaviors that it is guaranteed to have or not have. We're trying to understand it as, you know, through the lens of science. Science is what we use to take apart things that are too complicated. Engineering is how we build things up. I mean, if I teach CS 101, what is the most important thing we teach people? What is the most important tool we have for building complex software systems that are reliable and that can be built by teams and maintained over time? The single most important technique we have is modularity. The ability to say this large thing is made up of small things and these small things obey a contract where if you give me this input, I'm guaranteed to give you this output, and then we can work on them separately. So modularity is the key tool for developing complex systems that are reliable in classic software engineering. What is the key tool that has led to the recent explosive growth in machine learning? It's end-to-end optimization. It's take the data, have a blob, take the reward signal, and just propagate, propagate, propagate. And these things are very much at odds. And so one of the things we're going to have to reconcile as a field is how to get the reliability that comes from classical techniques like modularity that we use when we build software against the abilities that have come from optimization. And these things, it's not an either-or. It's not that you cannot build a system that has guarantees or that has structure or anything like that, but there are natural tensions. And if you go all in on either, you get some serious limitations. And figuring out how to combine these two wonderful pieces of progress we have is I think one of the central problems right now.

说得真好，这让我联想到自动驾驶汽车，我们显然在节目中邀请过很多嘉宾，我认为即使只看过去五六年嘉宾的轨迹，也明显有一个强烈的趋势走向端到端优化，尤其是当这些东西上线时，我们看到现在的 Waymo 和 Zoox，我们请过所有这些公司的人来，基本上我认为他们在说，随着他们越来越接近生产，他们做越来越多的端到端和训练。

That was really well said and it connects me to self-driving cars where we've obviously had a ton of guests on the show and I think even just looking at the arc of the guests over the last five or six years that we've been doing it, there's clearly been a strong trend towards end-to-end optimization especially as these things come online and we see Waymos now and Zooxs and you know we've had folks from all these companies come on and essentially I think they're saying that as they get closer and closer to production they do more and more end-to-end and training.

是的。

Yeah.

那么，你怎么看？我本以为你可能需要更多的模块化来为像自动驾驶汽车这样生死攸关的情况强制执行契约。

So, what do you make of that? I would have thought that you might need more modularity to enforce contracts for such a life or death case like a self-driving car.

是的。我的意思是，这是一个很好的问题，我认为这不仅仅局限于汽车。我认为总的来说，挑战在于我们将越来越信任 AI 做出有后果的决定。当我们构建系统时，在如何优化、如何架构之间会有权衡，在可靠性 vs 广度之类的事情之间会有权衡。所以我认为你会看到，那种存在一种架构能达到完美平衡的想法目前似乎并不成立。所以如果说把整个网络喂给 LLM 并希望可靠性涌现，那么智能的不同方面以不同速度发展，可靠性是慢的那个。所以如果这对你来说最重要，仅仅沿着这条路走下去可能不是最好的方法。而且你还会得到形状有点奇怪的智能。所以有一个流传的事情，某人在 Chipotle 与客户支持机器人交谈，问如何用 Python 反转链表。当然 LLM 能回答，但有趣的是它不应该，对吧？你实际上不想要那种横向性。所以你需要技术的力量与需求相匹配。当谈到自动驾驶汽车时，你确实需要安全，比如你想要的高可靠性位数非常高。所以你看这样的系统，突然它能否为你写十四行诗就不那么重要了，更重要的是它能够保证安全或有非常可靠的安全记录。

Yeah. I mean, it's a great question and I think it's not limited to cars. I think in general the challenge is that we are going to be increasingly trusting AI to make decisions that have consequences. And when we build systems, there are going to be trade-offs between how we optimize, how we architect, there are going to be trade-offs between things like reliability versus things like breadth. And so I think you're going to see this idea that there is sort of one architecture that hits the perfect balance doesn't seem to be true right now. So being able to say that feeding the whole web into an LLM and hoping for reliability to emerge, well the different facets of intelligence advance at different rates and reliability is like the slow one. So if that's the most important thing to you, just continuing down this path alone is probably not the best way to get there. And you also get sort of intelligences that have a bit of a weird shape. So there was this thing going around where somebody's talking to a customer support bot at Chipotle and asked about how to reverse a linked list in Python. And of course the LLM can answer it but interestingly like it shouldn't, right? You actually don't want that horizontality. And so you kind of need technologies where the strengths line up with the needs. And when you're talking about self-driving cars you really need safety, like the number of nines you want is really high. And so you look at a system like this and suddenly it's less important whether or not it can write sonnets for you and more important that it just be able to guarantee safety or have a really reliable safety record.

这里我认为存在一个真正的紧张关系：我们如何降低这些系统的错误率。如果做得不对，就意味着我们实际上再也无法保证任何东西。当然，也有一些关于控制系统的研究，你可以对它们的能力做出某种保证。比如伯克利的 Claire Tomlin 在保证无碰撞和飞行包线等方面的工作。我认为，作为一个挑战，建造一辆无论是否由 AI 控制、你能证明它不会撞车的汽车，会非常棒。这可能需要多种技术才能实现，但这正是我们应该努力的方向。

And here is a place where I think there is a real tension between how we drive down error rates in those systems. If not done correctly, it means we actually no longer have the ability to guarantee anything. And of course, there is work out there on being able to control systems where you can sort of make a guarantee about what they can do. So like at Berkeley, Claire Tomlin's work on being able to guarantee non-collisions and flight envelopes and things like that. And I think it would be great as a challenge, you know, to build a car, AI-controlled or not, where you can prove that you cannot crash it. Right? This is the kind of thing where one simple technique is not going to maybe get you there, but this is what we should be striving for.

有意思。不过我能想象这也有缺点。完全不能撞车这个约束可能太强了。

Interesting. I can imagine that comes with downsides though. To literally not be able to crash it might be too strong of a constraint.

嗯，这变成了一个社会性问题：你想要保证的边界是什么？我认为你指出这一点是对的，这确实可以很微妙。但我认为，能够讨论我们想要提供什么样的保证，这样的未来远比我们无法保证任何东西要好得多。

Well, as it becomes a sort of social question, what is the envelope you want to guarantee? And I think you're right to point out that can be nuanced. But I think the ability to have a discussion about what guarantees we would like to be able to provide is a much better future than one where we can't guarantee anything.

这大概是个很好的结束点。

That's probably a strong place to end.

好的。

Okay.

但我能再问你几个问题吗？如果我们能回到语言学家的角色？有趣。我一直在犹豫要不要提这个，但这对我来说挺有意思的。大概是因为我在句法分析上花了太多时间。显然，语言在某种程度上存在句法分析，这是真实的，或者说感觉是真实的。我想我是在内省。也许那是……

But could I ask you a few more questions if we could put back in the linguist? Fun. I just, I was debating whether or not to bring this up, but it's kind of interesting to me. I guess just because I spent so much time on syntactic parsing. Clearly, language has some sense of syntactic parsing that's real or feels real. I guess I'm introspecting. Maybe that's...

你在内省。抱歉，我也在内省。

You're introspecting. I'm sorry. I'm introspecting.

不，不，我不是那个意思。当我内省自己的大脑时，这种感觉非常强烈。这个观念是否以某种方式出现在语言模型中？因为它们也在建模语言，而这些语言显然来自人类，而人类在我们发明的语言中喜欢做某种句法分析。

No, no, but that's, I don't mean that in a bad way. It feels very present to me when I introspect my own brain. Does that notion somehow show up in LMs as they also model language that clearly comes from humans that like to do a certain sense of syntax in the languages that we invent?

是的，我认为这是一个非常好的问题。那么，让我们从已知的开始。我们最好先定义一下句法分析，因为我感觉有点跑偏了。

Yeah, I think it's a really good question. So, let's start with what we know. We know that we better sort of define syntactic parsing because I feel like I'm going a little wonky.

当然。嗯，我不知道这会不会出现在播客里，但如果有听众，他们可能——取决于他们小学毕业多久了——可能画过句子图解，把句子拆成主语、宾语，这个形容词修饰那个名词等等。那个描述词语之间层级关系的过程，基本上就是句法分析的本质。显然句法分析是存在的，对吧？你在谈论自然语言的句法分析。我们一直都在对编程语言做同样的事。完全正确。编程语言被设计成具有不同的歧义特性。它们通常被设计成无歧义的，而人类语言则充满歧义。所以一个句法分析器会接受一个句子，比如英语句子，然后生成一个表示词与词之间关系的结构，这种关系是层级性的，这很重要，因为例如你可能想要独立翻译子结构，作为分解问题的一种方式。如果没有这种结构，你基本上只能从左到右处理。而现代 LLM 大体上是从左到右工作的。我们知道的是，从科学上讲，这些层级结构似乎确实存在。显然你可以找到例外，比如荷兰语或瑞士德语中的交叉序列依赖。你也可以找到一些很难完全确定层级结构的例子，当然这正是句法学家们争论的地方。但似乎确实存在这些有规律的结构，结构之中还有结构。这在现象学上似乎是真实的。问题是：我们需要把它的工作原理编程进去吗？嗯，从 90 年代起我们就没有这样做了。我们有数据教会了分析器如何工作。而最近，也许你甚至不需要分析器。也许你只需要观察足够多的语言，任何规律都会潜在地出现在语言模型中。现在，如果你和句法学家讨论，他们会提出的论点是：语言理论中的许多概念，包括那些对计算机科学、形式语言理论、自动机理论产生重大影响的概念，语言通常被描述为上下文无关系统，这意味着下推自动机，也就是说原则上你可以无限嵌套，而任何有限深度的系统，比如 LLM，都会耗尽深度。但事实证明情况更复杂。首先，当结构变得复杂时，人们也会感到困惑。

Sure. Well, if, uh, I don't know if this is going to be on the podcast or not, but if there are listeners, they may, depending on how long ago they went to elementary school, they may have diagrammed sentences where you break up a sentence into like here's the subject and here's the object and this adjective modifies this noun and so on. And that process of describing how the words relate to each other in a hierarchical structure, that's basically the essence of syntactic parsing. And obviously syntactic parsing is a thing, right? You're talking about syntactic parsing of natural language. We do it to programming languages all the time. Totally. Programming languages are designed to have properties relating to ambiguity that are different. They are designed in general to be non-ambiguous, whereas human language is rampantly ambiguous. And so a syntactic parser would take a sentence, say in English, and basically produce a representation that showed the relationship between the words, and that relationship was hierarchical, which was important because, for example, you might want to do things like translate the substructures independently as a way of decomposing the problem. In the absence of that, you're basically going to be working left to right. And modern LLMs by and large work left to right. And the thing we know is that, scientifically, these hierarchical structures seem to exist. Obviously you can find exceptions, cross-serial dependencies in languages like Dutch or Swiss German. And you can find cases where it's hard to totally nail down exactly what the hierarchy is, and of course this is what syntacticians can debate. But there definitely seems to be these regular structures that have structures inside the structures. This seems to be real phenomenologically. The question is: do we need to program in how it works? Well, since the '90s we haven't. We've had data that taught the parser how it worked. And recently, maybe you don't even need the parser. Maybe you just observe enough language that whatever regularities there are will just show up latently in the LM. Now, the argument that would have been made if you talked to a syntactician: a lot of the constructs in language theory, including ones that have had a big impact in computer science, formal language theory, automata theory, language is generally described as a context-free system, which means a pushdown automaton, which means in principle you can nest and nest and nest, and any finite depth system like an LLM is going to run out of depth. But it just turns out it's more complicated than that. For one, people do get confused when the structures get complicated.

也许我们有栈深度。即使它是层级性的，我们也有处理限制，这些限制可以表现为栈深度。人们觉得容易理解的东西，比如尾递归，当然可以转化为迭代。所以也许这以某种方式避免了那些限制。

Maybe we've got stack depths. Even if it's hierarchical, we've got processing constraints that can manifest as stack depth. Things that people find easy to understand is like tail recursion, which of course you can transform into iteration. So maybe that somehow avoids those constraints.

所以，还有很长的路要走，但可以说层级句法现象似乎是真实的。语言模型似乎很擅长捕捉这些相关性。但这是一个很好的例子，说明拥有结构化表示可能会带来好处。它可能让你更快地学习语言，处理更棘手的情况，并在数据量较小的情况下更好地泛化。所以现在我们又回到了原点：现象是真实的，表示相对简洁。我们需要这种表示吗？在 AI 中，过去的答案是没有表示就无法取得进展。而今天的答案是，有了足够通用的东西，就不需要了。Transformer 并没有什么神奇之处。其他选择也能工作。它最终只是一个可参数化的电路，一个通用电路。你给它一些数据，它就会归纳出表示。现在问题变成了经验性的：它足够好吗？还是需要分析器来帮忙？对于人类语言的许多方面，答案似乎是通常不需要分析器。

So, there's a long way to go about saying hierarchical syntactic phenomena seem to be real. LMs seem to be pretty good at picking up on those correlations. But this is a good example of a case where actually having a structured representation might give you mileage. It might let you learn language faster. It might let you handle trickier cases. And it might let you generalize better to smaller data regimes. So this is really now we're back to the phenomenon is real. The representation is relatively parsimonious. Do we need the representation? And the answer in AI used to be you can't make progress without the representation. And the answer today is with a sufficiently general purpose thing. So there's nothing really magic about transformers. There are other choices that would work. It's ultimately just a parameterizable circuit, a general purpose circuit. You point it at some data and it induces a representation. And the question is now empirical: is it good enough or do you need the parser to help? And for many aspects of human language, the answer seems to be you often don't need the parser.

但我想另一个问题是：Transformer 是否以某种方式在其权重和偏置中重新发明了句法？

But I guess another question is: are the transformers reinventing syntax somehow within their weights and biases?

首先你得弄清楚那意味着什么。

First you'd have to figure out what that means.

所以我认为问题可能是：是否存在某种与移位归约解析器同构的子结构？

So I think the question would be something like: is there a substructure which is somehow isomorphic to a shift-reduce parser or something like that?

是的，我知道有人在做这个研究。这个问题之所以很难回答，是因为最终这些深层结构问题都表现为表面相关性。这两个词高度相关，但这两个词却不相关，尽管它们距离更近。我们将其归因于句法边界、嵌套或信息隐藏。但实际情况就是相关性要么朝这个方向，要么朝那个方向。像 Transformer 这样的通用系统学习的就是相关性。所以一旦系统能够足够准确地表现出这些相关性，就很难判断它是否通过正确的机制实现了等价。

Yeah, and I mean I know there are people working on this. There is a reason why this is a very hard question to ask, because ultimately these deep structural questions manifest as surface correlations. These two words are highly correlated, but these two words are not, even though they're closer. And we attribute that to a syntactic boundary or a nesting or an information hiding. But it simply is the case that the correlations go one way or the other way. And a general purpose system like a transformer learns correlations. So once the system can manifest those correlations correctly enough, it's hard to know whether it's doing it through the right mechanism to be able to say that there's an equivalence.

这里有一个有趣的教训——我这么说可能会惹恼很多音系学家和语音学家。但作为语言学中的一种粗略描述，音系学家和语音学家对于我们所听到的所有声音，以及我们在习得语言时所有流过我们的语言，往往有不同的看法。在音系学中，一个关键概念是认知经济性：大脑倾向于产生最小、最简洁的表征，学习和抽象都围绕着所谓的最小描述经济性。另一方面，语音学家更倾向于认为所有这些数据都以一种弥散和分布式的方式产生影响。所以当你听到大量语言时，你保留了什么呢？答案可能一端是全部保留——就像你听过的一切都有波形文件一样，对吧？这是一个相当强的说法，但那是极端之一。另一个极端是什么都没保留：你只保留了那个美丽过程的非常抽象的表征。而在我看来，答案几乎肯定在中间的某个位置。有启动实验展示了非常有趣的现象：如果你要对一个以前听过的无意义词做出反应，如果播放的是你之前听过的完全相同的录音，你的反应会更快。这告诉我们，这不仅仅是完全抽象。但我们也知道人们非常擅长进行抽象和泛化，以及在学习语言时（包括儿童习得母语时）如何泛化和过度泛化规则。所以似乎大脑同时保留了这些非常低级的信息片段和这些抽象。这使得我们很难去观察一个模仿行为的复杂电路，然后问：这和大脑中发生的事情是一样的吗？因为第一，我们并不确切知道大脑中发生了什么。第二，我们如何衡量呢？我们真正能衡量的只是：它是否捕捉到了这种相关性？当我们这样做时，答案是，当然，它捕捉到了相关性。这就是全部目的。

There's an interesting lesson — I'll probably make a bunch of phonologists and phoneticians angry at me by saying this. But as a caricature in linguistics, there is often a difference between what a phonologist and a phonetician will say about all the sound that we are constantly hearing, and as we acquire languages, just all the language that's flowing over us. In phonology, one of the key concepts is cognitive economy: the brain is driven to come up with minimal, parsimonious representations, and the learning and abstraction are all about what you might call a notion of economy of minimal description. On the other hand, phoneticians are much more likely to say all of this data impacts in a very diffuse and distributed way. And so this question of when you hear lots of language, what do you retain? The answer could be on one end, all of it — like there are wave files of everything you've ever heard, right? That's a pretty strong statement, but that's one extreme. The other extreme is none of it: you just retain this very abstracted representation of the beautiful process. And the answer seems to me to be almost surely in the middle somewhere. There are priming experiments that show really interesting things: if you're going to react to a nonsense word that you've heard before, you react faster if they play you the exact same recording as the one you heard it before. That tells you something about it being more than just the full abstraction. But we also know people are very capable of making abstractions and generalizing, and how people will generalize and overgeneralize rules when they're learning a language, including their own language when kids are acquiring language. And so it seems to be simultaneously true that the brain keeps these very low-level pieces of information around and also these abstractions. That makes it very hard to go to a complicated circuit that's mimicking the behavior and say, is this the same as what's going on in the brain? Because one, we don't know what's going on in the brain exactly. And two, how would we even measure that? All we can really measure is: did it capture this correlation? And when we do that, the answer is yeah, of course, it captured the correlation. That is the whole purpose.

嗯，我觉得语言学中有很多巧妙的技巧可以用来研究这些东西。比如观察不流利现象，或者我经常想到我妻子记错歌词的方式是语义性的——歌词在语义上相似——而我的大脑记错歌词的方式则更偏向语音相似。所以显然我们的大脑在编码音乐的方式上略有不同。我觉得你可以对机器也做类似的事情。

Well, I feel like there's all these clever tricks in linguistics to sort of get at this stuff. And you know, like looking at disfluencies, or I always think about how my wife misremembers songs in a semantic way — the lyrics are semantically similar — whereas my brain will misremember lyrics in a way that's more phonetically similar. And so clearly our brains are slightly different in how we encode music. I feel like you could run some of these same things on a machine.

当然，而且它们肯定不会像研究生那样感到无聊。

Sure, and certainly they wouldn't get bored like grad students doing it.

那么，我们如何知道人类拥有这些不同类型的语言知识呢？人们提出的论证之一——你提到了不流利现象——有一整套列表，我无法在短时间内充分说明。人们为语言结构的真实性提出了很多论证。你可以问：这种真实是统计上的真实吗？它是解释性的，还是像神经真实一样？它就在那里，是硬连线的。那是另一个问题。但就其解释性而言，你提到了像语义性记错与语音性或音系性记错。人们研究换位现象。音位的一个经典论证是像首音互换这样的现象，其中你换位音位、音节或单词。如果一个对象可用于语言操作，那就证明了它的连贯性。例如，句法结构的一个重要论证是：我能否用这个语块来回答问题？所以如果你说“猫在桌子下面睡觉”，我必须向你证明“在桌子下面”是一个单位。一种方法是问：猫在哪里睡觉？你回答“在桌子下面”。如果它可以作为答案，如果你在模式匹配这些句子时可以用“什么”这个词来替换它，无论它是如何工作的，那就证明了它的真实性。另一种论证方式是语义上的：“在桌子下面”是一个地点。而如果我问你关于“在桌子下”的问题，很难想出一个答案就是“在桌子下”的问题。如果你说，“在桌子下”是什么意思？指向“在桌子下”。我们做不到。所以语言学家为语言的模块性提出了这些论证。这些论证实际上并不是关于我们的大脑的。它们是关于语言的现象学。

Well, how do you know in a human that we have these different kinds of linguistic knowledge? One of the argumentations that people have — you mentioned things like disfluency — there's a whole list, and there's no way I can do it justice in a short amount of time. There's a whole list of arguments people have made for the reality of linguistic structure. You can say, is that reality a statistical reality? It's explanatory, or is it like a neural reality? It's in there and wired. That's a separate question. But in terms of it being explanatory, you know, you mentioned misremembering things like semantically versus phonetically or phonologically. People look at transpositions. One of the classic arguments for phonemes is that things like spoonerisms where you transpose phonemes or syllables or words. And if an object is available for a linguistic operation, that argues for its coherence. So for example, one of the big arguments for syntactic structure is: can I answer a question with this chunk? So if you say "the cat is sleeping under the table" and I have to argue to you that "under the table" is a unit. One way to do that is to say: well, where's the cat sleeping? And you say "under the table." Well, if it's available as an answer, if it can be replaced by the word "what" when you pattern match those sentences, however it is that works, then that argues for its reality. Another way you can argue is on semantic grounds: "under the table" is a place. Whereas if I ask you about "underthe," it's hard to come up with a question whose answer is "underthe." If you say, well, what does "underthe" mean? Point to "underthe." We can't do it. And so linguists have developed these arguments for the modularity of language. Those arguments are not about our brain really. They're phenomenological about the language.

现在当你接触到神经科学家时，你可以开始谈论大脑。这涉及到监测、探测和设计实验，这些实验偶然揭示了结构和组织，而且我们完全可以在机器上做同样的事情。

When you get to neuroscientists now, you can start to say things about the brain. That's about monitoring and probing and designing experiments that reveal incidentally the structure and organization, and that we can absolutely do on a machine as well.

酷。好吧。我希望我们保留这段。我觉得很有趣。

Cool. All right. Well, I hope we keep that in. That was fun for me.

我也觉得很有趣。

It's fun for me, too.

我很感激。是的，别担心。

I appreciate it. Yeah, no worries.

非常感谢收听本期 Gradient Descent。请继续关注未来的节目。

Thanks so much for listening to this episode of Gradient Descent. Please stay tuned for future episodes.