Percy Liang 谈基础模型、共享现实与 NLP 的演变

我确实认为我们生活在一个共享的世界里，如果每个人都拥有自己定制的模型，那实际上就是一个他们生活在其中的小虚拟世界，这基本上就是导致两极分化的原因。我认为这是一个我们需要对抗的问题。如果你考虑未来的这些语言模型，我认为我们与世界互动、获取信息以及在世界上采取行动的主要方式，很可能会通过这些模型来中介。所以这些模型必须与现实挂钩，而不仅仅是基于某种赚钱的广告方案，让人们基本上相信他们想相信的任何东西。我们需要某种共享的现实，哪怕没有别的原因，也因为现实世界要求这样。

I do think that we live in a shared world and if everyone has their own customized model which really is a little virtual world that they live in, that's basically how you get polarization. And I think that is a problem that we want to fight. If you think about each of these language models in the future, I think a primary way that we'll interact with the world and get information and also take actions in the world is probably going to be mediated by these models. So that better be tethered to reality, not just based on some moneymaking ad scheme that gets people to basically believe whatever they want. And there needs to be some sort of shared reality, if nothing else, because the real world demands it.

大家好，我是主持人 K Jun，我们是一个独立研究实验室，致力于开发模仿人类智能基础并能学习安全解决现实世界问题的 AI 智能体。在我们的播客中，我们采访研究人员，了解他们在论文和演讲中难以分享的幕后想法、观点和直觉。我们希望你能像我们在理解和构建心智的探索中一样学到很多。Percy Liang 是斯坦福大学计算机科学和统计学副教授。如今，他感兴趣的是理解基础模型如何工作，如何让它们更高效、模块化和鲁棒，以及它们如何改变人们与 AI 互动的方式。尽管他在基础模型出现之前就已经研究语言模型很长时间了，Percy 也是可重复性研究的坚定支持者，为此，他使用他的实验室开发的 Coda Lab 工作表平台将大部分研究论文作为可执行论文发布，并且他还发布了各种基准测试。非常感谢你来做客播客，我们真的很高兴邀请到你。你的职业生涯很长，嗯，实际上你的职业生涯很短——这都相对而言。中期职业，我想这才是合适的说法。中期职业。但你的职业生涯非常密集，涉及许多不同领域的有趣工作。所以我们总是从这个问题开始：你最初是如何发展你的研究兴趣的，也许是在读博之前和读博期间，然后随着时间的推移，这些兴趣是如何发展的？

Hey there, I'm your host K Jun and we are an independent research lab developing AI agents that mirror the fundamentals of humanlike intelligence and that can learn to safely solve problems in the real world. On our podcast, we interview researchers about their behind-the-scenes ideas, opinions, and intuitions that are hard to share in papers and talks. We hope you learn as much as we have in our quest to understand and build the mind. Percy Liang is an associate professor of computer science and statistics at Stanford. These days he's interested in understanding how foundation models work, how to make them more efficient, modular, and robust, and how they shift the way people interact with AI. Although he's been working on language models for long before foundation models appeared, Percy is also a big proponent of reproducible research and toward that end he shipped most of his research papers as executable papers using the Coda Lab worksheets platform that his lab developed, and he's also published a wide variety of benchmarks. Thank you so much for being on the podcast, we're really excited to have you. You've had a long career, well actually you've had a short career that has— it's all relative. Mid-career, I think that's the proper name for it. Mid-career. But it's been very, very densely packed with a lot of really interesting work across a variety of different things. So we always start with: how did you initially develop your research interests, maybe even going into your PhD and during your PhD, and then over time how has that developed?

也许我应该从读博之前开始说起。很久以前，本世纪初，也就是 2000 年代初期，我对 AI 感兴趣，但当时的 AI 非常不同。特别是，那时人们并不怎么使用这个词，因为正处于 AI 寒冬之一，如果你说你在做 AI，别人会投来奇怪的目光。所以当时说机器学习比较有趣。我记得第一个研究项目，可能和这次对话更相关，是关于 NLP 的。那时，我们的想法是构建模型来做词性标注或命名实体识别。现在这些被认为是无聊的东西，但当时真的很令人兴奋。其中一个关键挑战是缺乏数据，即标注数据。所以你知道，如果我们在大量原始文本上学习模型，我们可以开发出更好的特征来帮助分类。所以我做的第一个项目是训练一个我认为是语言模型的东西。具体来说，它是一个具有特定结构类型的隐马尔可夫模型，并在尽可能多的单词上训练它。它的酷之处在于，它确实提高了下游任务的准确性。而且，如果你查看语言模型内部的聚类，实际上非常有趣：它会将所有的城市名称和人名等分开。所以这可以说是第一次尝到‘嘿，这真的很涌现’的滋味，很有趣。当然，这种思路正是让我们走到今天的原因。那是 2005 年。然后我去了研究生院，我的兴趣有些分裂。我对机器学习理论非常感兴趣，同时也对无监督学习感兴趣，现在更多地被称为自监督学习。这个想法是，你可以利用原始文本，学习不同类型的结构，特别是句法树结构，以及它们如何涌现。我写了很多论文，有很多有趣的想法。我认为要在实际问题上站稳脚跟有点挑战性，因为这有点像读心游戏：你设计一个模型，在数据上训练，然后只是希望这些宾州树库结构会神奇地出现。但并没有理由一定会这样。所以我转向了。在博士快结束时，我转向了语义解析，这仍然满足了我从数据中学习语言结构的想法，这是我非常兴奋的，但有了实际应用。场景是问答：你问一个问题，比如‘加利福尼亚州最高的山是什么？’，然后给出答案，它需要找出一个数据库查询，即 SQL 查询，自然语言映射到它从而产生答案。然后你可以端到端地学习，并得到一个相当有趣的系统。所以那时我对构建实际应用非常兴奋。毕业后，我在谷歌待了一年，构建了语义解析器，这些解析器后来被用于支持 Google Now 或助手之类的东西。然后我开始在斯坦福工作。我早期很多工作都是关于语义解析：如何超越词性标注或表面线索，对自然语言进行更深层次的理解？自然语言句子组合映射到具有执行语义的逻辑形式，这个范式我沉浸了很长时间。然后我认为主要瓶颈之一是，虽然原则上这个范式工作得很好，你可以回答非常复杂的问题，甚至可能对今天的模型来说都很难，但互联网上的大部分知识并不是结构化形式。所以这里有一个急剧下降，很多知识这类方法无法获取。于是我开始思考更一般的问答和阅读理解。我们在 2016 年左右创建了 SQuAD 数据集，作为开拓这个领域的一种方式。然后开始转向思考评估。

Maybe I should start before my PhD actually. A long time ago, early still this millennium but early 2000s, I was interested in AI but AI was very different. In particular, people didn't really use the word back then because it was during one of these AI Winters where if you said you did AI, people gave you weird looks. So it was interesting machine learning to say. And I remember the first research project that maybe is more relevant to this conversation was on NLP. And back then, the idea was that we wanted to build these models to do part-of-speech tagging or named entity recognition. Right now that's considered boring stuff, but back then it was really exciting. And one of the key challenges was lack of data, labeled data that is. So you know that if we learn models on a huge amount of raw text, we could develop better features that could help with classification. So the first project I did was train what I would consider a language model. So in particular, it was a hidden Markov model with a certain type of structure and trained it on as many words as I could get a handle on. And what was cool about it is that it did improve accuracy on downstream tasks. And also, if you look at the clusters inside the language model, it was actually really interesting: it would separate all the city names and the people names and so on. So it was kind of the first taste of 'hey, this is really pretty emergent', let's say, and interesting. And of course that line of thinking has been the thing that has gotten us to where we are. That was 2005. And I went to grad school and I had a sort of a split interest. I was very interested in machine learning theory and I was also interested in unsupervised learning, nowadays called more self-supervised learning. The idea that you could take raw text and you could learn different types of structures, in particular syntactic tree structures, and how that would emerge. And I wrote a lot of papers and had a lot of interesting ideas. I think it proved a little bit challenging to get a hold of a foothold on the actual problem because it would be this sort of mind-reading game where you would design a model, you would train on data, and you would just hope that magically these Penn Treebank structures would come out. And there's no reason why that would have to be the case. So I pivoted. I think there was a period towards the end of my PhD where I pivoted to semantic parsing, which still satisfied the idea of learning language structures from data, which is something that I had been really excited about, but with a real application. So the setting was question answering: you ask a question like 'What's the tallest mountain in California?' and you would give it an answer, and then it would have to figure out essentially a database query, the SQL query, that the natural language maps to that would produce the answer. And then you can learn this end-to-end and actually get a pretty interesting system out of it. So I was very at that time excited about actually building real applications. After I graduated, I spent a year at Google where I built semantic parsers that then got used to power the Google Now or assistant things like that. And then I started at Stanford. A lot of my earlier work was on semantic parsing: how do you have deeper understanding of natural language beyond just part-of-speech tagging or superficial clues? And this idea of natural language sentence mapping compositionally to a logical form with execution semantics, that was the paradigm that I lived in for a very long time. Then I think one of the main bottlenecks was that while in principle this paradigm worked beautifully and you could answer really, really complicated questions that probably would even be difficult for models today, most of the knowledge on the internet is not in structured form. So that's where there was a sharp drop-off, where much of the knowledge was inaccessible to this class of methods. So then I started thinking about more just general question answering and reading comprehension. We created the SQuAD dataset around this time, 2016, as a way to kind of trailblaze into that area. And then it started shifting to thinking about evaluation.

那时，我花了很多时间思考机器学习的鲁棒性，因为有人怀疑深度学习方法在基准测试上表现很好，但在实际应用中就会崩溃，对吧？确实如此，无论是视觉还是语言领域的对抗样本，这些在排行榜上表现优异的系统一旦遇到域外情况就会失效。所以我研究了一段时间。

And at that time, I think I was spending a lot of time thinking about robustness of machine learning because there was a suspicion that deep learning methods were able to do really well on these benchmarks, but if you actually use them in real life, they would just fall apart, right? And this was true, you know, with adversarial examples both in vision but also in language. It seemed like these really high-performing systems that topped these leaderboards would actually just fall apart when they went out of domain. So I did that for a while.

然后，基础模型出现了。GPT-3 发布时，它让我大吃一惊：仅通过下一个词预测训练的语言模型，就能实现远超我想象的能力，比如零样本上下文学习，所有这些能力就这样涌现出来。这让我意识到，这是一次范式转变。那时我心想，我可以继续用各种方式破坏这些系统，但这不是重点。重点应该是真正理解这些系统，将它们用于应用，并理解其社会影响。

And then, you know, foundation models happened. GPT-3 came out and it sort of just blew my socks off in terms of the idea that you could train a language model just on next-token prediction, and you could get a model that did way more than I could imagine, just zero-shot in-context learning and all these capabilities just emerged. It really suggested to me that there was a paradigm shift. And then I think at that point, I sort of said, you know what, I could go on and break these systems in all sorts of different ways, but I think that's not where the action is. I think the action is really trying to understand these systems, harness them for applications, and understand the social impact.

所以在 2020 年左右，我又开始更多地思考这项技术的社会影响，这很及时，因为那时这项技术确实开始产生社会影响。也许值得讨论一下。于是我在 2021 年创立了基础模型研究中心，并做了更多评估工作。我可能涉猎了太多不同的事情，但我很乐意谈论其中任何一件。不过，这算是对一个简短问题的冗长回答。

So actually around that time, 2020, I again started thinking a lot more about the societal impact of this technology, which is timely because that's when this technology was really having a societal impact. Kind of worth maybe talking about. So I founded the Center for Research on Foundation Models in 2021, and then did a lot more evaluation work. I mean, I guess I just work on probably too many different things, but I'm happy to talk about any of that. But that's a long answer to a short question.

太棒了。我好奇的是，既然你从 2005 年就开始思考这些问题，那么有些旧想法呢？比如，你仍然对句法树结构感兴趣，或者仍然对语义解析感兴趣，或者这些更结构化的自然语言处理方法？还是说，你觉得这个新范式也改变了这一点，比如用这种更结构化的方式思考语言其实没有意义？

That's awesome. One thing I'm curious is, since you've been thinking about this stuff since 2005, there are some old ideas. Like, are you still interested in syntactic tree structures, or still interested in semantic parsing, or these more structured ways of dealing with natural language? Or do you feel like actually this new paradigm changes that as well, like it doesn't actually make sense to think about language in this more structured way?

是的，我认为这里有一些细微差别。总的来说，我认为很多纯粹的内部结构是不需要的。例如，句法分析，我觉得它们已经没那么有用了。即使在我做语义解析的时候，句法也是一个有趣的 formalism，它捕捉了语言的一个方面，但当你解决最终任务时，除了句法还有很多其他重要的东西。所以执着于句法结构并没有太大意义。

Yeah, I think there's some nuance here. By and large, I think a lot of the internal structures which are purely internal are not needed. So syntactic parsing, for example, I think I've sort of given up on them being that useful. Even when I was doing semantic parsing, I think that syntax was an interesting formalism that captured one aspect of language, but when you're solving end tasks, there are many other things besides syntax that matter. So fixating on syntactic structures didn't really make sense.

就语义解析而言，如果你考虑 2024 年的重新定义，它其实就是自然语言到代码，对吧？因为它的本质就是：根据自然语言生成可执行的代码。事实上，有一些关于潜在变量的想法，本质上和过去是一样的。例如，你可以把思维链看作是一种通向答案的潜在结构。所以，输入中存在一个你未必关心的潜在变量，但它有助于推导出答案——这基本上就是语义解析的抽象。在这个层面上，我认为许多框架仍然相关。现在大多数时候技术不同了，但有时，我的意思是，仍然有一个想法：如果你有一个潜在变量，并考虑写出边际似然，这仍然是有用的。我确实看到有时一些旧想法因此被重新发现。但你知道，这没问题。

I think in terms of semantic parsing, if you think about the 2024 rebranding, it's just natural language to code, right? Because that's what it was about: generating executable code given natural language that you can then execute. And in fact, there's been some ideas of latent variables that are still basically the same things in disguise from back then. For example, you can think about chain of thought as a sort of latent structure that leads up to the answer. So the general idea of having an input where there's a latent thing that you maybe don't necessarily care about that helps you derive the answer — that is basically the abstraction of semantic parsing. So at that level, I think many of the frameworks are still relevant. Now the techniques are different most of the time, but sometimes, I mean, there's still an idea: if you have a latent variable and you think about writing down the marginal likelihood, that is still a useful thing. And I do see that sometimes, some of the old ideas are rediscovered because of that. But you know, that's fine.

这真的很有趣，有点像旧想法被重新发现，但以新的方式，从新范式的第一性原理出发。那么，你感兴趣的事情现在是如何转变的？那是基础模型的开始，你专注于那个。现在，今年或未来，你真正兴奋的事情是什么？

That's really interesting, kind of like old ideas that get rediscovered but in a new way, from first principles in the new paradigm. Yeah. So how have the things you're interested in shifted towards now? So that was kind of the beginning of foundation models, great focusing on that. Like, what are the things now that you're really excited about this year or kind of going forward?

是的，也许先完成这个转变。我之前思考学习的鲁棒性：如何让模型鲁棒？我也在做语义解析：如何回答复杂的自然语言问题？而基础模型实际上同时回答了这两个问题。因为处理鲁棒性的方法就是训练足够多的数据来覆盖——不再有域外情况，或者通过尽可能多地训练数据来假装它不会发生，这非常有效。我们有一些论文表明，与许多以前的算法方法相比，拥有一个好的基础模型是首要的。而在语义解析方面，它完全改变了你所关注的方法和数据集类型。所以我非常认同。

Yeah, so maybe just actually complete the transition. So I thought about robustness of learning: how do we make the models robust? And I was working on semantic parsing: how do you answer complex natural language questions? And foundation models actually basically are the answer to both of them, in the sense that if the way you deal with robustness is that you just train on enough data that you sort of cover — there's no more out-of-domain anymore, or you try to pretend that it doesn't happen by training on as much data as you can, and it's very effective. We had some papers showing that compared to many of the previous methods which were algorithmic, just having a good foundation model was the first-order bit. And in terms of semantic parsing, it completely changes the game of what type of methods and datasets you even look at. So I was pretty bought in.

而且我认为，在我职业生涯的大部分时间里，思考如何让模型更好、更可靠——当然我们还没做到——但实际情况是，很多这类工作已经转移，至少重心已经转移到主要的基础模型开发者，比如 OpenAI 或 Google。所以，虽然我们显然可以发表论文，并创造新的有用想法——例如，今年我们有了新的优化器和数据选择方法——但从宏观影响来看，我倾向于更多关注那些指向事物可能不同结构的研究问题。

And I think, you know, going forward, for so much of my career, thinking about how to make models better and more reliable — and of course we're not there yet — but what has happened is that a lot of that work has shifted, at least the center of gravity has shifted towards major foundation model developers like OpenAI or Google. And so while we can obviously publish papers and still create new ideas that are then useful — for example, this year we had new optimizers and new ways of doing data selection — I think in terms of big picture impact, I think I'm gravitating more towards research problems that really point at a different structure about the way that things could be.

我认为学术界的一个特殊之处在于，我们能够不担心当前如何构建最好的系统、如何将其推向市场？而在某种意义上，这很大程度上是工业界做的事情。但我认为这或许不应该是未来的重点，而是思考社会如何构建——用最宏大的术语来说。所以，我会举一些例子来说明我的意思。

And one of the things that I think is special to academia is that we have the ability to not worry about, currently, okay, how do we build the best system, how do we get it out there? Whereas, you know, in some sense that is a lot of what industry does. But I think that isn't really, maybe should be the focus going forward, but thinking about how — I don't know — society could be structured, to put it in the most grand terms possible. So I'll give you some examples of what I mean by that to round it out.

也许从我们已经完成的具体工作开始，然后转向我们尚未完成但想做的事情。我们做了 HELM，即语言模型评估，我们获取了所有能拿到的模型，策划了一大套评估，然后以标准方式进行了评估。我认为这非常有影响力。而我没有完全意识到的一点是，由一个中立的第三方进行评估——尽管谷歌、OpenAI 或其他任何人都可以做——有其独特的贡献，即中立性和合法性是有价值的。这是一件事。

Maybe start with the concrete work that we have done, and then moving towards work that we haven't done but I'd like to do. We had HELM, the evaluation of language models, where we took all the models that are out there that we could get a hold of, we curated a large set of evaluations, and then we just did it in a standard way. This I think has been really influential. And the one thing I didn't maybe fully appreciate is the extent to which having a neutral third party that does the evaluations — even though Google or OpenAI or anyone could have done this — has a certain contribution, which is that the neutrality and the legitimacy of that is worth something. That is one thing.

我们做的另一个项目叫做基础模型透明度指数，这是评估……

Another project that we worked on is called the Foundation Models Transparency Index, and this is evaluating...

不是模型，而是组织。所以我们一直在思考负责任的人工智能和透明度问题，并感叹当前世界透明度不足。但我们没有只是抱怨，而是决定更科学地衡量这一点。我们制定了 100 个不同的指标，涵盖：人们是否谈论他们的数据、是否谈论他们的算力、劳动实践、是否进行适当的评估，一直到考虑他们的使用政策等等。这是一个非常全面的列表。我们基于这套指标评估了 10 个组织。这又是只能在主要实验室之外完成的事情，因为它具有我们拥有的中立性和视角。我认为从结构上讲，这是不可替代的。所以这些是我一直在思考的例子，这些不能在主要实验室完成，也不需要那么多 GPU。

Not models but organizations. And so we were thinking about responsible AI and transparency and lamenting the fact that there's not that much transparency in the current world. But rather than just complain about it, we decided to be more scientific and measure this. So we developed 100 different indicators spanning: did people talk about their data, talk about their compute, labor practices, do proper evaluations, and all the way to thinking about their usage policies and so on. So it's a very comprehensive list. And we evaluated 10 organizations based on the set. And this is again something that can only be done outside the major labs because it has the neutrality and the perspective that we have. I think it's just by construction not really replaceable. So these are examples which I've been thinking about which cannot be done at major labs and also doesn't require that many GPUs.

然后，我认为，更长远来看，我真正兴奋的是：关于如何构建基础模型的愿景。目前的现状是，世界上所有写书、写文章、拍照、创作内容的人，他们的内容被爬取成数据集，用于训练基础模型，然后服务于人和产品。这有很多结构性问题。一是内容创作者得不到任何认可或报酬，所以你会看到很多诉讼正在发生。另一个问题是，在决定这些模型行为方面存在大量的集中化，这再次缺乏透明度。所以我们不知道幕后发生了什么。我只是想：我们能否以不同的方式做事？我没有技术答案，只是描绘一个愿景。如果我们能够真正将预测归因于实际的训练来源呢？这实际上是我七年前研究过的东西，但方式更有限。你可以做数据归因，你可以可靠地做到。那么也许你可以建立一个更经济可行的系统，根据人们的贡献支付报酬，这可能会激励更好的数据质量。而且至少不会再有同样的诉讼，因为只要人们得到报酬，我想希望他们会更开心。这是一个方向。另一个方向是思考这些语言模型所体现的价值观，我认为这一点非常重要，不能仅仅掩盖在'我们正在对齐人类价值观，我们很安全'的伞下。因为这是一个如此复杂的结构，尤其是一个组织说'哦别担心，我们在处理'根本不是一个可行的前进方式。那么如何让这个过程更加民主？如何引出一些价值观，或者如何建立一个更具参与性的治理结构，让你获得更好的代表性，使得语言模型的价值观真正反映人们想要的，而不是关起门来的少数人决定的？所以这些是我想要推动的方向。如何让这一切运作起来，有很多技术以及激励政策问题，但这些是我感到兴奋的事情。

Then there is, I think, longer term what I'm really excited about: a vision of how foundation models can be built. So the current status quo is you have all these people in the world who write books, write essays, take pictures, create essentially content, which then gets scraped up into datasets that are used to train foundation models, then served to people and products. And this has many structural problems. One is that the content producers don't get any credit or pay, so that's why you see many lawsuits that are happening. Another problem is that there's a massive amount of centralization in determining these model behaviors, which is again lack of transparency. So we don't know what's happening behind the scenes. And I just wonder: how could we do things differently? So I don't have the technical answer, but just a kind of vision to paint out. So what if we were able to actually attribute the predictions to the actual training source? This is actually something I worked on seven years ago, but in a more limited fashion. You could do data attribution, you could do it reliably. Then maybe you could actually set up a more economically viable system where you pay people for their contributions, and that maybe incentivizes better data quality. And there wouldn't be the same lawsuits, at least because maybe as long as people are getting paid, I think hopefully they will be happier. That's one kind of direction. The other direction is thinking about the values that these language models embody, which is something I think is really important to foreground and not just sweep it under the umbrella of 'we're aligning human values and we're being safe.' Because that is such a complex construct that especially for a single organization to say 'oh don't worry, we're handling it' is just not a viable way forward. So how do you make this process more democratic? How can you elicit some values, or how do you have a governance structure that is more participatory and gets you better representation, so that the values of a language model are actually reflecting what people want, rather than whatever a few set of people behind closed doors decided? So those are the directions I want to push. There are many technical as well as incentive policy questions on how to make this all work, but those are some things I'm excited about.

你提到的一件事我真的很想深入探讨，那就是 HELM，实际上，关于语言模型的评估以及未来可能的样子。我认为你关于这项工作最好在实验室和主要公司之外完成的观点，可能是正确的地方。但我很好奇你对如何稳健地评估语言模型的看法，尤其是针对许多不同的任务。感觉这将是一个巨大的努力，试图理解几乎像标准化测试之类的东西，但针对的是比我们以前必须做的更广泛的能力类别。这感觉将是一个巨大的努力。你如何看待这种发展？你是怎么想的？

One of the things you mentioned that I really wanted to dig into is HELM, actually, in terms of the evaluation of language models and what that can look like going forward. I think that your point about it being work that is probably best done outside of the labs and outside of the major companies is probably the right place for it to be done. But I'm kind of curious to get your thoughts on how do we actually evaluate language models robustly, especially for lots of different tasks. And it feels like it's going to be this huge effort of trying to understand almost like standardized testing or something, but for a much, much wider class of capabilities than we've ever really had to do before. And this feels like it's going to be some huge effort. How do you see that kind of going forward? How are you thinking about it?

是的，是的。我认为标准化测试的类比很有趣，因为我们试图评估人类已经几千年了，但我们还没有真正解决这个问题。是的，我们正在获得足够通用的模型，这种粘性可能实际上——当然语言模型有其他结构，可能让它稍微容易一些，比如你可以让他们参加一个为期一周的考试，而不是在两小时内完成。我会尝试如下分解。我们需要在不同轴线上取得进展。一个轴线是评估的现实性。语言模型开始在企业应用和消费者应用中的许多不同地方被使用，社会影响，无论是准确性、益处还是风险，都应该根据部署环境来具体化。另一方面，评估通常围绕这些非常程式化的学术基准，这与实际用例相去甚远。所以一个方向是投资于更现实的用例。例如，如果它们被用于联络中心，或者用于从 PDF 中提取信息等，你如何真正在基准中捕捉到这一点？这是一个轴线。另一个轴线是开放生成的概念本身就很难。对吧？如果你有一个提示'给我写一篇关于第二次世界大战的文章'，你得到一篇文章，给那篇文章打分是你的英语老师会花相当多时间阅读并给出评论的事情。这与回答多项选择题完全不同。我们就是不知道如何做到这一点。还有一个问题是人们说'哦，就做人类评估吧。'这在某种意义上根本不是答案。首先，哪些人类？如果问题涉及关于，比如说，医疗数据的深层知识，你不能随便抓一个众包工人让他们做。所以需要专家。但即使你有专家，如何扩展？你做两两比较吗？你做某种艺术评分吗？你有反馈吗？所有这些关于如何确定评估应该是什么样子的问题都非常开放。然后我想评估的另一个方面是考虑安全性。所以我一直与 ML Commons 合作开发安全评估，这只是一个开放性问题：如何衡量偏见？再次，我认为我的倾向一直是将其置于实际用例中，因为那样你可以更有意义地谈论任何益处或风险。但当没有一个真正明确的正确答案时，这很难定义。你知道，一线希望是，评估现在，我认为鉴于这些基础模型，已经以某种方式成为了一等公民，而以前不是。以前总是'你训练一个模型，你评估。'你永远不会不评估——

Yeah, yeah. I think the standardized testing analogy is interesting because we've been trying to evaluate humans for a millennium and we haven't really figured that one out. Yeah, we're getting general-purpose enough models where that type of sticking maybe actually—of course the language models have other structures that maybe make it slightly easier, like you can actually subject them to a week-long exam as opposed to having done in two hours or something. I would try to break it up as follows. So we need progress along different axes. One axis is the realism of the evaluation. Language models are starting to be used in many different places in enterprises and consumer apps, and the societal impact, both in terms of accuracy, the benefits, and also the risks, really should be contextualized to where they're deployed. On the other hand, evaluation typically centers around these very stylized academic benchmarks, which is quite far removed from the actual use cases. So one direction is actually to invest in much more realistic use cases. For example, if they're used in a contact center or they're used to do information extraction from PDFs or something, how can you really capture that in the benchmark? So that's one axis. The other axis is the idea of open-ended generation is just hard. Right? If you have a prompt 'generate me an essay about World War II' and you get an essay, rating that essay is something that your English teacher would spend quite a bit of time trying to read and give comments and all that. That is a much different thing than answering multiple choice questions. And we just don't know how to do that. There's also the problem that people say 'oh just do human evaluation.' That is really not an answer in a sense. First of all, which humans? If the question involves deep knowledge about, let's say, medical data, you can't just grab a random crowd worker and have them do it. So it requires experts. But then even if you have the experts, how do you scale? Do you do pairwise comparison? Do you do some sort of like art rating? Do you have feedback? All these questions about how to even pin down what the evaluation should look like is pretty open. And then I guess there's another aspect of evaluation which is thinking about safety. So I've been involved with working with ML Commons on developing safety evaluations, and it's just an open-ended question: how do you measure bias? Again, I think my tendency has been to always situate in the actual use case because then you can more meaningfully talk about any sort of benefits or risks. But it's just hard to define when there's not a really clear correct answer. You know, the silver lining is that evaluation has now, I think in light of these foundation models, become more of a first-class citizen in a way that it was just not before. It was always 'you train a model, you evaluate.' You never evaluate without—

训练是因为如果你提出一个评估集，就没有模型可以评估。但现在我们有了这些通用模型，你可以真正专注于评估。我认为这非常强大，因为你可以想象去某个领域找专家，和你坐下来收集几百个例子——绝对不足以进行任何训练，但我认为足以进行有意义的评估。所以我认为我们能从评估中获得的保真度应该比以前高得多，只是因为数据需求少了很多。然后还有如何评估多模态模型、图像生成的问题？我们有一篇关于做图像生成评估的论文，这显然是一个很好的步骤。我认为还有很多工作要做，更不用说多语言评估和思考智能体，以及如何评估不仅是语言模型，还有使用工具的语言模型。我们还有一些工作，比如建立 ML 智能体基准，这是一个新的评估，用于评估智能体是否能完成一些机器学习任务。这是一个相当活跃的领域，所以希望我们作为领域能取得一些进展。

Training because if you come up with an evaluation set, there's no model to evaluate. But now we have these general-purpose models that you can actually just focus on evaluation. I think that is really powerful because you can imagine going out and getting experts in some domain to sit down with you and gather a few hundred examples — definitely not enough to do any sort of training, but enough, I think, to do meaningful evaluation. So I think the level of fidelity that we can get from evaluation should be much higher than before, just because the data demands are much less. And then there's a whole how do you evaluate multimodal models, image generation? We had a paper on doing image generation evaluation, which is obviously a good step. I think so there's a lot of work to be done, not to mention multilingual evaluation and thinking about agents and how to evaluate not just the language model but language models with tool use. We also have some work on setting up ML Agent Bench, which is a new evaluation for evaluating whether an agent can go on and do some machine learning task. It's a pretty vibrant space, so hopefully we'll make some progress as a field.

是的，这真的很有趣。上个季度我和大约 40 位创始人聊过，他们都在做智能体，他们遇到的头号问题就是评估太难了。我不知道如何判断我的智能体系统是否在变得更好。很多人用非常巧妙的方式解决了这个问题，比如创建自己的少量专家示例。但开放式生成这个问题真的很难。我认为我们通过主要使用问答作为我们许多智能体的基础来绕过它，但我很好奇你是否见过任何处理这个开放式生成问题的好方法。

Yeah, this is really interesting. I talked to maybe 40 founders last quarter who are working on agents, and the number one problem they all encounter is evaluation is too hard. And I don't know how to know if I'm making my agent system better or not. And a lot of people have solved it in really happy ways, like creating their own few expert examples. But this question of open-ended generation is really hard. I think we've gotten around it by mostly using question answering as the basis for a lot of our agents, but I'm curious if you've seen any nice ways of dealing with this open-ended generation problem.

是的，我们一直在探索模型分级评估，这变得越来越流行。首先声明：这些真的需要谨慎对待。但我认为，就像说人类评估不充分一样，说模型评估不充分也是对的，因为它实际上取决于你如何设置提示以及如何提问。所以具体来说，我认为就你的观点而言，如果你能把事情归结为问答，并且能更精确地说明你在寻找什么，那么基于模型的评估实际上可以变得可行，并且至少有望与你关心的内容相关。我还认为评估通常有两个目的，对吧？一个是绝对基准，比如‘领域进展如何’——一种统一的东西——另一个是我们只需要某种指标来爬山，这样我们就知道我们正在得到更好的模型。这两者实际上非常不同。所以即使例如困惑度显然是一个很差的绝对指标，无法告诉任何人模型能做什么，因为它不是真正可解释的，也不与实际下游结果相关。但它实际上，我认为，至少对于预训练模型来说是一个相当好的替代指标。而且我确实相信替代指标的力量。所以即使你没有正确的评估指标，如果你能开发出一个相关性足够好的替代指标，并且你没有自欺欺人，也没有玩弄它，那么我认为它可能非常有用。所以很多评估，例如，甚至 BLEU 分数实际上对某些类型的场景是有用的，对吧？如果你考虑 BLEU 分数，并且你只是在改变模型的一个轴，比如训练模型大小，我认为可能没问题。但如果你试图比较，它不是一个好的绝对指标，当然如果你试图玩弄它，那么指标就失去了意义。

Yeah, we've been exploring model-graded evaluation, which is becoming more popular. First disclaimer: these need to really be taken with a grain of salt. But I think just as saying human evaluation is inadequate, saying model evaluation is also inadequate, because it really depends on how you set up the prompts and how you ask the questions. So in particular, I think to your point, if you can distill things down into question answering and you can be more precise about what you're looking for, then model-based evaluation I think can actually become viable and at least hopefully correlate with what you care about. I also think that evaluation often has two purposes, right? One is an absolute benchmark of 'here's how the field is doing' — save a union kind of thing — and the other is that we just need some sort of metrics to hill climb on, so we know that we're getting better models. These are actually very distinct. So even for example, perplexity is obviously a very bad absolute metric for telling anyone anything about what the model can do, because it's not really interpretable and not correlated to actual downstream outcomes. But it actually is, I think, a fairly good surrogate at least for pre-trained models. And I think I do believe in the power of surrogate. So even if you don't have the right evaluation metric, if you can develop a surrogate that is well correlated enough and you are not fooling yourselves and not gaming it, then I think it could be very productive. And so many of the evaluations are, for example, even BLEU score is actually useful for certain types of regimes, right? If you think about a BLEU score and if you're changing the model but just varying one type of axis, let's say training model size, I think it's probably okay. But if you try to compare, it's not a good absolute metric, and certainly if you try to game it, then the metric loses meaning.

你认为这如何适用于我们开始拥有来自 Google、Anthropic、OpenAI 等公司的许多不同语言模型的情况？比如人们将如何决定应该使用哪个模型？这些模型相比起来有多好？这是否应该更公开、更广泛地以某种基准进行，还是最终只会被玩弄？每个人都会为自己的特定应用运行这些大型评估吗？你认为未来会是什么样子？

How do you think that applies as we start to have many different language models from Google and Anthropic and OpenAI and others? Like how are people going to be able to decide which model should I use? How good are these models compared to each other? Is this something that should be done more publicly and more broadly in some kind of benchmark, or does that end up just getting gamed? Is everyone just going to run these huge evaluations themselves for their particular application? What do you think that looks like in the future?

是的，我绝对认为这需要以受控的方式进行第三方评估。我想稍微抱怨一下，我认为每个人都在报告 MMLU 分数，但他们报告的方式——你可以——我不是在抱怨 MMLU，我是在抱怨你得到一些数字的方式，以及用于 MMLU 的提示都是不同的。有时你使用花哨的技巧，比如自一致性或思维链等等，你只是得到一个数字，但你不清楚你到底在测量什么。你不是在真正测量模型；你是在测量模型加上任何在其之上的提示工程，这可能不太有意义。我的理念是，我们应该绝对以标准方式测量一切，并且我们应该让事情透明。因为特别是如果你看生成式评估，我认为所有指标都将非常近似，我希望人们能够在事后说，‘啊，这个自动模型分级评估不好；我有一个更好的。’并且拥有模型的实际输出——实际上拥有你可以检查的输出，你可以做进一步分析——我认为这里至关重要。这就是 HELM 所做的：我们公开所有模型输出，这样人们就可以去研究它。但现在如果一切都是公开的，你就会担心，‘好吧，现在人们可以有意或无意地在其上训练，然后模型评估就失去了意义。’这确实是真的，但我认为解决方法是，你必须不断有新的评估，我认为我们无论如何都需要，因为随着这些模型的能力扩展，我们想要测量不同的东西。所以我们无论如何都需要新的评估，所以让我们创建它们，让旧的评估出于遗留原因而存在——你有它，你知道它做什么，但不要过于信任它作为客观评估。

Yeah, I absolutely think that needs to be third-party evaluation done in a controlled way. I think just to complain a little bit, I think that everyone reports MMLU scores, but the way that they're — and you could — I'm not complaining about MMLU, I'm complaining about the way that you get some numbers and the prompts that are used for MMLU are all different. And sometimes you use fancy tricks like self-consistency or chain of thought or whatever, and you just get a number, and it's not clear what you're exactly measuring. You're not really measuring the model; you're measuring the model plus any sort of prompt engineering that goes on top of that, which is not quite maybe meaningful. And my philosophy is that we should absolutely measure everything in a standard way, and we should make things transparent. Because especially if you look at generative evaluations, all the metrics I think are going to be really approximate, and I want the ability for people to go after the fact and say, 'Ah, this automatic model-graded evaluation is not good; I have a better one.' And having the outputs actually of the models — actually having the outputs where you can inspect them, you can do further analysis — I think is critical here. And this is what HELM does: we make all the model outputs public so that people can go and have fun with it. But now if everything is public, now you worry, 'Okay, well now people can just train on it, either accidentally or purposefully, and then the model evaluation loses meaning.' So that is true, but I think the way around that is you just have to have a constant of new evaluations, which I think we need anyway because as the capabilities of these models scale up, we want to measure different things. So we need fresh evaluations anyway, so let's create them and have the old ones just be sort of for legacy reasons — you have it, you know what it does, but don't trust it too much as an objective evaluation.

我实际上很喜欢这个想法，基本上随着能力的提升，不断改进评估基准。是的，是的。那么我想问题是，谁来资助这个？制作高质量的评估数据实际上相当昂贵，你需要真正思考任务，随着时间的推移更新它，并且对这些非常严格。而且，作为一名学者，你觉得你能为这样的事情获得资金吗？或者需要一些 AI 安全研究所，或者一些由政府设立的 AI 研究所需要资金？你怎么看？

I like that actually, as a kind of basically continually improving evaluation benchmarks as capabilities improve. Yeah, yeah. I guess the question then is, who funds this? It's actually kind of expensive to make high-quality evaluation data where you're really thinking about the task and you're updating it over time and you're being really rigorous about these. And also, as an academic, do you feel like you can get funding for something like this? Or there needs to be some AI safety institute that is like, or some AI institute that's like government set up that needs funding? How do you think about that?

是的，我的意思是，我认为这个领域需要更多的资金。如果你看看投入到模型构建与评估的资金量，那是

Yeah, I mean, I think there needs to be a lot more funding in this area. If you look at the amount of funding going into model builds versus evaluation, it's

完全失衡。投入到模型构建的资金比评估要多出好几个数量级。传统上，十年前，数据集并不被重视；人们想做的是建模和算法工作。但这种文化正在改变。NeurIPS 有了新的数据集和基准测试赛道，而且正在蓬勃发展。我认为这是一个认可基准测试重要性的好平台。我们需要的不只是学术资金来创建数据集，还需要更全面地思考如何规划这个领域。很多评估工作，比如我们在 HELM 做的，并不符合发表下一篇论文的目标；我们做这些是因为世界会受益。AI 研究所正在出现：英国有一个，美国有一个，欧盟也将有一个。最终结果如何还有待观察，但确实有进展。如果我们能把这些资源引导到资助推动基准测试的活动上，那就太好了。

Completely lopsided. The amount of funding going to model building is many orders of magnitude larger than evaluation. Traditionally, 10 years ago, datasets weren't really appreciated; people wanted to do the modeling and algorithms work. But that culture is changing. NeurIPS has a new datasets and benchmarks track, which is thriving. I think it's a good venue to recognize the importance of benchmarks. We need more than just academic funding for dataset creation, but also a holistic view of charting the space. A lot of evaluation work, like what we do at HELM, isn't aligned with publishing the next paper; we do it because the world benefits. AI institutes are emerging: the UK has one, the US has one, the EU will have one. It remains to be seen what comes out, but there is movement. If we channel that into funding benchmarking activities, that would be great.

从能力或鲁棒性评估转向，你最感兴趣的是什么？我知道 Alpaca 出自你的实验室，还有那个生成式智能体的论文。这些都是对能力和新技术的有趣探索。你如何看待这方面的探索？哪些看起来有前景，哪些没有，哪些被低估了，哪些被高估了？

Shifting away from evaluations on capability or robustness, what are you most interested in? I know Alpaca came out of your lab, and also the generative agents paper. These are interesting explorations into capabilities and new techniques. How are you thinking about exploration there? What feels promising, what does not, what feels underhyped versus overhyped?

让我澄清一下，以免混淆。智能体被讨论得很多。有两种类型。经典类型，比如 ML Agent Bench，是一个语言模型包裹在某种带有工具使用的架构中，能够比原始语言模型做更多事情。另一种类型以生成式智能体为代表，是关于模拟的。没有目标；人们问目标是什么。目标就是模拟并看看会发生什么。我们有一个由 25 个智能体组成的城市，每个都由 LLM 驱动，被提示去生活。它们互动，你会看到涌现的社会行为。这很迷人。将其规模化需要算力和快速推理，但你可能看到有趣的社会动态，比如政府形成。这在科学上会很棒。另外，如果模型能忠实地模拟人类，你就可以做关于人类行为的社会科学和经济学研究。很多研究很难，因为你需要受试者，不能给同一个人控制组和实验组，而且有些事不道德。有了 LLM，你就从这些限制中解放出来了。但需要注意的是，这些不是真实人类，所以不能直接得出结论，但它为洞察提供了另一个维度。我认为这被低估了，但很有趣。

Let me clarify before there's confusion. Agents are talked about a lot. There are two types. The classical type, like ML Agent Bench, is a language model wrapped in an architecture with tool use, able to do more than a raw LM. The other type is exemplified by generative agents, which are about simulation. There's no goal; people ask what the goal is. The goal is just to simulate and see what happens. We have a city of 25 agents, each backed by an LLM, prompted to live their lives. They interact, and you see emergent social behaviors. That's fascinating. Scaling this up would require compute and fast inference, but you might see interesting social dynamics, like governments forming. That would be scientifically awesome. Also, if models can faithfully simulate humans, you could do social science and economic studies on human behavior. Many studies are difficult because you need subjects, can't give the same person control and treatment, and some things are unethical. With LLMs, you're liberated from that. The caveat is these are not real humans, so you can't take conclusions directly, but it gives another vector for insights. That's underexplored but interesting.

这太有趣了。我第一次听说用多个智能体交互来模拟人类系统并进行科学研究。我们可能发现新的治理机制。这真的很吸引人。

That's super interesting. The first time I've heard of many agents interacting as a way to simulate human systems and do science. We could discover new mechanisms for governance. That's really interesting.

没错。例如，人们争论 COVID 政策。你可以模拟一个政策，让人们四处走动，每个人都感染 COVID，结果很糟糕。这种思路让我很不舒服。我看到了科学价值，比如发现新的政府体系。但我担心人们会把这些当作证据，而不去深究科学细节。就像所有从模拟到现实的问题一样，你能在多大程度上信任模拟？在机器人学中，模拟器很好；机器人会摔倒。在这里，你让一个文明崩溃，风险更大。所以，我们离用这个重写宪法还有一段时间。别太担心。但关键是，我们需要对结果持保留态度。

Exactly. For example, people debate COVID policies. You could simulate a policy, have people walk around, everyone gets COVID, and it's a bad idea. This line of thinking makes me uncomfortable. I see scientific value, like discovering a new government system. But my worry is people will take these as evidence and not dig into the scientific nuance. Like all sim-to-real problems, how much can you trust the simulation? In robotics, simulators are good; the robot falls over. Here, you make a civilization fall over, with bigger stakes. So yeah, there's a while before we rewrite the Constitution using this. Don't worry too much. But the point is we need to take results with a grain of salt.

这个想法太有趣了。我喜欢。那能力方面还有其他东西吗？有没有你感兴趣的领域或主要障碍？我们谈到了价值观方面的数据归因，但还有其他吗？

The idea is so interesting. I love it. What about other things on the capability side? Are there areas you're interested in or major blockers? We talked about data attribution in terms of values, but are there other things?

既然我们在讨论智能体，那就谈谈 ML Agent Bench。这是一个针对智能体的基准测试，旨在解决像 Kaggle 竞赛这样的任务。你给一个 Kaggle 竞赛智能体，准确率可能比基线提高 10%。我们的任务范围从相当简单（可能已在训练集中）到所有人都得 0% 的任务。追踪这个会很有趣。让我兴奋的是，作为 AI 研究者，这直接帮助了我。这不是一些我没有个人兴趣的随机 QA 任务。如果你能解决这个，它真的能帮到我。

Since we're on the topic of agents, let's talk about ML Agent Bench. It's a benchmark for agents meant to solve tasks like a Kaggle competition. You give a Kaggle competition agent, and accuracy might go up 10% over baseline. We have tasks ranging from fairly easy (probably in the training set) to things where everyone gets 0%. That will be interesting to track. What excites me is that as an AI researcher, this directly helps me. It's not some random QA task I have no personal interest in. If you can solve this, it can actually help me.

改进研究本身，所以那个反馈循环可能会很有趣。嗯，我就说到这里。

Improve research itself, and so that feedback loop could be interesting. Yeah, so I'll just leave it at that.

是的，同意。一种比较狂野的思考方式是：ML Agent B 就像是递归自我改进的第一步，其中递归涉及一群人类试图改进它。

Yeah, agreed. The kind of wild way of thinking about this is: ML Agent B is like the first step to recursive self-improvement, where the recursion involves a bunch of humans trying to improve it.

嗯，那正是我在想的，但我不想说出来。别担心，我替你说。不过我觉得，就像许多安全相关的事情一样，当你看到现实世界中的版本时，它并没有那么可怕，反而感觉合理得多。比如，如果你想拥有一个递归自我改进的 LLM，可以。那么现在每六个月当你进行新的六个月训练时，你会变得稍微好一点。实际上，很多这样的训练都有很长的周期，参与其中的人并不像科幻故事那样。

Yeah, that was what I was thinking, but I didn't want to say it. Don't worry, I'll say it for you. Although I think, like with many safety things, when you look at the real-world version of this, it's not so scary and it feels quite a bit more reasonable. Like, if you want to have a recursively self-improving LLM, cool. So now every six months when you're training your new six-month training run, you get a little bit better at it. There's actually really long time frames for a lot of these trainings, and people involved in things like it's not like a science fiction story.

是的，我认为可怕程度实际上取决于你如何看待细胞有多危险，就这么说吧。这其实是另一个话题。也许我们应该谈谈这个。我想问你的是：你怎么看这个问题？你怎么看？

Yeah, I think the scariness really depends on the way that you think about how dangerous cells are, just put it that way. Which is actually some other topic. Maybe we should touch on that. Something I wanted to ask you about: how do you think about that? How do you think about that?

嗯，首先我应该说，我们都应该非常开放，考虑所有非零概率的可能性，就像一个好的愿景。因为坦白说，我对这些模型的表现感到非常惊讶，我想每个人都被这些简单原则能走多远所震惊。所以我在某种程度上吸取了教训，不再认为‘哦，那永远不会成功，因为我做不到 Y 或 Z’，因为任何我想的……话虽如此，我认为关于风险的讨论缺乏严谨性。我理解，是的，未来风险有很多未知的未知，但仅仅援引‘未知的未知’这个词并不能让你免于至少尝试严谨。我一直在思考的一个特别问题是开放模型。因为作为科学家和研究者，我们所做的就是公开做事。我们希望社会受益，所以这算是核心价值主张。但与此同时，人们说，‘哦，如果你把这些开放模型都放出来，GPT-4 或 GPT-5 是开放的，那我们就完蛋了。’这就引出了一个问题：好吧，我们来谈谈。那么风险有多大？要从这一点思考，你需要考虑威胁模型是什么，对吧？你可以分解，例如，虚假信息、网络攻击、生物武器、化学武器。这些大概是人们喜欢考虑的一些常见威胁。还有更奇特的类型，比如失控和 AI 接管等等。但对于每一种，我认为思考风险的方式、监管的类型都需要更多考虑。我认为这个领域缺乏的一点就是对这些风险有多大的衡量，特别是考虑一个模型完全没有时的边际风险。例如，有些研究表明，你可以提示 Llama 2，它能给你合成生物武器的说明。好吧，如果你看到这个，就会说，‘不行不行，赶紧关掉。’我认为这显然是个错误的反应，因为如果你上谷歌，你实际上能找到所有这些信息。而且，即使你能找到配方，你还需要制造它，我所有的生物学家朋友都告诉我那才是困难的部分。没那么容易。想想看，对于一个研究者，你可以说，‘好吧，我怎么构建一个 LLM？’你问 ChatGPT，它能做到，它还会给你步骤。就像，‘哦，哇，现在我知道怎么训练语言模型了。’但实际上，显然还有很多复杂性，执行真的很重要。所以我认为我们需要仔细评估实际风险，而不是仅仅停留在表面。所以这又回到了评估以及我们如何衡量这些问题上。

Yeah, so first I should say that we should all be very open-minded and entertain all possibilities with non-zero probability, like a good vision. Because frankly, I've been very surprised by how these models work, and I think everyone has been surprised by how far you can get with these simple principles. So I've sort of learned my lesson in some ways to not think, 'Oh yeah, that's never going to work because I can't do Y or Z,' because anything I think... That said, I think there is a real lack of rigor in the discourse about the risk. And I understand that yes, future risk, there's a lot of unknown unknowns, but just invoking the word 'unknown unknown' doesn't get you off the hook for trying to at least be, you know, rigorous. And I think one particular thing I've been thinking about is open models. Because as a scientist and as a researcher, that's all we do: do things openly. We want society to benefit, and so that's sort of the core value proposition. But at the same time, people talk about, 'Oh well, if you put all these open models out, GPT-4 were or GPT-5 were open, then we'd be all doomed.' And so that raises the question: okay, so let's talk about it. So how much of a risk is it? And to think from that point, you need to think about what are the threat models, right? Okay, so you could break down, for example, disinformation, there's cyber attacks, there's bioweapons, there's chemical weapons. Those are I guess maybe some of the favorite ones that people like to think about. And there's more exotic varieties like loss of control and AI takeover and things like that. But for each of these, I think the type of way you think about the risk, the type of regulation, needs more consideration. And one thing I think is lacking in the space is just measurement of how much of the risks these are, and particularly think about marginal risk of a model not having it at all. So for example, there's these studies that show, well okay, you can prompt Llama 2 and it can give you instructions on how to synthesize bioweapons. Okay, and if you look at that, it's like, 'Okay, no no no, shut it all down.' I mean, that's I think obviously the wrong response, because if you go out on Google, you can actually find all these things. And also, even if you could find a recipe for it, you still have to manufacture it, and all my biologist friends tell me that that's actually the hard part. It's not that easy. Just think about, I think for a researcher, you can say, 'Okay, how do I build an LLM?' You ask ChatGPT, you can do this and it'll still give you the steps. It's like, 'Oh wow, now I know how to train a language model.' But actually, there's a lot obviously a lot more complexity, and the execution really matters. So I think we need to think carefully about assessing the actual risk, rather than just being somewhat superficial about it. And so again, it goes back to evaluation and how we measure these things.

这里有一个非常有趣的点，你谈到了开放模型。可能有一个相当合理的论点，即危言耸听的‘关闭它’实际上有利于并倾向于集中权力，走向对能力更严格的控制体制，而不是更开放的体制。所以如果我们不实际地评估风险，实际上有一个很大的危害，我认为人们没有谈到：危言耸听的大危害是它集中而非分散权力。这是其中的一部分。

There's actually a really interesting point here where you talk about open models. And there's probably a pretty reasonable argument that the fearmongering 'shut it down' actually benefits and tends to centralize power, and moves toward a more control regime around the capabilities, as opposed to a more open regime. So if we're not actually realistic about assessing risks, there's actually a big harm which I think people don't talk about: the big harm of fearmongering is that it centralizes rather than distributes power. That's a piece of it.

是的，是的，当然。我认为恐惧很多。恐惧与证据的比例极高，这可能导致各种我认为相当糟糕的监管提案。比如，如果你说，假设责任：模型开发者对下游发生的一切负责，那么你基本上就扼杀了开源，或者至少是强大的开源模型。或者如果你对模型有很多要求，即使是出于好意，也可能把小玩家排除在外，因为如果你每次发布模型都要进行 2000 万美元的评估工作，那谁能负担得起？所以我认为有很多后果需要考虑。这并不是说，根据风险，也许我们必须承担一些成本。我绝对不属于那种‘放任自流，一切都会好’的阵营。我确实认为我们需要负责任的做法。但我只是认为其中很多可以在开源中实现，我们需要再次更多地投资于安全和理解。而且我主张，实际上公开做事对安全更有利，因为至少如果你类比计算机安全，通过隐匿实现安全是行不通的。但如果你把东西公开，让每个人检查和审视，那么你实际上可以带来更安全的系统。现在 AI 有点不同；我不认为这个论点完全自动适用。但我认为我们可以从计算机安全中已经奏效的做法中获得一些启发。

Yeah, yeah, yeah, definitely. I think there's a lot of fear. The fear-to-evidence ratio is extremely high, which can lead to all sorts of, I think, fairly bad regulatory proposals. Like if you say, let's say liability: model developer is liable for everything that happens downstream, then you essentially kill open source, or at least strong open source models. Or if you have a lot of requirements for models, which might be even well-intentioned, it could shut out the small players, because if you have to do a $20 million exercise in evaluation every time you release a model, then who's going to be able to afford this? So I think there's a lot of consequences that need to be taken into account. And it's not to say that, depending on the risk, I mean maybe we have to incur some of the cost. And I'm not definitely in the camp where just let it all loose, everything will be fine. I do think we need responsible practices. But I just think that a lot of them can happen in the open source, and we need to invest again more into safety and understanding. And I've argued that actually having things done in open is better for safety, because at least if you look at analogy with computer security, security through obscurity doesn't work. But if you have things out in the open for everyone to inspect and scrutinize, then you can actually lead to more secure systems. Now AI is a bit different; I don't think the argument completely translates automatically. But I think that's some inspiration we can look at that has worked in computer security.

关于开源我有一个问题：我们现在处于一种某种程度上违背了开源所基于的先前原则的体制。例如，在软件开源中，任何人都可以真正为软件系统的能力做出贡献。而在这种模型体制中，情况并非如此。模型必须由一组人开发和训练，实际上很难在传统软件意义上做一个开源项目。我很好奇你对评估基准有什么想法：我能看到传统体制如何适用。但对于模型训练，你对未来如何适用有什么想法吗？

One question I have around open source: so we're in a regime that kind of violates the previous principles that open source was built on. For example, in software open source, anyone can really contribute to the capabilities of the software system. Whereas in this model regime, it's not really the case. The model has to be developed and trained by some set of people, and it's actually really hard to make an open-source project in the traditional software sense. And I'm curious if you have thoughts on like evaluation benchmarks: I can see how that traditional regime can apply. But for model training, do you have thoughts on going into the future how that could apply?

嗯，好的，我很高兴你提到这一点。注意我没有用‘开源模型’这个词，因为这不是开源软件，这是一个完全不同的……

Yeah, okay, I'm glad you brought this up. So notice that I didn't use the word 'open source models', and because this is not open source software, it's a completely different...

我能想到的最接近的类比是，这些是开放的二进制文件。模型的权重公开并不等同于源代码公开。你可以执行模型，但你不知道是否有后门，也不知道训练数据或模型的任何信息。这与拥有 Linux 的源代码形成鲜明对比，理论上你可以阅读数百万行代码，至少它是可审计的。所以从这个意义上说，它已经非常不同了。此外，即使训练数据公开，也明显不同，因为贡献的方式非常不同。在源代码中，你发现一个 bug，可以提交 PR，打补丁，人们可以回馈给这个不断改进的东西。而模型是由某个大组织训练、发布，然后它就冻结了。没有补救措施，没有 bug 修复。如果有人不喜欢某些东西，你无法真正改变它。你可以微调，这是一种非常不同的模式，而且微调实际上有一些优势，因为你不需要太多专业知识就能定制你的模型。所以这是可定制性的一大优点，但缺乏反向贡献使其与开源非常不同。我认为最接近开源的是拥有训练代码、数据和模型权重供所有人查看，然后你仍然需要算力来实际运行训练。算力部分肯定是个问题，但至少如果你有训练数据、模型代码和一些中间检查点，即使你不从头训练，也可以对模型进行审计分析。

The closest analogy I can think is that these are open binaries. The weights of a model being out there is not the same as source code being out there. You can execute the model, but you have no idea if there are backdoors, you have no idea what the training data or anything about the model is. This is in stark contrast to having the source code of Linux, where in theory you can go read the millions of lines of code if you wanted to; at least it's auditable. So that's a sense in which it's already very different. On top of that, even if the training data were out there, it's clearly different because the method of contribution is very different. In source code, you find a bug, you can submit a PR, you can patch it up, and people can contribute back to this thing that is constantly getting better over time. Whereas a model, some big organization trains it, releases it, and it's sort of frozen. There's no recourse, no bug fixes. If someone doesn't like something, you can't really change it. You can fine-tune, which is a very different mode, and fine-tuning actually has some advantages because you don't have to have very much expertise to really customize your model. So that's a big plus for customizability, but the lack of backward contributions makes it very different from open source. I think the closest thing to open source would be having the training code, the data, and the model weights available for everyone to see, and then you still just need the compute to actually run a training run. So the compute part is definitely a problem, but at least if you had the training data, the model code, and maybe some intermediate checkpoints, you can actually do an audit analysis on the model even if you weren't training it from scratch.

关于你提到的微调，我好奇的是，你对预训练和微调各自的作用有没有直觉？我们内部一直在做的是在预训练和微调中使用不同类型的数据，看看这对最终结果有什么影响。有一些有趣的早期发现，但还不太清楚。我很好奇你有什么想法。

On your point about fine-tuning, one thing I'm curious about is whether you have intuitions on what pre-training is doing versus what fine-tuning is doing. Something we've been running internally is using different types of data in pre-training versus in fine-tuning and seeing how that impacts the end results. Some interesting early findings, but not super clear. I'm curious if you have any thoughts.

微调和预训练，我认为至少在逻辑上占据非常不同的功能。如果你不从单个组织构建模型的角度，而是从生态系统层面来看，模型提供者构建一个原始模型，它非常好，拥有知识和推理能力，然后理想情况下会有大量不同的微调版本，针对特定组织或用户的需求进行定制。所以这是思考微调的一种方式。如果是在单个组织内部，那其实无所谓，只是完成任务而已。但基础模型更有趣的地方在于，存在这种接口或交接：你可以完成部分工作，然后交给别人，甚至可以交给多个人。这就是它的强大之处。还有关于模型合并的有趣工作。一旦你分叉，然后呢？你能把所有东西合并回来吗？Colin Raffel 有一些非常有趣的想法，关于如何将模型构建视为类似于软件的版本控制。这个类比还不完全成立，但我认为思考如何让模型构建基础设施更成熟是很有启发性的。

Fine-tuning and pre-training, I think at least logically occupy very different functions. If you think from less of a perspective of an individual organization building a model, but what happens at an ecosystem level, the model provider builds a raw model that's really good and has knowledge and reasoning, and then ideally there would be a plethora of different fine-tunes that happen which are customized to a particular organization's or user's needs. So that's one way to think about fine-tuning. If it's within a single organization, then it really doesn't matter; it's just getting the job done. But maybe what's more interesting about foundation models is that there is this interface or handoff that can happen: you can do part of the work and hand it to someone else, and you can hand it to multiple people. That's the powerful part. There's also really interesting work around model merging that happens. Once you fork, then what happens? Can you bring everything back together? Colin Raffel has some really interesting thoughts about how to think about model builds similar to version control for software. The analogy isn't quite there yet, but I think it's inspirational to think how we can make the model building infrastructure more mature.

你怎么看待学术界在提升能力、构建模型方面的作用，相对于商业大公司构建这些东西并开源？学术界的人是否还值得尝试推动能力前沿，还是我们进入了一个只能基于提供的工件进行构建和研究的阶段，而无法像过去那样推动能力？也许一个不那么二元化的框架：在能力方面，学术界扮演什么角色？学术界是否值得存在？有些人提议我们应该拥有一个大算力集群，训练我们自己的基础模型。这是值得的努力还是浪费？

How do you think about academia improving capabilities, building models, relative to commercial larger companies building these things and open-sourcing them? Is it even worthwhile for people in academia to be trying to push the frontier of capabilities, or are we entering a regime where we can only build on top of artifacts provided and study them, but not push capabilities in the same way as in the past? Maybe a less binary framing: where does academia play when it comes to capabilities? Is it worth having academia? Some people propose we should have a big compute cluster and train our own foundation models. Is that a worthwhile effort or a waste?

让我试着分解一下。没有大算力集群，学术界有多重功能。一是像往常一样，不断创造新的做事方式，验证它们，然后有人可以将其规模化。在小规模上做事，意图是停留在小规模，与在小规模上做事但意图是规模化，这两者是有区别的。例如，我们与 Chenry M 的学生在优化器方面有一些工作，实际上可以比 Adam 好两倍。这是我们在学术界可以做到的。我们还没有在大规模上证明它，但有人应该在大规模上尝试。还有 Flash Attention，这是我最喜欢的例子之一，它来自学术界，现在在工业界无处不在。所以仍然有空间为模型构建方式带来根本性变化。另一个例子是直接偏好优化（DPO），这是一项非常有影响力的工作，不需要太多算力。所以在方法方面有很多事情可以做。然后是评估，我们已经讨论过。作为一个中立方，深入思考评估，学术界做得一样好，甚至比拥有更大算力预算的人更好。然后是长期的事情，比如数据归因和重新设计整个激励系统。我认为工业界不会碰这些，因为这真的是在社会层面思考，而不是单个组织试图构建模型。这也是学术界可以发光的地方。

Let me try to break it down. Without the big compute cluster, academia has multiple functions. One is, as usual, constantly creating novel ways of doing things, proving them out, and someone can scale it up. There is a difference between doing things at small scale with the intention of doing things at small scale, and doing things at small scale with the intention of scaling up. For example, we had some work on an optimizer with Chenry M's students which can actually outperform Adam by a factor of two. This is something we can do in academia. We haven't proven it at scale, but someone should try it at scale. There's also Flash Attention, which is one of my favorite examples of something that came out of academia and is now everywhere in industry. So there is still space for producing fundamental changes to how model building works. Another example is Direct Preference Optimization (DPO), which is a really influential piece of work that doesn't need that much compute. So there are many things you can do on the method side. Then there's evaluation, which we already talked about. Being a neutral party and thinking deeply about evaluation is something academia is just as good at, if not better than people with a larger compute budget. Then there's the long-term stuff about data attribution and retooling the whole incentive system. I don't think industry is going to touch that because it's really thinking at a societal level rather than an individual organization trying to build a model. That's where academia can shine as well.

话虽如此，即使没有算力，我认为我们也能忙得不可开交，并且有很多有价值的生产性工作可做。然而，我确实认为，要让 AI 拥有比现在多得多的算力是必要的，因为目标不是训练一个新模型。我们并不幻想能拥有与 OpenAI、谷歌甚至一些初创公司同等的算力。但我认为，你需要足够的算力来进入一个想法已被验证的阶段，这是关键。而且这条线，我认为只会随着时间的推移而提高。所以，至少达到那个水平，才能让你继续构思、思考新事物并保持相关性。例如，如果你没有任何 GPU，即使你在考虑大规模的事情，你也无法做任何真正有用的事。因为你需要，比如说，尝试一种新架构，你可能需要达到几十亿参数才能验证它。当然，还有另一个挑战是扩展到 700 亿参数之类的。但至少你需要达到那个规模，你不能训练一个一百万参数的模型并期望它能泛化。所以总结一下，我认为学术界的目标始终应该是思考那些关键的概念性新想法，这些想法能推动事物前进并在大规模上具有相关性。而算力的数量应该与之成比例。我的意思是，很难给出一个确切的数字，但它需要足够大，以便你能够实际计算缩放定律，并且足够远，以便你能可信地外推其余部分。

All that said, I mean even without compute, I think we can be kept busy and there's a lot of valuable productive things. However, I do think that for AI to have way more compute than it has now is necessary, because I think the goal is not to train a new model. There's no delusion that we would have the same amount of compute as OpenAI or Google or even some of the startups. But I think you need enough to be in a regime where the idea has been proven out, and that's the key thing. And that line, I think, will only increase over time. So at least being at that level will allow you to continue to ideate and think of things and be relevant. For example, if you don't have any GPUs, no matter even if you're thinking about the large scale, you just can't do anything terribly useful. Because you need, let's say, to try out a new architecture, you should probably go up to a few billion parameters just to be able to prove it out. And hope that, of course, there's still another challenge to scale up to, let's say, 70 billion or something. But at least you need to go to that, and you can't train a one million parameter model and expect it to generalize. So in summary, I think that the goal of academia should always be to think about what are the key conceptually new ideas that can push things forward and be relevant at scale. And the amount of compute should be proportional. I mean, it's hard to give you a number exactly, but it needs to be enough so that you can actually compute the scaling laws and far out enough so that you can credibly extrapolate the rest.

有道理。这真的很有趣。早些时候你提到思考这些语言模型所体现的价值观，以及仔细思考与人类价值观对齐这个问题有多么重要。很多人都在谈论与人类价值观对齐，但显然人类有很多不同的价值观，这非常复杂。你提出了一个问题：如何让这个过程更加民主，即确定模型展现什么价值观，更加民主或更具参与性。你有什么想法吗？

That makes sense. That's really interesting. Earlier you had mentioned thinking about the values that these language models embody, how it's really important to think very carefully about this question of aligning to human values. A lot of people talk about aligning to human values, but obviously humans have lots of different values, it's very complex. And you posed this question of how would you make this process more democratic, figuring out what values a model exhibits, more democratic or more participatory. Do you have any thoughts on that?

是的，其中一部分更多是在治理层面：你如何确定要关注哪些人的价值观，如何聚合它们？已经有了一些初步工作。我认为集体智慧项目与 Anthropic 合作创建了这种集体宪法 AI，这很有趣。他们收集了一部由来自广泛人群的一千条陈述组成的宪法，并提示 Anthropic 的模型关注这部宪法。所以这是一个有趣的早期想法。我希望更进一步，因为在抽象原则层面，我认为没有太多分歧。好吧，这些语言模型应该是有帮助的、无害的、好的，听起来不错。所以真正重要的是具体细节：这段文本是否应该被视为有害，或者我是否应该允许它？以及所有细微差别，我认为你真正希望人们不仅投票，而且深思熟虑并辩论什么是最好的前进道路。有些事情可以归入个性化范畴，比如如果你喜欢简短的回答，我喜欢长的回答，我们不需要解决那个价值观问题。但也有一些事情，比如气候变化是否真实，我们不希望个性化，我们希望某种客观真理，那里需要一些深思熟虑。所以一种思考方式是，你有一群持有不同价值观的人，你试图聚合它们。我认为这可能只是第一步。但另一部分是：你是否真的可以将语言模型视为一种深思熟虑的媒介，让人们贡献他们的偏好和信念，同时反过来，接受关于这些观点的教育，并与他人分享和辩论，从而在社区中形成关于某些问题的共识？我认为那将是终极梦想：这些语言模型在某种意义上可以充当调解者。

Yeah, some of this is more at the governance level: how do you even figure out what people's values to pay attention to, how to aggregate them? There's been some preliminary work. I think the Collective Intelligence Project collaborated with Anthropic on creating this Collective Constitutional AI, which is interesting. They collected a constitution which consists of a thousand statements from a broad range of people, and prompted basically Anthropic's models to pay attention to that constitution. So that's an interesting early idea. I would hope to go a little bit further in the sense that at the level of abstract principles, I don't think there's too much disagreement. Okay, these language models should be helpful and harmless and fine, okay that sounds good. So it's really about the specifics: is this piece of text, should that be considered toxic or should I allow this? And all the nuances where I think you really want to get people to not just vote but also deliberate and debate over what is the best path forward. And there are some things that you can sweep under the umbrella of personalization, like if you prefer short responses, I prefer long responses, we don't need to sort out that value. But there's also some things, like whether climate change is real, that we don't want personalization, we want some sort of objective truth there, where there needs to be some deliberation. So one way to think about it is you have a bunch of people with different values and you're trying to aggregate. I think that's just maybe the first piece. But then the other piece is: can you really think about language models as a deliberation medium that allows people to sort of contribute their preferences and beliefs, but also going the other way, get educated about some of these views and share and debate with other people, so that there's sort of a consensus that forms among the community about some of these issues? I think that would be the ultimate dream: that these language models can be sort of mediator in some sense.

这真的很有趣。我稍微反驳一下，我们是否想要一个由单一模型调解的东西，还是我们想要更多的道德多样性？肯定会有一些地方，两组人肯定对某件事意见不一致，而你可能希望，也许梦想不是由一个模型来调解，而是许多许多不同的模型。我认为推动开源的原因之一就是它确实朝着这个方向推进：我们拥有的开源模型越多，它们越有竞争力，我们就越能拥有一个生态系统，以及观点和能力的多元化和多样性，等等。

That's really interesting. Do you, I mean pushing back on that slightly, like do we want something that is mediated by a single model, or do we want more of a diversity of morality? There's certainly going to be places where two groups of people definitely don't agree about a particular thing, and you want to have, like, maybe the dream is not a single model that is mediating, but maybe the dream is many, many different models. And I think one of the reasons for pushing for open source is that it does push in this direction: the more open source models we have and the more competitive they are, the more we have like an ecosystem and a plurality and diversity of views and capabilities, etc.

是的，这绝对是正确的，这也是开源模型的好处之一。我认为，当然目标不是让每个人都同意，但有一句话：求同存异。因为我确实认为我们生活在一个共享的世界里，如果每个人都有自己的定制模型，那实际上就是他们生活的小虚拟世界，我的意思是，这基本上就是导致两极分化的原因。我认为这是一个你想要对抗的问题。如果你考虑未来的每个语言模型，我认为我们与世界互动、获取信息以及在世界上采取行动的主要方式，很可能将由这些模型中介。所以这必须与现实挂钩，而不仅仅是基于某种赚钱的广告计划，让人们基本上相信他们想相信的任何东西。而且需要某种共享的现实，哪怕只是为了别的，因为现实世界要求这样。比如，有些选举即将发生，你不能生活在一个一半人认为这个人是总统，另一半人认为另一个人是总统的世界。但确实，在某种程度上，我觉得弄清楚多元与共识之间的微妙平衡是这里的关键问题之一。

Yeah, that's definitely right, and that's one of the benefits of open-source models. I think that one way, of course the goal is not to get everyone to agree, but I think there's a saying: agree to disagree. Because I do think that we live in a shared world, and if everyone has their own customized model, which really is a little virtual world that they live in, I mean that's basically how you get polarization. And I think that is a problem that you would want to fight. If you think about each of these language models in the future, I think our primary way that we'll interact with the world and get information and also take action in the world is probably going to be mediated by these models. So that better be tethered to reality, and not just based on some money-making ad scheme that gets people to basically believe whatever they want. And there needs to be some sort of shared reality, if nothing else, because the real world sort of demands it. Like there's some election that's going to happen, and you can't live in a world where half the people think this person's president and the other half think this other person's president. But yeah, to a point, I feel like figuring out that delicate dance between plurality and consensus is one of the key questions here.

这真的很有趣。是的，这是一种核心的二重性：你实际上两者都想要，但很难弄清楚。

That's really interesting. Yeah, it's kind of a core duality: you want both actually, but it's tricky to figure out.

是的，我认为你需要以某种方式划分问题。在某种程度上，人们可以拥有自己喜欢的音乐，如果他们不想听，可以不去关注另一种流派的音乐。但每个人都必须面对世界的政治现实和环境的现实。我的意思是，我不知道这是否百分之百正确，就像我们有相对论，你知道，它打破了旧的符号。有很多问题，那条线到底划在哪里，对吧？比如可能有些事情……

Yeah, I think you want to partition the issues somehow. In some way, people can have their own favorite music and not pay attention to the music from another genre if they don't feel like it. But everyone has to confront the political realities of the world and the environmental realities of the world. I mean, I don't know if that's 100% true, like we've got relativity, you know, it breaks up like old sort of sign. There's a lot like where exactly does that line fall, right? Like there might be things that sort of...

科学上感觉非常确定，然后有人反驳这种正统观点，比如地球绕着太阳转，我们为什么会想得不一样？你会得到范式转变。但那种范式转变是为了更好地理解现实。所以我们最终有了地球绕太阳转的范式转变，因为它是真的，至少就我们所知。也许还会有另一个范式转变。所以这是关于越来越接近现实，让人们越来越接近现实，我认为这也符合 Percy 所说的。

Scientifically feel very settled, and then someone pushes back this orthodox view that of course the Earth revolves around the sun, like why would we think differently? You get paradigm shifts. But that paradigm shift is about better understanding reality. So we ended up with this paradigm shift of the Earth revolves around the sun because it's true, at least as far as we can tell. Maybe there will be another paradigm shift. So this is about getting closer and closer to reality, enabling people to get closer and closer to reality, which I think is also true of what Percy is saying.

是的，Josh，我认为你提出了一个很好的观点，你需要允许那个疯狂的人去说一些别人都不同意的话。你不想要 100%的封锁，应该有空间。这可能是应该被构建到这个过程中的事情之一，比如语言模型的温度不应该为零。所以这是一个好观点，但我认为这可以构建到系统中；我们只需要有意识地去做。

Yeah, I mean Josh, I think you bring up a good point that you need to allow for this crazy person to go say something that everyone else disagrees with. You don't want 100% lockdown, and there should be space for that. And that's maybe one of the things that should be built into this process, like the temperature shouldn't be zero for the language model. So that's a good point, but I think that could be built into the system; we just have to be deliberate about it.

除了我们已经讨论过的，你对于重要的方向有没有什么强烈的直觉、理论或我们还没涉及的大问题？特别是，对于正在听的人，考虑是否加入你的实验室，应该研究什么？

Aside from what we have already talked about, do you have any strong intuitions or theories or big open questions that we haven't covered about important directions? And perhaps in particular, thinking about it as for people who are listening, thinking about whether they come join your lab, what sort of things to study?

是的，为了理解 X，有点像对学生的号召。我认为我们生活在一个如此激动人心的时代，感觉就像 90 年代互联网革命发生时。互联网应该如何发展，会产生什么影响，有很多不确定性，关于一个全新的新兴技术的基础如何奠定，这些问题极其重要。我只想鼓励每个人从更系统的角度思考问题，因为我认为很多机器学习研究者仍然倾向于认为，好吧，我们有这些模型，它们很酷，我们如何让它们更好？但我认为事物的布局方式以及谁有权力影响事物，是至关重要的问题，如果你只有纯粹的计算机科学思维，这些问题往往会被忽视。但与此同时，这些问题需要技术输入。我想谈的一件事，我就提出来，我们也在考虑语言模型的水印。如你所知，有些人知道这些水印在政策圈被广泛讨论作为一种工具；这是人们唯一能达成一致的东西，用于追踪生成的攻击并防范虚假信息等。但这是一个非常开放的技术方向：如何可靠地加水印。我们一直在为语言模型做一些工作，这些研究直接反馈到政策讨论中，因为能够可靠地做到和不能做到是有区别的。所以我认为很多技术工作确实对事物的设置方式有影响。再次回到 90 年代，思考互联网以及我们如何有这些抽象层次，我们有网络中立性，我们有开放标准。想象一个非常不同的世界，有一个组织拥有互联网，每个数据包都被检查是否有恶意用途。我的意思是，那是一个我们不想生活的非常不同的世界。想想今天 AI 的类比。我认为当时做出的许多决定，无论好坏，都一直伴随着我们。例如，互联网的安全是事后才想到的，我认为我们因此吃了些苦头。身份是我们没有很好理解的东西，这导致了很多问题。所以我认为令人兴奋的是思考在社会层面上为未来基础模型和 AI 构建技术栈。

Yeah, in order to understand X, kind of like a call for students. I think that we live in such an exciting time, and I feel like it's like the 90s where the internet revolution was happening. There was a lot of uncertainty how the internet should evolve, what kind of impact it will have, and many questions about how the foundations of a brand new nascent technology are laid are extremely important. I would just maybe encourage everyone to think about things from a more systemic perspective, because I think a lot of ML researchers still tend to think like, okay, we have these models, they're really cool, how do we make them better? But I think the way that things are laid out and who has the power to influence things are critical questions that I think often get neglected if you have a pure CS mindset. But at the same time, these are questions that need technical input. One thing I would talk about, which I'll just throw out there, is we've been also thinking about watermarking for language models. And as you know, some people know that these watermarks are talked about a lot in policy circles as a tool; it's the only thing that people could agree on for trying to track generated attacks and guard against disinformation and things like that. But this is such an open technical direction: how to watermark reliably. We've been doing some of the work for language models, and that research feeds directly into the policy discussion because there's a difference between being able to do it reliably and not being able to do it. So I think a lot of the technical work does have an impact on how things are set up. And again, going back to the 90s, thinking about the internet and how we have these levels of abstraction, we have net neutrality, we have open standards. I think just imagine a very different world where there's one organization that owns the internet and every single packet is inspected for malicious use. I mean, that's just a very different world that we don't want to live in. And think about the analog with AI today. I mean, I think there are many decisions which were made back then that were sort of stuck with, for better or worse. For example, security for the internet was sort of an afterthought, and I think we've suffered a bit. Identity is something that we don't really have a good sense of, and that has caused a lot of problems. So I think what's exciting is to think about architecting the technology stack at a societal level for foundation models and AI going forward.

我喜欢这个。这是一个很好的结束语，那就是我们在这个世界上的位置，我们有点在开始阶段，一切都在建设中，所以加入我们吧。是的，谢谢你 Percy。

I love that. That's a great note to end on, which is that this is where we are in the world, we're kind of at the beginning, and everything is getting built now, and so come join us. Yeah, thank you Percy.

是的，非常感谢。这很棒，这是一次很棒的对话，我们真的很感激。

Yeah, thank you very much. This was great, this was a great conversation, and we really appreciate it.

这是一次有趣的对话。感谢收听。如果你喜欢这个，请考虑在 Apple Podcast 上给我们评分和评论。在 Twitter 上，我是@KenKJN。下次见。

This was a fun conversation. Thanks for listening. If you like this, please consider giving us a rating and leaving a review on Apple Podcast. On Twitter, I'm @KenKJN. Until next time.