AI 的社会影响与模型开发的开放性

Nathan，欢迎来到超级数据科学播客。很高兴你能来。你从哪里打来的？

Nathan, welcome to the Super Data Science podcast. I'm excited to have you here. Where in the world are you calling in from?

我在加州奥克兰，还会待几周。谢谢邀请。

I'm in Oakland, California for a few more weeks. Thanks for having me.

很好，我的荣幸。那我们就直接进入技术话题吧。你目前是非营利组织艾伦人工智能研究所（AI2）的研究科学家，致力于基于人类反馈的强化学习（RLHF）和微调大语言模型，并且是开放和为了公共利益的。今年早些时候，你发布了 OLMo 模型，因其开放性而受到赞誉，包括提供预训练数据和训练代码。人们可能知道，大多数所谓的开源模型，比如 Llama 系列或 Gemma，并不像你的模型那样开放。公司越来越擅长谈论它，他们发布内部文件告诉人们如何沟通，但有些人还是会搞砸。是什么驱动你尽可能开放地开发这些 AI 工具？

Nice, my pleasure. So let's dig right into the technical stuff. You're currently a research scientist for the nonprofit Allen Institute for AI, where you work on reinforcement learning from human feedback (RLHF) and fine-tuning LLMs in the open and for the common good. Earlier this year you released the OLMo model, which has been praised for its openness, including providing pre-training data and training code. People are probably aware that for the most part, when there's supposedly open source models like the Llama series or Gemma, they're not open like yours is. Companies are getting better at talking about it, they release internal documents telling people how to communicate about it, but some people mess it up still. What are the reasons driving you behind having as much openness as possible with these AI development tools you create?

原因有很多。首先是希望有一个好的结果，这当然带有我个人的偏见，但主要是让更多人能够参与 AI 工作，让 AI 被理解，因为未来几十年它会非常强大，并确保不会因为不透明和人们不了解情况而产生风险。最大的担忧可能是企业垄断。如果 AI 像人们想象的那么强大，可能会导致比苹果和微软大 10 倍的公司出现，那时我们真的不知道现代经济体系会如何运作。所以就是分享成果，确保不会因为不透明和人们不了解情况而产生风险，同时也包括教育和让更多人参与进来，以及这些非常长期的社会变革。

It's various. It's wanting to have a good outcome, which is very biased by what I think is good, but it's just having a lot of people being able to work in AI, having AI be understood because it's going to be very powerful over these next few decades, and making sure there's not risks through obscurity and people not knowing what's going on. The worry is probably largest corporate capture. If AI is as powerful as people think it could be, it could result in companies that are 10 times as big as Apple and Microsoft, and then we don't really know how the modern economic system would work in that context. So just kind of spreading the love, making sure there's not risks through obscurity and people not knowing what's going on, but then also just like education and more people getting involved, and these very long-term kind of societal shifts.

说到大科技公司的事情，你在 Interconnects AI 通讯中最近发表了一篇关于 OpenAI 模型规范的文章。这份文件详细说明了他们如何通过 RLHF 引导模型达到目标行为。你能为可能不了解的听众大致解释一下 RLHF，以及你对这份模型规范文件的发现吗？

On that note of things happening at the big tech companies, in your Interconnects AI newsletter you recently published an article about OpenAI's model spec. This document details how they steer their model with RLHF toward their goal model behaviors. Can you elaborate on RLHF a bit in general for our listeners that maybe don't know it, and what your findings were about this model spec document?

是的，RLHF 是目前最流行的微调技术。对于深入研究语言模型的人来说，它很有趣，因为它是一种不同的损失函数。预训练完成后，你会听到指令微调或有监督微调，它们使用现代 NLP 核心的自回归损失函数。但 RLHF 引入了人为因素，让模型具有难以在数据中捕捉的人性。它也是一个非常广泛的损失函数，所以你可以尝试很多不同的东西。我们才刚刚开始理解，既然我们可以对这些模型进行策略梯度更新或不同类型的更新，我们能让它们变得多么不同且有用。RLHF 是做到这一点的工具，而且它似乎很可能会持续下去。如果你一年前采访我，我会说'哦，我不知道，我们会试试'，但现在每个人都在投资它。Scale AI 在基于 RLHF 的大量收入基础上进行了 F 轮融资；他们公司已经第二次转型。这个模型规范是一个新兴的企业方向，我认为更多公司会这样做。OpenAI 再次走在前列。这似乎是 John Schulman（他是近端策略优化论文的作者，也是语言模型出现之前深度强化学习领域的领先作者之一）和他们的产品团队的合作成果。它本质上是在说我们希望语言模型做什么，无论我们是否把技术细节做对，然后他们会随着更好地理解客户和公司文化而更新这份文件。作为一份独立的文件，它对我这样深入研究的人来说很有趣：他们给出了什么例子，他们给出了什么评论？他们谈到 NSFW 内容以及如何把握分寸，还有命令顺序，比如 OpenAI 对模型相对于客户能做什么有最终决定权，等等。把它放在一个地方很好，因为最终会有多个模型提供商做类似的事情，然后如果你货比三家，你可以看到谷歌希望他们的模型做什么，或者 Anthropic、OpenAI。这也是我们想在 AI2 尝试的事情，就是明确我们想做什么。我们在 RLHF 实践方面相当落后；我们没有那么大的人类数据预算，但我们希望能够说这些是我们的目标，我们会看看能否实现，然后我们可以记录目标如何随时间变化。这就像一个很好的反馈循环：RLHF 是一个混乱的过程，但这些是似乎可以用它处理的事情，无论它是否真的有效。

Yeah, so RLHF is the most popular fine-tuning technique right now. It's interesting to people deep in the weeds of language models because it's a different loss function. All this pre-training is done, then you hear about instruction fine-tuning or supervised fine-tuning, which uses the same autoregressive loss function that's at the core of modern NLP. But RLHF brings in a human factor, so it lets the models have some humanness that is hard to capture in data. It's also a really broad loss function, so there's a lot of different things you could try. We're still barely starting to understand that now we can do policy gradient updates or different types of updates to these models, how different can we make them in very useful ways. RLHF is a tool for doing this, and it seems very likely to stick around. If you had had me on a year ago, I would be like, 'Oh, I don't know, we're gonna try,' but at this point everyone's investing in it. Scale AI did a Series F round with tons of revenue built on RLHF; they've pivoted their company for a second time. This model spec thing is a nascent corporate direction thing, which I think a lot more companies will do. OpenAI is again at the forefront. It seems like a combo work from John Schulman, who's the author of the proximal policy optimization paper and one of the leading authors of deep RL as a field before language models came in, and also their product team. It's essentially trying to say what we want our language models to do, whether or not we get the technical details right, and then they're going to update this document over time as they better understand their customers and company culture. As a standalone document, it's interesting to people very in the weeds like me: what are the examples they give, what is the commentary they give? They talk about NSFW content and how it's hard to thread that needle, and the order of command, so like OpenAI has final say on what the model can do relative to the customer, and all these things. Just having it in one place is good because eventually we're going to have multiple model providers do stuff like this, and then if you're shopping around, you could see what does Google want their models to do, or Anthropic, OpenAI. It's even something that we want to try to do at AI2, just to be like what are we trying to do? We're pretty behind in terms of what the RLHF practices are; we don't have as big of a budget for human data, but we want to be able to say these are our goals and we're going to see if we can achieve them, and then we can document how our goals change over time. It's just kind of a nice feedback loop to be like RLHF is a messy process, but these are the sort of things that seem tractable to do with it, whether or not it actually works.

那么在 AI2，你发表了一篇热门论文《Zephyr：语言模型对齐的直接蒸馏》，关于如何利用蒸馏直接偏好优化（DDPO）在较小模型中进行意图对齐，这使得 70 亿参数的 Zephyr 模型超越了 700 亿参数的 Llama 2，参数多了 10 倍。这种方法与传统方法有何不同，其影响是什么？

So while at AI2, you've published a popular paper called 'Zephyr: Direct Distillation of Language Model Alignment' about how you leverage distilled direct preference optimization (DDPO) for intent alignment in smaller models, which allows this 7 billion parameter Zephyr model to surpass the Llama 2 70 billion on so 10 times as many parameters. How does this approach differ from traditional methods and what are the implications?

关键思想是我们使用更强的教师模型生成偏好数据，然后通过直接偏好优化将其蒸馏到较小的学生模型中。传统的 RLHF 方法通常需要单独的奖励模型和在线采样，计算成本很高。DDPO 通过直接使用教师的偏好优化策略简化了这一过程，使其更高效。这使得较小的模型能够达到有竞争力的性能，从而普及了高质量对齐模型的访问。

The key idea is that we use a stronger teacher model to generate preference data, and then we distill that into a smaller student model using direct preference optimization. Traditional RLHF methods often require a separate reward model and online sampling, which is computationally expensive. DDPO simplifies this by directly optimizing the policy using preferences from the teacher, making it more efficient. This allows smaller models to achieve competitive performance, democratizing access to high-quality aligned models.

关于可扩展性、透明度和可访问性的影响。是的，Zephyr 这篇论文主要关于直接偏好优化（DPO）论文，让它成为主流。有趣的是，就在两天前，Chris Manning 让我在他的课上做讲座，他说：‘哦，天哪，感谢你们让 DPO 看起来真实。’因为存在很大的时间差：DPO 论文在 2023 年 6 月左右发表，而 Zephyr 模型在 2023 年 9 月发布，是第一个用 DPO 方法取得突破的真正模型。当这么多实验室投入训练这些模型并为了公关和产品收益发布它们时，这时间很长。所以存在巨大的时间差，而且它确实建立在实验细节上的奇怪探索之上。比如，我们需要非常低的学习率。AI 界有个梗说 3e-4 是你唯一需要的学习率，它对一切有效，但这个模型和后来 AI2 的 Tulu 2 模型（700 亿参数，展示了规模）都使用了 5e-7 的学习率，这完全超出了当时大多数做微调和 AI 的人的正常范围。然后还有合成数据的概念，这是一种直接蒸馏的想法。有一个数据集叫 UltraFeedback，来自一个叫 OpenBMB 的团体，我想这是一个中国的研究小组。这个数据集直到今天，如果你与其他偏好数据集如 Stack Exchange、Stanford Human Preferences 或 Anthropic 的 HH-RLHF 数据集相比，它只是使用了我们在 UltraFeedback 数据上拥有的方法，该数据混合了来自 GPT-4、GPT-3.5、Llama 2 等模型的完成结果，然后用于创建所谓偏好的选择和拒绝完成结果由 GPT-4 标注。所以这个数据集比 Zephyr 晚了 6 个月。Tulu 仍然是我们看到的最好的。到本期节目播出时，我们会发布更多用 DPO 训练的模型，或者很快之后，我们仍然看到最好的结果来自使用这个数据集。所以随着对话的进行，我可能会继续敲我的鼓：我们需要让人们公开制作更多数据集，如果我们想继续在学术界和开源领域这样做，但要在很多渠道中这样做。但很明显，这个领域会进步很多。我们有这些 Llama 3 模型，我确信 Mistral 很快就会推出一些东西，但我们仍然在使用同一个数据集。它也是同类中的第一个。人们不会把第一个做得完美，尤其是当它是一个研究实验室时，但它的持久性令人印象深刻。

Implications for scalability, transparency, accessibility. Yeah, so this Zephyr paper is mostly about the Direct Preference Optimization (DPO) paper, kind of making it mainstream. It's funny, just two days ago, Chris Manning had me for a lecture in his class and he's like, 'Oh my God, thank you guys for making DPO seem real.' Because there was this big time lag: the DPO paper came out in about June of 2023, and then the Zephyr model was released in September of 2023 as the first real model to make a breakthrough with this DPO method. That's a long time when there are so many labs invested in training these models and releasing them for PR and product gain. So there's a huge time lag there, and it really built on strange exploration in terms of experimental details. Like, we need a really low learning rate. There's the meme in AI that 3e-4 is the only learning rate you need and it works for everything, but this model and later the Tulu 2 model at AI2, which was 70 billion parameters showing scale, both used a 5e-7 learning rate, which is just so outside the realm of normal for most people doing fine-tuning and anything with AI at the time. And then there's also this idea of synthetic data, which is this direct distillation idea. So there was a dataset called UltraFeedback from a group that's like OpenBMB, which I think is a research group based in China. That dataset, still to this day, if you compare to other preference datasets like Stack Exchange, Stanford Human Preferences, or Anthropic's HH-RLHF dataset, it's just using the methods that we have on this UltraFeedback data, which is a mix of completions from models like GPT-4, GPT-3.5, Llama 2, and then the chosen and rejected completions for creating so-called preferences are labeled by GPT-4. So this dataset is 6 months on from Zephyr. Tulu is still what we're seeing as the best. By the time this is aired, we'll release some more models trained with DPO or soon after, and we're still like the best results still come from using this dataset. So as this conversation goes on, I'll probably keep beating my drum, which is: we need to get people making more datasets in the open if we want to keep doing this academically and for open source, but do that in a lot of channels. But it's obvious the field will move so much. We have these Llama 3 models, I'm sure Mistral will come out with something soon, but we're still using the same one dataset. It was the first one of its class too. People don't get the first one perfect, especially when it's a research lab, but it's impressive the longevity of it.

除了 Zephyr，你在 AI2 还发布了 Tulu 2，如果我发音正确的话。第一个 'u' 上面是不是有一条线？

In addition to Zephyr, while at AI2 you also released Tulu 2, if I'm pronouncing that correctly. There's like an 'o' with a line over the first 'u'?

是的，它是一种杂交骆驼。Tulu 就是这个意思。我花了很长时间才学会这个。有两种骆驼，Tulu 是它们杂交的结果。名字不是我起的，但就是这么来的。所以我想它们是骆驼中的骡子。

Yeah, it's a hybrid camel. It's what a Tulu is. It took me so long to learn this. There are two types of camels, and the Tulu is what happens if you crossbreed them. I didn't come up with the name, but that's what it's named after. So they're the mule equivalent for camels, I think.

好吧，不管这个词的词源如何，Tulu 是一套用于使预训练语言模型适应下游任务和用户偏好的模型。Tulu 是如何在其他指令微调模型和方法失败的地方适应的呢？

Well, regardless of the etymology of the term, Tulu is a suite of models for adapting pre-trained language models to downstream tasks and user preferences. How does Tulu adapt where other instruction tuning models and methods fail?

是的，这实际上是这个项目的主体。它是关于理解最重要的指令数据，然后我们看到 Zephyr 出现了，所以让我们把 Zephyr 方法应用在上面。Zephyr 方法再次在这些模型上奏效，所以这有点像概念验证，证明 UltraFeedback 数据在不同规模下都有效。所以实际上是 700 亿规模，这再次是为了反驳 DPO 的怀疑者。每个人都像，‘哦，DPO 在 70 亿规模有效，但如果它不能扩展，没人会真正使用它。’然后一个月后，‘哦，看，我们做到了。’这有点好笑。但是有很多指令数据集，这变得非常混乱。我认为如果你深入指令微调的细节，你还会看到这些独立附属的人，他们在交易所有这些瓶子并上传到 Hugging Face，他们说，‘我们的数据集有来自 15 个不同地方的列表加上一些奇怪的过滤启发式方法的一百万个例子。’而 Tulu 是它的学术版本，专注于非常具体的评估指标，试图理解特别是代码和推理中一些更难用当前模型改进的东西，并进行严格的研究，研究如何改进它们。这有点像两阶段的事情，包装成一个结果：如何用我们拥有的所有数据改进这些，然后你有一个巨大的数据集，如何修剪它以保持最大性能。在 Tulu 方面，这看起来像是研究生们运行大量实验，深入细节。没有一个系统的答案。下一件事是做自动过滤。所以研究生们说，‘离开这个项目’，他们说，‘我们不能再手工做了’，他们正在研究基于嵌入的自动过滤方法，或者他们正在研究影响函数，我甚至不知道它们技术上是什么，但它是一种指令之间相似性的度量，并试图用来自动化过滤在线大量指令数据的过程。

Yeah, so this is really the bulk of this project. It was about understanding the most important instruction data out there, and then we saw Zephyr come out, so let's apply the Zephyr method on top of it. And like the Zephyr method again worked on these models, so it's kind of like a proof of concept that this UltraFeedback data works at different scales. So it's really the 70B scale, which again was to counter the DPO skeptics. Everyone's like, 'Oh, DPO works at 7B, no one's going to really use this if it doesn't scale up.' And like a month later, 'Oh look, we did it.' It was just kind of funny. But there's a lot of instruction datasets out there, and this becomes very messy. I think if you're deep in the weeds of instruction tuning, you also see these people that are like independent affiliations that are trading all these bottles and uploading them to Hugging Face, and they're like, 'We have a million examples in their dataset from this list of 15 different places plus some weird filtering heuristics.' And Tulu is like the academic version of that, which is focusing on very specific evaluation metrics, trying to understand particularly some things in code and reasoning that are harder to improve with the current models, and doing a rigorous study on how you can improve them. It's kind of a two-stage thing that's wrapped into one result: how you can improve these with all the data that we have, and then you have a giant dataset and how do you prune it down to maintain the maximum performance. And what this looks like in terms of Tulu is grad students running a ton of experiments and just getting really deep into the weeds. And there isn't a systematic answer. It's like the next thing is to do automatic filtering. So the grad students say, 'Coke away from this project,' and they're like, 'We can't do this by hand anymore,' and they're looking into ways of doing automatic filtering based on embeddings, or they're looking at influence functions, which I don't even know what they technically are, but it's like a measure of similarity between instructions and trying to use this to kind of automate the process of filtering through the vast amounts of instruction data online.

不错。是的，你在 AI2 做的所有事情都很酷。那一定是一个不可思议的工作场所，听起来你搬到那里是为了更好地利用这一点，搬到西雅图。

Nice. Yeah, really cool all the things you're doing at AI2. It must be an incredible place to work, and it sounds like you moving there is to take even better advantage of that, moving out to Seattle.

是的，很好。它就像是学术界和工业界混合的最后坚持者之一。就像每个工作场所，我说每个人都有工作，总有优点和缺点，但这是一个非常独特的地方。随着工业研究变得更加封闭，但我们没有工业研究的资源，所以我们在做事的方式上必须更聪明一些。

Yeah, it's good. It's like one of the last holdouts of academia and industry hybrid. Like every workplace, I say everyone has a job, there's always upsides and downsides, but it's a very unique place. As industry research has become more closed, but we don't have the resources of industry research, so we have to be a bit more clever in terms of how we do things.

在构建生成式 AI 应用程序时，您是否在优化延迟和降低推理成本之间左右为难？了解为什么越来越多的 ML 开发人员转向 AWS Trainium 和 Inferentia 来构建和服务他们的大语言模型。使用 AWS Trainium 芯片，您可以节省高达 50% 的训练成本，使用 AWS Inferentia 芯片，您可以节省高达 40% 的推理成本。Trainium 和 Inferentia 将帮助您实现更高的性能、更低的成本和更高的可持续性。查看节目说明中的链接了解更多信息。好了，现在回到我们的节目。

Are you stuck between optimizing latency and lowering your inference costs as you build your generative AI applications? Find out why more ML developers are moving toward AWS Trainium and Inferentia to build and serve their large language models. You can save up to 50% on training costs with AWS Trainium chips and up to 40% on inference costs with AWS Inferentia chips. Trainium and Inferentia will help you achieve higher performance, lower costs, and be more sustainable. Check out the links in the show notes to learn more. All right, now back to our show.

所以在 AI2 之前，相当长一段时间，因为你也曾在 Hugging Face 工作，但在此之前你在加州大学伯克利分校，你……

So prior to AI2, quite a bit, because you were also at Hugging Face, but prior to that you were at UC Berkeley and you...

聚焦在机器人与机器学习的交叉领域，这对我来说现在非常迷人。最近这个领域有一些非常激动人心的发展，比如英伟达的 Project Groot，用于人形机器人，结合了生成式 AI 和强化学习。还有麻省理工学院的衍生公司 Liquid AI 宣布，计划用液态神经网络彻底改变机器人技术。我甚至不知道他们是一家机器人公司。我见过他们，但不知道。那么，在 LLM 和机器人技术的交叉点上，什么让你感到兴奋？那里有什么前景？

Focused on the intersection of robotics and machine learning, which for me is personally super fascinating right now. There have been some really exciting developments in that space recently, things like NVIDIA's Project Groot for humanoid robots using generative AI and reinforcement learning. And there's also the announcement of an MIT spin-off, Liquid AI, which plans to revolutionize robotics with liquid neural networks. I didn't even know they were a robotics company. I saw them but I didn't know. So what's exciting for you at this intersection of LLMs and robotics? What's promising there for us?

我认为这一切真正开始的地方是 Google Brain 的机器人研究团队。他们仍然在做伟大的事情，但他们提前几年就接受了这一点：扩大我们的数据引擎，训练一些大模型，然后它就奏效了。随着人们在这个数据上投入资源，这显然会继续在越来越复杂的任务上发挥作用。存在一个产品市场契合度的问题，比如我不想买机器人。所以大多数细微的观点都来自产品方面。我相当有信心研究将继续取得进展。我认为过去几年机器人和机器学习的两大趋势是：一是利用某种大模型扩大数据收集，你可以获得有效的真实世界结果。谷歌展示了这一点，其他地方也复制了，他们做了这个开放数据集项目。但还有深度强化学习的实际成功领域，它已经缩小到这种机器人模拟，其中你有程序生成的世界，你为机器人进行模拟。我觉得我应该再做一个调查，但大约一年前我写了一篇博客文章，列出了很多成功的地方：无人机飞行、运动等等。DeepMind 有核聚变论文。所有这些非常疯狂的事情都有非常狭窄范围的深度强化学习帮助。所以我认为大多数机器人公司将利用这一点：我们在内部有机器人农场可以收集数据，然后是如何整合消费者数据的问题。如果你试图做人形机器人，这很难，因为在机械层面上，大多数人形机器人的力量如此之大，以至于很难让它们靠近人类。我认为这就是在 1X Robotics 工作的 Eric Jang 试图做的：制造力量更低的执行器，这样让它们靠近人类更安全。众所周知，波士顿动力的 Atlas，你不能让人类靠近它，因为如果它的手臂在运动时击中你，你会因为力量太大而飞过房间。那不安全。所以存在这个奇怪的最后一英里考虑。但后来我和一个家庭朋友聊天，他说：“远程操作可以拯救你。”所以如果你想在你的房子里有机器人，很明显它们在很多事情上不会工作，但你可以把劳动外包给印度，会有人乐意手动清空你的洗碗机，而不是机器人自动失败。这创造了劳动力市场的重新分配，套利成本等等。我实际上认为如果人们克服隐私问题，这可能会奏效。我中间跳过了一些东西：让机器人进入你的房子可能很长一段时间内都行不通，因为分布偏移。但我认为大多数认真对待这件事的人，如果分布偏移是这么大的问题，那就是你进行远程操作的时候。我尊重现在加入这个领域的很多人，因为在语言模型空间中有很多机会可以抓住，在数字应用和构建服务方面。那些仍在做基础研究或去机器人领域的人，你们是在做长期的事情。我认为 Eric Jang 特别说过：“这是对人形机器人 10 到 20 年的赌注。”我当时想，尊重你承担大风险，因为它确实看起来在朝着正确的方向发展。机器人技术，如果你去掉 Stable Diffusion 时刻和 ChatGPT 时刻，机器人技术的趋势线是一样的，只是缓慢上升，我们正在引入新东西。所以它没有那么大的轰动效应。轰动来自像 Elon 现在营销它的人。特斯拉 Optimus 可能会类似于 Autopilot：我不认为它完全像营销的那样，但他们有一个非常好的团队，他们正在建造一个很酷的机器人，这种不匹配将在下游以某种方式管理。

I think the place this all really started was Google Brain's research team on robotics. They're still doing great things, but they were years ahead in embracing this: let's scale up our data engine, let's train some big models, and then it kind of just worked. That will clearly continue on more and more complex tasks as people invest resources in this data. There's kind of a product-market fit issue, which is like, I don't want to buy a robot. So most of the nuanced takes come on the product side. I'm pretty confident that the research is going to keep going places. I think the two biggest trends in robotics and machine learning in the last few years, in my mind, are: one of them is scaling up data collection using some sort of large model, you can get real-world results that work. Google showed this, other places have replicated it, they did this open dataset project. But there's also deep RL's actual success area, which has narrowed down to this kind of simulation for robotics, where you have procedurally generated worlds and you simulate for robotics. I feel like I should try to do another survey on this, but I wrote a blog post about a year ago that was just listing a whole bunch of places where that has worked: drone flight, locomotion, other things. DeepMind had the nuclear fusion paper. There are all these really wild things that have really narrow-scoped deep RL helping with. So I think that's what most robotics companies will be leveraging: we have our robot farm internally that can collect data, and then it's the question of how do you integrate consumer data. If you're trying to do humanoids, it's hard because at a mechanical level, most humanoid robots have such high force that it's hard to have them around humans. I think this is what Eric Jang, who works at 1X Robotics, is trying to do: make actuators that are lower force so it's safer to have them around humans. Famously, Boston Dynamics' Atlas, you can't have humans around it because if its arm is doing a motion and it hits you, you go flying across the room because there's so much force. That's not safe. So there's this weird last-mile consideration. But then I was talking to a family friend, and he said, 'Teleoperation could save you.' So if you want to have humans in your house or robots in your house of any type, it's obvious that they're not going to work for many things, but you could outsource the labor to India, and there will be people that will happily empty your dishwasher manually instead of the robot failing to do it automatically. That creates a whole redistribution of the labor market, which arbitrages costs and stuff. I actually think that would probably work if people got over the privacy concerns. I kind of skipped something in the middle: having robots in your house probably won't work for a really long time because of the distribution shift. But I think most people who are serious about that, and then if the distribution shift is such a big problem, that's when you do teleoperation. I respect a lot of people that are joining this field right now because there's a lot of opportunity to grab in the language model space in terms of digital applications and building services. The people that are still doing fundamental research or the people that go to robotics, you all are taking the long-term thing. I think Eric Jang specifically was like, 'Yeah, this is a 10 to 20 year bet for humanoid robotics.' And I was like, respect for taking the big risk, because it does seem to be going in the right direction. Robotics has been, if you take away the Stable Diffusion moment and the ChatGPT moment, the robotics trend line is just the same, slowly going up, and we're pulling in new things. So it doesn't have as much of a splash factor. The splash is from people like Elon marketing it now. Tesla Optimus is probably going to be similar to Autopilot: I don't really think of it as exactly what it's marketed as, but they have a really good team there and they're building a cool robot, and that mismatch will be managed in some downstream way.

好的，明白了。所以基本上你认为我们还需要一段时间才能让人形机器人在家里做很多日常任务。所以我们将会看到越来越多的机器人应用是在工业领域。

Nice, gotcha. So basically you think that it's going to be some time before we have humanoid robots in our homes doing a lot of regular tasks. So where we're going to see more and more robotics applications is in industry, typically.

是的。所以有三家非常受欢迎的湾区机器人和 AI 初创公司：Dexterity、Ambi 和 Covariant。它们都与公司签订了各种物流任务的合同，比如拾取和放置、卸卡车或装载托盘，而且它们都运行得很好。亚马逊就是这样做的：亚马逊正在将其配送中心设置为机器人优先。他们从头开始建造整个配送中心，为机器人做好准备，而不是为人类做好准备，也不是用机器人替代人类。所以这一切都真的有效。问题是如何让它们离开生产线之类的东西，这完全不同。

Yeah. So there are three really popular Bay Area robotics and AI startups: Dexterity, Ambi, and Covariant. All of them have contracts with companies for various logistics tasks, like pick and place, unloading a truck, or loading a pallet, and they all work really well. Amazon does this: Amazon is setting up their fulfillment centers to be robot-first. They build entire fulfillment centers from the ground up to be ready for robots, rather than being ready for humans, rather than subbing robots in for where there were humans. So all of this really works. It's like how do you get them to leave the manufacturing line type of thing, which is just so different.

是的，是的。所以可能相关的是，我们现在更频繁地看到 LLM 在机器人应用中出现，比如之前的 Groot 解释，你刚才提到了 Covariant，他们也有一个非常酷的，他们的机器人基础模型 RFM-1。这些东西是实现人机交互的方式，让你的机器人拥有语言是途径。只是需要很高的可靠性才能让人想要购买它。

Yeah, yeah, yeah. And so related potentially because of how we're now seeing LLMs more frequently in robotics applications, like the Groot explanation from earlier, and you just mentioned Covariant, they had a really cool one too, their robotic foundation model RFM-1. That stuff is the way to get to human interaction, having language with your robot is the way. It's just going to take a lot of reliability for somebody to want to buy it.

这看起来很合乎逻辑。当然很酷。

It seems so logical. It's cool for sure.

那么关于 LLM，你在你的通讯中最近写了一篇关于 GPT-4o 的文章，其中提到了显著的改进，比如实时音频生成的延迟。在你看来，这些技术突破中哪些将深刻影响科技行业以外的领域，比如医疗保健、教育、金融，甚至可能进入机器人领域？哪两个？

And so related to LLMs, in your newsletter you recently wrote an article about GPT-4o that featured significant improvements such as latency in real-time audio generation. In your opinion, which of these technical breakthroughs will profoundly impact industries beyond the tech sector, like healthcare, education, finance, perhaps even making it into robotics? What are the two?

我认为音频是一个大突破。

I think audio is a big one.

我认为很明显，人们喜欢通过音频消费大量媒体。我们在 YouTube、电视和 TikTok 上都看到了这一点。看 TikTok 的人数与阅读 Substack 新闻通讯的人数相比非常低。但这消除了使用语言模型的门槛。GPT-250 存在时，可能会在每种语言上都表现得非常好。你可以把它放在一个不会说英语的孩子面前，他就有了一个完美的导师，即使大多数教育材料都是英语的。ChatGPT 已经学会了所有这些，它就在那里。教育的下游可及性非常高。当然有社会担忧，但也涉及更小的孩子。他们不需要能够写出连贯的问题。如果孩子要使用这个，你需要添加保护措施，但你可以列出无穷无尽的好处，尤其是与这些无延迟的机器对话，而孩子们非常聪明。我的父母可能会更困惑，或者需要更长时间适应一个可以被打断的 AI，但一个孩子可能 60 秒就搞定了，然后他们就自己探索了。他们甚至可能不会像对待正常人那样与它交谈；他们会以某种难以解析的奇怪方式提取信息。

I think it's clear that people like to consume a lot of media through audio. We've seen this with YouTube, TV, and TikTok. The number of people who watch TikTok versus read Substack newsletters is very low. But that removes a barrier to entry for using language models. GPT-250, when it exists, will probably be so good in every language. You could put it in front of a kid who doesn't speak English, and they have a perfect tutor for anything, even though most educational material is in English. ChatGPT has already learned all that, and it just exists. The downstream accessibility to education is just so high. There are social concerns, but also starting younger. They don't need to be able to write coherent questions. There are protections you need to add if kids are going to use this, but you could come up with an infinite list of benefits from talking to these machines with no latency, especially since kids are so clever. My parents would probably be more thrown off or take longer to adapt to an AI they can interrupt, but a kid probably figures it out in 60 seconds and then they're off. They probably won't even talk to it like a normal human; they'll extract information in some unparsable weird way.

是的，我认为你说得对。我仍然搞不懂 TikTok。我只是保护自己。我在科技行业太深了，不知道。所有科技孩子都受到这些应用的保护。真是个骗局。那么回到 RLHF，这不仅是你在艾伦研究所的重点，也是你之前在 Hugging Face 的重点。在你的论文《对齐天花板》中，你讨论了 RLHF 中的目标不匹配问题。以可重复的方式将奖励模型与人类偏好对齐面临哪些挑战？

Yeah, I think you're right. I still can't figure out TikTok. I just protect myself. I'm too deep in the tech industry to know. All the tech kids are protected from these apps. It's such a sham. So going back to RLHF, which has been a huge focus of yours, not only at the Allen Institute but also at Hugging Face where you were previously. In your paper 'The Alignment Ceiling', you discuss the issue of objective mismatch in RLHF. What are the challenges in aligning reward models with human preference in a reproducible manner?

这篇论文有个有趣的故事，因为它要追溯到我读博期间。我博士期间关于基于模型的强化学习的工作，比如目标不匹配论文，是我论文的核心。在基于模型的强化学习中，你学习一个策略和一个动力学模型。这更简单，因为评估环境更封闭。在经典强化学习中，你有模拟中的机器人任务，所以评估更固定。你可以认为，对策略好的动力学模型是针对策略调整的，而不是针对真实世界。在 RLHF 中，情况不同，因为你是在进行多阶段过程。你把语言模型作为策略，奖励模型作为你的环境，但它反映了人类想要的。所以多了一个环节，试图匹配人类想要的，然后你有奖励模型，接着策略从中提取信息。有很多类比。我一直在谈论信息流：如果你有这个正在训练的策略，奖励模型是某种过滤器、筛子或增益。你需要调整它。你通过一个黑箱传递信息，需要调整它以匹配人类实际想要的。这也是模型规范有趣的部分原因。进一步强调模型规范部分揭示的是，当大公司从人类那里收集偏好数据时，他们有 10 到 20 页的文档，说明在标注数据时应该优先考虑什么。我们一直未能看到的问题是，数据中告诉你要做的事情与最终模型实际做的事情之间的不匹配。如果你告诉它们优先考虑事实性或简洁性，即使数据中有这些，训练过程是否会产生一个这样做的模型？这就是对齐天花板：我们不知道我们的方法是否能够完美地与我们的期望对齐，因为我们在不同的模块中这样做。在深度强化学习时代，人们会问我们是否可以做端到端学习。如果我们只有一个目标，它能同时学习所有东西吗？这在现实的工程环境中似乎不太能扩展。唯一这样做的人是特斯拉的自动驾驶，但我们不确定。OpenAI、Gemini、Anthropic 的所有团队都有模块：一个 RLHF 团队、一个安全团队、一个预训练团队。这就是他们权衡这些事项的地方。如果你看这篇论文，每个边界都是你试图在他人优化的背景下设计自己的优化。这是一个数学问题：如果你在做多个优化问题，你永远无法得到完美的解决方案。

This paper is a fun story because it hearkens back to my PhD. My PhD work on model-based RL, like the objective mismatch paper, was a core paper of my thesis. In model-based RL, you learn a policy and a dynamics model. It's simpler because the evaluation regime is more closed. In classic RL, you have robot tasks in simulation, so the evaluation is more set. You can think of it as the dynamics model that is good for the policy is tuned to the policy, not to the real world. In RLHF, it's different because you're doing a multi-stage process. You have the language model as your policy, and the reward model as sort of your environment, but it mirrors what humans want. So there's this extra leg of trying to match what humans want, then you have the reward model, and then the policy extracting information from it. There are a lot of analogies. I've been talking about information flow: if you have this policy being trained, the reward model is some sort of filter or sieve or gain. You need to tune it. You're putting information through a black box and need to tune it to match what humans actually want. This is part of why the model spec is interesting. Doubling down on what the model spec partially reveals is that when big companies collect preference data from humans, they have 10 to 20 page documents on what to prioritize when labeling data. The question we haven't been able to see is the mismatch between what they tell you to do in the data and what the final model does. If you tell them to prioritize factuality or conciseness, even if the data has that, does the training process result in a model that does this? That's the alignment ceiling: we don't know if our methods could ever be perfectly aligned with our expectations because we're doing this in different modules. In deep RL days, people would ask if we can just do end-to-end learning. If we have one objective, can it learn everything at once? That doesn't seem to scale well in realistic engineering environments. The only people doing that is Tesla self-driving, and we don't know for sure. All the teams at OpenAI, Gemini, Anthropic have modules: an RLHF team, a safety team, a pre-training team. That's where they trade off these things. If you look at this paper, each of those boundaries is where you try to design your optimization in the context of other people's optimizations. It's a mathematical thing: you can never get a perfect solution if you're doing multiple optimization problems.

RLHF 的一个主要替代方案是宪法 AI 或基于 AI 反馈的强化学习。你在其他地方、论文和演讲中讨论过这个。你能详细说明这个想法吗？我知道 Anthropic 是它的主要支持者。有些人可能会担心，如果你用 AI 来评判 AI，真正的基准真理在哪里？

A big alternative to RLHF is constitutional AI or reinforcement learning from AI feedback. You've discussed that in other places, papers, talks. Can you elaborate on this idea? I know Anthropic is a big proponent of it. Some people might be concerned that if you're using AI to judge AI, where is the real ground truth?

宪法 AI 是最被误解的技术之一，主要是因为论文有些令人困惑。这篇论文中有两个主要部分：宪法 AI 和 RLAIF。RLAIF 是使用 AI 而不是人类来标注偏好的想法，这相当通用。宪法 AI 是一个两阶段过程，包含一些 RLHF 和一些指令修订。他们在论文中所做的是根据一系列原则修订指令数据集中的指令。这是一件事。然后他们重新进行指令微调。另一件事是他们根据原则的上下文重新进行偏好标注。这就是人们通常认为的。这只是添加合成数据的另一种方式。我认为他们现在可能已经远远超越了这一点；这篇论文相当老了。在这种情况下你可以做的其他一些事情……

Constitutional AI is one of the most misunderstood techniques, mostly because the paper is somewhat confusing. There are two major things in this paper: Constitutional AI and RLAIF. RLAIF is the idea of using AIs instead of humans to label preferences, which is pretty general. Constitutional AI is a two-stage process that does some RLHF and some instruction revision. What they do in the paper is revise the instructions in their instruction dataset with respect to a list of principles. That's one thing. Then they redo instruction fine-tuning. The other thing is they redo this preference label with the context of a principle. That's what people normally think about. It's just one more way of adding synthetic data. I think they've likely moved well beyond that at this point; the paper is pretty old. Some other sorts of things you could do in this case...

比如你可以通过修订来创建偏好数据。如果你有一堆语言模型的输出，你可以问它：‘这个事实正确吗？’如果它说不对，你就说：‘你能修正它吗？’如果它修正了，你就得到一个成对偏好，其中选择的偏好是修正后的文本，被拒绝的是原来需要修正的文本。这是越来越多生成合成数据的方法之一，其中 CI 是 Anthropic 引起很多关注的那个。这个名字很热门，它是最早的，但被严重误解了。我的意思是，直到几个月前我才真正理解它。它就是那种最终含义与实际情况不同的论文之一，这没关系。这种事常有。

Like you can do a revision to create a preference data. So if you have a bunch of completions from a language model, you could ask it, 'Is this factually correct?' And if it says no, you say, 'Can you fix it?' And if it fixes it, then you have a pairwise preference where the chosen preference is the fixed text and the rejected is the original one that needed to be fixed. This is one of a growing list of examples of ways that you could generate synthetic data, of which CI is the one that Anthropic got a lot of buzz on. It's a buzzy name, it was by far the earliest, but it's so misunderstood. I mean, I don't feel like I really understood it until a couple months ago. It's just one of those papers that ends up meaning something different than it actually is, which is fine. It happens.

好了，回到 RLHF。考虑到人类偏好固有的主观性，你如何确保聚合后的偏好能准确反映我们 AI 系统行为的期望结果？

Alright, so back to RLHF. Given the inherent subjectivity in human preferences, how do you ensure that the aggregated preferences accurately reflect the desired outcomes for our AI systems' behavior?

是的，我们最近内部一直在争论这个问题。本质上你可以把它表述为：标注者之间的分歧是信号还是错误？老实说，我更觉得它是一种信号，因为这些偏好非常模糊且多面。我们不知道每个人在做什么。有关于这种微调 RLHF 的研究，你会标注答案的多个部分，但最终很多都被简化为一对。我们对不同方面的关注权重不同，比如事实性、简洁性、有用性、诚实性——这些非常抽象的术语。所以如果被简化，就会有一些噪声。这在论文中可以看到，他们都报告了与标注者的一致性数字，研究人员将自己的数字与标注者比较时，一致性大约在 65%到 75%之间。我不认为这个问题会消失。人们看到，当使用语言模型时，一致性更高。我对这意味着什么没有强烈的看法。这可能只是放大偏见的情况之一。你不断用语言模型训练，它们的分歧更少，一致性更高，这表现为答案在可接受性方面的多样性可能下降。我们真的不知道。这又回到了我之前强调的：这就是为什么我们需要更清晰标注、带有良好元数据的开放偏好数据集。这样我们才能看到。我现在鼓励很多人做的一件事是：我们有三个非常好的语言模型。如果你要用 GPT-4 作为评判来标注数据集，那就用所有三个模型——也用 Claude 和 Gemini 来做。然后我们就能看到语言模型之间的分歧，并且我们可以大规模地获得这些数据。我们就能学到更多。

Yeah, we've been debating this a lot internally recently. It's essentially you could phrase it as: is the disagreement among labelers a signal or a bug? And it honestly feels more to me that it is kind of a signal, because of how vague and multifaceted these preferences are. It's like, we don't know what everyone's doing. There's research on this kind of fine-tuned RLHF where you label multiple pieces of an answer, but at the end of the day, a lot of it gets reduced to a pair. And we have different weights for how much we notice different things like factuality, conciseness, helpfulness, honesty—these very abstract terms. So if this gets reduced, there's going to be some noise. And that's seen in the papers where they all report these agreement numbers with their annotators, and it's somewhere between 65% and 75% agreement when the people doing research compare their numbers to the annotators. I don't think that's going away. People see that when they use a language model, it has higher agreement. I don't have a strong opinion on what that means. It could just be one of those things where you're amplifying biases. You keep training with language models, they have less disagreements, more agreement, and what that manifests as is the diversity of answers maybe going down in terms of what's acceptable. We really don't know. This is back to where I said I was going to beat this drum: this is why we need more clearly labeled, with good metadata, open preference datasets. So we can kind of see. Something that I'm encouraging a lot of people to do now is: we have three really good language models. If you're going to use GPT-4 as a judge to label a dataset, do it with all three of them—do it with Claude, do it with Gemini as well. And then we can kind of see what the disagreement between the language models is, and we get that at scale. We start to learn a lot more.

今天的播客节目由 CrawlBase 赞助，这是为数据科学家、AI 开发者和 Python 开发者量身定制的终极数据爬取和抓取平台。对于机器学习和 AI 来说，高质量数据当然至关重要。使用 CrawlBase，你获得一个强大且用户友好的解决方案，保证无缝集成、闪电般的性能和无与伦比的可靠性。CrawlBase 通过 2 分钟集成流程、AI 驱动效率和 99.99%的正常运行时间满足你的需求。CrawlBase 在绕过验证码、避免 IP 封锁和处理代理故障方面也表现出色，使其成为你所有数据需求的首选解决方案。使用特殊代码'superdatascience'（无空格）解锁 10,000 次免费请求。今天访问 CrawlBase，用业界最好的工具加速你的数据收集过程。很好，非常酷。你做的另一件与 RLHF 相关的很酷的事情是，你将其追溯到古代哲学和现代经济学，比如提到了亚里士多德和冯·诺依曼-摩根斯坦效用定理。我不太了解 VNM 效用定理，但这些历史基础如何影响当前的方法论？现代 AI 研究可以从这些早期理论中学到什么？

Today's podcast episode is brought to you by CrawlBase, the ultimate data crawling and scraping platform tailored for data scientists, AI developers, and Python developers. For ML and AI, high-quality data are of course essential. With CrawlBase, you get a powerful, user-friendly solution that guarantees seamless integration, lightning-fast performance, and unparalleled reliability. CrawlBase supports your needs with a 2-minute integration process, AI-powered efficiency, and 99.99% uptime. CrawlBase also excels in bypassing CAPTCHAs, avoiding IP blocks, and handling proxy failures, making them the go-to solution for all your data needs. Use the special code 'superdatascience' (no spaces) to unlock 10,000 free requests. Visit CrawlBase today and supercharge your data collection process with the best in the business. Nice, very cool. Another really cool thing you've done related to RLHF is you have traced it back to ancient philosophy and modern economics, so mentioning Aristotle and the von Neumann–Morgenstern utility theorem, for example. I don't really know what the VNM utility theorem is, but how do these historical foundations influence current methodologies, and what can modern AI research learn from these early theories?

是的，这是一篇有趣的论文，与我在伯克利开始合作的几位同事一起，现在我们分散各地。这都基于一个事实：强化学习有着非常深厚、多领域、跨学科的历史，可以追溯到很久以前。而偏好的概念在经济学中非常模糊。冯·诺依曼-摩根斯坦理论是一个基础理论，它本质上说你可以将所有行为或所有目标表示为概率和期望值分布，这让你可以对偏好进行期望值计算。然后引发了一系列争论：偏好是否真的存在，是否在任何这些事物中都是可处理的，或者由于偏好随时间、基于上下文的变化，它们是否真的可测量。所以我们拿这些来问很多问题，关于它们如何影响现代 RLHF 过程。比如：最终模型的偏好——我们将其映射到非常人性化的术语上——实际上更多是基于偏好模型、从互联网抓取的基础模型，而不是来自 Scale AI 之类公司的人类偏好？所以如果它更多基于互联网爬取，而不是他们从 Scale AI 获得的百万美元数据集，这对营销来说有点混乱，或者我们说我们在学习一个偏好模型，但它可能实际上并没有做那么多。其他事情：OpenAI 现在有大量用户数据，经济学文献对于从用户上下文或专业上下文（有人付费去做，并被要求以某种方式行事）生成训练数据有什么说法？所有这些如何混合？所以这真的是一长串问题，关于如果我们对人类偏好等做出宏大声明，为什么我们应该关注其他社会科学。

Yeah, so this is a fun paper with a few colleagues that I started working with at Berkeley, and now we're kind of spread out. This is all based on the fact that RL has a very deep, multi-field, multi-disciplinary history where it goes way back. And the notion of preference is a very vague thing in economics. The von Neumann–Morgenstern theory is a foundational thing that essentially says you can express all behaviors or all goals as probability and expected value distributions, which essentially lets you do expected value math over preferences. And then it led to a bunch of debates on whether or not preferences actually exist and are tractable in any of these things, or if they are actually measurable or not due to preference shift over time, based on context. So these are the kind of things that we take and ask a lot of questions on how this impacts the modern RLHF process. It's things like: Is the final model's preferences—which we're mapping onto very human terms—actually based more on the preference model, the base model which is scraped from the internet, than the human preferences that they get from somewhere like Scale AI? So if it's based more on the internet crawling than this million-dollar dataset they're getting from Scale AI, it's kind of confusing to the marketing, or we're saying we're learning a preference model but it might not actually do that much. Other things: OpenAI now has a ton of user data, and it's like, what does the economics literature say about generating data for training that comes from a user context or a professional context where someone is paid to do it and they're paid to act in a certain way? And how does all this mix? So it's really just a super long list of questions of why we should look at other social sciences if we're making grand claims about human preferences and all of these things.

很好，非常迷人，我们的听众有很多可以深入挖掘的内容。我计划中与 RLHF 相关的最后一个话题——我相信它会在对话中自然再次出现——但你提到 RLHF 甚至对微调都不稳健，因此从 GPT-4 和 Llama 2 等模型中移除安全层会破坏安全概念。你能详细说明这种脆弱性对未来 AI 系统开发和部署的影响吗？

Nice, well fascinating, tons to dig into there for our listeners. Final kind of topic that I've planned related to RLHF—I'm sure it'll come up again organically in the conversation—but you've mentioned that RLHF is not even robust to fine-tuning, and so removing the safety layer from models like GPT-4 and Llama 2 can break down the notion of safety. Can you elaborate on the implications of this fragility for the future development and deployment of AI systems?

是的，这是一个特定的研究方向。有几篇论文表明，如果你拿一个像 Zephyr 或 Tulu 这样的模型——我们之前提到过——如果它们在数据集中有安全内容，然后你再用少量数据进一步微调它们，你可以移除这些安全内容。这是一个大问题，因为它意味着我们通过 RLHF 放入的对齐并不深入；它是表面的。所以如果有人拿一个基础模型，针对特定任务进行微调，他们可能无意或有意地移除安全约束。这对开源模型有影响，任何人都可以微调它们。这也意味着对于前沿模型，我们需要更好的方法来使安全性更稳健，也许通过对抗训练或更基础的对齐方法。这种脆弱性表明我们不能仅仅依赖 RLHF 作为最终的安全层；我们需要考虑遏制、监控和其他保障措施。

Yeah, so this is a specific line of research. There were a few papers that showed that if you take a model like Zephyr or Tulu that we were mentioning, if they have safety in the dataset, if you then go and fine-tune them further on a small amount of data, you can remove that safety. And this is a big concern because it means that the alignment we put in through RLHF is not deeply ingrained; it's superficial. So if someone takes a base model and fine-tunes it for a specific task, they might inadvertently or intentionally remove the safety constraints. This has implications for open-source models, where anyone can fine-tune them. It also means that for frontier models, we need better methods to make safety more robust, perhaps through techniques like adversarial training or more fundamental approaches to alignment. The fragility suggests that we cannot rely solely on RLHF as a final safety layer; we need to think about containment, monitoring, and other safeguards.

如果你在另一个任务上微调模型，就会失去一些根深蒂固的行为。老实说，我认为这更多是标题党，而不是真正令人担忧，因为考虑到所用的算力，这并不奇怪。我们在数万亿个 token 上预训练这些模型，然后在几十亿个 token 上微调——这只是轻微的调整，轻微的调整可以被撤销是合理的。但在更大的实验室里，安全更关乎一个完整的系统，而不是单个模型。开源权重模型不安全对我来说不是什么大事——如果你把它们放在一个免费的端点上让所有人使用，那我当然不希望我的模型说希特勒的好话。但作为需要自己启动 GPU 的研究产物，我更开放地接受多样化的模型存在。在 Anthropic，安全不仅仅是 RLHF；你需要在预训练、偏好模型以及输出安全过滤器上都下功夫。ChatGPT 会读取基础模型生成的所有文本，然后有一个通过/不通过的机制，如果收到不通过信号就会重新措辞——这就像双重保险。研究人员需要推销他们的工作，但这并没有看起来那么严重。这更像是下游的商业模式问题，涉及责任归属。如果你在自己的硬件上微调 LLaMA 模型，那没问题，但 OpenAI 的微调 API——如果他们声称自己的模型是安全的，但任何通过他们 API 的微调都会使其不安全，那这就是一个商业问题。开源生态系统可能更好，因为它打破了责任链。我们会看到这如何发展；这一切都还太早，我们才进行了一年。

If you fine-tune a model on a different task, you'll lose some of the ingrained behaviors. Honestly, I think this is more clickbaity than actually worrisome, because it's not surprising given the compute applied. We pre-train these models on trillions of tokens, then fine-tune on a couple billion tokens—it's a slight nudge, and it makes sense that a slight nudge could be undone. But in bigger labs, safety is more about a complete system than a single model. Open-weight models being unsafe isn't a big deal to me—if you put them on a free endpoint for everyone, then I don't want my model saying good things about Hitler. But as a research artifact you spin up GPUs for yourself, I'm more open to diverse models. At Anthropic, safety isn't just RLHF; you need safety at pre-training, in the preference model, and then a safety filter on the output. ChatGPT reads all text from the base model and has a go/no-go that rephrases if it gets a no-go signal—it's like a double layer. Researchers need to market their work, but it's not as big a deal as it seems. It's more of a business problem downstream with liability. If you fine-tune a LLaMA model on your own hardware, that's fine, but OpenAI's fine-tuning API—if they claim their model is safe but any fine-tuning on their API makes it unsafe, that's a business problem. The open ecosystem might be better off because it breaks the liability chain. We'll see how this evolves; it's so early, we're only a year in.

这是我没想到的——如果你通过 OpenAI 的 API 微调他们的模型，你可能会移除一些安全措施。

That's something I hadn't thought of—if you fine-tune an OpenAI model via their API, you could potentially remove some of the safety stuff.

这很大程度上关乎 AI 的文化。在实际操作中，感觉事情就是会成功。许多人在一个无法进行真正假设检验的规模上工作。我们在 7B 规模上做 10 个实验，然后基于直觉训练一个 35B 参数的模型，因为我们没有基础设施进行彻底的测试——适当的随机化和严谨的细节。所以这从定义上就不那么科学。有很多'我们试试这个，因为感觉对，而且可能有效'的情况。此外，公司把他们的东西包装成伪宗教文物，还有各种 AGI 的讨论，这让它变得不科学。我的联合主持人 Tom 很喜欢这个话题，但我理解这个论点并同意。深度学习不可解释，从根本上使得科学很难进行。

A lot of this is about the culture of AI. On the ground, it feels like things will just work. Many people operate at a scale where real hypothesis testing doesn't work. We do 10 experiments at the 7B scale, then train a 35B parameter model based on intuitions because we don't have the infrastructure for thorough testing—proper randomization and rigorous details. So it's definitionally not that scientific. There's a lot of 'let's try this because it feels right and probably works.' Plus, companies cast narratives about their things being pseudo-religious artifacts and all the AGI talk, which makes it unscientific. My co-host Tom loves this topic, but I understand the argument and agree. Deep learning being uninterpretable fundamentally makes it hard to do science.

令人疯狂的是，即使是开发者也不知道像 GPT-4 这样的模型会有什么涌现能力。这跟炼金术的想法有关。

It's wild that even the developers don't know what emergent capabilities a model like GPT-4 will have. That relates to the alchemy idea.

去年 Ryland Schafer 有一篇最佳论文，基本上认为许多涌现特性是由于统计测量误差。基准测试有 25%的随机下限，所以当你最终获得信号时，会出现奇怪的统计现象。如果你绘制对数算力与性能的关系图，它应该是一条直线，但有了噪声下限，它看起来是平的然后突然上升。这篇论文认为大多数这些说法都是测量噪声。我认为它介于那个和现实之间——我们确实在最大的模型上发现了不可预测的东西，但我们呈现它们的方式因为基准测试的创建方式而美化了涌现假说。这是那种应该写成博客文章的文章，因为想法非常清晰，但我们必须经过学术评审周期，所以最终成了一篇论文。挺搞笑的。

There was a best paper by Ryland Schafer last year that essentially said many emergent properties are due to statistical measurement error. Benchmarks have a random floor of 25%, so weird statistical things emerge when you finally get signal. If you plot log compute vs performance, it should be a straight line, but with a noise floor, it looks flat then jumps up. The paper argues that most of these claims are measurement noise. I think it's somewhere between that and reality—we are discovering unpredictable things with the largest models, but the way we present them flatters the emergence hypothesis because of how benchmarks were created. It's one of those papers that should be a blog post because the idea is so clear, but we have to go through the academic gating cycle, so it ended up as a paper. Pretty funny.

我一直在提出这个理论，即微调很重要，即使它没有那么多原始算力，因为呈现信息的方式非常重要。我喜欢用《人类简史》这本书作类比——那都是历史课的内容，但他用一种方式重写了……

I've been putting forth this theory that fine-tuning is important even if it's not as much raw compute, because how you present information is so important. I like the analogy from the book Sapiens—it's history class stuff, but he rewrote it in a way that...

这太引人入胜了，简直就像有史以来最畅销的书之一。而基于人类反馈的强化学习（RLHF）正在小规模地做同样的事情。所有这些基础模型都有相似之处，但真正能引起人们共鸣的模型，其输出方式恰好非常吸引人。所以这是 RLHF 作为风格迁移的基本案例，而且让模型的流畅度恰到好处仍然很重要。然后还有 OpenAI 大肆炒作的其他东西——其实他们自己不炒作，但泄密者在炒作——比如所有那些 QAR 的东西，以及在微调阶段添加额外的搜索。这些都是获取全新类型数据的各种方法，也是我一开始提到的不同损失函数。这是一种利用我们不再使用自回归损失这一事实的方式，看看它能让我们创造出多少不同类型的语言模型或其他类型的机器学习模型。我需要做一个关于这个的演讲——为什么人们看好 RLHF。我还没做；我觉得我需要学很多，因为很难做出超过两页的幻灯片。用策略梯度更新而不是自回归损失，这到底意味着什么？听到一个如此精通 RLHF 的专家这样描述它，真是不可思议。OpenAI 的某个人做过一个演讲，说了这个——我想是纽约大学的教授，也在——可能是他最后一次——我不知道怎么说，但我可以稍后找到。他的演讲里有一张幻灯片，写着“语言建模 101”，他就是那样介绍 RLHF 的。我当时想，这是个好方法。

It was so compelling that it's like one of the best-selling books of all time. And RLHF is doing that at a small scale. All these base models have similar things in them, but the models that really resonate with people happen to output in a way that is really compelling. So that's the base case of RLHF as style transfer, and it's still important to get the flow of the model right. Then there's other stuff that OpenAI makes a lot of funny noise about—well, they don't make noise about it, but the leaks do—like all this QAR stuff and adding extra search at the fine-tuning phase. That's various ways of getting very new types of data, and it's what I was talking about at the beginning with this different loss function. It's a way to exploit the fact that we're no longer doing autoregressive loss and see how far that lets us create different types of language models or other types of ML models. I need to make a talk on this—why people are bullish on RLHF. I haven't done it yet; I think I need to learn a lot about it because it's hard to make it more than two slides. What does it actually mean that you're doing these policy gradient updates rather than that autoregressive loss? It's wild to hear someone who is so expert in RLHF describe it like that. Someone from OpenAI gave a talk that said this—I think it was the NYU professor who's also at—maybe his last—I don't know how to say it, but I can find it later. He had a slide in his talk which was like "Language Modeling 101," and that was how he presented RLHF. I was like, that's a good way to do it.

除了我们到目前为止讨论的所有专业工作之外，大约一年前，你写了一篇通讯文章，题为《幕后：现在在 AI 工作是什么感觉——恐惧、错失恐惧症以及 ChatGPT 驱动的科学外流》。我完全也有这种感觉。似乎很难找到坚实的地面，一些恒定的东西，你可以投入理解它，并且它会在未来几年对你的职业生涯大有裨益。一切似乎都变化得太快了，这很可怕。我不知道你是否想多谈谈那篇文章，以及基本上已经稳定下来一点了。我认为这是一个过渡期，通向我们现在所处的位置。

Beyond all your professional work which we've discussed so far, about a year ago you wrote a newsletter article called "Behind the Curtain: What It Feels Like to Work in AI Right Now—Fear, FOMO, and the Scientific Exodus Driven by ChatGPT." I totally feel this too. It seems so hard to find terra firma, something constant that you can just invest in understanding that will be great for your career for years to come. Everything seems to be moving so quickly, and it is scary. I don't know if you want to talk more about that article and basically settled down a bit. I think this was a transition period to where we're at now.

抱歉打断你一下，但节奏就是这么快。我认为有些基础的东西你仍然需要学习。学习如何使用语言模型是好的。这几乎就像我刚开始读博士的时候：学习任何与深度学习、PyTorch 以及所有这些相关的东西都是好的。我认为 Hugging Face Transformers 是一个起点。玩不同的模型是好的。我认为行业里流行一种说法：“哦，他们的代码不太好，不够优化。”但如果你是一个一年级研究生，你可以轻松地玩大量模型，而这就是他们的业务所在：让人们能够使用这些。这些东西会有回报的。我玩一些愚蠢的 AI 东西——我把我的通讯转换成 AI 生成的语音内容——而习惯使用所有这些现在是一项基础技能，会在 10 到 20 年内带来回报。因为很可能仍然会有像 OpenAI API 这样的东西；它只会变得更好。这就是 Karpathy 的观点：语言模型就像一种新的计算机处理器。它们只是一个值得习惯的基础计算单元。这篇文章是在我们都在重新适应这个的时候写的。我认为现在实际上好一点了。仍然有很多人在抱怨。昨天，Yann LeCun 发推文说：“如果你想在 AI 领域产生真正的影响，就不要研究语言模型。”我觉得有太多人在设门槛，不让人说：“去兴奋起来，去尝试吧。”我引用并反驳了相反的观点：你可以过上好生活，同时研究语言模型。这不知为何引来了黑子。他们说：“这是个糟糕的观点。”我就想，为什么？你只需要去玩这些东西。如果你在构建东西，那些噪音就没那么重要了，因为你实际上在做事情，而不是坐在那里被这些随机的发布轰炸。它们大多数都不重要。脚踏实地地做事是好的。我不知道这是否回答了你的问题，但我不期望这会改变。这主要是由风投的疯狂驱动的，直到那些获得荒谬融资轮次的公司开始倒闭。我们会一直处于这个周期中，总是有这么多发布，因为这就是所有这些初创公司为了获得客户和公关需要做的。直到这些公司开始倒闭，情况才会一样。之后是整合阶段和崩溃，但我们不知道那会是一年后还是五年后。

Sorry to cut you off a little bit, but the pace is just so high. I think there are fundamentals that you still need to learn. Learning how to use language models is good. It's almost like when I started my PhD: learning anything to do with deep learning and PyTorch and all these things is good. I think Hugging Face Transformers is a place to start with things. It's good to play with different models. I think it's an in-vogue thing in the industry to be like, "Oh, their code isn't very good, it's not very optimized." But if you're a first-year grad student, it's easy to play with a ton of models, and that's what their business is about: enabling people to use this. And those things pay off. I play with stupid AI things—I transform my newsletter into AI-generated voice stuff—and just getting used to working with all of these things is now the fundamental skill that will pay off in 10 to 20 years. Because there still probably will be something like an OpenAI API; it'll just be much better. And that's Karpathy's take: that language models are like a new computer processor type of thing. They're just a fundamental computing unit worth getting used to. This article was when we were all kind of readjusting to this. I think it's actually a bit better now. There's still a lot of people complaining. Yesterday, Yann LeCun tweeted something like, "If you want to have true impact in AI, don't work on language models." I just feel like there's so much gatekeeping against telling people, "Just go be excited and try things." I quote-tweeted the opposite: you could have a nice life and work in language models. It definitely brings out the haters for some reason. They're like, "This is a bad take." It's like, why? You just have to go play with things. If you're building things, it's much less important what the noise is, because you're actually doing things rather than sitting back and getting bombarded with these random releases. Most of them don't matter. It's good to get grounded in actually doing stuff. I don't know if that answers your question, but I don't expect this to change. This is mainly driven by VC psycho until these companies that get really ridiculous funding rounds start to die. We're going to be in this cycle where there are so many releases all the time, because that's what all these startups need to do to get customer inbound and PR. Until these companies start dying, it's going to be the same. After that is this aggregation phase and collapse, but we don't know if that's going to be a year or five years from now.

我喜欢你的观点，LLM 能创造东西，它们提供了许多我们现在可以构建而以前不能构建的不同类型的应用。它们的另一个巨大优势是它们让我的工作流程变得容易得多，无论是内联代码助手还是像 Claude 这样的东西，我经常用它来问关于世界的问题。所以这很令人兴奋，但有时也会让人害怕。值得一读那篇文章来理解要点。如果你是个学生，那篇文章真的是从 Hugging Face 的每个人集体恐慌中构建出来的。我当时就想，我就在一个下午把它写出来，把所有这些都放在一页上。我认为它很好地捕捉了那种事情像病毒一样传播的时刻。我不认为我能重现它，因为我们现在处于一个更稳定的状态。我们不再处于那个超高熵的状态。虽然嘈杂，但大多数人都习惯了。

I like your point that LLMs make things, they provide so many different kinds of applications that we can be building now that we couldn't before. Another great advantage of them is that it makes my workflows a lot easier, whether it's an inline code assistant or something like Claude which I use for questions about the world all the time. So it's exciting, but it can be scary at times. It's worth reading that article to understand what the gist was. If you were a student, that was really built up from everyone at Hugging Face just freaking out collectively. I was just like, I'm just gonna fire up one afternoon and put all of this on a page. I think it's a good capture of the moment when something kind of goes viral like that. I don't think I could recreate it because we're in much more of a steady state now. We're no longer in this super high entropy state. It's noisy, but most people are used to it.

在能够拥有更好的工作与生活平衡、不害怕事情、始终把心理健康放在第一位方面——这是你在网站上写过的——你已经成功地将你对烹饪、健身和健康的热情融入了你的日常生活。这对你的工作方式有什么影响，你认为这对实现工作与生活的平衡有帮助吗？

In terms of being able to have a better work-life balance, to not be fearful of things, to allow mental health to always come first—which is something you've written on your website—you've managed to integrate your passion for cooking, fitness, and health into your daily routine. How does this influence your approach to work, and do you think this is helpful for achieving work-life balance?

是的，我认为……

Yeah, I think the...

对我而言，现实的做法是，我有点刻意地部分保留了我仍然自我认同的身份，因为这有助于我保持健康。我从大学和成长时期就有很长的耐力运动历史，现在仍然在做一些这类事情。这就像一棵基本的决策树：我睡得好吗，是还是不是？如果答案是否定的，我在训练中就能感觉到。我只是想在这些事情上感觉良好，我足够投入，以至于很明显不熬夜工作很重要。我不需要熬夜，但显然事情会发生。我不认为我在研究生院或任何地方熬过夜。这就是我的方式：训练、越野跑、到户外去。这主要是找到适合你的方法。我确实认为这是值得的。这也是我想亲自到场的原因之一。很多计算机科学及相关领域的人非常擅长优化和组织他们的生活，以至于远程办公如此方便，我可能会过度这样做。比如，‘哦，下午 5 点了，我得去做我的锻炼了。’如果你必须步行 20 分钟到办公室，这就会多一点结构，让你重新掌控自己的生活。这对我帮助很大。你只需要在生活中拥有一些让你做到这一点的东西，有一些你无法控制的事情和工作之外的事情。但这很难。我认为在疫情之后，这方面仍在恢复中。我的意思是，你经常做远程采访。我不知道如何平衡这一点。我考虑过为我的博客做采访，我正在考虑是否只做面对面的采访，也就是去参加会议，带上麦克风，看看能否找到有趣的人。这显然会降低我的产出，但这只是试图在我的生活中制定这些规则，强化做那些真正在现实世界中的事情。所以我不太去想它，而且我认为没有完美的答案。

The realistic take on what I do is I kind of contrived, partially what I still self-identify as, because it helps me be healthy. I have a long history of endurance sports from college and growing up, and I still do some of this stuff. It's like a basic rule tree: do I sleep well, yes or no? If the answer is no, I feel it during training. I just wanted to feel good at these things, and I lean into this enough to make it obvious that not working late is important. I don't need to, but obviously things come up. I don't think I've ever pulled an all-nighter in grad school or anything. This is how it works for me: training, trail running, getting outside. It's mostly finding what works for you. I do think it's worthwhile. It's part of why I want to be in person. A lot of people in CS and related fields are so good at optimizing and organizing their life that being remote is so convenient, I can overdo this. Like, 'Oh well, it's 5:00 PM, I got to go do my workout now.' If you have to walk 20 minutes into the office, it's just a bit more structure to take back control of your life. That's going to help me a lot. You just need to have things in your life that make that the case, to have some sort of things that you don't control and things outside of work. It's hard though. I think it's still coming out of COVID in that regard. I mean, you do remote interviews a lot of the time. I don't know how to balance that. I've thought about interviews for my blog, and I'm considering whether or not I only do them in person, which is go to a conference, bring a microphone, and see if I can get somebody interesting. That obviously reduces my throughput, but it's just trying to make these rules in my life that reinforce doing things that are actually in the real world. So I don't think about it a lot, and I don't think there are perfect answers.

是的，这太疯狂了。对我个人来说，疫情，我的意思是，它仍然在很多你描述的方式上对我产生负面影响。因为我过去有去办公室、整天和同事在一起的日常，我真的很享受工作。现在我大部分时间在家办公，我甚至，你知道，你谈到了过度优化。我选了一个对面有 CrossFit 健身房的公寓，但现在我有点后悔这个决定，因为我可以在公寓里度过一整天，而我在典型的一天里唯一要做的事情就是去街对面的健身房。所以，这并不，我不像，是的，我过度优化了。我自己也在解决一些问题。

Yeah, it's crazy. For me personally, the pandemic was, I mean it still continues to have a negative impact on me in a lot of the ways that you're describing. Because I used to have this routine of going to the office and being around co-workers all day, and I really enjoyed being at work. Now I mostly work from a home office, and I've even, you know, you talked about over-optimizing. I picked an apartment that has a CrossFit gym across the street, but now I kind of regret that decision because I can spend my whole day in my apartment, and the only thing that I go to do on a typical day is go down to the gym that's across the street. So it's not, I'm not like, yeah, I've over-optimized. Working some things out there myself anyway.

总之，非常感谢你这次关于机器人研究，特别是 RLHF 的精彩而丰富的对话。在我让嘉宾离开之前，我总是会问一个书籍推荐。不知道你是否碰巧有一个给我们？

Anyway, so thank you so much for this great rich conversation on robotics research and RLHF in particular. Before I let my guest go, I always ask for a book recommendation. I don't know if you happen to have one for us.

我现在正在读一些我喜欢的东西。我正在读完《三体》系列。第二本和第三本，我认为没有第一本那么高的水准。第一本书中有两个时刻，我不会剧透，我认为它们是任何文学中最好的科幻时刻之一。所以《三体》真的值得一读，为了这些关键的时刻，然后它只是很棒、很搞笑的科幻小说，这有助于我们摆脱所讨论的这些话题。对于人工智能领域的人来说，一个及时的书是我正在读迈克尔·刘易斯的《走向无限》，关于山姆·班克曼-弗里德和 FTX 的事情。读这本书，特别是如果你读过一些幕后文章，就像我们将会看到关于人工智能的类似情况。我认为推特上有一些梗，人们说书名会是《并非始终坦诚》，并且是关于山姆·奥特曼的。但我认为山姆·奥特曼取得了真正的成就；他的成功不会消失。他可能只是有一些个人问题让事情变得更困难。但还会有其他书，描述这些人工智能公司以最戏剧性的方式兴衰。读这本书，确保你对完全胡说八道有良好的嗅觉，可能是好的。我可能可以挖出一本更经典的科技类书籍推荐，但这些是我正在读的，我很享受它们，这通常就足够了。

I'm reading some things that I like right now. So I'm finishing the Three-Body Problem series. The second and third books, I don't think have the peak level of the first book. The first book has two moments in it that I won't spoil, that I think are some of the best sci-fi moments in any literature. So Three-Body Problem is really worth reading for these kind of key moments, and then it's just great, hilarious sci-fi, which is good for getting out of these things that we talk about. And a timely one for people in AI is I've been reading Going Infinite by Michael Lewis on the Sam Bankman-Fried FTX stuff. Reading this, especially if you read some of the behind-the-curtain articles, it's like we're going to get this about AI. I think there are memes on Twitter where people are saying that the title is going to be 'Not Consistently Candid' and it's going to be about Sam Altman. But I think Sam Altman has achieved real things; his success is not going away. He might just have personal things that make it harder. But there are going to be other books where it's like these AI companies come and go in the most dramatic fashion. Just reading that to make sure that you have a good sniff test of total BS is probably good. I could probably dig up a more evergreen classic techie book recommendation, but those are things that I'm reading and I'm enjoying them, and that's normally good enough.

是的，很棒的推荐。《三体》在节目中多次被提及为最受欢迎的书，但《走向无限》是新的，听起来很棒。我喜欢迈克尔·刘易斯，我喜欢他很多年了。

Yeah, great recommendations. Three-Body Problem has come up quite a few times as a favorite book on the show, but Going Infinite is new and that sounds great. I love Michael Lewis, I've loved him for years.

是的，它得到了奇怪的评论，所以我花了一些时间才进入状态，但它读起来真的很好，就像他所有的书一样。所以我想，这只是，它显然有偏见，并不完美，但很扎实。他非常擅长把像卡尼曼和特沃斯基这样可能相当深奥的话题变成一本引人入胜的书。

Yeah, it got weird reviews so I took a second to get into it, but it reads really well like all of his books. So I'm like, this is just, it obviously has bias, it's not perfect, but it's solid. He's unreal at making what could be quite dense topics like Kahneman and Tversky into a page-turner.

太棒了。好的，那么关于本期节目之后如何关注你，我们已经谈到了你的新闻通讯 Interconnect DoAI，我们谈到了你的播客 The Retort。人们还应该通过什么方式关注你？

Awesome. All right, so in terms of following you after this episode, we've already talked about your newsletter Interconnect DoAI, we've talked about your podcast The Retort. How else should people follow you?

除了这些，这些真的是主要的东西。我在大多数平台上都是 NathanLambert，但有点不情愿。我真的只用推特来发一些随想。我会在其他渠道推广我的工作，但唯一额外的信息点就是推特。但我尽量把博客里的好东西留给自己。试着把它们浓缩成更少噪音是好的。总是很容易发更多推文，但就实际学习任何东西而言，你不一定会从发推文中获得更多。

Other than those, those are really the main things. I'm at NathanLambert on most platforms, but somewhat begrudgingly. I only really use Twitter for random thoughts. I'll promote my work on other channels, but the only other additional point of information is going to be Twitter. But I try to make the good things in the blog just for my own sake. It's good to try to condense them down to be less noise. It's always easy to tweet more, but you don't necessarily gain more from tweeting in terms of actually learning anything.

好的，Nathan，非常感谢你今天参加节目。你是一位非常棒的嘉宾，是的，我们真的很感激你抽出时间，希望几年后我们能再和你聊聊。

All right, Nathan, thank you so much for being on the show today. Such an awesome guest, and yeah, we really appreciate you taking the time, and hopefully we'll catch up with you again in a few years.

是的，谢谢你邀请我。这些问题很好。

Yeah, thanks for having me. This was good questions.

多么令人大开眼界的一集。Nathan 向我们介绍了 DDPO 如何在较小的模型中实现意图对齐，使 Zephyr 7B 在某些基准测试中超越 Llama 2 70B。他还谈到了 Dexterity、Ambi 和 Covariant 是机器人领域的主要参与者，但今天的人形机器人在许多日常场景中力量过大。他谈到了 RLHF 如何以更低的成本扩展微调，并帮助解决人类标注者之间的分歧是信号还是错误的问题。他还谈到了 RLHF 如何通过修复由于在 Reddit 等数据源上预训练 LLM 而产生的预训练偏差，从而产生积极的社会影响。像往常一样，你可以在 SuperDataScience.com/791 上获得所有节目笔记，包括本集的文字记录、视频录制、节目中提到的任何材料、Nathan 的社交媒体资料链接以及我自己的链接。

What an eye-opening episode. Nathan filled us in on how DDPO allows for intent alignment in smaller models, allowing Zephyr 7B to surpass Llama 2 70B on some benchmarks. He also talked about how Dexterity, Ambi, and Covariant are the big players in robotics, but that today's humanoid robots have too much force to be around in many everyday situations. He talked about how RLHF can scale up fine-tuning at a lower cost and help resolve whether disagreement among human labelers is a signal or a bug. And he talked about how RLHF can have a positive social impact by fixing the pre-training biases that crop up due to pre-training LLMs on data sources like Reddit. As always, you can get all the show notes, including the transcript for this episode, the video recording, any materials mentioned on the show, the URLs for Nathan's social media profiles, as well as my own, at SuperDataScience.com/791.

我喜欢与大家面对面交流，而不仅仅是通过社交媒体。下周我将参加多伦多的 Cision 大会，这是一个为期四天的会议。在会议的星期四，我将在内容创作者舞台上主持一个下午的会议。除了我主持的会议之外，你还可以看到其他精彩的演讲者，包括 AI 教父杰弗里·辛顿教授。我们还有 Perplexity 的 CEO Arvind Shivas、Cohere 的 CEO Aiden Gomez 以及网球传奇玛丽亚·莎拉波娃。感谢我的 Nebula 同事们支持我创作像本期超级数据科学这样的内容。当然，还要感谢超级数据科学团队的 Ivana、Mario、Natalie、Serge、Sylvia、Zara 和 Koll，他们制作了今天这期精彩的节目，让这个超级团队能够为你免费制作这个播客。我非常感激我们拥有的赞助商。你可以通过查看节目说明中的赞助商链接来支持这个节目。如果你这样做，对我们来说就是巨大的帮助。如果你自己有兴趣赞助一期节目，可以访问 jonkrohn.com/podcast 了解详情。否则，请与喜欢这期节目的人分享。在你收听的任何平台上评论这期节目。如果你还不是订阅者，请订阅。但最重要的是，继续收听。非常感谢你的收听，我希望我能继续制作你喜爱的节目，年复一年。下次再见，继续摇滚，我期待很快与你一起享受另一期超级数据科学播客。

I like to engage with me in person as opposed to just through social media. Next week I will be at the Cision conference in Toronto. It's a 4-day conference. On the Thursday of the conference, I'll be hosting an afternoon of sessions on the content creator stage. Beyond the sessions that I host, other amazing speakers you can check out include the Godfather of AI himself, Professor Geoffrey Hinton. We'll also have Arvind Shivas, the CEO of Perplexity; Aiden Gomez, CEO of Cohere; and the tennis legend Maria Sharapova. Thanks to my colleagues at Nebula for supporting me while I create content like this Super Data Science episode for you. And thanks, of course, to Ivana, Mario, Natalie, Serge, Sylvia, Zara, and Koll on the Super Data Science team for producing another fantastic episode for us today, for enabling that super team to create this free podcast for you. I'm so grateful to have the sponsors that we have. You can support this show by checking out our sponsors' links, which are in the show notes. It's a huge help to us if you do that. And if you yourself are interested in sponsoring an episode, you can get the details on how by making your way to jonkrohn.com/podcast. Otherwise, share this episode with people who would like it. Review the episode on whatever platform you listen to it on. Subscribe if you aren't already a subscriber. But most importantly, just keep on tuning in. So grateful to have you listening, and I hope I can continue to make episodes you love for years and years to come. Till next time, keep on rocking it out there, and I'm looking forward to enjoying another round of Super Data Science podcast with you very soon.