Mistral 7B：用小型开源 AI 模型改变游戏规则

过去一年，开源 AI 模型彻底改变了技术格局。法国一支由前 DeepMind 和 Meta 研究人员组成的小团队最近掀起了巨大波澜：Mistral。本周，Sarah 和我邀请到了 Mistral 的 CEO 兼联合创始人 Arthur Mensch，他最近发布了 Mistral 7B，一个采用 Apache 2 许可证的开源模型，改变了人们对小模型能力的认知。Arthur，欢迎来到 No Priors。

Open source AI models have completely changed the landscape of technology over the past year. One tiny team of ex-DeepMind and Meta researchers in France has made a huge splash recently: Mistral. This week, Sarah and I are joined by Arthur Mensch, the CEO and co-founder of Mistral, who recently released Mistral 7B, an Apache 2 licensed open source model that has changed people's mental models about what can be done with small models. Arthur, welcome to No Priors.

谢谢邀请，很高兴来到这里。

Thank you for inviting me, very glad to be here.

好的，就在六个月前我们见面时，你刚离开 DeepMind 创办 Mistral。看着 OpenAI、谷歌等公司积累的巨额资金和算力，还敢说‘我们也要参与这场游戏，而且这很重要’，这需要真正的勇气。跟我们说说创办 Mistral 的灵感吧。

Okay, so just six months ago when we met, you were leaving DeepMind to start Mistral. It takes real guts to look at the scale of dollars and compute that OpenAI and Google and others have amassed and say, 'We want to play in this game too, and it's important we do.' Tell us about the inspiration to start Mistral.

Guillaume、Timothée 和我很早就进入了这个领域。我们做机器学习已经十年了，确实知道从哪里起步，以及如何用有限的算力和资金做出好模型——虽然不是那么有限，但至少比我们之前所在的地方更有限。我想这就是我们开始的原因。我们所在的几家公司都朝着我们加入时未曾预料的方向发展，我们决定这是一个很好的机会，可以在欧洲创建一家独立公司，专注于让 AI 变得更好，专注于打造前沿 AI，并将开源 AI 作为核心价值。

Guillaume and Timothée and I were pretty early in the field. It had been 10 years that we had been doing machine learning, and we did know where to start from and how to make a good model with a limited amount of compute and money—well, not so limited, but at least more limited than where we were coming from. I think that's why we got it started. The various companies we were in moved into directions that we hadn't anticipated when we joined, and we decided that there would be a very good opportunity for creating something that would be a standalone company in Europe, focusing on making AI better, focusing on making frontier AI, and focusing on making open source AI as a core value.

也许我们可以逐一聊聊这些方面。在你成为 Chinchilla 缩放定律论文的合著者之前，你在机器学习领域有十年经验，早期还研究过混合专家模型的想法。能谈谈你在 DeepMind 的研究方向吗？

Maybe we can talk about each of those pieces. So 10 years in machine learning before you were a co-author on the Chinchilla scaling laws paper, you worked on mixture of experts ideas early. Can you talk a little bit about what your research directions were at DeepMind?

是的，我来自优化背景，所以过去十年我的重点一直是让算法更高效，更好地利用我们拥有的数据来制作预测性能良好的模型。当我到达 DeepMind 时，我加入了 LLM 团队，当时只有 10 个人，很快我就开始研究检索增强模型，与我的朋友 Sébastien Borgeaud（他仍在 DeepMind）共同领导了一篇名为 RETRO 的论文。其要点是在预训练期间使用非常大的数据库，这样我们就不必将知识强制注入模型本身，而是告诉模型它无论如何都可以访问外部记忆。效果相当不错，我们实际上可以降低困惑度。存在一些社区已经开始很好解决的局限性，当时检索方法还不是主流——现在它们已经完全成为主流了。这是我做的第一个项目。我还很快研究了稀疏混合专家模型，因为这与我擅长的领域——最优传输——相关。最优传输是一种设置，你需要将 token 分配给设备，并确保它们之间有良好的分配，这样设备就不会看到太多 token。事实证明，正确的方法是使用最优传输，这是一个数学框架。所以我开始研究将其引入稀疏混合专家模型，很快我们就转向了缩放定律：如何将某个规模有效的方法推广，预测它如何随规模、专家数量、看到的数据量而变化？这也是我与许多同事合作的工作，关于如何将密集模型的缩放定律适应到不仅要预测模型大小，还要预测专家数量的场景。然后，相关地，我参与了 Chinchilla 的工作，这是 LLM 历史上的一篇重要论文，同样与 Sébastien、Jordan、Lauren 等人合作。基本上，故事是每个人都在用太少的 token 训练模型，因为 2020 年的一篇论文执行得不太好。我们观察到，你可以纠正这一点，与其用很少的 token 训练非常大的模型，不如随着模型大小的增长而增加 token 数量。仔细想想，这很有道理，因为你不想让一个无限大的模型只看有限数量的 token，同样，你也不想让一个有限大小的模型看无限数量的 token——必须存在某种比例关系。

Yes, I come from an optimization background, so my focus has always been, for the last 10 years, to make algorithms more efficient and to use better the data that we have to make models with good prediction performances. When I arrived at DeepMind, I joined the LLM team, which was 10 people at the time, and very quickly I started to work on retrieval augmented models, with a paper called RETRO that I co-led with my friend Sébastien Borgeaud, who is still at DeepMind. The point was to use very large databases during pre-training so that we didn't force knowledge into the model itself, and we would tell the model that it would have access to an external memory anyway. It was working quite well; we could actually lower the perplexity. There were some limitations that the community has started to address quite well, and that was at the time when retrieval methods weren't really mainstream—now they've become completely mainstream. So that's the first project I did. I worked on Sparse Mixture of Experts also quite quickly, because that was related to my topic of expertise, which was optimal transport. Optimal transport is a setting where you have tokens you need to assign to devices, and you need to make sure that there's some good assignment between them so that the devices don't see too many tokens. As it turns out, the way you do it is with optimal transport, a mathematical framework to do it correctly. So I started to work on introducing this to Sparse Mixture of Experts, and very quickly we started to move on to scaling laws: how do you actually take the method that is working at a certain scale and try to predict how that will evolve with the scale, the number of experts, the amount of data you see? That's work I've done with many colleagues as well, on how do you adapt the scaling law for dense parameters for dense models to a setting where you want to predict the performance not only with relation to the size of the model but also the number of experts. Then, connectedly, I worked on Chinchilla, which is a major paper in the history of LLMs, also with Sébastien, Jordan, Lauren, and many other people. Basically, the story was that everybody was training models on too few tokens because of a paper from 2020 that happened to be not very well executed. What we observed is that you could actually correct that, and instead of training very large models on very few tokens, you should actually grow the number of tokens as you grow the size of the model. If you think about it, it makes a lot of sense because you don't want to have an infinite size model looking at a finite number of tokens, and similarly you don't want to have a finite size model looking at an infinite number of tokens—there must be some proportionality.

是的，没错。这是我们通过经验证明的，我认为这很快被采纳了，因为它纯粹是赢家：用同样的算力，你不仅会得到一个更好的模型，而且这个模型的推理成本会便宜四倍。所以这绝对是一个收益。但事实证明，我们做得还不够。这就是我们在 Mistral 所做的：我们意识到在进一步压缩模型方面还有很大机会。我的意思是，我们从 LLaMA 中看到这是可能的。我们用 Mistral 7B 展示的是，我们远远没有达到压缩的极限。我们通过制作一个非常非常小、推理成本极低、速度极快、可以在 MacBook Pro 上运行但仍然足够有用的模型，某种程度上纠正了这一点。这是我们在公司取得的首批成就之一。

Yeah, exactly. And that's something that we showed empirically, and I think that's something that was adopted very fast because it was like a pure win: for the same amount of compute, you would get a model that would be better, but also a model that would be four times cheaper to serve. So that was definitely a gain. And as it turns out, we didn't go far enough. That's what we did at Mistral: we realized that there was also a lot of opportunity in actually compressing models more. I mean, we've seen with LLaMA that it was actually possible. What we showed with Mistral 7B is that we were definitely far away from the limit of compression. We somehow corrected that by making a model very, very small, super cheap to serve, super fast, running on your MacBook Pro, but still good enough to be useful. And so that's one of the first achievements we made in the company.

是的，我想很多人对 Mistral 推出 7B 模型印象深刻，因为 A) 你们做得非常快，只用了几个月时间；B) 如果你看看运行这些模型的实际成本，显然有训练方面的成本，但还有推理成本，日常使用中，无论你是 OpenAI、LLaMA 还是 Mistral 的用户，运行模型的成本主要来自推理。而这通常也由模型大小驱动。所以我认为，推出这些性能出色的小模型，确实极大地改变了人们对可能性的看法。你能分享一下你认为这种性能会走向何方，或者你如何看待模型大小，包括更小但性能更强的模型，以及你们是否计划随着时间的推移构建像 GPT-4 或 GPT-5 那样非常大的模型？

Yeah, I think a lot of people were really impressed when Mistral came out with the 7B model because A) you did it very quickly, you know, you did it in a matter of a few months, but B) if you look at the cost of actually running these models, obviously there's the training side of it in terms of actually building the model, but then there's inference, and so much of the cost day-to-day if you're a user of OpenAI or a user of LLaMA or a user of Mistral, how much it costs to actually run the model—it's the inference. And that's also often driven by the size of the model. So I think by coming out with these small models that were very performant, it really made a huge change to how people thought about what was possible. Is there anything that you can share in terms of where you think this performance is going to go, or how you think about the sizes of models, both in terms of smaller more performant models as well as do you folks plan to build something very large, more like a GPT-4 or GPT-5 equivalent over time?

当然。我认为我们在 2022 年、2021 年看到的是，有几家公司真正专注于推动模型性能，如果你想提升性能，就需要扩大规模。但我们相信，在效率方面还有很大的改进空间，无论是训练还是推理。对于较小的模型，我们认为可以继续用有限的算力推动可能性的边界，使它们更易获取、运行成本更低。对于较大的模型，我们肯定计划构建非常大的模型，但我们希望以高效和开放的方式来做。我们不会透露具体计划，但我们正在努力。

Sure. So I think what we've seen in 2022, 2021, you had a few companies that were really focusing on pushing the performance of models, and if you want to push the performance, you need to scale up. But we believe that there is a lot of room for improvement in terms of efficiency, both in training and inference. For smaller models, we think we can continue to push the frontier of what's possible with limited compute, making them more accessible and cheaper to run. For larger models, we are definitely planning to build something very large, but we want to do it in a way that is efficient and open. We are not going to disclose specific plans, but we are working on it.

模型的纯性能，你不在乎推理成本，因为你不会使用模型，你只是看它们好不好，这纯粹是为了认证目的。但当你开始考虑部署和启用下游应用时，你就需要考虑运行时成本。所以你不仅担心获取模型的前期投入，还担心运行时成本。所以我认为推理成本和训练成本之间的系数确实取决于业务。作为一家打算拥有可行商业模式的公司，我们非常重视推理成本。我们认为，进入推理成本极低的阶段非常重要，这样你就可以运行智能体，基本上可以在所有用例中随处使用 AI 大语言模型，而不会受到成本的限制，而目前最大的模型就是这种情况。所以我们肯定考虑到了这一点，我们知道我们可以让一个 70 亿参数的模型变得非常好。这是肯定的，但这绝不是故事的终点。现在的问题是，我们是否训练更大的模型？答案显然是肯定的。特定模型大小能做的事情是有限的。我认为这个限制被低估了。但如果你想获得更强的推理能力，你确实需要转向更大的模型。转向更大模型的另一个好处是，它让你能够通过蒸馏或合成数据生成等各种技术训练出更好的小模型。所以这两件事是密切相关的。如果你想做出非常强大的小模型，你确实需要更大的模型。我们确实在训练更大的模型。

The pure performance of models, you don't care about inference because you're not going to use the model, you're just going to see whether they're good or not, and that's really for authenication purposes. But then when you start thinking about deployment and enabling downstream applications, then you need to think about what it is going to cost in runtime. So you're not only worried about the upfront payment you need to make to get the model, but you're also worried about the runtime. And so I think the coefficient that you put between the inference cost and the training cost is really business dependent. And as a company that intends to have a valid business model, we think a lot about inference cost. We think that it's super important to get to a regime where inference is super cheap so that you can run agents, you can basically use AI LLMs everywhere for all of your use cases and you're not blocked by cost, which is the case for the largest model currently. So that's definitely something we had in mind, and we knew that we could make a 7 billion parameter model very good. That's for sure, but it's definitely not the end of the story. Now the question is, do we train bigger models? And the answer is obviously yes. There's still a limit to what a certain model size can do. This limit was, I think, underestimated. But if you want to get to more reasoning capabilities, you do need to move into larger models. The other thing about moving into larger models is that it enables you to train smaller models that are better, through a variety of techniques like distillation or synthetic data generation. So these two things are quite related. If you want to make very strong small models, you do need to have bigger models. And we are indeed training larger models for sure.

你能谈谈你们在数据和标注方面的做法吗？因为我们刚刚谈到了另外两个维度。

Can you tell us about your approach to data and annotations? Because we kind of talked about the other two dimensions.

是的，我们谈到了算力，显然数据也至关重要。所以我们从开放网络入手。我们做了很多工作。我认为我们在获取优质数据方面做得不错。数据质量才是真正让模型变好的关键。我的意思是，数据和算法显然都很重要，但数据超级重要。我们非常重视这一点，而且我认为我们确实有一个非常好的数据集，这是肯定的。数据标注是另一个话题。它也与预训练有关。当你预训练一个模型时，你确实希望拥有最纯粹的知识、最纯粹的数据质量。当你想要对齐模型并指导它，让它遵循指令时——这对许多用例很有用，因为它使模型可控——你确实需要一定数量的人工标注或潜在的机器标注。所以我们开始在这方面下功夫。我们不是世界上制作优秀指令遵循模型的顶尖专家。我们肯定在加速，团队在这方面也越来越好。

Yes, so we've talked about compute, and obviously data is super critical. So we work from the open web. We do a lot of work. I think we do a good job at getting some good data. The data quality is really what makes the model good. I mean, data and algorithms obviously, but data is super important. We put a lot of focus on that, and I think we do have a very good dataset, that's for sure. Data annotation is another topic. It's also related to pre-training. When you pre-train a model, you really want to have the purest knowledge, the purest quality of data. When you want to align your model and instruct it, ask it to follow instructions, which is useful for many use cases because it makes it steerable, you do need to have a certain amount of either human-produced annotation or potentially machine-produced annotations. And so that's something that we start working on. We're not the top experts in the world in making good instruction-following models. We're definitely ramping up, and the team is getting better and better at that.

我们还没谈到的一点是，你们是一家开源公司，这与当今其他顶尖实验室非常不同。为什么这很重要？

One of the things we haven't talked about is the fact that you guys are an open source company, which is very different from the other labs working at the state of the art today. Why is that important?

如果你回顾过去 10 年机器学习的历史，它发展得非常快。我的意思是，我们从蹩脚的猫狗检测器发展到看起来基本具有人类智能的东西，记住这是如何发生的是很有用的。它之所以发生，是因为有很多学术实验室，很多工业实验室实际上在解决不同问题上投入了更多资金，并且直到 2020 年，大家都有充分的沟通，几乎完全透明。无论哪个实验室做了什么，即使是竞争实验室，实际上都会在 NeurIPS、ICML 上发表，每六个月我们都会聚在一起，获得新想法，想法会传播，每个人都会在他人工作的基础上继续建设。这就是我们从可能有趣的东西发展到非常有趣的东西的方式。但问题是，大约在 2020 年，一些公司开始在某些领域相当领先，并意识到可以积累一些价值，那时不透明性又回到了这个领域。我认为这是我们在软件领域已经观察到的循环，即开放与封闭之间的循环。我们再次观察到它。我们认为现在还为时过早，而且我们认为，进入这样一种不透明的体制——少数公司基本上在做同样的事情，只是不沟通，花费数十亿算力做完全相同的事情，而我们所关注的技术还没有完全成熟——对科学来说确实是有害的。它仍然不能很好地推理，记忆机制不太被理解，因果机制不太被理解，它不太可控，有很多偏见。我的意思是，它是不完整的。还有很多事情要做。我们仍然需要发明新技术。如果没有人谈论它，你怎么能发明新技术呢？为了发明新技术，你仍然需要花费大量资金去尝试那些可扩展的东西，而少数有钱可花的公司现在拒绝沟通。这是我们深感遗憾的事情，也是我们试图改变的事情。因为我们确实有相当多的资金可以花在算力上，我们确实有一些好主意。我们知道有一个庞大的社区在等待 AI 参与者，等待开源 AI 参与者的出现。我们非常感激 Meta 正在朝这个方向前进，通过做我们所做的事情，对我们创造的技术更加开放。我们想引导社区进入一个事情运作得更好、更安全的体制，因为受到更多审查。我们的真正意图是占据那个位置，改变游戏规则，因为我们不认为事情正朝着正确的方向发展。

If you look back at the history of machine learning in the last 10 years, it went very fast. I mean, we went from a poor cat-dog detector to something that basically looks human intelligent, and it's useful to remember how that happened. It happened because you had many academic labs, you had many industrial labs actually spending more money on different problems, and there was full communication, almost full transparency until 2020. Whatever was done in whatever lab, even in competing labs, was actually published at NeurIPS, published at ICML, and every six months we would all gather and get new ideas, ideas would circulate, and everybody would build on top of the work of others. And that's the way we went from something potentially interesting to something very interesting. But then the issue is that around 2020, some companies started to be quite ahead on some fields and realized that some value could be accrued, and at that point opacity made its way back into the field. And I think that's a cycle we've observed in software already, the cycle between openness and closeness. We are observing it again. We think that it's too early, and we think it's really damaging for the science to actually move into such an opaque regime where you have a few companies basically doing the same thing, just not communicating about it, spending billions of compute doing exactly the same thing, and where really the technology we're looking at is not working completely yet. Still, it doesn't reason well, memory mechanisms are not very well understood, causality mechanisms are not well understood, it's not super steerable, there's a lot of biases. I mean, it's incomplete. There are many things to be done. We still need to invent new techniques. And how are you going to invent new techniques if nobody is speaking about it? When in order to invent new techniques, you need to still spend some large amount of money to actually try things that scale, and the few companies that have the money to spend have now refused to communicate. That's something that we deeply regretted, and that's something that we are trying to change. Because we do have some substantial amount of money to actually spend on compute, we do have some good ideas. We know that there's a big community that is awaiting for AI players, for open source AI players to appear. And we're very grateful that Meta is moving into that direction by doing what we do, by being much more open about the technology we create. We want to steer the community into a regime where things just work better, where things are safer because put under more scrutiny. And really our intention there is to take that position and to change the rules of the game, because we don't think that this is moving into a proper direction.

这非常有趣，因为如果你看当前的讨论，真正的大型科技公司声称开源 AI 是危险的，这感觉真的是一种监管俘获。他们希望监管机构介入，以便他们能够限制创新并控制行业。而现实是，如果你看看全球健康、全球公平等问题，开源是让全人类以透明、开放的方式从这项技术中受益的最大潜在途径之一，人们可以真正理解和看到。你们如何处理安全、政策，以及思考在开源背景下思考安全的正确方式？

It's very interesting because if you look at the current discourse, the really big tech companies are claiming that open source AI is dangerous, and it feels like really a form of regulatory capture. They want regulators to step in so that they can constrain innovation and kind of control an industry. And the reality is, if you look at things like global health, global equity, open source is one of the biggest potential ways for all of humanity to benefit from this technology in a way that's transparent and open, and people can really understand and see. How do you approach safety and policy and thinking about the right ways to think about safety in the context of open source?

所以我们从非常务实的角度来处理这个问题。我们相信开源实际上让 AI 更安全，因为它允许更多的审查。当代码和模型开放时，研究人员、审计人员和社区可以检查它们是否存在偏见、漏洞和安全问题。这种透明度有助于更好地理解和更快地修复。相比之下，封闭系统可能会隐藏问题，直到它们变得严重。我们还认为，安全不应成为扼杀创新的借口。我们致力于负责任的发布，包括设置防护措施，但我们相信开放的好处大于风险，尤其是在技术仍在发展的时候。我们与政策制定者接触，倡导平衡的监管，既能促进创新，又能解决实际风险，而不是假设的风险。

So I think we approach it from a very pragmatic point of view. We believe that open source actually makes AI safer because it allows for more scrutiny. When the code and models are open, researchers, auditors, and the community can examine them for biases, vulnerabilities, and safety issues. This transparency leads to better understanding and faster fixes. In contrast, closed systems can hide problems until they become critical. We also think that safety should not be an excuse to stifle innovation. We are committed to responsible release, including putting safeguards in place, but we believe that the benefits of openness outweigh the risks, especially when the technology is still evolving. We engage with policymakers to advocate for balanced regulations that promote innovation while addressing real risks, not hypothetical ones.

所以问题是：开源今天的模型是否危险？它是否真的让不良行为者能够滥用模型？是否给了他们额外的边际能力去从事不良活动？我认为这个问题的答案是否定的。这是我的结论。我们看到了很多关于生物武器、核恐怖主义之类的想法。这很有趣，因为如果你真的假设这些论点是善意的——我认为在很多情况下人们是善意的，这是我们一直做的假设——并且如果你深入探讨这些论点，我们发现其实根本站不住脚。没有任何证据表明，LLM 在寻找可用于不良用途的知识方面比搜索引擎有边际优势。另一件事是，甚至没有证据表明知识是我们所讨论的实际滥用的瓶颈，而且这很可能不是事实。因此，要证明开源大语言模型实际上不安全，你需要证明它们比搜索引擎有边际改进，并且知识是创造的瓶颈。在这两种情况下，这两个问题的答案都是否定的。这意味着我们相信今天可以开源模型，实际上这是将事物置于高度审视之下的最佳方式，以便我们为可能成为超级智能的新一代模型做好准备。在这种情况下，我认为我们可以重新审视这些前提。但今天，我们谈论的只是对网络上广泛可用的知识的压缩，从边际上讲，我们并没有创造比以往更危险的东西。所以我认为存在一个权衡，一个需要进行的动态对话。这就是我们在 AI 安全峰会上讨论的内容。随着模型能力的增强，这需要重新审视，但今天，禁止开源并阻止其发生，实际上是一种强制监管捕获的方式。即使受益的参与者不希望它发生，但通过设计，如果你禁止小参与者以最有效的方式——即开源——做事，你确实会便利大型现有企业，这肯定会对生活有害。

So the question is: is open-sourcing today's models dangerous? Is it actually enabling bad actors to misuse the model? Is it giving them marginal extra capacities in pursuing their bad endeavors? I think the answer to this question is no. That's my conclusion. We've seen a lot of ideas around bioweapons, nuclear terrorism, and the like. It's very interesting because if you actually assume good faith of these arguments—and I think in many cases people are in good faith, and that's an assumption we always make—and if you try to go down the arguments, we realized that there was really nothing to it. Nothing shows that an LLM is actually marginally better than a search engine to find knowledge on topics that would enable bad use. The other thing is that it's not even proven, and it's certainly very likely not the case, that knowledge is the bottleneck for the actual misuse we're talking about. So to demonstrate that open-sourcing large language models is actually unsafe, you need to demonstrate that they have marginal improvement over a web engine and that knowledge is the bottleneck for creation. In both cases, the answer to these two questions is no. That means we believe we can open-source models today, and actually it's the best way of putting things under high scrutiny so that we are ready for potential new generations of models that could be superintelligent. In that case, I think we can revisit these premises. But today, we're really talking about a compression of knowledge that is widely available on the web, and marginally speaking, we're not creating anything that is more dangerous than before. So I think there's a trade-off, a dynamic conversation to be had. That's what we discussed at the AI Safety Summit. This needs to be revisited as model capacities build on, but today, banning open source and preventing it from happening is really a way to enforce regulatory capture. Even though the actors that would benefit from it don't want it to happen, by design, if you ban small actors from doing things in the most efficient way—which is open source—you do facilitate the life of the larger incumbents, and that's something that would be detrimental to life for sure.

你怎么看待那些任意提出的算力和规模限制？

What do you make of the arbitrary sort of compute and scale limits proposed?

这很有趣。我不完全知道它们是怎么来的。他们提出了这个阈值——按任何标准来看都是一个很高的阈值。因为如果你计算，如果你恶意假设这是 float64，它大约相当于 3 亿次最后运行的计算。所以这很高，不是我们甚至负担得起的，而且在未来几年我们也负担不起。所以它很高。这非常随意，因为谁告诉你超过 10^26 就会产生不良能力？模型开始出现不良行为？这绝对没有证据。将能力与规模联系起来也非常近似，因为它实际上取决于数据。数据集非常重要。如果你训练模型生成——我的意思是，有一个焦点是生物武器，所以假设我们想防止模型生成化合物，因为我们认为是不良行为的促成因素——我说过我们认为不是这样——但如果你确实想防止，你需要调整你的计算 FLOP 预算以适应我们正在处理的数据集。事实证明，他们就是这么做的，因为他们实际上为生物学制定了特定的 FLOP 预算，我想。所以你可以看到生物武器叙事在此基础上构建。但这完全是——我们应该真正关注能力，而不是市场前条件。我认为在某种程度上，对此仍有共识。所以每个人都知道它不完美，它是一个可能相当相关的代理，但肯定我们需要就如何衡量能力达成一致，就我们认为危险的能力达成一致。我认为我们在那个话题上彼此不同意，但这些应该是判断标准，而不是明显的市场前条件或你做的 FLOP 数量。

That's interesting. I don't exactly know how they came out. They came up with this threshold—it's a high threshold by any standard. Because if you compute, if you make the bad faith assumption that this is float64, it gives you approximately 300 million last run compute. So that's high, not something we can even afford and that we won't be able to afford for the coming years. So it's high. It's very arbitrary because who tells you that beyond 10^26 you end up with bad capacities? Models start to have the emergence of bad behaviors? That's definitely not proven. Relating capabilities to scale is also very approximate in the sense that it really depends on the data. The dataset is super important. If you train your model on generating—I mean, there's a focus on bioweapon, so let's say if we want to prevent models from generating chemical compounds because we think it's an enabler of bad behaviors—which I said we don't think is the case—but if you do want to prevent that, you need to adapt your compute flop budget to the dataset that we are working on. As it turns out, that's what they did because they actually made a specific flop budget for biology, I think. So you can see the bioweapon narrative building on that. But this is completely—we should really focus on capabilities and not pre-market conditions. I think to some extent there's still consensus around that. So everybody knows that it's imperfect, it's a proxy which is maybe fairly correlated, but definitely we need to come up with agreeing on how we measure capabilities, agreeing on what capabilities we deem dangerous. I think we don't agree with one another on that topic, but these should be the judge, and not obviously pre-market conditions or the number of flops that you do.

你知道为什么如此关注生物武器吗？我作为一个做了近十年生物学家的人问这个问题。当我看到构建病毒或实际做这些事情的复杂性时，我很惊讶社区如此关注这个特定例子。你知道为什么人们一直提起它吗？因为这实际上很难转化；它不是你可以操纵的数字东西。

Do you know why there's such a focus on bioweapons? I ask this as someone who worked for almost a decade as a biologist. When I look at how complex building viruses or actually doing these things is, I'm surprised that there's so much focus in the community on that specific example. Do you have any sense of the origins of why people keep bringing that up? Because it's actually hard to translate; it's not some digital thing that you manipulate.

我认为这是一个非常有趣的问题，老实说我没有答案。这几乎是认识论的问题。所以这个想法是如何出现的，它如何被政策人士放大，以及它如何最终被国家安全部门听到？我认为它始于 GPT-4 的附录 D 或类似的东西，在第 46 页左右——不完全是同样的数字——他们生成了一些化合物。然后他们说了一句小评论，说也许这不是我们想走的方向；我们不想有一个推理化合物的模型。所涉及的化合物并不危险，但随后他们做了这个观察，如果你在生物学文章上训练，这绝对是预期的：你肯定能够产生一些化合物。所以是预期的观察。然后不知何故，人们在此基础上构建。没有观察到任何东西，没有以适当形式发表科学研究，但政策论文开始引用非科学论文，声称这些是生物武器叙事实际上真实的科学证据。然后政策论文开始引用其他政策论文，突然之间你得到了大约 50 篇论文，说生物武器肯定会炸毁我们，这就是政策制定者最终读到的。我认为这就是我们今天所处的位置。所以不幸的是，有一些构建。我认为开源社区一直不够发声，因为他们没有预见到这一点，但现在这正在改变，我很高兴看到这一点。

I think it's a very interesting question, and honestly I don't have the answer. It's almost epistemology at that point. So how did this idea appear, and how did it get amplified by policy people, and how did it actually end up being heard by national security? I think it started somehow with GPT-4, annex D or something, on page 46 or something—not exactly the same numbers—where they generated some chemical compound. Then they said a small remark saying that maybe that's not the direction we want to take; we don't want to have a model that reasons about chemical compounds. The chemical compound in question wasn't dangerous, but then it made this observation which is definitely expected if you train on articles on biology: you're definitely going to be able to produce some chemical compounds. So expected observation. Then somehow people built things on top of it. Nothing was observed, no scientific studies in proper form were published, but then policy papers started to cite non-scientific papers arguing that these were scientific evidences that the bioweapon narrative was actually true. Then policy papers started to cite the other policy papers, and all of a sudden you end up with like 50 papers saying that for sure bioweapon is going to blow us up and that this is what the policymakers read at the end. I think that's how we ended up where we are today. So there's some construction to be made, unfortunately. I think the open source community hasn't been vocal enough because they didn't see it coming, but right now this is changing, and I'm very glad that it is.

这具有模仿性，就像世界刚刚经历了新冠疫情一样。

It's memetic, you have the factor of like the world just goes through the COVID-19 pandemic.

是的，集体创伤肯定在那段叙事中起了作用。那绝对是一次创伤——3000 万人死亡。我们当然不希望这种事再次发生。我不认为 AI 会成为引发下一次大流行的原因；那永远会是气候变化。过去是这样，现在可能也应该把重点放在那里，而不是关注那些由词元生成器带来的假设性、未经证实的生物风险。

Yeah, I mean the collective trauma for sure played a role in that narrative. That's definitely a traumatism—30 million people died. That's definitely something we don't want to reoccur again. I don't think AI is going to be the one triggering the next pandemic; it's always going to be climate change. That's the way it was, and that's probably where the focus should be, instead of focusing on hypothetical, non-proven biological risk by token generators.

如果生物武器在可预见的未来不是一个实际关切，那么关于护栏——我们希望 AI 模型实际生成什么——就存在真正的担忧。你怎么看？

If bioweapons is not a pragmatic concern in the visible future, there are real concerns around guardrails about what we want our AI models to actually generate. How do you think about that?

是的，我认为这是一个非常合理的担忧。模型可以输出任何类型的文本，在很多情况下你并不希望它输出任何文本。所以当你构建一个应用时，你需要考虑要在模型输出上设置什么护栏，可能还有输入。所以你需要一个系统来过滤无效的、你认为非法的输入和输出。我们设想的方式是创建一个模块化架构，让应用开发者可以使用，这意味着你提供原始模型——没有为了禁止某些输出空间而修改过的模型——然后你在上面提出新的过滤器，可以检测我们不想要的输出。比如，可以是色情内容，可以是仇恨言论——这些是你在聊天机器人中想要禁止的。但你不希望从原始模型中禁止这些，因为如果你想用原始模型来做审核，你希望模型了解这些东西。所以，认为模型应该行为良好，我认为是一个错误的假设。你需要假设模型应该知道一切，然后在此基础上，有一些模块来审核和防护模型。这就是我们的方法，这也是赋能应用开发者构建良好防护应用的方式。我们认为我们有责任制作非常好的模块，以便正确防护模型。这是平台的一部分，我们认为应该有不同初创公司在模型护栏领域进行健康竞争。而实现这种健康竞争的方式不是信任少数公司自己做安全，而是要求应用开发者遵守一些规则。所以聊天机器人不应该输出仇恨言论，这意味着应用开发者需要找到一个好的护栏解决方案。现在就有了竞争，其中存在提供最佳防护解决方案的经济利益。这就是我们认为生态系统应该运作的方式，也是我们定位自己的方式。这就是我们构建平台的方式，带有模块化过滤器和模块化机制来控制模型。

Yeah, so I think this is a very valid concern. Models can output any kind of text, and in many cases you don't want it to output any kind of text. So when you build an application, you need to think about the guardrails you want to put on the model output and potentially also on the input. So you need to have a system that filters inputs that are not valid, that you deem illegal, and outputs that are not valid. The way we do it in our mind is that you create a modular architecture that the application maker can use, which means you provide the raw model—the model that hasn't been altered to ban some of its output space—and then you propose new filters on top of that that can detect the output that we don't want. So it can be, I don't know, pornography, it can be hateful speech—these things you want to ban when you have a chatbot, for instance. But these things you don't want to ban from the raw model because if you want to use the raw model to do moderation, for instance, you want your model to know about this stuff. So really, assuming that the model should be well-behaved is, I think, a wrong assumption. You need to make the assumption that the model should know everything, and then on top of that, have some modules that moderate and guard the model. So that's the way we approach it, and it's a way of empowering the application maker in making a well-guarded application. We think it's our responsibility to make very good modules that allow guarding the model correctly. It's part of the platform, and we think it's the way—there should be some healthy competition in that domain of different startups working on guardrailing the models. And the way you make this healthy competition is not by trusting a couple of companies to do their own safety; it's rather the way you do it is to ask application makers to comply with some rules. So a chatbot should not output hateful speech, and that means that now the application makers need to find a good guardrailing solution. And now you have a competition where there is some economic interest in providing the best guarding solution. So that's the way we think the ecosystem should work, and that's the way we position ourselves. That's the way we build the platform with modular filters and modular mechanisms to control the model.

你们能如此深思熟虑，真是太好了。我认为当人们谈论安全时，他们实际上在谈论三个话题，有时会各说各话。第一类是这种审核或基于文本的风险——仇恨内容、非法内容、偏见等风险。第二类我们已经讨论过，是物理风险——比如生物武器或 AI 帮助脱轨火车或以某种方式干扰的能力。第三类是存在性或物种风险，这时人们开始谈论 AGI、新生命形式以及资源竞争或聚合之类的事情。

It's great that you folks are being so thoughtful about that. I think when people talk about safety, they really talk about three topics, and sometimes they talk past each other. One is this sort of moderation or text-based risk—risk of hateful content, illegal content, bias, etc. There's a second class which we talked about already, which is physical risk—things like bioweaponry or the ability of AI to help derail a train or interfere somehow. And then third is like existential or species risk, and that's when people start talking about AGI and new forms of life and resource competition or aggregation or things like that.

首先，我认为非常重要的是将这三件事分开处理，并强调我们对第一个问题有解决方案，第二个问题目前没有证据表明它实际存在，也没有证据表明它在不久的将来会存在。第三点，我认为那非常哲学化。显然，如果你能制造一个任意复杂度的系统，它可能开始做任何你不希望它做的事情。我们还没有达到模型具有任意复杂度的阶段，所以这对我来说非常抽象。我认为那仍然是——我的意思是，我们将进入一个智能体和 AI 相互交互的世界，这将产生很多复杂性。预期这种复杂性必然会导致崩溃——在机器学习中，当一切突然停止工作，因为你陷入局部最小值时，我们称之为崩溃。嗯，我不清楚复杂性会导致崩溃。通常复杂性会导致无所作为，因为没有自组织和构建某事的意志力。所以我不太担心存在风险。显然，这是一个动态的对话。如果我们能制造一个越来越智能的模型，那么也许你会达到奇点水平。没有任何证据表明我们正在朝着那个方向前进，正在实现它。所以我认为这是一个非常开放的对话，我们应该进行。我个人没有看到科学证据，作为科学家，我只相信我能看到的。

First of all, I think it's very important to address these three things separately, and to hammer home that our solutions exist for the first one, that the second one there's no evidence that it actually exists at this point, and no evidence that it will exist in the near future. The third point, I think that's very philosophical. Obviously, if you can make a system of arbitrary complexity, it can start doing anything that you don't want it to do. We are not at a stage where the model has arbitrary complexity, and so this is very abstract to me. I think that's still—I mean, we'll move onto a world with agents and AI interacting with one another, and that will create a lot of complexity. Anticipating that complexity will necessarily yield to a collapse—we call it a collapse in machine learning when suddenly everything stops working because you fall into a local minima. Well, it's unclear to me that complexity leads to a collapse. Usually complexity leads to doing nothing because there's no self-organization and no willpower to build something. So I'm not too worried about existential risk. Obviously, this is a dynamic conversation. If we can make a model which is growingly intelligent, then maybe you're at a singularity level. There's no evidence whatsoever that we are on the way of doing that, of making that happen. So I think it's a very open conversation we should have. I personally don't see the scientific evidence, and as a scientist, I trust only what I can see.

然后我猜你提到了智能体，我认为这是目前非常活跃的领域。感觉与智能体相关的很多事情仍然有点遥远。换句话说，感觉这是一个潜力巨大的领域，但还处于早期阶段。有没有什么重大的技术创新或你们正在做的事情，你认为会真正加速一个更广泛地转向基于智能体使用的世界？

And then I guess you mentioned agents, which I think is an area of a lot of activity right now. It feels like a number of things that are related to agents are still a little bit of ways in the future. In other words, it feels like an area with enormous promise but it's still quite early. Are there any big technological innovations or things that you're working on that you think will really help expedite a world that moves more towards agent-based use in a more broad sense?

我认为让模型更小绝对是让智能体工作的一种方式，因为智能体的一个问题是，如果你在 GPT-4 上运行一个智能体，你会很快花光钱。所以如果你将计算成本除以 100，你就可以开始构建更有趣的东西。我们在智能体上看到的是模式崩溃——不是很有趣的模式崩溃；它们开始重复自己，陷入循环。所以肯定有一些研究要做。有一些研究是关于让模型更擅长推理，让它们更能根据任务难度调整投入的计算量。这可以在某种程度上通过智能体解决。所以这绝对是我们正在探索的一个研究方向。

I think making a model smaller is definitely a way to make agents work, because one problem you have with agents is that very quickly, if you run an agent on GPT-4, you're going to run out of money very quickly. So if you divide by 100 the cost of compute, you can start to build more interesting things. What we see with agents is mode collapse—so not very interesting mode collapses; they start repeating themselves and they fall into loops. So definitely there's some research to be made there. There's some research to be made on making models more capable of reasoning and making them more capable of adapting the amount of compute they put onto the difficulty of the task. And this can be agent-solved somehow. So it's definitely an avenue of research that we're exploring.

回到 Mistral，你知道你稍微提到过的一件事是你们围绕训练的模型构建的平台。你能告诉我们更多关于这个平台以及一些……

Going back to Mistral, you know one of the things you've talked about a little bit is the platform that you've been building around the models that you train. Can you tell us a little bit more about that and some of...

这是发展的方向。是的，我们知道托管模型并不容易。我们知道在推理端有很多工作要做，以使服务高效。在训练端也有很多工作要做，因为你需要设计出内存高效的架构，例如，这就是 Mixtral 7B 擅长的，因为它有这种稀疏注意力机制，使其更节省内存。所以这是你在训练端可以做的一些工作。但为了充分发挥一个好模型的所有优势，你确实需要在推理部分做大量工作。因此，我们正在积极努力，使这部分尽可能高效，以构建一个非常具有成本效益的平台。所以你需要一个好的平台，拥有良好的代码和推理代码。你可以向客户提供的另一件事是跨客户的时间共享。所以，当你想尝试一个模型时，如果你想让其完全安全，你应该在云提供商的实例上启动它。但如果你只是想玩玩，你可以通过 API 访问，这样成本更低，因为单个 H100 就可以服务数百个客户。所以我认为有一些需求，大量的实验和 API 需求，这是我们开始构建的东西，同时还有我们面向其他企业客户的自托管平台。

The directions that's heading in. Yeah, so we know that hosting models isn't easy. We know there's a lot of work to be done on the inference side to make serving efficient. There's a lot of work to be done on the training side because you do need to come up with architectures that are memory efficient, for instance, that's what Mixtral 7B is good at because it has this sparse attention mechanism that makes it more memory efficient. So that's some work that you can do on the training side. But in order to realize all the benefit of a good model, you do need to work a lot on the inference part. And so we are actively working on that part to make it as efficient as possible, to build a platform that will be very cost efficient. And so you do need to have a good platform with good code, good inference code. The other thing that you can propose to customers is the fact that you do time sharing across customers. So when you want to play around with a model, if you want to make it completely safe, you should spin it up on an instance of a cloud provider. But if you just want to play around with it, you can access an API; it's going to be less costly because just a single H100 can serve hundreds of customers. So I think there is some demand, a lot of demand for experimentation and APIs, and that's something that we started to build alongside the self-hosted platform that we direct to other enterprise customers.

你的团队在法国。你之前说过，你认为有机会成立一家真正重要的 AI 公司，这家公司是法国和欧洲的，并为全球服务。我不知道在 Mistral 出现之前这是否是主流观点。你能谈谈为什么你认为这可能会成功吗？

Your team is based in France. You have said before that you think there's an opportunity for a really important AI company that is French and European and serving the world. I don't know if that is like a mainstream point of view before the arrival of Mistral. Can you talk about why you think that might work?

我认为欧洲在该领域的一些强项是人才。事实证明，法国、英国、波兰非常擅长培养数学家，而事实证明，数学家非常擅长做 AI。这意味着 AI 领域有很多法国人、英国人和波兰人，而且他们中的许多人希望留在欧洲。他们的家人在那里，食物更好，你有很多优势——我无法一一列举，那会太长。所以显然，我们看到了伦敦 AI 生态系统的出现，我认为很大程度上要归功于 DeepMind，然后在巴黎也要归功于 DeepMind 和在那里设立实验室的 Meta，以及许多开始回流的创业者。所以今天，我认为巴黎有数百家初创公司。这显然不是硅谷的水平，但我们开始拥有一个生态系统，有投资者，也有运营者进行投资。所以，让旧金山和湾区成功的同一种飞轮正在法国开始转动，我很高兴我们参与其中。

I think some very strong points of Europe on that domain is talent. As it turns out, France, UK, Poland are very good at training mathematicians, and as it turns out, mathematicians are very good at making AI. So that means that there are a lot of French people, English people, and Polish people in AI, and many of them want to stay in Europe. Their family is there, the food is better, you have many advantages—I can't list them, it would be too long. And so obviously we've been seeing the emergence of an AI ecosystem in London, I think very much thanks to DeepMind, and then in Paris also thanks to DeepMind and to Meta that settled a lab there, and to a lot of entrepreneurs that started to come back. So today we have, I think, hundreds of startups in Paris. This is not the level of Silicon Valley obviously, but we start to have an ecosystem in place with investors, with operators investing as well. So it's the same kind of flywheel that made San Francisco and the Bay Area successes is starting to spin in France, and I'm very glad that we are participating to it.

这是一次很棒的对话，Arthur。我一直觉得你很鼓舞人心。我非常非常感激能成为投资者。谢谢你接受采访。

It's been a great conversation, Arthur. I always find you inspiring. I'm very, very grateful to be an investor. Thanks for doing this.

好的，谢谢你邀请我，期待很快见到你。在 Twitter 上关注我们 @no_prior_pod，订阅我们的 YouTube 频道如果你想看到我们的脸，在 Apple Podcast、Spotify 或任何你收听的地方关注节目。这样你每周都能收到新剧集。并在 no-pri.com 注册邮件或查找每集的文字记录。

Well, thank you for having me, and looking forward to seeing you soon then. Find us on Twitter at @no_prior_pod, subscribe to our YouTube channel if you want to see our faces, follow the show on Apple Podcast, Spotify, or wherever you listen. That way you get a new episode every week. And sign up for emails or find transcripts for every episode at no-pri.com.