Mistral 发布 Boxtral TTS：新型架构实现高效语音生成

好的，欢迎来到 Lain Space。我们在演播室与我们的可靠联合主持人 Vibhu 在一起。欢迎。

Okay, welcome to Lain Space. We're here in the studio with our trusty co-host Vibhu. Welcome.

嘿，我对这一集很兴奋。

Hey, I'm excited for this one.

还有来自 Mistral 的 Guillaume 和 Pavan。欢迎。

As well as Guillaume and Pavan from Mistral. Welcome.

很高兴来到这里。

Excited to be here.

谢谢邀请。

Thank you for having us.

Pavan，你是 Mistral 音频研究的负责人，Guillaume，你是首席科学家。我们今天要宣布什么？我们这次发布与你们协调。

Pavan, you are leading audio research at Mistral and Guillaume, you're a chief scientist. What are we announcing today? We're sort of coordinating in this release with you guys.

是的，我们正在发布 Boxtral TTS。所以，这是我们的第一个生成语音的音频模型。这不是我们的第一个音频模型。我们之前发布过几个。夏天的时候我们发布了一个 Boxtral，那是我们的第一个音频模型，但它是转录模型，ASR。像人工翻译一样，我们在此基础上发布了一些更新，支持更多语言，还为我们的客户提供了很多表格堆叠功能，比如上下文偏置、说话人分离、转录中的时间戳。我们还有一些实时模型，不仅可以转录音频的结尾，还可以实时转录。而这里，这是音频领域的自然延伸。所以，基本上是语音生成。我们支持九种语言。这是一个非常小的模型，3B 模型，所以非常快，并且在同等性能上达到了最先进水平，与最好的模型处于同一水平，但在成本上效率更高，而且成本只是 150 尾模型的一小部分。他们总是喜欢这个模型真的存在。

Yeah, so we are releasing Boxtral TTS. So, it's our first audio model that generates speech. It's not our first audio model. We had a couple of releases before. We had one in the summer that was Boxtral, our first audio model, but it was a transcription model, ASR. Like a human translator, we released some updates on top of this, supporting more languages, also a lot of table stack features for our customers like context biasing, diarization, timestamping in the transcription. We also had some real-time model that can transcribe audio not just at the end of the audio. It doesn't need to feed your entire audio file, but can also come in real time. And here this is kind of the natural extension in the audio. So, basically speech generation. So, we support nine languages. And this is a pretty small model, a 3B model, so it's very fast, and also state-of-the-art in equal performance at the same level as the best model, but it's much more efficient in terms of cost, and also only a fraction of the cost of 150 tails. They're only always something that we like that this model really exists.

是的，内存相关？

Yeah, memory linked?

就是这样。

That is that.

是的，决策因素是什么？

Yeah, what's the decision factor?

这是个好问题。还会有更多。还会有更多。

It's a good question. There will be more. There will be more.

哦。是的，首先，关于你们已有的内容，有什么研究笔记要补充吗？

Ooh. Yeah, for one, any sort of research notes to add on what you have?

是的，也许我们稍后会在预测中深入探讨，但这是我们内部开发的新颖架构。我们在几个内部架构上迭代，最终得到了一个自回归流匹配架构，并且还有一个新的内部神经音频编解码器，它将音频转换为 1.5 Hz 的潜在令牌，包括语义令牌和声学令牌。是的，这是这个模型的新部分，我们非常兴奋它能有这么好的质量。就像你刚才提到的，它是一个 3B 模型。它基于我们几个月前发布的 Mistral 模型。它是一个主干。它主要用于 TTS 任务，但模型中也保留了原生文本能力。

Yeah, maybe we'll dive into it later in the forecast, too, but it's a novel architecture that we developed in-house. We iterated on several internal architectures and ended up with an autoregressive flow matching architecture, and also have a new in-house neural audio codec, which converts audio into 1.5 Hz latent tokens, semantic and acoustic tokens. And yeah, that's the new part about this model, and we're pretty excited that it came out with such good quality. And like you were just mentioning, yeah, it's a 3B model. It's based off of the Mistral model that we actually released just a few months back. And it's a trunk. And it's mainly meant for the TTS stuff, but the native text capabilities are also there in the model.

所以，有很多要讲的。我一直喜欢任何与新颖编码相关的东西，因为我认为这显然能提高很多效率，但有时也可能出现错误。你之前在 Gemini，从事语言模型的后训练工作。也许很多人对音频模型的总体经验比纯语言模型要少。当你加入 Mistral 并开始做这个时，你发现有什么需要从头重新审视的吗？

So, there's a lot to cover. I always love anything to do with novel encodings and all those things because I think that obviously increases a lot of efficiency, but also maybe bugs sometimes happen. You were previously at Gemini, and you worked on post-training for language models. And maybe a lot of people will have less experience with audio models just in general compared to pure language. What did you find that you have to sort of revisit from scratch as you joined Mistral and started doing this?

至少就音频而言，我认为有两个方面：音频理解和音频生成。音频理解，比如之前提到的 Walk Through 模型，是更早发布的。它们是 Vox 和 chat，我认为是去年七月发布的，以及后续的仅转录模型系列，是在一月发布的。这可以算作一个方面，而生成是另一个方面。我认为你也可以将它们视为统一的模型集，但目前这两种方法有些不同。关于你提出的音频如何输入模型的问题。在理解模型中，它实际上与我们发布的像素模型非常相似。那很棒。那是我加入 Mistral 后参与的第一个项目。非常不错。而 Vox 在精神上也非常相似。所以，我们通过一个音频编码器输入音频，类似于通过视觉编码器输入图像，它产生连续的嵌入，然后作为令牌输入到主要的 Transformer 解码器 Transformer 模型。是的，模型输出只是文本。所以在输出端，这类模型不需要做任何事。我认为生成部分的有趣之处在于，现在输出必须产生音频。我们的方法是这个神经音频编解码器，它将音频转换为这些潜在令牌，有很多现有文献和模型基于这种方法，我们对此做了一些稍微不同的设计决策，但归根结底，神经音频编解码器将音频转换为 12.5 Hz 的潜在集，每个潜在包含一个语义令牌和一组声学令牌。想法是，你取这些离散令牌，然后在输入端馈入。有几种方法可以在每一帧融合它们，但我们只是对嵌入求和，所以这有点像有 K 个不同的词汇表，并将它们全部组合起来，因为它们都对应于输入侧的一个音频帧。输出端是有趣的部分。在输出端，我不知道这是否是最流行的，但一种流行技术是使用深度 Transformer，因为你在每个时间步有 K 个令牌。就像文本一样，每个时间步只有一个令牌。所以，你只需从词汇表中预测令牌，得到概率。

At least when it comes to audio, I think the two buckets are audio understanding and audio generation. The audio understanding like the Walk Through models that came was mentioning that were released earlier. They are Vox and chat that were released I think July last year and the follow up transcription only models family that were released in January. That could be one bucket I guess and the generation is another bucket. I think you can also treat them as a unified set of models but currently the approaches are a little different between these two. To your question on how audio is fed to the model. In the understanding model it's very similar to actually pixel model that we also released I guess. That was amazing. It was pretty that was the first project I worked on after joining Mistral. It was pretty nice. And Vox still was very similar in spirit I guess. So we feed audio through an audio encoder similar to images through a vision encoder and it produces continuous embeddings and which are fed as tokens to the main transformer decoder transformer model. Yeah, and the model output is just text. So on the output side there is nothing that needs to be done in these kinds of models. I guess the interesting part about the generation stuff is the output now has to produce audio. And the approach that we have is this neural audio codec which converts audio into these latent tokens that is a lot of existing literature and lot of models which are based off of this kind of approach and we took a slightly different design decisions around this but at the end of the day the neural audio codec converts audio into a 12.5 hertz set of latents and each latent has a semantic token and a set of acoustic tokens. And the idea is that you take these discrete tokens and then feed it on the input side. There's several ways to fuse this at each frame but we just sum the embeddings so it's kind of like having K different vocabularies and kind of combine all of them because they all correspond to one audio frame on the input side. The output side is the interesting part. On the output side, that's not the I don't know if it's the most popular, but one popular technique is to have a depth transformer because you have K tokens at each time step. Like with text, you just have one token at each time step. So, you just do predict the token from the vocabulary with probabilities.

这是一个非常直接的技术。

This is a very straightforward technique.

非常直接。是的，但如果你有 K 个令牌，那么主要的事情就是并行预测所有令牌，但这行不通。至少效果不太好，因为音频具有更高的熵。人们使用的一种技术是深度 Transformer，你几乎有一个小的 Transformer，或者也可以是 LSTM RNN，但人们使用 Transformer，然后你以自回归方式预测 K 个令牌。

Very straightforward. Yeah, but if you have K tokens, then the main thing would be to predict all of them in parallel, but that doesn't work. At least that doesn't work that well because audio has more entropy. And one of the techniques people use is this depth transformer where you almost have a small transformer or it can be an LSTM RNN as well, but people use transformers and you predict the K tokens in auto-regressive fashion in that.

所以，这里有两个自回归过程。我们做的不同之处在于，我们没有采用自回归的 K 步预测，而是使用了流匹配模型。我们没有将其建模为离散的 token 集合，而是训练编解码器同时支持离散和连续，以获得这种灵活性。我们也尝试过离散方法，效果不错，但连续方法更好。所以我们采用了流匹配。这是一个流匹配头，它从主 Transformer 获取潜在表示，类似于扩散中的去噪，但在流匹配中，它是速度估计。从带噪的潜在表示一直走到对应 80 毫秒音频的音频潜在表示，然后通过声码器还原成 80 毫秒的音频帧。

So, you have two auto-regressive things going on. The thing we did differently is instead of having this auto-regressive K step prediction, we have a flow matching model. Instead of modeling this as a discrete token set, we train the codec to be both discrete and continuous to have this flexibility. We did try the discrete stuff too, and it works well, but the continuous stuff works just better. So we did this flow matching. It's a flow matching head which takes the latent from the main transformer and, like in diffusion, it's denoising, but in flow matching, it's a velocity estimate. You go from this noised latent all the way to the audio latent which corresponds to the 80-ms audio, and then it's sent through the vocoder to get back the 80-ms audio frame.

这是流匹配在音频中的首次应用吗？因为我通常是在图像领域看到这个。

Is this the first application of flow matching in audio? Because I usually come across this in images.

实际上，从某种意义上说，音频中已经有流匹配模型，但我觉得这种特定组合……我可能错了。可能有一些我没见过的工作。

Actually, in some sense, there are flow matching models in audio, but I think this specific combination... I could be wrong. There could be some work I haven't seen.

我在这方面没看到太多工作。所以我认为这是新颖的。很大程度上是因为现在的社区规模更大。我认为他们在扩散流匹配方面开创了很多工作，将其中一些想法应用到音频中很有趣。就我个人而言，关键部分是尝试……我的一个主要观点是，与文本不同，即使在视觉领域我认为也是如此，但在音频中绝对如此：还没有一个赢家模型。没有所谓的“这就是你应该做的方式”。它仍在演变。我认为人们仍在迭代，试图找出最佳的整体方案。我确信有一些模型是完全端到端的，比如 Nato Audio，但还没有收敛到正确的思考方式。这也使得这个领域探索起来非常令人兴奋。

I haven't seen much work in this. So I think it's novel. A lot of it is just a way bigger community like today. I think they pioneer a lot of these diffusion flow matching works, and it's interesting to adopt some of the ideas there into audio. Personally, that's the key part: trying out one of my main points is that unlike text, even in vision I think this is true, but in audio it's definitely true: there is no winner model yet. There is no 'this is the way you do things.' It's still evolving. I think people are still iterating and figuring out what's the best overall recipe. I'm pretty sure there are models which are also completely end-to-end like Nato Audio, but it's still not converged on the right way to think. That also makes the space pretty exciting to explore.

有哪些看待它的方式？有些方法可以对音频生成做扩散，但如果你想要实时生成，那对你采用的方法来说是个大事。另外，你如何评估你所关心的不同维度？

What are some of the ways to look at it? There are ways where you can do diffusion for audio generation, but if you want real-time generation, that's a big thing with the approach I'm assuming that you took. And also, how do you go about evaluating different axes of what you care about?

好问题。你可以对整个音频做流匹配扩散。我们甚至没有走那条路，因为一个主要应用是语音智能体，我们需要实时流式传输，这是用例之一。这不是唯一的用例，但它是我们想要实现的主要用例之一。所以我们为此选择了自回归方法。在自回归空间内，你可以逐块进行，或者……我个人更喜欢最简单的方法。所以我们尝试看看能否将音频作为另一个头添加到我们常规的 Transformer 解码器模型中，因为这样更容易实现最终的端到端音频-文本原生建模。效果很好。所以我们采用了那个方法。我们对流匹配头本身做了一些调整。我们有一种离散扩散的方法，效果也不错，但流匹配更好。

Good point. You can do just flow matching diffusion for the whole audio. We didn't even go down that path because one of the main applications is voice agents and we want real-time streaming, and that's the use case. That's not the only use case, but it's one of the primary use cases we want to get to. So we picked the autoregressive approach for that. Within the autoregressive space, you can do chunk by chunk or you can do... I personally prefer the approaches which are the simplest. So we tried to see if we can just add audio as another head to our regular transformer decoder model, because that makes it easier for eventual end-to-end modeling of audio-text native modeling. It works pretty well. So we went with that. We tinkered a little bit with the flow matching head itself. We had a discrete diffusion kind of approach, which also works well, but the flow matching one is better.

我只是好奇你如何看待这个整体研究方向。当你与音频团队合作时，你是设定一些高层参数然后让他们自由探索，还是你们之间如何运作？

I was just curious about how you also think about this overall direction of research. Do you, when you work with the audio team, set some high-level parameters and then let them explore whatever, or how does it work between you guys?

不，我认为运作方式是，我们共同确定哪些功能最重要。因为在音频领域你可以做很多很多事情。我们试图决定应该如何做。例如，最终我们想要构建这个全双工模型，但我们不会直接开始。这是我的一些项目来放出光束，但是……

No, I think the way it works is that we prioritize together what are the most important features. Because there are many, many things you can do in audio. We try to decide how we should do things. For instance, ultimately what we want to do is to build this full duplex model, but we are not going to start that directly. It's some of my project to put out the beam, but...

确认一下，全双工是指它可以在我说话的同时说话吗？

Just to confirm, full duplex means it can speak while I'm speaking?

好的，音频在你的音频集中。

Okay, audio in your audio set.

是的，是的。

Yeah, yeah.

所以，最终我们会达到那个目标，但对我们来说，我们决定一步一步来。我们从最重要的开始，我认为对客户来说也是如此，转录是最流行的用例。然后是语音生成，实时生成就在那之前。然后我们会尝试将所有东西结合在一起。但我们认为在合并之前，先分开并逐一优化每个能力也很重要。

So, ultimately we are going to get there, but for us we decided to take it step by step. We start with whatever is the most important, I think also for our customers, which is transcription is the most popular use case. And then there's speech generation, the real-time just a bit before that. And then we are going to try to combine everything all together. But we thought it was also important to separate things and optimize each capability one by one before we merge all of that together.

然后是超级全能模型。

And then the super omni model.

但有趣的是，正如我所说，当你研究这个 LM 的其他领域和其他东西时，有很多领域我认为不那么有趣。例如，很多地方基本上只是围绕数据或创建新环境，以及很多容易的事情，但研究可能不那么有趣。而在音频中，有很多实际构建这个模型的方法。有很多方法可以绕开它。这听起来，我认为，真的很有趣。我们在语音生成方面也尝试了多种方法。有趣的是，尽管它们非常不同，但最终结果相当接近。但流匹配结果更自然，所以我们对此很满意。

But interestingly, as I was saying, when you work on some other domains of this LM and other things, there are many areas where I think it's not as interesting. For instance, many places it's essentially just around data or creating new environments and a lot of kind of easy things, but things where I think the research is maybe not as interesting. While in audio, there are so many ways to actually build this model. So many ways to go around it. This sounds, I think, is really interesting. On what we also tried for speech generation is a picture of multiple approaches. What was interesting is that even though they were extremely different, they ended up being at the end of the day pretty close. But the flow matching turned out to be quite more natural, so we are happy with this.

有没有直觉上的原因，为什么流匹配在某种自然的基本潜在维度上能更好地建模语音？

Is there intuition why flow matching just models speech better in some natural fundamental latent dimension?

不，我认为主要问题是，即使在特定的时间步，也存在一个需要预测的分布。比如你变调的方式。所以你已经知道你在说的词。在文本空间中，简单来说，这个词映射到单个 token。大多数情况下是这样。所以没有太多……你只需选择这个词。但在音频中，即使是同一个词，即使是你自己的声音，也可以有多种不同的变调方式。我认为任何能很好建模这个分布的方法，流匹配是其中之一——它绝对不是唯一的方法——但它是一个效果相当不错的方法。我认为这样更好。

No, I think the main thing is that even at a particular time step, there is a distribution of things to be predicted. Like the way you inflect. So you already know the word that you're speaking. In text space, let's say the word maps to just a single token for simplicity. In most cases, it does. So there is not a lot of... you just pick the word. But within audio, even the same word, even with your own voice, could be inflected in so many different ways. I think any approach which models this distribution well, and flow matching is one of them—it's not the only one at all—but it's the one which works pretty reasonably well. I think that's better.

所以，你必须在几个不同的簇中进行选择，每个簇对应某种特定的语调或发音方式，你不能预测平均值，因为那对应模糊的语音，但你必须选一个，然后变得清晰。

So, you have to pick across several different clusters, each corresponding to some specific way you would inflect or pronounce that thing, and you can't predict the mean because that corresponds to blurred speech, but you have to pick one and then sharp.

条件推理。

Conditional inference.

对，没错。

Yeah, exactly.

这些都属于‘不流畅’（disfluencies）这个术语的范畴吗？

Is that all covered under disfluencies, which is the normal term of art?

不流畅、停顿、语调。顺便说一句，我得感谢 Sophia 安排这一切，包括这些很好的笔记，因为我对音频不太熟悉。

Disfluencies, pauses, intonations. By the way, I would have to thank Sophia for setting all this up, including some of these really good notes, because I'm less familiar with the audios.

不，不，不。我认为不流畅肯定是这种现象之一。不流畅更像是……

No, no, no. I think disfluencies are definitely one such phenomenon. Disfluencies is more like...

就是‘嗯’、‘啊’之类的。

Which is 'ums', 'ahs'.

对，‘嗯’、‘啊’，还有重复，比如‘你觉得……像……你这样做填充词，你在思考，所以你会重复这个词’。

Yeah, 'ums', 'ahs', and also repeats like 'you feel like... like you do this filler words you're thinking so you repeat the word'.

好的。而语调则是……它由说话的所有这些组成。

Okay. Whereas intonation is like a... it's up of speak all this okay.

是的，所以我认为将其建模为分布有很多熵，任何有助于处理它的技术，而深度 Transformer 是一种条件建模方式，Transformer 实际上很擅长这个，但它是迷你 Transformer，所以我认为对我们来说效果也很好。只是主要考虑是，当你有一个深度 Transformer 时，如果有 K 个 token，你需要进行 K 步自回归，所以即使它很小，也是 K 步，非常重。但通过流匹配，我们能够显著减少步数，所以我们可以在 4 步或 16 步内完成推理，效果很好，而且还有更常规的技术可以进一步减少，比如极端情况下一步完成。我们还没做到，但至少这个框架本身支持更高效的推理。

And yeah, so I think there is a lot of entropy in modeling it as a distribution, and any technique which helps with it, and the depth transformer is a conditional way of modeling this, and transformers are actually good at it, but it's a mini transformer, so I think that work pretty well too for us. It's just that the main consideration is when you have a depth transformer, if you have K tokens, you need to do K auto-regressive steps, so even though it's a small thing, it's K steps, which is very heavy. But flow matching we were able to cut it down significantly, so we are able to do the inference in 4 steps or 16 steps, and it works pretty well, and there are more normal techniques to bring it down even further to like in the extreme case of one step. Like we're not doing it yet, but at least the framework lends itself to more efficient inference.

是的，图像领域的人做了了不起的工作。

Yeah, and the image guys have done incredible work.

现在你只需发送提示，就能得到一张图像。

Now you just send the prompt and you get an image.

是的，令人惊讶的是，我觉得图像模型实验室并没有在生产中充分使用这些技术。我认为很多都是研究演示，但今天我在手机上还无法使用。

Yeah, surprisingly not enough I think image model labs use those techniques in production. I think it's a lot of research demos but nothing I can use on my phone today.

问题是，与音频相比，视觉社区已经做了很多工作。对我来说，我认为有很多学习信息和很多方法可以进一步改进这个范式。所以我不知道这是不是我们的第一个版本，但我们有很多方法可以做得更好、更高效、更具成本效益。

The thing is, there has been so much work done in the vision community compared to audio. To me, I think there are so many learning infos and so many things we can do to actually improve this paradigm further. So I don't know if it's our first version, but we have so many ways to exist much better, much more efficient, cost-efficient.

是的。

Yeah.

所以当然你确实有这种感觉，但还有很多事情可以做，但我觉得这只是……

So really certainly you feel that all of course, but there are still so many things that can be done, but I feel that it's just...

我还应该提一下，对于刚接触流匹配的人，我认为创造者是一个叫 Alex 的人。他在 NeurIPS 上，可能两年前，做了一个非常好的研讨会，大约一小时关于流匹配。我推荐大家去看看。这是另一件事，对吧？在效率方面，我想你选择 3.6B 骨干、2.4B，是为了适应某种硬件约束。是什么约束？

I should also mention for those who are newer to flow matching, I think the creator is a guy named Alex. He did a very good workshop at NeurIPS maybe two years ago, about one hour on flow matching. I recommend people look that up. That's the other thing, right? The efficiency wise, I imagine the reason you picked a 3.6B backbone, 2.4B, you are trying to fit some kind of hardware constraints. What are they?

不一定。我认为我们在模型中关心的一点是效率。所以我们有很多单独的模型。我们有一个模型，我们知道它非常小、非常高效。我们还有一个小模型，也非常好、非常高效。我认为其他公司可能会采取的方法是用一个非常通用的模型做所有事情，但那也会很昂贵。另外我想说的是，如果你关心这个特定用例，你可以使用这个模型。它只做这件事，但非常擅长且高效。这就是为什么我们可以访问模型。或者你可以访问我们的 CR，它们非常擅长这个，而且比另一个包含很多你不需要功能的通用模型更具成本效益。所以我们既做通用模型，也做更定制化的模型。

Not necessarily. I think something we care about in our models is that they are efficient. So we have a lot of separate models for instance. We have this model that we know is very small, very efficient. We also have a small model that is very good, very efficient as well. And I think an approach that maybe other companies are going to take is to have a very general model that will do a bit of everything, but that is also going to be expensive. And the other thing I want to say is if you care about this specific use case, you can actually use this model. It just does that. It's extremely good at it but also very efficient. That's why we can actually access models. Or you'll be able to access our CR that are really good at that and will be much more cost-effective than another general model that contains a lot of capabilities you don't really need. So we're doing general models but also more customized models.

它与其他 TTS 模型相比如何？你完全开源，就这样发布……

How does it compare to other TTS models? You're going full open wave, you're just dropping it like...

我认为它相当不错。

I think it's pretty good.

是的，我认为它相当不错。它绝对是最好的之一。

Yeah, I think it's pretty good. It's definitely one of the best.

当然，当然。它可能是目前最好的开源模型。

For sure, for sure. It's probably the best open source model right now.

你只是在自夸吧？

You are just hyping yourselves, right?

是的。

Yeah.

为什么是现在？它如何融入 Mistral 的更大愿景？你怎么看语音智能体？你怎么看语音……

Why now? How does it fit into broader Mistral vision? How do you see voice agents? How do you see voice like...

我觉得每年都听到‘好了，你是语音，你是语音。’有很多架构上的东西。你在解决很多端到端延迟问题，但你认为语音的未来方向是什么？

I think every year I've heard 'okay, you're a voice, you're a voice.' There's a lot of architectural stuff. There's a lot of end-to-end latency that you're solving, but where do you see voice heading?

我们有很多客户要求语音功能。这也是我们想要构建它的原因。这个领域有趣的是，从某种意义上说，如果你拿像转录这样简单的事情来说，它看起来不应该是模型很难做到的事情。它本质上只是模式识别，是分类。这些模型非常擅长分类，对吧？然而，当你与它们对话时，它还没有达到那个水平。你无法像与人交谈那样与它们交谈。也许人们没有意识到这一点。在英语中，它仍然比其他任何语言好得多。即使与法语相比，例如，如果你用法语与这个模型对话，当你看到人们与这个模型对话时，他们会说得很慢。他们会尽可能清晰地发音。所以，这不自然。我们还没有达到这个程度。也许下一代不会知道这一点，但我们这个年龄段的人在与这个模型对话时，总会保持说得很慢的偏见，即使可能在几年后，也许明年，就不再需要了。但有趣的是，即使对于像法语、西班牙语、德语这样不是低资源语言的语言，你有很多音频数据，它们仍然不够好。所以，我认为原因只是没有像其他模态（如视觉或编码）那样投入那么多的精力或努力。但是，是的，还有很多进步空间。

We had so many customers asking for voice. That's also why we wanted to build it. What's kind of interesting in this domain is that in a sense, if you take something simple like transcription, it doesn't seem like something that should be very hard to do for a model. It's just essentially pattern recognition. It's classification. These models are very good at classifying, right? Nonetheless, when you talk to them, it's not there yet. You don't talk to them the same way you talk to a person. Maybe people don't realize it. In English, it's still much better than in any other language. Even compared to French, for instance, if you talk to this model in French, when you see people talking to this model, they will talk very slow. They will articulate as much as they can. So, it's not natural. We are not yet to this point. Maybe the next generation will not know this, but people our age will always keep this bias of speaking very slowly when they talk to this model, even if maybe in a couple of years, maybe next year, it will not be necessary anymore. But what's interesting is to see that even for languages like French, Spanish, German, that are not low-resource languages, you have a lot of audio with this. They still aren't as good. So, I think the reason for this is just that there is not as much energy or effort that has been put into some other modalities like vision or coding. But yeah, there is still a lot of progress to be done.

我认为这只是需要做一些工作的问题。有一条清晰的路径，我觉得，可以到达那里。

I think it's just a question of doing some work on this. There is a clear path, I think, to get there.

这真的很迷人，因为我之前做过 Google Assistant 的工作，但当你退一步看，这很迷人。就在不久之前，大概四五年吧，现在完全是音频输入输出，函数调用和整个过程端到端地以非常自然的方式完成。但正如你所说，还有改进空间。尽管取得了所有进展，当你与这些智能体机器人或语音模式对话时，仍然不像在和人说话。还有差距。我认为这是好事，但我觉得即使使用现有的技术栈，我们应该很快就能实现这种非常自然的语音对话能力，我们也希望能达到那个目标。在客户方面，因为当人们与这些智能体对话时，通常只是打字。有时非常明确，比如你想写代码，并且很清楚你希望模型如何实现你的想法。但你必须花很多时间打字。所以效率不高，而音频是一种自然的界面，只是还没实现，但我认为至少它会实现的。

It's really fascinating because I worked on Google Assistant a while back, but it's kind of when you take a step back, it's fascinating. It's not that long ago, like 4 or 5 years ago, and now it's completely audio in and audio out, and the function calling and the whole thing happens end-to-end in a very natural way. And still there are ways to go, as you were saying. Even despite all the progress, it's not like you're speaking to a person when you talk to any of these agent bots or voice mode. There's still a gap. I think that's the great thing, but I feel like with even the existing stack, we should be able to get to this very natural speech conversational abilities soon enough, and we'll hope to get there. On the customer side, because when you talk to these agents, usually people are just writing to them. Sometimes it's very clear, for instance, you want to write code and you have a very clear idea of how you want the model to implement what you had in mind. But then you have to spend a lot of time writing. So it's not really efficient, whereas audio is really a natural interface that is just not there yet, but I think it's just going to be there at least.

是的，我认为这正是我们与客户合作的主要内容。他们来找我们通常出于不同原因。一个原因是他们有很多隐私顾虑。他们的数据非常敏感，不希望数据离开公司，希望留在公司内部。所以我们帮助他们内部部署模型，无论是在本地还是私有云上。这样他们就不担心数据被交给第三方或泄露。有时许多公司有不同敏感级别的数据，比如一级、二级、三级。三级数据可以发送到云端，一级不行。这就造成了异构的工作流程，很麻烦。你不能把某些数据发到云端，但另一些可以。所以当我们为他们部署模型时，他们就没有这些顾虑了。他们不担心会泄露。一切都简单多了。所以我们在这方面帮助他们。这是一个价值主张，但另一个是，当客户使用现成的闭源模型时，很可惜的是他们没有利用自己收集了四年甚至几十年的数据。这么多数据，有时是数万亿个 token，在非常特定的领域，他们的领域，这些数据在公共互联网上是找不到的。闭源模型实际上无法访问这些数据，而这些数据会非常有用。所以如果他们使用闭源模型，基本上就没有从这些年收集的洞察和数据中受益。他们可以在推理时将其放入上下文，但这仍然不如直接在模型上训练好。所以这就是我们帮助他们做的事情。我们实际上为他们提供 Mistral purchases，基本上就是本周在 GTC 上宣布的。我们为他们提供一个平台，有很多工具来帮助他们处理数据、进行训练。这实际上和我们在科学团队中使用的是一样的。所以这是一个经过实战考验的基础设施，比如大量高效的训练代码库，用于继续预训练、微调，甚至做好 SFT。所以我们帮助他们使用与科学团队构建和使用的相同工具。因为这些工具我们已经用了两年，所以真的经过实战考验且非常复杂。我们给他们的是与科学团队内部用于构建自己 AI 的相同工具。这带来了很大的不同。我认为有时客户和许多普通人没有意识到，当你用自己的数据微调模型时，模型会变得好多少。你可以有一个模型从这里开始，闭源模型从这里开始，但如果你微调它，它可以走得更远。然后你就有了很大的优势。模型基于你整个公司的知识进行训练，所以它知道一切。你不必在每个查询中都提供 1 万个 token 的上下文。所以容易得多。

Yeah, I mean I think this is a lot of what we are doing with our own customers. Very often they come to us for different reasons. One reason is sometimes they have a lot of privacy concerns. They have data that is very sensitive. They don't want data to leave the company. They want it to stay inside the company. So we help them deploy models in-house, either on-premise or on private cloud. So they are not worried that it's given to a third party or that there is some leakage. Sometimes many companies have different sensitivity levels of data, like tier one, tier two, tier three. Tier three can send it to the cloud, tier one can't. So it creates heterogeneous workflows that are kind of annoying. You cannot send some data to the cloud, but this one you can. So when we deploy the model for them, they don't have these considerations. They are not worried that this is going to leak. Everything is much easier. So we help them with that. That's one value proposition, but the other is very often, when customers use off-the-shelf closed models, what's very sad is that they are not leveraging the data they've been collecting for 4 years or for decades. So much data, sometimes trillions of tokens of data in a very specific domain, their domain, which is data that you will not find on the public internet. Data on which a closed model actually does not have access to and which is going to be really good. So if they are using closed-source models, they are basically not benefiting from all these insights, all these data they've collected through years. They can always give it into context at inference, but it's still not as good as if you actually train the model on it. So that's basically what we help them to do. We actually provide them with Mistral purchases, basically what we announced at GTC this week. So we provide them with a platform with a lot of tools to actually help them process data, train on that. It's actually the same thing that we are using in the science team. So it's a very battle-tested infrastructure, like a lot of efficient training codebase for continue pre-training, fine-tuning, even doing SFT well. So we help them do this using the same tools as what our science team is building and using. Since it's tools that we've been using for 2 years now, it's really battle-tested and sophisticated. So we are giving them the same thing that our science team uses internally to actually build their own AI. And it makes a really big difference. I think sometimes customers and many people in general don't realize how much better the model becomes when you fine-tune it on your own data. You can have a model starting here, and a closed-source model starting here, but if you actually fine-tune it, it can go much further. And then you have a very big advantage. The model is trained on your entire company knowledge, so it knows everything. You don't have to feed 10K tokens of context at every query. So it's much easier.

我认为使用闭源模型真的很可惜，因为你没有充分利用自己的数据，而且当你本可以使用多年来收集的所有宝贵数据时，你却要和所有竞争对手使用同一个模型。所以我们帮助客户解决这个问题。我们有很多解决方案，工程师会进入公司，查看客户面临的问题、他们正在努力解决的事情，以及我们应该如何解决。我们共同帮助他们解决问题。我认为我们的方法与其他一些公司和竞争对手略有不同。我们不只是发布一个端点并在上面加些东西，或者只给一个检查点。我们与客户密切合作，查看他们的问题，帮助他们解决，并针对他们面临的问题制定量身定制的解决方案。例如，有时客户想要一个在某种亚洲语言上表现非常好的模型。如果你看一些已发布的模型，它们可能无法很好地用那种语言说或写，因为那种语言可能只占训练数据的 0.1%。它在训练中被包含，但非常少。所以我们做的是为他们训练一个新模型，其中那种语言占混合数据的 50%，这样强多了。它知道所有方言和俚语。这就是我们能做的事情的一个例子，而且非常定制化。有些客户想要一个能做音频且函数调用能力很强的 3D 模型，用于厨房或汽车。他们希望它离线工作，因为汽车里不一定有网络。所以我们可以构建这些解决方案。互联网上没有现成的模型。你有非常通用的模型，推理能力很强，但对于特定事情，他们总是想要特定的解决方案。有时他们来找我们，是因为他们尝试了一些闭源模型，做了原型，对性能很满意，然后想投入生产，却发现极其昂贵。你无法推广这个。所以他们回来找我们说，‘你能帮我们用便宜得多的方式构建同样的东西吗？’我们有时可以通过微调模型，部署在他们自己的服务器上，构建出便宜 10 倍的东西，而且性能更好，也更便宜。

I think using a closed-source model is really sad because you are not leveraging all your data, and you will be using the same model as all your competitors when you could use everything you have been collecting for years, which is really valuable. So we help customers with this. We have many solutions deployed for engineers who go into the company, look at the problems customers are facing, what they are struggling with, and what we should do to solve it. We help them solve them together. I think our approach is a bit different from some companies and competitors. We don't just release an endpoint and put some stuff on top of it, or just give a checkpoint. We work very closely with customers, look at the issues they have, help them solve them, and make tailored solutions for the problem they are facing. For example, sometimes we have customers who want a really good model performing well on some Asian languages. If you take some of the shipped models, they can't speak or write well in that language because it might be only 0.1% of the mixture. It was included during training but very little. So what we did is we trained a new model for them where that language was 50% of the mix, which is much stronger. It knows all the dialects and slang. That's an example of what we can do, and it's really arbitrary custom. Some customers wanted a 3D model that can do audio with very good function calling, something you want in the kitchen or car. They wanted it to be offline because in the car you don't necessarily have internet access. So we can build these solutions. There is no out-of-the-box model on the internet for this. You have very general models that are strong at reasoning, but for specific things, they always want specific solutions. Sometimes they come to us because they experiment with some closed-source model, get a prototype, are happy with the performance, and then want to go to production. Then they realize it's extremely expensive. You cannot push this. So they come back to us and say, 'Can you help us build the same thing but much cheaper?' And we can sometimes build something 10x cheaper by just fine-tuning a model on-prem on their own server, which is also much cheaper.

这就是错失的机会。把钱都赚了。

That's the missed opportunity right there. Take all the money.

除此之外，我们确实发布了开放权重模型，让人们可以自己做。但没多少人会费心去做。他们不会的。他们会让我们来做。

Outside of that, we do put open-weight models so people can do this themselves. But not enough people go out of their way to do it. They're not going to. They're going to ask us to do it.

我明白怎么回事了。

I see how it works.

最初，我们公司刚起步时并不具备竞争力，因为我们的策略和现在不完全一样。但我们最初低估的是部署这些模型的复杂性，比如将它们连接到所有东西以确保能访问公司知识。两年前我们就看到客户在这方面挣扎，现在情况复杂得多，因为现在不仅有文本和简单的指令遵循，还有推理、智能体、工具、多模态音频。所以比以前复杂得多，而即使在那时对客户来说也很难。所以他们真的需要支持，这就是为什么我们总是提供一些职位来帮助他们部署。

Initially, we weren't competitive at the beginning of the company because our strategy was not exactly the same as it is today. But what we underestimated initially is the complexity of deploying these models, like connecting them to everything to ensure they have access to company knowledge. We were seeing customers struggling with this two years ago, and now things are much more complicated because now you don't just have text and simple instruction following; you have reasoning, agents, tools, multi-modal audio. So it's much more complicated than before, and even back then it was hard for customers. So they really need support, and this is why we provide always some for the position as well to help them deploy this.

我很好奇，人们也会做语音微调吗？

I'm curious, is there also voice fine-tuning that people do?

和在这个熔炉里一样，我们也有一个统一的框架。希望就像我们今年早些时候发布的 Vox 语音转文字，甚至去年发布的 Vox 聊天。我认为有一个很大的生态系统，人们在微调 Whisper，而人们希望用 Vox 做同样的事情。它比 Whisper 强得多，平台提供那种微调，可以是任何类型的微调。例如，有时人们想支持新语言，比如小众语言，我们希望自己原生覆盖，但如果有一种语言你有数据并且想微调，我认为这是一个很好的用例。另一个用例是相同的语言，比如英语，但以非常特定领域的方式，带有术语、行话、医学内容。

Same as in this forge, we also had a unified framework. The hope is like the Vox speech-to-text that we released earlier this year, and even the Vox chat that we released last year. I think there's a big rich ecosystem of people fine-tuning Whisper, and people want the same thing with Vox. It's much stronger than Whisper, and the platform offers that kind of fine-tuning, which could be any kind of fine-tuning. For instance, sometimes people want to support new languages, tail languages, which we hope to cover ourselves natively, but if there is a language where you have data and you want to fine-tune it, I think this is a good use case. The other use case is the same language, like even English, but in a very domain-specific way with terminology, jargon, medical stuff.

是的，术语、行话、医学内容。

Yeah, terminology, jargon, medical stuff.

完全正确。还有特定的声学条件，比如有很多噪音。模型在大多数条件下表现不错，但你总可以做得更好。这些是一些你可以进一步改进的用例，这也是一个很好的用例。对于我们的文本转语音，我们刚刚发布，所以很快也会支持。我认为这是一个类似的用例。你想扩展文本转语音模型的东西有点不同，比如语音个性化、企业语音适配。许多企业需要非常特定的语调、非常特定的个性，所有这些都是微调的好用例。

Exactly. And also the specific acoustic conditions, like there's a lot of noise. The model will do decently in most conditions, but you can always make it better. Those are some of the use cases where you can improve it even further, and that's one good use case for this. And for our text-to-speech, we're just releasing it, so we'll have support for that soon too. I think it's a similar use case. It's a little different the kind of things that you want to extend a text-to-speech model to, which could be like voice personalization, voice adaptation for enterprises. Many enterprises need very specific kind of tone, very specific kind of personality for this kind of voice, and all of those are good use cases for fine-tuning.

这正是我想问你的。我们从未在这里讨论过语音克隆。它有多重要？比如，我可以克隆一个名人的声音，但……

This is what I was going to ask you. We never talked about voice cloning here. How important is it? Like, I can just clone a famous person's voice, okay, but...

主要用例是企业个性化。企业需要大量定制化。你不希望所有企业都用同一个声音。每个企业都想要一个定制化、专业化的东西，代表他们的品牌和安全考虑。你在医疗领域部署的共情助手，与客户支持机器人中的东西会非常不同，也与更对话式的方面不同。我认为这些是企业期望的定制化，至少从我们这边来看是主要用例。

The main use case would be for enterprise personalization. Enterprises need a lot of customization. You don't want the same voice for all enterprises. Each enterprise wants a customized, specialized something that is representative of their brand and also their safety considerations. The kind of thing you would deploy as an empathetic assistant in the context of healthcare would be very different from the kind of thing that would be in a customer support bot, and different from more conversational aspects. I think those are the customizations you would expect from enterprise, and that's the main use case at least from our side.

我的基本例子是，你不想打两个客服电话，听到完全一样的声音。

My base example is you don't want to call two customer services and have the same exact voice.

你知道，这肯定会很奇怪。但在技术方面，我觉得 VoxTro 有几个点很有意思。他是这篇论文的忠实粉丝，说这是他读过最好的 ASR 论文。

You know, it's just going to be weird. But also, on the technical side, there are a few things about VoxTro that I thought were pretty interesting. He's a big fan of this paper. He said it's the best ASR paper he's ever read.

是的，我已经把这篇语音论文吹捧得够多了。我们在某个地方讨论过它。

Yeah, I've hyped up this voice paper enough. We covered it somewhere.

但一个重点是：Whisper 以 30 秒处理著称，你们把它扩展到了 40 分钟。论文里有很多关于如何实现的细节，甚至包括填充的必要性等小细节，还有相关的合成数据生成。我想知道你们能否分享关于新文本转语音的类似内容？如何生成长篇连贯的输出？有什么亮点吗——会有论文吗？

But a big thing: Whisper is known for 30-second processing. You extended this to 40 minutes. There was a lot of good detail in the paper about how this was done, even little niches like how padding is very much needed. The synthetic data generation around this. I'm wondering if you can share the same about the new text-to-speech? How do you generate long-form coherent output? And any gems—is there going to be a paper?

是的，会有一份技术报告。我认为总结是：这篇论文中的一些考虑是因为我们以 Whisper 编码器为起点。现在我们有了内部编码器，比如一月份发布的实时模型，也有一份技术报告。那个模型采用双流架构——一个有趣的架构，你应该看看。它有一个因果编码器，我认为社区里没有强大的多语言因果编码器，所以我们认为这是一个很好的贡献。我们从头训练了它。我们的全栈现在已经足够成熟，可以训练超强编码器，像填充这样的考虑是 Whisper 编码器的特性。现在我们内部训练编码器，设计考虑就不同了。对于文本转语音，它依赖于原始的自回归解码器主干。考虑几乎相同。模型以 12.5 Hz 处理音频，所以 1 秒对应 12.5 个 token。1 分钟大约 720 个 token。在 8K 上下文窗口中可以获得最多 10 分钟，在 30K 上下文窗口中可以获得半小时。32K 上下文是我们非常舒适的训练长度。我们可以扩展到更长，到 128K。所以你可以自然地看到它如何扩展到甚至一小时的生成。我们需要数据配方和算法来在如此长的上下文中连贯工作，但技术类似于文本长上下文建模。关键区别在于它做流匹配自回归，而不是文本 token 预测。

Yeah, there will be a technical report. I think the summary is that some considerations in this paper were because we started with the Whisper encoder as the starting point. Now we have in-house encoders, like the real-time model we released in January, which also had a technical report. That model has a dual-stream architecture—an interesting architecture, you should check it out. It has a causal encoder, and I don't think there's any strong multilingual causal encoder out in the community, so we thought it's a good contribution. We trained it from scratch. Our full stack is now mature enough to train super strong encoders, and considerations like padding are a function of the Whisper encoder. Now that we train encoders in-house, the design considerations are different. For text-to-speech, it leans on the original autoregressive decoder backbone. The considerations are almost identical. The model processes audio at 12.5 Hz, so 1 second maps to 12.5 tokens. 1 minute is about 720 tokens. You can get up to 10 minutes in an 8K context window, and half an hour in a 30K context window. 32K context is something we're very comfortable training on. We can extend it much longer, to 128K. So you can naturally see how it extends to even hour-long generations. We need the data recipe and algorithm to work coherently through such long contexts, but the techniques are similar to text long-context modeling. The key difference is that it does flow matching autoregressively instead of text token prediction.

好的，我想语音问题差不多了。但我有一个关于 Mistral Small 的大问题。

Okay, I think that was most of the voice questions. But I have a big question on Mistral Small.

Mistral Small。来吧。

Mistral Small. Let's go.

那么什么是 Small？我们如何定义 Small？这是什么？我记得 Mistral 7B 还能在我的笔记本上运行，现在已经装不下了。

So what is Small? How do we define Small? What is this? I remember the days of Mistral 7B on my laptop. It's not fitting on my laptop anymore.

我可以在大笔记本上运行，但这是另一个问题。如果我告诉你我的运动的能量相当于足够的活跃小马尾巴，这是真的。我们确实让你以同样的名字赢了。但我们可以叫它 Medium，但我觉得它只是一个日语词。我会用这个来麻烦你。我想。但这是我们的一个模型，Mistral of Experts。它是一个结合不同模型的模型。以前，我们有一个通用的 Mistral 模型，指令跟随用另一个单独的 Destral 模型。我们还有另一个用于推理的模型 Magistral。所以这些是 Mistral 不同团队构建的独立产物。现在我们基本上在合并所有这些。甚至 Pextral，我们的第一个视觉模型，也是一个单独的模型。我们内部的做法是，一个团队专注于一个能力，构建一个模型。然后一旦成熟，我们决定将其合并到主模型中。所以这是第一次我们基本上把所有东西合并成一个。当时还有一些东西我们没有时间合并，比如更多能力，比如函数调用——在 Mistral Small 正式版中会好得多。但这是我们的最新模型，也是我们正在开发的。当然，还有更大的版本。

I could run it on the big laptop, but it's a different question. If I tell you the energy equivalent of my sport is enough active pony tails, which is true. We did let you win that in the same name. But yeah, we could have called it medium, but I think it's just a Japanese word. I'll trouble you with this. I suppose. But yeah, it's a model we released, Mistral of Experts. It's a model that combines different models. Before, what we were doing is that we had one general model for Mistral, doing instruction following with a separate model that was Destral. We had another model for reasoning, Magistral. So these were separate artifacts built by different teams at Mistral. And now what we're doing is basically merging all of this. It was even Pextral, the first vision model we had, was like a separate model. The way we do things internally is that we have one team focus on one capability, build one model. Then once mature enough, we decide to merge this into the main picture. So here it was the first time we basically merged all of this into one. There are some other things we didn't have time to merge at the time, for instance, more capabilities like function calling—it's going to be much better in Mistral Small proper one. But yeah, so it's our latest model and the one we're working on. Of course, there are larger versions of this.

是的，关键点：它非常稀疏，6B 活跃参数，所以服务效率很高。256K 上下文。我觉得有趣的是这种在不同团队开发独立能力然后合并的总体思路。这会走向何方？我们看到这次合并了五个东西。接下来五个团队是什么？

And yeah, key things: it's very sparse, 6B active, so pretty efficient to serve. 256K context. I think what's interesting is this general theory of developing individual capabilities in different teams and then merging them. Where is this going to end up? Like we've seen five things put together in this. What are the next five teams?

我认为实际上 OpenAI 已经有点偏离了最初的全能模型愿景。这是他们当初推销的，对吧？所有模态。但我觉得你们可能会做到。

I think actually OpenAI has kind of gone away from the original vision of the omni model. This is what they were selling, right? All modalities. But I feel like you might do it.

我的意思是，我认为有些模态并不完全明显。例如，对于音频转录，使用这么大的模型没有意义。如果你只是想转录文本，那会非常低效。你可能只需要构建一个 1B 或 3B 的模型。性能基本相同，而且便宜得多。所以这就是为什么我们想要一个单独的模型来做这件事。我认为问题是：如果你通过语音与模型对话并问非常复杂的问题，你如何做到这一点？另一方面，对于级联系统，你是否想在堆栈中放置一个做单一关键任务的模型？不是这个问题的竞争者。我认为，如果你朝那个方向做，那当然是可能的。

I mean, I think there are some modalities where it's not completely obvious. For instance, for audio transcription, it makes no sense to use a model this large. If you just want to transcribe text, it would be very inefficient. You probably just want to build a 1B or 3B model. Performance would be essentially the same, and it would be incredibly cheaper. So that's why we want a separate model that just does this. I think the question is: if you are talking to your model by speech and asking very complex questions, how do you do this? On the other hand, for cascaded systems, do you want to put a model that does one key on the stack? Not a competitor of this question. I think on the way, if you do that thing in that direction, that's the right possible, of course.

但对我们来说，下一步想整合到模型中的能力是更多的编程、更多的推理。我认为还有一些人们不太谈论但对不同行业客户很重要的能力，比如法律、金融、计算机辅助设计。这些是现成模型没有优先考虑的事情，因为没有相关基准，但让模型在编程上变得更好并不难——你只需要做工作：获取一些数据，处理一下。所以是的。

But yeah, I think for us the next step abilities we want to integrate into these models are more coding, more reasoning. I think more capabilities that people don't talk much about but are important for our customers in different industries, for instance, things like legal, finance, computer-aided design. These are things that models out of the box don't prioritize because there is no benchmark on that, but it's not hard to make a better model for coding—you just have to do the work: source some data, process it. So yeah.

我们还有其他东西要整合进来。

We have other things we merge into this.

我认为对于语音来说，过去一年左右的关键点，比如 VALL-E 和 Google Imagine 这些，是把语音和视频结合起来，对吧？人们不理解空间音频，因为大多数 TTS 就像‘我在完美录音室质量的麦克风前说话’，但当你有了视频，声音会移动。

I think for voice, the key thing over the last year or so with VALL-E and Google Imagine and all these things is joining voice with video, right? People don't understand spatial audio because most TTS is just like 'I'm speaking to a microphone in perfect studio quality,' but when you have video, the voice moves around.

确实如此。考虑因素也有些不同，因为那是一个独立的成品，你拿到整个东西然后消费它，但在对话场景中，你需要极低的延迟。流式传输会是主要考虑之一。

That's true. The consideration is also a little different in the sense that there it's like a standalone artifact where you get the whole thing and consume it, but in a conversational setting, you need extreme low latency. Streaming would be one of the primary considerations.

是的。你可以只靠这个就建立一家大公司，所以不需要做语音那部分。但我只是说在融合模态这个主题上，我觉得‘哇，我之前没想到……’直到去年年中，每个人都在做这些流水线：‘好吧，我们把一个 TTS 模型和一个语音东西、一个唇形同步东西拼在一起。’不。

Yeah. You can build a giant company just doing that. So you don't need to do the voice stuff. But I was just saying on the theme of merging modalities, that is something I'm like 'wow, I didn't...' Everyone up until mid last year was just doing these pipelines of 'Okay, we'll stitch a TTS model with a voice thing and a lip-sync thing and what have you.' No.

就是一个巨大的模型。

Just one giant model.

是的。

Yeah.

我有一个两部分的问题。第一，它仍然是开放的。看起来开源仍然是你们工作的核心。我必须提一下你们的论文。2024 年 1 月，这是唯一的……当他们推出专家模型时，就像关于如何做好 MoE 的基础研究。论文出来了。对任何人来说都是很好的论文，但这只是题外话。

I have a two-part question. One is, it's still open. It seems like open source is still very core to what you guys do. And I just have to plug your paper. Jan 2024, this is the only... When they stroll out experts like very fundamental research on how to do good MoEs. Paper comes out. Very good paper for anyone, but that's just a side tangent.

不，这个东西引起了……我们带回了……我的意思是，8x22 就像开源界的核弹。

No, this thing caused... We bring back... I mean, 8x22 was like the nuclear bomb for open source.

我觉得它需要 7B 更多。所以，7B 更多？

Like, I think it takes 7B more. So, 7B more?

是的，是的，是的。但这对 7B 来说是一个很大的提升。

Yeah, yeah, yeah. But, this is a big up for 7B.

是的，我不记得这个。我记得我觉得不是一月，对吧？好像是 NeurIPS。

Yeah, I don't remember this. I remember I don't think it was January, right? It was like NeurIPS.

它是在 NeurIPS 期间发布的。然后 NeurIPS 上的每个人都在……2023 年 12 月，但我觉得模型也更新了。我想可能更早一点。

It dropped during NeurIPS. And then everyone in NeurIPS was there... December 2023, but I think yeah, the model was updated as well. I think it was a bit earlier.

是的，可能只是一个小更新。

Yeah, it's just a little update probably.

是的，我知道，但你有话要说。

Yeah, I know, but you have a point to make.

不，我的意思是，你得查一下。但我想更广泛地听听你们对开源的想法。当你之前问到下一步是什么，其他团队在做什么时，你们推出了 Le Stral。这让我很惊讶。我当时想，这不符合我对 Mistral 的认知。

No, I mean, you got to check that. But then I just want to hear more broadly on open source for you guys. And when you had asked earlier about what's next, what are the other side teams working on, you put out Le Stral. This one was a surprise. I was like, I don't... This doesn't fit my mental model of Mistral.

是的，我的意思是，首先对于开源整体来说，我认为这确实与公司的历程有关。我们一开始就拥抱开源，甚至在此之前。在此之前，当我在 Meta 时，我们发布了 Llama。我认为很值得看到的是，在此之前，大多数研究人员，比如大学，是不可能研究 LLM 的。外面没有 LLM。如果你看看 Llama 开源后开发的许多技术，比如所有这些后训练方法，甚至 DPO 类型的性能优化。所有这些都是由能够访问这个模型的人完成的，没有这个模型就不可能做到。所以，这确实有助于更快地进步。因此，我们真的想为这个开源生态系统做出贡献。我认为这是一个强烈的信号，也很有影响力。我认为开源社区的所有这些论文确实在帮助整个科学界更快地发展。所以我们想为这个生态系统做贡献。这就是为什么我们发布非常详细的技术报告，关于 Mistral 和我们模型的第一个版本。我发布了很多原因，有效的方法，无效的方法等等。我认为这很有帮助。对于音频模型，我们也分享了很多细节，很多关于实时模型的细节。所以我们真的想继续这样做。基本上，我们属于这个分享科学的社区。我认为我们真的不想生活在一个最聪明、最好的模型只被关在门后，只被某个公司访问，而那个公司有权决定谁能使用它们的世界里。我认为那是一个可怕的未来，我们不想生活在其中。我们真的希望这个模型能被任何人访问。我们希望智能能被任何可以使用它的人使用和访问。所以，是的。这就是为什么我们推动这个使命，所以它是一个模型……是的，就是这样。我们开始它，是的，所以它是开源的，不是第一个模型，也不是最好的。对于 Mistral，我认为也是朝着这个方向的一步。所以，它有点不同于我们通常发布的东西，但我们内部有一个小团队在研究它们，形式化证明，形式化数学。所以，我认为这是我们普遍关心的一个主题。我们也在研究推理。我认为我们开始得太早了，在 LLM 之前，没有 LLM 做推理非常困难，尤其是当你处理形式化系统时，因为你可以获得的数据量微乎其微。写形式化证明的人是一个非常小的社区。但我们喜欢它的原因是，我认为如果你看看人们在推理方面做的事情，通常你可以使用的问题是可以验证输出的问题。例如，所有 AIME 问题，解决方案是 1 到 1000 之间的一个数字，所以你可以通过与参考比较来验证。或者是一个表达式，你可以将模型生成的输出表达式与参考进行比较，但许多数学问题和大多数推理问题没有简单的方法来验证解决方案。

Yeah, I mean, first for open source in general, I think it's really something which maybe looks to the journey of the company. I think we started it for once this we saw we have an open source thing with us since the beginning and even before this. So, before this, when I was at Meta, we released Llama. And I think what was really nice to see that before this for most researchers, like universities, it wasn't possible to work on LLMs. There was no LLM outside. And if you look at many of the techniques that were developed after for instance Llama was open source, like all these post-training approaches, even DPO type performance optimization. All of these were done by people that had access to this model or it would have been impossible to do without this model. So, it's really making sense to move faster. So, we really want to contribute to this open source ecosystem. I think that's a deep signal and also like very lot of impact. All these papers that are I think in the open source community are really helping the science community as a whole to move faster. So, we want to contribute to this ecosystem. That's why we are releasing very detailed technical reports on Mistral and our first season in the model. And I mean I'm publishing a lot of reasons, things that worked, things that did not work as well, etc. I think it's helpful. On the yeah, so for the audio model also to share a lot of details, share a lot of them for the real-time model. On the yeah, so we really want to continue this. Basically, we belong to this community of people who share science. I think we really don't want to be living in a world where the smartest models, the best models are only behind closed doors, only accessible to a sure company that we have the power to decide who can use them. And I think it's kind of a scary future we don't want to live in. We really want this model to be accessible to anyone. We want intelligence to be used and accessible by anyone that can use it. So, yeah. So, that's why we are pushing for this mission and so it's a model on the yeah, so that's so yeah. We're starting it yes, so it's open source not the first model and so not the best. On the yeah, Mistral I think is also one step into this direction. So, it's kind of yeah, a bit different than what we are usually releasing, but we have a small team internally working on them, formal proving, formal math. So, I think a subject we care about in general. And we're working on reasoning. I think we started too early before LLMs doing reasoning without LLMs is very hard, especially when you work with formal systems because the amount of data you have is negligible. It's a very small community of people writing like formal proofs. But the reason why we like it is because I think there is, if you look at what people are doing with reasoning, is that I mean the problems that you can use are usually going to be problems where you can verify the output. So, for instance, all these AIME problem where the solution is a number between 1 and like 1,000, so you can verify compare this with a reference. Or it's an expression you can actually compare the output expression generated by your model with a reference, but there are many most of the math problem and most of the reasoning problem there is no like way to easily verify the solution.

如果问题是证明 F 是连续的，你就不能拿它跟一个参考答案比较，对吧？如果是证明某个命题成立或证明某些性质，你没法合成地验证你的证明是否正确。所以很难应用，因为这里没有可验证的奖励。你可以提供一个裁判来检查你的证明，但这很难，而且可能会发生奖励破解，所以很困难。你可以提供一个参考证明，但同一个命题有很多种证明方式。所以，如果模型因为证明不同而得到负奖励，但可能它仍然是一个合法的证明，只是不同而已。这样就行不通。而 Lean 和形式化证明的好处是，你完全不用担心这个。只要它们在 Lean 中编译通过，功能上就是等价的，对吧？

If the question is to show that F is continuous, you cannot compare it with a reference, right? If it's a proof that this is true or prove these properties, there is no way to synthetically verify the correctness of your proof. So, it's hard to apply because there is no verifiable reward here. What you could provide is a judge that will look at your proof, but it's very hard and you could lose some reward hacking happening there, so it's difficult. You could provide a reference proof, but there are many ways to prove the same thing. So, if the model gives a negative reward because it's a different proof, but maybe it's still a legit proof just different. So, it's not going to work well. What's nice with Lean and formal proving is that you don't have to worry about this whatsoever. They're all functional as long as they compile in Lean and functionally the same, right?

完全正确。就像程序一样：如果编译通过，就是正确的。

Exactly. It's like a program: if it compiles, it's correct.

是的，这很容易，而且你可以应用到任何类型……

Yeah, it's very easy and you can apply this on any kind...

只是规模太小了；没有人会真的去做。

It's just way too small; no human will actually go and do it.

是的，正是如此。唯一能做这件事的人是一个很小的群体，他们正在攻读相关博士学位。所以规模非常小，这有点可惜，因为它实际上非常有用，不仅在数学领域，在软件验证中也是如此。例如，如今的软件验证市场非常小。很少有行业会做这个并需要它。通常只有像制造飞机、机器人这样的公司，他们绝对要确保安全——生命攸关——但很少有人会正式验证他们软件的正确性。我认为一个原因就是它太难了。你是在想 TLA+吗？那是有些人用来做软件验证的语言。

Yeah, that's exactly it. The only people that can do it are a very small community of people doing a PhD on that. So, it's super small and it's kind of sad because it's actually very useful, not just in math but also in software verification. For instance, software verification today is a really tiny market. Very, very few industries work on this and will need that. It's usually going to be companies like building airplanes, robotics, things where they absolutely want to be sure—life depends on this—but it's very rare that people formally verify the correctness of their software. I think one reason is simply that it's super hard to do. Are you thinking of TLA+? It's the language that some people use for software verification.

不，我知道它被用作参考，但 Coq……是的，我认为人们不更多使用它、这个行业没有它本应那么大的原因就是它非常难。但现在有了编码智能体，情况会非常不同。我们会看到更多这样的应用。所以，我认为是的，未来这个行业会随着那些理论模型变得大得多。所以，这里也有一点预见。我想在这方面工作，因为它就像在证明一个定理，而我往里面放了一个函数。所以，实际上是同样的工具。

No, I know it's used as a reference, but Coq... yeah, the reason I think why people don't use it more and why this industry is not as big as it could be is because it is very hard. But now with coding agents that are there, it's going to be very different. We are going to see much more of this. So, I think yes, the industry is going to be much larger in the future with those theoretical models. So, here also anticipating this a little bit. Wanted to work on that because it's proving like a master and I'm putting like a function in it. So, it's actually the same tools.

是的，是的。我的一个理论是，因为证明耗时很长，它实际上只是长程推理、连贯性和规划的一个代理。很多人会说，‘好吧，这是给喜欢数学的人用的。它是一种小众的数学语言。谁在乎呢？’但实际上，如果你把它作为后训练和推理的数据混合的一部分，它可能会在其他所有领域都带来提升。我认为这一点尚未被充分探索；还没有人真正发表一篇明确的论文来说明这是如何泛化的。

Yeah, yeah. One of my theories is that, because the proofs take so long, it's actually just a proxy for long horizon reasoning and coherence and planning maybe. A lot of people will say, 'Okay, it's for people who like math. It's a niche math language. Who cares?' But actually, if you use this as part of your data mix for post-training and reasoning, it might spike everywhere else. And I think that's under-explored; no one's really put out a definitive paper on how this generalizes.

是的，绝对如此。我认为这已经是我们正在看到的情况了。例如，如果你在数学上做一些推理，然后在 Coq 中做，是的，我在早期阶段就看到了这一点。所以，肯定存在某种迁移，某种涌现现象。而且我认为有趣的是，不仅仅是这个主题；从长远来看，这与编码智能体有很多联系。因为有时模型会看到一个需要证明的定理，它非常复杂。然后它可以主动说，‘我要证明这三个引理。我要提出三个引理，然后并行证明每个引理，用子智能体并行证明三个，同时我还要证明主定理，就像你在家里看到这三个引理一样。’所以，你可以采用这种子智能体方法，这非常有趣。即使你没能证明其中一个引理，你实际上可能可以用 60 来证明正常的引理二。所以，这里你会得到部分奖励。这比整个任务只得到零奖励要稀疏度低一些。所以，这非常有趣。我想我们可以在这里停一下。

Yeah, absolutely. And I think that's kind of what we're seeing already. For instance, if you do some reasoning on math and then do it in Coq, yeah, I just saw that through in the early stage. So, there definitely is some transfer, some sort of emergence that happens. And I think what's also interesting, it's not just the topic; in the long run, there is a lot of connection with this on encoding agents. Because sometimes a model can see a theorem that it has to prove, it's very complex. Then it can take the initiative to say, 'I'm going to prove these three lemmas. I'm going to suggest three lemmas and I'm going to in parallel prove each lemma, so three of them in parallel with sub-agents, but I'm also going to prove the main theorem as you see in the three lemmas at home.' So, you can do this sub-agent approach, which is pretty interesting. You can even if you fail to prove one of the lemmas, you can actually maybe use 60 to prove the normal lemma two. So, you get some partial reward here. So, it's a bit less sparse than if you just get a zero reward for the entire thing. So, it's pretty interesting. I think we can actually stop here.

是的，对于专门的模型来说，这也是一个有趣的案例，对吧？比如你展示的成本问题就很有意思。是的，在分数上类似，你花了 30、70、150、300 美元。

Yeah, it's also an interesting case just for specialized models in general, right? Like the cost thing you show is pretty interesting. Like yeah, similar score-wise, you're 30, 70, 150, 300 bucks.

和那个相比。

Compared to something like that.

我认为成本有点不公平，对吧？因为这是推理成本。那个是在此基础上加了他们的利润。但你知道，我们不知道其他情况。

I think cost is a bit unfair, right? Because this one is at inference cost. This one is there with their margins on top of it. But you know, we don't know anything else.

我确实想进一步追问这个。所以，不是关于成本，而是你提到，嗯，这是一个拥有可验证长上下文推理的好方法。还有哪些前沿领域，我知道你们内部肯定在探索？有很多人在推动预训练、Scaling 强化学习，把算力投入到训练预算的一半以上都用于强化学习。你们在这个领域看到的研究前沿是什么？

I did want to actually push on that more. So, not on cost, but you mentioned about okay, it's a great way to have verifiable long context reasoning. What are other frontiers that you know, I'm sure you guys are working on internally? There's a lot of push of people pushing back on pre-training, scaling RL, pushing compute towards having more than half of your training budget all on RL. Where are you guys seeing the frontier of research in that?

你是指强化学习吗？

You mean with RL?

就是在基础模型训练方面，接下来……你们做的一件事实际上是从头开始做基础研究，对吧？所以，你们可能对未来的预测有很好的视角。

Just in foundation model training in the next... One thing that you guys do actually is you do fundamental research from the ground up, right? So, you probably have a really good look at where you can forecast this out.

是的，但我认为对我们来说，我们仍然在预训练方面做了很多工作。我们离预训练的任何饱和状态都非常非常远。我认为任何形式的预训练都会比我们之前做的一切都迈出一大步。所以，我们对此非常兴奋。另一方面，我认为现在我们需要越来越多地考虑能够支持这些非常长轨迹的算法。我认为，例如，当使用 GPU 时，它实际上不适用于这种策略，这最初还可以，因为你解决的是可以在几千个 token 内解决的数学问题。所以，模型可以很快地生成它们。所以，当你进行更新时，模型永远不会偏离太远。仍然不会太远，但现在当你转向这类需要数小时、比如 6 小时才能获得奖励的问题时，你的模型就需要配合。所以，你必须带来支持这一点的新的基础设施，因为你的算法……所以现在我们内部正在努力构建一些基础设施，以预见我们在未来 6 个月内将面临的这种极低场景。

Yeah, but I think for us, we are still working a lot on the pre-training side. We are very, very far from any sort of saturation on the pre-training. I think any sort of pre-training would be like a big step up compared to everything we have done before. So, we are pretty excited about this. And I think on the other side, now we have more and more to think about this algorithm that will actually support these very long trajectories. I think when it was for instance GPU for instance, it doesn't really work with this any bit of policy, which was okay initially because you are solving math problems that can be solved in like a few thousand tokens. So, the model can actually generate them pretty quickly. So, when you are doing your updates, the model is never too far off. Still not too far off, but now when you are moving towards this kind of problems where something takes hours, like 6 hours to get to reward, then your model is cooperative. Please see so you have to actually bring me accuracy that new infrastructure that supports this because then your algorithm... so now everything we're doing internally we're trying to build some infra that will actually anticipate this what we have in like a 6-month window which is this extremely low scenarios under at pace that they can.

我想当我们创办 Mistral 时，我和 Timothy 的一部分想法是重建一个非常棒的环境，让人们可以带着充足的资源做他们喜欢的研究。所以那很好。我认为 ChatGPT 出现后情况发生了很大变化。在那之后变得非常困难。现在也是一样，但没错，那很好，我想我们也想重现之前的一些文化。

I think when we started Mistral, part of me and maybe also Timothy wanted to recreate this very nice environment where people are there so they can do research they like with a lot of resources. So it was nice. I think things changed a lot when ChatGPT came out. After that, it was very difficult. This time is also the same again, but yeah, it was nice, and I think we also want to recreate part of this culture we had before.

接近尾声，我们显然认为你们在做非常了不起的工作。你们对开源和语音有着令人印象深刻的愿景。你们在招聘什么职位？接下来你们希望什么样的人加入公司？

Coming to the end of it, we're just obviously I think you guys are doing incredible work. You have a very impressive vision for open source and for voice. What are you hiring for? What's the next thing you're looking for that you're trying to join the company?

是的，我们在科学团队招聘很多人。我们在所有办公室都在招人。我们的总部在法国巴黎。我们在伦敦有一个小团队，在帕洛阿尔托也有一个团队。最近我们在波兰华沙开设了一些办公室，在苏黎世也有一间，在纽约也有业务，很快在旧金山也会有一个。所以我们遍布各地，也远程招聘。我们正在扩大团队，试图招聘非常优秀的人。我想我们想保持团队仍然相当小，因为我们认为小团队非常高效且敏捷。所以是的。

Yeah, so we are hiring a lot of people in our science team. We are hiring in all our offices. We have our HQ in France, in Paris. We have a small team in London, a team in Palo Alto as well. Recently we opened some offices in Warsaw in Poland, also one in Zurich, and we also have some presence in New York as well, and soon one in San Francisco. So we are a bit everywhere, also hiring people remotely. We are growing the team, trying to hire very strong people. I think we want to stay—the team is still a fairly small team, but I think we want to keep it that way because we find it quite efficient to have a small team and be very agile. So yeah.

好的，让我们聚焦科学这一点。我们实际上是科学的坚定信仰者。我们启动了新的科学部门，专门专注于 AI for Science。你认为哪些领域最有前景？

Okay, let's focus on science for the point. We actually are strong believers in science. We started our new science part that focuses specifically on AI for science. What areas do you think are the most promising?

你现在最关注的是我们已经开始做的事情，我们可能在几个月内能分享更多细节。我们正在探索 AI for Science，有很多领域我们认为如果应用 AI 会取得极其有前景的成果。有很多低垂的果实。你只需要找到那些 AI 尚未被应用的领域。这通常很难，因为在这些领域工作的人不一定了解这些模型的能力。他们不知道 AI 能做得有多好。你只需要将他们与完全匹配的研究人员配对，知道该做什么。但这种配对我们正在自然地与客户一起进行。所以我们有一些密切合作的公司。例如，我们的合作伙伴之一 ISM Electrónica。我们与他们一起做研究，他们有大量非常有趣的问题。物理学、科学、材料科学中的问题，他们基本上是唯一在研究这些问题的人，因为他们正在做别人没有做的事情。所以有很多领域 AI 实际上可以带来革命性的变化。你只需要思考并熟悉 AI 能做什么以及如何应用它。所以是的，这是我们与合作伙伴和客户一起建模的事情。所以 AI for Science 非常有趣。

What you are most critical about right now on something we have started doing and we'll probably be able to share more about this in a couple of months is that we are exploring AI for science, and there are a lot of areas where we think that you could get some extremely promising results if you apply AI in these domains. There are a lot of low-hanging fruits. You just have to find these domains where AI has not yet been applied. And it's usually hard to do because the people working in these domains don't necessarily know the capability of these models. They don't know how well AI would work. You just have to pair them with exactly the researchers matching exactly what to do. But this matching we are kind of doing it naturally with our customers. So we have some companies we work very closely with. For instance, ISM Electrónica, one of our partners. We are doing some research with them, and they have tons of extremely interesting problems. Problems in physics, in science, material science that they are essentially the only ones to work on because they are doing something no one else is doing. So there are many domains where AI can actually revolutionize things. You just have to think about it and be familiar with what AI can do well and how to apply it. So yeah, it's something we are modeling with our partners, with our customers. So AI for science is very interesting.

是的。好的，那么对于部署工程师来说，什么造就一个好的部署工程师？他们需要什么？人们在哪里失败？

Yeah. Okay, and then for deployed, what makes a good deployed engineer? What do they need? Where do people fail?

我认为通常你需要非常熟悉技术的人。不一定需要很多研究专长，但要非常擅长使用这些模型，知道如何进行微调，知道如何启动强化学习流水线。这并不容易。这是大多数客户、大多数公司无法自己完成的事情。所以在这里，我认为我们需要那些喜欢解决问题、对解决复杂具体问题感到兴奋的人。这基本上是应用科学。所以我认为这与研究所需的技能没有太大区别，因为本质上你是在试图找到客户尚未解决的问题的解决方案。有时很容易，有时你需要做很多工作。你必须创建合成数据，找到一些边缘情况。所以这取决于问题。但你还需要一点耐心。要有创造力。我认为实际上技能组合非常相似。

I think it's usually you need people that are very familiar with the tech. And not necessarily with a lot of research expertise, but that are actually pretty good at using this model, that can actually, you know, know how to do fine-tuning, know how to start some RL pipeline. And it's not easy. It's something that most customers, the majority of companies would not be able to do on their own. So here I think we need people that are, you know, that like to solve problems, that are excited about actually solving some complex, very concrete problem. It's applied science basically. And yeah, so I think it's not too different from the skills you need in research, because essentially you are trying to find solutions to problems that customers have not yet solved. Sometimes it's easy, sometimes you have to do the work. You have to create synthetic data, find some edge cases. So it can be, yeah, depends on the problem. But you need also a bit of patience. Be creative. I think very similar skill sets actually.

他们所做工作的多样性总是让我惊讶。从他们在行业中遇到的各种事情都有。我觉得这非常有趣。

The diversity of the work they do always surprises me. It goes all the way from the kind of stuff they encounter in industries. It's just very interesting, I think.

有什么有趣的成功轶事吗？

Any fun success anecdotes?

我的意思是，是的，比如在边缘设备上训练一个小模型，只做一件特定的事情。我们也在用一些特定语言训练一些非常大的模型。让模型非常擅长某些工具，比如计算机辅助设计之类的东西。

I mean, yeah, it can be like actually training this small model on edge that just does one specific thing. And we are training some very large model with some specific languages as well. Making models really good at some tools like for instance computer-aided design, these kinds of things.

那是否也与视觉结合？比如芯片的缺陷检测或工厂中的识别。多样性可以是任何可以部署这些基础模型的地方。所以，是的，就是让它在特定环境中工作。基本上，就是不惜一切代价让它在特定工作流程中增加价值。

Is that in pairing with vision as well? Like defect detection for chips or in factories identifying things. The diversity could be anything where you can deploy these foundation models. So, yeah. The work to make it work in that specific setting. Basically, whatever it takes to make it add value in that specific workflow.

是的。而且它贯穿整个技术栈，对吧？比如甚至只是拉起网站。你必须选择……

Yeah. And it goes kind of across the stack, right? Like even just pulling up the website. Like you have to choose...

它太广泛了。算力太广泛了。

It's so broad. Compute is so broad.

我们甚至还没谈到 Mistral 5。我们有一个实时编码 CLI 工具。你们实际上是最早做的一件事，我认为是 Mistral 智能体。智能体构建器，你可以通过 API 提供它等等。我猜前向部署人员会帮助构建它之类的。

We didn't even touch on Mistral 5. We have a live coding CLI tool. One thing you guys were actually like I think the first to do was Mistral agents. The agent builder, you can serve it via API and all that. And I'm guessing forward deploy people will help build that out and stuff.

是的。这也是为什么我们做很多事情，但我认为这也是价值主张的一部分，有时客户总是非常谨慎地对待他们的数据。他们不想信任太多合作伙伴。信任一个合作伙伴处理代码，把数据交给另一个第三方处理音频等等。所以他们喜欢这样。但在这里，他们真正喜欢我们方法的是我们可以帮助他们处理任何事情。所以他们不必把所有东西都发送到那么多云上。所以是的。

Yeah. It's also why we are doing many things, but I think that's also part of the value proposition that sometimes customers are always very extremely careful about their data. And they don't want to trust so many partners. Trusting one partner for code, giving your data to another third party for audios and so on. So they like this. But here what they really like with our approach is that we can help them on anything. So they don't have to send all that out to so many clouds. So yeah.

我认为部署工程师的数量可能比研究科学家多几个数量级。他们不需要你的全部经验，但对客户来说仍然非常有价值。

I think that there can be many orders of magnitude more FDEs than research scientists. And like they don't need your full experience, but they're still super valuable to customers.

我的意思是，实际上这两个团队仍然紧密交织。首先，他们经常使用相同的工具、相同的数据管道等等。

I mean in practice these two teams are still quite intertwined. I mean very often first of all they are using the same tools, the same data pipeline and everything.

对同一个团队来说，从解决方案团队获得反馈非常有帮助，因为他们可以说：‘看，这些客户正试图这样做，但行不通。我们能不能在下一个版本中展示一下？’

On the it's it's very helpful for the same team to get the feedback from the solution team because they can say, "Look at these customers are trying to do this. This is not working. Can we maybe show in the next version?"

是的。这基本上就是你的真实世界评估。

Yeah. So this is basically your real-world eval.

嗯。

Yeah.

这是真实世界的评估。不像那种情况，比如你只在实验室工作，模型发布后，却没有人为客户做这种检查工作。你根本不知道模型在处理这种边缘案例时表现如何。比如，即使在你之前的工作中也是如此，对吧？所以，那些非常学术化的公开基准测试和真实案例之间存在着巨大的差距。

It's real-world eval. It's not something For instance, if you are just working in the lab, it's just shipped model, but you don't do this work of people examining for customers. You have no idea whether your model is good at this edge case. Like for instance, in the you even in your before this, right? So there is a very get big gap between the public benchmarks that are very like academic on the the real cases.

真实案例非常多样化。在客户的具体场景中，你可以进行微调，比如先评估，创建一个可靠的评估基准，然后根据他们的音频类型进行测量。例如，一个用例就是让孩子们说出一个词，然后你需要判断他们说得对不对，这有点像针对孩子的强化学习。所以，应用科学家和工程师会处理各种不同的用例和想法，使其变得更好，然后我们将这些经验融入基础模型本身。这样，模型开箱即用效果就更好了。

The real cases are just very diverse. And in the specific context of customer you can fine-tune and make it like uh first like evaluate create a solid eval benchmark and then measure in the context of their the kind of audios like for instance, one use case is literally just the the word for kids and they have to just say it out. It's a very specific thing. You're just saying one word and then you have to like you you'll grade the kid whether they did it right or it's like RL for kids. But uh so there's very diverse use cases and ideas that they the applied scientists and engineers will go and make it better and then from the learnings we incorporate it into the base model itself. So it's it's just better out of the box.

是的。这是一个很好的闭环系统，你知道，基础模型的评估都只是真实情况的代理。你永远找不到一个完美的评估标准。像那种单个词的转录，并不合理，也不是你想要去拟合的东西。

Yeah. It's a good full circle system, you know, like the the foundation model eval's are all just proxies of what you really You're never going to have one that's It doesn't make sense for there to be a one-word transcription like that. It's it's not something you want to fit on.

完美。那么，大家都应该去看看 Michelle 提供的所有内容，并试试我们在节目笔记中链接的 TTS 模型。非常感谢你的到来。

Perfect. Well, everyone should go check out everything Michelle has to offer and try the TTS model which we'll link in the show notes. But, thank you so much for coming.

谢谢。非常高兴来到这里，各位。

Thanks. It's such a pleasure to be here, guys.