从开发者体验到智能体体验：Modal 的起源故事

我们实际上已经改变了 SDK 团队的方向，从思考开发者体验转向思考智能体体验。我们认为适用于 DX 的同样好处也适用于 AX，为什么你要让一个智能体阅读数百个 Kubernetes 文件并编写甚至没有类型的 YAML，而它只需在装饰器中做几个更改，就能获得这种自我配置的运行时，能够实时看到其更改的效果。

We've actually changed our SDK team to think about agent experience instead of developer experience. And we think that the same benefits that apply for DX also actually apply for AX, which is why would you have an agent read through hundreds of Kubernetes files and write YAML that's not even typed when it can basically make a couple changes in a decorator and it gets this sort of self-provisioning runtime of being able to see its changes live in action.

在进入今天的节目之前，我有一小段话要对听众说。谢谢你们。如果没有你们选择点击并收听我们的内容，我们就无法为您带来您明显想要的 AI 工程、科学和娱乐内容。几乎每天都有赞助商联系我们。但幸运的是，有足够多的订阅者让我们在没有广告的情况下维持这一切，我们希望保持这种状态。但我只想请大家帮一个忙。您能做的最有力、完全免费的事情就是点击订阅按钮。这是我唯一会请求您做的事情，它对我以及每周努力为您带来 Inspace 的团队来说意义重大。如果您订阅了，我保证我们会永不停止努力让节目变得更好。现在，让我们进入正题。

Before we get into today's episode, I just have a small message for listeners. Thank you. We would not be able to bring you the AI engineering, science, and entertainment content that you so clearly want if you didn't choose to also click in and tune into our content. We've been approached by sponsors on an almost daily basis. But fortunately, enough of you actually subscribe to us to keep all this sustainable without ads, and we want to keep it that way. But I just have one favor to ask all of you. The single most powerful, completely free thing you can do is to click that subscribe button. It's the only thing I'll ever ask of you, and it means absolutely everything to me and my team that works so hard to bring the Inspace to you each and every week. If you do it, I promise you, we'll never stop working to make the show even better. Now, let's get into it.

我们请到了 Moto 的 CTO Ashhat，与 Vivu 一起。恭喜你获得 CC。

We're here with Ashhat of Moto, CTO of Moto, together with Vivu. Congrats on your CC.

谢谢。

Thank you.

你昨天的派对太棒了。

Your party yesterday was amazing.

是的。

Yeah.

所有的照片和赠品。

All the photos and all the swag.

我们有一堆艺术装置，看到我们的产品放在基座上，旁边像罗丹的作品，挺有趣的。

We had a bunch of art installations which is kind of fun seeing like our products on pedestals next to like Rodan.

非常好。非常好。当你开始时，它并不是 GPU 推理公司。我的意思是也许你心里有数。带我们回到起源故事。

Very nice. Very nice. When you started it was not the GPU inference company. I mean maybe it was in your mind. Take us back to the origin story.

我实际上是通过一位投资者第一次见到 CEO Eric，那时 Eric 已经在考虑构建一种新的运行时，他思考为什么工作流编排产品这么难用？因为必须在 Kubernetes 上运行它们。Kubernetes 难以管理，不是为突发性和自定义镜像而构建的，而且开发者体验很糟糕。

I actually first met Eric who's the CEO through an investor and back then Eric was already thinking about building a new kind of runtime and he got there thinking through why are workflow orchestration products so hard to use? It's because you have to run them on Kubernetes. Kubernetes is hard to manage. It's not built for burstiness and custom images and has a terrible developer experience.

我插一句，给新听众。我们两年前采访过 Eric，那里有更多关于 Spotify 等事情的故事。我实际上是通过 data council 遇到 Eric 的，因为他做了关于你们那种无服务器容器栈的演讲，那是我第一次觉得“好吧，我需要认真对待模型”的时刻，但仍然很不清楚，我真的需要所有这些仅仅为了我的数据管道吗？

And I'll inject for listeners who are new. We interviewed Eric two years ago and there's a bit more of the story there from Spotify and all those things. And I actually came across Eric through data council because he did that talk on the sort of serverless container stack that you guys did which was like that was my first like okay I need to take models very seriously moment but it was still very unclear like do I actually need all this for just my data pipelines?

是的，我的意思是，最初我们在想，如果我们构建一个更好的运行时。它本身就是一个非常有用的原语。有很多事情可以通过无服务器函数解决，比如你可以做 ETL 工作、任务队列、所有这种突发性处理，事实证明每家公司都有需求。但我们也认为这是一个原语，我们可以在此基础上构建一整套非常有价值的产品。所以数据工程可能是第一个，但那时我们也在考虑推理，更经典的推理，比如计算机视觉和运行 XGBoost 之类的东西，但我们在 ChatGPT 出现前一年就将 GPU 添加到了产品中。我们只是没想到它会变得那么重要。

Yeah, I mean initially what we were thinking about was if we build a better runtime. It's a very useful primitive in itself. There's a lot of things that get solved by serverless functions like you can do ETL stuff, you can do job queues, you can do all this bursty processing which it turns out every company had needs for. But then we also were thinking about this as a primitive that we can build a whole collection of products on which are very virt. So perhaps data engineering would have been the first one but we were thinking about inference back then it was more classical inference like computer vision stuff and running XGBoost and whatnot but we added GPUs to the product a year before ChatGPT came out. We just didn't think it would be that big of a deal.

是的。就像添加 A100。有没有什么早期的关键问题真正激发了你构建它的原因？

Yeah. Just like add A100. Was there any early key problem that really sparked off why you built it?

是的，主要是现有的工具没有一个是为了提供真正出色的开发者体验而构建的，而且我们看到的很多工作负载普遍非常计算密集，需要更多资源，所以你需要频繁地上下突发，而 Kubernetes 是为慢速扩展设计的，更多用于 Web 服务器用例。此外，这些工作负载运行的环境也更加专业化。比如有时需要加速器，有时需要不同类型的镜像，这是我们在许多公司看到的一致情况。那将是下一步。

Yeah, primarily it's just none of the tooling that was out there was built for one a really great developer experience and also there's a general trend of a lot of the workloads that we were seeing were very compute heavy like they need a lot more resources so you need to burst up and down a lot versus like Kubernetes designed for slow scaling and more for web server use cases. And also there's just a lot more specialization in what kinds of environments these workloads run in. Like sometimes they need accelerators, sometimes need different kinds of images and this is just like a consistent thing that we saw across a lot of companies. That would be the next step.

是的。很好。我不知道这在早期故事中占多大比重，但我在 Temporo 时写过一篇关于基础设施软件定义基础设施之类的文章。

Yeah. Be nice. I don't know how much this factored into the early story, but I wrote a post when I was at Temporo about infrastructure software defined infrastructure or something like that.

自我配置。是的。我甚至不记得自己的文章，然后你把我放到了登陆页面上。

The self-provisioning self. Yeah. I couldn't even remember my own post and then you put me on the landing page.

是的，我们真的很喜欢这个术语，所以我们就偷来了，因为你的洞察是，一切都可以放在与代码相邻的装饰器中，对吧？那是原始故事的重要部分，还是只是一个 DX 层？

Yeah, we really like the term and so we stole it because you had the insight that everything can just be in decorators next collocated with the code, right? Was that a big part of the original story or was just like a DX layer?

那非常重要，因为我们真的不想让人们花太多时间编写 YAML，而且似乎你可以真正浓缩你所做工作的表面积，将其放入代码中，这样你就可以像操作其他代码一样操作它，构建更具表现力和动态性的东西。所以是的，那始终是一个非常重要的部分。

That was really important because we really didn't want people to spend so much time writing YAML and it seemed like you could really condense the surface area of what you're doing, put it in code so you can actually operate on it just like you can operate in other code, build stuff that's more expressive and dynamic. So yeah, that was always a very important part.

然后反对意见是，这是一个 DSL，你是闭源的，我被锁定在 modal 中。

Then the push back is this is a DSL, it's you're closed source, I am locked in to modal.

是的，我们从未真正因此受到反对，因为 modal 的好处是你可以带来任何你拥有的代码，当然 DSL 是用于配置你使用的硬件、如何扩展的配置层，但你仍然拥有代码。即使现在我们做推理，这也一直是我们故事的重要部分。

Yeah, we never really got push back for that because the nice thing about modal is you can bring whatever code you have and sure the DSL is the sort of configuration layer for what hardware you're using, how you're scaling things up, but you still own the code. And that's been an important part of our story even as we do inference now.

是的。你认为今天有多少保持不变？比如如果你今天要构建一些东西，DevX 显然非常重要，但我觉得你知道很多已经改变了，只需将其连接到智能体，使用云代码，让 codeex 实现一个工具。有一些非常智能体原生的原语，与我亲自做这件事是不同的，对吧？

Yeah. How much of it do you think still stays the same today? Like if you were to build something today, DevX obviously very important, but I feel like you know a lot of this has kind of been changed with just hook it up to an agent, have Claude Code, have codeex implement a tool. There's very agent native primitives that are kind of different than if I'm doing this myself, right?

我们实际上已经改变了 SDK 团队的方向，从思考开发者体验转向思考智能体体验。我们认为适用于 DX 的同样好处也适用于 AX，为什么你要让一个智能体阅读数百个 Kubernetes 文件并编写甚至没有类型的 YAML，而它只需在装饰器中做几个更改，就能获得这种自我配置的运行时，能够实时看到其更改的效果。是的，从我们交谈过的客户来看，他们实际上发现 modal 用于智能体比在其他基础上操作要快得多。

We've actually changed our SDK team to think about agent experience instead of developer experience. And we think that the same benefits that apply for DX also actually apply for AX, which is why would you have an agent read through hundreds of Kubernetes files and write YAML that's not even typed when it can basically make a couple changes in a decorator and it gets this sort of self-provisioning runtime of being able to see its changes live in action. Yeah, it just seems from the customers we talked to they actually find the model is way faster used for agents versus operating on a different substrate.

因为你要把基础设施需求与运行它的代码放在一起。

Because you collocate the infrastructure requirements to the code that runs it.

嗯，现在的负面论点是没人再看代码了，所以没意义。

Well, the negative thesis now is that nobody's looking at their code anymore. So there's no point.

是的，人们确实不看代码。我们仍然认为很重要的一点是可观测性。你的仪表盘有多好？当然，我们把很多功能推到了 CLI 上，这样智能体可以自行调查，但你仍然需要人类去解读发生了什么并做出判断。我觉得这现在可能比看代码本身更重要。

Yeah. I mean people aren't looking at code. One thing we actually still see is really important is observability. How good is your dashboard? And of course, we push a lot of it to the CLI so the agents can do their own investigation, but you still need humans to go interpret what's going on and make judgment calls. I feel like that's maybe more important now than looking at the code itself.

是的，因为你可以把代码当作黑盒，然后利用它产生的可观测行为，直接提示修改。

Yes, because you can try to treat the code as a black box and then use the observable action that comes out of it and just prompt a change.

我认为需要一点克制才能不专门化，说我想发布一个新原语，然后保持通用。人们问你你是做什么的，你回答我不知道，我们能做这个能做那个。我很好奇，如果问你 Modal 到底是做什么的，哪怕在高层次上？你们做很多事：沙盒、GPU，什么都有。你怎么回答？

I think it takes a bit of restraint to not specialize, to say I want to ship a new primitive and then just be general purpose. People ask you what are you for, you're like I don't know, we can do this, we can do that. I'd be curious, if we were to ask you like what is Modal for even at a high level? There's a lot you guys do: sandboxes, GPUs, everything. How do you answer?

Modal 是一个从头为 AI 应用构建的云平台。目前它涵盖推理、训练、批处理和沙盒工作负载，但我们还在构建更多功能。

Modal is a cloud platform built from scratch for AI applications. Right now it covers inference, training, batch processing, and sandbox workloads, but we're building a lot more.

我注意到你没提 Web 服务器，所以始终在线的大规模 Kubernetes 这类东西仍然有作用。

I noticed you didn't say web server, so there is still a role for the always-on large scale Kubernetes type of things.

当然。我们不想和 Render 这类公司竞争。我们的差异化在于需要专用算力、需要频繁伸缩的工作负载。它们的形态不同。

Absolutely. We're not trying to compete with the Renders of the world. Our differentiator is workloads that need specialized compute, need to scale up and down a lot. They're shaped differently.

我觉得你们很多是和初创公司一起构建的，对吧？他们创新很多。即使在你们最新的博客文章里，提到的客户——Cognition、Decagon、Ramp——都在和你们一起创新。这不是 AWS 直接在做的事。

I think you're building a lot of it alongside the startups, right? They're innovating quite a bit. Even in your latest blog post, the customers you mentioned — Cognition, Decagon, Ramp — they're innovating with you. That's not something AWS is doing directly.

完全正确。我们是个小团队，行动很快。我们的工程师和客户一起工作，解决问题。

Absolutely. We're a small team, we can move really fast. Our engineers work with customers and figure things out.

我在 Cognition 的第一周，走进去看到有人穿着 Modal 的 T 恤。我说，你在这儿干嘛？他们说，‘我嵌入在 Cognition 里。’

My first week at Cognition, I walked in and there was someone wearing a Modal shirt. I was like, what are you doing here? They said, 'I'm embedded inside of Cognition.'

那是 Payton。我们派他过去，因为沟通延迟太高了。

That was Payton. We sent him over because the latency of communication was too high otherwise.

分布式节点。你得放一个在共置位置。

Distributed node. You have to place one in collocate.

我有直接的个人经历。3 年前我做了 Small Developer，受 CloudOne 启发。我想你当时给我做了入职。我需要一些突发算力，试用了 Modal，体验相当不错。显然我出现在了董事会会议的分析里。

I had a direct personal experience. I worked on Small Developer 3 years ago, inspired by CloudOne. I think you onboarded me at some point. I needed some bursty compute, tried using Modal, and it was a pretty pleasant experience. Apparently I showed up in the board meeting analytics.

你在 Hacker News 上火了，我们流量暴增。你用 Small Developer 的方式是用 Modal 函数来运行东西。那是个好用例。

You blew up on Hacker News and we got a big traffic spike. The way you used Small Developer was with Modal functions for running stuff. That was a good use case.

对我来说那就是原型 Cognition，对吧？

To me that was proto-Cognition, right?

要是我坚持下来就好了。如果你画出科技树，就像，嗯，很可能这事会发生。

If only I had stuck to it. If you draw the tech tree out, it's like, yeah, probably this will happen.

他离……那么近。

He was so close to...

与此同时，我们在和一群需要沙盒功能的客户交谈。那是 2023 年。

At the same time, we were talking to a bunch of customers who needed something like sandboxing. This was 2023.

所以之后我们需要一个新 API。

So we needed a new API after that.

我们在 2023 年 5 月构建了沙盒，在任何人知道这会成为一件事之前。我们发布的第一个例子是拿 Small Developer 放进循环里，让智能体可以自我迭代。

We built sandboxes in May 2023, before anyone even knew this was going to be a thing. The first example we published was we took Small Developer and put it in a loop so the agent could iterate on itself.

循环现在很难。

Loops are hard these days.

循环在什么时候？2023 年？

Loops in when was this? 2023?

是的。

Yeah.

小菜一碟。

Small chick.

那是 2023 年中。模型不是为这些构建的。它们没有经过后训练来理解循环、自我修正，工具调用虽然有但不太好。模型在 10 次迭代后就会发散，产生不了有意义的东西。

It was mid 2023. The models were not built for any of this. They weren't post-trained to understand looping, self-correction, and tool calling was there but not great. The models would diverge after 10 iterations and not produce anything meaningful.

跟三年前的自己说，答案是收集所有失败，构建基准，收集所有示例，构建口头环境，以 100 亿美元卖给 Meta，然后训练一个模型，以 600 亿美元卖给 Elon。这是个赚钱机器。没那么难。但很难有那种内在信念，相信这些东西会变得那么好——事后看来很明显。我们当时还在做什么别的？

Talking to myself three years ago, the answer is collect all the failures, build a benchmark, collect all the examples, build the oral environment, sell it for $10 billion to Meta, then train a model and sell that for $60 billion to Elon. It's a money machine. It's not that hard. But it's hard to have that inherent conviction that this stuff will get that much better in retrospect. It's so obvious. What else were we doing back then?

那是你们沙盒之旅的开始。我觉得直到去年才火起来。

That was the start of your sandboxing journey. I feel like it didn't blow up until last year.

有几年很安静。

There was a couple years of quietness.

没错。非常被低估的产品价值。我在 Charles 加入 Modal 之前和 Modal 的体验：我在黑客马拉松遇到一个人，他坚持要运行一个没地方托管的小模型。他说有家很酷的公司 Modal，可以启动一个 GPU 沙盒，我们扔上去，它接受 Hugging Face 链接。那里就有很多价值：即时托管，启动，关闭，它会保持冷状态，但几天后运行演示，它会重新起来。所有这些事后看来仍然是今天需要的。

Exactly. Very underrated product value. My experience with Modal before Charles joined Modal: I met this guy at a hackathon, he insisted we wanted to run some small model not hosted anywhere. He said there's this cool company Modal, they'll spin up a GPU sandbox, we can throw it on there, it takes a Hugging Face link. There's so much value right there: instant hosting, spin up, spin down, it'll stay cold, but you run the demo a few days later, it'll come back up. All this stuff in retrospect is still what we needed today.

今天仍然需要。工作负载形态变化很大。我们为大规模生产环境的人运行东西。不是从 0 到 1 的扩展，而是如何在给定区域从 1000 个 GPU 弹性扩展到 1500 个。这是同样形态的问题。

It's still needed today. Workload shapes have changed a lot. We run stuff for people with massive production scale. It's not about scaling from 0 to 1, but how do we scale elastically from 1000 to 1500 GPUs very quickly in a given region. It's the same shaped problem.

比如你看 Cursor Composer，他们每隔几小时就会对模型做一次强化学习。你们有整套的 RL 推理 gym 之类的。当你看这类工作负载时，基本上就是在做训练运行，需要每小时扩展和缩减数千块 GPU，对吧？这就是我们需要它的例子，对吧？

So you look at say Cursor Composer, right? They had a, we'll do RL on a model every couple hours. You guys have a whole version of RL inference gym and whatnot. When you look at workloads like that, you're basically doing train runs where you need to scale up, scale down every hour, thousands of GPUs, right? That's the example for we do need it, right?

嗯，实际上我想先退一步，谈谈人们现在如何使用 Modal。因为我们最大的用例实际上是弹性推理，而我们最初找到产品市场契合点的是自定义模型的推理。所以我们有点避开了大语言模型领域。我们为像 Suno（音频）、Runway（视频）、机器人、计算生物学等公司提供服务，这些公司在别处训练自己的模型。但 Modal 是最好的部署黑盒，可以根据流量模式的变化扩展到任意数量的 GPU。我们发现所有这些公司的流量模式都非常不可预测。比如，有些天公司会发布产品，他们需要多得多的资源，而且他们部署的不仅仅是一个模型；这些公司会在不同地区部署许多不同的模型，因此自动扩展问题变得更加困难，因为你必须在特定区域内进行扩展，而这些周期是错开的，所以不同时间需要在不同地区扩展。这就是我们的……

Yeah, well actually I'll take a step back and maybe talk about how people use Modal today. Because our biggest use case actually is elastic inference, and the thing we first found product-market fit with was inference for custom models. So we kind of stayed away from the LLM space. We were serving companies like Suno for audio, Runway for video, robotics, comp bio companies that train their own model elsewhere. But Modal is the best black box for deployment, scaling to however many GPUs you need as your traffic pattern changes. And we saw all of them actually have a very unpredictable traffic pattern. It's like, some days the company will do a launch and they'll need way more, and it's not just one model that they deploy; all these companies deploy lots of different models in different regions, and so the autoscaling problem becomes even harder because then you have to scale within a certain region and those cycles sort of are offset, so different times you need to scale up in different regions. So that's our sort of...

这本身就是一个巨大的类别。有很多推理提供商提供这种服务，Fireworks 将其作为服务提供，还有 Together、Baseten 等，至少目前在大语言模型领域已经形成了自己的细分市场。

And that in and of itself is a huge category. There's a bunch of inference providers which provide this, Fireworks does this as a service, together whatnot, Baseten, that's kind of carved into its own niche for language models at least right now.

是的，我们真正专精的是自动扩展方面，因为我们发现并非所有人都能实现自动扩展，我们在技术层面做了更深入的研究。我们还在产品中加入了 GPU 快照功能，这样你可以获取 GPU 状态，比如你的 Torch 编译器模型快照，然后下次冷启动会快得多。所以回到你的问题，这就是为什么推理需要很大的突发性。但人们也会做很多按需训练，比如强化学习，你的 rollout 就像你说的那样是突发性的。人们还会做很多批处理作业。所以我们看到很多公司在进行训练运行之前，需要数千块 GPU 来运行编码之类的任务。我认为这些事情突发性更强。我同意智能体没有那么突发。沙盒是突发性的，除非你在做强化学习。强化学习极其突发。

Yeah, I mean the thing that we have actually specialized in is the autoscaling aspect, because we found that it's not universally true that everyone else can autoscale, and we've gone deeper into it on the tech side. But we've incorporated GPU snapshotting into the product, so you can actually take the GPU state, like your torch compiler model snapshot it, and next cold starts way faster. So going back to your question, that's why you need a lot of burstiness for inference. But then people also do a lot of on-demand training, like for RL stuff, your rollouts are bursty as you said. People also do a lot of batch jobs. So we'll see a lot of companies, before they have a training run, they'll need thousands of GPUs to run encoding or something like that. And I think those things are much more bursty. I agree that agents are not that bursty. Sandboxes are, except when you're doing RL. RL is insanely bursty.

是的。

Yeah.

是的。比如在做 rollout 时，有时需要 10 万个沙盒。

Yeah. Like when you're doing rollouts, you sometimes need 100,000 sandboxes.

是的。我很好奇你是否看到了持续学习的早期迹象。像我们的朋友 Engram 最近宣布了这一点。他们正在尝试做训练。这似乎也是一种不同的工作负载，对吧？如果你 24/7 进行训练，那么人们之间使用 GPU 的方式会非常奇怪，但这似乎是你们会做的事情。

Yeah. I'm curious if you've seen early sparks of continual learning. There's some people like our friends Engram recently announced this. They're trying to do training. That also seems like a different workload, right? If you're doing training 24/7, per se, there's a very weird dynamic of how you're using GPUs between people and whatnot, but seems like something you guys would work for.

正如你所说，我们很幸运能与许多前沿客户合作，他们正在利用我们提供的基本组件，并尝试以非常有趣的方式使用它们，比如持续学习。随着技术的进步，如果更多人需要，其中一些可能会成为我们产品的一部分。但我们只是在等待这一切如何发展。

As you said, we're fortunate to work with a number of customers at the frontier and grab some of our customers, and they are taking the primitives we have and trying to use them in very interesting ways like continual learning. It's possible as the stuff gets better some of that will be part of our offering as well if more people need it. But we're just waiting to see how all this shakes out.

在沙盒之后，你们添加了哪个基本组件作为故事的下一步？

Is there a primitive that you added after sandboxing that was the next step in the story?

我想我们一直在更深入地研究大语言模型推理，因为我们意识到我们在自动扩展方面的一些优势，尤其是在不同地区，是其他地方没有的。我们的差距在于我们没有在模型层本身下功夫，我们就像一个黑盒。我们意识到，通过拥有优秀的人才来研究这一切，我们实际上可以达到前沿水平的模型性能。我们实际上已经开源了很多工作。最近我们分享了关于 Dlash 的工作，这是一个基于块的推测器，我们已经全部开源了。所以你可以通过使用开源的 vLLM 获得与专有提供商相同的性能。我们接下来考虑的是……

I guess we've been going much deeper into LLM inference because we realize that some of the advantages we have with autoscaling, especially in different regions, are not present elsewhere. And the place where we had a gap was we weren't working on the model layer itself, like we were a black box. And we realized that we actually can get to frontier-level model performance by having great people who work on all of this. And we've actually been open sourcing a lot of our work. Recently we shared our work on Dlash, which is a block-based speculator, and we've open sourced all of it. So you can get by using open source vLLM, you can get the same performance as you would with one of the proprietary providers. And the next thing we're thinking about here...

我觉得这篇博文也很有趣，对吧？我认为你在这里提出了一个观点，不是观点，而是推测解码到底能带来什么效果。你想从中指出人们应该知道什么吗？

I thought this was actually an interesting blog post as well, right? I think in here you make a claim, not a claim, just that how effective speculative decoding really gets you anything. You want to point out from this around what people should know?

嗯，当然。高层次的总结，需要描述一下什么是推测解码吗？

Uh, yeah, absolutely. I mean the high-level summaries, would it help to describe what speculative decoding is?

是的，我想我们讨论过 EGO 和 Hydra 这些东西，但那是两年前了。我觉得再说说也无妨。

Yes, I think like we've covered like EGO and all this like Hydra and all those things, but it was like two years ago. I think it doesn't hurt.

推测解码是：你有一个较小的模型，称为草稿模型，它提前预测大模型的 token，然后大模型验证所有预测的 token。它更快的原因是，如果你一次预测一个 token，你会受到内存带宽的限制，但如果你能批量验证草稿模型，那么计算效率会更高，速度也更快。只要你的草稿模型产生大量可被接受的 token（称为接受长度），你就能获得数倍于原始模型速度的提升。这就是我们在这里强调的：人们经常谈论让内核更快，但改进内核只能带来几个百分点的提升，而增加接受长度实际上能带来 2 到 4 倍的乘法级减速，而不会对性能造成太大影响。

Speculative decoding is: you have a smaller model called a draft model that predicts tokens ahead of the bigger model, and then you have the bigger model verify all the tokens that are predicted. The reason it's faster is if you're predicting one token at a time, you're kind of bound by memory bandwidth, but if you can batch the verification of the draft model, then you're much more efficient using compute and it's faster. And as long as your draft model is producing a lot of tokens that can get accepted, which is called the accept length, you can get speed up that's multiple times of the original model speed. And that's what we highlight here: people talk a lot about making kernels faster, but improving kernels only gives you a few percentage points of improvement, and increasing accept length literally is a multiplicative decrease of 2 to 4x without much head-on performance.

是的。

Yeah.

是的。我想可能，我的意思是你运行了第二个模型，对吧？所以可能在计算上更昂贵，但是……

Yeah. I think it maybe, I mean you are running a second model, right? So maybe it's more expensive in compute, but...

我指的是质量，但没错。我认为是这样，质量性能没有下降，因为你永远不会接受一个……

I meant quality, but yeah. I mean I think so, there's no drop in quality performance because you're always, you're never accepting a token that...

更好或相同。

Better or same.

正是。是的。

Exactly. Yeah.

对。是的。所以我们一直在研究 Dlash，这是一个基于块的推测器。它不是一次预测一个 token，而是预测一个块。我们一直在开源我们的工作。我们接下来的目标是帮助人们训练推测器和自定义模型。

Right. Yeah. And so we've been working a bunch on Dlash, which is a block-based speculator. So it's instead of predicting one token at a time, it's predicting a block. And we've been open sourcing our work with it. The next thing for us here is for helping people train speculators and custom models.

嗯，这通常是非常依赖全职员工和支持部署工程师的事情，比如你和客户合作，帮助他们完成这些工作。我们推出自动端点的愿景是让前沿级别的性能对每个人都可用。所以我们提到了这个公告，算是预告了一下。接下来我们要推出的基本上是，当你运行一个自动端点时，我们会镜像流量……你想从高层次解释一下自动端点吗？

Uh it's something that traditionally is very FTE driven, support deployed engineer driven, like you work with customers and help them do that. And our vision for this is why we launched auto endpoints is we want to make frontier level performance available to everyone. And so we mentioned this announcement, we kind of teased it. The next thing we're launching is basically as you run an auto endpoint, we shadowed traffic and... Do you want to explain auto endpoints at a high level?

是的。所以这要回到你的 Modal 理念，你接触代码，但有时人们实际上不想碰代码，他们想要一个开箱即用的端点，具备 Modal 的所有优秀性能和可扩展性。所以我们通过一种从 UI 或 CLI 创建端点的方式简化了这一点，它内置了我们讨论过的所有优化，比如去闪烁功能，并且完全透明。我们给你代码，你可以自己运行，如果你愿意，还可以退出到完整的 Modal 体验，我们看到随着用户变得成熟，他们确实想调整模型、微调东西，你仍然可以做所有这些，它不是黑箱。是的，接下来我们在帖子中预告的是，我们如何在你无需与人沟通的情况下，让你的草稿模型随着数据分布的变化而演进，从而为你提供超越这一点的价值。

Yeah. So this is going back to your modal is you touch the code, but sometimes people actually don't want to touch the code and they want to get started with an endpoint that works and has all the great performance and scalability that modal has. So we've made that easier with basically a way to create an endpoint from our UI, from the CLI, that has all of our optimizations that we talked about, like the deflash stuff already baked in, and there's full transparency. So we give you the code, you can go run it yourself, and if you want you can eject out into the full modal experience, which we see as people get sophisticated, they do want to tweak the models, they want to fine-tune stuff, you can still do all of that, it's not a black box. And yeah, the next thing as we tease later in the post is how do we give you value even beyond this in terms of having your draft models evolve as your data distribution evolves, again without having to talk to a person.

我想直接理解一下，显然你们有 GPU，有兼容的端点，提供开放模型。如果有人自己做，你们提供的增量是什么？你们在高效推理方面做了很多出色的开源工作。相比而言，比如我拿同样的模型 GLM 5.2 2 FB8，用现成的推理引擎 VLM SG lang，获得类似容量和成本的计算资源，接入这样的东西除了扩展的好处之外，还提供了什么样的增量？

I guess just to understand it directly, obviously you have the GPUs, you have an endpoint that's compatible, you serve open model. If someone was to do this themselves, what's the delta that you guys provide? So you do a lot of open-source great work on effective inference. How does it compare to say I take the same model, GLM 5.2 2 FB8, take off-the-shelf inference engine VLM SG lang, get compute of similar capacity, similar cost? What's the kind of delta that plugging into something like this offers, outside of the benefit of scaling?

这很有趣，因为我们采取了开源贡献并向上游提交的方法。我们与 SG lang 团队密切合作。我们实际上希望我们团队提出的改进能够存在并开源，供其他人甚至在 Modal 之外使用。对我们的好处是，我们有一个拥有丰富专业知识的团队，如果你遇到某些尚未解决的问题，我们的团队可以首先帮你获得那种性能。另一件事是，通过这些端点，我们比任何人都更具弹性，你可以真正缩放到零，真正实现突发性，在实践中，这对人们来说比仅仅找到 GPU 并在上面运行模型代码要重要得多。

It's kind of interesting because we've taken the approach of open sourcing our contributions and upstreaming them. We work closely with the SG lang team. We actually want the improvements that our team comes up with to be there and open source for others to use even outside of modal. The benefit to us is we have a team that has significant expertise in terms of if you do have something that is not there, our team can help you get that performance first. The other thing is with these endpoints we are way more elastic as you said than anyone else, and you have true scaling to zero, you have true burstiness, and in practice that matters a lot more to people than just finding the GPU and running model code on it.

是的，我要说的是，实际上并不像我说得那么简单，说起来容易做起来难，对吧？我认为对普通人来说，仍然很难凭直觉检查使用不同的组合，你可以做很多组合。这些权衡从表面上看并不明显。

Yeah, and I will say it's actually not that straightforward to just like what I said is easier said than done, right? It's I think still for the average person still hard to just gut check using different, there's quite a bit of combinations you can make there. The trade-offs aren't really known at face value.

是的，我的意思是不止这些。我认为运行生产级推理本身就是一个困难的推理问题，即使你减去自动扩展，还要控制尾延迟之类的事情，并确保每个请求至少被交付一次等等。这里有很多创新可以做。

Yeah, I mean it's not just that. I think it's that running production grade inference is a hard inference problem, even if you subtract out the autoscaling, controlling things like tail latency and making sure every request is delivered at least once and whatnot. There's a lot of innovation that you can do here.

我认为非常有趣的是，随着你成为一个完整的云服务，你开始侵入别人的地盘了。

I think it's very interesting that you're starting to encroach on like as you become a full cloud, you're starting to encroach on other people's turf.

嗯。

Mhm.

你们不会做什么？

What will you not do?

嗯，我们希望跟随用户，确保他们得到一个一切协同工作的平台。所以现在我们主要关注模型生命周期和智能体生命周期。既包括从数据准备到训练再到推理，也包括如果我想要部署一个后台智能体，比如从沙箱到持久存储，还有很多其他东西。

Well, we want to follow our users and make sure they get a platform that has everything that works well together. So right now we're kind of focused on the model life cycle and the agent life cycle. So both like going from data prep to training to inference, and also if I want to deploy a background agent, let's say sandbox to persistent storage, a whole bunch of other stuff.

我们和做 Open Inspect 的 Cole 聊过。是的。

We talked to Cole who did open inspect. Yeah.

是的。显然 Real Inspect 也在 Modal 上。

Yeah. And obviously real inspect also is on modal.

是的。所以 Ramp Inspect 是一个后台智能体的绝佳例子，非常成功，因为他们能够使用一些原语，比如快照和快速扩展，使系统感觉反应灵敏且运行良好。

Yeah. So ramp inspect was a great example of a background agent that was really successful because they were able to use some of the primitives like snapshotting and fast scaling to make something that feels really reactive and works well.

是的，那就是 RAM 的新 CTO。

Yeah, that's the new CTO of RAM right there.

是的，Rahul。真的非常有趣。嗯，是的，你知道，我认为都非常乐观。比如，我的一个反思是，我最初并没有意识到，因为显然当我遇到你们时，你们在 GPU 方面涉足不深，而现在你们完全专注于推理。我在今年 GTC 的 Jensen 主题演讲中强调的一点是我们所谓的推理拐点，对吧？比如在 AI 工作负载或机器学习工作负载中，过去 GPU 与 CPU 的比例大约是 8:1，现在更像是 1:1，这很有趣，因为智能体很大程度上被阻塞或调用 CPU 密集型任务。实际的限制因素在 GPU 和 CPU 之间来回摇摆，比以前频繁得多。过去全是 GPU，偶尔有 CPU，GPU，CPU，现在则是持续交替，你必须将所有东西放在一起。

Yeah, Rahul. It was really really fun. Um yeah, I mean you know I think all very bullish. Like you know one of my reflections was also I did not originally, because obviously when I met you guys you weren't that much in the GPU game, and now you're all about inference. And one of the points that I hinged on for Jensen's keynote at GTC this year was what we're calling like the inference inflection, right? That let's say in AI workloads or machine learning workloads it used to be like let's call 8:1 GPU to CPU, and now it's more like 1:1, which is interesting because of how much agents basically are blocked or call out to CPU heavy stuff. The actual limiting factor swings back and forth from GPU to CPU a lot more than it used to be. All GPU and then occasional CPU, GPU, CPU, and now it's like just constantly, and you just have to collocate everything.

是的。这也是我们再次看到 Modal 吸引人的地方之一，我们建立了一个跨越 17 个云服务商的能力池。所以我们非常擅长在全球各种云能力上运行。

Yeah. And that's one of the things that actually again we see is something appealing about modal, which is we've built this capacity pool that spans 17 cloud providers. So we're very good at running on various kinds of cloud capacity across the world.

你们没有自己的数据中心。

You don't have your own data centers.

我们没有自己的数据中心。我们只是运行在许多 NeoCloud 和提供商之上。

We don't have our own data centers. We just run across a lot of NeoClouds and providers.

是的。你在做数学计算，然后想，转折点在哪里……

Yeah. You're running the math and you're like what's the cut over point where you're like...

是的。这是个好问题。部分原因是我们认为自己的差异化在于软件层，轻资产并专注于软件帮助我们快速行动。到目前为止效果很好，因为有很多其他人在建设数据中心，我们能够与他们有效合作，并再次专注于让我们与众不同的东西。

Yeah. It's a good question. I mean part of it is we see our differentiator in the software layer and being capital light and focusing on the software helps us move really fast. So far it's worked out well because there are so many other people building data centers that we're able to work effectively with them and again focus on what makes us special.

17 个让你有时能接触到本地提供商……最有趣的是哪个？

17 gets you into like the local providers sometimes like... Who's the most interesting one?

实际上 Neo Cloud 比你想象的要多得多，它们都有不同程度的可靠性，这就是为什么我们投入大量时间在上面构建自己的可靠性层。所以如果 GPU 掉线或发生其他事情，用户工作负载不会受到影响，这实际上让我们能够使用比你自己作为用户所能使用的多得多的容量。

There actually a lot more Neo clouds than you expect and they all have various degrees of reliability, and that's why something we've invested a lot of time in is actually building our own reliability layer on top. So if the GPU falls off the bus or something happens, user workloads are not affected, and that actually lets us use a lot more capacity than you as a user would be able to.

拥有这个很有用，因为现在每个人都知道你处于哪一层，你基本上是在优化成为所有云的超级云。

It's a useful thing to have because like now everyone knows what layer you are and you sort of optimize for being the super cloud of all clouds.

是的，就是这个想法。所以当你提到共置时，那是另一个有趣的事情，我们看到的一件事是，当人们想要非常特定位置的 CPU 或 GPU 时，他们会来找我们，比如他们希望它固定在欧盟。

Yeah, that's the idea. And so I guess when you mentioned collocation, that's another interesting thing where one thing we've seen is people come to us when they want very specifically located CPUs or GPUs, like they want it pinned in the EU.

是数据本地化问题还是性能问题，还是什么？

Data locality thing or performance or what?

要么是数据本地化，要么是延迟。是的。

It's either data locality or latency. Yeah.

就像你希望你的沙盒在 Modal 中运行，你希望它们紧挨着那个。这在所有那些事情中都很重要。所以你有点意外地，我不知道是不是意外，但你已经为智能体表达自己构建了完美的原语。然后几乎很有趣的是，每一个额外的开发都只涉及更多的文件系统，更多的 CPU，就像你已经拥有的东西。我不太了解是否有任何有趣的网络用途，但你在网络方面也做了一些很好的工作。

Like you want your sandboxes running in Modal, you want them right next to that. That is important in all those things. So you've kind of accidentally, I don't know if it's accident, but you've built the perfect primitive for agents to express themselves. And then it's almost very funny how every extra development just involves more file system, just involves more CPU, just like the things that you already have. I don't know much about if there's any networking usages that are interesting, but you've also done some good work on networking.

是的，我的意思正是如此。我们只是把算力、存储和网络拿来，在那层上构建人们需要的东西。

Yeah, I mean that's exactly right. We're sort of just taking compute, storage, and networking and building stuff on that layer for again the stuff people need.

我们看到一些有趣的网络问题出现。一个是人们实际上想要沙盒。所以我们有一个类似 Docker Compose 的东西。

We see a few interesting network things coming up. One is people actually want sandboxes. So we have a Docker Compose type thing.

类似于 Docker 集群之类的东西。抱歉。Docker swarm。它叫什么？

For like a Docker cluster type thing. Sorry. Docker swarm. So what is it called?

Compose。

Compose.

Compose 之类的东西。

Compose type thing.

是的。所以实际上如果你想要 Docker Compose，我们的沙盒现在支持一个叫做 sidecar 的东西。所以一个沙盒实际上是一个容器组，你可以在沙盒中运行多个容器。这也很有用，因为回到网络问题，人们想要对沙盒的出站网络有大量控制。他们可能想运行中间人代理来记录强化学习的东西，或者控制如何向域出口、注入凭证。是的，所以我们不得不自己构建很多这些东西。

Yeah. So actually if you want Docker Compose, our sandboxes now support this thing called sidecar. So a sandbox is actually a pod of containers and you can run multiple containers in the sandbox. Also useful because going back to networking, people want a lot of control over outbound networking from a sandbox. They might want to run a man-in-the-middle proxy for logging stuff for RL or controlling how egress can happen to a domain, injecting credentials. And yeah, so we've kind of had to build a lot of that stuff ourselves.

是的。

Yeah.

但有时人们实际上希望跨多个节点的沙盒相互通信。这是我们看到的一个新兴事物。我们出于不同的原因支持了这一点，是的，我们会看看这是否变得安全。

But then also sometimes people actually want sandboxes spanning multiple nodes to talk to each other. Which is an emerging thing we're seeing. We have support for that for a different reason and yeah we'll see if that becomes safe.

就像开放的套接字，直接像 NTLS？

Like just an open socket, it's directly like NTLS?

我们确实支持这一点，你可以在沙盒内暴露一个隧道，然后你可以暴露给公共互联网，或者它可以是，你可以在上面加一个 HTTP 奇数层。但我们有一个叫做 I6PN 的东西，我们还没谈到过，这是一个使用 IPv6 地址的覆盖网络。所以如果同一个工作空间内的 Modal 容器，当这个启用时，实际上可以使用这个私有 IPv6 地址相互寻址，没有其他人可以。这有点像容器的私有网络。我们实际上构建它是因为我们需要它作为分布式训练产品的原语。所以我们有另一个功能，你可以给一个函数添加一个装饰器，然后你得到一个 GPU 集群，它们有 RDMA 网络，所以你可以运行一个真正无服务器的分布式训练作业，我们需要覆盖网络来实现这一点。但后来我们看到人们出于其他原因使用它，我很好奇会看到什么。

We do support that which is you can expose a tunnel inside a sandbox and then you can either expose to the public internet or it can be, you can add an HTTP odd layer above it. But we have this thing called I6PN which we haven't talked about, which is this overlay network using IPv6 addresses. So if Modal containers within the same workspace, when this is enabled, can actually address each other using this private IPv6 address and no one else can. It's like sort of private networking for containers. We actually built it because we needed it as a primitive for our distributed training product. So we have this other feature which is you can add a decorator to a function and you get a cluster of GPUs and they have RDMA networking, so you can run a distributed training job that's truly serverless, and we need the overlay network for that. But then we've seen that people are using it for other reasons and I'm kind of intrigued to see.

是的，人们会用它做什么？构建原语，让人们自己去发现，对吧？

Yeah, what would people do with it? Build primitives and let people figure it out, right?

他们读了文档，我们挑选，让我用它做你从未打算过的事情。这甚至不在我们的文档页面上。人们不知怎么找到了它，并且正在使用它。

They read the docs, we pick, let me use that for something that you never intended. This is literally not even in our docs page. People somehow found it and they're using it.

我的意思是，你用 RDMA 对比 TCP 的方式描述得很好，但只是大规模强化学习的传输速度变化……

I mean the way you portrayed it with RDMA versus TCP, very well laid out, but just the transfer speed change at scale for RL like...

是的，你内置了它。我肯定有人在你们真正把它做成产品之前就发现它效率高得多，对吧？

Yeah, you have it built in. I'm sure someone found it to be a lot more efficient before you actually made a thing out of it, right?

是的。而且不是吹毛求疵，我想覆盖网络实际上是 TCP 覆盖网络。我们拥有它的原因是，在之上建立 RDMA 网络之前，你需要它来进行 RDMA 的密钥交换。但后来人们发现了 TCP 部分。

Yeah. And not to split hairs, I guess the overlay network actually is the TCP overlay network. The reason we have that is you need that to do the key exchange for RDMA before you set up the RDMA network on top of that. But then people found the TCP part.

我能告诉你这对我来说是一个大顿悟时刻吗？因为我为世界博览会审阅了 2200 份投稿。然后我从 John Osterhalt 那里得到了这个，我不知道你是否知道 John Osterhalt 这个名字。他出版过，他是一位知名教授，出版了很多有趣的软件设计书籍，这是他选择提交的演讲。是关于 RDMA 在 imprint 上的，我想，你不会想到这个有点像操作系统的人会关心 RDMA。我的意思是，这对我来说有道理，因为云，对吧？就像你移动 KV 缓存的方式以及你能多高效地做到，你多高效地把权重从训练 GPU 移动到推理 GPU，强化学习有很多自由度，这基本上是一个移动内存、调度的系统问题。

Can I tell you this is like a big aha moment for me? Because I reviewed 2,200 submissions for the World's Fair. And then I got this from John Osterhalt, who I don't know if you know John Osterhalt by name. He published, he's a well-known professor, published a lot of interesting software design books, and this is the talk he chose to submit. It's on RDMA at imprint, and I'm like, you wouldn't think that this guy who is like kind of operating systems guy would care about RDMA. I mean it makes sense to me because cloud, right? Like the way you move around your KV cache and how efficiently you can do it, how efficiently you move your weights from your training GPUs to your inference GPUs, and RL is there's a lot of degrees of freedom and it is basically a systems problem of moving memory around, scheduling.

这显示了我对网络知识的理解有多原始。这也属于 WireGuard 的领域吗？

This shows you how primitive my understanding of networking stuff is. Is this like the domain of WireGuard as well?

不完全是，所以是相邻的。

Not quite, so adjacent.

嗯，所以解释一切。当然，当然。我们如何在 GPU 之间移动内存？

Um, so explain everything. Sure, sure. How do we move memory around GPUs?

哦抱歉，是的，那是内存。抱歉，我更多是在说，也许我五分钟前在说你建立的私有 IPv6 寻址。它基本上是一个 VPN 吗？

Oh sorry, yeah that is memory. Sorry, I was talking more and maybe I was talking like 5 minutes back about the private IPv6 addressing that you've set up. Is it basically a VPN?

是的，有点像 VPN，是的，WireGuard 是……是的，你说得对。它在同一个领域。

Yeah, it's sort of like a VPN and yeah, WireGuard is... Yeah, you're right. It is in the same space.

是的，你已经转到话题了。

Yeah, you already moved on to topics.

WireGuard 是加密的，而这个是……

WireGuard is encrypted and this is...

你不需要加密。

You don't need encryption.

是的，它没有加密。这是主要区别。这是 TCP，我们有 eBPF 程序，会根据你是否被允许来拒绝或允许 TCP 连接。过去涉及一个全尺寸的 car，但现在你在 Linux 内核中有 eBPF。

Yes, it's not encrypted. That's the main difference. This is TCP and we have eBPF programs that will reject or allow the TCP connection based on whether you're allowed to do it. Used to involve a full-size car but now you have eBPF in the Linux kernel.

是的。我不知道这是否自然延续了话题，就像我对分布式训练的怀疑是，人们花很多钱在连接 GPU 的线缆上，即使这样也不够快，那是瓶颈。你的网络够快吗？

Yeah. I don't know if this is a natural follow-on to the topic of like my skepticism on distributed training is that well, people spend a lot of money on cables to hook up GPUs and even that is not fast enough and that's the bottleneck. Is your networking fast enough?

是的。所以我想你说的是那种完全分布式训练，比如 dialoc 之类跨……那会是，是的，那是极端情况。是的。你处于中间，然后其他人会在他们的实际数据中心里有像 Melanox 线缆这样的东西。

Yeah. So I guess you're talking about sort of fully distributed training like a dialoc or something which is cross... that would be yes, that's the extreme. Yeah. You're kind of in the middle and then other people would have like the Melanox cables up in their actual data center.

当你在 Modal 上运行多节点训练时，RDMA——我认为是 Mellanox 或 InfiniBand——就像，你也使用 RDMA，但基本上这是一种绕过 TCP 网络堆栈、在节点之间更快传输数据的方式。而且我们拥有，我想，大约 3 太比特每秒的内部网络，这是所需的标准。

When you run multi-node training on Modal, the RDMA — I think Mellanox or InfiniBand — is like, you also use RDMA, but basically it's a way to bypass the TCP networking stack and transfer stuff much faster from one node to another. And we have, I think, like 3 terabits per second internal networking, which is the standard that's needed.

好的，所以我误解了 TCP 堆栈的哪一部分。

Okay, so I misunderstood what part of the stack of TCP.

是的。

Yeah.

我的意思是，非常令人印象深刻的工作。所以实际上你们是在将 Modal 的理念扩展到训练集群。

I mean, very impressive work. So effectively you're extending the Modal philosophy to the training cluster.

是的。而且我们显然不是针对大规模预训练运行。我们构建多节点训练是为了应对我们看到的大量小规模后训练。比如人们正在对中等规模的 Qwen 模型进行后训练，以便在推理中获得更高质量。这非常适合这类场景。

Yeah. And we're not going for obviously large-scale pre-training runs. The thing that we've built multi-node training for is we see a lot of smaller-scale post-training. Like people are post-training medium-sized Qwen models so they can get higher quality on inference. This is a perfect fit for something like that.

是的，这就是我对许多实验室如何探索后训练分支并最终合并他们发现的东西的印象。

Yeah, that is my impression of how a lot of these labs explore branches in post-training and then eventually merge whatever they find.

是的，我们看到的另一个多节点训练用例是，即使你有一个大集群，你的研究人员仍然在进行小规模运行，而弹性在那里更为重要。这实际上是自动研究的当前限制因素，即你基本上需要给你的模型一些 GPU。我们有一篇关于自动研究的博客文章，Modal 被证明是一个相当好的基础。

Yeah, the other use case we've seen for multi-node training is even if you have a big cluster, your researchers are still doing small runs, and having elasticity there matters a lot more. This is actually the current limiting factor for auto research, which is you basically need to give your model some GPUs. We have a blog post on auto research and Modal is, yeah, turns out to be a pretty good substrate for that.

所以我的印象是自动研究意味着很多事情，比如如果有什么新词的话……

So my impression is auto research means many things, like if anything coins...

目前它仍然像科学展览，对吧？实际上并不像，我不知道有多少人真正在做这个。

Right now it's still science fair, right? Not actually like, I don't know how many people are actually doing this.

是的，我也这么想。

Yeah, I thought the same thing.

是的，你会知道的。

Yeah, you would know.

我们内部的训练和推理团队实际上都大量使用了这种通用模式。比如我们有一个内部仓库叫 auto inference，本质上我们使用这个框架自动化了我们自己的 FTE 工作，即智能体会启动一系列不同的扫描。它甚至会运行像 Nvidia Nsight 分析器这样的工具，调整配置，然后找到正确的东西。它会将你的 GPU 从 H200 换成 B200，而且效果非常好。

We like our internal both training and inference teams actually use this sort of the general shape of this quite a bit. Like we have this one internal repo called auto inference, which is essentially we've automated our own FTE efforts using this harness, which is the agent will just spin up a sweep of different things. It'll even run like Nvidia Nsight profiler and it'll like tweak configs and it'll arrive at the right thing. It'll change your GPUs from H200 to B200 and actually works really well.

不错。

Nice.

顺便说一句，我很欣赏你们的 FDE 如此技术化，以至于你们必须做这些事情。

By the way, I enjoy that your FDE is so technical that you have to do these things.

这与其他人的 FDE 非常不同。

It's very different from FDE from other people.

是的。对于我们的 FDE 团队，他们本质上就像是应用推理研究员或应用训练研究员。

Yeah. For our FDE team, they're essentially like applied inference researchers or applied training researchers.

有人告诉我，他们必须既能构建又能销售。他们必须销售吗？还是他们只是擅长？这更像是售后类型的事情。

Someone told me like they have to be able to build but they also have to be able to sell. Do they have to sell or are they like, they're good? This is like post-sale type of thing.

确实如此。能够与客户交谈并有效互动非常重要。同样的事情，你知道的。

It does. Being able to talk to a customer and engage effectively with them matters a lot. The same thing, you know.

但这并不是真正的销售工作。我们还为他们配备了解决方案架构师，他们更偏向售前。

But it's not really a sort of sales thing. We pair them with solutions architects as well that are more on the pre-sales side.

好的，让我们多花点时间在自动研究上。这是我今年的一大重点。这会走向何方？你知道，人们探索得够多了吗？你知道，有所有这些漂亮的图表，改进，改进，然后它有点趋于平稳，然后你找到下一个东西。这基本上是比正常训练高一个抽象层次吗？我们是这么想的，还是你有不同的看法，比如模型级训练与基本上像 AI 驱动的超参数搜索？

Okay, let's spend a bit more time on auto research. This is a big focus for me for this year. Where does this go? You know, like have people explored enough? You know, there's all these beautiful charts of improve, improve, then it sort of levels off a bit, and then you find the next thing. Is this basically sort of one abstraction up from normal training? Is that how we think about it, or do you think about it differently, like model-level training versus basically like AI-driven hyperparameter search?

有些人称之为神经架构搜索之类的，对吧？

Some people call it neural architecture search or whatever, right?

是的。

Yeah.

我的意思是，我看到人们用它做的事情远未达到架构层面。它基本上是在调整参数，但本质上是由某种模型直觉引导的超参数扫描。所以它比你拥有的任何其他扫描器都高效得多。是的，我的意思是，这只是你想把算力花在哪里的问题，对吧？

I mean, the stuff I've seen people do with it is nowhere near the architecture level. It's pretty much tweaking parameters, but it's basically a hyperparameter sweep that's guided by some sort of model intuition. So it's much more efficient than whatever other sweeper you would have. Yeah, I mean, it's just a question of where you want to spend your compute, right?

因为是的，你可以在这上面投入无限的资金，然后不知怎么地你就会撞上莎士比亚，你知道的。

Because yeah, you can just throw infinite amounts of money on this and somehow you'll bang on Shakespeare, you know.

是的。互联网猴子。

Yeah. Internet monkey.

是的，我的意思是，这对 Modal 非常好，而且我认为智能体能够启动其他智能体也非常重要。它们可以启动自己的基础设施，这对你非常有利。语言模型在生成 Modal 代码方面有多好？你知道，现有预训练的好处是你就在数据中。

Yeah, I mean, so very good for Modal, and I think it's also very important that agents can spin up other agents. They can spin up their own infrastructure, like very, very good for you. How good are LMs at generating Modal code? Like, you know, the benefit of existing pre-training is that you are in the data.

它们实际上出奇地好。我认为在 Claude 4 之前它们不行，但现在它们能够开箱即用地一次性完成。我们正在考虑发布一个 Modal 基准测试，用于语言模型尚不能完成的更难的事情，也许……

They're actually surprisingly good. I think pre-Claude 4 they were not, and then now they're able to one-shot stuff out of the box. We're playing around with releasing like a Modal bench for the harder things that the LMs cannot do yet, and maybe...

举个例子？

What's an example of that?

我认为有时智能体在没有正确指导和技能的情况下难以处理的事情是，如何使用我们其他的可观测性工具，比如如何看到某个东西失败了，如何查看日志然后更新正确的东西。这有点像关于那个的推理。

I think the things that sometimes agents struggle with, without right guidance and a skill, is how to use the rest of our observability, like how to see something is failing, how do you look at the logs and then update the right thing. It's sort of reasoning about that.

但它们能够一次性完成……

But they're able to one-shot like...

是的，你不能只是给它添加一个技能。

Yeah, you can't just add a skill to it.

是的。所以我们现在有一个模型技能，这实际上就是我们构建这个 Modal 基准测试的原因。就是为了发现这类问题，以便我们在调优技能中解决它们。

Yeah. So we have a model skill now, which is kind of actually why we built this Modal bench. It's to find things like that so we can address them in our tuning skill.

不，不，我的意思是这很好。你们面临任何短缺吗？你知道，我们经常谈论 GPU 短缺，但也包括 CPU、内存。

No, no, I mean it's good. Are you facing any shortages? You know, we talk a lot about GPU shortages, but also CPU, also memory.

是的。

Yeah.

我们经历了很多增长，这意味着我们必须在主动容量规划方面做得更好。

We have had a lot of growth, which means that we've had to be much better about proactive capacity planning.

是的。所以我们有一个……

Yeah. So we have a...

顺便说一句，这就像 MBA 的梦想工作。就是规划这些东西。我想上次你和我谈过这个。

Which by the way, like it's like an MBA's dream job. It's like just planning this stuff. I think last time you and I talked about something maybe about this.

是的。我的意思是，我们有一个非常能干的团队，我们称之为算力策略角色。所以，如果这里有听众想从事算力策略工作。

Yeah. I mean, we have a really competent team of people that we call the role compute strategy. So yeah, if anyone listening here wants to work on compute strategy.

我喜欢，我的意思是普通人称之为 FPNA 之类的。

I like, I mean the normies call it FPNA or something.

嗯，这更像是，不是 FPNA。它涉及很多有趣的财务问题，比如一年期和三年期预留之间的混合比例是多少？我们如何预测自己的容量？我们基本上如何做，尤其是因为我们的容量在不同 GPU 类型和不同区域之间非常可互换。就像你基本上必须对很多内容进行建模。而且你还必须对供应链将如何演变有自己的看法，然后基于此下注。Token 经济学。

Well, it's more, it's not FPNA. It's, there's a lot of interesting financial questions of like, what is the blend between one-year and three-year reservations? How do we forecast our own capacity? How do we basically, especially since our capacity is very fungible across different GPU types and different regions. Like you basically have to model a lot of it. And you also have to have an opinion on how the supply chain is going to evolve and then you have to take bets based on that. Tokconomics.

嗯，这可能不算一个真正的观点，但我试着想还有哪些行业遇到过这类问题。航空公司的燃油对冲——西南航空曾因一次大胆的燃油押注而长期保持超低成本，远超其他航司。

Yeah, this is like probably not a real point, but I was trying to think about what other industries have had this kind of problem. Airlines with fuel hedging—for a long time Southwest made a hero fuel bet and was super low cost compared to everyone else.

是的，我也想过这个。我们其实也挺享受这个过程的。对我们来说，算力业务整体上也非常注重容量管理。这样才能实现优秀的单位经济，同时随着时间的推移为客户释放更多价值。我们现在正在构建的一个功能是，如果客户不关心延迟，他们可以获得更便宜的价格，结果会在 24 小时内返回——本质上就是一个批处理层。

Yeah, I had thought about that. We're having a fun time too, you know. The compute business in general for us is also about being very good at capacity management. That is how you have great unit economics, but also over time how you can unlock more value for customers. One of the things we're building now is a way for customers to get much cheaper pricing if they don't care about latency—they'll get results back in the next 24 hours or so, essentially a batch tier.

是的。这些是我们拥有的杠杆，因为我们控制着整个堆栈和调度，可以给人们足够的……嗯，我觉得它们没有达到应有的流行度。前沿实验室都有这些 API，但它们的普及度还不够。

Yeah. And those are levers we have because we control the whole stack and scheduling to give people sufficient... Yeah, I feel like they're not as popular as they should be. The frontier labs have all those APIs, but they're not as popular as they should be.

我们看到对这种功能的需求实际上并非来自语言模型领域，尽管有时人们想运行评估和做合成数据准备，那确实有意义。但需求主要来自很多非语言模型公司，比如做计算生物学的人——他们没运行过真正的大规模批处理作业，也不在乎什么时候拿到结果。

The demand we see for something like that is actually not for LMs, although sometimes people want to run evals and do synthetic data prep, and there it makes sense. But it's from a lot of non-LM companies, like people doing computational bio—they haven't run really big batch jobs and they don't care about when they get it back.

是的。这也有点像停机问题——能否及时完成。

Yeah. And it's also like a cousin to the stopping problem of will this finish in time.

是的，你可以设定边界，可以给人们提供保障。

Yeah, you can bound it, you can give people on it.

是的。我觉得有趣的是 Modal 的下一阶段——既然你们已经站稳脚跟，成为领先公司中知名的算力玩家，人们对你们有什么期待？你们有一个推理发布周，我们聊了一些发布内容。还有什么人们应该知道的？

Yeah. I think what's interesting is the next phase of Modal—what do people expect from you now that you're established and a well-known compute player among leading companies? You had an inference launch week and we talked a little about the launches. What else should people know?

我们正在构建能让用户生活更轻松的基元。例如，在语言模型推理方面，将有成千上万的公司进行后训练并部署开源模型用于推理。所以我们正在思考什么样的产品形态最适合这一点，这涉及从我们的训练平台到能获得前沿性能的端点，全程无需与人沟通。在其他垂直领域情况有所不同——我们也看到很多实时音视频需求，这就是为什么我们在开发带有回退机制的区域路由，以便让 GPU 尽可能靠近用户，实现低延迟视频流。在智能体方面，我们仍在与客户紧密合作，因为他们的需求变化非常快。除了沙箱和持久化文件系统，人们在构建生产级智能体时还需要很多其他东西。所以我们正在考虑那些适合的组件。

We are building primitives that make our users' lives much easier. For example, with LM inference, thousands more companies are going to post-train their own models and deploy open source models for inference. So we're thinking a lot about what the best product shape is for that, and that involves everything from our training gym to endpoints that get frontier-level performance, without having to talk to anyone. It looks somewhat different on other verticals—we're also seeing a lot of real-time audio video stuff, which is why we're working on things like regional routing with fallbacks so you can get GPUs as close to users as possible for low latency video streaming. On the agent side, we're still working closely with our customers because stuff is changing so fast in terms of what they need. Beyond sandboxes and persistent file systems, there are a lot of other things people need from this agent stack as they build production agents. So we're thinking about those other things that fit in there.

我想问那些其他东西是什么。

I want to ask what the other things are.

嗯，现在可能可以分享。我经常思考云的主要组件——计算、存储、网络。到目前为止，前几代云都还好。智能体有什么本质不同，以至于需要新的权限级别？很多人……我就随便说说，希望能激发点想法。现在的新级别就是 Claude Code 所做的——危险地跳过权限或按命令允许列表，对吧？有时它们有这种自适应思考模式，就像‘相信我，兄弟，我来帮你调用。’就是这样吗？基本上就是语言模型中介的权限。

Yeah, probably share right now. I think a lot about the principal components of cloud—compute, storage, networking. So far for the first couple generations of cloud, it's fine. What's qualitatively different about agents that you need some new permission level? A lot of people... I'll just kind of spew tokens at you until it sparks something. The new level now is whatever Claude Code does—dangerously skip permissions or allow list by command, right? And sometimes they have this adaptive thinking mode where it's like 'just trust me, bro, I will make the calls for you.' Is that it? Basically LM-mediated permission.

现在你把它与目标循环起来，然后泛滥成灾。

Now you're looping it with a goal and flooding row.

是的。我的意思是，我对沙箱级别的语言模型中介权限持怀疑态度，因为你确实需要硬边界。显然有人可以窃取数据。但也许那是老派思维。也许我们是恐龙。也许 AI 操作系统或大语言模型操作系统的内核就是一个该死的大语言模型。这让你感到不舒服，但这就是给大语言模型加糖霜。想象一个完美的球形奶牛大语言模型，对吧？让它也许……我想测试边界。显然我不相信那个，但我想看看我错在哪里。那是非共识。

Yeah. I mean, I'm skeptical of LM-mediated permission for stuff at the sandbox level because you do want hard boundaries. Obviously someone can exfiltrate stuff. But maybe that's old school thinking. Maybe we're the dinosaurs. Maybe the AI OS or the LLM OS is really the kernel is a goddamn LM. It makes you feel uncomfortable, but that's what frosting the LM is. Imagine a spherical cow perfect LLM, right? Let it maybe... I want to test the boundaries. Obviously I don't believe that, but I want to see where I'm wrong. That's the non-consensus.

是的。我的意思是，我认为你总是需要硬护栏，并且可以搭配软护栏，对吧？然后协商并调解。

Yeah. I mean, I think you always need hard guardrails when you want, and you can pair those with softer guardrails, right? And ask a deal and mediate it.

最后我想听听你对 Modal 之外生态系统的评论。管理智能体——人人都有。Gemini、OpenAI、Claude——对你很有用，但这也是他们开始侵入你领域的方式。发生了什么？

I'll end with a couple of your commentary on the ecosystem outside of Modal. Manage agents—everyone has one. Gemini, OpenAI, Claude—very useful for you but also it's their way of starting to edge into your space. What's going on?

是的。我的意思是，我们非常高兴能与 Anthropic 和其他一些基础实验室合作。如果你刚开始构建智能体，Belting 是一个很好的起点。但当你要构建更生产级的东西时——比如像 RAMP 这样的公司，他们在我们这里运行他们的会计智能体，他们的外部智能体——你需要对计算基元有更多控制，比如如何处理智能体可以访问的不同文件，如何快照和恢复，如何控制网络。也许你想要 GPU。到了那个阶段，你可能会想要一个专门的沙箱提供商来提供这些东西，这就是我们试图扮演的角色。我们对框架没有太多意见——不管它是在云管理智能体中运行，然后你把它连接到 Modal 沙箱，还是你在 Modal 沙箱中运行框架。我们会看看人们最终会如何选择。

Yeah. I mean, we're very excited to partner with Anthropic and some of the other foundation labs. Belting is a great place to start if you're starting out building an agent. But when you get to building something more production grade—like a company like RAMP that runs their accounting agent on us, their external facing agent—you need a lot more control over your compute primitives on things like how do you process different files that the agent has access to, how do you snapshot and restore, how do you control the networking. Maybe you want GPUs. When you get to that point, you kind of want a specialized sandbox provider that gives you those things, and that's the role we are trying to play. We don't really have an opinion on the harness—whether it runs in a cloud-managed agent and you hook it up to Modal sandbox, you run the harness in Modal sandbox. We'll see where people converge with that.

是的。你对元框架有什么看法吗？它只是这些东西之上的另一层。

Yeah. Do you have any opinions on the meta harnesses? It's just another layer on top of these things.

你是指像 OpenPipe 这样的？OpenPipe 是一个。我想 Vercel 有一个，我现在记不起名字了。Fred shot 也有一个。

You mean like OpenPipe? OpenPipe is one. I think Vercel had one which I can't remember the name of right now. Fred shot had one.

嗯，然后最近对我来说是 Databricks 收购了 Omnien。所有这些都像是某种元艺术，类似伪智能体云之类的东西。

Uh and then to me most recently was Data Bricks that had Omnien. All these are sort of meta artists like kind of pseudo agent cloud type things.

我个人没有怎么玩过它们，也没有深入构建。

I personally have not played around with them and built into that.

我的意思是，只要消耗更多基础设施，一切对 Modal 都是利好。这就是为什么我们专注于基础设施层。这是我们的相对优势所在，也是一个难题。

I mean everything's bullish modal as long as it consumes more infra. That's why we're focusing on the infra layer. It's somewhere where our relative competences and also it's a hard problem to solve.

是的。我的意思是，作为一个基础设施从业者，虽然没有你那么投入，但总的来说，这是基础设施领域最激动人心的时刻。实际上有一段时间很无聊，你没法让人们为数据基础设施兴奋起来——比如 Eric 在数据委员会上，大家只是看视频，就像‘看我启动了这么多沙盒’，但没人在乎。

Yeah. I mean I will say like just generally reflecting on I don't know if you if there's other topics on model but like just generally reflecting as an infra person not as intense as you but in that field this has like been the most exciting time in infra like it was boring actually for a while and you couldn't really get people excited about data infrastructure like Eric would get on data council everyone just watched the video and like it's like look at how many sandboxes I spin up and no one gave a crap.

是啊。

Yeah.

而现在，每个人都在乎了。

And like now everyone gives a crap.

确实如此。这是一个非常激动人心的时刻，我认为很大程度上是由所有这些事物所需的规模驱动的。

That's true. It is a very exciting time and I think a lot of that's driven by just the amount of scale all of this stuff needs.

我认为你们的很多举措和产品方向事后看来都很有道理，这是最好的情况，但我自己不一定能想到。我们需要预测，你知道吧？我的意思是，有很多东西你甚至看不到，对吧？比如你们有批处理、语音、多模态。但还有什么在等着我们，或者你认为事情会走向何方？

I think like a lot of your initiatives, a lot of your product directions make sense in retrospect, which is like the best kind, but I wouldn't necessarily have thought about it myself which we need the predictions, you know? I mean, I think there's a lot that you just don't even see, right? like you have the batch, you have the voice, you have the multimodel. But what else is coming up for us or where do you see things going?

是的，总的来说，很明显正在发生巨大的转变。我认为有一件事对人们来说不那么明显，因为语言模型推理被谈论得太多，而且我们也与很多从事药物发现和计算生物学的公司合作——比如 Chai Discovery 世界，那里可能会发生大事。我们与很多机器人公司合作，它们实际上正在将机器人投入实际部署并取得良好成果。有没有气隙 Modal？有没有像本地气隙之类的版本？

Yeah, I mean in general it's clear that there's a huge shift happening. I think one thing that's not as obvious to people because LM inference gets talked about so much and is also we work a lot of companies that are doing things like drug discovery and computational bio like the chai discoveries world big things are probably going to happen there. We work a lot of robotics companies that are actually putting robots in active deployments and getting good results out of them. Is there air gap modal? Is there a version that is like on-prem airgapped whatever?

不，我们只做云。是的。好的。但我的意思是，你所说的就是，因为你专注于原语，而且它们是好的原语，所以你找到了用例等等。实际上，这可能让你一直稍微远离语言模型。

No, we cloud only. Yeah. Okay. But yeah, I mean so what you're saying is like because you're focused on primitives and they're good primitives, you find use cases and all these kinds of things. Actually probably diversifies you a little bit away from LM all the time.

是的，绝对。我们的目标不仅仅是服务于网站上的 L1 拳头市场。音频、生物图像上的螺栓。是的。我的意思是这里有很多。有 QITTS 定制。哦，Chatter Box。还有定制 Whisper。

Yeah, absolutely. Our goal isn't to only serve the L1 fist market just on the website. The audio the bolts on the bio images. Yeah. I mean there's a lot here. There's QITTS customizing. Oh, chatter box. There was a customizing whisper.

是的。是的。这个屏幕让我想起了一个倒下的竞争对手。Replicate。

Yeah. Yeah. This screen reminds me of a fallen competitor. Which Replicate.

嗯。你对发生的事情有什么事后分析？我们一直避开的一件事就是为模型提供 API，因为我认为提供模型 API 最终会服务于一个非常业余的市场，粘性低得多，而我们一直想为那些构建产品、需要更多灵活性而不仅仅是 API 的公司服务。你可以为模型构建 API，这很明显，但你可以把它包装成一个更全功能的后端来运行。

Mhm. What's your postmortem on what happened? This is one thing we've kind of stayed away from is providing an API for models because I think providing model APIs is some of it ends up serving like a really hobbyist market which is much less sticky and we've always wanted to build for companies that are building sort of products and need sort of more flexibility that's not just an API and which you can build an API for a model and this is clearly what it is but you can wrap it into more fully functioning backend that you run.

是的。所以，实际上我们所有的例子都不是‘启动这个模型，给你一个 API 令牌，用它’。它们实际上都是代码。

Yeah. So, actually all of our examples, it's not that spin up this model, here's an API token, use it. They're actually all code.

好的。

Okay.

所以，关键是这是一个示例启动代码。

And so, the point is that this is an example starter code.

是的。但你可以随意调整它。如果你是一家像计算生物学这样的公司，等等。是的。

Yeah. But you can tweak it however you want. And if you're like a company building a product like computational bio, whatnot. Yeah.

我想为听众梳理一下。是的。什么时候它不再是‘哦，你只是一个 API 调用，你只是一个 API 的包装器’，而变成你所说的产品？那个层是什么？显然有更多代码行，但除此之外，人们添加了什么实质内容使其成为更高级的东西？我认为有一点选择效应：很多想要深入那一层的公司可能正在构建更差异化的东西。我认为一个例子是，我们最初与 LM 和朋友们合作，与那些构建自己的后训练框架的公司合作，或者 Ramp 早期在训练自己的分词器，并在 LLaVA 中替换分词器等等。我并不是说那在这种情况下是成功的。

I guess I'm trying to tease out for listeners. Yeah. when does it stop becoming oh you're just an API call and you're just a wrapper on an API to becoming what you call a product right like what is that layer like what like you know obviously more lines of code but like beyond that what is the substance that people add that qualifies it to be something more I think there's a little bit of like a selection effect of like a lot of companies who do want to get deeper into that level are probably building something that's more differentiated and um I think an example is like we with LM and friends originally we worked with companies that were building their own post-training frameworks or they were ramp actually early in the day was training their own tokenizer and like swapping out the tokenizer in lava and whatnot. I'm not saying that was successful in that case.

一个更好的例子是，比如 u，因为嗯，他们不用 Modal 进行训练。是的。但他们用 Modal 进行所有推理，这是因为他们有完全自定义的模型架构，这意味着他们必须在代码层面调整那些不是 API 的东西。有趣的是，我们最近请来了 XAI Grog 团队的 Ethan，他预测视频生成模型的下一层是一个更好的模型或智能体来编排视频模型。语言模型骨干可以使用工具和编写代码，比如我可以从 Grok 制作 6 秒或 10 秒的视频，但实际上我想要 6 分钟的视频，我不会通过常规视频生成来实现。

A better example is like let's say u because um does not use for training Mikey on a pod. Yeah. But they they use modal for all their inference and that's because they have like a custom they have completely custom model architecture and that means that they have to be at the code level and tweak things that are not um you know it's an API. It's interesting as well like we had Ethan most recently on the XAI Grog team make a prediction that actually like the next tier in videog model it's a better model or agent that orchestrates video models. language model backbone that can use tools and write code like yes I can make my six-second video or my 10-second video from grock but actually I want my 6-minute video and I'm not going there through normal video gen.

是的，这很有趣。实际上，我们有 GPU 沙盒，最近看到一些公司在做类似智能体进行视频操作的东西，或者给它 F，那不像你需要给它 Adobe。

Yeah that's interesting I actually so we have GPU sandboxes and recently have seen a few companies doing sort of agents that do video manipulation or give it f like that's not you need to give it Adobe.

是的。我没想到它实际上会是一个视频制作的东西。在我的脑海里，这些东西更倾向于编辑。是的。我显然经常思考这个问题。Luma Luma 智能体是视频制作的一个版本，但你知道，它是一次性的。

Yeah. I hadn't put it together with like it would actually be a video production thing. In my mind, these things were going more towards editing and Yeah. I think about this a lot obviously. Luma Luma agent is a version of this for video production, but you know, it's a one-off.

我本来想听听你对最近新闻中其他一些事情的快速看法，看看你有没有什么有趣的见解。GitPod 很像，你知道，他们处于不同的市场，有点像 CI/CD 市场，但实际上技术上非常令人印象深刻。我不知道你是否真正看过他们。

I was going to get your quick takes on some other stuff that happens in recent news and see if you have anything interesting. GitPod very like somewhat like you know different market they're in like sort of like the CI/CD market but actually technically very impressive. I don't know if you've taken a real look at them.

是的。我们团队有人和 GitPod 团队聊过，他们技术上非常强。

Yeah. We've people on our team have talked to the GitPod team and they're technically very strong.

是的。实际上，我们在 Modal 对 CI 市场也非常看好，因为随着更多智能体、编码智能体的出现，它们会运行更多的 CI，那里的原语可以更好。我认为有很多浪费的 CI。

Yeah. I actually am we're very bullish at modal on the CI market as well because as there's more agents coding agents they're going to run a lot more CI and the primitives there can be much better. I think there's a lot of wasted CI.

是的。

Yeah.

那么，是不是就像我们得筛选一下，在改进智能体的 CI 方面，最高优先级的事情是什么？

So is it just like let's filter what is the highest order bit here in improving CI for agents?

嗯，CI 中有很多时间浪费在准备制品、依赖项之类的事情上。显然构建系统对此有帮助，但如果你有像内存快照和恢复这样的原语，能不能更高效地运行 CI？

Well, there's a lot of wasted time in CI on preparing your artifacts and getting you to basically preparing your dependencies and whatnot. And obviously build systems help with that, but if you have primitives that are like memory snapshot and restore, can you just run CI more efficiently?

哦，好的，有意思。是的，我是说另一种形式的按需算力。

Oh, okay, okay, interesting. Yeah. I mean another form of like on-demand compute.

对，没错。

Yeah. Exactly.

嗯。

Yeah.

它同样需要同一个平台。

It needs the same platform again.

是的。对于不知道的人来说，GitPod 更名为 Owner 了。这事儿还挺大的。我其实在 Cognition 半开玩笑地拉过警报，我说你们应该认真对待这帮人，因为他们基础设施做得非常好。

Yeah. So for those who don't know, GitPod rebranded to Owner. There was this whole thing. I actually sort of semi-sounded the alarm at Cognition. I was like you should take these guys seriously because they're infred very good.

是的。但他们后来加入了 OpenAI，大概我们会看到 Owner 团队推出的 Codex Cloud，我觉得那会非常非常强。对我来说，像这样的团队能够搭建网络和安全边界，让你的智能体各自拥有自己的云，这基本上就是你在做的事情。我只是想做个类比或者找出差异，如果你研究过他们的话。哲学上的区别是什么？我的感觉是，也许他们没有在正确的时间瞄准正确的市场，因为我们也很幸运，智能体式用例正好起飞了，而且更像是沙盒形态的东西，而不是一个设置。

Yeah. But then they joined OpenAI and presumably we'll see Codex Cloud from the Owner team, which I think would be very, very strong. To me, teams like that can set up the networking and the secure boundaries for your agents to have their own cloud each effectively, which is what you're doing kind of. And I'm just trying to draw the analogy or the differences if you have studied them. What is the philosophical difference? My sense is maybe they didn't go after the right market at the right time because we also got lucky with the agentic use case really taking off and dating more of a sandbox-shaped thing than a setting.

沙盒是有效的。沙盒就像是构建时沙盒与运行时沙盒，实际上运行时更好。

Sandboxes work. Is sandbox it's just like build-time sandboxes versus runtime sandboxes, and actually it turned out runtime was better.

对，区别在于运行时沙盒在如何配置镜像、如何挂载存储等方面有不同的配置面。

Right, and the difference there is runtime sandboxes have a different configuration surface of how you configure images, how you attach storage, and...

是的，这很迷人。其他人，比如 Astro，还有 OpenAI，还有 Python 工具生态系统。你仍然看好基于 Python 构建吗？另外最近 Modular 被高通收购了。你有什么看法？

Yeah, it's fascinating. Other people, Astro, also OpenAI, also Python tooling ecosystem. Are you still sort of bullish building on top of Python? Also recently Modular got bought by Qualcomm. Any of your takes there?

是的，我们最初用 Python 作为第一个 SDK 语言，因为那是人们做数据和机器学习用的语言。我现在也有 Go 和 TypeScript 的 SDK，而且我们的运行时完全与语言无关。它是用 Rust 写的，但绝不绑定 Python。我们还没看到……我认为在推理和训练方面，人们仍然非常 Python。有趣的是，在智能体方面，人们更多地使用我们的 TypeScript SDK，因为他们实际上不做任何机器学习。我认为我们不会很快需要超越这些，因为 Python 和 TypeScript 仍然占主导地位。

Yeah, I mean we had Python as our first SDK language because that was the language that people did data and ML in. I actually now have Go and TypeScript SDKs as well, and our runtime is completely language-agnostic. It is in Rust but it's not tied to Python by any means. We haven't seen... I think with inference and training stuff, people are still very Python. And the interesting thing with the agent stuff is people use our TypeScript SDK a lot more because they're not actually doing anything ML. I don't think we'll have to go beyond that super soon because Python and TypeScript are still dominant.

世界上最后两种语言。

The last two languages in the world.

是的。

Yeah.

就这样。

That's it.

嗯，英语和提示工程是……

Well, English and prompting is...

英语和提示工程。我偶尔会和那些试图构建新语言的人聊天。他们甚至说，OpenAI 的董事长 Brett Taylor 也说过，我们需要一种用于语言模型的新语言。但还没人找到，我也一直在找。你知道，Python 和 TypeScript 有大量数据，但它们作为语言本身也非常不完美。那么我的结论是，我认为 Modal 曾经在开发者体验上下了大赌注，而你们已经把团队转向了智能体体验。现在是这样的方向吗？你认为整个公司、独角兽、甚至多个独角兽可以仅仅靠提供更好的智能体体验来建立吗？还需要别的吗？

English and prompting. I occasionally talk to people who try to build new languages. They're like, even Brett Taylor, who is chairman of OpenAI, we need a new language for LMs. So no one has come across one, and I keep looking. You know, Python and TypeScript have a lot of data, but also they are very imperfect as languages themselves. Then my close is I think Modal used to be a big bet on developer experience, and you've pivoted the team to agent experience. Is it like the way now? Do you think entire companies and unicorns, multi-unicorns can be built on just having better agent experience? Do you need something else?

这是我们身份的重要组成部分。这不仅仅是战术层面的智能体如何使用 CLI，还包括启动一个东西有多容易？当你想要启动一个新服务并投入生产时，你的迭代时间是多少？在实践中，这对人们来说非常重要，而且我认为它将继续重要。人们构建东西的速度越来越快，如果你给他们快速完成且没有开销的方法，那么……对我来说，争论点在于你是否需要做一些对开发者体验和智能体体验有根本性不同的事情？你似乎站在它们很相似的一边。

It's a big part of our identity. It's not just the very tactical how does an agent use the CLI, but it's also how easy is it to spin something up? What is your iteration time when you want to spin up a new service and you want to get something going in prod? In practice that matters a lot to people, and I think it will continue to matter. People are building stuff even faster, and if you give them ways to do it quickly and not have overhead, then... I think the debate for me has been do you do anything differently that is very fundamentally different for developer experience versus agent experience? You seem to be on the side of they're like...

我们有一篇关于这个的博客文章。

We have a blog post on that.

相似度大概 0.9 之类的。是的，我的意思是，对我们来说，主要的转变是，正如我所说，我们构建了这个基准 Modal Bench 来观察智能体欠缺的地方，然后实际上如果它们试图获取某个功能，就增加产品的表面，比如也许这应该只是一个 CLI。

Co-sign similarity on like 0.9 or whatever. Yeah, I mean pretty much it's the main shift for us has been as I said we built this benchmark Modal Bench to see where agents are lacking, and actually literally add surface area to a product if they're reaching for something like maybe this should just be a CLI.

是的，它们会幻觉出自己的功能。

Yeah, they hallucinate their own features.

是的，有时候这很有道理。如果它们试图获取这个东西，那就是产品反馈。给它们。然后实际上，我们过去只在 UI 中有日志和指标，现在也把这些都移到 CLI 中，这样它们就能以那种形式访问了。

Yeah, and sometimes it makes sense. If they're reaching for this thing, it's product feedback. Give it to them. And then actually moving, we used to only have logs and metrics in our UI, just moving all those things to CLI as well, so they're accessible in that form.

就这么简单。

Simple as that.

酷。非常感谢。是的，这是一个很好的更新，我能理解为什么你们如此成功。这确实是专注，同时也是非常好的执行。

Cool. Thank you so much. Yeah, this is a great update and I can see why you guys have succeeded so much. It is really focus but also really good execution.

谢谢。我的意思是，我们还有很长的路要走。

Thanks. I mean we have a long way to go.

好的。谢谢。酷。

All right. Thank you. Cool.