Engineering Manager for Your Agents: Building Trust in AI-Generated Code
打开互动全文版(中英对照 + 朗读 + 问答)→Lauren Tan 分享如何把 AI 智能体当作工程团队成员,通过验证建立信任,并实现数百个自动合并 PR 的规模化开发。
Lauren Tan on treating AI agents like engineering reports, using verification to build trust, and scaling to hundreds of auto-merged PRs.
而有了 Grok Bot,我们真的很想从一个全新的视角来看,如果你的智能体更像一个同事,会是什么样子?
And with Grok Bot, we really wanted to take a fresh perspective on what it would look like if your agent was really more like a colleague?
嗯。
Mhm.
于是我开始思考,怎样才能让我的智能体在我睡觉时自动帮我修复 bug?这就是我们早期一些实验的灵感来源,构建一个更持久的机器人。如果你在早期投入很多,我认为工程师在智能体世界中的角色演变是,就像我提到的,你有点像你智能体的工程经理。但我认为,作为工程师,我们现在很大一部分责任也是保持代码库处于一种状态,让产品经理、设计师也能贡献高质量的代码。
And so I started thinking about how would I make it so my agents could just automatically fix bugs for me while I sleep? And so that was the inspiration for some of our earlier experiments with building a more persistent bot. If you invest a lot in having early the way I see engineers' roles evolve in the agent world is, like I mentioned, you're sort of like an engineering manager for your agents. But I think a big part of our responsibility now as engineers is also to keep the code base in a condition where product managers, designers can contribute high-quality code.
好的,那么我假设你能看到我的屏幕。
Okay, so I assume you can see my screen.
是的。嗯,没问题。
Yes. Yeah, we're good.
所以,今天我认为,在我使用智能体编写代码的过程中,最大的主题是:你如何信任它?我相信很多人也有同样的经历。特别是如果你是一位写了很久代码的工程师,你对做好工程有很多意见和学到的教训。当你看到智能体只是即兴发挥、猜测、产生幻觉,自信地声称他们第一百次找到了确凿证据,但实际上并不是真正的问题时,你会失去很多信任。而当你对智能体没有多少信任时,我觉得你真的无法充分利用它们。对我来说,这和管理的类比是相似的。所以如果我是一个团队的工程经理,我团队里有一群工程师,我不信任他们,那么我的操作模式就会是微观管理,对吧?我将不得不花很多时间监督我的下属,检查他们是否做好了工作,是否没有将 bug 发布到生产环境。
So yeah, today I think the big theme for me as I've been using agents to write code, and I'm sure a lot of you have had the same experience as well, is how do you trust it? Especially if you are an engineer that's been writing code for a very long time, you have a lot of opinions and lessons that you've learned about doing good engineering. And when you see agents just winging it and guessing, hallucinating, confidently stating that they found the smoking gun for the hundredth time, but it's actually not the real problem, you lose a lot of trust. And when you don't have much trust in your agents, I feel like you really can't get the most out of them. And for me, the parallel is with management. So if I'm an engineering manager for a team, and I have a team of engineers on my team, and I don't trust them, then the mode of operation I'm going to be in is going to be like micromanagement, right? I'll have to spend a lot of time looking over my reports' shoulders and checking that they're doing their work well, that they're not shipping bugs to production.
所以我画了这张图,因为它不是很科学,但这就是我想象自己和我的智能体使用之旅。所以,快退到大约一年前,当没有人或不是很多人使用智能体编码时,我想你会进入这样一种模式:你与一个或几个智能体紧密相连,你发现自己不断试图理解你的智能体在做什么,你非常非常地在循环中。你在观察每一个输出。你坐在那里提示,你真的无法并行化 beyond that,因为再次,你没有那种信任,对吧?当你甚至不信任一个智能体的输出时,你不可能去 100 个智能体,像生成 100 个智能体。
And so, I drew this chart because it's not a very scientific chart, but this is how I imagine myself and my journey through using agents. So, fast backwards like a year or so, when nobody or not many people were using agents to code, I think you get into this mode where you are very heavily in the loop with one or several, like a handful of agents, and you find yourself just constantly trying to understand what your agents are doing, and you're very very in loop. You're watching every single output. You are sitting there prompting, and you really can't parallelize beyond that because again, you don't have that trust, right? You can't go to 100 agents, like spawn 100 agents when you don't even trust the output of one agent.
所以,在过去 5 个月里,我觉得我真的能够攀升这条信任曲线,现在我已经到了这样的地步:我实际上有——这么说有点吓人,让我听起来像个吹牛的人,但我保证我不是——但我现在实际上让我的智能体自动为我合并 PR,这说起来很疯狂,但我今天醒来,有大约 20 个 PR 已经落地,我只是在 main 分支上审查了它们,它们已经落地了。而且它们很好。那么,我是如何达到这一步的?这基本上就是我今天想谈的。再次,如果你有问题,Colin,随时可以插话。
So, over the past 5 months, I feel like I've really been able to ascend this trust curve, and now I'm at the point where I actually have—this sounds kind of scary to say this, and it makes me sound like a slot artist, but I promise I'm not—but I actually have my agents now auto-merging PRs for me, which is like a wild thing to say, but I woke up today, and there were like 20 PRs landed, and I just reviewed them on main, like they were already landed. And they were good. So, how did I get to that point? It's basically what I wanted to talk about today. And again, feel free to jump in if you have questions, Colin.
哦,当然,我得展示这张图。
Oh yeah, of course, I got to show this chart.
哪里——哦,不要信任——有人请求控制我的电脑。可能不会那样做。但是的,所以这张图,我分享这张图不是为了炫耀,而是为了展示旅程。所以你可以看到曲线有点与我在 Cursor 能够落地的贡献成反比。所以,我 5 个月前加入。5 个月前,我的第一个月,我不是很高效,因为我在学习代码库。不知道到底发生了什么。随着我对我的智能体越来越有信心,我真的能够提高我的生产力。再次,上个月,我发布了 1,000 个 PR,这很荒谬。然后这个月,才 12 号,我已经有将近 800 个 PR 落地。所以,速度肯定很高。我分享很多人肯定会质疑这些代码有多少实际上是好的。我认为这绝对值得质疑。但是的,我认为如果你设置好你的智能体,你肯定可以达到非常相似的水平。所以,我将谈论我们如何做到这一点。
Where—oh, do not trust to—someone requested to control my computer. Probably won't do that. But yeah, so this chart, I'm sharing this chart not to flex, but to show the journey. So you can see the curve sort of inversely matches the contributions I've been able to land at Cursor. So, I joined 5 months ago. And 5 months ago, my first month, I was not very productive because I was learning the code base. Didn't know what the heck was going on. And as I got more confident in my agents, I've really been able to ramp up my productivity. And again, last month, I shipped 1,000 PRs, which is ridiculous. And then this month, we're only on the 12th, I'm already at almost 800 PRs landed. So, the velocity is definitely high. And I share a lot of you will definitely be questioning how much of this code is actually good. And I think that's definitely fair to question. But yeah, I think if you set up your agents well, you can definitely get to a very similar level. And so, I'm going to talk about how we do that.
所以,对我来说,我也很好奇 Colin 的经验,但对我来说,我认为当你与智能体合作时,你工具箱中最重要的技能是验证。所谓验证,我指的是智能体能够实际运行代码,或获取 CPU 跟踪或堆快照,或打开 iOS 模拟器,等等,无论你的应用如何暴露给用户,它都能做同样的事情,并真正运行和测试,验证它能工作。因为这才是真正闭环的事情。它不保证你的智能体写出好的代码,但它至少允许它们写出正确的代码,这是能够信任你的智能体的一个非常大的进步。
So, for me, I think I'm curious about Colin's experience as well, but for me, I think the most important skill that you should have in your toolbox when you work with agents is verification. And by verification I mean the ability for an agent to actually run the code or take CPU traces or heap snapshots or open an iOS simulator, whatever, however your application is exposed to users it can do the same thing and run it for real and actually test and verify it'll work. Because that's the thing that really closes the loop. It doesn't guarantee your agent writes good code but it allows them to at least write correct code which is a really big step forward for being able to trust your agent.
我可以分享一个我们在 Cursor 内部的例子。哎呀。让我打开这个。让我把这个放大。让我全屏。好了。所以对于 Cursor 的智能体窗口,这实际上是一个有趣的故事,但当我 5 个月前加入 Cursor 时,我本应加入一个不同的团队。我本应加入云智能体团队。但后来因为我在 React 方面有很多经验,而智能体窗口是一个 React 应用,我被要求基本上帮助智能体窗口的工作。但当时并没有很多技能可以帮助我。所以我发现自己,好吧,Agent Swindows 将在一周内发布。好吧,我们有一个非常紧的截止日期。然后——我只是坐在那里,好吧,我要打开 Chrome DevTools,取一个跟踪,自己看看,试着理解这个火焰图。记住,我才第一周。所以,我不知道我在看什么,不知道什么——我的意思是,我有一些想法,但代码库对我来说完全是新的。
I can share one example that we have within Cursor. Oops. Let me open this. Let me make this bigger. Let me go full screen. There you go. So for Cursor's agent window, this is actually an interesting story but when I joined Cursor five months ago, I was supposed to join a different team. I was supposed to join the cloud agents team. But then since I have a lot of experience working on React and the agents window is a React application, I was asked to basically help out with the agent window work. But there wasn't really a lot of skills to help me. So, I just found myself that okay, Agent Swindows is going to launch in like a week. All right, we have a really tight deadline. And there was—I was just sitting there like okay, I'm going to open up the Chrome DevTools and just take a trace, look at it myself, and try to make sense of this flame graph. And keep in mind, I was just in my first week. So, I had no idea what I was looking at, no idea what—I mean, I had some idea, but the code base was completely fresh to me.
然后我意识到,我的智能体也完全不知道。我会截个图,试着下载 trace 或者发给智能体,说“嗯,大概长这样”。然后它会很自信地说就是某个东西。接着我试着去修,结果发现根本不是那个东西。所以这个过程非常非常慢。如果你自己做过任何性能优化的工作,或者哪怕只是用智能体开发过、但没有验证技能,那你就是验证者。你就是瓶颈。你让智能体做一件事,它跑去写一些代码。然后你打开本地的开发构建,开始说“哦,这不行”。然后你得复制粘贴截图或者控制台报错之类的。接着你的智能体慢慢处理这些信息,试着理解并修复问题。但你始终在循环里,始终是瓶颈。所以根本没法并行。
And I realized my agent had no idea either. I would take a screenshot to try to download a trace or send it to the agent and be like, "Yeah, it kind of looks like this." And it would confidently state it's this thing. Then I'd try to fix that, and it turns out that's not the actual thing. So this was a very, very slow process. If you've ever done any performance work yourself, or even just development with an agent where you don't have a verification skill, you are the verifier. You're the bottleneck. You tell your agent to do something, and then it goes off and writes some code. Then you open up your local dev build, and then you start to say, "Oh, it doesn't work." Then you've got to copy-paste screenshots or console errors or whatever. And then your agent slowly works with that and tries to understand it and fix the thing. But then you're constantly just in the loop and being a bottleneck. So there's really no way to parallelize.
所以 Control Glass 是我为 Cursor 构建的最早的技能之一。顺便说一下,Glass 是我们内部对 Agent Windows 的代号,但其实就是 Cursor 吧。这个技能的代码本身非常有意思。你的智能体可以很轻松地帮你做一个。如果你在构建 Electron 应用、Web 应用甚至 iOS 应用,你可以教你的智能体如何使用 Chrome DevTools 协议,或者 Apple 也有一些工具可以用来运行模拟器、抓取 trace 以及进行程序化控制。所以这非常有用。
So Control Glass was one of the first skills I built for Cursor. And Glass, by the way, is the code name for Agent Windows that we use internally, but it's just Cursor, I guess. So this skill, the code itself is super interesting. Your agent can very easily make one for you. If you're building an Electron app or a web app or even iOS applications, you can teach your agent how to use the Chrome DevTools protocol, or Apple has some utilities as well for running the simulator and taking traces and controlling things programmatically. So that's really useful.
但我真正想聊的是这个东西。readme 在哪?这个技能附带了一个非常独特的文件,叫 feature map(功能地图)。故事是这样的:我构建了这个技能,现在智能体能够真正运行智能体窗口、抓取 trace 之类的了。但它完全不知道智能体窗口是什么。所以有人会说“哦,左边栏有点卡”,或者“右边的 PR 标签页不工作”,然后智能体就到处乱撞。它会花大量时间翻代码,想搞清楚这个功能在哪、我在 UI 上怎么才能到那里,这让它基本上完全没用。我在本地运行这个技能,它会启动一个开发构建,但然后就一直在空转——它试着点这里,却不知道怎么到达那些地方。体验非常糟糕。
But one thing I actually want to talk about is this thing. Where is the readme? So this skill comes with this very unique file called a feature map. So the story is, I built this skill, and now the agent was able to actually run the agent window and take traces and whatnot. But it had no idea what the agent window was. So someone would say, "Oh, the left sidebar is laggy," or "The PR tab on the right side is not working," and the agent would just be flailing around. It would spend a lot of time trying to look up the code and figure out where this feature is, how do I actually get to it on the UI, which made it basically completely useless. I would run this skill locally and it would spawn a dev build, but then it would just be churning — it would try to click here, it wouldn't know how to get to things. It was just an awful experience.
所以是谁在我屏幕上放箭头?这个 feature map 真的非常有用,因为它教智能体如何到达你所有的功能。在我做的插件 Pstack 里——你在 Google 上搜 Pstack Cursor 就能找到——那个插件里有一个 create verification 技能,它能帮你为自己搭建类似的东西,包括 feature map。它会真正去探索代码,构建出这个初始的 feature map,告诉你的智能体如何到达你所有的不同功能。这非常强大,因为现在有了这些用户反馈,你甚至可以映射一个模糊的报告,甚至一张截图。
So who was putting arrows on my screen? Yeah, this feature map has been really useful because it teaches the agent how to get to all of the features that you have. And in Pstack, the plugin that I've made — if you search for Pstack Cursor on Google, you'll find it — there is a create verification skill in that plugin where it actually helps you set up something like this for yourself, including the feature map. So it will actually explore the code and build up this initial feature map that tells your agent how to get to all of the different features that you have. And this is extremely powerful because now that you have these user reports that come in, you can actually map even a vague report or even a screenshot.
我们在 Cursor 内部就有这样一个 Slack 频道,很多人给我们反馈智能体窗口、rock bar 之类的问题。很多时候报告非常糟糕、质量很低。经常是我们收到一张截图,然后有人就发“???”,意思是这是什么?没有这个的话,你的智能体只会说“我完全不知道”。但有了这样的 feature map,它就有更多的上下文和理解,知道如何真正导航、如何到达所有不同的功能。比如侧边栏——侧边栏是什么?里面有哪些不同的子功能。从用户的角度看,怎么到达那里,所有的键盘快捷键。甚至 DOM 元素,或者你通过 CDP 选择元素时用的属性,全都在里面。所以再说一次,这对智能体来说真的非常非常强大。
We have this internally at Cursor where we have a Slack channel with lots of people giving us feedback on the agent window and the rock bar and whatnot. And oftentimes the report is very bad, very low quality. Very often we get a screenshot and then someone just says "???", like, what is this? And without this, your agent says, "I have no clue." But with a feature map like this, it has a lot more context and understanding of how to actually navigate, how to get to all the different features. For example, the sidebar — what is the sidebar? All the different sub-features that are present in it. From the user's point of view, here's how to get to it, all the different keyboard shortcuts. Even the DOM elements, or the attributes that you use for selecting things through the CDP, are all there. So again, this is really, really powerful for agents.
Pstack 不仅附带那个 create verification 技能,还有一个 maintain verification 技能,这样你可以让它保持最新。
And Pstack ships that create verification skill, but also a maintain verification skill, so you can keep this up-to-date.
酷。我正想问你那是怎么创建的。你介意再多讲讲在 Pstack 语境下那个过程吗?也许顺便给不熟悉的人介绍一下 Pstack 是什么。
Cool. Yeah, I was just going to ask how you created that. So do you mind sharing a little bit more about that process in the context of Pstack? And maybe just what Pstack is for the folks who aren't familiar.
好,Pstack 挺有意思的,因为首先这个名字有点傻。Pstack 里的 P 是 potato——土豆袋。因为有个挺有名的人,Garry Tan,Y Combinator 的 CEO,他做了一个叫 Gstack 的插件。Garry stack。有趣的是我们同姓。我们没有亲戚关系。但我觉得调侃一下 Garry 会挺好玩,把 Pstack 做成我版本的他的插件,不过按我自己的一套工程实践来定制。
Yeah, so Pstack is pretty interesting because, well, first of all, the name is kind of goofy. The P in Pstack is like potato — potato sack. Because there is a pretty famous person, Garry Tan, who is the CEO of Y Combinator, and he's come up with this plugin called Gstack. Garry stack. And funnily enough, we share the last name. We have no relation. But I thought it'd be funny to poke fun at Garry and make Pstack my version of his plugin, but kind of tailor it to my own set of engineering practices.
但说实话,我从来没打算做 Pstack。它只是从一堆技能开始的。我先做了那个 Control Glass 技能。然后我又做了一个叫 How 的技能,也是通过观察智能体注意到的。在我试着往上爬的早期,我超级亲力亲为。我基本上是在极度地挑智能体的刺。我会告诉它“这个功能坏了。这是 bug 报告。为什么它不工作?”而很多时候智能体会很自信地说“哦,肯定是这个,对吧?肯定是这个东西。”然后我注意到,当我查看实际的工具调用时,它根本没有在读应该受影响的代码。这让我极度怀疑。那一刻我就想,我不能再信任这个智能体了,因为它完全在幻觉。我觉得很容易积累起这种不信任,然后感到无助,好像你不知道怎么帮你的智能体成功。
But I honestly never set out to build Pstack. It just started with a bunch of skills. I started out with that Control Glass skill. And then I started with another skill called How, which I also noticed through observing agents. In the early days of me trying to climb this ladder, I was super in the loop. I was basically nitpicking my agents to an extreme degree. I would tell it, "This feature has stopped working. Here's a bug report. Why isn't it working?" And very often the agent would just confidently state, "Oh, it has to be this, right? It has to be this thing." And I noticed, when I looked at the actual tool calls, it wasn't actually reading the code that should be affected. And that made me extremely suspicious. At that point I was just like, I can't trust this agent anymore because it's just completely hallucinating. And I think it's very easy to build up that distrust and kind of feel helpless, like you don't know how to help your agents succeed.
但再次强调,我认为管理类比非常有帮助,因为想象一下,如果你是工程团队的管理者,团队里有一位非常擅长编码的工程师,但完全没有业务联系,你刚雇佣他们,他们大概 5 秒前才入职。那么你如何教这个人变得高效?你通过技能来做到。技能只是,你知道,就是 markdown,对吧?但它编码了大量信息、指令,你可以真正从智能体中提取大量智能,嗯,Twitter 上有些人称之为把智能体拉到不同的潜在空间。这有点像一种花哨的说法,就是说因为 LLM 是预测下一个词,当你一开始给它一些高质量的 token,它就可以在更智能的更高空间上进行模式匹配。所以这是一个非常有趣的模型。但是的,我构建 peace 是非常非常渐进的。所以开始只是真正观察智能体,所有失败的不同失败模式。每次我看到那个,我就想,好吧,我就把它变成一个技能。好了,别再幻觉了。真的去搜索查找代码。使用大量子智能体。是的,别再猜测了。
But again, I think the management analogy is super helpful because imagine if you were a manager of an engineering team and you had an engineer in your team who was a really good coder, no business contacts whatsoever, you know, they just you just hired them and they onboarded, you know, like 5 seconds ago. And so how do you actually teach that person to be effective? So how you do that is through skill. Skill being just, you know, it's just markdown, right? But you know, it codes a lot of information, instructions, a lot of you can really draw out a lot of intelligence from an agent by well, some people on Twitter call it like, you know, pull the agent to a different latent space. Which is kind of like a fancy way of just saying like since LLMs are sort of like they predict the next token. When you give it some high-quality tokens to begin with then, you know, it can kind of pattern match on like a higher space that's, you know, smarter. So that's like a very interesting model there. But yeah, I built peace that very, very incrementally. So started with just really observing how agents, you know, all the failed different failure modes of that agents were having. And every time I saw that, I was just like, okay, I'm just going to make that a skill. All right, like stop hallucinating. Actually go and search up look up the code. Use a lot of sub agents. And yeah, stop guessing.
是的,这说得通。我有个后续问题,既为我自己也为聊天室里的很多人。我想它分两部分。一是你如何维护这些技能?产品会随时间变化。显然有很多人在代码库上发布代码。那么这些技能如何维护?其次,你如何知道你的验证足够好?比如你信任你构建的验证循环,当它们完成时你会信任它们。
Yeah, that makes sense. One follow question here both for myself and for a bunch of people in the chat. So I guess it's two parts. So one is like how do you maintain these skills? So like the product changes over time. Obviously there's a lot of people who are shipping against the code base. So how do these skills get maintained? And then second to that is like how do you know when your verification is good enough? Like in you know you can trust that the verification loops that you've built are going to I guess you trust that that would put when they're done.
是的,也许我会谈谈,我想我有些真正怀疑的东西。也许我先从这个开始。那么我如何维护这些技能?如果你不熟悉这个概念,eval 本质上是一种方式,我想我的心理模型是它就像智能体的单元测试。你实际上可以制作自己的 eval。你不需要特殊的框架。你可以构建一个,取决于你想有多科学和多严谨。我的屏幕是红色的。
Yeah, maybe I'll talk about I think I have something I really doubt. Maybe I'll start with this one first. So like how do I maintain these skills? So if you're not familiar with this concept and eval is essentially like a way to I think the mental model I have is like it's like a unit test for an agent. And you can actually make your own evals. You don't need like a special framework for them. You can build you can you can build one depending on like you know how scientific and how rigorous you want to be. My screen is red.
是的,有一个小按钮。抱歉。
Yeah, there's a little button. Sorry.
你是想禁用绘图还是什么?我看不到我的屏幕。
Are you trying to disable the drawing or something? I can't see my screen.
是的,抱歉。如果你们不在屏幕上绘图那就太好了,但有一个小按钮在,是的,小下拉菜单。我喜欢清除。是的。
Yeah, sorry. If you guys could not draw on the screen that would be great but there's a little button in the Yeah, the the little drop down. I like clear. Yeah.
好的,是的。
Okay, yeah.
你搞定了。完美。继续。
You got it. Perfect. Continue.
是的,所以 eval 基本上是对技能进行单元测试的一种方式。实际上在 pstack 中,我们在土豆模式下发布,如果你搜索的话有一个叫 eval playbook 的剧本。它实际上相当严谨。但本质上我做的是生成许多不同的子智能体。我让我的主协调智能体为我想让技能做的事情制定一个评分标准。然后它生成所有这些子智能体,并为它们创建单独的目录,这些目录被巧妙地命名,以免让子智能体知道它正在被评估,因为智能体实际上能看出来,当它们看出来时,它们会改变行为。但它做了一堆类似的事情,本质上是的,测试我正在制作或更改的技能是否真的在做我认为它做的事情。cursor 的一个非常好的地方是我们支持许多不同的模型。所以你可以实际上在各种不同的模型上评估你的技能。你知道,了解它在不同矩阵上的表现如何。特别是对于你使用的模型。所以我经常这样做。每次我修改一个技能,我都会运行其中一个。比如 eval playbook。并确保你知道,它实际上导致我想要的结果。但我要说维护技能实际上相当难。我认为它需要很多品味和观察。所以你需要非常擅长做后座司机。你明白我的意思吗?就像如果你做结对,如果你曾经做过结对编程,你看着同事编码,你就像你可能会做得更好。你知道,你可以做你知道,你喜欢为什么你不这样做,对吧?你问同事很多问题。这里有点类似。你不想只是一个被动的观察者智能体。你想在初始阶段非常处于驾驶座,当你建立自己的技能集时。你知道,显然你可以使用像 pstack 这样的东西,但如果你在建立自己的技能集,我认为打开所有工具调用,阅读代码,阅读所有智能体行为及其思考块,是看到它们失败的好方法。对吧?比如它们在哪里被完成?然后你可以为此构建一个技能。然后对于验证,你如何信任它,我认为这也是一个非常相似的迭代循环,你知道,就像我实际上对验证技能做了同样的过程,我实际上得到所以,关于 eval 的一个有趣的事情是你可以爬山。意思是你的 eval 可以产生一个分数,对吧?你可以让你的协调者产生一个分数,但你也可以有一个不同模型的评判智能体来交叉参考,确保第一个模型没有偏见,对吧?评判所有运行事物的子智能体的模型。但你也可以爬山。所以,意思是你可以使用 cursor 中的 {slash} loop,你可以说,“好的,继续在这个 eval 上循环,直到一切都是 10 分(满分 10 分)。”举个例子。我对控制技能做了基本上相同的方法。所以,我有点非常放手,实际上。所以,你知道,我有点构建了我那样构建那个技能,比如那个技能中的 CLI。随着时间的推移,它变得非常好。但是的,一开始绝对不是超级顺利。它需要大量迭代。我认为这里有一个类比,嗯,我在绘图上的另一个幻灯片中做了这个类比。但我认为就像你知道,作为一名工程师,你现在更像你,也许像管理者,或者我喜欢的类比是,你像餐厅里的厨师。
Yes, so evals are a way to unit test your skills basically. And actually in pstack we ship under poteto-mode there's a playbook if you search for it called eval playbook. And it's uh It's like not it's actually pretty pretty rigorous the way it's done. But essentially what I do is I spawn a lot of different sub agents. I have like my main coordinator agent come up with a rubric for what I want the skill to do. And then it spawns all these sub agents and it creates individual directories for them which are cleverly named to not let the sub agent know that it's being evaluated because agents can actually tell and when they do they change their behavior. But it does a bunch of stuff like that to essentially yeah, like test whether or not the skill I'm making or changing is actually doing what I think it does. And one of the really nice things about cursor is that we are we have we support so many different models. So you can actually eval your skill across all sorts of different models. And you know, get a sense of how well it performs across that different matrix. Especially for the models that you use. So I do this a lot. Every time I modify a skill, I will run one of these. Like the eval playbook. And make sure that you know, it's actually leading to a result I want. But I will say like maintaining skills is actually pretty hard. It requires I think a lot of taste and observation. So you kind of need to be very good at being a backseat driver. You know what I mean? Like if you do pair if you've ever done pair programming for example and you watch a co-worker code and you just like you you could probably do this better. You know, you could do you know, you like why did you not do this, right? You you ask a lot of questions to your co-worker. And it's kind of a similar thing here. You like you don't want to just be a passive observer agent. You want to be very in the driver seat in the initial stages when you're building up your own set of skills. You know, obviously you can use something like pstack, but if you're building your own set of skills, it's very I think, you know, opening up the all the tool calls and like reading the code and reading all the agent behavior and their thinking blocks is a really great way to see where they fail. Right? Like what what you know, where are they being done? And then you can go and build a skill for that. And then with verification, how you trust it is it's I think it's also a very similar iteration loop where, you know, like I actually did the same process for verifying the verification skill where I actually get So, one thing that's interesting about E valves is that you can sort of hill climb them. Meaning that your E valve can produce a score, right? A score that you can get your coordinator to produce, but also you can have a judge agent of a different model to kind of cross-reference and make sure that the first model is not being biased, right? The model that's judging all of the sub agents that are running the thing. But you can also like hill climb. So, meaning that you can you can use like {slash} loop in cursor and you can say, "Okay, keep looping on this E valve, right? Until everything is 10 out of 10." As an example. And I did the same the basically the same approach with the control skill. And so, I kind of it was very It was very hands-off, actually. So, you know, I kind of built I built that skill that way, like the CLI in that skill. And over time it's gotten really good. But yeah, it was definitely not super smooth at the beginning. It required a lot of iteration. And I think there's an analogy here for me, which is Well, I make this analogy later in a different slide on my drawing here. But I think of it like you know, as a as a engineer now, you're sort of more like you like maybe a manager or the analogy I like is like you're like a chef in a restaurant.
你知道,你是主厨。你不再自己烹饪所有食物了。你有一个厨师团队,对吧?你有流水线厨师,你有副主厨,你有所有这些不同的岗位。而你的职责是真正营造环境。你负责布置厨房。你负责给不同的人分配任务。所以是的,这是一种非常有趣的工作方式。但没错,我就是这样构建这些验证技能的。
You know, you're the head chef. You're not cooking all the food yourself anymore. You have a team of cooks, right? You have line cooks, you have a sous chef, you have all these different stations. And it's your job to really set the environment. You're in charge of setting up the kitchen. You're in charge of giving tasks to different people. So yeah, it's a very interesting way of working. But yeah, that's how I basically built these verification skills.
是的,就那一点,我想再深入一层。那么,假设我们想为某件事构建一个评估或技能,并希望它能自行改进,我想这就是你的建议。你是在像工作树那样隔离的环境中,用子智能体和审查智能体等来做吗?它是在某种云托管环境中发生的吗?更实际的步骤是什么?如果我想去做这件事,为某件事建立一个验证系统,我会怎么做,或者从哪里开始?
Yeah, just one thought there and I could go try to go one layer deeper. So, let's say we wanted to build an eval or a skill for something and we wanted to kind of get better on its own, which is what I think you're suggesting. Are you doing that in like a work tree kind of isolated with like the sub agents and then the reviewer agents and all that? Is it happening like in some type of cloud-hosted environment? Like what are the more practical steps? If I wanted to go do this and set up a verification system for something, what would I do or where would I start?
我认为最好的开始方式是本地,因为你可以观察。你绝对可以观察你的智能体在做什么。所以,如果你在为自己构建验证技能,我绝对会从本地开始,让你的智能体启动应用程序,无论是 CLI 还是桌面应用或其他什么。这样你就可以实际观察,对吧?你可以看到智能体如何与应用程序交互。你可以看到它如何调用不同的 API 来与应用程序交互。但对我个人而言,我基本上一直全力投入,大部分精力都放在云智能体上,因为它们非常强大。而 Cursor 真正强大的地方其实是云智能体,如果你花一点时间设置环境,这些控制技能、这些验证技能会带来巨大的红利,因为它不仅仅是让你作为单个工程师变得更好,它实际上提升了整个团队甚至整个公司的水平,因为你可以开始思考云智能体,开始思考自动化,自动完成一些事情,比如——我可能稍后会详细讲,但我先提一下——例如,我经常谈到我们有一个叫 Benny 的智能体,它接收我们收到的所有错误报告,然后自动在云中运行,打开一个云——它是桌面版,在自己的计算机上运行 Cursor,并使用相同的控制技能与应用程序交互,尝试复现错误或用户报告,对吧?这非常强大,因为我可以立即从中获得大量信息。就像这个例子中,你可以看到 Benny 实际上复现了错误,但它在主分支上已经修复了。所以,它实际上确认我们已经修复了这个问题,我只需要发布另一个 Cursor 构建版本。所以,这是巨大的信息,我不必去和智能体坐在一起花一个小时试图弄清楚:这是修复了吗?还是没修复?所以,你可以最大化节省时间,但我团队中的每个人都能从中受益。公司里的每个人都能从中受益。所以,绝对认为使用云智能体是非常强大的。但没错,这是一个旅程。你必须先信任它,对吧?在达到这个阶段之前,这又回到了我在这里说的,这非常困难——几乎不可能,我绝对鼓励你不要试图跳跃——如果你还在这个区域,你不想跳到‘我现在要生成一百或一千个云智能体’,因为你只会浪费大量 token。而且会非常昂贵。
I think that the best way to start is local because you can observe. You can definitely observe what your agents are doing. So, if you're building a verification skill for yourself, I would definitely start local and just have your agent bring up the application, whether it's a CLI or a desktop app or whatever. And so, you can actually observe, right? You can see how the agent is interacting with the application. You can see how it calls the different APIs that allow it to interact with the application. But for me personally, I have basically been all in, mostly all in, on cloud agents because they're extremely powerful. And the really powerful thing about Cursor is the cloud agents actually, where if you spend a little bit of time setting up your environment, these control skills, these verification skills pay a huge amount of dividend because it's not just something that makes you as a single engineer better, it actually levels up your whole team and even your whole company because you can actually start thinking about cloud agents and start thinking about automations that automatically do things like—I'll probably get—I kind of talk about this a bit later, but I'll just kind of get into it—where, for example, I talk a lot about this agent we have called Benny, right, who takes all of the bug reports that we get and it automatically goes off in the cloud, opens up a cloud—it's desktop, it runs Cursor in its own computer, and it uses the same control skills to interact with the application and try to reproduce the bug or the user report, right? And this is so powerful because at once I can immediately get so much information from this automatically. Like here in this example, you can see that Benny actually reproduced the bug, but it's already fixed on main. So, it actually confirms that we fixed this problem already, and all I need to do is just release another build of Cursor. So, that's like huge information there that I didn't have to go off and sit with an agent and spend an hour trying to figure out: Is this fixed? Is this not fixed? So, you can max so much time, but everybody on my team benefits from this. Everybody at the company benefits from this. So, definitely think that keeping these using cloud agents is super powerful. But yeah, it's like a journey. You have to trust it first, right? Before you get to this point, and that goes back to what I was saying here, where it's very hard—it's almost impossible, and I would definitely encourage you not to try to jump from—you know, like, if you're still in this zone, you don't want to jump to like, 'I'm going to spawn a hundred or a thousand of cloud agents right now,' because you're just going to waste a lot of tokens. And it's going to be extremely expensive.
是的,那么到目前为止总结一下,基本上如果我们想走你走过的旅程,那就是从验证开始,构建一些技能和一些方法来确定智能体至少产生正确的代码,就像你说的,代码好不好可能是另一个问题,但从技术上讲,它通过查看堆栈跟踪、查看应用中的实际行为等来解决问题。然后一旦我们在本地信任它,我们就可以开始考虑扩展到云中,运行更多智能体,它们自己捕捉信号,对吧?所以,无论是收到的错误报告还是其他什么,它们可以去接手并解决问题,然后给我们返回一个 PR。然后也许最后一步是自动合并 PR,这是你目前所处的阶段。
Yeah, so just to kind of recap so far, basically if we wanted to go on the journey that you've kind of gone on, it would be just start with verification, building some skills and some ways of determining that the agents are producing at least like correct code, whether like you said, whether it's good code or not is maybe a separate question, but like it's technically solving the problem by looking at stack traces, looking at the actual behavior in the app, and so on. And then once we trust it locally, then we can start to think about scaling into the cloud and running more agents that are picking up signals, I guess, on their own, right? So, whether that's like a bug report that comes in or something, they can go and pick it up and solve the problem and give us back a PR. And then maybe the last step is like auto-merging the PRs, which is where you're at.
也许不是每个人都在这个阶段。
Maybe not where everyone is at.
然后在主分支上对它们进行面试,但这样大致正确吗?
And then interviewing them on main, but is that about right?
是的,没错。我想是的,这就是我画这条曲线的原因,对吧?因为这基本上描述了我的旅程,你知道,当我开始时几乎只能使用几个智能体,我只是观察每一件事。我认为从这里到那里真的没有捷径,因为这真的关乎你对智能体的个人信任程度。对吧?显然,作为一名工程师,你不想把垃圾代码直接扔进生产环境。那么,你如何真正建立这种信任?这需要很多品味和判断力。但你知道,像 Pstack 这样的插件绝对能帮助你更快上手。所以,我想这就像如果你信任我并信任 Pstack,那么推而广之,你也许可以信任你的智能体。但如果你不信任我——我绝对不鼓励人们盲目信任我——你知道,如果你建立自己的技能集,你显然可以看看 Pstack 并分叉它,使其成为你自己的,改进技能,绝对鼓励那样做。但对我来说,这真的全是关于——它总是回到信任。你知道,我们在这个聊天中的每个人都有不同的工程标准。在我们的代码库中,有不同的事情对我们很重要。当你能够将所有这些编码到技能中,并且你可以验证你的智能体确实在执行它们时,那让你真正能够攀升这条曲线。并开始自动化事情。还有另一件事我想谈谈,如果有——
Yeah, exactly. I think yeah, that's why I drew this curve, right? Because this basically describes my journey of, you know, when I started barely could use a couple agents, and I was just observing every single thing. I think there's really no shortcut for going from here to there, because this is really about your personal level of trust in the agents. Right? Obviously, as an engineer, you don't want to just slop code into production. So, how do you actually build up that trust? Takes a lot of, I guess, taste and judgment. But you know, like I think plugins like Pstack definitely kind of help you get up to speed much quicker. And so, I guess it's like if you trust me and you trust Pstack, then by extension, you can maybe trust your agents. But if you don't trust me—and I definitely would not encourage people to blindly trust me—you know, if you build up your own set of skills that you can obviously take a look at Pstack and kind of fork it, make it your own, improve the skills, definitely encourage that. But for me, it's really all about—it just keeps coming back to trust. You know, every one of us here in this chat have a different standard for engineering. And there are different things that are important for us in our codebase. And when you are able to encode all of that into skills and you can verify that your agents actually doing them, that allows you to really kind of ascend this curve. And start automating things. There's another piece I wanted to talk about if there's—
是的,说吧。
Yeah, go for it.
我会在过程中回答更多问题。是的,我觉得这还有第三部分我还没谈到,挺有意思的,就是重构和重写。我觉得业界最有争议的话题之一就是该不该重写你的应用。因为工程师很容易这样,尤其是当你加入一家公司,看到代码库时,你会想:“天哪,这太烂了。谁写的这代码?太糟糕了。我想全部重写。”这是一种非常普遍的倾向,很多人,在智能体出现之前,甚至可以说现在,肯定会劝你不要重写。但我其实想论证一下为什么你可能要考虑重写。因为我觉得这真的取决于情况。我认为棕地应用其实处境不错,尤其是如果它们已经搭建得很好。最近我跟一些人聊过,但我只是在观察。我注意到一个相似之处,就是很多大科技公司的问题现在成了所有人的问题。大科技公司的问题,比如我在Meta工作时,我们有一个巨大的单一代码库。我们有,我不知道,成千上万的工程师在键盘上敲打、发布代码。Meta有很多非常优秀的工程师,但我会说,你会惊讶于代码质量其实并不那么好。所以我经常开玩笑说,在AI懒惰之前,我们有人的懒惰。所以我认为很多大科技的基础设施,比如Meta或谷歌的,真正的大科技公司其实是为那种情况设计的,你要照顾团队里能力最差的工程师,对吧?你构建框架、构建约定、构建护栏、限制凭证,这样你的实习生就不会清空生产数据库。如果你已经有那种级别的基础设施,我认为你的智能体其实已经可以做得非常扎实。因为护栏已经到位,智能体不会造成混乱,或者不会在你的代码库里造成太大混乱。而且你总是可以添加更多护栏。但我认为绿地应用,尤其是全新的应用,在我看来是最大的风险。同时也是最大的机会。因为如果你用氛围编程做一个项目或原型,就像我们为Grok Bot做的那样,Grok Bot是很快搭建起来的。如果你还没听说过Grok Bot,那是我们昨天刚发布的新应用。真的很酷。它让你可以编排和创建拥有自己身份的独立智能体,你可以编排它们。超级酷。一定要去看看。但没错,那是一个非常绿地的应用,就像大多数原型一样。所以它是用氛围编程很快做出来的。人类根本没读代码。我最近发了一条推文,说了些关于有机架构的话。也许我会找到它。但想法是,当你有一个完全用氛围编程做的应用时,你基本上没有任何护栏。所以你的智能体,当你给它们一个任务时,它们会用最方便的方法解决。随着时间的推移,你会陷入这种情况:代码库失控,因为你不理解它。你的智能体大概在某种程度上理解它,但它们构建的东西是为走捷径而优化的。你会受苦,那个应用会有很多问题。所以我认为用非常强的约束来启动你的代码库是非常必要的。因为当你有一个可以信任的代码库,当你有真正帮助智能体写好代码的护栏时,你就可以进入曲线的这个部分:我今天醒来,我的智能体合并了20个PR。那是因为我投入了大量时间,我昨天算了一下,超过600个PR,当时我把整个Grok Bot重构成我一直在构建的新架构。是的,我已经到了真的不再看代码的地步。我说这话不只是为了卖你token,而是因为达到那个地步需要很多工作。我花了很多token才让代码库达到这个程度,让我不再需要看它。但我非常兴奋,因为潜力不仅惠及我,还惠及所有为Grok Bot做贡献的人。它还让设计师、产品经理甚至GTM人员能够为Grok Bot添加功能。我不必担心,不必半夜醒来担心有人刚合并了一个性能回归。我有很多约束,在Grok Bot里写代码时CI其实很烦人,但智能体吸收了所有这些烦恼。不过没错,我很乐意谈谈那具体是什么。
I'll pick up more questions as I go. Yeah, I think there's a third part to this which I haven't talked about yet, which is kind of an interesting one, which is like refactoring and rewriting. Like, one of the most controversial topics in the industry, I think, is like should you rewrite your app or not? Because I think engineers are very prone to this where especially when you join a company, you come in and you see the codebase and you're like, "Man, this is shit. Who wrote this code? It's terrible. I want to rewrite the whole thing." There is a very common inclination and I think a lot of people, before agents, and arguably even now, will definitely discourage you from rewriting stuff. But I'm actually here to make a case for why you might want to consider it. Because I think it really depends. Brownfield applications I think are actually in a pretty good spot, especially if they're set up well already. And like recently I've been talking to some people, but I was just observing. I just noticed this parallel, which is that a lot of big tech company problems are now everybody's problems. And the big tech company problem, like when I was working at Meta, we had this giant monorepo. We had like, I don't know, tens of thousands of engineers just banging on their keyboards and shipping code. And a lot of really great engineers at Meta, but I'll say like, you'll be surprised that the code quality is actually not that good. And so I often joke that before AI sloth, we had human sloth. And so I think a lot of big tech infra like what Meta has or Google, really big tech companies are actually designed for that, where you're catering to the least capable engineer on your team, right? You build frameworks, you build conventions, you build guardrails, you restrict credentials so that your intern doesn't wipe your production database. If you have that level of infra already, I think your agents can actually already do a very solid job. Because the guardrails are already in place for agents to not cause havoc or not cause too much havoc in your codebase. And you can always add more guardrails. But I think greenfield applications, especially brand new applications, are the biggest risk, in my opinion. And also the greatest opportunity. Because if you vibe code a project or a prototype, like we did for Grok Bot, Grok Bot was spun up very very quickly. And if you haven't heard of Grok Bot, it's our new application we just launched yesterday. It's really cool. It lets you orchestrate and create individual agents that have their own identity and you can orchestrate them. It's super cool. Definitely check it out. But yeah, that was a very greenfield application, like most prototypes are. So it was vibe coded very quickly. Humans were not reading the code at all. And I had this tweet recently where I said something about organic architecture. Maybe I'll find it. But the idea is that when you have a completely vibe coded application, you essentially have no guardrails whatsoever. So your agents, when you give them a task, they will just solve it in whatever method is the most convenient. And over time you get into this situation where you have a codebase that is spiraling out of control because you don't understand it. Your agents understand it, I guess, in a way, but they've built something that is optimized for shortcuts. And you will suffer, you will have a lot of issues with that application. So I think starting your codebase with very strong strains is very much needed. Because when you have a codebase that you can trust, when you have guardrails that actually help your agents write good code, you can get into this part of the curve where I woke up today and I had like 20 PRs merged by my agents. And that's because I invested a lot of time, over 600 PRs I calculated yesterday, when I refactored all of Grok Bot to this new architecture that I've been building. And yeah, I've gotten to a point where I really don't look at the code anymore. And I say that not just to sell you tokens, but because it took a lot of work to get to that point. I spent a lot of tokens to get the codebase to this point where I no longer have to look at it. But I'm very excited because of the potential where it doesn't just benefit me. It benefits everyone contributing to Grok Bot. And it also empowers designers and product managers and even GTM people to add features to Grok Bot. And I don't have to worry, I don't have to wake up in the middle of the night and worry like, oh someone's just merged a perf regression. I have a ton of constraints and CI is actually very annoying to write code in Grok Bot, but agents absorb all of that annoyance. But yeah, I'm happy to talk about what exactly that is.
主持人:是的,在我们进入这部分之前,我想问一个问题,就是关于你的CI是什么样子,或者一些约束,还有平均PR大小。我之前看到有人问这个。只是给大家一个大概的印象。不一定要数学上的平均值,就是一般PR的大小,是只有几行代码还是怎样。
Yeah, I think one question before we get into this part here is just around that element of like what your CI looks like or maybe some of the constraints and then also like the average PR size. I saw a question about that earlier. Just to give people a glance. It doesn't have to be like mathematically average but just like what generally the size of a PR is if it's only a couple lines of code or yeah.
我觉得这取决于情况。也许我试图以一种我不会——
I think it depends. Maybe I'm trying to do this in a way where I'm not going to like—
主持人:对对,如果你不想分享实际数字,比如实际平均值,那就不必——
Yeah yeah you don't have to share the actual number if you like the actual average just—
这是个不错的基准。
This is a fine benchmark.
但就像我们有——好吧。这没那么有趣,但有趣的是,Grok Bot和Cursor里的虚拟化其实是由pretext驱动的,这是一个有人构建的新库。真的很有意思。你应该去看看。但那不是很重要。平均PR大小我其实不知道。我可能——我不知道我是否想点开这些。
But like we have— So okay. This is not that interesting but fun fact is that virtualization in Grok Bot and in Cursor is actually powered by pretext, which is a sort of new library that someone's built. That's really interesting. You should check it out. But that's not really that important. I think the average PR size I actually don't know. I probably— I don't know if I want to click on these.
呃,我大概可以,但我想说,它们的规模可以从几百行或 50 行到一千行不等,取决于具体在做什么。呃,所以就像这里,我实际上在删除一堆文件,所以我预计这主要就是删除操作。呃,但没错,这确实有变化。没有固定的……是的。
Uh I probably can but I would say like they can range anywhere from a few hundred lines or 50 lines to like a thousand depending on what the thing is doing. Uh so like here I'm actually like deleting a bunch of files so I expect that it's just this like mostly deletion. Uh but yeah it it kind of varies. There's no like yeah.
它们有硬性上限吗,还是都是 50 行左右的 PR?
Is there like a hard cap for how long they can be or are they all like 50 line PRs?
绝对没有硬性上限,但我确实鼓励我的智能体把工作拆分成多个 PR。呃,我这样做主要是因为我喜欢……我想也许现在这更难做到了,在智能体的世界里,你有那么多提交。但我喜欢 git 历史是一个丰富的上下文来源这个想法。呃,我喜欢每个 PR 以原子方式描述那一小块东西在做什么,这也让我更容易回滚更改,并弄清楚,哦,我发布了一个 bug,它在这里,对吧?而不是在一个 40,000 行的 PR 里,谁知道里面塞了什么。呃,但没错,我对 PR 大小没有硬性上限。
There's definitely no hard cap but I I do encourage my agents to split up their work into multiple PRs. Uh I do that mostly because uh I like I like the idea of I guess maybe this is much harder to do now is in the world of agents and you have like so many commits. But I like the idea that you know the get history is a very rich source of context. Uh and I like the I like each PR to sort of atomically describe what that small piece of thing is doing, which also makes it easier for me to revert changes and like figure out, you know, oh, I shipped a bug and it's this it's here, right? It's not in this 40,000 line PR where you know who knows what landed in there. Uh, but yeah, I don't have a hard cap on PR size.
酷。然后,呃,是的,还有一个关于 CI 的快速问题。所以,再次,你不必深入屏幕共享你的 CI 做什么,但一般来说,你能描述一下 CI 的样子吗,或者它有多严格?
Cool. And then, um, yeah, also quick question on like CI. So, again, you don't have to go into like uh, the screen share of like your your CI does, but just generally, would you describe what the CI kind of looks like uh, or how strict it is?
呃,是的,所以呃,好吧,具体到 Grok Bot,Dune 是我们为 Grok Bot 构建的架构的俏皮代号。嗯,CI 看起来挺烦人的,因为什么都有检查。所以,就像字面上,我有呃,呃,好吧,如果你读过任何 React,例如,你知道,你知道 React 中最大的坑之一就是 useEffect。呃,所以在 Dune 和 Grok Bot 中,我们禁用了 useEffect。所以,Dune 就是你可以……Dune 的心智模型,呃,你可以把它想象成用于 Electron 应用的 Next.js,它是为智能体编写而设计的。呃,它是为我们智能体驱动的应用定制的。嗯,所以 CI 检查非常具体,比如,你知道,不要使用 useEffect。它被禁用了,CI 会失败并冲你吼。我们有一些更有趣的,人们可能会挑眉,比如我实际上也禁用了代码注释。呃,这很有趣。嗯,但我注意到 99% 的情况下,智能体只是写一些描述历史事情的代码注释,而这些实际上与代码完全无关。嗯,比如它经常说,哦,Lawrence 说你永远不应该这样做。现在这出现在代码注释里。我就想,什么?为什么?那不是作为持久的全局规则说的。我只是说你这个 PR 很烂,你应该改那部分。呃,智能体并不太理解这一点,令人惊讶。呃,或者它们假设太多,以非常愚蠢的方式做事。所以就像,是的,我们禁止了一切你能想象到的智能体不擅长的事情。呃,所以我们在智能体窗口中经常受苦的一个例子是,我们有呃,你知道,如果你用过智能体窗口,你肯定见过性能问题,你知道,我们一直在努力修复它们。呃,但这就像一场无休止的斗争,因为有太多拉取请求被合并。任何一个都可能使性能、稳定性或可靠性回退。呃,你知道,智能体窗口还没有这个架构。我计划把这一学习成果带回去,并重构那里的一切。呃,但它经常回退,因为呃,有一个例子是我们进程间的隔离非常差。所以就像在 Electron 上,你有一个渲染线程来渲染 UI,但你也有一个主线程可以运行其他不需要阻塞渲染器的代码。嗯,但我们在分离这些方面做得很差。所以经常你不小心就有代码被拉到渲染线程上运行。呃,然后突然之间你就与渲染器竞争,你知道,如果你想达到 60 FPS,每一帧绘制必须在 16 毫秒内完成。所以每帧的截止时间非常非常小。如果你想要一个非常流畅的产品。当你开始构建,不小心引入计算量很大或大量 IO 的东西时,你就会遇到很多卡顿。你的 FPS 真的下降。你开始丢帧。你会得到超过 16 毫秒的长任务,然后体验就非常卡顿。所以,我们在构建 Electron 应用中学到的所有模式,我们都编码到这个框架中,它变成了硬性失败。所以,我在 Grok Bot 中字面上有一个名为 electron main、electron renderer 的目录,我们有一个导入 CI,我想,我们实际上检查依赖图,确保你不会意外地从另一个目录导入代码。所以,这由 CI 以及 Bobbots 强制执行,Bobbots 是我们的类似 Cursor 的代码审查工具,在 CI 上运行。你知道,在我们的 agents MD 中,到处都是。所以,我我我我,我在这里有这个东西,我谈到,你知道,我认为构建良好代码库有多个层次。显然,代码库是其中之一,如果你有这样的架构,它极其严格,你知道,构建功能的方式非常传统。那是最强大的执行级别,因为智能体就是喜欢复制现有模式。所以,Grok Bot 中的一个例子是我们有这些称为 feature 的概念,我们有入口点和转录卡片。就像你在聊天中看到的所有卡片。这些都是框架中的名词,我想。所以有一种非常传统的创建方式。例如,一个 feature 都在一个目录中。所以所有贡献该功能的代码都住在一个目录里。所以都集中在一个地方。这使得超级容易,你知道,智能体不必 grep 查找并试图弄清楚所有东西在哪里。它只是看着 feature,哦,好的,我在 Grok Bot 中处理 onboarding 功能。我就只在这个目录里工作。对于 80% 的工作,它主要就封装在那里。
Uh, yeah, so uh, well, specifically for Grok Bot, so Dune is the is the sort of cheeky code code name for the architecture that we built for Grok Bot. Um, the CI looks pretty annoying because there's checks for everything. So, like literally, I have um, uh, well, if you've read any React for example, you know, you know that one of the biggest foot guns in React is use effect. Uh, so in uh, Dune and in Grok Bot, we've banned use effect. So, Dune is just you can the the mental model of what Dune is, uh, you can kind of think of it as like Next.js for uh, Electron apps and it's designed for agents to write. Uh, and it's like custom for, you know, our agent powered applications. Um, so the CI checks are very like specific to that, like, you know, don't use use effect. It's it's it's banned, like CI will fail uh, and yell at you. We have like some of the more interesting ones that people might raise eyebrows is like I actually banned code comments as well. Uh, which is very interesting. Um, but I've noticed that a 99% of the time agents just write code comments that kind of describe some historical thing that is actually totally irrelevant to the code. Um like it will often say like, you know, "Oh, Lawrence said you should never do this." And it's now in the in the code comment. I'm like, "What? Like why what that was I didn't say that as like a durable, you know, global rule. I just meant like your this PR sucks and you should change that part." Uh agents don't really understand this that well, surprisingly. Uh and or they kind of assume too much and kind of do things in like very stupid ways. So like yeah, we just ban everything everything you can imagine like the agents are bad at we ban. Uh so one example that we actually suffer a lot in the agents window is we have uh, you know, you know, if you've used agents window, you've definitely seen performance issues and you know, we're constantly trying to fix them. Uh but it's like a it's a never-ending struggle because there's so many pull requests that get merged. Every any one of them could just regress performance or stability or reliability. Uh, you know, the agents window doesn't have this architecture yet. I plan to do bring this learning back there and kind of refactor everything there. Uh but uh it just regresses super often uh because uh there's there's one example is like we have very poor um isolation between processes. So like on you know, on Electron, you have a renderer thread that renders your UI, but you also have like a main thread that you can run other code that you know, doesn't need to block the renderer. Um but we do a poor job of separating those things. And so often times you just accidentally have code that gets pulled into running on the renderer thread. Uh and then all of a sudden you're competing with the the renderer that you know, that has a very if you want like 60 FPS, you have to every frame that gets drawn has to be done in 16 milliseconds. So very very small, you know, deadline per frame. If you want, you know, a very smooth product. And when you start building bringing in accidentally bringing in, you know, things that are like very computationally heavy or they have a lot of IO, then you just get into like a lot of jank. And your your FPS really drops. You start, you know, losing frames. You get long tasks that take more than 16 milliseconds and you just get this really choppy experience. So, all of those patterns that we've learned basically building Electron apps, we've encoded into this framework and it becomes like a hard failure. So, I literally in in Grok Bot, we literally have a directory called electron main, electron renderer, and we have import CI, I guess, where we actually check the dependency graph to make sure you're not accidentally importing code from one directory to another. So, that's enforced by CI as well as Bobbots, which is our which cursors like code review tool that runs on CI. You know, in our agents MD, it's everywhere. Like so, I I I I um I have this thing here where I talk about like you know, like there are multiple layers, I think, for building a good code base. Obviously, the code base is one where if you have an architecture like this, where it's extremely strict, you know, the the the way to build features is very conventional. That's like the strongest strongest level of enforcement because agents just love to copy existing patterns. So, one example of this in Grok Bot is like we have this these concepts called like a feature and we have entry points and transcript cards. Like all you know, the cards that you see in the chat. These are all like like nouns, I guess, in in the framework. And so there's a very conventional way of creating them. And so like a feature is all in a single directory as an example. And so all of the code that contributes to that feature lives in one directory. So it's all co-located in one place. Makes it super easy, you know, agents don't have to like grep around and try to figure out like where all the things are. It just looks at the feature and like, "Oh, okay, I'm working on the onboarding feature in Grok Bot. I'm just going to work in this directory." And for 80% of the work, it's mostly just very encapsulated there.
但就像,它再次是为,你知道,最笨的智能体设计的。就像你不必思考。好吧,这个框架的关键原则之一是,最短的路径就是最好的路径。因为这完全符合智能体喜欢写代码的方式。
But like it's like very it's like designed again for, you know, like the dumbest agent. Like you don't have to think. All right, the the the One of the key principles I have for this framework is like the shortest the shortest path is the best path. So because that plays exactly to how agents love to write code.
它们喜欢走捷径,真的。你知道,它们会找到解决问题的最快方法。那为什么不把那个方法变成解决问题的最佳方法呢?所以我可能不会深入所有具体细节。这个框架更像是一组想法和原则的集合,而不是会开源的东西。你可以,你知道,截图这个,如果你愿意的话,然后告诉你的智能体也为你构建类似的东西。嗯,是的,但这真的全是关于分层。你知道,代码库是一部分,有功能、目录,以及,你知道,导入阻止——不应该被导入的导入依赖。但它全都强制执行这些,还有静态分析。所以有 CI 检查。我们有很多针对我们观察到的坏模式的 lint 规则。编译器诊断。bug bot 里也有规则,嗯,我觉得像三、四、五是比较软的,对吧?这两个实际上会让 CI 变红,对吧?所以,你知道,有一个硬约束,智能体不能随便写烂代码。对于规则、技能和 bug bot,你的智能体仍然可能忘记,好吧?你仍然可能,或者它可能不总是持续应用它们。所以,我喜欢把它们分层,但我不喜欢依赖它们作为唯一的执行来源,因为它非常非常软,对吧?如果你只有规则、bug bot、技能和代码风格指南,你的代码库变成完全垃圾只是时间问题。我很抱歉这么说,但是,呃,我绝对推荐,是的,你知道,投资于那些可以硬性强制执行的东西,对吧?这就是为什么,你知道,也许你使用的技术栈的选择也非常重要。嗯,比如我认为,Rust 有点在变得,你知道,它又变得超级流行了,呃,因为编译器太严格了,对吧?编译器强制执行许多不同的东西,你知道,有一个借用检查器你必须安抚。只要确保你的智能体不写不安全的代码块,呃,你或多或少可以有点信心,如果代码编译通过,它可能就能工作,而且很好。呃,但你会看到,它给了你那种信任和信心,你作为人类工程师不再需要自己去检查它。你知道,你依赖代码和静态分析来真正让那变得,呃,顺畅得多。嗯,我想最糟糕的部分,最糟糕的处境是如果你被困在代码审查之地,你实际上通过字面上的人类说,你知道,阅读代码并说:“好的,你不应该这样做。”来强制执行代码库中的所有约束、不变量。对吧?每次你必须这样做时,你应该把这视为代码异味,像一种反模式。你应该说:“好的,而不是我在 PR 上评论,我如何把它变成硬规则?”好吧,我如何把它变成 lint 规则?我如何把它变成 CI 失败?或者我如何甚至彻底消除这个问题?呃,我可以谈谈我做的另一次迁移,但我可能在这里暂停。
Like they like to take shortcuts, really. You know, they'll find the quickest way to solve the problem. So why not make that the best way to solve the problem? So I probably won't get into all the specific details. And this framework is really more of a collection of ideas and principles rather than something that will open source. You can, you know, screenshot this, I guess, if you want and tell your agent to build something like this for you, too. Um yeah, but it's really all about the layers. You know, like the code base is one part with features and directories and, you know, import blocking import dependencies that shouldn't be imported. But and it all enforces that and static analysis. So like there's CI checks. We have a lot of lints for bad patterns that we observe. Compiler diagnostics. There's also rules in bug bot which are, um, I think like three, four, five are more soft, right? These two actually make CI red, right? So, you know, there's a hard constraint where the agent can't just write crappy code. For rules and skills and bug bot, your agents can still forget, all right? You can still or it may not always consistently apply them. So, I like to layer them, but I don't like to rely on them as the only source of enforcement because it's very, very soft, right? And if you only have rules and bug bot and skills and a style guide for your code, it will only be a matter of time before your code base looks like complete trash. I'm sorry to say that, but, uh, I definitely recommend, yeah, that, you know, investing in things that can be hard enforced, right? And this is why, you know, maybe uh, the choice of tech stack that you use is also very important. Um, like I think, for example, Rust is sort of making, you know, it's like getting super popular again, uh, because the compiler is so strict, right? The compiler enforces so many different things, you know, there's a borrow checker that you have to appease. And if as long as you make sure your agents don't write unsafe code blocks, uh, you can more or less feel somewhat confident that if the code compiles, it probably works and it's good. Uh, but you'll see, it gives you that level of trust and confidence that you as a human engineer no longer need to go and check it yourself. You know, you rely on code and static analysis to actually make that, uh, a lot smoother. Um, and I guess the worst part the worst place to be in is if you are stuck in code review land where you actually enforce all of the constraints, the invariants in your code base by literally the human person saying, you know, reading the code and be like, "Okay, you should not do this." Right? Every time you have to do that, you should consider that as a code smell, like an anti-pattern. And you should say, "Okay, instead of me commenting on the PR, how do I turn this into a hard rule?" All right, how do I turn this into a lint rule? How do I turn this into a CI failure? Or how do I even categorically eliminate this problem uh entirely? Uh I can talk about another migration I've done, but I'll probably pause here.
当然。是的,我觉得说实话,我现在就处于你刚才描述的那个阶段,就是我没有所有这些规则。呃,所以这次会议后我有一些事情要做,以便能够扩展我的智能体。我肯定是在,呃,你知道,也许本地有几个并行的,所以本地两到三个。我相信这里大多数人也是一样,所以呃是的,我知道我们只剩几分钟了。呃,Lauren,你还有什么想强调的吗?我显然有很多问题,所以我可以再收集一些,但我想给你几分钟,如果你还有什么想谈的。
Sure. Yeah, I feel like that's where I am, to be honest, is what you're describing right now, which is that like I don't have all of these rules. Uh so, I have some things to go do after this session in terms of being able to scale my agents. I'm definitely on like the uh you know, maybe a couple of parallel ones locally stage, so like two to three locally. And I'm sure most people here are on the same, so uh yeah, I know we only have a couple minutes left. Uh Lauren, was there anything else that you wanted to highlight? I obviously there's lots of questions, so I can grab more, but I want to give you a few minutes if there's anything else you want to talk about.
嗯,我觉得我已经唠叨了很多,所以也许我们就回答问题吧。
Well, I think I've been yapping for quite a lot, so maybe let's just do questions.
好的,酷。呃,我有个问题,呃,有几次被提到,就是关于 token 使用量。所以问题是,你描述的东西对于使用正常 token 用量的人来说是现实的吗?他们并没有,你知道,基本上无限的 token 来工作。
Okay, cool. Uh one question that I had uh a couple of came up a couple times was just around like token usage. So, the question is like is what you're describing a realistic thing for people who are on you know, uh a normal set of token usage, they don't have, you know, basically unlimited tokens uh to work with.
嗯,我觉得这是个非常好的点。我的意思是,显然,你知道,我在一个 AI 实验室工作,我们有无限的 token,所以我绝对不能说什么每个人都应该以和我完全相同的方式来做这件事。我认为达到这个点是可能的,而不用,你知道,花大钱。但是,你知道,如果你是工程领导者,或者你知道,你领导一个初创公司,嗯,我认为主要是一个 ROI 的问题。嗯,就像,呃,是的,你在前期阶段在 token 上花了很多钱。你知道,重构代码库会消耗大量 token。呃,添加所有这些内容会消耗大量 token。但是,如果我们正走向一个智能体编写所有代码的世界,而且,你知道,你想保持非常精简,对吧?你不想不得不招聘,你不想成为,你不想变得像 Meta 那样,对吧?我的意思是,你不想成为一个 10,000 人的工程组织,因为我的意思是,那是个很酷的问题,但你也需要,你有太多开销。有规划,你知道,就像你,这,呃,个人而言我不会,呃,这不是超级有趣。但是,嗯,我认为你想保持非常灵活,对吧?你想,你想,智能体就是让你做以前做不到的事情。这对我来说才是智能体的价值。你知道,不仅仅是把 token 扔到每一件小事上,但嗯,对我来说,我以前做不到的事情就是像这样在代码库中强制执行这种级别的约束。对吧?就像我只是一个人,你知道,呃,我自己构建这个框架并完成所有重构和测试所有内容并验证,你知道,就像运行,想象一下只有我,对吧?不,在前智能体时代,就像运行,你知道,靠我自己,那会花我很长时间,对吧?而且我的薪水相当高,对吧?所以,你知道,我认为工程领导者可能会问的问题是,你知道,就像有什么,有一个权衡:你是雇人来做这个,还是你花 token 来设置一个代码库,以便即使是最天真的,对吧,最笨的智能体也能做好工作。当你真正达到这个点时,即使不是,你知道,寓言大小的智能体,也能出色地编写代码。这也给我个人带来了很多红利,因为我不仅赋予了自己能力,而且再次,像产品经理、设计师、不熟悉 Grok 的工程师也能以可持续的方式做出贡献。所以,我认为,是的,这绝对是一个权衡。
Well, I think that's a really good point. I mean, like obviously, you know, I work at an AI lab where we have unlimited tokens, so uh I definitely cannot say that, you know, this is something everyone should do in the exact same way that I did it. I think it's possible to get to this point without, you know, breaking the bank. But, you know, if you are like an engineering leader or, you know, you're you have a startup that you lead, um I think the main is a question of ROI. Um and it's like uh yes, you spend a lot of money on tokens in the upfront stage. You know, you like refactoring a code base is going to take a lot of tokens. Uh adding all these things uh is going to take a bunch of tokens. But, if we're heading to a world where agents are writing all the code, and, you know, you want to be very lean, right? You don't want to have to hire You don't want to be You don't want to become like Meta, right? Like I mean like in terms of you don't want to become a 10,000-person engineering org because I mean, that's a cool problem to have, but also you need you have so much overhead. There's like planning, you know, like you it's it's uh personally I I wouldn't uh it's not super fun. But, um I think you want to stay very nimble, right? And you want to you want to be like agents are all about allowing you to do things that you couldn't do before. That's really to me like the value of agents. You know, it's not just throwing tokens on every single little thing, but um to me like the thing I couldn't do before is like enforce this level of constraints in a code base by myself. Right? Like I'm just a single person, you know, uh it would have taken me years to build this framework uh and do all the refactoring and test everything myself and verify, you know, like run Imagine if there was just me, right? No in in pre-agent era, just like running, you know, by my It would take me so long, right? And my salary is pretty high, right? Like So, you know, the the question I think an engineering leader might have is just then, you know, like what is there's a tradeoff of do you hire someone to do this, or do you spend the tokens to set up a code base so that even the the most naive, right, the dumbest agents can do a good job. And when you actually get to this point, like even agents that are not, you know, fable size, do an excellent job of writing code. And this pays a lot of dividends as well for me in personally where I have empowered not just myself, but again, like PMs, designers, engineers who are not familiar with Grok to just contribute in a way that is sustainable. So, I think, yeah, it's definitely like a trade-off for sure.
你知道,没有什么是免费的,这是肯定的。而且 token 相当贵。但其实,我不知道你们有多少人看到了,我们今天发布了 Grok 4.6。非常激动。终于出来了。所以,是的,Grok 4.6 会是一个很棒的——它非常非常聪明。在基准测试上表现非常好。而且 token 是一样的——希望我没说错,但我相信每 token 的成本和 4.5 一样。所以,你实际上以同样的成本获得了更多的智能。我认为这是 Cursor 试图——Cursor 和 SpaceX AI 真正优化的领域,就像成本与智能的帕累托前沿。我们不一定想构建有史以来最大的模型,因为运行起来极其昂贵。关键在于如何找到那个最佳平衡点,对吧?你不需要一个巨大的模型,但它就是超级聪明,而且推理成本不高。但嗯,我想总结一下,我觉得如果你自己做分析,感觉相当正面。你会得到相当正面的 ROI,投资这类东西不仅能赋能你自己,还能让你的整个团队生产力大大提高,对吧?想象一下,如果你有一支像我这样的工程师大军,每天交付大量改进和 bug 修复。那真是令人兴奋。
You know, like nothing is free, for sure. And tokens are pretty expensive. But actually, I don't know how many of you have seen this, but we actually announced Grok 4.6 today. So, very exciting. Finally out. So, yeah, Grok 4.6 would be like a great—it's very, very smart. It's really good on the benchmarks. And it's the same—the tokens—I hopefully I'm not saying this incorrectly, but I believe the cost per token is the same as 4.5. So, you're actually getting more intelligence for the same cost. I think this is an area that Cursor tries to—Cursor and SpaceX AI try to really optimize for, like that Pareto frontier of cost versus intelligence. We don't necessarily want to build the biggest model ever because that is extremely expensive to run. It's really about how do you find that sweet spot, right? You don't need a giant model, but it's just super smart, and it's not very expensive for inference. But yeah, I think to kind of round it up, I think it's—if you do your own analysis, I feel like it's pretty positive. It'll be pretty positive that the ROI you get from investing in stuff like this just empowers not just yourself, but your whole team to be so much more productive, right? Like imagine if you have an army of engineers like me who are shipping so many improvements and bug fixes every day. All right, like that is pretty exciting.
酷。在结束前最后一个问题。这个问题是给电话会议中产品部门的人的。假设我们真的有一支像 Lauren 这样的工程师大军在快速交付。我很好奇,产品团队或公司其他职能部门如何跟上,因为如果你们交付得这么快,他们是否更多地使用 AI 来完成工作?你能说多少就说多少。显然你不在那个角色,但只是好奇那是怎么运作的。
Cool. One last question before we wrap up. This one is for the people in product on the call. So, let's say we do have an army of engineers who are shipping like Lauren. I'm just curious, how is the product team or other functions of your company keeping up given that if you're shipping so quickly, are they using AI more to do their jobs? As much as you can speak to that. I obviously you're not in that role, but just curious about how that works.
我认为这正是 Grok Bot 实际上非常强大的地方。在 Grok Bot 之前,显然 Cursor 只有 Cursor。我们只有智能体窗口、CLI、IDE。这些都是真正的高级用户工具,对吧?它们是为开发者设计的,所以非常以开发者为中心。你可以在其中做知识工作,但 UI 并没有真正为此优化。所以,我们实际上并没有——嗯,我认为很多 GTM、产品的人可能用过 Cursor 来做他们的工作,但对他们来说绝对不是愉快的体验。我认为现在有了 Grok Bot,它基本上成了非技术人员的 Cursor 时刻,在我看来。这是一种非常非常易于访问的方式来使用智能体,在一个非常舒适、非常熟悉的界面中。它看起来像 iMessage。而且非常有趣。你可以给你的智能体起名字。你可以以非常自然的方式进行编排,每个智能体就像一个人,现在你有一个智能体团队在工作——例如,你每个账户有一个智能体来管理。或者如果你是 PM,你可以有一个智能体总结 Lauren 昨晚做的所有工作,然后你就知道了我做了什么,对吧?所以,我认为我们的 PM 们大量利用这一点,他们也在交付代码。所以,很多时候他们只会说,‘哦,这里有个 bug。我修好了。你能看看吗?’然后我去审查,实际上它完美无缺。我就说,‘好的,盖章。’所以,我认为这表明 Dune 架构是站得住脚的,对吧?所有严格的约束允许非工程专家在高水平上做出贡献。所以,我觉得我已经看到这带来了很多回报,设计师和 PM 们能够直接交付功能。这让 Grok Bot 团队超级快,对吧?我们可以如此快速地交付。我们有很多计划,所以我很兴奋能交付更多东西。
I think this is where Grok Bot has been actually exceedingly powerful. Before Grok Bot, obviously Cursor only had Cursor. We only had agents window, we had a CLI, we had an IDE. And these are really power user tools, right? They're designed for developers, so it's very developer centric. You can do knowledge work in them, but the UI is not really optimized for that. So, we actually didn't really have—well, I think a lot of people in GTM, product, they might have used Cursor to do their work, but it definitely wasn't a delightful experience for them. I think now with Grok Bot, it's become—Grok Bot is basically the Cursor moment for people who are not in tech in my opinion. It's a very, very accessible way to use agents in a very comfortable, very familiar interface. It looks like iMessage. And it's very fun too. You can give your agent a name. You can do orchestration in a very natural way where each agent is like a person and now you have a team of agents working—you have one agent per account that you manage as an example. Or if you're a PM, you can have an agent that summarizes all the work that Lauren did last night and then now you know what I did, right? So, I think our PMs are leveraging that a lot and they're shipping code, too. So, often times they will just say, 'Oh, here's a bug. I fixed it. Can you look at it?' And then I'll go review it and actually it's just perfect. I'm like, 'Okay, stamp.' So, I think that shows that the Dune architecture is holding up, right? All the really strict constraints allow people who are not experts in engineering to contribute at a high level. So, I feel like I'm already seeing that pay off a lot where designers and PMs are just able to ship features directly. And that just makes the Grok Bot team super fast, right? Where we can ship so quickly. And we have a lot planned, so I'm very excited to ship more stuff.