产品经理为何再度吃香:Tulsee Doshi 谈打造 Gemini

Why PMs Are Suddenly Valuable Again: Tulsee Doshi on Building Gemini

塔尔西·多希 Tulsee Doshi · Lenny’s Podcast · 2026-11-08 · 约 59 分钟 · 原视频 ↗

打开互动全文版(中英对照 + 朗读 + 问答)→

本期速览 · Overview

Gemini 产品负责人 Tulsee Doshi 解析模型研发为何既是科学也是魔法,以及这对她招募的产品经理意味着什么。

Tulsee Doshi, Head of Product for Gemini, explains why model development is part science and part magic, and what that means for the PMs she hires.

要点 · TL;DR

核心观点 · Key points

反共识 · Contrarian takes

本期章节 · Chapters(共 39)

全文 · Full transcript(中英对照)

模型开发不可预测性 Model development unpredictability

Host

模型是不可预测的。这有点像斯内普的魔药课。你可以把所有原料都放进去,但你也得预料到会有一点魔法,而那魔法会让人惊讶。

The model isn't predictable. It's kind of like Snape's potions class. You can put all the ingredients in, but you also have to expect that there will be a little bit of magic, and that magic is going to be surprising.

Host

你最近招产品经理时,有没有什么是你越来越看重的?

As you're hiring PMs these days, is there anything you look for more and more?

Tulsee

我们在找那些能理解我们模型或其他模型“气质”的产品经理。我想要有自己判断、每天都在玩这些模型的产品经理。理想情况下,你会想用自己真正兴奋的东西去压力测试模型。

We are looking for PMs who understand the vibes of our models or other models. I want PMs who have an opinion, who are playing with these models every day. You ideally want to stress test the model on things that you are excited about.

Host

很多人觉得谷歌现在落后于排名前二的两家实验室。

A lot of people feel that Google lags behind the top two labs right now.

Tulsee

我其实认为我们确实想赢下前沿这场对话。我不认为我们应该说谷歌没有参与前沿。

I actually think we do want to win the frontier conversation. I don't think we should say that Google is not participating in the frontier.

Host

你有一个关于要提前多久构建的框架,你把它叫作 60-20-20 法则。

You have this framework around how far ahead to build. You call it the 602020 rule.

Tulsee

传统的创业智慧就是,做一个能用的最小可行产品。但我认为你其实想构建一些今天未必总能正常工作的功能。

Traditional startup wisdom. You're just like build an MVP that works. But I think you want to actually build features that aren't necessarily always working today.

Host

感觉现在每个上线的 AI 助手都在用谷歌那些诱人的数据,比如 Gmail、谷歌日历、我的文档。我很好奇,为什么谷歌还没推出一个特别惊艳的东西。

It feels like every AI assistant launching these days is taking all this juicy Google data like Gmail, Google Calendar, my Docs. I'm curious just why Google hasn't launched something awesome yet.

Tulsee

团队正在做一些很棒的东西,但我认为你说得对,这方面我们还有更多工作要做。

The team is working on some amazing things, but I think you are right that there is more work we need to do here on this front.

Host

Tulsee,非常感谢你来,欢迎来到播客。

Tulsee, thank you so much for being here and welcome to the podcast.

Tulsee

谢谢,我很高兴能来。

Thanks. I'm excited to be here.

Tulsee旗下模型 Models under Tulsee's umbrella

Host

你是谷歌前沿模型的产品负责人,其中包括大家熟悉的 Gemini,但还有一堆人们可能不太熟悉的模型。请介绍一下你负责的其他模型。

You are head of product for Google's frontier models, which includes Gemini, which people know, but there's also a bunch of other models that people may not be familiar with. Share the other models that are under your umbrella.

Tulsee

好,我们有一组模型,我们称之为生成式媒体模型。这些模型专注于视频编辑与生成、图像编辑与生成。如果你用过 Nano Banana、我们的音乐模型 Liria,我们还有 Gemma,我们的一套开放模型。然后还有一组其实属于 Gemini 旗下、但我觉得我们谈得没那么多的模型,就是我们的音频模型。比如对话、音频理解、文本转语音、翻译,都是我们构建前沿模型产品组合的一部分。

Yeah, so we have a set of models we kind of call our gen media models. These are our models that focus on video editing and generation, image editing and generation. If you've used Nano Banana, Liria, our music model, we also have Gemma, our suite of open models. And then one set of models that actually falls under our Gemini umbrella, but I feel like we don't talk about as much is our audio models. So dialogue, audio understanding, text-to-speech, translation, all a part of the portfolio of how we build frontier models.

模型如何打造 How models are made

Host

好,我想花点时间聊聊模型。大家都看到模型发布,感觉每周都有一个模型发布。你有一个很独特的视角,能看到这些模型从研究到预训练、后训练、发布的全过程。你觉得人们可能误解了模型是如何创造的哪一点,或者有什么会让人惊讶的地方?

Okay, so I want to spend a little time on models. Everyone sees models launch. There's a model launch every week. It feels like you have a really unique lens in being able to see these models from research to pre-training, post-training launch. What's something that you think people maybe misunderstand about how models are created, or something that may surprise people about how models are made?

Tulsee

是个好问题。我想说的是,尤其是从产品管理的角度看,通常你谈任何产品的产品管理,包括模型,你会谈模型的目标是什么,你想设定哪些能力,你想推动哪些结果,你想在一组评估上达到什么目标。而当我跟客户甚至内部用户交流时,他们常问我,你设定的目标是什么,然后他们多少会期待我们能确保模型以非常传统产品的方式实现那些目标。对我来说,模型开发以及做模型产品经理常常令人惊讶但也令人兴奋的一点是,你应该那样做,你应该有明确的意图,但模型是不可预测的。所以这有点像斯内普的魔药课,对吧?你可以把所有原料都放进去,你应该仔细想清楚你期待什么,但你也得预料到会有一点魔法。而那魔法会让人惊讶,那魔法会是独特的。所以作为产品经理,你的工作不只是塑造进入模型的东西,还要看另一端出来什么,真正识别出模型在哪里闪光,在哪里以让你惊讶的方式显得独特,然后把它塑造成对用户很棒的东西。

Yeah, it's a good question. I mean maybe what I'll say, especially coming from a product management lens, is one thing that I think, you know, when you typically talk about product management for anything including a model, you talk about what are the goals you have for a model, right, what are the capabilities you want to set, what are the outcomes you want to drive, what goals do you want to hit on a set of evals. And often when I talk to customers or even internal users, you know they'll ask me, okay, what are the goals that you're setting, and they'll sort of expect that we can really make sure that the model will deliver on those goals in a very traditional product sense. And I think for me what has often been surprising but also exciting about model development and being a PM for the model is you should do that and you should have an intentional view, but the model isn't predictable. And so it's kind of like Snape's potions class, right? So you can put all the ingredients in and you should be thoughtful about what you expect, but you also have to expect that there will be a little bit of magic. And that magic is going to be surprising and that magic is going to be unique. And so your job as a PM is not just to try to shape what goes into the model, but it is also to look at what comes out the other side and really identify where does the model shine, where does it actually look unique in ways that surprise you, and then be able to take that and shape that into something that is amazing for users.

Host

真有意思,就像我们写好的这段代码,我们喂给它数据,我们并不完全知道会发生什么、它是怎么运作的。我最近请了一位 Anthropic 的可解释性研究人员来播客,我们聊了很多“模型是长出来的,不是造出来的”这个想法。我复述一下你说的:研究人员对下一个模型有这些目标,我们要更擅长写作,我们要试着更幽默,我们要试着成为更好的工程师。而你说最终结果究竟如何是不可预测的。他们把这些想法放进去,改变训练用的数据。关于这种意想不到的、怎么说呢,训练过程,还有什么可以补充的吗?

It's so interesting that it's just like this code that we have written that we feed data and we need to like, we don't exactly know what's going to happen and how it works. I recently had an interpretability research person on the podcast from Anthropic and we talked a lot about this idea that models are grown not built. And just to kind of reflect back what you're saying, there's these goals that researchers have for this next model. We're going to get better at writing. We're going to try to be funnier. We're going to try to become better engineers. And you're saying it's unpredictable exactly what ends up at the end. They put in these ideas. They change the data that it's trained on. Is there anything more there about just this kind of like unexpected, I don't know, process of training?

Tulsee

当然,这些模型背后有出色的研究,很多方面是可预测的,因为我们会持续监控这些训练运行。我们有研究人员跟踪它们,判断何时干预,观察不同指标随着训练推进而上升。所以我们在尽可能把它变成一门科学。但对我来说,作为产品经理真正了不起的一点是,你也需要真正理解模型另一端的“气质”。是的,你可以知道模型在某个能力上提升了,但那能力究竟在哪里闪光?它在下游产品中如何展现?你和模型交流时,它的行为和人格给你什么感觉?这些方面有很多我们确实会构建进训练里,但当我们理解模型、评估它、测试它时,我们总会学到更多。我想,我们总能对模型能做什么有新的认识。

You know, of course there's brilliant research that goes into these models, and so many aspects of it are predictable in the sense that we are consistently monitoring these runs. We have researchers who are tracking them, who are identifying when to intervene, who are seeing different metrics climb up as the run progresses. And so we are, you know, making it as much of a science as possible. But I think for me what is really amazing as a PM is you also need to really understand the vibes of the model on the other side of it too. So yes, you can understand that a model has improved on a certain capability, but where does that capability really shine through? How does that showcase then downstream in a product? How do you feel communicating with the model in terms of its behavior and persona? And there's a number of these aspects that we definitely build into training, but also as we understand the model and as we evaluate it and as we test it, we always learn more. We always learn something new, I think, about what the model can do.

赞助商:Work OS Sponsor: Work OS

Host

本期节目由本季首席赞助商 Work OS 呈现。OpenAI、Anthropic、Cursor、Replit、Sierra、Clay 以及数百家其他成功公司有什么共同点?它们都由 Work OS 提供支持。如果你在做企业级产品,你一定体会过集成单点登录、SCIM、RBAC 和审计日志的痛苦。Work OS 把这些交易阻碍变成了为 B2B SaaS 打造的开箱即用 API。而现在有一种新型用户需要访问你的应用:智能体。AI 智能体需要像人一样注册、认证并获得权限。Work OS 全都能处理。Agent OS 为智能体提供有范围、可撤销的身份,通过 OMD 进行智能体注册,还有 Airlock 来评判每个智能体动作,允许、拒绝或升级给人类处理。

This episode is brought to you by this season's presenting sponsor, Work OS. What do OpenAI, Anthropic, Cursor, Replit, Sierra, Clay, and hundreds of other winning companies have in common? They're all powered by Work OS. If you're building for enterprises, you have felt the pain of integrating single sign-on, SCIM, RBAC, and audit logs. Work OS turns those deal blockers into drop-in APIs built for B2B SaaS. And now there's a new kind of user that needs access to your app. Agents. AI agents need to register, authenticate, and get permissions just like a person. Work OS handles it all. Agent OS to give agents scoped, revocable identities, agent registration with OMD, and Airlock to judge every agent action and allow, deny, or escalate to a human.

赞助商信息 Sponsor Message

Host

我注意到,我投资的每一家向高端市场扩张的初创公司最终都会使用 WorkOS,因为他们是最好的。访问 workos.com,让你的应用为企业级做好准备。

I've noticed that every startup that I am an investor in that expands up market ends up using WorkOS because they're the best. Go to workos.com to make your app enterprise ready.

谷歌AI战略 Google's AI Strategy

Host

今天我想稍微谈谈谷歌在 AI 竞赛中的位置。感觉很多人认为谷歌目前落后于两大顶尖实验室——Anthropic 和 OpenAI,而且谷歌有时会表现得最出色,然后感觉又有点放缓。我很好奇你能分享关于谷歌在 AI 竞赛中的战略,以及外界可能不了解的你们内部是如何思考这些事情的。

Today I want to talk a little bit about Google's place in the AI race. It feels like a lot of people feel that Google lags behind the top two labs right now, Anthropic and OpenAI, and Google has these moments of being the best and then it kind of feels like it slows down a bit. I'm curious what you can share about just Google's strategy in the AI race and what maybe people don't understand on the inside of how you all think about this stuff.

Tulsee

首先,希望这次对话播出时,你们已经在使用 Gemini 4 Argon 了,这很令人兴奋。所以我真的希望我们是在讨论一个人们乐于使用的前沿 AI 模型。我认为 Gemini 4 在一定程度上体现了我们的战略,对吧?因为它确实是我们持续构建前沿实验室、前沿模型系列、前沿能力集的推动力。我认为如果你看谷歌的战略,它确实是全栈的,对吧?所以我认为谷歌的强大之处之一是从硬件一直到服务数十亿用户的终端产品。我记得这个数字是我们有 14 款产品用户超过十亿。我应该核实一下这个数字,但我想是这样的。所以如果你考虑这一点,我们实际上从硬件一直到建模,再到这种终端用户应用,我们的最终目标不仅仅是构建令人惊叹的前沿体验——这当然是目标的一部分。但也是要构建跨越成本-质量帕累托曲线的模型,能够服务不同生活方式阶段的不同用户。无论是来获取信息的搜索用户,还是试图构建下一个惊人产品的工程师,还是试图保护关键基础设施的网络防御者。我认为我们的目标是确保我们有模型服务这些用例范围,并且我们实际上是在这些用户的反馈中构建的。

First of all, hopefully by the time this conversation goes out, you will be using Gemini 4 Argon, which is exciting. And so I really do hope we're having a conversation about a frontier AI model that folks are excited to use. And I think Gemini 4 speaks a little bit to our strategy, right? Because it really is our push to continue to build a frontier lab, a frontier set of models, a frontier set of capabilities on our path. I think if you look at Google's strategy it really is full stack, right? So one of the things that I think is powerful about Google is all the way from hardware to end products that serve billions of users. I think the stat is like we have 14 products that are above a billion users. I should check that number but I think that's the stat. And so if you think about that, we actually go all the way from the hardware to then the modeling to then this kind of end user application and our goal ultimately is to not just build amazing frontier experiences which is definitely a part of the goal. But it is also to build models across this pareto curve of cost to quality that can serve different users at different points in their lifestyle. Whether that be a search user who is coming for information, right? Whether it is an engineer who is trying to build the next amazing thing, whether it is for a cyber defender who is trying to protect a critical piece of infrastructure. I think our goal is to make sure that we have models that serve that range of use cases and that we're actually building with feedback from these users.

多线并进取胜 Winning on Multiple Fronts

Host

所以我听到的是,谷歌可以在很多方面获胜。有每 token 成本的最有效帕累托前沿。有最聪明的模型。还有只是让你的产品更好。例如,AI 模式、AI 概览。所以,并不是说你必须在人们可以使用的最高级 AI 模型(如聊天机器人)上做到最好。基本上,你可以在所有这些其他领域获胜,但听起来你也试图获胜并拥有最先进的模型。

So what I'm hearing is there's so many vectors where Google can win. There's the most efficient kind of pareto frontier per token cost. There's the smartest model. There's also just making your products better. For example, AI mode, AI overviews. And so, it's not that you need to be the best at the most advanced AI model that people can use like chatbot. Basically, you can win in all these other areas, but it sounds like you're also trying to win and have the most state-of-the-art model.

Tulsee

我实际上认为我们确实想赢得前沿对话,对吧?我不认为我们应该说谷歌没有参与前沿,我认为我们正在构建越来越好的模型,我认为 Gemini 4 Argon 在我们所处的位置上 definitely 处于前沿。但我认为一件了不起的事情是能够构建广泛的经验和为用户构建多样化的体验。我认为对谷歌来说,我们的最终目标是构建真正让人们生活更美好的惊人体验,对吧?我认为这就是最终目标,无论你为前沿的哪个部分而构建。

I actually think we do want to win the frontier conversation, right? I don't think we should say that Google is not participating in the frontier and I think we are building better and better models and I think Gemini 4 Argon definitely is at the frontier in the place that we are at. But I think one thing that is amazing is being able to build a breadth of experiences and build a diversity of experiences for users. And I think for Google, our goal ultimately is to build amazing experiences that actually make people's lives better, right? And that is, I think, ultimately the goal, whatever part of the frontier you're building for.

AI安全与责任 AI Safety and Responsibility

Host

谷歌产品确实让我的生活更美好。就像你的生活中有多少是在谷歌生态系统中,这很惊人。甚至不是 AI 的东西,只是谷歌日历、Gmail,所有这些。就像谷歌文档。呃,所以它正在发生。

Google products definitely make my life better. It's just like amazing how much of your life is within a Google ecosystem. Like not even the AI stuff, but just Google Calendar, Gmail, all these things. You're just like Google Docs. Uh, so it's happening.

Tulsee

这就对了。这就对了。

There you go. There we go.

Host

嗯,我想谈谈另一个广泛的 AI 问题,然后我会深入更多产品具体的东西。如今一个热门话题是 AI 安全,以及这种恐惧:AI 会伤害人类。随着它越来越聪明,它会造成损害。你对此有非常有趣的视角,因为你在谷歌负责负责任 AI 团队的产品六年。这是你在这个角色之前做的。所以你思考这个问题很长时间了,比大多数人长得多。我很好奇你对这场辩论的看法。只是我不知道,你是否乐观我们会解决这个问题?你认为人们可能不了解关于 AI 安全辩论的什么?

Um, I want to talk about one other broad AI question, then I'm going to dig into more product specific stuff. A big topic of conversation these days is AI safety and this kind of fear that AI is going to harm humans. It's going to cause damage as it gets smarter and smarter. You have a really interesting lens on this because you led product for the responsible AI team at Google for six years. This is what you did right before this role. And so you've been thinking about this for a long time, a lot longer than most people. I'm curious just to get your take on this debate. Just I don't know, are you optimistic we will figure this out? What do you think people maybe don't understand about this debate around AI safety?

Tulsee

我们这里的安全团队非常出色。看到他们从模型初始训练到部署方式所投入的用心,令人惊叹。我认为安全已经嵌入到那个过程中,对吧?我们需要——我认为当我们谈论如何安全部署模型时,重要的一点实际上是思考围绕这些模型的正确缓解措施和结构是什么,对吧?例如,非常重要的一点是考虑监控,对吧?所以对我们来说,实际部署监控器来监控模型的思维链,并实际识别模型是否与其指导行动不一致,或者是否一致,然后能够适当干预。这实际上是那种缓解措施,让我们对部署这些模型更有信心,但也让我们更好地理解我们可能在哪里看到不一致,并更好地随着时间的推移改进模型。所以我认为像监控这样的事情让我对我们如何推出模型更有信心。我认为我们在谈论这场辩论时没有足够谈论的另一件事是 AI 在加固我们系统方面可以发挥的作用,对吧?所以我们在 Gemini 4 上做的一件事是积极与许多网络防御者合作,包括实际上甚至与现在是谷歌公司的 Wiz 合作,真正思考我们如何能够识别和修补全球系统中的漏洞,对吧?我们实际上有一个有趣的案例研究——Wiz 有一个“Scan for Good”项目,他们实际上积极与全球系统合作,识别潜在问题,他们与一个医院系统合作,识别了一个漏洞,然后修补了它。正是这种事情,我认为如果我们能够实际利用这些工具做更多,我们就能确保我们的基础设施和系统是稳健的,对吧?我认为随着整个领域的发展,这也非常关键。

Our safety teams here are incredible. And it's been incredible to see the intentionality that they've been putting in all the way from initial trainings of the model to how we deploy them. And I think safety is embedded in that process, right? And we need to be—I think the part that is important actually as we talk about how do you deploy models safely is actually thinking through what are the right sort of mitigations and structures around these models, right? So for example, one of the things that's really important is thinking about monitoring, right? And so for us actually deploying monitors that can monitor the model's chain of thought and actually identify whether the model is misaligned with its directed actions or whether it is aligned and then being able to intervene appropriately. That's actually the kind of mitigation that gives us more confidence in deploying these models, but also allows us to better understand where we might see misalignment and better actually improve the models over time. And so I think things like monitoring give me more confidence in how we're putting our models out. I think the other thing that we don't talk about enough when we talk about this debate is also the role that AI can play in hardening our systems, right? And so one of the things that we're doing with Gemini 4 is like actively partnering with a number of cyber defenders including actually just even partnering with Wiz that is now a Google company to really think through how can we actually identify and patch vulnerabilities in systems across the globe, right? We actually have this interesting case study—Wiz has this Scan for Good program where they're actually like actively working with systems across the world to identify potential issues and they worked with a hospital system to identify a vulnerability and then patch it. And it's that kind of thing that I think if we can actually leverage these tools to do more of, we can actually make sure that our infrastructure and our systems are robust, right? Which is also, I think, really critical as this whole space grows.

AI在网络防御 AI in Cyber Defense

Host

上周在 All-In 上实际上有一场关于这个问题的对话,现在 AI 在网络方面变得非常擅长,人们将更多地投资于网络防御和用于保护的 AI,所以将会有大量资金用于保护他们的系统免受这种越来越聪明的 AI 的侵害。

There's actually a conversation on All-In this last week about this whole issue that now that AI is getting really good at cyber, people are going to be investing a lot more in cyber defense and AI for protecting and so there's going to be a lot of money being spent on protecting their systems from this ever smarter AI.

AI安全与乐观 AI Safety and Optimism

Host

你们在那里投资真的很酷。所以我想回到这个问题:你觉得我们会解决这个担忧和问题吗?我们觉得很多人认为 AI 会接管世界,造成很多损害。你觉得这是一个可解决的问题吗?你乐观吗?还是说你经常思考这个问题并感到担忧?

So it's really cool that you guys are investing there. So I guess just coming back to the question of do you feel like we'll figure out this concern and this issue? We feel a lot of people feel that AI is going to take over the world, cause a lot of damage. Do you feel like this is a solvable problem? Are you optimistic? Or is it something you think a lot about and are kind of worried about?

Tulsee

也许你的问题的答案是两者兼有,因为我认为我们需要持续思考这个问题,对吧?我认为乐观的方式——没有更好的说法——通往乐观的道路是通过思考它,并且在通往那里的道路上非常小心、非常深思熟虑、非常有意识。所以这就是我乐观的来源:它更多来自于我是否觉得我们有人在有意识地、深思熟虑地思考安全故事是什么样子。我认为这是关键的威胁——你需要很多人感到担忧、进行对话,并真正审视我们如何构建模型,这就是我们必须采取的逐步过程。

Maybe the answer to your question is both, in the sense that I think we need to be consistently thinking about it, right? I think the way to optimism here—for lack of a better way of putting it—the path to optimism is through thinking about it and being very careful and very thoughtful and very intentional on the path to get there. And so that is my source of optimism: it comes more from do I feel like we have people who are being intentional and thoughtful about what the safety story looks like. And I think that is the critical threat—you need a number of people to be worried and to be having conversations and to be actually interrogating how we're building our models, and that's the step-by-step process we have to take.

Host

太棒了。有趣的是,我不认为我读过谷歌模型黑入任何人的报道,我认为这表明你们一直在思考这个问题很长时间了。我不知道,不管好坏,但感觉它正在起作用,无论你们在做什么。

Awesome. And what's interesting is I don't think I've read a story of Google's models hacking into anyone, which I think shows that this is something you all have been thinking about for a long time. I don't know, for better or worse, but it feels like it's working, whatever you all are doing.

Tulsee

我们喜欢关于修复漏洞的故事。这对我们来说是个好位置。

We like stories about fixing vulnerabilities. That's a good place for us to be.

产品复杂性与UX平衡 Product Complexity and UX Balance

Host

好的。所以从产品角度来看,AI 正在发生的一件有趣的事情是产品变得非常复杂。例如,谷歌因为很难找到你们做的所有神奇功能而受到很多批评。就像很多人不知道 AI Studio 甚至存在。很多人找不到 VIA 或 Genie 以及所有这些。同样,Codex——ChatGPT 最近推出了 dots——现在你去 Codex 的界面,它就像标签页、下拉菜单和点,简直失控了。Claude Code 也是一样。它变得非常复杂。那个 co-work,然后他们放弃了 co-work,有趣的是我在一个有很多产品经理的 WhatsApp 群里,有一个来自某个实验室的产品经理。

Okay. So one of the interesting things that's happening kind of from a product perspective with AI is products are getting very complicated. For example, Google gets a lot of flack for it being hard to find all the magical things you all do. Like a lot of people don't know AI Studio even exists. A lot of people can't find VIA or Genie and all these things. Similarly, Codex—ChatGPT recently launched dots—and now you go to the interface of Codex and it's like tabs and drop-downs and dots and it's just out of control. And Claude Code is the same thing. It's just getting very complicated. That co-work and then they drop co-work and interestingly I'm in this WhatsApp group with a bunch of PMs and there's a PM from one of the labs.

Tulsee

你想加入这个 WhatsApp 群吗?抱歉。

You want to be in this WhatsApp group? Sorry.

Host

好的,我们谈谈。他分享说有一种紧迫感,要尽快将前沿 AI 能力提供给用户。部分是为了保持竞争力,部分是因为他们还不知道它具体用于什么。所以先把它交给人们,弄清楚他们用它做什么,然后再改进用户体验。所以有一种有意识的‘先把它发布出去,以后再想办法如何打造出色的用户体验’。对于这种平衡——让产品非常易用 versus 尽快发布——你有什么想法?这是你处理的事情吗?

Okay, let's talk. And he shared that there's this kind of urgency to get frontier power of AI out to users as fast as possible. Partly to stay competitive, partly because they don't know exactly what it's used for yet. So let's get it to people, figure out what they're using it for, and then make the UX better. So there's this kind of intentional 'let's just get it out there, figure it out later, how to make it a great UX.' Thoughts on this kind of balance of making the product really easy to use versus just getting things out as quickly as possible? And is that something you deal with?

Tulsee

两个想法,也许三个。第一个我要说的是,我确实认为我们需要尝试新的范式和新的用户体验方法,如果你愿意的话,关于人们如何与模型互动,对吧?所以你提到了例如 Genie 或 VO,我们现在正在构建 Omni。就像如果你看看 Flow——Flow 是我们一直在设计的产品,思考视频编辑和生成、音乐编辑和生成。所以我认为我们正在尝试——NotebookLM 是一个不同形态的好例子。我认为我们的实验室团队正在积极思考如何为不同的个体尝试不同的形态和不同的用户体验来接触这些模型。所以我确实认为我们需要不断创新和尝试用户体验的样子。同时我认为你的观点也是对的,即我认为取决于你的用户是谁,你也想要不同的形态,对吧?所以就像我认为例如 AI Mode,我们谈论的是数十亿用户来到搜索问他们的问题。我认为你确实需要在那里有一个每个用户都能访问的某种用户体验,对吧?同时另一方面,当你有像 Anti-Gravity 这样的东西时,你是在为高级用户构建,你是在为知道如何使用斜杠命令并且能够跨终端和 IDE 积极互动的用户构建,你希望能够支持用户能做的广泛范围。所以我认为取决于你的用户是谁,你在思考用户体验方面有一定的灵活性,对吧?

Two thoughts, maybe three. The first one I'll say is I do think we need to be playing with new paradigms and new UX approaches, if you will, to how people will engage with models, right? So you mentioned for example Genie or VO, which we're now building Omni. Like if you look at for example Flow—Flow is a product that we've been designing thinking about video editing and generation, music editing and generation. And so I think we are trying to play with—NotebookLM is a good example of a different form factor. I think our labs teams are actively thinking about how do we try different form factors and different UX for different individuals to engage these models. And so I do think we need to keep innovating and playing around with what UX looks like. I think at the same time your point is also right, which is like I think depending on who your user is you want a different form factor too, right? So like I think for example with AI Mode we're talking about billions of users coming to search asking their questions. I think you do need to have a certain UX there that is accessible to every user, right? At the same time on the other end of the spectrum when you have something like Anti-Gravity you are trying to build for a power user and you are trying to build for users who know how to use slash commands and can actively engage across both the terminal and the IDE, and you want to be able to support a wide range of what a user can do. And so I think depending on who your user is you have a certain level of flexibility with how you think about the UX, right?

Host

有道理。基本上有不同的服务,针对你能多彻底改变事物的不同层次。

That makes sense. Basically there's different services for different levels of how radically you can change things.

Tulsee

是的,我认为另一部分也是,我认为把模型本身也看作有一种用户体验也很重要,对吧?所以作为模型的产品经理,我经常思考的一件事是实际使用模型的用户体验是什么,对吧?所以模型有多透明,它与你沟通得如何,它回来问你什么问题——这些实际上都以自己的方式属于用户体验。所以我认为思考如何将关于用户体验的对话带入实际的模型开发过程本身也非常有趣。

Yeah, and I think the other part too is I think it's also important to think about the model as having a UX also, right? So one of the things I think about a lot as a PM for the model is what is the UX of actually using the model, right? So how transparent is the model, how well does it communicate with you, what questions does it come back and ask you—these are all actually UX in its own way. And so I think it's also really interesting to think about how do we bring conversations about UX into the actual model development process itself.

Host

这真的很有趣。基本上模型的个性就是模型的用户体验。作为产品经理,你如何与研究人员合作来影响这项工作?是不是像‘这是关于它说话方式的反馈,这里应该更详细一点’以及‘这是它可以不同格式化内容的方式’?

That is really interesting. Basically the personality of the model is the UX of the model. How do you as a PM inform that work with the researchers? Is it like 'Here's feedback on the way it's talking and it should be a little more verbose here' and 'Here's ways it could format things differently'?

Tulsee

我认为产品经理在这里的角色实际上非常棒,因为你实际上在塑造我们希望模型行为走向的方向,对吧?所以我们有一个团队在这里研究模型行为,我们一起思考的很多内容是查看客户反馈和用户反馈,了解他们如何互动,但也思考对 Gemini 来说什么是好的,对吧?以及我们如何制定一个规范,如果你愿意的话,为我们正在做的——那是什么样子,然后能够将其转化为我们可以评估模型的评分标准,能够将其转化为我们可以实际用于模型的数据,能够将其转化为我们的奖励建模。所以这实际上是产品经理和研究团队之间非常好的合作。

I think the role of the PM here is actually really awesome because you are actually getting to shape what is the direction of where we want to go with the model's behavior, right? And so we have a team that works on model behavior here and a lot of what we've been thinking about together is looking at customer feedback and user feedback for how they engage, but also thinking through what does good look like for Gemini specifically, right? And how do we develop a spec, if you will, for what we're—what does that look like, and then being able to translate that to rubrics that we can then evaluate the model on, to be able to translate that into data that we can actually leverage for the model, to be able to translate that into our reward modeling. And so it's been actually a really nice collaboration between PM and the research team.

AI能力过剩 AI Capability Overhang

Host

我经常听到的一件事是,即使我们停止模型的开发,它也能做很多我们尚未做的事情。存在这种过剩。这是你经常思考的事情吗?感觉这是这项工作的一大部分——将 AI 模型的能力暴露给用户。

Something I hear a lot is that even if we stopped development on the model there's so much it can do that we are not yet doing. There's this overhang. Is that something you think a lot about? Is that—it feels like that's a big part of this job is exposing the powers of the AI model to users.

Tulsee

是的。

Yeah.

新前沿与客户反馈 The Next Frontier and Customer Feedback

Tulsee

还有搞清楚什么是新的,对吧?下一批令人兴奋的方向在哪里——下一个前沿在哪里?我们想要投入和构建哪些能力?我觉得这本身也是一件很有意思的事。而且我认为这其中很多工作其实是和我们的内部客户紧密协作的,比如搜索、Gemini 应用、YouTube 或 Labs,同时也要和外部客户合作,真正理解他们要往哪走、他们的需求是什么、什么会真正改变他们的体验,以及我们如何真正建立起那个反馈飞轮。

And figuring out what is new, right? And where is the next set of exciting—where is the next frontier? What are the capabilities that we're trying to invest in and build? I think that is a fun activity here also. And I think a lot of this actually is really working hand in hand with our internal customers, if you think about search or the Gemini app or YouTube or Labs, but also working with external customers to really understand where are they going, what are their needs, what would meaningfully change their experience, and how do we actually build that flywheel of feedback too.

产品经理回归 The Return of Product Managers

Host

顺着这个思路,我一直在 Twitter 上特别关注的另一个很有意思的趋势,是人们又重新意识到产品经理的价值了。我们经历过那样一个阶段,就是觉得我们不再需要 PM 了。

Along these lines, one of the other really interesting trends I've been watching on Twitter in particular is people realizing the value of product managers again. We went through this phase of just like, we don't need PMs anymore.

Tulsee

对,没错。

Yeah, exactly.

Host

就好像,好吧,我们什么都能做了,不需要 PM 了,全都建出来吧,PM 太糟糕了。而原因——稍微补充一点背景——是现在构建东西太容易了,人们就在造一堆没用的东西、行不通的东西、没人想用的东西。于是人们意识到,判断什么值得构建是一门非常特定的技能:人们有哪些值得解决的问题,做完之后“好”是什么样、“出色”是什么样,以及如何让人们围绕一个统一的愿景达成一致。你注意到这一点了吗?对这股重新兴起的 PM 崛起,你怎么看?

It's just like, okay, we can build anything. We don't need PMs. Let's just build it all. PMs are so bad. And the reason—just to add a little context there—is now that it's so easy to build, people are just building all this nonsense and things that don't work, things people don't want to use. And people are realizing there's a very specific skill to figure out what is worth building, what problems people have that are worth solving, what great looks like, what good looks like once you're done, aligning people behind one cohesive vision. Is that something you're noticing? Thoughts on that kind of PM ascendance that has started to happen again?

Tulsee

我喜欢“PM 崛起”这个说法,我要拿来用。对,我觉得你说得非常好。你看,我显然有偏见,因为我自己就是 PM,所以我愿意相信 PM 崛起已经到来、我们回来了。但我其实觉得正如你所说,当你能构建任何东西、而且能如此轻松地构建任何东西时,构建出卓越的东西本身就成了独特的价值增量,对吧?品味现在变得非常重要。能有人进来,真正从噪音中分辨出信号,找到那个能真正解锁卓越体验的洞见。而这正是一个优秀 PM 所做的,带来那份洞见并帮助解锁。你提到的另一点,我在 Google 这样同时构建多个产品的地方尤其感受深刻:当构建如此容易时,能凝聚人心、推动共识、把混乱变成秩序的人,我觉得变得非常重要,因为如果没有这一点,以我们现在能拥有的构建速度,速度本身就可能带来混乱——如果你无法真正带来清晰度、带来结构的话。

I like the phrase PM ascendance. I'm going to use that. Yeah, I think you actually said it very well, right? Look, I'm obviously biased because I am a PM, so I would like to believe that PM ascendance is here and we're back. But I actually think exactly what you said, which is when you can build anything and you can build anything so easily, building something great actually is now a unique value add, right? And taste is now really important. Someone who can come in and actually really pull out what is the signal from the noise, right? And what is that insight that will actually unlock a great experience. And that is truly what a good PM does, right, is bring that insight and help unlock. I think the other part that you noted that I particularly feel in a place like Google where we are building a number of products is again, when this is so easy, someone who can bring people together and drive consensus and turn chaos into order, I think becomes actually really important, because in the absence of that and the ability to build as quickly as we now can, speed can basically introduce chaos if you're not actually able to bring clarity to that and bring structure to that.

招聘PM看什么 What to Look for When Hiring PMs

Host

那么说到这个,你现在招 PM 时,有没有什么是你越来越看重的、在你招的 PM 身上可能正在上升的技能,是你觉得要真正成功非常重要的?或者宽泛地说,你觉得要胜任研究型 PM 这个角色,哪些技能非常重要?

So on that, as you're hiring PMs these days, is there anything you look for more and more, a skill that might be trending up in the PMs that you hire that you think is really important to be really successful? Or just broadly, skills that you find are really important to be successful in this role of being a research PM?

Tulsee

对,我觉得我们越来越看重这样的 PM——怎么说呢——能理解我们模型或其他模型“气质”的人。我想要有自己观点的 PM,每天在玩这些模型、有自己的用例来判断一个模型是否真的好、亲自把玩并压力测试这些能力、能扎进去看数据、能灵活上手、撸起袖子干的人。所以我觉得这既包括努力建立这种直觉的 PM,也包括在技术上有足够能力、愿意像我说的那样撸起袖子深入进去真正理解数据、构建一个 eval、真正坐下来把这些东西拼起来的人。我觉得这非常重要。另外我想说的是,敏捷且善于适应的 PM 也非常重要。这个领域每两周就在变。你两个月前写的路线图,你会再翻出来重写,对吧?所以我认为,那些持续关注环境、关注市场格局,并不断重新审视我们在做什么、为什么这么做,而不是假设存在某种固定行动计划的 PM,非常重要。

Yeah, I mean I think more and more so we are looking for PMs who, for lack of a better way of putting it, understand the vibes of our models or other models. I want PMs who have an opinion, who are playing with these models every day, who have their own use cases for how they judge whether a model is really good, who play with and stress test these capabilities themselves, who are capable of diving in and looking at data and getting scrappy and rolling up their sleeves. And so I think it's both PMs who are trying to build that intuition as well as who are technical enough in the sense of being willing to, like I said, roll up sleeves, go in, really understand the data, build an eval, actually sit down and pull those pieces together. I think that's really important. The other thing I'll say is I think PMs who are agile and adaptive are also really important. I think the space is changing every two weeks. A roadmap you wrote two months ago, you're going to be looking at it again and rewriting it, right? And so I think PMs who are constantly keeping an eye out on the environment, on the market landscape, and actually consistently reinterrogating what are we doing and why are we doing it, as opposed to assuming that there's a fixed kind of plan of action.

如何写首个评估 How to Write Your First Eval

Host

太棒了。好,那么就是气质、eval、适应力。我想稍微深入探讨一下。关于 eval,假设有人真的不知道怎么写出一个 eval、不知道怎么着手写自己的第一个 eval,有什么建议能开始提升写 eval 这项技能?

This is awesome. Okay, so vibes, eval, adaptive. I want to poke at these a little bit. So with evals, say someone doesn't really know how to write an eval, how to approach writing an eval, their first eval. What are some tips to start getting better at the skill of writing evals?

Tulsee

对,说实话,我们在内部和我们的 PM 也还在摸索这件事。我想说,我们看到有些 PM 在这方面比其他人更强,所以我们正在努力提升整个团队的能力。对我来说,一个关键点是哪怕从五个 prompt 开始,对吧?你不必一开始就搞很大。我们常常会想,哦,我得构建某种大规模评估或基准。我觉得往往哪怕只是一个迷你评估,做到 20 个 prompt,能真正压力测试一组能力,让你能看到你的模型和另一个模型之间、或模型 A 和模型 B 之间的信号,我觉得这已经非常强大,而且是一种解锁,因为它不需要你对所有环节都那么严苛地思考,对吧?从简单开始。甚至就从你自己开始——你作为评分者,你看一个 prompt,你试着判断回答里的结果好不好。尤其当你谈论这些长程、多步、多轮的轨迹时。哪怕只是能识别出你如何思考“好”,其实都非常难。所以哪怕只是得到那组好例子,就已经能让你感觉到,哦,这就是我想要达到的。它会给你自我的清晰度,然后你再去想怎么把它规模化。

Yeah, and to be honest, we're still figuring that out internally with our PMs as well. And I would say we see some PMs who are stronger at this than others. And so we're kind of working to upskill our whole team here. I think for me a big thing is start with even just five prompts, right? You don't have to start large. I think we often think, oh I need to build some sort of massive large-scale evaluation or benchmark. I think often even just a mini evaluation where you get to like 20 prompts that can really stress test a set of capabilities, where you can see signal between your model and another model, or model A and model B. I think that already is so powerful and that's an unlock, because it doesn't require you to think as critically about all of the pieces, right? Start simple. Start with even just yourself—you as the rater, you looking at a prompt, you trying to decide if the result in the response is good. Especially when you're talking about these long multi-step, multi-turn trajectories. Just even being able to identify how you're thinking about good is really hard actually. And so even getting that set of good examples will already give you a sense of, oh, this is what I'm trying to get to. It'll give you self clarity, and then you figure out how to scale that up.

评估与直觉 Evals vs Vibes

Host

太棒了。好,这真是很好的建议。大概六个月前 Twitter 上有过一场关于 eval 与气质之争的辩论,有意思的是——你的建议是两者其实都很重要:结构化的 eval 评分标准,以及练好捕捉气质的能力。那么关于气质这块,说实话这是我自己也很纠结的地方。当我拿到一个新模型时,我从来不太确定该拿它做什么,才能理解它到底哪里变强了。所以我就只是,你好,试试我以前做过的一些事。这个不错,这个很棒。

Awesome. Okay, that is really good advice. There was this debate I don't know six months ago on Twitter around eval versus vibes, and it's interesting—your advice is both are actually really important: structured eval rubrics and also getting good at getting the vibes. So on that vibes piece, this is something I struggle with, to be honest. When I get a new model, I never know exactly what to do with it to understand what it's gotten better at. So I'm just like, hello, let's try some things I've done in the past. This is cool. This is great.

评估模型改进 Evaluating Model Improvements

Host

对于那些想更好地掌握这项特定技能的人,你有什么建议?这项技能就是理解模型在哪些方面变得更好了,以及它的感觉如何,也许与之前的模型相比发生了怎样的变化。

What are some tips you have for folks that want to get better at this very specific skill of understanding where a model has gone better and the vibes and how they've shifted maybe from a previous model?

Tulsee

这不一定有一个正确答案,因为理想情况下,你想用你感兴趣的东西对模型进行压力测试,对吧?我认为,这也是你能看到模型神奇之处的地方。所以我实际上看到不同的人对此有非常非常不同的看法,对吧?例如,在 Twitter 上,你会看到很多人尝试 3D 动画和更具视觉美感的东西,因为这是一种通过视觉来评估模型运行情况的方法。你会看到其他人让模型进行写作,无论是创意写作、写文章还是综合信息。你还会看到有人用它进行非常困难的财务分析。所以我认为,部分原因是找到正确的任务,让你觉得今天某些模型已经显示出生命的迹象和光芒,但在你定义的完美标准下它们并不完美。这给了你一些灵活性,可以看到不同模型以不同方式处理问题。我认为这已经让你对模型的表现有了一些了解。

There isn't necessarily a one right answer to this because you ideally want to stress test the model on things that you are excited about, right? That's where you're going to also see magic in the model, I think. And so I actually see that different people have very very different notions of this, right? So you'll see, for example, on Twitter a lot of people trying the kind of like 3D animations and more visual type aesthetics because that's one way to kind of visually gauge a sense of how models are operating. You'll see others throw models at writing, whether that be creative writing or writing essays or synthesizing information. You'll see others throw it at just like really hard like financial analysis. And so I think part of it is like finding the right task where you feel like today certain models are already showing signs of life and showing light, but they're not perfect under your definition of perfect. And that gives you some malleability to see different models handle the problem differently. And that already gives you I think some insight into how the models perform.

Host

这是很好的建议。所以我听到的是,坚持你真正关心的事情,观察模型在这件事上如何进化。可能是为你写产品需求文档,可能是写战略文档,可能是生成图形。

This is great advice. So what I'm hearing is stick to things you actually care about and watch the models evolve on how good they are at this thing. Could be writing a PRD for you, could be writing strategy doc, could be generating graphics.

评估中的肤色量表 Skin Tone Scale in Evaluations

Host

好的。我有几个评估故事,是你的一位同事分享的,关于你的一些趣事。

Okay. I have a couple eval stories that one of your colleagues shared about that are fun facts about you.

Tulsee

哦,天哪。

Oh boy.

Host

这是 Eli Collins,产品副总裁。他是你在 Google DeepMind 的同事。好的。其中一个是他参与了一个叫做菲茨帕特里克肤色量表和小僧肤色量表的工作,这是人们在模型工作和评估方式上的一个重大转变。请分享这个故事,这项工作是什么。

So this is Eli Collins VP product. He's a colleague of yours at Google Deepmind. Okay. So one is he worked on something called the Fitzpatrick scale and the monk skin tone and this was a big shift in how people in how models worked around and eval part of this share this story what this work was.

Tulsee

是的。就像你之前提到的,我在负责任 AI 领域工作了六年。其中很大一部分时间,我们都在关注肤色和肤色代表性在我们的模型和产品中的体现。我们意识到的一件事是,早期人们使用的一种量表是菲茨帕特里克量表,它只有六种肤色。这就是我们评估产品有效性的方式。但实际上,六种肤色无法捕捉世界上肤色的多样性。不仅如此,它实际上是为完全不同的目的而构建的。它是为医疗保健目的而构建的。因此,由于它试图解决的问题,它实际上极度偏向非常浅的肤色。所以,它不仅只有六种肤色,而且如果你试图代表全球人口,它甚至不是正确的六种肤色。因此,我们与哈佛大学的一位教授 Monk 教授合作。他一直从人类学和社会学的角度研究肤色特征。他通过多年的用户访谈和研究,建立了一个包含 10 种肤色的量表。我们将这组肤色纳入我们的评估中,并实际上重新设计了我们评估产品和模型的许多方式,以反映这个 10 种肤色的量表。并且实际上利用它来改进我们在 Pixel 相机和图像生成等方面的工作,更加有意识地表现肤色。

Yeah. So I like like you noted earlier, I spent six years in responsible AI. Um and a big part of that time we were looking at skin tone and skin tone representation um in our models um in our products. And one of the things we realized is that early on um there was a scale that individuals were using, the Fitzpatrick scale, which was six skin tones. And that was how we were evaluating uh the effectiveness of product. And realistically, six skin tones cannot capture the diversity of skin tones in the world. Not only that, it was actually built for an entirely different purpose. Um, it was built for like healthcare purposes. And so, it actually skewed extremely light skin tone uh given the problems that it was trying to solve. And so, it actually not only was only six skin tones, it was also not even the six like correct skin tones if you're trying to represent like a global population. And so, we worked with a professor at Harvard, um, Professor Monk. uh he had been from like a more anthropology sociology perspective looking at characterizing a set of skin tones. And so he had a scale of 10 skin tones that he had built through a number of user interviews and uh research that he had been doing over years. And we pulled that set of skin tones into our evaluations and actually redid a number of the ways that we evaluated our products and our models to reflect that 10 skin tone scale. um and actually use that to improve things like our efforts in the pixel camera like some of our image generation uh to be more intentional about how skin tones are represented.

整体评估方法 Holistic Evaluation Approach

Host

太棒了。你在这里产生了重大的影响。你认为还有什么可能对人们有用,当他们听到这个并认为可能需要重新思考他们可能过时的评估。

That is awesome. Uh that is some serious impact you had here. What um what else do you think is maybe useful to people for people to hear is they hear this and they think re maybe have to rethink their evals that may be out of date.

Tulsee

是的,我认为评估的一个有趣之处在于,尤其是现在,你必须整体地考虑它们,对吧?例如,我们做的一件事是训练评估模型表现的评判者。评判者的质量也很重要,对吧?所以你的模型需要表现良好,但评判者如何评价模型也需要表现良好,这样你才能有准确的感知,对吧?你是否在生产环境中评估,包括所有的缓解措施和过滤器,可能会影响你的模型性能,对吧?模型的部署方式也可能影响模型性能。因此,在评估最终用户实际体验时,所有这些部分都需要考虑。实际上,我经常和我们的音频团队讨论这个问题,因为音频很复杂。你谈论的不仅是模型所说的实际内容,还有声音的自然度、音质和语调,这些是部署实际上可以真正影响的东西,即使你的模型很好。因此,拥有一个干净的设置,真正进行端到端评估变得超级重要。

Yeah, I mean I think one thing that's kind of interesting about evals in general is especially now is you have to kind of think about them holistically, right? So one of the things that we do for example is you train you have judges that evaluate the models and how well they perform. The quality of the judge also matters, right? So your model needs to be performing well, but also how your judge rates the model also needs to be performing well for you to have an accurate sense, right? Whether you evaluate on your production stack with all of your mitigations and filters might affect your model performance, right? How your model is served might impact your model performance. And so all of these pieces need to be thought through as you're trying to evaluate what an end user will actually experience. And like this is something that I talk about a lot with our audio team actually because audio is complex. You're talking not just about the actual content that the model is saying, but also the naturalness of the voice and the tenor of the voice and its inflections and these are things that serving actually can really affect even if your model is good. And so having a clean setup that you're actually evaluating end to end becomes super super important.

赞助商信息 Sponsor Message

Host

本季度的支持赞助商是 DX。现在贵公司的每个人都终于在使用 AI 了,工程领导者面临的问题不再是我们是否在使用 AI,而是它是否带来了回报?通过 DX,您可以衡量和基准测试 AI 采用在整个产品生命周期中的影响。DX 向您展示 AI 在哪些方面帮助了开发人员,在哪些方面创造了新的瓶颈。您还可以使用它来评估 AI 供应商、监控成本和许可证,并确切了解是什么限制了您的智能体。这就是为什么包括 Snowflake、Sony 和 BNY 在内的数百家企业使用 DX 平台来衡量 AI 对开发人员生产力的影响。访问 getdx.com/lenny 获取 DX 的演示。那是 getdx.com/lenny。

This season's supporting sponsor is DX. Now that everybody at your company is finally using AI, the question that engineering leaders face is no longer are we using AI, but is it paying off? With DX, you can measure and benchmark the impact of AI adoption across your full product life cycle. DX shows you where AI is helping developers and where it's creating new bottlenecks. You can also use it to evaluate AI vendors, monitor costs and licenses, and see exactly what's limiting your agents. That's why hundreds of enterprises, including Snowflake, Sony, and BNY, use the DX platform to measure AI's impact on developer productivity. Visit getdx.com/lenny to get a demo of DX. That's getdx.com/lenny.

Flash团队早期研究 Early Research on Flash Teams

Host

这就像是 Tulsi 趣事角。我听到的另一个有趣的故事是关于群体智能的。现在每个人都在谈论智能体群体、Hugging Face,所有这些它们可能串通做坏事的东西。显然,12 年前在斯坦福,你就在这方面做了一堆研究,远在任何人考虑这个潜在问题之前。请分享你当时做了什么以及那个故事。

This is like Tulsi fun fact corner. There's another interesting story I heard which is around swarms. So right now everyone's talking about agent swarms, hugging face, all these things where they all kind of collude to do bad things. Apparently 12 years ago at Stanford, you did a bunch of research on this way before anyone was thinking about this potential issue. Shared what you did there and that story.

Tulsee

是的,我的意思是,这实际上是一种非常不同的框架和思考方式。所以当我在斯坦福时,我和 Michael Bernstein 教授做了一些研究,他非常了不起,围绕的想法是他当时称之为“闪电团队”的概念,想法是如何更好地将一组众包工人聚集在一起,并有效地整合一套技能和个体来实际完成一项任务。这里的想法是,你知道,不同的人有不同的优势。

Yeah, I mean it's actually a very different kind of framing and thinking through which is so when I was at Stanford I did some research with professor Michael Bernstein who's incredible around this idea of like he was calling them at the time flash teams and the idea was how do you better bring together a set of like crowdsourced workers and actually efficiently pull together a set of skill sets and individuals to actually complete a task. And the idea here was, you know, different individuals have different strengths.

Flash团队与智能体 Flash Teams and Agents

Tulsee

你能否真正有意识地整合这些,使这种大规模的众包努力更有效、更高效?现在回想起来,那项研究真是令人惊叹。当时,我们是在人的背景下进行那项研究的。我们与 mTurk 和个人合作,将他们聚集在一起,研究如何创建一个有效的生态系统,以及如何创建一个有效的闪电团队,对吧?想法是,你可以快速组建一个团队,也可以快速解散它,并且对团队中的个人和你自己都极其高效。而现在,我们能够用智能体来复制那个世界。因此,将之前的世界映射到当前的世界,思考如何应用一些相同的原则,如果你愿意的话,但实际上是关于如何建立一个有效的子智能体系统,这很有趣。

Can you actually be really intentional about how you pull that together to make this full effort of crowdsourcing at scale more effective and efficient? And it's amazing to think about that research now. At the time, we were doing that research in the context of people. We were working with mTurk and individuals to pull them together into how to create an effective ecosystem and how to create an effective flash team, right? The idea being like you can spin up a team and spin it down and make it extremely efficient for both the individuals in the team and yourself. And now we're kind of able to replicate that world with agents. And so it's been interesting to kind of map that previous world to this current one in terms of like how do we apply some of the same principles, if you will, but to like how you actually set up an effective system of sub-agents.

Host

真正有趣的是,我不知道你读了多少关于整个 Hugging Face 事件的报道,但智能体们自己弄清楚每个角色的方式,它们决定你是领导者,我将牺牲自己。它们能够如此良好地合作,并自己创建这些闪电团队,而这一切都没有经过训练,这真是太有趣了。它们只是说让我们弄清楚这个。

What's really interesting is I don't know how much you've read about this whole Hugging Face incident, but the way that agents figured out on their own what each one's role is going to be and they decided you're the leader. I'm going to go sacrifice myself. And it's so interesting how well they are able to work together and create these flash teams on their own without any of this being trained into them. They're just let's figure this out.

Tulsee

我认为闪电团队的整体概念以及你可以真正调动个人组成一个高效团队的想法,即使只是短暂的一刻,然后实际上解散它,这实际上是一个令人兴奋的原则,我们已经在思考,但现在在我们所处的世界中更加重要。

I think the whole notion of a flash team and this idea that you can actually like mobilize individuals into something that can be highly effective even if it's only for a moment in time and then actually like take that out is actually like a really kind of exciting principle that we were already thinking about but now exists even more importantly in the world we're in right now.

Host

是的。好吧。我认为 Genomechanical Turk 要关闭了。我觉得它不再必要了。这不是很疯狂吗?从 12 年前到今天,真是太有趣了。

Yeah. All right. The genomechanical Turk I think is shutting down. Like I don't think it's necessary anymore. Isn't that crazy? It's just so interesting from like 12 years ago to today.

Tulsee

你听说有很多变化吗?

Have you heard lots have changed?

Host

好的。那么,换个方向,你有一个框架,我认为对人们会很有帮助,关于在 AI 发展程度方面要提前构建多少,以及你的产品应该假设 AI 会达到什么程度。你称之为 60-20-20 规则。解释一下。

Okay. So, going in somewhat of a different direction, you have this framework that I think is going to be really helpful to people around how far ahead to build in terms of how far AI is and how far your product should assume AI is going to be. You call it the 60 20 20 rule. Explain that.

Tulsee

我认为这里的数字可能是可替换的,但论点是一样的,那就是当你今天在构建,作为产品开发者使用这些模型构建时,重要的是要记住这些模型在不断变化。所以你今天拥有的能力不会是明天或后天的能力。因此,你实际上想要构建的功能不一定总是今天就能工作,对吧?所以通常当谈到传统创业智慧时,我认为你只是构建一个能工作的 MVP,对吧?但我认为你实际上想要构建的是,比如你构建的 60% 现在就能工作,你的原型至少能运行,然后有一组大约 20% 的功能,你觉得它不完美但足够好,我能看到模型在不久的将来可能达到这里,然后这里是我希望模型达到的地方。这里我看到了希望的曙光,我觉得如果我能突破这个,那将是一个惊人的产品体验。我认为如果你以这种方式构建,你是在构建一个今天就能工作的产品,但你也在为随着模型变得更好而构建神奇的产品体验做好准备,并且随着模型成长而成长。

I think the numbers are probably fungible here, but I think the thesis is the same, which is I think as you're building today and as you're a product developer building using these models, I think the important thing is to keep in mind that these models are consistently changing. And so the capabilities that you have today will not be the capabilities you have tomorrow or the next day. And so you want to actually build features that aren't necessarily always working today, right? So typically when you talk about traditional startup wisdom I think you're just like build an MVP that works right but I think what you want to do is actually build such that like 60% of what you've built works now and your prototype is at least functional right then there's a set of maybe the 20% where you're like it's not perfect but it like works well enough and I can see how the models might get here in the very near future and then here's where I want the models to be. And here's I see glimmers of light and I feel like if I could crack this, like this would be an amazing product experience. And I think if you build in that way, you are building a product that works today, but you're setting yourself up to build a magical product experience as the models become better and also to grow as the models grow too.

Host

我听过很多关于提前构建多少的建议,最常见的是构建比模型将要达到的水平提前三个月。这感觉更实用,因为你不能只赌未来。你还需要今天发布一些很棒且能工作的东西。

I've heard a lot of advice along these lines of how far ahead to build and the most common is just like build like three months ahead of where the models are going to be. And this feels like a lot more practical where you can't just bet on the future. You also need to ship some things today that are great and working.

Tulsee

是的,就是试图找到那个平衡,对吧?而且我认为,作为从事模型工作的人,我个人也觉得很难准确告诉你模型三个月后会是什么样。所以我认为更多地使用你自己的代理指标,比如我的产品在哪里?我在哪里看到希望的曙光?我在哪里看到它弥合差距,我认为这是一种更具体的方式来问同样的问题。

Yeah, it's trying to find that balance, right? And I think also I find personally hard to know where the models as someone who works on the model to tell you exactly where the model is going to be three months from now. And so I think using more of your own proxies of like where is my product? Where do I see glimmers of light? Where do I see it kind of bridging that gap I think is a more concrete way to hopefully like ask that same question.

Host

这让我想起另一种思考如何在产品团队中创建产品赌注组合的方式。过去的方式是,有一些雄心勃勃的大胆赌注,但也有很多低风险、可能推动指标的东西,你想把它们混合在一起,这就像是那个的新版本,其中

This reminds me of just another way of thinking about how to kind of create a portfolio of product bets on a product team. Like the way it used to be is like have some ambitious bold bets but also have a lot of stuff that's very low risk likely to move the metric and you want to mix them together and it's like this is the new version of that where it's

Tulsee

一个关于你如何思考的投资组合

a portfolio of how you're thinking about

Host

就像这个在六个月内会奏效,我向你保证

of just like this will work in six months I promise you

Tulsee

另一件有趣的事情是,我认为思考如何确保你的产品对模型变得更好具有敏捷性。我认为我们有时在客户身上看到的另一件事是,他们围绕模型当时的状态构建了一个非常僵化的系统,而现在模型已经变得好多了,对吧?那么你如何从模型当时的状态转变为能够适应并持续迭代它,你知道

the other thing that's interesting too is like I think thinking about how you make sure that your product is agile to the model getting better. I think one thing we've also seen with customers sometimes is like they've built a very rigid system around for example where the model was and now the model has gotten so much better right so how do you go from where the model was to like being adaptable to kind of consist continuing to iterate it with you know

Host

是的,我听到的常见建议是,模型会把你的 harness 当早餐吃掉,你构建的所有这些东西、所有优化都会消失,因为模型变得更聪明,他们不得不拿走所有这些。我认为 Boris Cherny 分享了一些建议,就像偶尔摆脱你所有的技能,看看会发生什么,因为现在它往往和你内置的所有这些东西一样好。

yeah the common kind of advice I hear is that just the model's going to eat your harness for breakfast that all these things you built all these optimizations are just going to they'll go away because the model gets smarter and they kind of have to take all these things away there's some advice that I think Boris Cherny was sharing of just like get rid of all your skills once in a while and just see what happens because often it's now as good as all these things you've built into it.

Tulsee

是的。或者就像持续重新测试,对吧?因为你甚至可能因为试图将新模型强行套入现有结构而阻碍了从新模型中获得的东西。

Yeah. Or just like like re consistently retest, right? Because you you might be even hampering what you're getting out of the model out of a new model because you're trying to force fit it into your existing structure.

Host

让我问你一个完全不相关的问题,因为我认为这可能有助于人们理解这一点。有一个术语 harness,人们经常听到这个想法,有一个模型,有一个位于模型之上的 harness。理解 harness 的最简单方法是什么?

Let me ask you a completely unrelated question just because I think this might be helpful for folks to understand this. There's this term harness that people often hear this idea of there's a model, there's a harness that sits on top of the model. What's the simplest way to understand what a harness is?

Tulsee

是的。我的意思是,我认为 harness 本质上是模型的包装器,对吧?所以,把它想象成系统指令、工具、技能、模型可以访问的上下文。以及它如何利用这些东西,对吧?所以,你基本上想要做的是给模型一个操作环境。而这正是你的 harness 所构建的。

Yeah. I mean, I think of a harness as like the wrapper around the model essentially, right? So, think of it as like what are the system instructions, the tools, the skills, the context that the model has access to. And how can it leverage those things, right? And so, what you want to basically do is give the model an environment in which to operate. And that's really what your harness builds out.

构建AI框架 Building AI Harness

Tulsee

所以本质上,它是一个系统提示,加上它能访问的工具来做事,再加上它能访问的其他上下文、其他数据、上下文信息,以及技能。对吧?所有这些元素组合在一起,就构建出了你的 harness。

So essentially it's a system prompt plus tools it has access to to do things, and then whatever other context, other data, context information that it has access to, and skills. Right? Like all of these things kind of come together to build out your harness.

Host

太棒了。好的,这非常有帮助。那么我还想听听你的想法:显然变化很大,很多人都在试图弄清楚如何在这个新世界中茁壮成长。你是如何提升自己并帮助你的团队更好地理解他们需要擅长什么?你帮助团队发展和在这个新世界中生存并茁壮成长的方式有哪些?

Awesome. Okay, that is very helpful. So one other thing I wanted to get your thoughts on: there's a lot of change, obviously, a lot of people are trying to figure out how to thrive in this new world. How are you upskilling and helping your team get better at and kind of understand what they need to be good at in this new world? What are just ways you're helping your team develop and kind of survive and thrive in this?

团队技能提升与信心建设 Team Upskilling and Confidence Building

Tulsee

生存并茁壮成长。是的。我的意思是,老实说,我觉得在很多方面其实是我的团队在帮助我的团队。嗯,我认为当你看看我团队的产品经理时,你会发现有些人真的投入其中,在利用我们的内部工具方面积累了很多技能,他们经常摆弄模型。而且我认为他们对团队其他成员来说是宝贵的资产,因为他们构建了其他人可以使用的技能。我们开始做得更多的一件事是,如何分享你构建的、在日常工作中给你带来价值的技能给其他人,对吧?这样我们就能构建一个完整的资源库来帮助产品经理团队。那些在审视数据方面变得更强、更自信的产品经理,对吧?真正地查看数据并理解更长的追踪和轨迹。我们如何与其他产品经理举办研讨会,以便我们共同建立更多的信心?所以对我来说,很多都是关于建立信心。我们如何建立每个人对我们拥有的工具的信心,以及他们有效利用这些工具的能力。我认为这是我们正在团队中越来越多地尝试建立的东西。

Survive and thrive. Yeah. I mean, honestly, I feel like my team is helping my team in a lot of this actually. Um, I think when you look at the PMs on my team, you see some who have really leaned in and built a lot of skill in leveraging our internal tools, and they're individuals who play around with the models a lot. And they actually are, I think, amazing assets to the rest of my team because they build skills that others can use. And one thing we've been starting to do more of is like, how do you share a skill that you've built that is giving you some value in your day-to-day job with others, right? So that we can actually build out an entire repository that helps the PM team. PMs who have gotten stronger and more confident in eyeballing data, right? And really looking at it and understanding longer traces and trajectories. How can we run workshops with other PMs so that we kind of all collectively build more of that confidence? And so for me, a lot of it is about confidence building. How do we build everyone's confidence in the tools that we have and their ability to leverage them effectively. And I think that's something that we're trying to build more and more across the team.

Host

那么再深入一层,具体是什么呢?你做了什么?你说到了研讨会,就是让擅长某件事的产品经理给其他产品经理开研讨会。除此之外,还有什么非常实际的做法来帮助人们更好地掌握这些?

So going one level deeper there, what is it? What is it you do? So you said workshops, there's like PMs that are good at a thing, teach a workshop to the rest of the PMs. Is there anything else just like very practically that you do to help people get better at these things?

Tulsee

实际上,我认为要想在这些方面做得更好,最重要的是——我这么说是因为我知道自己刚开始这样做,但还没有完全做到——实际上是给人们时间和空间。我认为当你只是不停地冲刺时,很难创造出空间来真正退后一步,确保你熟悉这些工具。但一旦你做到了,它会在另一方面加速你 10 倍,对吧?所以很多工作就是推动团队说,嘿,我们能不能利用 Jet Ski(我们的内部工具)来做这个?嘿,我有一个技能用来做这个,你想试试吗?实际上就是建立这样一套技能。而且实际上,我认为有时最实际的做法就是实际构建一个技能来做某事——当我说技能时,我不是指人的技能,而是指做这件事的技术技能——然后这个技能被分享给其他人,这建立了更多信心,然后导致其他人构建另一个技能或改进那个技能,现在你就有了这个迭代飞轮,我认为这是好的。

The biggest thing I would say actually that is needed to get better at these things—and I say this knowing that I am starting to do this but we are not quite there yet—is actually giving people the time and space. I think when you're just sprinting constantly, it is hard to create the space to actually take a step back and make sure you know the tools well. But once you do, then it accelerates you 10x, right, on the other side. And so a lot of it is about pushing the team to say like, hey, could we leverage Jet Ski, our internal tool, to do this, right? And hey, I have a skill that I use to do this, do you want to try? And actually building that kind of set of skills, if you will. And actually, I think sometimes the most practical thing to do is actually build a skill to do something—and when I say skill, I don't mean like the human skill, I mean like the technical skill to do it—and then that skill gets shared to others, that builds more confidence, that then leads to someone else building another skill or improving that skill, and now you have this iteration flywheel which I think is good.

20%时间与工作坊 20% Time and Workshops

Host

谷歌还在做 20% 时间吗?因为那感觉像是花时间做这些事情的好地方。现在还是这样吗,还是不如以前了?

Does Google still do the 20% thing? Because that feels like a really good place to spend time on this stuff. Is that still a thing or less so these days?

Tulsee

其实我不知道。我确实觉得我所有的产品经理都在像 150% 那样工作。所以也有这个因素。

I don't know actually. I do feel like all my PMs are working like 150%. So there is also that.

Host

好吧。所以你可以做 100%,从 150% 中拿出你的 20%。

Okay. So you could do 100, you get your 20% out of the 150%.

Tulsee

是的。或者你可以做 170% 吧。不过,我认为我们正在做的一件事是 11 月的一个为期两天的研讨会,我们将把所有产品经理从日常工作中抽出来两天,真正给他们空间来构建和摆弄我们拥有的工具。我认为即使给自己那样的思维空间,说“嘿,这是我做过的一件事”。我听到人们分享并且自己也开始采用的另一个经验法则是:如果你必须做某件事两次,你需要自己做吗?对吧?或者你能外包吗?有些事情,是的,显然像写两次策略,你可能还是应该自己写。但还有其他方面,比如,这有多可复制,我能用模型多少来帮助我完成这个任务?

Yeah. Or you could do 170% I guess instead. But no, I think actually one of the things we're working on is actually a two-day workshop in November where we're going to actually pull all the PMs out of their day job for two days and just actually give them the space to build and play with the tools that we have. I think you just giving yourself even that mind space to be like, hey, here's something I did. The other rule of thumb that I've heard people share and I've been starting to adopt myself is like, if you have to do something twice, do you need to do it yourself? Right? Or can you outsource it? And some things, yes, obviously like writing a strategy twice, you should probably still be writing it yourself. But there's other aspects of this where it's like, actually, how replicable is this and how much can I use the models to help me with this task?

AI新习惯 New Habits with AI

Host

是的,我一直有这样一个想法:最重要的新习惯之一是在刺激和反应之间插入一个步骤。比如说,我有一件事要做,然后我就去做。你必须插入这个问题:AI 能帮我做这个吗?然后尝试。我发现过去几个月我实际上更强烈地建立了这个习惯,它带来了巨大的不同,因为然后你开始构建所有这些小的工作流程和自动化。我做了很多 Grok 机器人操作,就像一旦你意识到你应该越来越多地这样做,你真的开始从中受益,而不是像我不知道该怎么用它。

Yeah, there's this thought I've been having that one of the most important new habits to build is to insert between stimulus and response. Say, I have a thing to do and then I'm going to go do it. You have to insert this question: Can AI do this for me? And then try. And I find that like I've actually built that habit much more strongly over the past few months and it makes a huge difference because then you start building all these little workflows and automations. I'm doing a lot of Grok botting and it's just like once you realize you should be doing that more and more, you really start to get the benefits of this versus like I don't know what to do with it.

Tulsee

你知道我发现的是,你真的只需要一个效果很好的例子就能改变你的心态。我认为至少对我来说就是这样,我觉得很难摆脱只做我知道怎么做的思维模式,直到我开始说,哦,实际上,就像你说的,设定那个中间期,实际上,模型能帮我做这个吗?当你拿一个实例这样做,然后你说,实际上,这对我真的很有帮助,你开始说,哦,实际上,现在我要试着把这个扔给模型,现在我要试着把这个扔给模型。因为你对它能为你做什么的信心增长了。

And you know what I found is like you really just need one example where that works really well to change your mindset. And I think that's what happened to me at least is like I feel like it was hard for me to get out of my mindset of just doing what I knew how to do until I started being like, oh, actually, to your point, like set that interim period of actually, can the model help me do this? And the minute you take one instance to do that and you're like, actually, this was really helpful for me, you start being like, oh, actually, now I'm going to try throwing this at the model and now I'm going to try throwing this at the model. Because your confidence in what it can do for you grows.

Host

是的。是的,这正是我的经验。一旦你看到它能做什么,那个习惯就强化了,就像神经通路开始建立,哦对,等等,让我们看看 Grok Bot 能不能帮我做这个。让我沿着这些思路问你一个问题。我最近请了 Josh Woodward 上播客。我问了他同样的问题。我很好奇你的答案会是什么。感觉现在每个推出的 AI 助手都在获取所有这些丰富的谷歌数据,我的 Gmail、我的谷歌日历、我的文档。你注册,连接你所有的谷歌数据,用它做所有这些神奇的事情。有 Grok Bot、Muse、Instinct、dots。现在,谷歌还没有人们真正——我知道有个 Spark,我想,它试图成为这个。

Yeah. Yeah, that's exactly my experience. Once you see what it can do, that habit reinforced like the neural pathway starts to build of like, oh yeah, wait, let's see if Grok Bot can do this for me. Let me actually ask you a question along these lines. I had Josh Woodward on the podcast recently. I asked him the same thing. I'm curious what your answer would be. It feels like every AI assistant launching these days is taking all this juicy Google data, my Gmail, my Google calendar, my docs. You like sign up, you connect all your Google data, does all these magical things with it. There's Grok Bot, there's Muse, there's Instinct, dots. Now, Google does not yet have anything that people get really—I know there's Spark, I think, which is trying to be this.

谷歌延迟发布 Google's delayed launch

Host

我很好奇,为什么 Google 还没推出什么惊艳的东西,反而是让别人拿走我所有的 Google 数据,用它做出这些神奇的事情。

I'm curious just why Google hasn't launched something awesome yet, versus letting everybody take all my Google data and do these magical things with it.

Tulsee

嗯,你看,我们希望用 Google 的数据来驱动神奇的体验,我觉得这是真的。至于“惊艳”长什么样,我想说敬请期待。团队正在做一些了不起的东西,所以我对这件事能走到哪里非常兴奋。我们也在琢磨的,一部分就是如何到用户所在的地方去,在哪里构建这种体验,以及它应该是什么样子。我愿意相信,在我们的 Google 产品里,已经有不少很棒的体验了,但我觉得你说得对,这方面我们还有很多工作要做。

Yeah, I mean, look, we want Google data to power magical experiences. I think that is true. In terms of what awesome looks like, I think stay tuned. The team is working on some amazing things, so I'm really excited about where this can go. Part of what we're trying to figure out also is where to meet users where they are, where to build that experience, and what that looks like. I like to think there are already a number of awesome experiences you can already have in our Google products, but I think you are right that there is more work we need to do here on this front.

Host

好,这跟他说的差不多,就是“关注这个领域”。

All right, that was similar to what he said, which is watch the space.

Tulsee

关注这个领域。

Watch the space.

Host

而你们所有人的优势,跟 Apple 类似,就是一旦你做到了,分发优势非常强,事情可以很快地变得非常好。

And the advantage you all have, similar to Apple, is once you get there, the distribution advantage is very strong, and things can go really well really quickly.

倦怠与内卷跑步机 Burnout and the treadmill of doing more

Host

好,回到这个工作量的问题。我一直在跟更多嘉宾聊的一个想法是:正因为 AI 让我们能做更多,所有人就都在做更多,于是出现了大量倦怠,因为大家会想,糟了,我得去给我的智能体解阻塞;我本可以做更多;这些事本可以都在发生。而且上面还有压力:多做点,再多做点,AI 能让我们做更多。你在团队里有没有做过什么,帮助大家避免倦怠,不要被卷进这台永无止境的“做更多”的跑步机?

Okay, so coming back to this workload question, something I've been trying to talk with more guests about is this idea that just as AI allows us to do more, everyone's just doing more, and there's a lot of burnout that happens because it's just like, oh no, I got to unblock my agent. Oh, I could be doing so much more. All these things could be happening. And there's pressure from above: do more, do more. AI can let us do more. Is there anything you've done on your team to help people avoid burnout, not get hooked on this never-ending treadmill of doing more?

Tulsee

我觉得一部分原因其实是,我们现在确实需要做更多,因为竞争很激烈,而且这是历史上非常罕见的时刻,一项全新的技术能带来这么多可能。所以多少有点:好吧,现在正是该全力长跑的时候。但我也想到那条推文,马拉松和冲刺合在一起,那个词怎么说来着?

And I think part of it is actually we need to do more now, because the competition is fierce, and it's this really rare time in history when a whole new technology enables so much more. So there's a bit of like, okay, this is actually the time to go really long. But then there's also this tweet I think about, what's the word for a marathon and a sprint?

Host

对,马……冲刺。

Yeah, mar... sprint.

Tulsee

拼起来的。

Parent.

Host

没错。

Exactly.

Tulsee

嗯,sprint 加 marathon。嗯,对,我觉得,如果你问我的团队,他们大概会说:喂,我们工作量很大。我觉得现在团队里每一个人都在把所有的时间投进去,无一例外。所以我确实觉得,这是我正在问自己的一个问题:我们怎么为团队创造一个可持续的工作环境?所以我也在思考这件事,敬请期待。但有一点我要说,至少对我个人而言,倦怠不只是来自工作量大,我觉得它来自两个地方。一是当你感觉事情反复折腾、混乱无序的时候,我觉得那会带来倦怠。另一个带来倦怠的是你不喜欢的工作。当你觉得自己做了很多,却像是苦力活或者浪费时间,我觉得那会带来倦怠。当你在做自己热爱或兴奋的事情时,是的,会很累,我们也该想办法可持续,但我觉得它也会带来更多快乐。所以接着我们之前聊的这些工具能怎么帮忙,我其实挺乐观:如果我们把这件事搭好,这些工具反而能缓解倦怠,因为它们能去掉你工作里那些苦力活或令人沮丧的部分,让你有更多时间思考、更多时间创造、更多时间做你真正想做的部分。希望这在倦怠问题上能带来一点解锁,尤其是当你在这份工作上投入这么多小时的时候。

Um, sprinton. Um, yeah, I think, I mean, if you ask my team, I think they would say, hello, we are working a lot. I think every single person on the team right now is putting in all of the hours, across the board. And so I do think this is an area that I'm actually asking myself about: how do we create a sustainable work environment for the team? So stay tuned as I also think about this. But one thing I'll say, at least for me personally, is I think burnout comes not just from working a lot. It comes from, I think, two places. One, I think it comes from when you feel like there's churn and chaos. I think that introduces burnout. The other thing that introduces burnout is work that you don't enjoy. When you feel like you were doing a lot of work and it feels like grunt work or a waste of time, I think that introduces burnout. When you are working on something you're passionate about or excited about, yes, it is exhausting, and we should figure out sustainability, but I think it also brings more joy. And so I think, actually, to our previous conversation about how these tools can help, I'm hopeful that if we can set this up well, these tools actually help with burnout, because they remove some of the grunt work or the frustrating parts of your job and actually enable you to spend more time thinking, more time being creative, and more time doing the parts of the job you actually want to do. And that, hopefully, is a bit of an unlock when it comes to burnout, especially if you're spending this many hours on the job.

Host

对,我做过一个关于科技从业者感受的调查。有一点很清楚:工时更长不等于倦怠。很多人正以职业生涯中最拼的状态工作,但他们也正享受着职业生涯中最快乐的时光。太有意思了。他们在构建东西,能做更多事,尤其是产品经理。你不用再那么干等着工程师。你就是在构建、在做原型、在跟有意思的客户聊、在思考有意思的问题。

Yeah, I did this survey of how tech workers are feeling. And one thing that's clear is more hours does not equal burnout. A lot of people are working the hardest they've ever worked, but they're having the best time they've ever had in their career. It's so fun. They're building. They're able to do a lot more, especially PMs. You don't have to sit around and wait for engineers as much. You're just building, you're prototyping, you're talking to interesting customers, you're thinking about interesting problems.

Tulsee

我觉得团队里很多人,我愿意相信,他们留在这里就是因为想要这个。他们真的被这件事驱动、被它激励。所以对我来说,问题就是:好,我们怎么把他们工作里那些容易引发倦怠的摩擦部分拿掉?那具体该怎么做?

And I think a lot of the team, I like to think, is here because they want that. They're really motivated and excited by that. And so for me it's about, okay, how do we pull out the friction parts of their work that are likely to induce burnout? And what does that look like?

Host

同时也要找到那种有意义、他们看重、愿意花很多小时去做的工作,而不是不得不做的工作。

And also find work that is meaningful and that they value and are excited to spend lots of hours on, versus they have to.

Tulsee

而且这对每个产品经理来说也不一样。如果你从我团队里挑两个产品经理,我觉得他们告诉你的、关于自己为什么每天来上班、从什么里获得价值,会有实质性的不同。所以我觉得,作为组织负责人,我的一部分工作也是去思考每个人各自的动机是什么,以及我们怎么确保他们在这样的安排里也能成功。

And that's different for every PM, too. If you pick two PMs on my team, I think they would tell you meaningfully different things about why they come to work every day and what they find value out of. And so I feel like part of it for me as an org leader also is to think about what motivates the individual and how do we make sure that they are set up for success in that setup too.

AI垃圾与AI文档 AI slop and AI-written docs

Host

现在人们面临的另一个挑战是 AI 垃圾内容。基本上就是大量由 AI 写成的文档被发给他们,要在无穷无尽的文档里翻找,变得非常烦人。以前很多问题在于花时间写文档,现在问题变成了读文档,因为全是 AI 写的东西。你们团队有没有建立什么流程或政策,来帮助避免这种 AI 生成内容在团队间泛滥?

Another challenge people have these days is AI slop. Basically lots of docs being sent to them that are written by AI, and it's becoming very annoying to have to sift through endless docs. It used to be that a lot of the issue was spending time writing docs, and now it's like reading docs is the problem, because it's just all this AI-written stuff. Do you have any kind of, I don't know, processes or policies you put in place in your team to help avoid this rise of AI-written content across the teams?

Tulsee

我们经常讨论的一件事是:这些文档在哪里有用。比如,我觉得 AI 生成的文档真正有帮助的地方,是在你发布某个东西之后,假设两周后你想拉一份最新情况:谁在用、怎么用、反馈是什么。这种文档正是 AI 擅长的,因为它做这件事比你快得多。它会去从各种仪表盘和指标里拉数据,给你一份更新,拉出优势和劣势,还会从 Twitter 分析、Reddit 之类的地方抓内容。所以我们讨论的一件事就是:要有意识地分清,文档里哪些内容是 AI 帮你产出的、确实有用的,哪些内容应该来自你自己。哪里是洞察?你应该提供的是洞察,模型应该提供的是数据。我觉得如果你能把这两者分开,希望就能在这里找到合适的平衡,把好处发挥出来。

One of the things that we talk about a lot is where are these docs useful. For example, I think where AI-generated docs are actually quite helpful is after you launch something, and let's say two weeks later you want to pull the latest on who's using it, how are they using it, what's the feedback. That's the kind of document AI is fantastic for, because it will actually do that so much faster than you. It will go and pull from various dashboards and metrics, it will give you an update, it'll pull strengths and weaknesses. It'll pull from Twitter analysis and Reddit or whatever. And so I think one of the things we talk about is: be intentional about where AI is helping you drive content in the doc that is helpful, versus where is it content that should come from you. Where is it insight? What you should be providing is the insight. What the model should be providing is the data. And so I think if you can separate those two things, then hopefully you find the right balance of the benefit here.

AI对产品管理的影响 AI's Impact on Product Management

Tulsee

我不会——我想说的另一点是,因此我们在如何培养正确的技能来获取正确的数据上投入了更多,以便我们能很好地构建这些工件,从而 hopefully 为创造性思维和创造性写作创造更多空间,我们希望更多人花时间做这些事。

I wouldn't— I would say the other thing is we're investing a lot more as a result of that in how do you create the right skills to get the right data, so that we can structure these artifacts really well, that then hopefully creates more space for the creative thinking and the creative writing that we want more people to spend time doing.

Host

这其实正是我想问的。所以就产品经理的角色而言,AI 显然在原型制作上产生了巨大影响。你现在可以无休止地做原型,而不需要设计师或工程师来构建东西。所以这显然是 AI 改变该角色的一个方面。你刚才提到的另一个领域是数据。产品经理可以做更多的数据分析。那么除了原型制作之外,你认为过去几年 AI 最大改变产品经理工作的领域是什么?

That's actually something I was wanting to ask about. So in terms of the PM role, AI has clearly made a big dent on prototyping. You can prototype till the cows come home now versus having a designer or an engineer build a thing. So that's clearly a place the role has changed with AI. The other area which you just mentioned is data. PMs can do a lot more data analysis. What would be like behind prototyping? What are the biggest areas AI has most changed the job of a PM in the past couple years would you say?

Tulsee

是的,原型制作是一个很好的例子。我认为你说得对,数据科学可能——我觉得你更有能力随时掌握数据来做出数据驱动的决策。我还认为,实际上,回到我们之前的对话,你在社交媒体上经常看到这类帖子,比如“空白是新的 PRD”,不管空白是什么——原型是新的 PRD,评估是新的 PRD,数据是新的 PRD。但我认为这实际上有意义地表明,产品经理已经从仅仅写下需求然后假设别人——无论是数据科学家、工程师、研究员还是设计师——会弄清楚背后的实质,向前迈进了一步。你已经把这一步推进得更远,说你的工作不再只是写下一组需求。你的工作实际上是更进一步,让需求有实质内容,无论这种实质以什么形式呈现。

Yeah, prototyping is a great one. I think you're right that data science is probably— I think you are more empowered to have the data at your fingertips to make data-driven decisions. I also think actually, to our earlier conversation, you see these on social a lot, these kinds of posts of like, oh, blank is the new PRD, whatever blank is—a prototype is the new PRD, evals are the new PRD, data is the new PRD. But I think actually what that meaningfully shows is you've moved the PM from being someone who just writes a set of requirements down and then assumes that somebody else—whether that is a data scientist or an engineer or a researcher or designer—will figure out the meat behind it. You've moved that one step further to say your job is no longer just writing down a set of requirements. Your job is actually going one step further to have some heft to that, whatever format that heft takes.

Host

太棒了。好的,这与我所听到的很多内容一致。Tulsi,你还有什么想分享的吗?在我们进入非常激动人心的闪电轮之前,关于我们讨论过的内容,你还有什么想强调或想说的吗?

Awesome. Okay, that aligns with a lot of what I'm hearing. Tulsi, is there anything else that you wanted to share? Is there anything else you think you want to double down on that we've talked about that you want to say before we get to our very exciting lightning round?

Tulsee

没有,我觉得很好。

No, I think that's great.

快问快答:书籍 Lightning Round: Books

Host

好的,我们做到了。那么,Tulsi,我们到了非常激动人心的闪电轮。我有五个问题问你。你准备好了吗?这里会播放一些后期制作添加的音乐。

Okay, we did it. Well, with that, Tulsi, we reached our very exciting lightning round. I've got five questions for you. You ready? There's some music that plays over this that's added in post-production.

Tulsee

你是说听起来像这样吗?

You mean does it sound like this?

Host

正是那样。我们应该把它和你的鼓演奏对齐。好的,第一个问题。你发现自己最常向别人推荐的两三本书是什么?

It's exactly that. We should align it with your drum playing. Okay, first question. What are two or three books that you find yourself recommending most to other people?

Tulsee

好的。那么,一本实际上是——我认为严格来说它可能被称为中篇小说,但不知道你是否读过《呼吸》,那是一组科幻故事。其中有一个特别的故事,讲述了一群人试图在平台决定弃用后仍保持一组数字头像存活。我喜欢那本书,因为我认为实际上,尤其是现在,我们正在思考个人将如何与智能体互动,我们与技术建立什么样的关系,以及作为人类如此紧密地与数字世界互动意味着什么。我认为这是一个写得非常优美的故事,而且它引发了许多开放性问题。所以我认为这是一本。另一本在产品经理圈子里可能更标准的是《创新公司》,我很喜欢。那是皮克斯的故事。但我认为它实际上很大程度上讲述了如何创造良好的文化,这是我经常思考的事情,比如如何创造一种真正鼓励人们跌倒后重新站起来的文化?如何创造一种鼓励人们用想法创新并推动前沿的文化,这显然是我们这里经常思考的事情。我最近还非常喜欢《Acatar》系列,但那是什么系列?

Okay. So, one actually is— I think it's actually technically a novella is probably what it's called, but I don't know if you've read Exhalation, which is a set of science fiction stories. And there's a particular story in that that actually speaks to a set of individuals who are trying to keep a set of digital avatars alive even after the platform decides to deprecate. And I love that book because I think actually, especially now in a world where we're thinking about how individuals are going to act with agents and what kinds of relationships we're building with technology and what does it mean to be human and interact with the digital world so closely. I think it's just a really beautifully written story and I think it also evokes a lot of open questions. So I think that's one. Another that's maybe more standard in PM circles is Creativity Inc., which I love. It's the Pixar story. But I think it actually speaks a lot to how do you create good culture, which is something I think a lot about, like how do you create culture that actually encourages people to fall down and get back up? How do you create culture that encourages people to be creative with their ideas and push the frontier, which is obviously something we're thinking a lot about here. I also just have been really into the Acatar series recently, but that's— what series?

Host

那是浪漫奇幻。所以,对于任何只想找点有趣读物的人,你应该读《玻璃王座》或《荆棘与玫瑰的王庭》。

That's romantic fantasy. So, for anyone who's just looking for something fun to read, you should read The Throne of Glass or A Court of Thorns and Roses.

Tulsee

我喜欢这里推荐的多样性。

I love the spectrum of recommendations in this.

快问快答:影视 Lightning Round: Movies and TV

Host

好的。你最近非常喜欢的电影或电视节目是什么?

Okay. Favorite recent movie or TV show you've really enjoyed?

Tulsee

你知道,实际上,这是一个很难的问题,因为我最近看了很多糟糕的电影和电视节目,这很不幸,我不会推荐。所以,就这样。但我真的很喜欢 Apple TV 上的《人生切割术》。在风格和内容上都非常出色。所以那可能是首选。

You know, actually, this was a really hard question because I've watched a lot of bad movies and TV shows recently, which is unfortunate, actually, that I would not recommend. So, there's that. But I did really enjoy Severance on Apple TV. Incredibly well done, both stylistically and content-wise. So that's probably top of the list.

快问快答:AI产品 Lightning Round: AI Product

Host

除了 Gemini,你现在最喜欢的 AI 产品是什么?

Favorite AI product right now other than Gemini?

Tulsee

Whisper Flow。我不知道你是否在用,但我有手腕问题,所以长时间打字有困难。因此 Whisper Flow 对我来说实际上是改变游戏规则的。

Whisper Flow. I don't know if you're using it, but I have wrist problems, so I have trouble typing for long stretches. And so Whisper Flow has been game-changing for me, actually.

Host

太棒了。如果你订阅 Lenny 的新闻通讯,你可以免费获得一年 Whisper Flow。插播,插播,插播。

Awesome. You get a year free of Whisper Flow if you become a subscriber of Lenny's newsletter. Plug, plug, plug.

Tulsee

哦,我喜欢这个。

Oh, I like this.

Host

这就对了。我也喜欢的产品的自然推荐。这就对了。

There you go. Organic recommendations of products I also love. There we go.

Tulsee

不,我喜欢 Whisper Flow。一直在用。按下功能键,开始说话,它就能神奇地转录一切。

No, I love Whisper Flow. Use it all the time. Press function, just start talking, and it's just magically transcribes everything.

Host

太神奇了。

Amazing.

快问快答:人生格言 Lightning Round: Life Mottos

Host

你有没有一个最喜欢的人生格言,在工作或生活中经常回想起来?

Do you have a favorite life motto that you often come back to in work or in life?

Tulsee

我想有两个不同的。一个可能更老套,那就是一切事情的发生都有原因。我认为这实际上是我个人的一大人生格言,就像你可以努力工作,你会尽力而为,但有时一扇门会关闭,另一扇窗会打开,你处于某个位置是因为那注定要在那里。不过,我认为与此相关的是,我喜欢经常和我交谈的人谈论“寻求机缘巧合”。我认为这实际上与第一个有关,但你应该总是努力让自己处于能够获得幸运的位置。而这样做就是让自己处于挑战自己的位置,让自己走出舒适区,尝试新事物。那些总是你更有可能获得幸运的地方。

Two that are different, I guess. One is maybe more cliché, which is everything happens for a reason. I think that's actually just a big life motto for me personally, which is like you can work hard, you will do your best, but sometimes a door will close and another window will open, and you're in a position because that is meant to be there. I think related to that though, I love often talking to folks that I talk to about seek serendipity. I think that actually relates to the first one, but you should always try to put yourself in a position where you have the ability to get lucky. And doing that is about finding yourself in positions where you're challenging yourself, where you're putting yourself out of your comfort zone, where you're trying something new. Those are always places where you are more likely to get lucky.

Host

有趣的事实,我大约 15 年前做过一个 TEDx 演讲,叫做《失去机缘巧合》。

Fun fact, I gave a TEDx talk called Losing Serendipity maybe 15 years ago at this point.

Tulsee

哦,那么我们是试图失去机缘巧合还是找到它?

Oh, so we're trying to lose serendipity or find it?

Host

它是关于我们如何失去它,这很可悲,我们需要生活中更多的机缘巧合。

It was about how we are losing it and that is sad and that we need more serendipity in our life.

Tulsee

我是机缘巧合的忠实粉丝。

I'm a big fan of serendipity.

Host

这个词真好。而且,那部电影也不错。

Such a great word. Also, the movie is pretty good.

Tulsee

你看过。好的。它像一部浪漫喜剧。

You've seen it. Okay. It's like a romcom.

职业反思与AI机缘 Career reflections and AI serendipity

Tulsee

但我觉得这就是我迄今为止看待整个职业生涯的方式,也是我今后会继续的方式。实际上,这很有趣,因为即使在使用这些 AI 工具时,我觉得那是另一件我——

But I think that's how I've looked at my whole career so far, and I think how I'll continue to. And actually, it's fun because even with using these AI tools, I think that's another thing that I—

Host

是的。AI 带来了太多意外之喜。最后一个问题。你的一位同事告诉了我关于你的另一个趣事:你是一名受过训练的印度古典舞舞者。

Yeah. So much serendipity with AI. Final question. One of your colleagues told me another fun fact about you: that you're a trained classical Indian dancer.

Tulsee

天哪。

Damn.

Host

而且你还导演过面向一万多人的文化节。

And you've directed cultural festivals for 10,000 plus people.

Tulsee

我不知道他怎么知道的,但没错,我可能告诉过他。

And I don't know how he knows that, but yeah, I probably told him that.

Host

好的。这个人说,这或许能解释你如何在开发前沿模型的混乱中保持如此冷静。嗯,我不知道,也许可以分享一些关于舞蹈,你生活中舞蹈的部分。

Okay. And this person said that they may explain how you stay so calm amidst the chaos of frontier model development. Uh I don't know, maybe share some about this dance, this dance part of your life.

舞蹈作为文化与个人锚点 Dance as a cultural and personal anchor

Tulsee

是的,我热爱舞蹈。我觉得那是难得的时刻之一,你真的无法想其他任何事情。你必须专注于动作和音乐。所以对我来说,这是一个让我的大脑在一段时间内转向不同方向的机会。嗯,所以我喜欢它。而且我觉得对我来说,它也是一个文化连接点。嗯,舞蹈是我遇到一些最好朋友的方式。呃,也是我继续与许多朋友社交的方式。我策划这个节日。我在俄勒冈州波特兰策划了多年,那是我的家乡。

Yeah, I mean I love dance. I think it's one of those rare moments where you really can't think about anything else. You have to focus on the movement and the music. And so for me, it's an opportunity for my brain to actually just have a different direction of travel for a period of time. Um, and so I love it. And I think for me also it is a cultural connection point. Um, dance is how I've met some of my best friends. Uh, it's how I continue to engage socially with so many of my friends. I plan this festival. I've planned it for years uh in Portland, Oregon, which is where I'm from.

Tulsee

呃,它叫印度文化节。嗯,印度节,我们呃,每年八月举行。事实上,这是我第一年没有参加这个节日,因为我有一个七个月大的儿子。呃,所以,谢谢。所以今年比平时更分心一些,但自从我三岁起就在那个节日上表演。嗯,然后组织它,它一直是我生活的重要组成部分,你知道,如何将舞蹈、文化和社区结合在一起?

Uh it's called the India Cultural Festival. Um India Festival, we uh it happens every August. And in fact, this was the first year I did not participate in the festival because I have a seven-month-old son. Uh so, thank you. So, it was a little bit more distracted this year than normal, but it's for since I I've been performing at that festival since I was three. um and then organizing it and it's just been a big part of my life of you know how do you bring dance and culture and community together?

趣事与听众互动 Fun facts and listener engagement

Host

Tulsee,你真是充满了趣事,我觉得还有很多我甚至不知道的

Tulsa, you're so full of fun facts that I feel like there's much more I don't even know that

Tulsee

我就是来分享趣事的。

I'm here for the fun facts.

Host

呃,最后一个问题。听众如何能对你有所帮助?

Uh final question. How can listeners be useful to you?

Tulsee

嗯,哇。呃,好问题。我是说,使用 Gemini,呃,使用 Argon,嗯,但也使用我们的各种产品。你知道,实际上我刚才在谈论音频。如果你是 Gboard 上的 Pixel 用户,试试我们的 rambler 功能。它很棒。你知道,在手机上打字很棒。如果你在使用文本转语音模型,如果你在使用对话功能,如果你在使用我们的 omni 模型,所以试试它们。给我们你的反馈,对吧?嗯,我们一直在寻找反馈。无论你是通过公司提供反馈,你知道,通过分享反馈,无论是你给我们发私信,嗯,还是公开发布,我认为我们希望更多人来给我们提供正确的反馈。

Um wow. Uh great question. I mean a use Gemini uh and use Argon um but also use use our smattering of products. You know, actually I was talking about audio just now. If you are a pixel user on Gboard, like try using our rambler feature. It's awesome. You know, typing on the phone is great. If you are um using uh text-to-speech models, if you're using dialogue, if you're using our omni models, so so try them. Give us our your feedback, right? Um we're constantly looking for it. whether you're giving that feedback uh via a company that you are you know sharing feedback via why whether it's you're sending us you know a DM um whether you're posting it publicly I think we want more people uh giving us giving us the right feedback

致谢收尾 Closing thanks

Host

呃,Tulsee,太棒了,你太棒了,非常感谢你来这里

Uh Tulsi that was awesome you're awesome thank you so much for being here

Tulsee

不,谢谢你邀请我,这很棒

No thank you for having me this was great

Host

这是我的荣幸,大家再见,非常感谢收听,如果你觉得这有价值,可以在 Apple Podcasts、Spotify 或你最喜欢的播客应用上订阅本节目。另外,请考虑给我们评分或留下评论,因为这真的能帮助其他听众找到这个播客。你可以在 lennispodcast.com 找到所有往期节目或了解更多关于节目的信息。下集见。

It was my pleasure bye everyone thank you so much for listening if you found this valuable you can subscribe to the show on Apple Podcasts Spotify or your favorite podcast app. Also, please consider giving us a rating or leaving a review as that really helps other listeners find the podcast. You can find all past episodes or learn more about the show at lennispodcast.com. See you in the next episode.

互动版:逐字朗读 + 针对本期提问 →