为金融业打造 ChatGPT：OpenAI 的企业级实践

今天我们请到了 Joseph Kim，他是 OpenAI 的 ChatGPT 金融负责人之一，以及我们的联合主持人、Latent Space 的编辑总监 Richard，来到演播室。这是一期特别节目。我们在纽约的 AI Finance 活动现场。Joseph 刚刚做了一场精彩的演讲。我想我们不想重复大部分内容，所以大家真的应该去看，但请用 30 秒简单介绍一下你讲的内容，然后我们会留下线索。大家可以去看演讲。

Today we have Joseph Kim, one of the heads of ChatGPT finance from OpenAI, and our co-host and head of editorial from Latent Space, Richard, in the studio. It's a special episode. We're here in New York at AI Finance. Joseph just gave a great keynote. I think we don't want to repeat most of the information, so honestly go watch it, but give like a 30-second of what you talked about and then we'll leave the breadcrumbs. You guys can go watch the keynote.

当然，我可以快速用 30 秒介绍一下我们刚才聊的内容。第一，我们刚刚发布了一款新产品，ChatGPT for financial services。这是一款面向企业的产品，真正试图把 ChatGPT 的强大功能、你在消费级产品中喜欢使用的工具和功能，同时加入企业层，以便支持公司特定的模板，以及像 Dilupa、S&P 等数据源。非常令人兴奋。我们一直在与设计合作伙伴摩根士丹利和 Evercore 合作，反响非常热烈。这是演讲的前半部分。演讲的后半部分则更具前瞻性，深入思考我们在模型改进方面看到的一些趋势线，并将这些趋势线延伸到未来，试着思考一下，到 2027 年中、2027 年底，这些模型将能做什么，这对于那些在模型之上构建产品的人意味着什么，以及你如何开始为几个月后即将到来的智能做规划。

Sure, yeah, I can give a quick 30-second intro to what we just chatted about. One, I would say we just released a new product, ChatGPT for financial services. This is an enterprise-focused product that's really trying to bring the power of ChatGPT, the tools and things that you really like to use in the consumer product, but also add in the enterprise layer so that firm-specific templates, sources like Dilupa, S&P, etc. And so really exciting. We've been working with our design partners Morgan Stanley and Evercore on that, and reception to that's been incredible. So that was kind of the first half of the talk. And in the second half of the talk, really much more forward-looking, so thinking deeply about some of the trendlines that we're seeing in terms of model improvements, and drawing those trendlines out to the future and trying to think a little about, okay, like what are these models going to be able to do at the end of mid-2027, end of 2027, and what does that mean for folks that are building products on top of that model, and how can you kind of start to plan ahead to the intelligence that's coming in just a matter of months.

我觉得这是个有趣的时刻，因为你实际上来自研究领域。你现在负责很多产品和工程，但请告诉我们，打造一款金融专用产品需要什么。你在演讲中提到，当你推出 ChatGPT work 时，用户有所增加。Codex 开始向大众开放。人们开始将其用于金融，现在你正在推出你的首款金融产品。金融推理数据是什么样的？金融专用模型需要哪些要素？

I think it's a fun time because you actually come from the research side of things. So you now lead a lot of the product and engineering, but tell us what it takes to make a financial-specific product. You mentioned in the keynote that there was sort of an uptick in users when you launched ChatGPT work. So, Codex started to become available to the masses. People start using it for finance, and now you're shipping your first finance product. What is like financial reasoning data like? What goes into finance specific models?

是的，这是个好问题。我要说的是，我们为金融推理分析所做的所有训练数据，都是在上游进行的，进入我们所谓的“大跑”，也就是主跑，最终形成类似 6 Astra 或 61 Astra 等模型。因此，即使我们向模型添加了大量金融数据，这些数据也会进入主模型，因此可供那些想要在其上构建的人使用。这对我们非常重要。在 OpenAI，我们真的希望这种智能不仅可用于我们内部以第一方方式构建的产品，也可用于那些想要在此基础上创建初创公司、建立业务的人，以及希望将其直接整合到公司中的金融机构。因此，我们将让所有这些智能都可访问。我想你稍微提到了产品和研究，我认为这是 OpenAI 的核心。内部，我们称自己为研究和部署公司。这意味着我们故意按顺序构建，研究优先，部署其次。因此，当我们思考最终改变人们知识工作方式时，我们按这个顺序工作，对吧？我们考虑模型能力、它们的发展方向、它们能实现哪些新事物。这是第一点。第二点，我们深入思考我们应该构建哪些产品功能和界面，以使模型智能更广泛地可及。我想你提到了我演讲中的一张幻灯片，展示了 OpenAI 内部 Codex 的采用情况。有趣的是，你可以看到软件工程的采用曲线，稳步上升。人们开始使用并将其融入日常工作。但如果你看金融、法律和销售，几乎为零，然后突然跳到 90%。如果你看那条线，看看跳跃时发生了什么，我们在二月份发布了 Codex 桌面应用，那时采用率就跃升了。因此，我认为这显示了共同开发这些东西的重要性，产品的目的是让模型智能尽可能多地惠及用户。

Yeah, it's a great question. One thing I will say is that all of the training data that we do for financial reasoning analysis, we do upstream into what we call the big run, which is the main run that becomes something like 6 Astra or 61 Astra, etc. And so, even though we're adding a bunch of financial data to the model, it's going into the main model, and so therefore it's available to folks that want to build on top of it. And that is something that's very important to us. At OpenAI, we really do want that intelligence to be accessible not only to our products that we're building in-house in first-party ways, but also for folks that want to build startups, build businesses on top of that, as well as financial firms that want to integrate that directly into their companies. And so, we will make all that intelligence accessible there. I think you touched a little bit on product and research, and I think that's really core to what we do at OpenAI. Internally, we call ourselves a research and deployment company. And so, what that means and we kind of structure that in order on purpose, like research comes first, and then deployment comes second. And so, as we think about, at the end of the day trying to change the way that people do knowledge work, we kind of work in that order, right? We think about the model capabilities, where they're headed, what kind of new things they enable. So, that's one. And then two, we think very deeply about the product features and surface areas that we should build to make that model intelligence accessible to more people. And I think you alluded to a slide in my keynote where we had this chart of Codex adoption within OpenAI. And it's funny because you can see the software engineering adoption curve, and it's just like up into the right pretty steady. People are picking it up and incorporating into their daily routines. But if you look at finance, legal, and sales, it's pretty much at zero, and suddenly it kind of jumps up to 90% and if you look at that line and try to see what happened around the time where it jumped up, we had released Codex desktop app in February and that was when that adoption jumped up. And so, I do think it kind of shows the importance of co-developing those things and the purpose of the product is to make that model intelligence accessible to as many users as possible.

那么，在 Codex 用于金融应用的数据增长中，你学到了哪些东西，然后应用到金融数据的后训练中？

So, in that uptick of data for using Codex for financial applications, what sort of things did you learn that you then applied into the post-training of financial data?

是的，这是个好问题。我想说，我可以谈谈我们如何构建 ChatGPT for financial services。这非常独特。我们还没有构建很多第一方产品。我想说 Codex 是我们最大和主要的一个，这是我们第一次开发一个产品，其中产品、模型硬度和研究都在一个屋檐下，这确实给了我们很大的灵活性来构建东西。我认为过去 6 个月左右的工作中最大的收获是，在企业中，你必须应对你所处行业的现实，并围绕它构建。我认为这一点很容易被忽视。如果你只是从第一性原理思考，你可以得出一个有意义的结论，谈论它，并用逻辑和理性来辩护。但如果你在开始构建时忽略了你所处的世界状态以及行业内的力量，无论是竞争力量还是人们有自己的目标。如果你不考虑这些，你就不会构建出正确的产品，对吧？所以，我们在 ChatGPT for financial services 中看到了这一点。我们考虑了很多。我举个例子。数据。数据是我们思考构建产品的重要部分。

Yeah, it's a good question. I would say, yeah, I could talk a little bit about how we went about building ChatGPT for financial services. It was very unique. We haven't built a whole lot of first-party products. I would say Codex was really our biggest and main one and it was the first time we developed a product where the product, the model hardness and the research was all kind of under one house and under one roof and it really gives us a lot of flexibility to go about building things. I think the biggest takeaway that I have from having worked on this for the last 6 months or so is in enterprise, you have to deal with the reality of the industry that you're in and go build around that. And I think that piece is easy to ignore. Like if you just think about it from first principles, you can come to a conclusion that makes sense and you talk about it and you can defend it with logic and reason. But if you ignore the state of the world that you're in when you start building and the forces that are at play within the industry, it's either competitive forces, people have their own objectives. And if you just don't think about that, you're not going to build the right product, right? And so, we saw that with ChatGPT for financial services. We thought about that a lot. I'll give you one example. Data. Data is like a huge part of how we thought about building the product.

数据是那种从第一性原理上很难为产品辩护的东西。当我在谈论构建和开发这个产品时，你会说：“嘿，我想把季度财报电话会议记录引入产品并打包。”人们首先会说：“看，商店里已经有 MCP 连接了。你打包这个能提供什么增量价值？”纸面上看很糟糕。他们可能已经有许可证了，所以会觉得付了两次钱。而且他们可能有 MCP。那你为什么要构建这个？现实是我们测试了所有这些 MCP。我们发现错误率非常非常高。我们连接的一些 MCP——一些大牌——错误率高达 40% 以上，对吧？所以 MCP 是那种纸面上看起来很好，但你必须去看数据，看看实际发生了什么。所以对于数据，我们说：“嘿，这真的很糟糕。我们大规模解决这个问题的唯一方法就是把数据引入内部，端到端地控制体验。”你额外支付的钱来打包数据是值得的，对吧？所以这是一个我们必须做出的产品决策，非常行业特定。它引起了客户的共鸣。客户喜欢它。他们首先提到的就是数据。所以它奏效了。

And data was one of those things where on first principles it was a little bit tough when I was talking about building and working on this product to defend because, you know, you say, "Hey, I want to go and bring quarterly earnings transcripts into the product and bundle it." And the first thing people are going to say is, "Look, there's an MCP connection in the store. What is the incremental value that you're providing by bundling that in?" On paper it looks bad. They might already have a license for it, so they might feel like they're paying twice for it. And they might have an MCP. So why are you building this? And the reality is we went and tested all these MCPs. And what we found was the error rates were really, really high. Some of the MCPs that we connected—some big names—had 40% plus error rate on their MCPs, right? And so MCPs are one of those things where it looks way on paper, but you have to go look at the data and see what's actually going on. And so for data we're like, "Hey look, this is really bad. The only way we can go about fixing this at scale is to bring the data in house, just control that experience end to end." And the extra money that you pay on top to bundle that data is worth it, right? And so that was a product decision that we had to make that was very industry specific. And it's been resonating with customers. Customers love it. The number one thing they call out is the data. And so it worked.

所以你在这方面取得了不错的成功。给我们讲讲技术方面的事情，对于那些为特定服务构建工具适配的人，对吧？你们出去改变了 Codex，做了后训练，做了任何你们做的事情来打造面向金融服务的 ChatGPT。如果有人要为另一个领域做同样的事情，比如法律、销售等等，那里的最佳实践是什么？你学到了什么？仅仅采用现成的东西并真正围绕它打造某个产品需要什么？

So, you had good success in this. Tell us about the technical side of things for people building a harness adaptation for a specific service, right? So, you guys went out, you changed Codex, you did post training, you did whatever you did to make ChatGPT for financial services. If someone was to do the same for another domain, say legal, sales, whatever, what are the best practices there? What did you learn? What goes into just taking off the shelf and actually making a certain product around it?

是的，我很喜欢这个问题。我在主题演讲中也试图提到这一点，但有趣的是，谈论后训练的东西很有趣，因为它是我们独有的，能让人兴奋，但我们构建的 95% 是绝对可以由初创公司或其他公司完成的事情。我们并不真的需要利用我们在模型中做出改变的能力来构建出色的产品，对吧？所以我认为提到这一点很好。我想说的是两方面。第一，我们构建的大部分产品都是基于 Codex——或者我现在应该说 ChatGPT——的插件架构和插件原语。这是我们投入了大量时间和精力的事情。OpenAI 是一家有趣的公司，我们在内部面临的问题与初创公司在 OpenAI 之上构建时面临的问题相同。所以，在很多方面，我的团队就像是一家在 OpenAI 技术之上构建的初创公司，对吧？因此，即使在内部，我们也需要找到可扩展的方法，让像我这样的团队能够站在巨人的肩膀上，在 Codex 之上构建，对吧？Codex 非常棒。它有很好的记忆。它有很好的技能。它是持久的。Codex 有很多美妙之处，但我们团队的论点是 Codex 能让你达到市场上成功产品的 80%，但缺少 20%。所以，我们需要找到某种方法让我的团队去构建那 20% 才能真正获胜，因为论点的另一半是，如果你只是拿 Codex 并把它作为金融产品发布，它会失败。所以那 20% 实际上是关键。所以，我想说的是，我们基于插件架构构建。它使用的是任何想要在 ChatGPT 之上构建的构建者都可以使用的相同原语。我在演讲中有一句话，但我真的认为将会有一个建立在 ChatGPT 插件之上的十亿美元业务，对吧？就像你将能够利用所有那些横向功能，然后在上面添加你的秘密武器。那个秘密武器可能是数据，可能是 UI 更改，可能是集成，但那个秘密武器真的很有价值，我认为那里有十亿美元的价值可以构建。

Yeah, it's a—yeah, I like the question a lot. I tried to allude to this in my keynote as well, but the interesting thing is, it's fun to talk about the post training stuff because it's unique to what we do and gets people excited, but 95% of what we built was something that absolutely a startup could do, another company could do. And we didn't really have to go leverage our ability to go make changes in the model to go about building fantastic product, right? And so I think it's a great call to kind of mention that piece. What I would say is twofold. One is most of the product that we built was built on top of the Codex—or I should say ChatGPT now—plugin architecture and plugin primitives. And that is something that we've been investing a lot of time and energy in. OpenAI is like an interesting company where we kind of face the same problems internally that a startup would face if they were to build on top of OpenAI. So, in a lot of ways, my team is kind of like a startup that's building on top of OpenAI technology, right? And so, even internally, we need to find scalable ways for teams like myself to stand on the shoulders of giants and build on top of Codex, right? Codex is fantastic. It's got great memory. It's got good skills. It's persistent. There's a lot of things that are wonderful about Codex, but the thesis of my team was Codex will get you about 80% of the way to a product that would be successful in the market, but the 20% is missing. And so, we need to find some way for my team to go build that 20% to actually go win, because the other half of the thesis was if you just took Codex and launched it as a finance product, it would fail. And so, that 20% is actually pivotal. And so, what I would say is we went and we built on top of the plugin architecture. It's using the same primitives that are available to any of the builders that want to go build on top of ChatGPT. And I had a line in my presentation, but I really do think that there's going to be a billion-dollar business built on top of a ChatGPT plugin, right? Like you're going to be able to leverage all of that horizontal functionality and just add your secret sauce on top. And that secret sauce might be data, it might be UI changes, it could be an integration, but that secret sauce is really valuable and I think there's a billion dollars worth of value to be built there.

所以，是的，这对行业来说是一个有趣的时刻，因为我们现在处于编排变得更加自动化的时代，你知道，智能体团队出去做事。但对于金融，显然涉及很多风险。所以，你能谈谈你融入产品的安全性吗？

So, yes, it's an interesting time for the industry because we're at a time now where orchestration is becoming a lot more automated, you know, teams of agents going out and doing things. But for finance, obviously, there are a lot of risks involved in that. So, can you talk a bit about the safety sort of that you've baked into the product?

是的，安全非常非常重要。我想说它只会变得更加重要，因为智能体已经变得更加自主和独立。我们预计这一趋势将持续到 27 年和 28 年。对于我们在上个月构建和发布的产品，已经有很多关于安全的范式，以及围绕审计工作、确保模型不会犯影响业务结果的错误进行深思熟虑。我们能够大力依靠这些。我想说的是，我们发布的现有产品是面向该领域知识工作者的生产力工具。所以背后的理念是有一个深刻的假设，即始终有人在环中，对吧？所以那里的想法是，你是一名金融分析师，你有一大堆当天想要完成的事情。就像软件工程师使用 Codex 那样，对于每一项，你做的第一件事就是尝试用 ChatGPT 构建它，对吧？它可能让你完成 30%，也可能让你完成 80%，取决于任务，但始终假设有人在环中主持那项工作。这使得我们很容易构建一个产品，我们可以将审计审查的责任交到人们手中，变成人们的责任。这对我们很有效，而且很容易沟通。所以那很棒。我要说我们正在深入思考接下来会发生什么。而接下来发生的事情看起来不同于一个始终由人类监控的生产力工具。

Yeah, safety is super, super important. I would say that it's only going to get more important because the agents are already getting more autonomous and more independent. And I think that trend we expect it to continue into '27 and into '28. I would say for the product that we built and launched last month, there was a bunch of paradigms that already existed as it pertains to safety and being thoughtful around auditing the work, making sure that the model's not making a mistake that's going to impact business outcomes. And we were able to lean heavily into those. And what I would say is the existing product that we launched is a productivity tool for knowledge workers working in this space. And so the idea behind it is there's a deep assumption that there's always a human in the loop, right? So the idea there is, you're a financial analyst and you have a laundry list of things you want to get done that day. And just like how a software engineer would do with Codex, for every one of those items, the first thing you do is try to build it with ChatGPT, right? And it might get you 30% of the way there, it might get you 80% of the way there depending on the task, but there's always this presumption of humans in the loop that are presiding with that work. And that made it very easy for us to build a product where we could pass off the responsibility of auditing review into the people's hands and into people's responsibilities. And that worked for us well and it was easy to communicate. So that was wonderful. I will say we're thinking very deeply about what comes next. And what comes next looks different than a productivity tool that sits and is always being monitored by a human.

我觉得在思考下一步时，我们主要在想两件事。一是如何把产品的安全与保障特性塑造成人们能直观理解的东西。而我认为人们非常直观就能理解的，就是如何与其他人协作。所以我们的论点是：如果我们把产品做得非常像你和同事、实习生、身边的人共事的方式，那人们就能很快、很容易地上手学会。因此我们需要把它的安全特性塑造成像一个人。

And I think as we think about what comes next, there are kind of two things we're thinking about. One is how do we shape the security and safety profile of the product into something that people understand intuitively. And something I would say people understand very intuitively is how to work with other people. So our thesis is: if we make the product very similar to how you work with your co-workers, your interns, people around you, that is something people can pick up and learn very quickly, very easily. And therefore we need to shape the security profile of that to look like a person.

这里我要点出几件事。一是“读写权限”这个概念——对人也是如此。比如你是一家公司的新软件工程师；我在 Google 时，你需要一个叫 readability 的东西。所以在你能动手碰系统之前，已经有这些制衡机制和系统存在。也就是说，已经有系统在监控人的工作。那么如果模型的行为更像人，你就可以复用这些系统。这让与这些金融机构沟通、推进这件事变得非常容易。这是我们接下来会重点投入的方向。

And so there are a few things I'll call out there. One is there's this notion of read-write — even for people. Like, you're a new software engineer at a company; I was at Google, you need something called readability. And so there are these checks and balances and systems in place before you can start mucking around touching the system. So there are already systems in place to monitor the work of people. So if models behave more like people, then you can leverage those systems. And that makes it very easy to talk about and work through this with these financial institutions. So that's something we're leaning into pretty heavily coming up.

最后我要说的是，至少在银行业这个我们起步的语境里，已经有一套系统：初级分析师做大量基础工作，他们的工作交给经理（associate），经理审核这些工作，然后经理再添上自己的独门秘方、他们的魔法、他们的推理，再交给副总裁（VP），VP 再往上到总监、董事总经理。所以存在这样一种层级体系，其中隐含着工作核查。这套东西早就建好了，用来防范人犯的错误，对吧？人犯的错误和模型一样多。所以那里已经有系统了。我们在构建金融产品时，也会重点依托这些系统。

And the very last thing I would say is, at least in the context of banking, which is where we kind of started, there's already this system in place for junior analysts doing a lot of the groundwork. Their work gets passed off to associates. Associates are reviewing the work. Then the associates are kind of adding their secret sauce, their magic, their reasoning. That gets passed off to the VP. VP goes up to the director, managing director. So there's this kind of hierarchical system where there's this implicit work checking that's happening. That's already been built out to try to prevent against mistakes that humans make, right? Humans make just as many mistakes as models. And so there are systems that are built there. And so we're also leaning heavily on those systems as it pertains to building financial products.

好。我想说存在这样一种误解：在金融界，一个错误真的会不断放大。比如你提交了错误的东西，整个职业生涯就岌岌可危。人们应该如何看待开始使用它？如果你是一名分析师，你该如何使用这些工具？你有没有什么类比来说明我们现在所处的位置？就像自动驾驶有分级一样。我记得你有一页幻灯片说 2027 年就看不到 PowerPoint 了，很多人会在流程中使用 AI 工具。

Okay. I'd say there's this sort of misconception where in the finance world, one mistake really compounds. Like your whole career is at risk if you submit something wrong. How should people think about getting started in using it? So if you're an analyst, how should you use these tools? And do you have any analogies towards where we're at right now? There are like levels of self-autonomy in self-driving. One of your slides I think said 2027, you see no more PowerPoint. A lot of people will be using AI tools in the loop.

那么人们应该如何看待我们现在的位置和未来的走向？如果他们抱着“一个错误，我的职业生涯就完了”这种误解，该如何采用这些工具？显然有人类在环，但随着他们需要卸载越来越多的工作，他们又该如何看待这件事？

So how should people think about where we are now and where things are going, and how should they adopt these tools if they have that misconception of like one mistake, my career's over? Obviously, there's human in the loop, but how should they also think about it going forward as they need to offload more and more?

好，这个问题很好。对此我有两个视角，它们基于不同的假设。我认为“犯一个错职业生涯就完了”这个观念是不正确的，我有一条思路可以展开。但让我先假设它是真的，对吧？在它是真的情况下，我想说，并非所有错误都是同等的，也并非所有模型能力都是同等的。

Yeah. No, that's wonderful. I have two perspectives on this, and they start with different assumptions. I think the notion that my career's over if you make one mistake is an incorrect notion. And I have a line of thought that follows that. But let me first assume that it's true, right? And in the case that it's true, what I would say is not all mistakes are created equally and not all model capabilities are created equally.

所以当你想到模型或人可能犯的、可能导致职业生涯终结的错误时，通常是在你升到某个层级之后。比如你是 VP，风险就比初级分析师高，因为人们默认到此为止的工作都已经核查过了。所以如果董事总经理看到一个错误，那就是个大问题。这个错误能一路传到董事总经理那里，说明很多安全网都失效了，对吧？

And so when you think about the mistake that a model or a human could make that could be career ending, it's usually once you get up to a certain level. Like if you're a VP, the stakes are higher than when you're a junior analyst, because there's a presumption of work having been checked up to that point. So if an MD sees a mistake, that's a huge problem. A lot of the safety nets have failed for that mistake to make it up to an MD, right?

但我想说的是，那种终结职业生涯的错误，是指你拿到一条事实错误的信息，而这条事实信息改变了结论。这显然是你可能犯的最大一类错误。我想说，我们在细节注意力上看到了很大进步。幻觉曾经是 2022、2023 年经常出现的大词，如今几乎不怎么提了。我想说，在事实性错误方面，我们开始看到模型可靠性的大幅提升。所以那种会让你被开除的严重错误，模型在这方面正变得越来越好，而且会继续变好。

But what I would say is, the career-ending mistake is something where you get a piece of information factually incorrect, and that piece of factual information changes the conclusion. That is obviously the biggest type of mistake you can make. And what I would say is we're seeing a lot of progress in attention to detail. Hallucination used to be a big term that came up in 2022, 2023; that doesn't come up nearly as often nowadays. I would say we're starting to see huge improvements in the reliability of models as it pertains to factual mistakes. So the egregious mistake that would get you fired — models are getting much better at that and continue to get much better.

所以我其实相当有信心：这种持续改进，加上现有的系统核查，会让那类错误一路传到董事总经理那里的概率变得非常低，对吧？

And so I feel actually quite confident that a combination of that continual improvement plus the existing system checks in place will make it such that that type of mistake — the chance of that propagating up to an MD — is just very low probability, right?

我想说，模型今天欠缺、仍需改进的，其实不是那种终结职业生涯的错误，而是涉及人类判断和品味的部分。比如：这页幻灯片看起来很丑，对吧？它没有错，但排版上颜色有点不对，某处看起来有点粗糙，因此这不是一页好幻灯片。这不像事实性错误那样是终结职业生涯的错误，但它很重要。它仍然影响那份工作的有用程度。

What I would say the models are lacking today, and they still need to improve, is actually not the career-ending mistake, but the bit that involves human judgment and taste. So it's: this slide looks ugly, right? It's not wrong, but it's formatted — the colors look a little bit off. Something about it looks sloppy, and therefore this is not a good slide. And that's not a career-ending mistake in the way a factual mistake is, but it matters. It still matters in terms of how useful that work is.

所以在思考迈向 2027 的进展时，我想说我预期模型会在人类品味这一块变得好得多。这也是我在主题演讲中暗示的：我认为模型会在那些更软的目标函数上、在人类品味方面变得好得多。它不是“对还是错”，而是比较性的：有没有比这更好的？我认为模型未来会沿着这个维度变得好得多。

And so as we think about the progression going into 2027, what I would say is I expect the model to get much better at the human taste component. And that's what I was alluding to in my keynote: I think the model will get much better at human taste on things that are softer objective functions. It's not "is it right or wrong?" It's comparative: is there something that's better than this? And I think models are going to get much better along that dimension moving forward.

怎么说？我们怎么解决品味问题？我们怎么把品味训练进模型里？

How so? How do we solve taste? How do we train taste into the models?

好，说来有趣。很早很早，在国际象棋的早期，我们训练模型唯一依赖的东西本质上就是品味，对吧？那时有基于人类反馈的强化学习，而人类反馈始终是成对信号：给定两个选项，哪个更好？所以强化学习的早期是建立在“用人类品味来训练”这一理念上的。然后显然有了可验证奖励和环境之类的东西——我们构建了大量机制来做其他类型的奖励和其他类型的强化学习。

Yeah. So it's funny. Very early on, in the early early chess days, the only thing we trained models on was essentially taste, right? There was reinforcement learning through human feedback, and the human feedback was always a pairwise signal. It was: given two options, which one's better? And so the early days of reinforcement learning were built on the notion of training on human taste. And then obviously things like verifiable rewards and environments — we built a lot of machinery to do other types of rewards and other types of RL.

所以，人们有点忘了，早期强化学习（RL）的很多工作都是围绕人类品味展开的。那么，如何基于人类品味进行训练？我认为技术上的答案很简单。实际上的答案显然更复杂。实际上的答案就是训练一个奖励模型，对吧？就是训练一个奖励模型。你让一群人看成对的东西，然后说哪个更好。这仍然有效。

And so, people kind of forgot that a lot of the early days of RL, reinforcement learning, were all around human taste. And so, how do you train on human taste? I think the technical answer is quite simple. I think the practical answer obviously is more complicated. The practical answer is you just train a reward model, right? You just train a reward model. You get a bunch of humans to look at pairs of things and say which one is better. And this still works.

有趣的是，我们在幻灯片上就是这么做的，对吧？如何让这些模型更擅长生成演示文稿？我们只是让一群人看成对的幻灯片，然后说哪个更好看。而人类实际上非常非常擅长审美判断，对吧？我们发现，人类实际上比模型强得多。人类可以看两张幻灯片，几秒钟内就能说这张更好看。而如果你有一个模型，比如一个评分模型，试图做同样的事情，它会花费大量推理 token，这个像素这样，那个东西那样，然后可能得出错误的结论。因此，人类在让这些模型变得更好方面仍然非常有价值。所以，我们将大量依赖人类偏好来灌输品味之类的东西。这是一点。

So, fun fact, we did this for our slides, right? How do you go about making these models better at generating presentations? Well, we just get a bunch of humans to look at pairs of slides and say which one looks better. And humans are actually really, really good at aesthetic judgment, right? Actually much better than models, is something that we discovered. A human can look at two slides and within seconds say this one looks better. And if you have a model, like a grading model, try to go about doing that, it's going to spend a whole bunch of reasoning tokens, this pixel this way, this thing's that, and then maybe comes to the wrong conclusion. And so, humans are still really, really valuable in making these models better. And so, we will rely a lot on human preference to imbue things like taste. So, that's one.

我要补充的最后一点是，这有先例，这让我们相信这个策略会奏效。那个先例就是编程。模型已经——我想说编程还没有完全解决，但假设我们已经完成了 80%，并且正在最后 20% 上取得进展。编程是可验证奖励和人类品味的结合，对吧？可验证奖励是，代码能编译吗？有语法错误吗？函数的输出符合预期吗？所以这些可验证奖励是存在的，但很多好的软件工程是干净的系统设计？你构建的系统是否长期可维护？是否容易进行单元测试？是否遵循了社区或行业绝大多数人认为正确的范式？

The last thing I'll add is there's precedent for this, which makes us confident that this strategy will work. And that precedent I would say is coding. The models have gotten—I would say coding is not fully solved, but let's say we're 80% of the way there and we're making progress on the last 20%. Coding is a combination of verifiable rewards and human taste, right? And so, the verifiable rewards are, you know, does the code compile? Are there syntax errors? Is the output of the function expected? And so there's these verifiable rewards that are in place, but a lot of good software engineering is this clean system design? Did you build a system that's maintainable over time? Is it easy to unit test? Is it following paradigms that the vast majority of the community or the industry thinks is the right paradigm?

如果你从这个角度看，很多金融工作看起来非常相似。比如，我有一个 Excel 模型，它能加总吗？有引用错误吗？有一大堆可验证的东西可以检查，以确保工作通过第一轮健全性检查，但也有很多是“这是我会构建这个模型的方式吗？”我开始在金融领域工作时学到的一件事是，有一种叫做模型感觉的概念，就是你扰动输入，然后观察输出。有经验的金融专业人士可以扰动输入，观察输出，并理解数据流经这个模型的方式是否感觉良好，对吧？这是一种人类偏好，看起来很像“这段代码看起来干净吗？写得好吗？”我认为我们在编程方面的成功让我有信心，我们也能解决金融领域的许多这类问题。

And if you look at it from that lens, a lot of financial work looks very similar. It's the, you know, I have an Excel model, does it sum up? Are there reference errors? There's a whole bunch of verifiable things that we can check to make sure that the work is passing the first pass sanity check, but there's also a lot of is this the way I would have built this model? Something I learned about as I started working in finance is there's this notion of the feel of the model, which is, you know, you perturb the inputs and you look at the outputs. And an experienced financial professional can perturb the inputs, look at the outputs and understand does the data flow through this model in a way that feels good, right? And that is a human preference that looks a lot like does this code look clean? Is it well written? And I think our success in coding gives me confidence that we can solve a lot of these problems in finance as well.

是的，我们可以提一下 Cognition 的 Frontier Code。所以，不只是 SWE-bench 那种“答案正确吗？”，这也是“你会合并这段代码吗？”所以，你可以有多个正确的 PR 通过，但你会合并哪一个？这就是为什么你有时会看到“推理更低更好”的图表，但这也是一个品味基准，对吧？

Yeah, we can shout out Frontier Code from Cognition. So, it's not only like SWE-bench is very, you know, is the answer correct? This is also would you merge the code? So, you can have multiple answers that are proper PRs that pass, but which one would you merge? And that's why you sometimes see the charts of lower reasoning is better, but it's a taste benchmark as well, right?

是的。你另一个我觉得有趣的 2027 年预测是，你训练的模型将在投资论点以及为投资和交易生成想法方面变得好得多。但显然，这是人类——经验丰富的金融专业人士非常擅长的，他们对这类事情有直觉，但这往往与主流观点相悖。否则，获得高回报的机会就更小。那么，在训练智能体做这类工作方面，你如何解决这个问题？

Yeah. And one of your other 2027 predictions I thought was interesting was that the models that you're training will become much better at investment thesis and generating ideas for investment and trading. But obviously, that's something that humans are—well, experienced financial professionals are very good at and they have an intuition for that kind of thing, but it's often counter to the prevailing wisdom out there. Otherwise, there's less of a chance of it making a great return. So, how are you tackling that problem in terms of training the agents to do that kind of work?

是的。我想说，到目前为止，AI 在金融领域的绝大多数应用都集中在初级分析师的工作上。我认为这有两个原因。一是你必须根据当今模型的能力来构建产品。所以，如果你看看当今模型的能力，它们非常擅长接受相当具体的指令并执行，对吧？比如，给我做一张幻灯片，我希望它看起来这样，让它好看，用我们的模板，诸如此类。模型非常擅长做这些事情，因此，我们围绕这些构建了产品体验。

Yeah. I would say that up until today, the vast majority of applications of AI in finance have been really focused on the junior analyst's work. And I think that's for two reasons. One is you have to build the product based on the capabilities of the model of today. And so, if you look at the capabilities of the model today, they're really quite good at taking pretty concrete instructions of what to do and executing, right? So, build me a slide, I want it to look this way, make it look good, use our template, these types of things. The model's quite capable of doing those things and so therefore, we've built product experiences around that.

我要指出的另一点是，人们强烈倾向于让模型擅长人类不擅长或不喜欢做的事情，对吧？所以，我交谈过的大多数人——显然有例外，但很多人真的不喜欢构建 Excel 模型。这是一项繁琐的工作，非常注重细节，容易出错。幻灯片也是如此。你在复制徽标，把它们非常精确地放在页面的特定位置。这是很多人不喜欢做的非常繁琐的工作。因此，这促使我们专注于其中一些任务，对吧？这是两个因素的结合。现有模型能力在某些类型的任务上相当不错，而人类喜欢做什么和不喜欢做什么的偏好也推动了产品的方向，对吧？这是这两个因素的结合。

And the other thing I'll call out is there's a strong preference to make the model good at things that humans are not good at or don't like to do, right? And so, most humans I talked to—there's obviously exceptions, but a lot of the humans I talked to really don't like building Excel models. It's tedious work and it's very detail oriented, it's easy to get wrong. And similarly with slides. You're copying logos, you're placing them very precisely on a specific part of the page. That's very tedious work that a lot of people don't like to do. And so, that's been part of the impetus for us to go and focus on some of these tasks, right? It's a combination of two of those things. There's the existing model capability is pretty good at certain types of task and also human preferences of what they like to do and not like to do helps drive the direction of the product as well, right? It's a combination of those two things.

我认为我们已经解决了这两个方面很多容易摘的果子。所以，进入 2027 年，我们将开始向上移动，对吧？我们将从初级分析师能做的任务，到一些助理能做的任务，再到一些董事总经理会做的任务，沿着这个链条向上移动。而随着你向上移动，我想说目标会变得更加模糊。从“这明显是一张好幻灯片或坏幻灯片，或者是我们会放在客户面前的东西”，变成“嗯，我不知道这个投资想法好不好”。而且我需要等 3 年才能真正给你一个关于市场表现的清晰具体的答案。所以目标开始变得更加模糊。

I think that we've solved a lot of the low-hanging fruit of those two things. And so, going into 2027, we're going to start to move up the stack, right? We're going to go from the task that a junior analyst can do to some of the tasks that an associate can do to some of the tasks that an MD would do and move up that chain. And as you move up the chain, what I would say is the objectives become much fuzzier. It goes from this is clearly a good or bad slide, or something that we'd put in front of a client or not, to well, I don't know if this investment idea is good. And it would take me 3 years of waiting to really be able to give you a clear concrete answer of what the market has done. And so the objectives start to become fuzzier.

所以，训练模型的方式，以及你思考如何让模型变得更好的方式，这些事情都变得更困难了。从研究的角度来看，它变得更难了。但我想说，这并非不可能。所以需要的是大量的研究专注。需要大量优质的数据。需要来自该领域专家的许多反馈。我们真的希望获得 P99.9 类型的反馈，让模型变得和人类一样好。但我不认为这是一个无法克服的问题。所需要的是基础模型的持续 Scaling（规模扩张），以及专注。来自研究实验室的专注，他们认为这是一个重要的问题去解决。而我在我的团队中正在做的一件事，就是让这成为 OpenAI 的一个重点领域和优先事项。我认为，由于这两件事的结合，我有信心我们在 2027 年将取得很大进展。

And so the way you go about training the model, and the way you think about how to make the model better, those things become more difficult. It becomes more difficult from a research perspective. But it's not impossible, is what I would say. And so what it takes is a lot of research focus. It takes a lot of wonderful data. It takes a lot of feedback from experts in the field. We really want to get the P99.9 type of feedback to make the model as good as humans. But I don't think it's an insurmountable problem. And what it takes is continued scaling of the base model, and focus. Focus from the research labs who think that this is an important problem to go solve. And one of the things that I'm doing on my team is making this an area of focus and priority for OpenAI. And I think because of the combination of those two things, I feel confident we'll make a lot of strides in 2027.

你对当前任何金融基准有什么评论吗？

Do you have any comments on any of the current financial benchmarks?

所以，我认为我们仍然处于这样的状态：你知道，这在数字上说得通吗？你如何考虑制作自己的基准，加入品味？我认为这也是对在这个领域工作的人的一个公开呼吁，让他们开发自己的基准。

So, I think we're still in the state of, you know, does this check out numbers-wise? How do you think about making your own, adding in taste? I think this is also an open call for people working in this space to work on their own benchmarks.

与此相关的一个问题是，你说今天为模型构建系统，但你也给出了很多对未来的预测。你如何对你今天为人们构建的金融服务设置护栏、限制和检查，以便他们确信，你知道，这不会失控？

And a sort of follow along to that is, you said build systems for models today, but you also give a lot of predictions for the future. How do you put guardrails, limitations, checks on financial services that you build for people today, just so they're confident in, you know, this is not like going off the rails?

是的，有道理。让我们谈谈评估。评估是我非常关心的事情。我花了很多时间在评估上。我想与大家分享的一个先验是，构建一个好的评估是非常困难的。对于没有做过评估的人来说，这并不明显。要构建一个真正与你试图测量的东西相关的评估，比如你试图测量的技能、你试图测量的能力，非常困难。因为你组合起来的任何东西都只是那个东西的代理，对吧？而且它只是一个维度。所以，这是一点。第二点是，要正确获取答案键或评分方式非常繁琐。如果你没有正确掌握评分方式，这个评估就很容易被破解，很容易被奖励破解。因此，你可能会在评估上得分很高，但在实际底层技能或目标上表现很差，对吧？所以，我要分享的先验是，评估真的很难，不仅在金融工作中，而是全面如此。我想说的是，我们认为今天还没有一个真正好的金融工作评估。有一堆现有的先前工作。我会引用我们做的一些工作，即 GDP 评估，它更广泛地涉及知识工作。如果你们看 Astra 的公告，我们没有谈论 GDP 评估。我们知道这个评估已经完全饱和了，而未饱和的部分基本上是有问题的，对吧？所以，我们实际上不再报告它，不再跟踪它。因此，对于 GPT-6 Astra 来说，一个棘手的事情是，我们没有过多谈论金融评估。我们内部有一些，但没有过多谈论，当时也不想分享。它们即将到来。但我们没有好的方法在这些评估上展示金融任务的进展。有一个有趣的事情，我认为评估中存在一种偏见：如果数字看起来不错，研究人员就会假设模型变得更好了，他们不会去调查。如果你做了一个更强大的模型，而评估数字看起来不好或没有增加到你想要的规模，那么你就会去调查，对吧？所以，这有点像调查这些评估时固有的偏见。类似的事情发生在 GPT-6 上，我们在金融评估上运行 GPT-6，数字并没有太大变化。我们就像，“哦，这是怎么回事？让我们看看这个评估。”我们对这些不同的评估进行了深入调查，发现很多地方都有问题。我们基本上发现我们已经使评估饱和了，我们做错的问题是由错误的答案键、指定不清的提示或评估的其他缺陷导致的。所以，总而言之，我的团队目前正在做的一件事是尝试构建一个更好的评估，真正预测自动化特定职业工作的能力。所以，想法是，我们将有一个类似投资银行评估的东西，预测，你知道，如果你在这个评估上得分高，你可能可以自动化投资银行初级分析师的很多工作。

Yeah, makes sense. Let's talk a little bit about evals. Evals is something near and dear to my heart. I spend a lot of time working on evals. I would say that the prior that I would share with people is building a good eval is incredibly hard. That's not obvious to people who haven't worked on an eval before. To build an eval that actually correlates with the thing that you're actually trying to measure, like the skill that you're trying to measure, capability you're trying to measure, very difficult. Because whatever you put together is just a proxy for whatever that thing is, right? And it's just one dimension. So, that's one. Two is it's very tedious to get the answer key or the way that you grade this correctly. And if you don't get the way that you grade the answers correctly, it's an eval that's very easy to hack and very easy to reward hack. And therefore, you could score very highly on the eval, but actually do really poor on the actual underlying skill or objective that you're trying to measure, right? And so, the prior that I will share is evals are really hard, not just in financial work, but just across the board. And what I would say is we don't think that there's a really good eval today for financial work. There's a bunch of existing prior work. I'll reference some of the work that we did, which is a GDP eval, which is like a little bit more broadly in knowledge work. If you guys look at the Astra announcement, we don't talk about GDP eval. We know that the eval is completely saturated, and the pieces that are not saturated are essentially broken, right? And so, we actually don't report on that, we don't track it anymore. And so, one of the tricky things for GPT-6 Astra was we didn't talk a whole lot about financial evals. We have some internally that we didn't talk too much about and didn't want to share at the time yet. They're coming. But there was no good way for us to show progress on financial tasks on these evals. There's a funny thing like I think there's this bias in evals where if the numbers look good, the researchers assume that the model's gotten better and they don't investigate it. If you made a much more capable model and the eval numbers look bad or haven't increased in the scale that you want, then you go investigate, right? So, this is kind of like bias baked into investigation of these evals. Something like this happened with GPT-6 where we ran GPT-6 on a set of financial evals and the numbers didn't really change much. We're like, "Oh, what's going on here? Let's take a look at this eval." We did a deep dive into these different evals and lots of things were broken. We basically found that we had saturated the evals and the questions that we were getting wrong were driven by incorrect answer keys, poorly specified prompts, or just some other flaw with the eval. And so all that to say like one of the things that my team is working on currently is to try to build a much better eval that actually predicts the ability to automate the work of that specific profession. And so the idea is like we'll have something like an investment banking eval that predicts, you know, if you score well on this eval, you could probably automate a lot of the work of a junior analyst in investment banking.

我们很快就要结束了。我想问一下，对金融领域的人、Latent Space 的听众有什么呼吁吗？你的团队可能在招聘，但最后一件事，人们如何能对你有所帮助？

We got to wrap here soon. I want to ask any callouts for people in the finance space, the Latent Space audience, your team is probably hiring, but last thing to put out there, how can people be helpful to you?

是的，太好了。我想说，首先，请查看 ChatGPT 金融服务。它是一个企业产品，但我们正在努力使其在专业计划中向更多专业消费者提供，所以希望我们会有一些很酷的公告。第二，是的，我的团队正在招聘。我们看到产品有惊人的客户吸引力，因此我们正在快速招聘和扩展团队。这是一个非常多元化的团队，有许多不同的角色，所以我想快速提几个。显然有标准的前端后端工程，所以是传统产品构建。这是一点。第二，我们非常需要优秀的产品经理。我认为在某些方面，伟大的产品管理是试图在这个领域获胜的主要瓶颈，所以我们正在招聘投资银行、股票研究、资产管理等各个领域的优秀产品经理。然后我想特别提一下另一个角色，我们正在招聘一个我们称之为主题专家的人。你知道，这非常重要。我们在编码方面做得很好的一件事是，训练模型并使其变得更好的人理解什么是好的代码，对吧？这对我们来说是一个巨大的顺风，因为我们正在使模型在这些方面变得更好。

Yeah, wonderful. Well, I would say first and foremost please check out ChatGPT Financial Services. It is an enterprise product, but we're trying very hard to make it available to more prosumers on the pro plan and so hopefully we'll have some cool announcements there. Two, yes, my team is hiring. We're seeing incredible customer traction with the product and because of that we're hiring and scaling the team rapidly. It's a very diverse team with lots of different roles and so I just want to call out a few of them very quickly. There's obviously the standard front end back end engineering, so kind of traditional product building. So that's one. Two, we're very much looking for wonderful product managers. I think in some ways great product management is the main bottleneck in trying to go win in this space and so we're trying to hire wonderful product managers in investment banking, equity research, asset management, across the board we're trying to hire product managers. And then the other role that I want to give a special shout out to is we're hiring someone we call a subject matter expert. And you know, it's really important. One of the things that we did well in coding is, you know, the people that were training the models and making them better understood what good code looks like, right? And that was a huge tailwind for us as we're making the models better at these things.

在让模型更擅长金融方面，我们并没有那个优势。所以我们正在扩充团队，努力招聘很多能够带来金融专业知识的人，与工程师、研究员、产品经理合作，让模型和产品在你们的特定领域变得更好。所以如果这些内容引起了你的共鸣，我强烈建议你联系我。我可以在 LinkedIn 上找到。我们也在招聘页面上发布了这些职位。很容易找到。

We don't have that advantage when it comes to making the models better at finance. And so we're building out and trying to hire a lot of folks that are able to bring that financial expertise and work with the engineers, work with the researchers, work with the product managers to make the model and the product better in your specific fields. And so if any of those resonates with you, I highly recommend that you reach out to me. I'm available on LinkedIn. We also have these job listings posted on our careers page. It's pretty easy to find.

太好了。嗯，你回答了我的问题。我本来想问什么时候我能试用这个，但它会来到 Pro Pro Pro Pro Pro

Great. Well, you answered my question. I was going to ask when can I have a play with this, but it is coming to Pro Pro Pro Pro Pro

很难让更多人用上它。

Very hard to make it available to more people.

非常常见的 OpenAI 做法，你知道，这很合理，你们先为企业开发东西，然后这些会逐渐下放到常规模型。它会来到 Pro 计划。所以我预计对于你的通用用途，未来它在控制台相关方面会变得更好。

Very common opening eye stuff, you know, it makes sense you make stuff for enterprise and then this trickles down to the regular models. It comes to Pro plan. So I expect for your general usage down the line it'll get better at console stuff.

太棒了。非常高兴你能来。去看看主题演讲吧。你讲了很多。你给出了很好的预测。你谈了一些有趣的我们修复了预训练问题的事情。所以我会给你们留下线索去看，但感谢你的参与。

Awesome ball. Very excited to have you. Check out the keynote. You gave a lot. You gave good predictions. You talked a little bit about some juicy we fixed pre-training issues. So I'll leave breadcrumbs for you guys to go watch, but thanks for joining.

是的，太棒了。非常感谢。

Yeah, awesome. Thank you so much.