在扩展超级智能的同时保持人性

大家下午好，欢迎收看新一期的《View From the Top》。今天我很荣幸为大家介绍我们的嘉宾 Sam Altman，他是 OpenAI 的联合创始人兼 CEO，也是当下塑造世界如何看待人工智能的最有影响力的声音之一。正如很多人所知，Sam 的职业生涯始于创业者，他联合创办了一家叫 Loopt 的公司，之后又成为投资人和创业者。后来他担任 Y Combinator 的总裁，与许多早期创始人密切合作，帮助塑造了下一代初创公司。OpenAI 已成为全球最受关注的公司之一，而 Sam 对这项技术走向的看法，持续影响着商业、政策和文化的讨论。今天，Sam 将分享他在我们这个时代最重大的技术变革中心所形成的视角。我们非常高兴他能来到现场。请大家和我一起欢迎 Sam Altman。

Good afternoon everyone, and welcome to another View From the Top. It's my pleasure today to introduce our guest, Sam Altman, who is a co-founder and the CEO of OpenAI and one of the most prominent voices right now shaping how the world thinks about artificial intelligence. As many of you know, Sam began his career as an entrepreneur, co-founding a company called Loopt before becoming an investor and an entrepreneur. He went on to serve as president of Y Combinator, where he worked closely with many early-stage founders and helped shape the next generation of startups. OpenAI has become one of the most closely watched companies in the world, and Sam's views on where this technology is headed continue to influence conversations across business, policy, and culture. Today, Sam is going to share the perspective he has built at the center of one of the most consequential technological shifts of our time. We could not be happier to have him with us today. Please join me in welcoming Sam Altman.

你好，Sam。

Hi Sam.

你好。

Hello.

欢迎来到商学院。

Welcome to the GSB.

谢谢你们的邀请。

Thank you for having me.

我们都很高兴你能来。为了做准备，我一直在和一个 AI 版的你对话，当然是由 Codex 做的。它非常有用，回答了我大部分问题。但我得说，我仍然觉得没有什么能比得上一场面对面的对话。所以，谢谢你。

We are all so excited to have you here. One of the things I've been doing to get ready is I've been talking to an AI version of you, built by Codex, of course. And it's been very helpful. Answered most of my questions. But I have to say, I still don't think anything beats an in-person conversation. So, thank you.

很高兴来到这里。

Excited to be here.

Sam，在座很多人知道你离开斯坦福去创办第一家公司，但可能很少有人知道，你差一点就去投行实习了。

Sam, many in this audience know you left Stanford to build your first startup, but fewer probably know that you almost did an internship in investment banking instead.

是的，确实如此。

I did. That's true.

你能带我们回到那个决定，讲讲你当时的心态吗？

Could you take us back to that decision and your mindset at the time?

你知道，我从来不想当投行家，但那是所有酷小孩都想做的事，我成了同辈压力的受害者。不，如果你们中有人想当投行家，那祝你们顺利。那只是不符合我的天性。我是个书呆子。但我到了斯坦福，所有最聪明的孩子都在努力成为投行家，那才是酷的事。我就想，好吧，我想当个聪明孩子，也许我也该想进投行。所以，纯粹因为同辈压力，我真的走了全套流程，拿到了实习，感觉自己超级酷。当时他们会把你飞到纽约，有一场面试，然后有一顿特别好的晚餐，非常气派。我从没经历过那样的事，这让我更加觉得这一定是我该走的路。然后我心里越来越有一种恐惧感，我就是觉得自己不会喜欢。与此同时，我一直在和几个朋友做这个项目，当时甚至还不算一家初创公司。然后 Y Combinator 说：“我们给你一笔钱供你夏天生活，你可以试着做一家初创公司。”我就想：“好吧，这是宇宙给我的信号。我应该去试试。”

You know, I never wanted to be an investment banker, but it was what all the cool kids wanted to do, and I fell victim to peer pressure. No, if any of you want to be investment bankers, more power to you. It was just not my organic thing. I was a nerd. But I got to Stanford and all of the smartest kids were trying to become investment bankers, and that was the cool thing. And I was like, well, I want to be a smart kid. Maybe I should want to go into investment banking, too. So, for no reason other than peer pressure, literally, I did the whole thing and got an internship and felt super cool. At the time they would fly you out to New York and there was this interview and then they have this super nice dinner, and it was very impressive. I'd never experienced anything like that, and that made me think even more that this must be the right path for me. And then I had this increasing feeling of dread, and I just didn't think I was going to like it. Meanwhile, I had been building this sort of not even a startup at the time, but this project with some friends. And then Y Combinator was like, "We will give you money for the summer to live on and you can try to do a startup." And I was like, "Okay, this is a sign from the universe. I should try to go do that."

你加入了 Y Combinator 最早的一批，最终成为 YC 的总裁。从经营一家初创公司到执掌全球最著名的创业加速器，这段经历教会了你什么？

And you joined the very first batch of Y Combinator and eventually became president of YC. What did that experience teach you, going from running a startup to running the world's most famous startup accelerator?

对我来说，做初创公司最棒的事情之一——我退学时很难过，因为我热爱学校，我也担心自己不会再继续学到东西。而做初创公司最酷的一点是，它过去是、现在仍然是我经历过学习速率最高的环境。我觉得如果你们——也许在这个场合说这个不太合适——但当时有人对我说过一句话，我想可能是 Paul Graham，或者类似的人，他对我说：“你做一个月初创公司，就能拿到一整个 MBA。”我没有 MBA，所以没法说这是不是真的，但我确实很快学到了很多关于商业、产品、客户、销售以及其他一切的东西，这些对我之后的职业生涯非常重要，我觉得这些在学校里根本学不到。这就是那种只能边做边学的东西，也许没有别的更好的学习方式。所以初创公司第一段时期的学习速率简直难以置信。我这辈子绝对没有在别的时期学到过更多东西。经营 YC 是另一种学习。那几乎像一份政府工作。你并不对这些公司拥有权力，但你要努力建设一个健康的社区，确保每个人都茁壮成长。所以那是另一种有趣且非常不同的学习经历。我喜欢——之前的一切，你们都会在自己的人生中发现，你会在一切之上构建，不管它们看起来多么不相关。所以我在经营初创公司时学到的很多东西对 YC 有帮助，但那是非常不同的工作。

One of the best things for me about doing a startup — I was sad to drop out of school because I loved school, and I was also worried I just was not going to keep learning stuff. And one of the coolest things for me about doing a startup is it was, and still is, the highest rate of learning environment I had ever been in. I think if you're — this is maybe the wrong room to say this in — but a thing that someone said to me, I think it was maybe Paul Graham, someone like that, said to me around then: "You will get an entire MBA in one month of doing a startup." And I don't have an MBA, so I can't say whether or not that's true, but I did learn a lot very quickly about business and product and customers and sales and all of this other stuff that has been very important to the rest of my career, that I think I just wouldn't have been able to learn in school. It was one of these things that you can just learn by doing, and maybe there's no other good way to learn it. So the rate of learning for that first period of a startup was unbelievable. I definitely have never learned more in any other period in my life. Running YC was its own kind of learning. That's almost like a government job. You don't have power over these companies, but you're trying to build a healthy community and make sure everyone thrives. So that was another interesting and very different learning experience. And I like — everything that comes before, and you'll all find this out in your own lives, you build on everything no matter how unrelated they seem. So I learned a lot of things about running a startup that were helpful for YC, but it was a very different job.

你是怎么学会那些与经营初创公司如此不同的技能的？你觉得自己是在工作中学会的，还是有导师？你实际上是怎么做到的？

How did you go about learning all of those skills that were so different than running a startup? Like do you feel like you learned them on the job or you had mentors? How did you actually go about that?

我是说，肯定有很棒的导师，但我认为如果一份工作已经被做过很多次，你也许可以在课堂上学到——也许你能学到很多有用的技能和有用的知识。但像任何一家具体的初创公司，或者经营 Y Combinator，或者任何其他没有多少人做过的这类工作，你只能边做边学。我坚信，人们在准备做一件事上花的时间太多了，不如直接去做，边做边学。所以你会犯很多错误，你会学得很快。只要你真的以一种——我想人们称之为刻意练习——的精神去对待，也就是做大量实验，然后真正反思学到的东西，无论正面还是负面，并试着更新你的模型，而不是胡乱挣扎。我认为学习速率会非常高，而且如果你直接跳进去做，人们几乎总会惊讶于自己能多快变得擅长。

I mean, definitely had great mentors, but I think if a job has been done many times before, you can maybe learn it in a classroom — maybe you can learn a lot of skills that are helpful and a lot of knowledge that's helpful. But I think things like any specific startup, or running Y Combinator, or any other kind of job that not that many people have done — you're just going to learn as you go. And I am a big believer that people spend too much time trying to prepare to do the thing than just doing the thing and learning as they go. So you're going to make a lot of mistakes, you're going to learn very quickly. As long as you really approach it with a spirit of — I think people call it deliberate practice — where you are running a lot of experiments and then actually reflecting on the learnings, positive or negative, and trying to update your model, and not just random flailing. I think the rate of learning is super high, and people almost always surprise themselves at how quickly you can get good at it if you just jump in and do the thing.

YC 让你近距离看到了各种雄心勃勃的想法。然而在 2015 年你联合创办 OpenAI 时，大多数人都告诉你，通用人工智能是不可能的，至少在可预见的未来如此。你觉得当时你和你的团队看到了什么别人没看到的东西？

YC gave you a front-row seat to all kinds of ambitious ideas. Yet in 2015, when you co-founded OpenAI, most people told you artificial general intelligence was impossible, at least for the foreseeable future. What do you think you and your team saw then that other people were missing?

深度学习是有效的。

Deep learning was working.

所以我觉得——其实我在斯坦福读书时，在 AI 实验室工作，也计划把 AI 作为职业生涯。我清楚地记得 2005 年，一位斯坦福计算机科学教授告诉我，关于 AI 有很多事，我们唯一确定的是，它可能至少还需要一个世纪，但唯一确定的是深度学习行不通。你知道，就像，好吧，我相信你，你很厉害。然后 2012 年出现了著名的 AlexNet 结果。到 2014 年，我们看到深度学习似乎在各种方面都随着规模变大而变得更好。但世界完全不接受这一点。很难夸大当时世界是多么认为这不可能成功。你知道，现在网上仍然能找到一些人，他们会花所有时间谈论深度学习行不通，是花招之类的。现在，那算是激进观点。当时，激进观点是这绝不可能达到任何严肃的智能水平，这很荒谬。但我已经学到——这在一定程度上是我职业生涯的核心洞见——规模会带来令人惊讶的效应。当你看到某种漂亮的指数级缩放时，你应该继续推进。而人类直觉在这方面做得不好。在所有这些不同的事情中——单个公司的规模回报、像 Y Combinator 这样的网络、神经网络，还有很多其他例子，但为了节省时间我们跳过——我一直发现这一点，并且我已经形成了高度的个人信念去继续押注。所以当我们看到这行得通，看到它随规模变大而变好时，我们当时并不知道它随规模变好有多么可预测——那是几年后才知道的。似乎我们应该继续推进。也许它会在某个时候崩溃。我们不确定——有好几次我们几乎失去信心，以为撞到了某种墙，但我们只是继续前进。当时对几乎任何不在 OpenAI 内部的人来说，这太不显而易见了。似乎没人相信——这并非字面事实，显然有人相信——但这是一个非常不受欢迎的观点。

So I think it— I actually, when I was a student at Stanford, I worked in the AI lab and I was planning to work on AI for my career. And I remember very memorably in 2005 a Stanford CS professor telling me that the one thing we knew for sure—there were a lot of things about AI—it was probably going to take another century at least, but the one thing we knew for sure is that deep learning was not going to work. And you know, it's like, okay, I believe you. You're impressive. And then in 2012, there was the famous AlexNet result. And then by 2014, we had seen that deep learning seemed to get better with scale in all these different ways. And the world didn't accept this at all. It's actually hard to overstate how much the world was just like, there is no way this works. You know, there are still a few people online that you can find that will spend all their time talking about how deep learning is not going to work and it's a parlor trick and this and that. At this point, that's sort of the radical view. At the time, the radical view was there is no way this is going to get to any serious level of intelligence. This is a ridiculous thing. But I had kind of learned—and this has been, to the degree there has been, a central insight of my career—that there are surprising effects to scale. And that when you see a beautifully scaling exponential of some sort, you are supposed to keep going. And something about human intuition does not do a good job with that. And in all of these different things—the scaling returns of individual companies, networks like Y Combinator, neural networks, lots of other examples I could give, but we'll skip in the interest of time—I have just always found this, and it is something that I have developed a high personal conviction to continue to bet on. And so when we saw that this was working and we saw that it got better with scale, we didn't know at the time how predictably better it got with scale—that came a few years later. It seemed like we should keep pushing on it. And you know, maybe it would break at some point. We weren't sure—there were various times where we almost lost conviction and thought we were hitting some sort of wall, but we just kept going. And it was so nonobvious to pretty much anyone that was not inside the walls of OpenAI at that time. It didn't seem to believe it—that's not literally true, there obviously were people who did—but it was a very unpopular opinion.

OpenAI 最初是一个非营利研究实验室，无意成为产品公司，而今天每周有超过十亿人使用 ChatGPT。随着 OpenAI 获得巨大影响力，你自己的领导方式是如何演变的？

OpenAI starts as a nonprofit research lab with no intention of becoming a product company, and today over a billion people use ChatGPT every week. How has your own leadership had to evolve as OpenAI has gained tremendous influence?

像 2010 年代中期硅谷的许多其他科技人士一样，我迷上了研究实验室的想法。那时所有人都在谈论硅谷如何迷失了研究的方向，我们需要回到施乐帕克或贝尔实验室，或者你最喜欢的例子。那是一件很酷的事情。我一直认为研究很棒。我一直想运营一个研究实验室，也一直对 AI 感兴趣。所以这本来算是我的退休项目。头五年——真的是我能想象的最酷的工作——我们只有一小群人，是一个研究实验室。我们在许多事情上是对的，但我们错判了 AI 一旦开发出来会如何发展。所以我们真的没想到最可能的路径是我们将不得不建立一家产品公司。我们主要以为我们会开发出这种不可思议的超级智能——我知道现在听起来很荒谬——然后我们就不必做所有艰苦的工作，想办法把它有效地交到人们手中并让社会受益。就像很久以前那个关于创业公司如何运作的梗：做第一件事，做第二件事，问号问号问号，盈利。我们的版本是：做研究，构建超级智能，许多问号，人们受益并快乐。而技术走向了一条与我们最初想象非常不同的道路。我认为这是一条好得多的道路，但这意味着我们确实必须将其产品化。这也是我认为这是一个好方向的原因之一：这更像是一种正常的经济力量，而不是我们当时想象的那样。这就像电力或晶体管或其他重大科学发现，使许多其他公司、技术和科学发现得以发生。我们必须把基础设施——即构建像 ChatGPT 这样的产品，以及构建比人类历史上任何项目都更大的算力——交到人们手中。所以到 2020 年底、2021 年初，我们清楚地意识到我们将不得不建立一家大公司。没想到会这么快。然后当我们推出 ChatGPT 时——大约四年前——一夜之间所有人都说，好吧，我们要很快成为一家大公司。

Like many other tech people in Silicon Valley in the mid-2010s, I got enamored with the idea of research labs. This is a time when everybody was talking about how Silicon Valley had lost the plot on research and we needed to get back to Xerox PARC or Bell Labs or whatever your favorite example was. It was kind of a cool thing to talk about and do. And I always thought research was awesome. I always kind of wanted to run a research lab and I was always interested in AI. So this was kind of supposed to be my retirement project. And for the first like 5 years—I mean truly the coolest job I could imagine—it was like there was a small set of us. It was a research lab. We were right about many things, but we were wrong about the way that we thought AI was going to go once developed. And so it truly hadn't occurred to us that the most likely path was that we were going to have to build a product company. We mostly thought that we were going to develop this unbelievable superintelligence—and I realize this sounds ridiculous now—and that we wouldn't then have to do all of the hard work of figuring out how to get it usefully into people's hands and for society to benefit. It was like there's that meme from a long time ago of like, do the first thing, do the second thing, question mark, question mark, question mark, profit—about how startups were going to work. Ours was like, do research, build superintelligence, many question marks, people are benefited and happy. And the technology went in a very different path than we originally thought it was going to. I think it's a much better one, but it means we did have to productize this. And this is one reason I think it's a good direction: this is much more like a normal economic force than we imagined at the time. This is like electricity or transistors or some other major scientific discovery that enables many other companies, technologies, scientific discoveries to happen. And we have to put infrastructure—which is building products like ChatGPT, and also building an amount of compute bigger than any project humanity has ever taken on in history—into people's hands. So it became clear by, I would say, 2020, late 2020, early 2021, that we were going to have to build a big company. Didn't think it would happen so quickly. And then when we launched ChatGPT, which was almost four years ago, then overnight everyone was like, okay, we're going to be a big company very fast.

你们确实是一家非常大的公司。我很好奇，如今对你来说最难做的是哪些决策？

Very big company you are. I'm curious to know what sorts of decisions are hardest for you to make today.

我想那会占用我们剩下的时间。我很乐意谈论整个话题，但首先想到的是这个——人们叫法不同：暂停、节奏、智能速度限制。但根本上，如果你相信——我非常相信——人们应该始终掌控未来，我们不应有意将决策权不可逆地交给 AI，也不应意外失去控制。同时，你也相信未来权力需要相当分散。这两者说的是不同版本的：我们希望让我们走到今天的民主多元理想继续下去，我们不希望看到太多权力集中在 IQ 100 亿的 AI 手中，或集中在少数拥有 AI 的人手中，他们可以为所欲为而其他人不能。如果你相信这两点，那么我认为你就会遇到非常困难的问题：我们能让 AI 系统比最聪明的人聪明多少、多快，以及我们需要在多大程度上增强人们变得更聪明或理解这些系统的能力？我认为这是可以解决的，只是需要时间。

I think that could take the rest of our time. I'd be happy to talk about the whole but the number one that comes to mind is this—people call it different things: pausing, pacing, intelligence speed limits. But fundamentally, if you believe—which I very much do—that people should always be in control of the future and we should not either intentionally hand over irreversibly decision-making power to AI or accidentally lose control. And also, you believe that power needs to be pretty distributed in the future. Both of which are saying a different version of: we want the kind of democratic pluralistic ideals that have gotten us this far to continue, and we don't want to see too much power concentration either in an AI with IQ 10 billion or in a small number of people that have AI and can do whatever they want while everybody else can't. If you believe both of those, then I think you get to very difficult questions about how much smarter than the smartest people can we let an AI system get, how quickly, and how much do we need to enhance people's ability to be smarter or to understand these systems? And I think this is solvable. It just takes time.

过去六个月，随着你看到能力加速提升，你自己对于做一个务实的中间派以及 AI 安全的看法是如何演变的？

How has your own thinking about being a pragmatic centrist and AI safety evolved over the past six months as you've seen capabilities accelerate?

我仍然相信我们最早关于 AI 应如何在社会中部署的许多信条。我仍然相信迭代部署。我仍然相信社会必须在我们开发这项技术的过程中与之正面交锋，尽管这很困难，变化速度也惊人。我也仍然相信，人极具韧性、适应力强，而且说实话，人们并不那么在意机器所做的工作。他们太关注人了，以至于未来——即便未来的技术图景看起来非常不同——未来的社会形态会比人们想象的更接近今天。话虽如此，看着过去六个月的进展速度，我上述某些信念的坚定程度受到了考验。在某些狭窄的领域，比如数学推理，就我而言我们现在已经拥有超级智能了，而我没想到它会来得这么快。如果我推演再有一两年类似速度的进展——我认为在世界现状以及公司、政府和社会层面的对齐都不发生任何改变的情况下，我们就会迎来这样的进展——那么在我看来，什么都不改变是行不通的。系统变得如此强大、如此之快，如果我们想守住我刚才提到的那两点，我们就必须改变从这里开始如何控制进展节奏的思路。

I still believe in a lot of our earliest tenets about how we think AI should get deployed in society. I still believe in iterative deployment. I still believe that society has got to contend with this technology as we develop it, even though it's difficult and an incredible rate of change. And I still believe that people are incredibly resilient and adaptable and honestly don't care that much about the work of machines. They're just so focused on people that the future is going to look — even the technological landscape of the future looks very different — the societal shape of the future is going to be much more similar to today than people think. All that said, watching the rate of progress the last six months has tested the strength of my convictions on some of those things. In some narrow senses, certainly mathematical reasoning, as far as I'm concerned we have superintelligence now, and I didn't think it was going to happen this fast. And if I play out another year or two of a similar rate of progress — which I think, without any changes to the current landscape in the world and the alignment at the level of companies and governments and societies, we will have — then it seems to me like that's not going to work, to make no changes to how we do things now. The systems are just getting so capable so quickly that if we want to hold on to those two things I mentioned a little bit ago, we're going to have to make some changes to how we think about pacing progress from here.

人们对这件事有非常强烈的看法。我很好奇，今天这场争论在 OpenAI 内部是如何展开的？

People have very strong opinions about this. I'm curious, how does that debate play out inside of OpenAI today?

我想说，越多人看到并感受到模型变得有多好，争论的分歧就越小，越多人会说：“好吧，我们得想想未来的战略，以及要以某种不同于过去的方式做出我们想要的安全保证需要什么。”我意识到，OpenAI 内部的很多争论——我觉得这对理解社会中的意见分歧也是个很好的借鉴——曾经有很多关于模型会变得多好的争论。很快，这就被重新表述为一场关于正确战略是什么的争论。一旦那个问题有了答案，也就是“好吧，模型即将变得非凡”，那么关于“我们该怎么办”的问题空间就变小了。不过还是有一些。

I'd say the more people see and feel how good the models are getting, the less spread there is in the debate, and the more people are like, "Okay, we're going to have to think about the strategy for the future and what it takes to make the safety guarantees we want to make somewhat differently than we did in the past." I realized that a lot of the debate inside of OpenAI — and I think it's also a good learning for the difference in opinion in society — there was a lot of debate about how good the models were going to get. How quickly that got reframed as a debate about what the right strategy was. Once that debate is answered and it's like, okay, the models are about to get extraordinary, then there's less space on the what-should-we-do-about-it questions. Still some.

你最近在联合国就 AI 安全发表了讲话，并强调了政府在监管 AI 方面的重要性，但传统上政策和技术的变革节奏相当不同。那么你认为 AI 实验室和政府有哪些方式可以更好地合作？

You recently spoke at the UN about AI safety and you emphasized the importance of the government when it comes to regulating AI, but traditionally policy and technological change move at pretty different paces. So what do you think are some ways that AI labs and the government can better work together?

我其实对目前的趋势挺满意的。不是最近——我原本认为大多数政府里的大多数人没有足够认真地对待这件事。过去几个月情况变了，现在我认为对话正在展开。我们拭目以待是否真会有事情发生，但我原本认为没有任何好事处在可能发生的轨道上，而现在我认为大概会有了。

I'm actually pretty happy with how it's trending. Not recently — I didn't think most people in most governments took it seriously enough. The last few months it's changed, and now I think the conversation is being had. We'll see if something actually happens, but I didn't think anything good was on a trajectory likely to happen, and now I think probably yes.

还有一个关于个人或组织层面 AI 安全的问题。今天人们或企业领导者可以做些什么，来更好地为 AI 的潜在风险做准备？

There's also this question about AI safety at an individual or organizational level. What can people or business leaders do today to better prepare themselves for AI's potential risks?

这涉及很多层面。我想说的一点——为了不列出一长串 30 件事——网络安全，我甚至不知道该用什么足够有冲击力的词，但向我们逼近的这个问题，我认为仍然是我们作为组织领导者、国家或个人都还没有认真对待的。今年早些时候有那么一刻，就是所谓的“Mythos 恐慌”或人们怎么称呼它，我认为情况确实有所好转——人们肯定修补了一些漏洞——但这也是又一个人们没有真正直觉理解指数增长的例子。就好像，好吧，有这些模型，它们能找到一堆安全漏洞。那些大概就是所有存在的安全漏洞，或者模型将来能发现的所有安全漏洞。我不认为大多数人会说：“好吧，模型会变得聪明得多。它们大概能找到漏洞，疯狂的漏洞，超出我们通常认为黑客能做的威胁模型的漏洞。硬件层面的疯狂东西，等等。”于是人们修了一些漏洞，安全专家还在恐慌，CISO 还在恐慌，但大体而言我认为大多数组织领导者已经某种程度上翻篇了，而模型还在继续进步。我不再认为我们能够只是修补所有漏洞。我认为我们将不得不彻底切换防御模式，用同样的技术来构建一种新型的防御盾。我认为这是一个巨大的安全与安保问题。而我对它得到的关注相对如此之少感到有些震惊。

There's so many levels of that. I think the one — just so that I don't go through a laundry list of 30 things — the cybersecurity, I don't even know what a dramatic enough word to use, but the problem coming towards us I think is still something that we have not, as organizational leaders or countries or individuals, taken seriously. There was a moment earlier this year where it was like the Mythos freakout or whatever people called it, and I think things did get somewhat better — people definitely patched some bugs — but it was another one of these cases where people didn't really intuit the exponential. So it's like, okay, there are these models, they can find a bunch of security vulnerabilities. Those are probably all of the security vulnerabilities that exist, or all the security vulnerabilities that models are ever going to find. And I don't think most people sort of said, "Okay, the models are going to get much smarter. They can probably find bugs, like crazy bugs, you know, bugs that are outside of the normal threat model of what we think a hacker can do. Hardware-level insane stuff, whatever." And so people fix some bugs, and security professionals are still freaking out, CISOs are still freaking out, but largely speaking I think most organizational leaders have somewhat moved on, and the models have kept going. And I no longer think we're going to be able to just patch all the bugs. I think we're going to have to completely switch the defense model to using the same technology for a new kind of defense shield. And I think it's a huge safety and security issue. And I am somewhat astonished at how relatively little attention it's getting.

回到在座各位，作为斯坦福商学院的学生，我们就在构建 AI 未来的人旁边，但并不总是清楚我们该扮演什么角色。那么对于如何塑造接下来发生的事，你会给我们什么建议？

And to bring it back to the people in this room, as business school students at Stanford, we're right by the people building the future of AI, but it's not always clear where we fit in. So what advice would you give us on how we can shape what comes next?

我诚实的看法是，变化不会像人们想的那么快、那么大。确实，今天由人完成的大量任务将由 AI 做得更好、更便宜、更快、完全不同。也确实，当前工作的很大一部分——至少按今天的方式来做——将完全由 AI 完成。过去 10 天观察数学界的反应其实很有意思。第零天的反应是“这是假的”。第一天的反应是“我们完蛋了”。

My honest belief is that it's not going to be as different as quickly as people think. It is totally true that huge numbers of tasks that are done today by people will be done better, cheaper, faster, completely differently by AI. And it's also true that wide swaths of current jobs will — the way they're done today at least — be entirely done by AI. It's actually been interesting to watch over the last 10 days the reaction of the mathematical community. The day-zero reaction was like, this is fake. The day-one reaction was like, we're cooked.

我们的领域已经结束，我们告诉研究生们要转专业。第三天的反应可能是‘一开始太糟糕了’。我这么做，不应该有人被允许——你知道，如果这是更深入理解数学的代价，我们不想要。但到了第五天左右，情况有了很大转变，我开始听到数学家们说，‘是的，有一种数学工作结束了，但我们大多数人做的并不是解决千禧年大奖难题——我们大多数人从未做过那个。我们做的大部分是弄清楚如何理解知识，如何交流它，如何以人类可理解的方式呈现它。’现在我们已经看到很多人说，‘实际上，我将作为一名数学家拥有一种不同但希望更好的职业生涯，我明白 AI 将从这里开始在很大程度上推动数学发现。但就确定工作内容、其意义、如何呈现、如何解释、如何写出伟大的证明、如何做所有这些其他事情而言——我认为将会有一整套全新的东西。’这是我认为我们所有人最快看到的一个领域经历这个循环，但它的版本在其他领域发生得更慢。你知道，我确信你们许多人看到了过去两年计算机程序员所经历的事情，从说‘我们的工作结束了’，到实际上‘看来我们将继续向那些能让计算机做事的人支付越来越高的薪水。只是工作的日常将会不同。’顾问们经历了一段时期，他们说‘这完全结束了。’然后他们发现，无论 AI 模型在制作漂亮的 PowerPoint 幻灯片方面多么出色——它们确实擅长——人们真的很想与另一个人互动，几乎没有人想让 AI 顾问出现并告诉他们该做什么。你明白这个道理。这种情况会持续发生。所以我不认为会出现许多人谈论的那种工作末日。在人们做的任务和某些类型的工作方面，会有大量的短期变化。但如果你现在正步入世界去从事某项工作，我认为你想在变化程度最高的时期开始你的职业生涯，你想要最开放的运动场。我认为有潜力、有动力、有想法、有看待事物的新方式和新技能的人，在地面震动最剧烈时表现相对最好。我对 MBA 学生的长期失望是，他们总是想去做五年前很酷的事情，而不是做新的事情。我认为这是一个去做新事情的好时机。

And our field is over, and we're telling our grad students to switch majors. The day three reaction was maybe like, 'It's horrible at opening.' I did this and no one should be allowed to—you know, if this is the price of understanding more mathematics, we don't want it. But then by maybe day five it shifted a lot, and I started hearing from mathematicians who were like, 'Yes, there is one kind of math job that is over, but mostly what we did was not solve millennium prize problems—most of us never did that. Most of what we did was figure out how to make sense of knowledge, how to communicate it, how to present it in a human intelligible way.' And we've now seen a lot of people say, 'Actually, I'm going to have a different but hopefully better kind of career as a mathematician, where I get that AI is going to drive mathematical discovery from here to a significant extent. But in terms of figuring out what to work on, what it means, how to present it, how to explain it, how to write a great proof, how to do all these other things—I think there's going to be a whole new set of stuff.' And that was the fastest that I think we've all gotten to watch a field go through this cycle, but versions of it happen more slowly in other fields. You know, I'm sure many of you watched what happened in the last two years with computer programmers saying, 'Our job is over,' to actually, 'Looks like we're going to keep paying ever higher salaries to people that can get computers to do things. It's just the day-to-day of the job is going to be different.' Consultants went through a period where they were like, 'This is totally over.' And then they found out that actually, no matter how good the AI models are at making nice-looking PowerPoint slides—which they are good at—people really want to interact with another person, and almost no one wants the AI consultant to show up and tell them what to do. And you get the point. This is going to keep happening. So I don't think there's going to be a job apocalypse in the way that many people talk about. There will be huge amounts of short-term change in terms of the tasks people do and some kinds of jobs. But if you are heading into the world right now to go work on something, I think you want to be starting your career at a time of the highest degree of change and you want the most open playing field. I think people with potential and drive and ideas and a new way of looking at things and new skills do relatively the best when the ground is shaking the most. My chronic disappointment in MBA students is that they always want to just go do the thing that was cool like five years ago and not do the new thing. And I think this is a great time to go do the new thing.

所以，去尝试新事物吧。即便现在 AI 是热门之选。

So go do the new thing. Even if AI is the cool thing right now.

嗯，我觉得现在不是再创建一个 OpenAI 的好时机。但我认为，有许多公司可以建立在 OpenAI 已向世界推出的技术之上，我希望这些公司能比 OpenAI 更大、更重要。

Yeah. I mean, I don't think this is a good time to start another copy of OpenAI. But I think there are many companies that can be built on top of technology that OpenAI has put out into the world that will be, I hope, much bigger and more important than OpenAI.

现在我想转而谈谈 AI 对我们作为人的身份及未来潜力的影响。AI 让人们几乎可以无需先前经验就能构建任何东西。在这个世界里，即使 AI 能替人们完成某些事情，人们还应该学会自己去做哪些事呢？

I want to shift now to AI's impact on who we are as humans and future potential. AI makes it possible for people to build almost anything without prior experience. In this world, what should people still learn to do for themselves even if AI can do it for them?

举个例子，有人曾向我强调学会某项技能对我个人很重要，尽管那些是计算机。比如，我二年级的老师坚信我必须学会写连笔字。我大概是最后一代被迫学习连笔书写的学生了，当时我就想：“我这辈子肯定用不上这个，我敢肯定。我现在连签名都不用连笔，如果现在让我写，我可能都写不出来了。”我接着说：“如果你让我学这个是因为我字写得不好，需要锻炼精细动作技能，或者七岁孩子能表达的任何理由，我能接受。但如果你认为我将来真的会因为连笔字本身而需要它，那我可不相信。”老师回答：“不，作为社会的一员，你将来会需要的。你必须会写连笔字。”好吧。几年后，另一位老师对我说：“你必须学会写作。”我心想：“我根本不想成为作家。”那位老师说：“那没关系，但写作主要是学习如何思考的重要工具。”不管我是不是说：“我要成为数学家。”老师都说：“你仍然需要学会思考，学会组织思想。即使你长大后从不为了给别人看而写作，学会思考也是一项重要的技能。”我的领悟是，这两次经历都有重要的教训。学习有助于思考的技能极其宝贵。而那些明显将被机器取代，不再受人们重视的技能，显然就不那么有价值了。这些道理显而易见，对吧？所以，大多数人会同意，我们仍应学习这些通用技能，而不应去学那些AI能做得非常好的事情。但接下来的问题是，随着我们作为人类发展通用技能，进行自我锻炼——未来哪些领域仍属于人类。首先，我认为显而易见但值得说明的是：人类的价值观因其人性而具有内在价值。我们似乎真的不太关心机器为了自身目的做什么——我认为这很好。无论AI变得多么强大，这一点都将持续。某种意义上，科学史就是人类主角光环不断遭遇失望的历史。我们曾以为太阳绕地球转，我们确实处于宇宙的中心，后来发现整个太阳系，再后来是整个银河系，这一过程持续不断，直到多元宇宙，你不得不眯起眼睛想，为什么人类会是中心。但我们感觉自己非常重要，确实是自身故事的中心。我认为这很好，而且这种趋势会继续下去。

So an example of a thing that someone promised me was important to learn to do for myself, even though they were computers, is my second grade teacher was absolutely convinced I had to learn to write cursive. I was like the last generation of students that had to learn to write cursive, and there was a whole—I was like, 'I am never going to use this again. I am so sure. I don't even sign my name in cursive. I couldn't write cursive today probably if I had to.' And I was like, 'If you're making me learn this because I have bad handwriting and I need to learn fine motor skills or however a seven-year-old would articulate that, I can believe it. But if you think I'm actually going to need to learn cursive for the sake of cursive, then I don't believe you.' And my teacher was like, 'Nope, you're going to need this as a functioning member of society. You have to be able to write cursive.' Okay. And then I had another teacher a couple years later who was like, 'You have to learn to write.' And I was like, 'I have no desire to be a writer.' And that teacher was like, 'Well, that's fine, but writing is mostly an important tool to learn how to think.' And whether or not I was like, 'I'm going to be a mathematician.' And the teacher was like, 'Well, you still have to learn how to think and you still have to learn how to structure ideas. And even if you never write as a grown-up for anyone else to read, it is an important skill to learn how to think.' And my takeaway is that there was an important learning from both of those. Still learning skills that help you think is incredibly valuable. Learning skills that are clearly going to be in the realm of machines and not something that people value are clearly not valuable. And those are kind of obvious, right? So I think most people would agree directionally that we should still learn these general things. We should not learn things the AI is going to do really well. But then there's a question of we develop our general skills, we do our exercise as people—what remains in the domain of people in the future. Number one, I think obvious but worth stating: human values have inherent value because they are human. We really do not seem to care—and I think this is great—about what machines do for their own purposes. And I think no matter how good AI gets this will continue to be the case. In some sense, the history of science is this history of successive disappointments to the main character energy of humanity. We used to think that the sun rotated around the earth and that we were really at the center of that, and then we realized there was this whole solar system and then there was this whole galaxy and this goes on and on and you get to this multiverse and you have to squint and wonder why people are the center of it. But we feel like we are very much at the center of what matters and we are the center of our own stories for sure. And I think this is great and I think this will keep going.

我认为，无论 AI 在能力上走多远，只要我们把安全做好，只要不出现那种失控事件，人类的价值对我们所有人都始终重要，因为背后是一个个人。工作也会重要，因为是人去做。我们永远都会这样在意。所以，想清楚如何在场、如何对他人有用、如何创造出对他人至关重要的东西——这件事会永远持续下去，而我认为学会做到这一点，是一项出人意料地可以习得的技能。

And I think that no matter how far AI goes from a capability perspective, as long as we get the safety right, as long as we don't have one of these loss-of-control incidents, human values will matter to all of us because there is a person behind them. Work will matter because a person does it. And we're just always going to care that way. So figuring out how to be there, how to be useful to other people, how to create stuff of central importance to other people — that's going to go on forever, and I think learning how to do that is a surprisingly learnable skill.

假设我们在 AI 的分配和发展上都做对了。你认为今天面向未来、人们仍然严重低估我们解决能力的问题是什么？

And let's say we get the distribution and development of AI right. What do you think is a problem facing the future today that people still dramatically underestimate our ability to solve?

这个问题问得时机挺有意思，因为过去几天我和你的对话里，连那些怀疑论者都说，你知道吗，到这一步我明白了——AI 会解决所有知识问题。我不完全同意，但方向上我也有点这种感觉。什么叫过上好的人生，让每个人都能有选择、自由、灵活度，仍然可以犯错，但人们回头看、拉远视角时能说，是啊，我过了很棒的一生——我认为这仍然会很难。我觉得某种意义上它会一直很难。也许难本身很重要。我希望未来几年，我们更多推向一种新文艺复兴的感觉，而不是新工业革命。但我认为 AI 的很多能量都奔着工业革命去了：我们要自动化一切，完美地把人从等式里拿掉，让经济跑得更快更好，给每个人发点钱，由 AI 决定什么重要，人只是机器里运转得更顺滑的齿轮，而机器会变成某种不可思议的莫洛克式怪物。我希望这一局我们能做对。

Well, it's a funny time to ask that because in my conversations with you the last few days, even the people who are skeptics are like, you know what, at this point I get it — AI is going to solve every piece of knowledge. And I don't quite agree with that, but I directionally kind of feel that way. What it means to have a great life and enable everybody to have the choice, freedom, flexibility to still make mistakes but then when people look back and zoom out say, yeah, I had a great life — I think that will still be hard. I think that will always be hard in some sense. Maybe it's important that it's hard. I am hopeful that as these next few years roll on, we push much more towards what feels like a new Renaissance than a new Industrial Revolution. But I think a lot of the energy in AI has been towards the Industrial Revolution: we're going to automate everything, we're just going to perfectly take humans out of the equation and make the economy run faster and better, and we're going to give everybody some money and the AI will decide what's important, and people will just be a cog running more smoothly in a machine, and the machine is going to be this incredible Moloch thing. And I hope we get that one right.

你认为在通往这种文艺复兴式未来与工业革命式未来之间，最大的障碍是什么？

What do you think is the biggest thing standing in the way between this Renaissance version of the future versus this Industrial Revolution style version of the future?

我其实认为人们想要的是文艺复兴版本。我认为人们想要创造力、探索事物的能力、发现知识的能力，想要这种广阔开放的新自由、新方向和新探索空间。我认为社会和经济的很多压力都太偏向冷酷的效率，所以大概需要有来自公司、政府或全体人民的力量去对抗它。

I actually think people want the Renaissance version. I think people want the creativity, the ability to explore stuff, the ability to discover knowledge, the ability to have this wide-open space of new freedom and directions and exploration. I think a lot of the pressures of society and the economy are just so positioned toward ruthless efficiency that there will probably have to be some force against that from the companies or the governments or people as a whole.

现在我们把时间交给观众提问。

We're going to turn it over to our audience questions now.

好的。嗨 Sam，我叫 David Lee，是 2027 届 MBA 学生，来自湾区。我的问题是：OpenAI 有远见地在 2024 和 2025 年重金投入算力，我认为这项投资正通过 Astra 和 ChatGPT 的速度得到回报。你们现在押注什么基础设施来推进 OpenAI 的使命？

Okay. Hey Sam, my name is David Lee. I'm part of the MBA class of 2027 and I'm originally from the Bay Area. My question for you is: OpenAI had the foresight to invest heavily in compute in 2024 and 2025, and I think that investment is paying off with Astra and the speed of ChatGPT. What infrastructure are you all betting on now to further the mission of OpenAI?

机器人。人们经常谈论 AI 模型的递归自我改进。还有另一个循环：如果你有能造更多机器人的机器人，然后有能有效建造更多数据中心的数据中心，包括运营芯片晶圆厂之类，你就能一路造出多得多的算力，直到建更多发电厂。我认为这件事和 AI 研究里潜在的递归循环一样大，或者几乎一样大。它被谈论得少得多。这是其一。另一个是我们做了自己的芯片，叫 Jalapeño，后面还有一款，从每瓦智能的角度看，相对市场上任何其他东西都是巨大的胜利。我们会继续推进，而随着模型变聪明，它们能帮我们设计越来越聪明的芯片。所以这是另一个循环。我们在整条供应链上投入很多——晶圆厂、电力、所有设备和数据中心。所以我们会做很多这方面的事，但这大概是三大领域。

Robots. So I think people talk a lot about recursive self-improvement for AI models. There is another loop, which is: if you have robots that can build more robots, and then data centers that can effectively build more data centers, including operating chip fab facilities and such, you can make a lot more compute all the way down to building more power plants. But I think this is as big of a deal, or almost as big of a deal, as the potential recursive loop in AI research. It gets much less talk. So that's one. Another is we made our own chip called Jalapeño, and we have another one coming after it that are huge wins from an intelligence-per-watt perspective relative to anything else out in the market. And we will continue to push there, and as the models get smarter they'll be able to help us design smarter and smarter chips. So that's another one of these loops. We're investing a lot in the whole supply chain — fabs, power, all the equipment and data centers. So we'll do a lot of that, but those are probably the three big areas.

嗨 Sam，我是 Sri，来自印度，2027 届 MBA。你 30 岁生日时写过一份人生经验清单，叫《日子很长，但十年很短》。现在你 41 岁，也当了父亲，我很好奇：你会对那份清单做什么修改？有没有哪条你会重写？

Hi Sam, I'm Sri, MBA class of '27 from India. On your 30th birthday, you wrote a list of life lessons called "The days are long but the decades are short." And now that you're 41 and a father, I was curious: what would you change about that list? Are there any that you would rewrite?

我记得我写过那个。里面每一条我不记得了，但我现在可以给一个很短的版本。我记得它挺长的。大概会是：组建家庭，构建对你重要的东西，花时间享受乐趣、做你觉得充实的事，保持健康。会非常短。我有点觉得，这种事你就该听老人的，他们看起来确实说得对。

I remember that I wrote that. I don't remember everything in there, but I could do a very short version right now. I think it was pretty long. It would be like: have a family, build something that is important to you, spend time having fun and doing the things that you find fulfilling, stay healthy. It would be very short. I kind of think this is a case where you should just listen to old people when they look like they were all right.

你好 Sam，感谢你的时间。我叫 Jerko，2028 届，来自秘鲁。我的问题是：2020 年你关于 AI 将如何发展所持的哪个强烈信念后来被证明是错的？它如何改变了你运营 OpenAI 的方式？

Hello Sam, thank you for the time. My name is Jerko. I'm from the class of 2028. I'm from Peru. And I have a question on: what is a strong belief you held back in 2020 about how AI will unfold that has come to be wrong, and how did it change how you run OpenAI?

你说从什么时候？

From when, did you say?

从 2020 年？2020 年底。

From 2020? Late 2020.

是的。我没想到我们能走到那样的模型——更好的说法是：我以为对齐问题会更难。从现在起它可能仍然会非常难。事实上，我怀疑它会难得多，我也认为从我们接下来要走向的地方看，它极其重要。但事实是我们得到了在某些重要方面超越人类的模型，而对齐到目前为止比预期容易——我本来会高度确信地说，这两者中有一个是错的。现在我担心的是，有些人看着我们已经达到的位置说，到 n+1 还会继续，对齐会一直容易。我认为这完全不显然。我也认为对齐出错的潜在代价已经大幅上升。我越来越相信，有一段时间我们可以把这些系统放进沙箱，我认为我们可以基于沙箱做出一些安全保证；然后当这失效时——因为模型变得太聪明，我们根本无法理解沙箱还意味着什么——那时我认为可监控性会在一段时间内起作用。

Yeah. I didn't think that we were going to be able to get to models that were — I guess a better version of saying it is: I thought the alignment problem was going to be harder. It may still turn out to be very hard from now. In fact, I suspect it will get much harder, and I also think it is really incredibly important from where we go from here. But the fact that we got to models that are superhuman in some important ways, and alignment has so far been easier than expected — I would have said one of those two was wrong with a high degree of certainty. Now a thing I'm worried about is that some people look at where we've gotten and say, for n plus one it's going to keep going and alignment is going to stay easy. And I think it's actually not at all obvious to me that that's the case. And I also think that the potential cost of alignment being wrong has gone up enormously. I increasingly believe that for a while we can sandbox these systems and I think we can make some safety guarantees based off the sandbox, and then when that fails because the models get so smart we can't really understand what a sandbox would even mean, then I think we'll have monitorability that'll work for a while.

但最终，我之前提到那个速度限制的原因，不是别的，正是对齐——我认为它实际上会起作用。我不认为有任何我们足够聪明能理解的对齐，能真正自信地说“可以运行这个模型”，并且人们能保持对未来的控制。我担心世界上一些 AI 项目对对齐如此自信，鉴于到目前为止它一直很容易，以至于他们太愿意往前冲了。

But eventually, and the reason I was talking about that speed limit earlier, nothing but alignment—I think actually will work. I don't think any alignment that we're smart enough to understand and can really say with confidence, "It's okay to run this model," and people will stay in control of the future. And I worry that some AI efforts around the world are so confident in alignment, given how easy it's been so far, that they're too willing to run ahead.

嘿 Sam，感谢你来到这里。我是 Josh Island，MBA 2027 届。你最近在 Decoded 播客上说，你认为世界应该接受一些坏事发生，以换取 AI 的好处。当你想到你儿子将要成长的世界时，你认为我们应该具体接受哪些坏事，又在哪里划清界限？

Hey Sam, thanks for being here. I'm Josh Island, MBA class of 2027. You said recently on the Decoded podcast that you think that the world should accept some bad things happening in exchange for the benefits of AI. As you think about the world that your son's going to be growing up in, what are the bad things you think we should be okay with specifically versus where you draw the line?

是的。我希望我当时措辞不同。我本意在我脑子里似乎如此明显，以至于毫无争议。但我明白为什么它没有那样传达。人们会有开源模型，它们会输出一堆仇恨言论、在互联网论坛上捣乱、入侵一些系统，并代表其用户以某种方式尝试欺诈。例如，会有新一代电话诈骗，比上一代糟糕得多。我认为有人会说，如果那会发生，那就禁止所有开源模型。我在那个播客中试图表达但我觉得被剪掉的观点是，禁止所有开源模型将对社会的创新和自由造成极高的代价，我们不应该这样做。但如果你要有开源模型，如果你要让人们在本地电脑上运行它们，无论你多么擅长全球智能体监控，总会有一些你希望没人做过的事情。我不会接受说完全不要开源模型的交易。现在有一个阈值，我认为社会需要辩论，超过这个阈值，模型能造成足够的伤害，我们就不给人们随意使用它们的自由。明显的例子是，没有人应该能够瘫痪互联网。没有人应该能够对他人造成身体伤害。没有人应该能够制造一种新型生物恐怖袭击并释放到世界上。我从那件事中学到的一点是，如果你告诉人们接受任何数量的坏事发生，他们只会说我不想要任何坏事发生，他们会失去自由也很重要这个微妙之处，真正重要的是我们在哪里划线。就像如果我们回到电话诈骗的例子，你可以直接拿走所有电话，就永远不会有电话诈骗，但我们认为那不值得。所以这就是我想表达的观点。我不认为我们作为一个社会应该承担大风险。事实上，这就是为什么我认为 AI 不同于以往任何技术：它是在一个人能造成伤害的量上真正的新一步。随之而来会有许多不舒服的后果，关于社会必须如何适应以及我们确实必须放弃的一些自由。但我不认为这意味着我们不应该在我们的电脑上运行神经网络。

Yeah. I wish I had phrased that answer differently. I like the way I meant it seemed so obvious in my head that it was non-controversial. But I see why it didn't come across that way. People are going to have open source models that are going to output a bunch of hate speech and troll internet forums and that are going to hack into some systems and that are going to try to commit fraud in some sorts of ways on behalf of their user. There will be a new generation of phone scams, for example, that would be much worse than the previous generation. And I think there are people who would say if that's going to happen, let's ban all open source models. And the point I was trying to make in that podcast, but I think got clipped out, was banning all open source models would be an extremely high cost to innovation and liberty in society and we shouldn't do it. But if you are going to have open source models and if you're going to let people run these locally on their computer, no matter how good you are at global agent monitoring, there's going to be some stuff that you wish someone hadn't done. And I wouldn't take the trade of saying no open source models whatsoever. Now there is some threshold, and this is where I think society needs a debate, above which the models can do sufficient harm that we don't give people the liberty to use them however they want. And obvious examples are no one should get to take down the internet. No one should be able to cause physical harm to another person. No one should be able to make a new kind of bio terror attack that they put out into the world. One of my learnings from that thing is if you tell people to accept any amount of bad things happening, they'll just say I don't want anything bad happening at all, and they will lose the subtlety of like liberty is important too and what really matters is where we draw the line. Like if we go back to the example of scams on phones, you could just take away all phones and no one would ever have a phone scam, and we decide that's not worth it. So that's the point I was trying to make. I don't think we should take big risks as a society. And in fact, that's why I think AI is different than any technology that has come before it: it is a really new step in the amount of harm one person can cause. And there are a bunch of uncomfortable consequences that are going to follow from that about how society has to adapt and some freedoms we do have to give up. But I don't think it means we shouldn't get to run neural networks on our computers.

嗨 Sam，我是 Shiraz，MBA 2028 届。感谢你和我们在一起。我是作为软件开发者来的。所以我的一个问题是，随着模型能力随时间提高，我们看到的一个问题是软件护城河正日益被越来越好的 AI 模型吞噬。所以对于下一代崭露头角的企业家，你有什么建议，关于软件中什么样的护城河，或者一般来说对于软件开发者，我们可以致力于什么样的技术或什么样的软件来在未来建立初创公司或创业项目？

Hi Sam, I'm Shiraz, MBA class of 2028. Thanks for being here with us. So I'm coming in as a software developer. So one of my questions was, as model capabilities get better over time, one of the problems we're seeing is that software moats are increasingly being swallowed by better and better AI models. So for the next generation of budding entrepreneurs, what advice would you have on what kind of moats within software or generally for people that are software developers, what kind of technologies or what kind of software can we work on to build a startup or an entrepreneurial venture in the future?

粗略地说，我认为历史上软件护城河并不多。还有其他类型的护城河。有像 Facebook 那样的网络效应。有像 Google 那样的规模效应。有像许多企业软件公司拥有的数据护城河。但软件本身从来都不是一个超级强大的护城河。你用它来获得临时优势，以建立企业传统护城河之一。我有点认为这只会以同样的方式继续下去。

To a first approximation, I think there haven't been that many software moats in history. There are other kinds of moats. There's network effects like Facebook or something. There are scale effects like Google. There are data moats like a lot of enterprise software companies have. But software itself has never been a super strong moat. You use it to get a temporary advantage to build up one of the traditional moats of a business. And I kind of think that's just going to keep going the same way.

我是 Adam。我也是 MBA。我还是 OpenAI 基金会的幕僚长。酷。只是提一下。所以我有一个与此相关或后续的问题。随着 AI 变得更强大，在让技术广泛惠及全人类与确保其安全开发和部署之间，存在日益增长的张力。所以我想知道，你如何看待开放权重和封闭权重模型之间的权衡，以及要发生什么你才会觉得可以放心地将最强大的模型开放权重？

I'm Adam. I'm an MBA too. I'm also a chief of staff at the OpenAI Foundation. Cool. Just throwing that out there. So I've kind of got a bit of a related or follow-up question to this one. As AI becomes more powerful, there's a growing tension between making the technology broadly accessible to all of humanity while also ensuring it's developed and deployed safely. So I wonder how do you think about the trade-off between open-weight and closed-weight models and what would have to be true for you to feel comfortable open-weighting the very most capable models?

嗯，我认为开放权重模型对社会非常重要，直到相当高的能力水平，而社会为开放权重模型的好处究竟会接受多少损害，这个问题在未来几年将受到非常激烈的辩论。这也是一个移动的目标，因为社会建立的韧性越多——比如 OpenAI 基金会，我希望，将做很多工作来帮助预防未来的大流行病——然后也许世界可以拥有更强大的生物模型，因为我们可以比今天更好地应对。所以这不是一个静态的问题。随着世界用 AI 建立更多韧性，它会演变。但我不认为我们可以负责任地在短期内开源我们当前最好的内部模型。我在想世界必须发生什么——比如互联网运作的整体网络防御模式必须改变，我们必须建立这个生物层，还有一长串其他事情。现在，这是一个有趣且活生生的问题的原因是：任何人真的相信那个模型比将开源那个级别模型的人领先多少？不是超级多。所以我认为这种封闭源代码、封闭权重模型的滚动优势，帮助世界建立韧性并防御开放模型——我认为世界确实应该得到开放模型——是超级重要的。而且部分——我不知道你在 OpenAI 基金会工作。这不是托，但我会接受。

Well, I think open-weight models are very important for society to have up to some fairly high level of capability, and the question of exactly how much damage will society accept for the benefits of open-weight models is something that is going to get very heavily debated in the next couple of years. It's also a moving target because the more that society builds up resilience—like OpenAI Foundation is, I think hopefully, going to do a lot of work to help prevent against future pandemics—and then maybe the world can have more capable bio models because we can respond better than we could today. So it's not a static question. It'll evolve as the world builds up more resilience with AI. But I don't think we could responsibly open source our current best internal model anytime soon. I'm trying to think what would have to happen to the world—like the total cyber defense model of the way the internet works would have to change, we'd have to build up this bio layer, and a long list of other stuff too. Now the reason that this is an interesting and live question is: how far ahead does anyone really believe that model is from someone who will open source a model of that level? Not super. And so I think this whatever this rolling advantage is of closed-source, closed-weight models to help the world build the resilience and defend against the open models that I think the world does deserve is super important. And part of—I did not know you work for the OpenAI Foundation. This is not a plant, but I'll take it.

我是 Sam。首先，感谢你今天与我们同在。去年你参加 David Perell 的“How I Write”播客时，分享了你那套模拟笔记系统——你整天带着笔记本，写下你的想法。在你的生活中，无论是个人还是职业方面，有没有哪些领域你试图保持更低调、更模拟，以保留你作为一个人和专业人士的一些技能？你会建议我们如何遵循同样的路径？

Name is Sam. First of all, thank you for being with us today. Last year when you joined David Perell's 'How I Write' podcast, you shared the analog note-taking system you have where you carry a notebook with you all day and write down your thoughts. Are there any areas in your life, both personal and professional, where you try to stay more low-tech and more analog to preserve some skills you have as a person and as a professional? And what would be your suggestions for us to follow the same path?

比如记事本。我觉得我在这点上就是错了，我可能应该改用电脑，但这是旧习惯使然。我认为真正重要的模拟东西是那些老生常谈——老人总是对的。比如花很多时间和他人相处，一起吃饭，多去大自然，锻炼身体。有一长串模拟的东西已经行之有效很久了，但我觉得都是显而易见的东西。而试图说“哦，你知道，有人用记事本写字”，那他们可能只是错了。那个你大概可以停止。

Like the notepad. I think I'm just wrong there and I probably should switch to a computer, but it's an old force of habit. I think the analog stuff that matters is all of the stuff with speaking of like old people are always right. It's like spend a lot of time with other people, have meals together, go out into nature a lot, exercise. There's a long list of analog stuff that has worked for a long time, but I think it's all the obvious stuff. And trying to be like, 'Oh, you know, someone writes with a notepad,' like they're probably just wrong. That one you can probably just stop.

Sam，今天我还有最后一个问题。在商学院，我们谈论改变生活、改变组织、改变世界。当你审视当今世界时，你认为未来的领导者需要关注什么来推动变革？

Sam, I have one final question for you today. At the GSB, we talk about changing lives, changing organizations, and changing the world. As you look at the world today, what do you think future leaders need to focus on to drive change?

我最明显的答案就是 AI。如果我们选其中最容易的、或者最有吸引力的一个来谈改变组织：我认为现在有可能设计与以往不同的组织，让 AI 相对于组织中每个人的比例大大提高。我觉得这是一个未被充分探索的研究领域——今天建立尽可能最好的组织意味着什么。我认为这非常非常不同。这纯粹是推测；我从未研究过这是否正确，但我的感觉是，大多数组织的设计方式源于模仿军队。我们建立这种非常等级化的命令控制结构，是为了在超过一定规模后还能扩展。那是我们拥有的唯一一个能做到这一点的例子。组织似乎有了一些这样的特点，然后在过去几十年里，科技行业和其他行业也对此进行了创新，并有所改进，但它仍然受到那个基本原则的启发。而观察这些 AI 系统时，让我极其感兴趣的一件事是，如果你看——我想是 10,000 到 10,000 个智能体解决纳维-斯托克斯方程——它们合作的方式是我能想象到的任何组织都无法比拟的。没有领导者。没有争功。完美对齐。没有争吵。每个人都在努力解决这个目标。人类有一些美丽的缺陷，这意味着我们永远做不到那样。人们会以许多与过去相同的方式工作。显然，我不希望那只是一个无情的世界，但我认为仍然有一些东西可以从中学习。当我们在 OpenAI 内部观察时，真正拥抱 AI 的团队规模小得多，他们改变了工作方式。我相信你们都会是那些去弄清楚如何设计新组织的人。我认为世界将真正从中受益。

My very obvious answer is AI. If we pick the easiest one of those, or the most attractive one, to talk about changing organizations: the degree to which I think it's possible to design a different kind of organization now with a lot of AI relative to each person in it. I think it's just an underexplored area of study—what it means to set up the best possible organization today. I think it's very, very different. This is pure speculation; I've never studied to know if this is true or not, but my sense is that the way most organizations are designed came from copying the military. There was a way that we set up this very hierarchical command-and-control thing to scale beyond a certain point. That was the one example we had of something that could do it. Organizations seem to have some of that, and then over the last few decades, the tech industry and other industries too have innovated on that and somewhat improved it, but it's still inspired by that same basic principle. And one of the things that's been extremely interesting to me about watching these AI systems is if you look at—I think it was 10,000 to 10,000 agents that solve Navier-Stokes—they work together in a way that is unlike any organization I could imagine. There was no leader. There were no credit squabbles. There was perfect alignment. There was no bickering. Everyone was just trying to solve this goal. And there are beautiful flaws in humans that are going to mean we never do that. People are going to work in many of the same ways that they've worked in the past. And I wouldn't want to just be that kind of emotionless world, obviously, but I think there is still something to learn from that. And as we have watched inside of OpenAI, teams that really embrace AI and are much smaller change the way that they work. I'm sure you all will be the people that go figure out how to design new organizations. And I think the world will really benefit from it.

Sam，这真是一次愉快的经历。

Sam, this has been a pleasure.

谢谢。

Thank you.

请和我一起感谢 Sam Altman。

Please join me in thanking Sam Altman.