AI 未来：硬件、规模扩展与入门指南

如果你展望未来几年，用于运行你自己模型的硬件会变得比人们预期的更快。我认为这将解锁的能力是，他们能够扩展这些模型，你会看到与以往截然不同的行为。在 OpenAI，我们有时会看到这一点。例如，我们有一篇关于无监督学习的论文，你训练一个语言模型来预测亚马逊评论中的下一个字符。仅仅通过学会预测亚马逊评论中的下一个字符，它不知何故就学会了一个最先进的情绪分析分类器。想想看，这有点疯狂：你只是被告知要预测下一个字符。如果你被告知要做好这件事，你首先要做的是学习单词的拼写和标点符号。接下来，你开始学习语义，如果你有实际的能力的话。如果你用一个稍微小一点的模型，这种效果就会消失。如果你用一个稍微大一点的模型会发生什么？我们不知道，因为我们现在还无法运行这些模型。但在未来几年，我们将能够做到。

Now if you look forward to what's going to happen over upcoming years, the hardware for these applications for running your own models is going to get fast faster than people expect. I think that what that's going to unlock is they're going to be able to scale up these models, and you're going to see qualitatively different behaviors from what you've seen so far. At OpenAI, we see this sometimes. For example, we had a paper on unsupervised learning where you train a language model to predict the next character in Amazon reviews. Just by learning to predict the next character in Amazon reviews, somehow it learned a state-of-the-art sentiment analysis classifier. It's kind of crazy if you think about it: you just were told to predict the next character. If you were told to do this well, the first thing you do is learn the spelling of words and punctuation. The next thing you do, you start to learn semantics, if you have actual capacity there. And this effect goes away if you use a slightly smaller model. What happens if you have a slightly larger model? We don't know because we can't run those models yet. But in upcoming years, we'll be able to.

你们认为人工智能中最有前途但尚未充分探索的领域是什么？如果我们想让它更快地发展，人们应该研究哪些他们目前没有在做的领域？

What do you guys think are the most promising under-explored areas in AI? If we're trying to make it come faster, what should people be working on that they're not?

是的，人工智能的许多领域我们已经有了相当程度的发展。有一些基础研究只是在分类、规划、强化学习方面。人们所做的是，他们试图发明问题，比如解决一些复杂的游戏，如围棋或星际争霸，然后他们试图给他们的模型添加额外的功能来应对这些问题。但我认为，真正理解现有方法及其局限性的研究非常少。例如，在深度学习领域，长期以来人们一直认为，要并行化计算，你需要在每个设备上塞入尽可能小的批次。事实上，百度完成了一项令人印象深刻的工程壮举，他们采用了循环神经网络，并实现了 GPU 汇编代码，以确保每个 GPU 上可以容纳批次大小为 1。尽管所有聪明的人都在研究这个问题，但直到最近，Facebook 才悄悄地审视了一个非常基本的分类问题。在他们名为“ImageNet 在一小时内”的伟大努力中，他们表明，如果你实际采用进行图像分类的代码并修复所有错误，你实际上可以使用更大的批次大小，从而可能更快地完成分类。这不是人们想看到的那种性感的研究，但我认为这种研究在现阶段将最大程度地推动该领域的发展。

Yeah, so many areas of AI that we already developed by quite a bit. There is some basic research in just classification, planning, getting reinforcement learning. And what people do is they kind of try to invent problems, such as solving some complicated games like Go or StarCraft, and they try to add extra features to their models to combat those problems. But I think there is very little research happening on actually understanding the existing methods and their limits. So for example, it was a long-held belief in deep learning that to parallelize your computation, you need to cram as small batches as possible on every device. In fact, Baidu did this impressive engineering feat where they took recurrent neural networks and implemented GPU assembly code to make sure that you can fit batch size one on every GPU. And despite all the smart people working on this problem, it was only very recently that Facebook took a quiet look at a very basic problem of classification. In their great effort called ImageNet in one hour, they showed that if you actually take code that does image classification and fix all the bugs, you actually can get away with much larger batch sizes and therefore finish the classification probably much faster. It's not the kind of sexy research that people want to see, but this kind of research at this point will advance the field the most.

Greg，你在最初的回答中提到了硬件。在短期内，你预见到哪些实际的创新会发生？

Greg, you mentioned hardware in your initial answer. In the near term, what are the actual innovations that you foresee happening?

最大的变化是，我们一直试图制造得非常快的计算机是建立在冯·诺依曼架构上的通用计算机：基本上是一个处理器，一个大内存，以及两者之间的瓶颈。随着我们现在开始做的应用，突然间你可以利用大规模并行计算。这些模型能够运行得最快的架构将有点像大脑，在那里你有一堆神经元，它们都有自己的内存就在附近，它们都与邻居交流，也许还有某种长程跳跃连接。没有人真正有动力去开发这样的硬件。我们已经看到的是，当你把你的神经网络从 CPU 上运行转移到 GPU 上时，突然间你有一千个 CUDA 核心并行运行，你可以获得巨大的性能提升。现在，如果你转向更类似大脑的专用硬件，并与一堆微小的核心并行运行，你将能够以疯狂的速度运行这些模型。

The big change is that the kinds of computers we've been trying to make really fast are general-purpose computers built on the von Neumann architecture: basically a processor, a big memory, and a bottleneck between the two. With the applications we're starting to do now, suddenly you can start making use of massively parallel compute. The architectures that these models can run on the fastest are going to look kind of like the brain, where you have a bunch of neurons that all have their own memory right near them, and they all talk to their neighbors, and maybe there's some kind of longer-range skip connections. No one's really had an incentive to develop hardware like this. What we've seen is that when you move your neural networks from running on a CPU to a GPU, suddenly you have a thousand CUDA cores running in parallel, and you can get massive performance boost. Now if you move to specialized hardware that is much more brain-like and runs in parallel with a bunch of tiny little cores, you're going to be able to run these models insanely faster.

我认为在 Twitter 和 Facebook 上被问到的最常见的问题之一是如何进入人工智能领域。你们能给我们一个入门指南，告诉一个大学计算机专业的学生应该从哪里开始吗？

I think one of the most common questions or threads of questions that were asked on Twitter and Facebook was generally how to get into AI. Could you guys give us just a primer of where someone should start if they're just a CS major in college?

是的，当然。这实际上取决于你想做的项目的性质。我可以告诉你一些关于我们项目的情况，它本质上是在为 Dota 2 开发大规模的强化学习。大部分工作实际上是工程：把我们已经实现的算法进行扩展。这通常是我们实验中获得改进的最快方式。所以基本上，为我们的团队成为一名优秀的工程师比，比如说，人们花大部分时间在流程上实现异国模型要有价值得多。

Yeah, absolutely. So it really depends on the nature of the project that you would like to do. I can tell you a bit about our project, which is essentially developing large-scale reinforcement learning for Dota 2. The majority of the work is actually engineering: taking the algorithms that we have already implemented and trying to scale them up. It's usually the fastest way to get improvement in our experiments. So essentially, becoming a good engineer for our team is much more valuable than, for example, people spending most of their time implementing exotic models in terms of flow.

只是想呼应一下，因为我经常听到这样的说法：‘我的梦想是在 OpenAI 工作，但我得去拿一个人工智能博士学位，所以五七年后再见吧。’如果一个人只是一个非常扎实的工程师，但完全没有人工智能经验，这样的人需要多长时间才能在我们寻找的 OpenAI 工作中变得高效？

Just to echo this, because I hear this come up all the time: 'It's my dream to work at OpenAI, but I gotta go get an AI PhD, so I'll see you in like five or seven years.' If people are just a really solid engineer but with no experience at all with AI, how long does it take someone like that to become productive for the kind of work at OpenAI that we're looking for?

这样的人实际上可以从第一天就开始高效工作。来到 OpenAI 的工程师各不相同；他们最终专攻的领域有一个范围。有些人专注于构建基础设施。这种基础设施可以包括我们在云平台上运行的大型 Kubernetes 部署，构建工具和监控，以及管理这个底层。这实际上看起来很像运营一家初创公司。很多在这方面最成功的人都有在初创公司或生产环境中大规模运营的丰富经验。还有下一层次，更接近实际的机器学习。如果你考虑机器学习系统的样子，它们往往在核心是一个神奇的机器学习黑盒子。你实际上试图让这个核心尽可能小，因为机器学习非常困难，消耗大量算力，而且很难知道里面发生了什么。所以你希望它尽可能简单，但然后你用尽可能多的工程来包围它。那么百分之多少...

Someone like that can actually become productive from day one. There are different engineers who show up at OpenAI; there's a spectrum of where they end up specializing. Some people focus on building out infrastructure. This infrastructure can range from having a big Kubernetes deployment that we run on top of a cloud platform, building tooling and monitoring, and managing this underlying layer. It actually looks quite a bit like running a startup. A lot of people who are most successful at that have quite a bit of experience running large-scale in a startup or production environment. There's kind of a next level of getting closer to the actual machine learning. If you think of how machine learning systems look, they tend to be this magical black box of machine learning at the core. You actually try to make that core as small as possible because machine learning is really hard, eats a lot of compute, and it's really hard to tell what's going on there. So you want it to be as simple as possible, but then you surround it by as much engineering as you possibly can. So what percent of...

你们觉得 Dota 2 项目的工作，本质上更像是人们通常理解的机器学习科学还是工程？就日常工作而言，这类工作几乎不存在。相比花在工程上的数人月，可能只有几人周花在那上面。而且我认为，也许在机器学习方面押对了一些注，是的，而且这些注往往关乎不该做什么，而不是该做什么。

The work on the Dota 2 project, would you guys say was what people would really think of as like machine learning science versus engineering essentially? Like as far as day-to-day work goes, this kind of work was almost non-existent. Those like might be a few person-weeks spent on that compared to like person-months spent on engineering. And I think maybe placing some good bets was good bets on the machine learning side, yeah, and they're often about what not to do rather than what to do.

所以在项目一开始，我们就知道要解决一个游戏，一个困难的游戏。我们不确定具体选哪个，因为这些都是测试算法极限的好平台。游戏的一个好处是——要说明的是，你们两位是关键人物，整个团队大约十个人，密度大约十个人。这些是算法的好测试床，看看极限在哪里，真正推动可能性的边界。而且可以肯定的是，当你完成时，你知道你完成了，这是非常二元和可测试的。所以我们选择游戏的方式是上 Twitch，浏览全球最受欢迎的游戏列表。从第一名开始，你知道，是《英雄联盟》。但《英雄联盟》的问题是不在 Linux 上运行，也没有游戏 API，这些小事情实际上以奇怪的方式成为 AI 进步的最大障碍。所以往下看列表，Dota 实际上是第一个具备所有正确属性的：在 Linux 上运行，有庞大的回放解析社区，有内置的 Lua API。实际上这个 API 是为构建模组而设计的，而不是为构建机器人，但我们想，也许可以用它来构建机器人。Valve 这家公司的一个优点是，他们非常喜欢开放可破解的游戏，让人们去做各种自定义的东西。所以从哲学上讲，这非常适合合作。我们实际上在 11 月就做了初步选择，但当时还在做其他项目，所以直到 12 月底才真正开始。有趣的是，完全巧合的是，12 月中旬 Valve 发布了一个新的专注于机器人的 API，他们说，嘿，我们的机器人出了名的差，也许社区能解决这个问题，所以我们会专门为此构建一个 API。这真是宇宙的巧合，结果非常好。我们并没有与这个 API 的开发者密切接触，但整个过程中，到了项目最后，你还能怎么办呢？

So at the very beginning of the project, we knew we wanted to solve a game, a hard game. We didn't know exactly which one we wanted to do, because these are great test beds for pushing the limits of our algorithms. And one of the great things about games, to be clear, you guys are two of the key people, the entire team was like ten people, the density about ten people. And these are good test beds for algorithms, see what the limits are, to really push the limit of what's possible. And for sure that when you've done it, you know that you've done it, it's very binary and testable. So actually the way that we selected the game was we went on Twitch and just looked down the list of most popular games in the world. Starting, you know, number one is League of Legends. The thing about League of Legends is it doesn't run on Linux and it doesn't have a game API, and little things like that actually are the biggest barrier to making AI progress in odd ways. And so looking down the list, Dota actually was the first one that kind of had all the right properties: runs on Linux, it has a big community around replay parsing, there's a built-in Lua API. Actually this API was meant for scripting for building mods rather than for building bots, and we were like, but we could probably use it to build bots. And one of the great things about Valve as a company is that they're very into having these open hackable games where people can go and do a bunch of custom things. So kind of philosophically, it was very much the right kind of company to be working with. So we actually did this initial selection back in November, and we were working on some other projects at the time, so it didn't really get started until late in December. And one of the funny things is that by total coincidence, in mid-December Valve released a new bot-focused API, and they were saying, hey, our bots are famously bad, maybe the community can solve this problem, so we'll actually build an API specific for it for people to do this. And that was just one of those coincidences of the universe that just worked out extremely well. We weren't in close contact with the developer of this API, but kind of all throughout, at the very end of the project, well, what are you gonna do?

第一件事是我们必须非常熟悉这个游戏 API，确保理解所有细微语义和所有不同的边界情况，还要确保能大规模运行并把它变成一个舒适的开发环境。所以当时只有我们两个人。一个人在用机器人 API 构建脚本机器人。基本上就是：学习所有游戏规则，深入思考其运作方式。写这个的 ROFL 一生只玩过三四局 Dota，但他看过超过一千小时的 Dota 游戏视频，现在写出了世界上最好的 Dota 脚本机器人。这涉及大量用 Lua 编写代码，非常熟悉所有细节。与此同时，我在研究如何把它变成 Docker 容器。他们有一套完整的构建流程。结果发现 Steam 一次只能离线两周，而且他们不断推送新补丁。你需要从手动下载游戏变成自动化的可重复流程。游戏完整文件大约 17 GB，而我们的 Docker 注册表只支持 5 GB 层。所以我不得不写一个工具，把文件分成 5 GB 的 tar 包，放到 S3 上再取回来。所以很多工作就是弄清楚正确的工作流程和正确的抽象。

The first thing was we had to become very familiar with this game API to make sure we understood all the little semantics and all the different corner cases, and also to make sure that we could run this thing at large scale and turn it into a pleasant development environment. So at the time it was just two of us. One person was working with the bot API building a scripted bot. So basically this is: learn all the game rules, think really hard about how it works. This particular person who wrote it, ROFL, has played about three or four games of Dota in his life, but he's watched over a thousand hours of Dota gameplay and has now written the best Dota scripted bot in the world. And that involved a lot of just writing this thing in Lua, getting very intimately familiar with all those details. In the meanwhile, what I was working on was trying to figure out how to turn this thing into a Docker container. So they had this whole build process. Turns out that Steam can only be in offline mode for two weeks at a time, they push new patches all the time. You needed to go from this manually downloading the game to actually an automated repeatable process. It turns out that the full game files are about 17 gigabytes, and our Docker registry can only support 5 gigabyte layers. So I had to write a thing to chunk things up into 5 gigabyte tarballs and put those in S3 and set them back down. So a bunch of things there was really just about figuring out what the right workflow is, what the right abstractions are.

下一步是，我们知道要用 TensorFlow 和 Python 写机器人。怎么做到？为什么？因为机器学习……这其实很有意思：进步的最高阶位就是拥有游戏 API。还有就是：你能用熟悉且易于迭代的工具吗？在现代机器学习框架出现之前，除了个别情况，代码都在 MATLAB 里，如果你有新想法，要花两个月才能实现。祝你好运。所以关键在于迭代速度。如果能进入 Python 世界，我们有这些积累的高质量算法代码库，周围有那么多酷东西，那是最优体验。

And then the next step was, well, we know we want to be writing our bots in TensorFlow and Python. How do you get that? Why was that? Well, because machine learning... You know, that is actually quite interesting: a lot of the highest order bit on progress is just having the game API. It's also: can you use tools that are familiar and easy to iterate with? Before the world of modern machine learning frameworks, except for right there, their code in MATLAB, if you had a new idea it would take you two months to do it. Good luck making progress. So it was really all about iteration speed. And so if you can get into the Python world, well, we have these large code bases that we built up of high quality algorithms, there's just so much cool stuff built around it that that's like the optimal experience.

下一步是把脚本机器人移植到 Python。我的做法是直接把所有 Lua 文件重命名为 .py，注释掉代码，然后一个函数一个函数地取消注释。运行函数，遇到异常，就去取消注释它依赖的代码。尽可能机械地，我试图像一个人工转译器。Lua 是一索引，Python 是零索引，你得处理这个。Lua 不区分数组类型和字典类型，所以你得简单地对这两者做质量保证。但大部分工作完全可以机械完成。这很好，因为我不用理解任何游戏逻辑，不用理解底层发生了什么。我基本上就是直接移植，然后它就拼凑起来了。但最后你会有一小部分函数没有实现，那就是实际的 API 调用。所以我最终得到一个文件，里面有一堆虚拟调用，我确切知道需要哪些调用。然后我在 gRPC 之上实现了一个基于 protobuf 的协议，每个 tick 游戏会转储完整的游戏状态，通过网络发送，在 Python 中重新组装成内存中的状态对象，然后所有这些 API 方法都在 Python 中实现。最后，这听起来有点像弗兰肯斯坦的过程，但实际上效果很好。

So the next step was to port the scripted bot into Python. The way I did that was I literally just renamed all of the Lua files to .py, commented out the code, and then started uncommenting function by function. Then you run the function, you get an exception, you then go and uncomment whatever code it depends on. As mechanically as possible, I tried to be like a human transpiler. You know, Lua has one-index, Python zero-index, you have to do that. Lua doesn't distinguish between an array type and a dictionary type, so you kind of have to simply QA those two. But for the most part, it is something that could have been totally mechanically done. And it's great because I didn't have to understand any of the game logic, I didn't have to understand anything that was going on under the hood. I could basically just pour it over and it just kind of came together. But then you end up with a small set of functions that you do not have implementations of, which are the actual API calls. So I ended up with a file with a bunch of dummy calls, and I knew exactly which calls I needed. Then I implemented on top of gRPC a protobuf-based protocol where on every tick the game would dump the full game state, send it over the wire, reassemble that into an in-memory state object in Python, and then all of these API methods would be implemented in Python. So at the end of this, it sounds like a bit of a Frankenstein process, but it actually worked really well.

那看起来就像一个典型的 Gym 环境。你只需要说 'Jim，把这个做成 Dota 环境 ID'，然后你就能玩 Dota 了。你的 Python 代码只需要调用某个实现了胶水 API 的对象，然后这些角色就会在屏幕上按你的意愿行动。所以我当时主要在做这类纯工程方面的工作。实际上，我和 Shamone、Cup、J 以及其他人在这个项目上合作。大多数人都是在这个 API 之上进行开发，不需要深入了解底层的实现细节。所以，我要讲讲我个人在这个项目中的一个机器学习贡献。因为我的背景主要是创业公司的工程，构建大型基础设施，而不是机器学习。我绝对不是机器学习博士，我甚至没读完大学。所以，当我把基础设施做到一个相对稳定的状态后，我觉得，好吧，我不需要一直救火了。我有时间真正专注于深入研究机器学习了。

Something that looked just like a typical opening I'd gym environment. And so all you have to do is you say 'Jim, make this Dota environment ID' and suddenly you're playing Dota. And your Python code just has to call into some object that implements the glue API, and suddenly these characters are running around the screen doing what you want. So this was a lot of the kind of thing that I was working on, the pure engineering side. And actually, you know, I went on with Shamone, Cup, J, and others on the project. Most people were building on top of this API and really didn't have to dig into any of the underlying implementation details. So personally, my one machine learning contribution to the project I'll tell you about, because my background is primarily startup engineering, building large infrastructure, not machine learning. Definitely not a machine learning PhD; I didn't even finish college. So I kind of reached a point where I got the infrastructure into a pretty stable point. I felt like, all right, I don't have to be fighting fires very constantly. I have some time to actually focus on digging into the machine learning.

我们特别感兴趣的一个部分是行为克隆。我们构建的系统之一就是每天下载所有发布的回放。这个游戏的运作方式是，大约有 150 万场回放可供公开下载，但 Valve 会在两周后清除它们。所以你必须有一个发现过程，把它们存到 S3 的某个地方。最初我们每天下载所有回放，但发现每天大约有 2TB 的数据，累积得很快。所以我们最终只筛选出最顶尖玩家的回放。但我们想真正利用这些数据，解析它们，并用它们来克隆机器人的行为。所以我花了很多时间在这个完整的流程上：下载回放、解析、迭代，然后训练模型，尝试预测行为。

One particular piece that we were interested in doing was behavioral cloning. So we had one of the systems that we had built was to go and download all of the replays that are published each day. The way this game works is that there are about 1.5 million replays that are available for public download, but Valve clears them out after two weeks. So you have to have some discovery process, you have to stick them in S3 somewhere. Originally we were downloading all of them every day and realized that was about two terabytes worth of data a day, which adds up quite quickly. So we ended up filtering down to the most expert players. But we wanted to actually take this data, parse it, and use it to clone the behavior for a bot. So I spent a lot of time with this whole pipeline: download the replays, parse them, iterate on that, then train a model and try to predict what the behavior would be.

首先，我觉得非常有趣的一点是，做机器学习时你会遇到一种不同的工作流程。当软件工程师加入 OpenAI 时，有很多事情让他们感到惊讶。例如，如果你看一个典型的研究工作流程，你会看到很多文件名为 'experiment1'、'experiment2'、'experiment3'、'experiment4'，你一看，它们只是同一件事的轻微分支。你会想，'这就是版本控制的用途吗？' 做完这个克隆项目后，我确切地知道了为什么。因为问题是，如果你有一个新想法——好吧，我已经让这个东西工作了，现在我要尝试一些稍微不同的东西——当你做新东西时，机器学习是非常二元的。一开始它根本不工作，你不知道为什么；或者它有点工作，但有一些奇怪的表现，你不确定：是 bug 吗？还是这个数据集就是这样？你就是不知道。所以如果你已经让它工作了，然后你做了一个改动，你总是想回去和之前运行的东西比较。所以你实际上希望新东西和旧东西同时运行。如果你不断地暂存和切换分支，那你只会很痛苦。有很多这样的工作流程问题。你只能撞墙，然后你才恍然大悟。

You know, first it's just like one thing I find very interesting is the different workflow that you end up with when doing machine learning. There are a bunch of things where when software engineers join OpenAI that are just very surprising. For example, if you look at a typical research workflow, you'll see a lot of files named like 'experiment1', 'experiment2', 'experiment3', 'experiment4', and you look at them and they're just slight forks of the same thing. And you're like, 'Is this what version control is for?' After doing this cloning project, I learned exactly why. Because the thing is, if you have a new idea — okay, I've kind of got this thing working and now I'm going to try something slightly different — as you're doing the new thing, machine learning is something very binary. At the start it just doesn't work at all and you don't know why, or it kind of works but it has some weird performance and you're not sure exactly: is it a bug? Is it just how this dataset works? You just don't know. So if you've gotten it working at all and then you make a change, you're always going to want to go back and compare to the previous thing you've had running. So you actually do want the new thing running side by side with the old thing. And if you're constantly stashing and checking out, then you're just going to be sad. There are a lot of workflow issues like that. You just got to bang your head against the wall and then you see, 'Oh, I've been enlightened.'

在我们继续故事之前，你能解释一下在游戏中训练机器人的基础知识吗？比如你实际上是如何给它反馈让它学习的？

Before we progress further on the story, can you just explain the basics of training a bot in a game? Like how are you actually giving it the feedback so that it learns?

这并不高深，尽管强化学习听起来很 fancy。本质上，我们有一个机器人，它观察环境中的一些状态，并根据这些状态执行一些动作。根据它执行的动作，它继续游戏，最终要么表现好要么表现差。这是我们可以用一个数字来量化的，这更多是一个工程问题而不是研究问题：如何量化机器人的表现好坏。你需要想出一个指标。然后机器人会得到关于它表现好坏的反馈，并尝试选择能产生高正向反馈、高奖励的动作。

It's no rocket science, even though reinforcement learning sounds fancy. Essentially, what's happening is we have a bot which observes some state in the environment and performs some actions based on that state. Based on those actions that it executes, it continues playing and eventually either does well or poorly. That's something that we can quantify in a number, and that's more of an engineering problem than a research problem: how to quantify how good the bot is doing. You need to come up with a metric. Then the bot gets feedback on whether it is doing good or not, and tries to select the actions that yield high positive feedback, high reward.

为了让我们了解它的效果：机器人通过与自己比赛来变得更好。一旦一切就绪，第 N 天的机器人比第 N-1 天的机器人强多少？

And to give us a sense for how well that works: the bot plays against itself to get better. Once you had everything working, how good would the bot from day N be against a bot from day N-1?

我想有一个故事可以说明这些技术能达到什么效果。当我们开始这个项目时，我们的目标并不是真正做研究。我的意思是，在某种程度上是，但我们非常目标导向。我们只想解决问题。我们想打败游戏内置的机器人。一开始，Sanders 每天都在编写 Rafa，Rafa 是一个极其脚本化的机器人。所以他只是写逻辑：'我认为机器人应该这样做。' 他会说，'这是一个小兵，这是一个...' 我以为我能做到。他花了大约三个月的时间。而 Rafa 实际上是一个非常优秀的工程师，所以我们有一个非常好的脚本机器人。然后发生了什么？他到了无法再改进的地步。所以我们先尝试了别的东西。我当时实际上在度假，但另一位工程师 Coupe，在我的整个假期中——我觉得这非常令人惊讶——我相信没什么。我回来时，这已经是强化学习机器人了，而且它实际上在仅仅一周的工程努力后，可能两周，就打败了我们的脚本机器人。但与脚本机器人的开发相比，这只是非常小的工作量。所以实际上，我们的机器人没有对游戏做任何假设，却足够好地理解了游戏结构，打败了我们手工制作的任何东西，这真是太神奇了。

I guess we have a story that kind of illustrates what to expect from those techniques. So when we started this project, our goal wasn't to really do research. I mean, at some high level it was, but we were very goal-oriented. All we wanted to do was solve the problem. We wanted to beat the game's built-in bots. The way it started was, every day Sanders was just coding Rafa, and Rafa was an incredibly scripted bot. So he would just write the logic: 'I think this is what the bot should do.' He'd say, 'He's a creep, he's a...' I thought I could do that. And he spent like three months of his time. And Rafa is actually a really good engineer, so we had a really good scripted bot. So what happened then? He got to the point where he couldn't improve it much more. So we tried something else first. I was actually on vacation at the time, but there was another engineer, Coupe, who throughout my vacation — which I found super surprising — I believe there is nothing. I come back and this is the reinforcement learning bot, and actually it's beating our scripted bot after like a week worth of engineering effort, possibly two weeks. But it's just something very miniature compared to the development of the scripted bot. So actually our bot, which didn't have any assumptions baked about the game, figured out the game structure well enough to beat anything that we could craft by hand, which was pretty amazing to see.

那你是在什么时候决定参加比赛的？

And at what point do you decide to compete in the tournament?

哦，好吧，也许我应该把我的故事讲完。抱歉，有点长了，但没关系。我很快就会讲完。所以，就结束我的机器学习贡献吧。我基本上花了一个月真正学习工作流程，得到了一些有生命迹象的东西，比如跑到中间，你知道它在做什么。它太好了。而且很明显，当你只是做克隆时，这些……

Oh, well, so I maybe I should finish up my story. Sorry, it was running a bit long, but it's good. I will get good shortly. So just finish up my machine learning contribution. I basically spent about a month really learning the workflow, got something that showed signs of life, like run to the middle and you know what it's doing. It's so good. And it's very clear when you're just doing cloning that these...

算法学会模仿所见而非实际意图，所以它们会感到困惑，试图做某种卡兵，但小兵不在那里。我把它做到了能可靠卡兵的程度，然后交给 Jay，他用强化学习进行微调。突然之间，它从只理解动作而非意图，变成了真正知道自己在做什么，并且拥有了大家见过的最好的卡兵。那是我在这个项目中唯一的机器学习贡献。

Algorithms learn to imitate what they see rather than the actual intent, so they get kind of confused and try to do some sort of creep blocking, but the creeps wouldn't be around. I got it to the point where it was creep blocking pretty reliably, and then I turned it over to Jay, who used reinforcement learning to fine-tune that. Suddenly it went from only understanding the actions rather than the intent to really knowing what it was doing, and it had the best creep block anyone has seen. That was my one machine learning contribution in the project.

项目最重要的部分之一是有一个记分板。我们在墙上放了一个指标，即衡量机器人胜率的真实技能评分。我们把它贴在墙上，每周人们尝试各种想法。有些有效，有些提升了性能，最终我们得到了一条非常平滑、几乎线性的曲线。我们把它发在博客里，这实际上意味着机器人实力随时间呈指数级增长。部分原因是有时我们只是把同一个实验训练更长时间——通常我们的实验持续长达两周——但也有很多次是我们有了新想法，尝试了别的东西，做了调整，添加了功能，或者移除了不必要的组件。

One of the most important parts of the project was having a scoreboard. We had a metric on the wall, which was the true skill rating that measures the win rate of your bots versus others. We put that on the wall, and each week people tried all the ideas. Some worked, some improved performance, and we ended up with a very smooth, almost linear curve. We posted it in a blog post, and that really meant an exponential increase in the strength of the bot over time. Part of that was sometimes we just trained the same experiment for longer—typically our experiments lasted up to two weeks—but also a lot of those were when we had a new idea, tried something else, made a tweak, added a feature, or removed a component that wasn't necessary.

我们选择了 1v1 的目标，我不记得确切时间，但一定是春天或初夏。但我们真的不知道能否实现。不像构建工程系统时，你仔细思考所有组件，分解成子系统，并通过组件完成百分比衡量进度；这里你需要尝试想法，这在某种程度上是不可预测的。项目最重要的变化之一是管理方式。最初，我们每周或每两周写下里程碑，比如‘在这个日期前打败这个人’。这些里程碑来了又去，你却没有实现。这完全不自然。相反，我们转向了‘下周我们要尝试的所有事情’。这是个好见解。你这样做，如果你没有完成所有你说过的事，你应该感到难过并做得更多。如果你做了所有事但没成功，那也公平，但你实现了你想实现的目标。

We chose the goal of 1v1, I don't recall exactly when, but it must have been in the spring or maybe early summer. But we really didn't know if we were going to be able to make it. Unlike when you're building an engineering system, where you think hard about all the components, decompose it into subsystems, and measure progress by the percentage of components built, here you have ideas that you need to try out, and it's sort of unpredictable. One of the most important changes to the project was the way management was happening. Initially, we wrote down milestones like 'beat this person by this date' on a weekly or bi-weekly basis. Those milestones would come and go, and you wouldn't achieve them. It's completely unnatural. Instead, we shifted to 'what are all the things we're going to try by next week.' That's a good insight. You do that, and if you didn't do everything you said, you should feel bad and do more. If you did all those and it didn't work, then fair enough, but you achieved what you wanted to achieve.

即使到了国际邀请赛，前两周是我们的截止点。那时我们能做的已经不多了。我们要做有史以来最大的实验，把所有算力放在一个篮子里，看看结果如何。两周前，机器人有多好？很差。有时我们需要我们测试的职业选手，但也不总是。具体来说，7 月 8 日是我们第一次战胜半职业测试者。然后我们变得稍微稳定一些，但他去度假了，所以他在某个地方用一台不太好的笔记本电脑。我们一直打败他，但那不是非常可靠的数据。国际邀请赛前一周，我们真的不知道我们变得多好。我们知道真实评分在上升。

Even going into The International, two weeks before was our cutoff. At that point, there wasn't much more we could do. We were going to do our biggest experiment ever, put all of our compute into one basket, and see where it goes. Two weeks out, how good was the bot? It was bad. Sometimes we needed professionals that we had testing, but not always. To be specific, July 8th was when we had our first win against our semi-pro tester. Then we were kind of more consistent with it, but he went on vacation, so he was on some laptop somewhere that was not very good. We were consistently beating him, but that was not very reliable data. The week before The International, we didn't really know how good we were getting. We knew the true skill was going up.

上一次 OpenAI 员工打败机器人是什么时候？我觉得我们不太擅长 Dota。大约一两个月前，它能打败所有 OpenAI 的人。两周前，它有一次能打败半职业选手。所以四周前是它第一次打败半职业选手。两周前，我们仍然无法真正知道。我们当时不知道它有多好。我们只知道我们能偶尔打败半职业选手。进入国际邀请赛时，我们觉得有五五开的几率。我想我们一直告诉 Sam，这些事情的概率，你永远不能真正相信概率，你只能相信概率的趋势。即使那个趋势也在剧烈波动。你们每晚都给我发短信：‘我们要输了’，‘没机会’，‘我们肯定能赢每一局’。所以很明显，我们自己对将要发生的事情的估计是不准确的。

When was the last time an OpenAI employee beat the bot? I think we weren't very good at Dota. About a month or two out, it could beat all the OpenAI people. Two weeks out, it could at one time beat a semi-pro. So four weeks was the first time it beat the semi-pro. Two weeks out, we still couldn't really find out. We didn't know how good it was at that time. We just knew we could beat our semi-pro occasionally. Going into The International, we figured there was a 50/50 shot. I think we were telling Sam the whole way that the probability with these things, you never really trust the probabilities, you just trust the trend of the probabilities. Even that was swinging wildly. You guys texted me every night: 'We're going to lose,' 'No chance,' 'We're definitely going to win every game.' So it was very clear our own estimates of what was going to happen were miscalibrated.

在整个 TI 周，我们仍然不知道。大部分团队去了西雅图。我们躲在赛事场馆附近。我们在 Key Arena 的地下室得到了一个更衣室。我们都有制作人徽章，所以可以跳过排队进入后台区域。那实际上是一个改造成拍摄区的更衣室。我们都在那里放了自己的笔记本电脑。他们还会时不时带职业选手进来。我们有一个完整的拍摄设备。我们和职业选手对战。我们设置了一个隔断，只是一块黑布，把团队和选手隔开，团队坐在那里想着能不能打败这个职业选手，尽量保持安静。周一，他们带来了三个或两个职业选手和一个非常高排名的分析师。我们打了第一局，真的不知道会发生什么。我们以 3-0 打败了这个人。这对 OpenAI 的每个人来说都非常激动。

Throughout the week of TI, we still didn't know. Most of the team went to Seattle. We were holed up near the stadium where the event was happening. We were given a locker room in the basement of Key Arena. We all had production badges, so we could skip the line and go to the backstage area. It was literally a locker room converted into a filming area. We all had our laptops in there. They would also bring in pro players every so often. We had a whole filming set up. We played against the pros. We had a partition set up, just a black cloth, between the team sitting there wondering if we could beat this pro, trying to be as quiet as possible. On Monday, they brought three or two pros and one very high-ranked analyst. We had our first game, and we really didn't know what was going to happen. We beat this person 3-0. This was a very exciting thing for everyone at OpenAI.

你还记得那场比赛的细节吗？差距有多大？

You remember the details of that one? What was the margin?

我们有所有数据，但 Valve 请来了第二位职业选手 Pyke，他和我们的机器人对战。我们赢了他一次、两次，然后他赢了。我们一看比赛就知道发生了什么。他积累了大量魔棒充能。这个物品会积累充能，他积累的充能比我们的机器人见过的都多。结果发现我们的设置有个小漏洞。基本上，你的机器人没准备好应对某个阈值。

I mean we have all the data, but Valve brought in a second pro named Pyke, and he played the bot. We beat him once, we beat him twice, and then he beat us. Looking at the game, we knew exactly what had happened. Essentially, he accumulated a bunch of wand charges. This item accumulates charges, and he accumulated more charges than our bot had ever seen before. It turns out there was a small bug in our setup. Basically, some threshold that your bot was not ready for.

所以基本上是某个你的机器人没准备好的阈值？

So basically some threshold that your bot was not ready for?

具体来说，根本原因是他选择了早期魔棒出装，而我们之前从未做过这种出装。所以我们的机器人从未见过这种特定出装，也没有机会探索它的意义。它从未学会积攒和使用魔棒充能。它很擅长计算谁会赢，但没认出那个维度。然后我们和另一位职业选手打了第三场比赛，3-0 赢了。

I'd say very specifically the root cause was that he had gone for an early wand build, and we had just never done that item build before. So our bot had never seen this particular item build, and it never had a chance to explore what it means. It had never learned to save up stick charges and use them. It's very good at calculating who will win a fight, but it didn't recognize that axis. So then we played a third match against another pro and went 3-0.

职业选手的反应很有意思。我们得到了混合反应。有人说这是最酷的东西，有人说这很蠢永远不会用。但那天晚上职业选手离开后，他们花了四个小时讨论机器人和它的意义。情绪很激动。他们以前从未被电脑打败过，所以难以置信。有个选手最终赢了机器人，但一开始说它没用，后来冷静下来，说这对练习很有帮助。

It's interesting getting the pros' reactions. We got a mix. Some said it's the coolest thing ever, one said it's stupid and would never use it. But after the pros left that night, they spent four hours talking about the bots and what it meant. The flavors were highly emotional. They were never beaten by the computer before, so it's unbelievable. One player eventually beat the bot but initially said it was useless, then calmed down and said it's great for practice.

在你的机器人第一次输掉后，他们开始讨论用反直觉的策略来打败它了吗？

After your bot lost that first time, did they start talking about counterintuitive strategies to beat it?

我不认为职业选手对此感兴趣。他们主要感兴趣的是如何让自己变得更强。但活动结束后，我们组织了一场大型局域网派对，有 50 台电脑运行机器人。我们让人类对抗我们的机器人，他们找到了所有漏洞。我们预料到了，因为机器人只能从环境中学习。有些东西它从未见过，那些就是可被利用的。我们对下一步很兴奋，就是 5v5，因为 5v5 本身就是一个巨大的漏洞。它关乎利用对方团队，出现在他们意想不到的地方。所以我们知道必须正面解决这些问题。

I don't think pro players are interested in that. They are mostly interested in how it lets them get better at the game. But there was a point after the event where we set up a big LAN party with 50 computers running the bot. We unleashed this form of humans against our bot, and they found all the exploits. We expected them because the bot can only learn as well as the environment allows. There are some things it just never sees, and those will be exploitable. We are excited about our next step, which is 5v5, because 5v5 is one giant exploit. It's about exploiting the other team, being where they don't expect you. So we know we will have to solve those problems head-on for 5v5.

训练过程中有趣的一点是，我们的很多工作就是找出漏洞，然后做个小调整来修复。我现在认为机器学习系统是一种让人类程序员杠杆效应大幅提升的方式。通常，构建系统时，你构建组件一、二、三，组件四的边际收益和组件一差不多。但在这里，很多早期工作只是从糟糕变成不那么糟糕。一旦我们在国际邀请赛上输给了 Pyke，我们知道根本原因是它从未见过那种出装。我们只需要做个调整，把它加入我们的出装列表，然后它就会在接下来的时间里模拟这个场景。

One thing interesting about the training process is that a lot of our job was seeing what the exploits were and then making a small tweak to fix them. The way I now think about machine learning systems is that they are a way to make the leverage of human programmers go way up. Normally, when building a system, you build components one, two, three, and the marginal return on component four is similar to component one. But here, a lot of the early stuff went from crappy to slightly less crappy. Once we were at the International and had this loss to Pyke, we knew the root cause was it had never seen that item build. All we had to do was make a tweak to add that to our list of item builds, and then it played out this scenario for the next however long.

你能从技术角度解释一下这个调整是如何工作的吗？我的印象是你们已经运行了数百万场比赛，所以它已经学会了所有东西。有些人说这些网络很灰暗，不知道如何操作。你们是怎么进去改变东西的？

Can you walk me through how that tweak works technically? My impression is that you've been running a million games, so it has learned all this stuff. Some people talk about these networks being very gray and not knowing how to manipulate them. How are you getting in there and changing things?

从高层次看，你可以把这比作教一个人。你看到小孩做数学，把加法和减法搞混了。你看着说，'这个符号是你没看清的。' 同样，我们的机器人从未见过那种魔棒出装。我们只需要说，当机器人玩游戏选择物品时，我们只需增加采样那种它从未见过的特定出装的概率。当它和用那种出装的对手玩几局，或者自己用几次，它就会更熟悉那种出装带来的游戏后果。从技术层面看，这些模型是一个黑箱，输入一串数字，输出一串数字。它在映射上很聪明，但这就是你得到的。然后你把这个当作原语，在上面构建，这样学习内部需要做的工作就尽可能少。我们注意到我们忘记了一件事，就是传入数据...

On a high level, you can compare this to teaching a human. You see a kid doing math and confusing addition with subtraction. You look and say, 'This symbol is what you're not seeing clearly.' Similarly, our bot had never seen that wand build. All we had to do was say that when the bot plays games and chooses items, we just need to add some probability of sampling that specific build it has never seen. When it plays a couple of games against opponents that use that build, or uses it itself, it becomes more comfortable with the in-game consequences of that build. On a technical level, these models are a black box that takes in a list of numbers and outputs a list of numbers. It's very smart at that mapping, but that's what you get. Then you think of this as your primitive, and you build on top of it so that as little work as possible has to be done inside the learning. One thing we noticed we had forgotten was passing in data that...

这对应着我传递的传送可见性。作为人类，你能看到有人传送出去。我们的机器人没有这个特征。传入的数字列表没有那个特征。所以你需要做的就是添加它。特征向量原本有多长，现在多了一个特征。机器人没有把它识别为屏幕上的东西。它看不到屏幕，它接收的是机器人 API 传来的数据。所以它确实只得到我们给它的数据。因此，特征工程很大程度上取决于我们。你要尽可能让事情变得简单，这样它内部需要做的工作就尽可能少。这样它就能把精力——你可以把它看作固定容量——花在策略学习上，还是花在映射、选择攻击哪个小兵上？你想把它花在解析像素上吗？归根结底，我认为我们作为系统设计师的很多工作，就是把尽可能多的模型容量和学习能力推向问题中有趣的部分，那些你无法编写脚本、无法进行任何处理的部分。所以这是一个层面：很多工作最终是识别哪些特征缺失，以合适的方式设计观察和动作空间。

Corresponds to the person I was passing in the visibility of a teleport. As a human, you can see when someone's teleporting out. Our bots just did not have that feature. That list of numbers passed in did not have that feature. So one of the things you need to do is just add it. That kind of goes from your feature vector, however long that was, and now it's got one more feature on. And the bot wasn't recognizing that as an on-screen thing. It doesn't see the screen; it's passed data from the bot API. So it really is given whatever data we give it. So it's kind of on us to do some of this feature engineering. You want to do as much as you can to make it as easy as possible so that it has to do as little work inside as possible. So it can spend — you think of it as you get some fixed capacity. Do you want to spend that on learning the strategy? Do you want to spend it on learning how to map, choose which creep you want to hit? You want to spend that on trying to parse pixels? At the end of the day, I think a lot of our job as the system designers here is to push as much of that model capacity, as much of the learning, towards the interesting parts of the problem that you can't script, that you can't possibly do any processing for. So that's one level: a lot of the work ends up being identifying which features aren't there, kind of engineering the observation and action spaces in an appropriate way.

另一件事是另一个层面。实际情况是：周一我们在那里，大家吃了晚饭，然后 Ramon、Cup、Raul 和我——可能还有一两个人——熬夜对我们的运行实验进行了手术。这非常像一次生产故障，所有人都全力以赴去改进。

Another thing is another level. The way this actually happened was: we were there on Monday, people got dinner, and then Ramon, Cup, Raul, and I — and maybe one or two others — stayed up all night to do surgery on our running experiment. So it was very much like a production outage, everyone all hands on deck trying to go and make improvements.

具体来说，深入一点，让你感受一下在这个领域工作的感觉：那是非常疲惫的一周。每天都是与职业选手会面，兴奋起来，然后编写下一版实验代码。因为实际上，实验的日常迭代不足以击败下一个选手。第二天面对职业选手——就在那天早上下载网络的新参数——就能击败他，但前一天还不行。你是怎么判断的？这几乎又是巧合。可能还有更深层的原因，但这一周的全貌是：我们周一进行了比赛，输了。澄清一下，你们是在比赛中还是不在？我们做的是一个特殊活动，与 Dendi 对战，他是有史以来最顶尖的选手之一。在那里，我们也测试了其他职业选手。明白了。现在就在这里。好吧，看看我们表现如何。明白了。所以周一发生了，你开始训练它。这个实验可能是在前一周或几周前启动的。我们已经运行这个实验一段时间了，我们的基础设施本意是从头开始运行实验——从完全随机开始，运行，两周后看看效果。但我们没有两周时间了。所以我们不得不进行手术，非常仔细地检查提交的每一个字符，确保没有 bug，因为一旦搞砸，就没时间了。你无能为力。这不是那种你稍微聪明一点就能热修复并搞定一切的情况。实际上，你必须让这个东西放着，让它慢慢成熟。所以周一过去了。我们运行着这个经过手术的实验。第二天我们稍微喘息了一下，只和一些排名较低的选手对战——解说员和社区名人——但没有挑战机器人的极限。周三下午 1 点，Valve 的联系人过来说：“嘿，我要给你们安排 RTZ 和 Sumail，”他们基本上是世界上最顶尖的选手。我说：“能不能推到周四？”他说：“他们的日程已经排满了。你们只能按安排来。”我们被安排在下午 4 点与他们比赛。

Specifically, to zoom in and give you a feel for what this felt like working on the world: it was a very tiring week. Every day was just meeting with the pros, getting excited, and then coding up the next version of the experiment. Because actually, day-to-day iteration of the experiment was not good enough to beat the next player. The next day's professional — just that morning to download the new parameters of the network — and it would be good enough to beat it, but the day before it wasn't. How were you discerning that? Again, something of almost a coincidence. There might be something a little bit deeper, but the full story of the week was: we did the Monday play, and we lost. Just to clarify, are you guys in the competition or not? The thing we did was a special event to play against Dendi, who's one of the best players of all time. While we were there, we also tested against all these other pros. Gotcha. Physically here right now. Okay, let's see how we do. Got it. So Monday happens, you start training it. This experiment we kicked off maybe the prior week, weeks before, something like that. We've been running this experiment for a while, and our infrastructure is really meant for running an experiment from scratch — start from complete randomness, run it, and two weeks later go see how it does. We didn't have two weeks anymore. So we had to do this surgery and very carefully read every single character of your commit to make sure you're not going to have any bugs, because if you mess it up, we're out of time. There's nothing you can do. It's not one of those things where if you're just a little bit more clever, you can go and do a hot patch and have everything be good. It's literally the case that you gotta let this thing sit there, it's got to bake. So Monday came and went. We were running this experiment that we performed surgery on. The next day we got a little bit of reprieve where we just played against some lower-ranked players — commentators and popular in the community — but not pushing the limit of our bots. On Wednesday at 1:00 PM, our contact from Valve came by and said, "Hey, I'm gonna get you RTZ and Sumail," who are basically the top players in the world. I was like, "Could we push them off to Thursday maybe?" And he was like, "Their schedules are booked. You're gonna get them when you get them." And we were scheduled to get them at 4:00 PM.

好的，我们检查了我们的机器人，看看它的表现如何，我们一直在沿途评估它。我们让我们的半职业选手测试它，他说：‘这个机器人完全坏了。’哦不。我们脑子里闪过各种念头，比如手术中是不是出了 bug。他给我们看了问题。他说：‘看，第一波这个点承受了大量不必要的伤害，这没有任何优势。我要冲过去杀了它，让你看看这有多容易。’他冲过去想干掉它，结果他输了。

Okay, so we looked at our bot to see how it was doing, and we kind of been along the way gauging it. We tested it against our semi-pro player, and he said, 'This broad bot is completely broken.' Oh no. And you know, kind of pictures of maybe we had a bug during the surgery went through our head. And he showed us the issue. He said, 'Look, first wave the spot takes a bunch of damage it doesn't have to take. There's no advantage to that. I'm gonna run it, I'm gonna go kill it. I'll show you how easy it is.' He ran it to kill it, and he lost.

好的，别跳步。解释一下发生了什么。

Okay, and don't jump ahead. Explain what happened.

他打了五次，每次都输了，直到他终于找到了利用它的方法。我们意识到这个机器人学会了一种诱饵策略：假装自己是个很笨的机器人，不知道自己在做什么，然后当对手冲过来杀你时，你突然转身变成超级机器人。这其实是个糟糕的策略，如果你真的很强的话。但我想它对阵它一直在对抗的整个机器人群体是有效的，而人类直到那天才见过这种策略。所以我们觉得这是其中一个主要例子，说明我们并没有明确激励的东西，但它们却都学会了。

So he played it five times, and he lost each time, until he finally did figure out how to exploit it. And we realized what was going on was that this bot had learned a strategy of baiting: pretend to be a really dumb bot, you don't know what you're doing, and then when the person comes in to kill you, you just turn around and go super bot. It was legitimately a bad strategy, you know, if you're really really good. But I guess it was good against the whole population of bots that it was playing against, and you had never seen it until that day. So we expected it was like one of the major examples of the things that we kind of didn't have explicitly incentive for, and yet they both actually learned them.

本质上，这挺搞笑的，因为当机器人和它自己的其他版本对战时，这只是一个好的战斗策略。但它对人类产生了非常有趣的心理影响，因为我们的策略是不上当，稍微等一等，因为机器人有劣势。但他却想：‘看这个最蠢的机器人，我要去杀了它。’所以它对人类产生了有趣的心理影响，我觉得它好像几乎知道自己是机器人，知道别人怎么攻击它。看到机器人似乎玩弄玩家的情绪，这很有趣。当然实际上并非如此，但看起来就是这样。

And essentially, I mean, this is just kind of funny because of course when the bot played against its other versions, it was just a good fighting strategy. But it had a very interesting psychological effect on humans because after you, our strategy was not to fall for the bait, it was kind of to wait it out a little bit because the bot has a disadvantage. But he's like, 'Okay, look at this stupidest bot, I'm gonna go for a kill.' So it kind of had an interesting psychological effect on humans, which I thought was like it kind of almost knows it's a bot, yeah, it knows how it's attacked. Yeah, you know, it's funny to see a bot which kind of seems like it's playing with emotions of the player. Of course it was not what actually happened, but it seemed this way.

现在我们面临两难：周三下午 1 点，这些顶尖选手下午 4 点就要来了，我们有一个坏掉的机器人。我们该怎么办？我们知道周一的机器人不够好，肯定不行。所以我们做的第一件事就是写：周一的机器人在第一波表现不错，这个新机器人之后是超级机器人。好吧，那我们能不能把两者拼接起来？我们已经有了一些做类似事情的代码，所以我们重新启用了它。然后在三个小时里，J 花时间做了一个非常仔细的拼接：先运行第一个机器人，然后在正确的时间切换到第二个机器人。这基本上就是机器人一先打一段时间，然后切换。他在比赛前 20 分钟完成了，就在决赛、半职业选手 Santa Cruz 之前。‘太好了，至少我们及时完成了。’但另一个问题是：我们到底怎么修复这个机器人？

So now we were faced with the dilemma: it's 1:00 p.m. on Wednesday, these best players are going to show up at 4 p.m., we have a broken bot. What are we gonna do? And we know that our Monday bot is not gonna be good enough, we know it's not going to cut it. And so the first thing we do is we write: well, Monday bot is pretty good at the first wave, this new bot is a super bot thereafter. Okay, so can we stitch the two together? So we already had some code for doing something similar, so we kind of revived that. And then in the three hours, J spent his time doing a very careful stitch where you run the first bot and then you cut over at the right time to the second bot. And this is literally just like bot one plays the first amount of time, and then literally just that. And he finished it 20 minutes before the pro, before the grand final, semi-pro Santa Cruz. Like, 'This is great, so at least we got that done in the nick of time.' But the other question was: how do we actually fix the bot?

实际上我刚刚完成了一个方面，因为我们也不确定从一个机器人切换到另一个时会发生什么。所以我站在正在玩它的职业选手旁边，看着切换的时刻。我说‘嗯’，当然这可能是完全没必要的，但我们不确定会发生什么。所以我不知道故事的那部分。

And I mean, I actually just finished like one aspect because we were also kind of uncertain what happens when you switch over from one bot to the other. So I was actually standing by the pro who was playing it, and I was looking at the time at the moment when it was switching. I was like, 'Yeah,' and of course it was probably completely unnecessary, but we weren't sure what would happen there. So I didn't know about that part of the story.

那么如何真正修复它的问题：有一点争论，也许我们应该放弃这个，切换回我们旧的实验，让它跑更长时间。我忘了是谁建议的，但有人说：‘我觉得我们只需要让它跑更长时间，因为机器人学会了诱饵策略。而反制策略就是不上当。’好吧，整个过程都是这样。所以我们让它额外跑了三个小时。然后我们首先让我们的 TZ 来玩我们的切换机器人，就是那种弗兰肯机器人，结果它打败了他三次。我们说：‘好吧，试试这个另一个机器人，看看额外训练三个小时的结果。’因为我们的半职业测试者至少验证了它看起来修好了。

So the question of how do you actually fix it: there was a little bit of debate of like maybe we should abandon ship on this, switch back to our old experiment, run that one for longer. And I forget who suggested it, but someone's like, 'I think we just have to let it run for longer, because you learn a strategy of baiting. Well, the counter strategy for that is just don't bait.' Okay, well the whole time. And so we got that run for the additional three hours. And so we first played our TZ who showed up on our switch bot, you know, kind of the Franken bot, and you know that beat him three times. And we're like, 'All right, let's try out this other bot and just see what happened with the additional three hours of training.' Because you know, our semi-pro tester at least validated that like it looks like it's fixed.

在那三个小时的训练中，它实际上同时玩了多少局游戏？

And in that three hours of training, how many games is it actually playing simultaneously?

好问题。相当多，是的。

That's a good question. Quite a bit, yeah.

好的。然后我们让这个新机器人和我们的 TZ 对战。我不知道结果会怎样，果然它打败了他。他很喜欢，玩得很开心。那天他玩得很尽兴。我记得可能打了十局，但我觉得他只是觉得‘这一切太酷了’。我们那天本来也应该有 Sumail，但由于日程安排问题，他不得不参加一个小组讨论，所以时间没安排好。但 Arteezy 和他的教练（也是 Sumail 的教练）都说：‘是的，Sumail 会打败这个机器人，肯定会。也许第一局他会有点困难，但之后你就麻烦了。’所以我说：‘好吧，我们还有一天时间想办法。’

Okay. And so we played this new bot against our TZ. I didn't know how it was gonna do, and sure enough it beats him. And he loved it, he was having a lot of fun. He ended up getting loving games that day. And I remember maybe was ten, but I think that he's just like, 'All this is so cool.' We were supposed to have Sumail that day as well, but due to a scheduling snafu, he had to be at some panel, and so like time didn't work out. But Arteezy and his coach, who also coaches Sumail, both said, 'Yeah, Sumail's gonna beat this bot, like it's gonna happen. You know, maybe he'll have a little bit trouble to figure it out for the first game, but like after that, you're in trouble.' And so I'm like, 'All right, we've got one more day to figure out what to do.'

那我们怎么安排一顿好的晚餐？我们休息了一下，和家里的人放松了一下。然后早上我们下载了网络的新参数，让它继续玩。我们就待着，让它运行，让它玩。这和我习惯的工程截止日期完全相反。通常你会一直工作到最后一分钟。所以你们想要像你们那样，睡个整觉，轻松自在？

So what do we do for some nice dinner? We kind of rested, kind of like no slack with some people at home. And then in the morning we download the new parameters of the network and just let it play. So we hung out and just let it go, just let it play. It's the exact opposite of how I'm used to engineering deadlines happening. Yeah, normally it's your work right up until the minute. So you guys wanted like you guys were getting like full night's of sleep, nice and relaxed?

不，绝对不是。好吧，我不喜欢这样。前一天晚上，我们找到休息放松的那天前两个晚上，那晚的情况是这样的：我们一整天都在处理问题，情绪大起大落，累得要死。到了午夜，我们开始工作。好吧，我们需要做所有这些改动。比如我们午夜讨论的一件事，我们四个人开始工作，大家都累得不行，我们检查所有要添加到实验中的注释。实际上有两个人一起看，因为太累了，我们不相信一个人。他们就在看那些常数。早上 6 点，我在更新模型，这有很多讨厌的 off-by-one 索引问题，虽然代码很短，但花了六个小时。六个小时到凌晨 3 点左右，我们接了个电话，因为一些机器开始超过某些限制，我们试图让他们提高限制。然后到了早上 6 点。

No, absolutely not. Okay, so I don't like this. Clear the night before, the night we found that like two nights before the day where we got the rest relaxation, the night looked something like the following: we had a full day of dealing with the problems and kind of like emotional highs and lows, absolutely knackered. Come midnight, we start working. Okay, we need to make all those changes. Like the one thing that we talked about around midnight, we start with four people and we are all so tired that like you know, we look through all the comments that we are gonna add to the experiments. There are actually two people looking at them because we didn't trust a single person given how tired we are. So they're like looking at those constants. 6:00 a.m., as I was doing this like updating the model, which is a lot of nasty like off-by-one indexing things, even though it is a short call, it took me like six hours. Six hours to the somewhere around 3 a.m., we had like a phone call either because it turns out a certain number of machines start exceeding some limits, we try to make them raise the limits. And around 6 a.m.

我们准备好了，可以部署了。然后飞行部分只是一个人的工作。所以 Yak 就在那里思考，点击部署，并修复所有出现的问题。我待在那里只是为了确保合作不会出问题。最终在上午 11 点实验开始运行，我们去睡觉，下午 4 点左右醒来。它训练了超过 24 小时。我想直到比赛前总共训练了一天半。抱歉，重复一下时间线。所以周一我们打了第一组比赛，输了，当晚做了调整。周二开始比赛，然后周三下午 4 点我们和 RTZ 比赛，之后又训练了更长时间。之后我们没做任何改动，可能有一些小的。然后周四我们和 Sumail 比赛。所以我认为周二到周三那晚我们做的改动较少。有很多不同的工作同时进行，最终汇聚在一起。

We are okay, we are ready to deploy this. And then there was the flying, it's just a one-man job. So Yak was just thinking, clicking deploy, and fixing all the issues that came up. I was staying around just to make sure that the coop doesn't fall asleep. Eventually at 11 a.m. the experiment was running, and we went to sleep, woke up at 4 p.m. or something. It had over 24 hours to train. I think it ended up being like one and a half days until the game. Sorry, yeah, just repeat the timeline. So this was Monday when we played the first set of games, had the loss, did the surgery that night. It played starting on Tuesday, then Wednesday 4 p.m. is when we played RTZ, and then trained for longer. I don't think we made any changes after that, maybe some small ones. Then on Thursday we played Sumail. So I think Tuesday to Wednesday was the night we made fewer changes. There was quite a bit of different work going on that all came together at once.

我认为有一件事非常重要：我们的一位团队成员，他的昵称是 SCI，是一位非常著名的编程竞赛选手，他花了很多时间观察机器人的比赛，看它为什么在这种情况做出奇怪的行为，有哪些奇怪的调整，并真正获得直觉，比如‘哦，因为我们以这种方式表示这个特征，所以如果我们把它改成另一种方式，它就会以不同的方式工作。’我认为这几乎是一个非常人性化的过程，就像观察专家玩游戏，试图找出构成宏观选择的所有微观决策。开始对你构建的系统拥有这种非常不同的关系是很有趣的。通常，你的目标是让一切都非常可观察，对一切设置指标，如果有什么不可理解，就增加日志记录。这就是你设计系统的方式。但在这里，对于周围的部件你可以这样做，但对于这个机器学习核心，你真的必须从行为层面去理解它。

One thing I think was really important: one of our team members, his handle is SCI, who is a very well-known programming competition competitor, was spending a lot of time just watching the bot play and seeing why it does this weird thing in this case, what are all the weird tweaks, and really getting intuitions for, 'Oh, because we're representing this feature in this way, so if we change it to this other thing, it's going to work in a different way.' I think this is almost a very human-like process of watching an expert play the game and trying to figure out what are all the little micro-decisions that go into this macro choice. It's interesting starting to have this very different relationship to the system you build. Normally, the way you do it, your goal is to have everything be very observable, put metrics on everything, and if something is not understandable, add more logging. That's how you design systems. But here, you have that for the surrounding bits, but for this machine learning core, you really do have to understand it at more of a behavioral level.

有没有让你困惑的时候，你会想，‘这以一种我们没预料到的方式在创造’，它可能甚至有效，但你不知道它为什么或如何做出那个选择？

Was it ever stumping you, where you're just like, 'That's being creative in a way that we didn't expect it to,' and it may be even working, but you don't know why or how it decided to make that choice?

有几个小的。在项目的早期，我们有职业选手测试机器人的下一个版本。他说，‘是的，机器人在补刀方面非常强。’故事中还有另一个我认为有趣的部分。为了看看我们的半职业测试者（他在过去几个月里和这个机器人打了数百场比赛）与 RTZ 相比如何，我们让他和 RTZ 比赛。Artesia 整场比赛都领先，每次补刀都比 RTZ 快 500 毫秒。所以我们的半职业选手说，‘好吧，我最后一搏，试试机器人经常对我用的那个策略。’那是一个策略，你做一些复杂的事情，然后三波兵线压对手，把他逼出塔，你有回血，不需要进去击杀。他做到了，而且成功了。这是机器人教会了他一个可以用来对付人类的策略。我认为这非常有趣，是一个很好的例子，说明你可以从这些系统中得到什么：它们可以发现非常不明显的策略，并且实际上可以教给人类。

A few small ones. In the early days of the project, we had professionals playing the next iteration of the bot. He's like, 'Yeah, the bot is really good at creeping.' There is also one other part of the story that I think is interesting. To see how well our semi-pro tester, who had played hundreds of games against this bot over the past couple months, benchmarks relative to RTZ, we had him play against RTZ. Artesia was up the whole game, beating him to the last hit by 500 milliseconds every single time. So our semi-pro was like, 'All right, I've got one last-ditch effort to try this strategy that the bot always does to me.' It's some strategy where you do something complicated, then triple wave your opponent, get him out of the tower, you have regen, you don't need to go in for the kill. He did it, and it worked. This was the bot had taught him the strategy that you could use against a human. I think that was very interesting, a good example of the kinds of things you can get out of these systems: they can discover very non-obvious strategies that can actually be taught to humans.

和 Sumail 的比赛怎么样？

And how did it go with Sumail?

和 Sumail 的比赛，我们保持不败。我想那天是 5-0。有一件有趣的事：我们可能在未来几周写博客，但自那以后我们实际上和很多职业选手比赛过。我们的机器人需求量很大，一些职业选手在直播中玩它。所以我们更好地观察了人类从完全无法击败它，到如果和它玩足够久，实际上可以变得相当好的过程。那里有一组非常有趣的统计数据，我稍后会提取和分析。

With Sumail, we went undefeated. I think it was 5-0 that day. One thing that's actually interesting: we'll probably blog about this in upcoming weeks, but we've actually been playing against a bunch of pros since then. Our bot has been in very high demand, and some of these pros have been live streaming it. So we've gotten a better sense of watching as humans go from being completely unable to beat it to, if you play against it for long enough, you can actually get pretty good. There's a very interesting set of stats there that I'll be pulling and analyzing in a bit.

现在有没有人类能持续击败这个机器人？

Are there humans that consistently beat the bot today?

是的，我想有一个人有大约 20%的胜率。那个玩家打了数百场比赛，找到了利用它的策略。他在机器人擅长的方面变得和机器人一样好，但事实证明他打了数百场比赛。像这样的顶级玩家，他能击败大多数人类。都是职业选手，好吗？不只是某个擅长击败机器人的随机小孩。看待这件事的方式是，成为职业视频游戏玩家是一个相当高的门槛。有一些人真的很喜欢它，如果你和它打数百场比赛，你会变得非常非常擅长它做的事情。

Yeah, I think there's one who has like a 20% win rate or something. That player played hundreds of games and just finds strategies to exploit it. He becomes essentially as good as the bot at what the bot does, but it turns out that he played hundreds of games against it. A top player like this, he beats most humans. All professionals, okay? It's not just some random kid who's good at beating the bot. The way to think about this is that being a professional video game player is a pretty high bar. There are some who have really liked it, and if you're playing hundreds of games against it, you're going to get very, very good at the things that it does.

和 RTZ 聊天时，我问他，‘它有没有改变你的打法？’他说他认为它为他做的是帮助他更专注，因为当你在线上补刀时，现在突然变得非常粗暴，因为你已经做了很多，变得非常擅长。我认为一个非常有趣的事情是，如何提高人类的打法，能否改变人类的打法？我认为我们开始看到一些积极的答案。

Talking to RTZ, I was asking him, 'Has it changed your play style at all?' He said he thinks the thing it's done for him is it's helped him focus more, because while you're just there in lane last-hitting, now suddenly that's just so rude, because you have been doing it so much, you've gotten so good at it. I think one really interesting thing to see is how can you improve human play style, can you change human play style? I think we're starting to see some positive answers in that direction.

我知道我们快没时间了。我可以做一个快速问答，快速过一下特别的问题。实际上，关于你需要培养哪些技能的问题，你有一个非常具体的清单吗？至少在我们数据团队，我们发现一些分布式系统的知识非常有用，因为我们构建了很多这样的系统，而且这些系统很容易做得不好。另一个我们发现非常重要的东西是实际上编写无错误的代码。

I know we're almost out of time. I could do a little lightning round, just quickly go through special questions. Actually, to the question of what kind of skills you need to work on, do you have a very specific list of things that we found very useful, at least in the data team? Some knowledge of distributed systems, because we build a lot of those, and those are easy to not do properly. Another thing that we found very important is actually writing bug-free code, essentially.

在计算机科学界，大家普遍认为每个人都会犯错，但在这里，尽量减少错误更为重要，因为它们很难调试。具体来说，许多错误表现为训练性能下降，而要得到那个数字需要一天时间。在超参数调整的忙碌中，很容易忽略。所以主要的调试方式实际上是阅读代码。每个错误的代价都很高，因此编写正确、无错误的代码对我们来说非常重要。我们有时会牺牲良好的工程实践（如代码模块化）来使代码更短、更简单，本质上就是减少可能出错的代码行数。

It's kind of taken for granted in the computer science community that everybody makes bugs, but here it's even more important to minimize them because they're very hard to debug. Specifically, many bugs manifest in lower training performance, and to get that number takes a day. In a flurry of hyperparameter changes, it's really easy to miss. So the primary way of debugging is actually reading the code. Every bug has a very high cost associated with it, so writing correct, bug-free code is quite important to us. We sometimes sacrifice good engineering practices like code modularity to make our code shorter and simpler, essentially having fewer lines where you can make bugs.

主要技能是良好的工程能力，但如果有人觉得需要补习数学，比如线性代数和基础统计学，尤其是在做实验时，很容易犯基本的统计错误。线性代数是你需要了解的大部分内容，再加上基本优化，才能理解这些模型中发生的事情。但与成为一名优秀的工程师相比，这些技能相对容易掌握，至少在我们正在做的这类项目中是这样。

And the primary skills are good engineering, but if somebody feels like they need to brush up on math, like linear algebra and basic statistics, especially when doing experiments, it's easy to make elementary statistics mistakes. Linear algebra is most of what you need to know, along with basic optimization to follow what's happening in those models. But compared to being a good engineer, that's quite easy to pick up, at least in a project like the one we're doing.

我想谈谈一些我认为非常重要的非技术技能。一个是，如果你像我一样来自工程背景，就需要保持谦逊。你参与的项目中，你不再是传统意义上的技术专家。当你为医生构建产品时，你可以和十位医生交流，无论你构建什么，都可能对他们的工作流程有宝贵价值，因为医生通常无法自己构建软件工具。但在机器学习研究中，你合作的每个人都非常技术化，可以自己构建工具。然而，如果你在正确的地方注入工程纪律，在正确的时间构建正确的工具，审视工作流程并思考另一种方式，那才是你真正能增加价值的地方。所以关键在于知道何时注入工程纪律，也知道何时不该注入。正如 Sherman 所说，有时我们确实想要最短的代码，因为我们非常害怕错误，这可能会导致与你预期不同的选择。

I wanted to talk about some non-technical skills that I think are really important. One is that there's a real humility required if you're coming from an engineering background like I am. You're working on projects where you're no longer the technical expert in the way that you're used to. When you build a product for doctors, you can talk to ten doctors and whatever you build will probably be a valuable addition to their workflow because doctors can't really build their own software tools. But with machine learning research, everyone you work with is very technical and can build their own tools. However, if you inject engineering discipline in the right place, build the right tool at the right time, and look at the workflow to think about doing it in another way, that's where you can really add value. So it's about knowing when to inject engineering discipline, but also knowing when not to. To Sherman's point, sometimes we really want the shortest code because we're terrified of bugs, and that can yield different choices than you might expect.

在 OpenAI，谁每行代码的错误最少？

Who writes the least bugs per line of code at OpenAI?

这是个有争议的问题。我不会说是我。我觉得我导致的异常错误最少。我的错误通常会导致异常，所以没问题。你不想要的是那些导致正确性问题的事情，比如性能下降 10%。

That's a contentious question. I'm not going to say me. I think I have the least amount of bugs that cause exceptions. My bugs usually cause exceptions, so that's fine. What you don't want are things that cause correctness issues, where performance gets 10% worse.

Tim Beco 问：非技术人员如何帮助 AI 初创公司？

Tim Beco asks: How can non-technical people be helpful to AI startups?

我认为重要的一点是，目前关于 AI 有很多噪音，很难区分真假。仅仅自我教育就是一件非常重要的事情。很明显，AI 将产生巨大的影响。只需看看已经创造出来的东西，并在没有新技术发展的情况下进行推断，很明显 AI 将被融入许多不同的系统。有很多伦理问题需要解决。在这些对话中发声并自我教育非常重要。然后看看我们接下来能开发什么，这才是真正变革性东西的来源。

I think one important thing is that for AI generally right now, there's a lot of noise, and it can be hard to distinguish what is real from what's not. Simply educating yourself is a pretty important thing. It's very clear that AI is going to have a pretty big impact. Just look at what's already being created and extrapolate that without any new technology development, and it's pretty clear AI will be baked into lots of different systems. There are a lot of ethical issues to work through. Being a voice in those conversations and educating yourself is really important. Then you look at what we'll be able to develop next, and that's where the really transformative stuff will come.

我曾在 Greg 的 RescueTime 报告中看到一篇文章，感到非常震惊。对于长时间工作有什么建议吗？

I once saw a post in Greg's RescueTime report and was pretty shocked. Any advice for working such long hours?

我认为最大化坐在电脑前的时间不是一个好目标。对我来说，我这样做是因为我热爱它。我在世界上最喜欢的活动就是进入状态，编写代码并产出有意义、有价值的东西。作为二阶效应，这可能是有益的，但我不会说这是产生影响的方式。更具体地说，我见过人们超级高效的唯一方式是他们做自己热爱的事情。没有其他东西能让你长期坚持下去。

I think it's not a good goal to maximize the number of hours you sit at your computer. For me, I do it because I love it. The activity I love most in the world is when you're in the zone, writing code and producing something meaningful and worthwhile. As a second-order effect, it can be good, but I wouldn't say that is the way to have an impact. More specifically, the only way I've ever seen people be super productive is if they're doing something they love. There is nothing else that will sustain you over a long enough period of time.

许多初创公司是否过度使用 AI 这个词只是为了在媒体上好看？

Is the term AI overused by many startups just to look good in the press?

是的。

Yes.

随着 AI 开始做其他所有事情，最后剩下的工作是什么？

What is the last job that will remain as AI starts to do everything else?

最后的人类工作将是 AI 最难做的事情。总的来说，我认为实际上不是 AI 研究员。AI 研究员会先于它消失。当你问人们这个问题时，非常有趣。每个人都倾向于说自己的工作是最难的。但我实际上认为 AI 研究员是你希望让这些系统变得非常好的工作之一。

The last human job is going to be the hardest thing for AI to do. In general, I think it's actually not AI researcher. The AI researcher will kind of go before it. It's very interesting when you ask people this question. Everyone tends to say whatever their job is is the hardest one. But I actually think that AI researchers are one that you're going to want to make these systems very good.

你能解释一下玩电子游戏与构建 AGI 之间的联系吗？

Can you connect the dots between how playing video games is relevant to building AGI?

这实际上可能是最让我惊讶的事情之一，游戏被用于 AI 研究的程度。你真正想要的是在复杂环境中运行的算法，它们可以学习技能，并且你想增加这些技能的复杂性。你要么推动环境，要么推动复杂性，要么扩展算法。游戏很棒，因为它们是一个预先打包好的环境，其他人类花时间制作，首先投入了大量复杂性，确保有实际的智力问题需要解决，或者有趣的机械挑战，你可以从中获取人类数据。

It's actually maybe one of the most surprising things to me, the degree to which games end up being used for AI research. The real thing you want is to have algorithms that operate in complex environments where they can learn skills, and you want to increase the complexity of those skills. You either push the environment, push the complexity, scale the algorithms up. Games are great because they are a prepackaged environment that other humans have spent time making, first of all putting in a lot of complexity, making sure there are actual intellectual things to solve, or interesting mechanical challenges that you can get human data from.

在它们上面设定基线水平，这样你就能确切知道它们有多难。它们非常方便，不像机器人技术那样无法完全在虚拟环境中运行。这意味着你可以扩展它们，运行多个副本，所以它们是非常方便的试验台。我认为你会看到很多工作将在游戏中进行，但目标当然是将它带出游戏，真正用于解决现实世界的问题，能够与人类互动并做有用的事情。所以我认为它们是非常好的起点。我非常喜欢这个 Dota 项目的一点是，把它带给所有这些职业选手，未来我们都会与超级先进的 AI 系统互动。现在，我认为我们对于它们如何运作、在哪里失败、与它们互动是什么感觉，还没有很好的直觉。这是一种非常低风险的方式，让你第一次接触非常先进的 AI 技术。

Level baselines on them so you know exactly how hard they are. They're very nice, unlike something like robotics where you can't run them entirely virtually. That means you can scale them up and run many copies of them, so they're a very convenient testbed. I think you're going to see a lot of work done in games, but the goal is of course to bring it out of the game and actually use it to solve problems in the real world, to be able to interact with humans and do useful things. So I think they're a very good sort of starter. One thing I really like about this Dota project and bringing it to all these pros is that we're all going to be interacting with super advanced AI systems in the future. Right now, I don't think we have good intuitions as to how they operate, where they fail, what it's like to interact with them. This is a very low-stakes way of having your first interaction with very advanced AI technology.

酷。如果有人想加入 OpenAI，他们应该怎么做？

Cool. If someone wants to get involved with OpenAI, what should they do?

哦，嗯，我们在网站上有招聘信息。我想我给出的关于如何在 OpenAI 找到工作的建议非常针对特定的职位，我们那里大部分是大型扩展力量和让工程师离开。是的，总的来说，我们寻找那些在各自技术专长上非常出色的人，我们可以利用很多不同的专长。

Oh, well, we have a job posting on our website. I guess the tips I'm giving about how to get a job at OpenAI are very geared towards a specific job, most of what we have there, which is a large scaling force and letting engineer go. Yeah, and in general we look for people who are very good at whatever technical axis they specialize in, and we can use lots of different specialties.

太好了，好的。谢谢大家。只是想重申一下，每个人都认为他们必须是 AI 博士，其实不是。这两位都不是。好的，非常感谢。谢谢，是的，谢谢。

Great, alright. Thanks guys. Just to echo that, like everyone thinks they have to be an AI PhD, not true. Neither of these guys are. Alright, thanks a lot. Thanks, yeah, thank you.