为何下一次 AI 突破可能来自物理学

到目前为止，AI 领域大多数重大突破都来自用更多数据训练更大的模型。但这不是唯一的出路。今天的嘉宾 Max Welling 认为，更好的 AI 可能来自一个非常不同的地方：物理学。Max 是 KUSP AI 的联合创始人兼 CTO，也是阿姆斯特丹大学的教授。在 KUSP AI，他的团队正在利用生成式 AI 设计全新的材料，用于半导体、电池、碳捕集和其他科学应用。但我们的对话不止于 AI for science。Max 认为，物理学不仅仅是 AI 可以应用的另一个领域。物理学家为理解自然世界而发展出的数学思想——热力学、波传播、对称破缺等概念——也可能帮助塑造下一代 AI 架构。以下是 Max 谈为什么物理学可能塑造下一代 AI。

So far, most of the biggest breakthroughs in AI have come from training larger models on even more data. But that's not the only way forward. Today's guest, Max Welling, believes that better AI may come from someplace very different: physics. Max is the co-founder and CTO of KUSP AI and a professor at the University of Amsterdam. At KUSP AI, his team is using generative AI to design entirely new materials for semiconductors, batteries, carbon capture, and other scientific applications. But our conversation goes beyond AI for science. Max argues that physics isn't just another domain where AI can be applied. The mathematical ideas physicists have developed to understand the natural world—concepts like thermodynamics, wave propagation, and symmetry breaking—may also help shape the next generation of AI architectures. Here's Max on why physics may shape the next generation of AI.

现在在大脑中确实观察到了波，因为我们从单电极测量发展到了数百或数千个电极的测量。当然，在单个电极中，你无法看到移动的波，即行波。但如果你放入数百或数千个电极，你实际上可以看到这些行波。现在人们已经在各处观察到了它们。在神经网络中，一直有点难以确保信息从输入层一直传播到输出层，穿过数千层。所以问题变成了：你如何深入时间进行通信或深入时间进行推理，或者如何与你大脑中非常遥远的事物进行通信？因此问题变成了：我们能否真正开始利用这种现象，这种我们在大脑中看到的波现象，并且我们能否也开始在神经网络中使用它？这就是我们一直在开发的工作。

Waves are actually seen in the brain now because we've gone from single electrode measurements to maybe hundreds or thousands of electrode measurements. And of course, in a single electrode, you cannot see a moving wave, a traveling wave. But if you put hundreds or thousands in them, you can actually see these traveling waves. And people have now observed them everywhere. In neural networks, it's always been a little hard to make sure that the information travels all the way from the input to the output layers through thousands of layers. So the question becomes: how do you communicate very deep into time or reason very deep into time, or how do you communicate with things that are very far away in your brain? And so the question becomes: can we actually start using this phenomenon, this wave phenomenon that we see in the brain, and can we also start to use it in neural networks? And so this has been the work that we've been developing.

我是 Sam Cherington，这里是 TwiML AI 播客。十多年来，我一直通过这样的对话探索塑造 AI 未来的想法和创新，帮助你理解什么是真实的、什么是下一个、什么重要。让我们开始吧。

I'm Sam Cherington and this is the TwiML AI podcast. For over a decade, I've been exploring the ideas and innovation shaping the future of AI through conversations like this one that help you understand what's real, what's next, and what matters. Let's jump in.

Max，很高兴再次与你连线。好久不见了。

Max, it's so great to be on the line with you again. It's been a while.

很高兴回来，Sam。我很期待我们的讨论。

It's great to be back, Sam. I'm looking forward to our discussion.

我也是。我们的听众可以回顾我们 2019 年和 2020 年的对话，当时我们讨论了你当时的工作，我认为这些工作仍然回响在你今天的工作中：几何神经网络、规范等变神经网络等等。但我想请你向我们介绍一下你从那以后一直在做什么。已经过去挺久了。

I am as well. Our audience can look up our conversations from I think 2019 and 2020 where we covered what you were working on at the time and I think still echoes into your work today, geometric neural networks, and gauge equivariant neural networks and the like. But I'd love to have you kind of catch us up on what you've been up to since. It's been quite a while.

是的，实际上等变性这个主题肯定延续下来了。事实上，我发现等变性在化学和材料科学中得到了非常富有成果的应用。所以在化学和材料科学中，人们训练神经网络模型来预测原子上的力，因为人们想要让原子随时间演化，以计算它们的性质，这被称为分子动力学。通常你需要使用量子力学来计算这些力，因为很大一部分贡献来自电子，而电子非常轻。所以你需要用量子力学来处理它们。但如果你有 10 个或更多电子，求解所谓的薛定谔方程基本上就完全不可行了。所以人们提出了近似方法，比如密度泛函理论，简称 DFT，其发明者因此获得了诺贝尔奖。但现在人们做的是训练代理模型。他们提供使用这种昂贵的量子力学近似计算的数据，然后使用神经网络来 shortcut 计算，预测该计算的结果，但速度要快得多。换句话说，相对于这些量子力学近似，加速了三到四个数量级，效率更高。在这些模型中，因为世界是三维且对称的，如果我旋转一个分子，所有的力都会随之旋转。所以我们现在可以把用于图像的相同思想，放入这些分子、这些预测力的模型中，我们可以使用等变性。所以这就是为什么我——也因为我的背景是科学，我在理论物理获得了博士学位——我想，好吧，这是我的旧热情和新热情的完美统一，可以把它们结合起来。那时我开始对 AI for science 感兴趣。

Yeah, actually the equivariance theme has definitely continued. In fact, I found out that equivariance was used very fruitfully in chemistry and material science. So in chemistry and material science, people train neural network models to predict the forces on atoms because people want to evolve atoms forward in time in order to compute their properties, which is called molecular dynamics. And typically you need to use quantum mechanics to compute these forces because a large contribution comes from the electrons and electrons are very light. And so you need to treat them with quantum mechanics. But if you get 10 electrons or more, it basically becomes completely unfeasible to solve the so-called Schrödinger equation. So people have come up with approximations like density functional theory, known as DFT, and the inventors of that got the Nobel Prize for that. But what people do now is they train surrogates. So they provide data computing this using this expensive approximation to quantum mechanics. And then they use neural networks to shortcut the computation, to predict the outcome of that computation but at a much more accelerated pace. So in other words, three orders or four orders of magnitude acceleration, more efficiency relative to these quantum mechanical approximations. And in those models, because the world is three-dimensional and symmetric, if I rotate a molecule, all the forces will rotate with it. And so we could now put the same ideas that we used for images, we could put them in these molecules, these models that predict the forces, and we could use equivariance. And so that's why I kind of—also because my background is in science, I did my PhD in theoretical physics—I thought, okay, this is a perfect unification of my old passion and my new passion and can put it together. And that's when I started to be interested in AI for science.

所以这直接促成了 KUSP AI 的创立？

So that led pretty directly to the founding of KUSP AI?

实际上，我先在微软研究院做了两年副总裁，因为他们在阿姆斯特丹建立 AI for science 实验室，所以我帮助推进了这件事。但两年后我想创办一家初创公司。我之前已经做过一家初创公司，但那家实际上被高通收购了，然后我在高通待了一段时间。我真的很喜欢初创公司，那种动态的环境和你所能产生的影响。我想和我的联合创始人 Chad Edwards 一起以硅谷的方式来做这件事。于是我们在 2024 年创办了 KUSP AI。

Actually first I spent two years at Microsoft Research as a VP because they were building their AI for science lab in Amsterdam and so I helped that along. But after two years I wanted to start a startup. I already did a startup a while ago but this was actually the startup that got acquired by Qualcomm and then I spent some time at Qualcomm. And I really like startups, the dynamical environment and the impact you can make. And I wanted to do it the Silicon Valley way together with my co-founder Chad Edwards. And so we started KUSP AI in 2024.

请谈谈你们从那以后取得的进展。公司今天大概是什么样子？

Talk a little bit about the progress that you've made since then. What is the kind of shape of the company today?

实际上这是一段巨大的旅程。就像坐过山车。我们大约两年前开始，2024 年 5 月，春季。是的，我们以约 3000 万美元的良好初始投资起步，从中我们可以聘请一支优秀的团队。所以团队现在已经增长到约 50 人，分布在不同的地区。总部在阿姆斯特丹和剑桥都有。实际上总部 officially 在剑桥，但最初的两个实验室是阿姆斯特丹和剑桥，因为 Chad 来自剑桥，我来自阿姆斯特丹。我们现在在伦敦和柏林也有实验室。我们还在向亚洲和北美扩张。

It's been a huge ride actually. It's a roller coaster. So we started I think about two years ago. So 24, May, spring 24. Yeah, we started with a good initial investment of about 30 million from which we could hire an excellent team. So the team has grown to about 50 people right now across different geographies. So there's a headquarters both in Amsterdam and in Cambridge. Actually the headquarter officially is in Cambridge but the two initial labs were Amsterdam and Cambridge because Chad is from Cambridge and I'm from Amsterdam. We now also have labs in London and Berlin. And we're also expanding into Asia and North America.

明白了。毫不意外，你的顾问名单有点像名人录，Jeff Hinton 和 Yann LeCun 名列前茅。

Got it. And no surprise, your list of advisers is a bit of a who's who with Jeff Hinton and Yann LeCun at the top of the list.

是的。顾问们实际上非常棒。我们有 Jeff Hinton 和 Yann LeCun。我们还加入了 Martin van den Brink 和 Lord Brown。Lord Brown 是 BP 的前 CEO，Martin van den Brink 是 ASML 的前总裁兼 CTO。他们都已经退休，喜欢把时间花在新的初创公司上并帮助他们。然后是 Varity Harding。她在为英国政府工作，我想也在 DeepMind。然后是 Kristen Pson，她发起了 Materials Project，她也在那方面提供建议。

Yes. Yeah, the advisers are actually fantastic. So we have Jeff Hinton and Yann LeCun. We added to that also Martin van den Brink and Lord Brown. So Lord Brown is the former CEO of BP and Martin van den Brink is the former president and CTO of ASML. They're both retired and they like to spend their time with new startups and help them along. And then there's Varity Harding. She's working for the UK government and also DeepMind I think. And then Kristen Pson who has initiated the Materials Project and she's also advising on that side.

明白了。

Got it.

那么让我们深入探讨一下你将你开发的这套原创工作应用到材料上的技术和方式。你是如何开始这项工作的？

So let's dig into the technology and the approach that you're taking to apply this original set of work that you developed to materials. How did you get started with that effort?

所以我认为机会在于——对我来说，一个深深着迷的事实是，你组合原子的方式并非无限多。而宇宙只弄清楚了其中一部分，因为它们自然形成，我猜是在宇宙中。但还有更多你可以自己设计的、具有各种奇特性质的材料。当我们创办这家公司时，查德和我都关心气候问题。我们现在仍然关心。所以我们觉得迫切需要加速向更可持续能源的转型，同时努力去除大气中的二氧化碳。所以不是很多人知道，到 2015 年，我们当然希望完全碳中和，但之后仍有 50 到 100 年，我们每年必须去除大约我们当前排放量的一半。所以那是每年 200 亿吨。那是巨大的量。那相当于日内瓦湖装满液态二氧化碳——一个巨大的量。而我们没有那样的技术，因为实际上很难去除，因为它在大气中太稀薄了。它也非常昂贵，而两个最昂贵的因素是能源和用于从大气中去除二氧化碳的吸附材料。所以我们一直在做的第一批项目是改进从大气中去除二氧化碳的材料，这些被称为金属有机框架。实际上，这就是今年获得诺贝尔化学奖的材料。所以我们建立了一个平台——我可以更详细地讲——但我们建立了一个利用 AI 设计这些材料的平台。所以你可以把它想象成一个搜索引擎，但它不是在搜索现有的文档。它搜索已知和未知的材料。所以如果还没有合适的材料，它实际上会从头完全设计它们。而且有很多组件。它是智能体式的。所以有一个智能体坐在它的核心，协调一个长期的计算。它搜索现有数据库。它生成全新的分子。它还使用各种工具评估所有这些分子。在这个生成和评估中，等变性和我们多年来在我的学术实验室开发的各种方法发挥着非常重要的作用。

So I think the opportunity is—to me there's a deep fascination with the fact that there isn't a sheer infinite amount of possibilities in which you can put together atoms. And the universe has only figured out so many of them because they form naturally, I guess, in the universe. But there are many more that you can design yourself with all sorts of exotic properties. When we started this company, both Chad and I were concerned about the climate. We still are. And so we felt there's a strong need to accelerate the energy transition to more sustainable energy sources as well as trying to take the carbon dioxide that's in the atmosphere out. So not many people know that by 2015, of course, we'd really like to be completely carbon neutral, but after that there is still 50 to 100 years where we have to take out every year about half of what we currently put in. So that's 20 gigatons a year. That's an enormous amount. That's kind of the size of Lake Geneva filled with liquid carbon dioxide—a gigantic amount. And we don't have the technology for that because it's actually very hard to take it out because it's so dilute in the atmosphere. It's very expensive too, and the two most expensive factors are energy and the sorbent material that you use in order to take the carbon dioxide out of the atmosphere. And so the first projects we've been doing were improving the materials that take out this carbon dioxide from the atmosphere, and these are called metal-organic frameworks. Actually, this is the material that this year won the Nobel Prize in Chemistry. And so we built a platform—I can go into much more detail—but we built a platform that designs these materials with the use of AI. So you can think of it as a search engine, but it's not searching over existing documents that exist. It searches over known and unknown materials. So it actually completely designs them from scratch if there are no suitable ones already available. And there are many components. It's agentic. So there is an agent sitting at the core of it who orchestrates a long computation. It searches through existing databases. It generates entirely new molecules. And it also evaluates all these molecules with all sorts of tools. And in this generation and evaluation, equivariants and all sorts of methods that we have developed over the years in my academic lab play a very important role.

澄清一下，你提到了一种获得诺贝尔奖的分子。你参与了那个特定分子的发现吗？

To be clear, you mentioned a molecule that won the Nobel Prize. Were you involved in the discovery of that particular molecule?

不，不。我希望是我。那些是化学家做的，他们发现了它。我想是 Kitagawa 教授、Yaghi 教授和 Robson 教授——我想是这三位，我不太确定最后一位。但这是一类分子，其中有一个金属节点，金属配合物，在图的顶点上有各种原子，然后有所谓的连接体，也是有机配合物，连接这些顶点。它们非常多孔。所以它们中间有很大的孔洞，表面积巨大，所以如果你吹入空气，空气中的分子——水、氮气和二氧化碳，但二氧化碳只占很小一部分——它们倾向于粘在侧壁上。所以你需要制造一个分子——你需要设计一个分子，实际上只有二氧化碳粘在这些孔内，其余通过。所以这是一个设计问题。而且当它满了，你想摇晃或加热它来释放出来，以便你可以实际重用那个材料。我们添加的是一种为特定目的微调或设计分子的方法。所以人们在实验室里实际上已经制造了大约 10 万个这样的分子，并验证了它们的结构。所以我们现在能做的是，基本上为非常具体的任务提出一个全新的分子，然后在实验室里制造它，然后用于那个特定任务。我应该说明，KUSP 不仅在做 MOFs。事实上，这只是我们合作的第一组分子。我们已经扩展到半导体。我们正在做大量关于钙钛矿的工作，这是用于太阳能电池板的材料——改进的太阳能电池板。我们研究用于新芯片材料的半导体。我们研究电池材料、燃料电池。我们还研究从水中去除 PFAS，我们目前用于此的分子组又是金属有机框架。

No. No. I wish. Those were chemists who did that, who discovered it. I think Professor Kitagawa, Professor Yaghi and Professor Robson—I think those are the three, I'm not quite sure of the last one. But this is a class of molecules where there is a metal complex in a metal node with all sorts of atoms along it at the vertices of a graph, and then there are so-called linkers which are also organic complexes which are connecting these vertices on the graph. And they're extremely porous. So they have very large holes in the middle with an enormous surface area, and so if you blow air atmosphere through it, the molecules in the air—which is water, nitrogen and carbon dioxide, but carbon dioxide is only a small fraction of that—they tend to stick to the sides. And so then what you need to make is a molecule—you need to design a molecule where actually only the carbon dioxide sticks inside these pores and the rest goes through. So that's a design question. And also when it's full, you want to shake it or heat it to get it out so that you can actually reuse that particular material. What we added to this is a way to fine-tune or design a molecule for a particular purpose. So people have actually made maybe around 100,000 of these molecules in labs and verified their structure. And so what we can do is we can now basically come up with an entirely new molecule for a very specific task and then make it in the lab and then use that for that particular task. I should say KUSP is not only working on MOFs. In fact, this was just the first set of molecules that we worked with. We have hence expanded to semiconductors. We're doing a lot of work on perovskites, which are materials for solar panels—improved solar panels. We look at semiconductors for new chip materials. We look at battery materials, fuel cells. We also look at removing PFAS from water, and the current set of molecules we use for that is again metal-organic frameworks.

你们是与对这些特定分子感兴趣的其他公司合作，还是你们自己探索，然后如果发现什么，再找合作伙伴并可能授权给他们？关于商业模式的思考是什么？

Do you partner with other companies that have an interest in these particular molecules, or are you out exploring and then if you find something you will find partners and maybe license to them? What's the thinking around the business model?

所以我们喜欢与合作伙伴合作，因为材料类别非常广泛，每个类别都有自己的超级专家专注于那些特定领域，他们要么在学术界，要么在公司。当然，我们也喜欢在实际合成这些材料上合作。所以我们与学术实验室合作，也与这些公司内部的实验室合作。因此我们建立一个生态系统或网络，我们的引擎实际上可以帮助所有这些不同材料类别设计材料，然后我们与那些公司合作实际制造。所以那是一种合作模式。但我们也运行内部项目。例如，关于用于碳捕获的金属有机框架的项目，我们内部自筹资金运行，然后我们现在在半导体方面还有另一个项目，我们运行它，所以如果我们发现了一些奇妙的东西，我们拥有知识产权，然后我们可以为那个知识产权找到客户。最重要的是，我认为最重要的是我们发现了某些东西，让人们有一定的信心，我们能做到这一点，我们真正拥有这个过程，我们知道如何做，然后我们与客户合作，为他们的特定需求设计材料。

So we like to work with partners because there's a very broad class of materials and every class has its own super experts that focus on those particular areas, and they're either in academia or they're in companies. And of course we also like to partner on the actual synthesis of these materials. So we partner with academic labs but also with the labs inside of these companies. And so we build an ecosystem or a network where our engine can actually help in all of these different material classes design the materials and then we work with those companies to actually make it. So that's a partnership model. But there's also internal projects that we run. So for instance the project on metal-organic frameworks for carbon capture we ran self-funded internally, and then we also have another project now in the semiconductor side where we run it and so if we discover something fantastic with selfancer we got the IP and then we can find customers for that IP. Most important, I think the most important thing is that we discover something that gives people a certain confidence that we can do this, we actually really own this process and we know how to do it, and so then we work with customers to design materials for their specific needs.

你稍微谈到了这种扫描科学文献的生成性或有机性质，并利用它来识别潜在分子，但我们也谈到了你工作的一些几何含义。

You talked a little bit about the generative or organic nature of this scanning scientific literature and using that to identify potential molecules, but we've also talked about some of the geometric implications of your work.

你提到过可能使用模拟。你能再讲讲识别这些分子的端到端流程吗？

You alluded to the potential use of simulation. Can you talk a little bit more about the kind of end-to-end process of identifying these molecules?

好的，很乐意。我想这里有一连串的步骤，对吧？第一步就像在搜索引擎里，你实际上输入一个请求。你基本上说我要一种材料，这些是它应该具备的全部性质，这些是它不应该具备的性质。你把这个作为查询提交。然后，如果你对希望这次搜索如何进行有先验知识，你也可以告诉它，也许用这些工具，也许按这个顺序来。所以你还可以给它一些指令。

Yeah, happy to. I guess there's a sequence of things that happens, right? So the first thing is, like in a search engine, you actually type a request. So you basically say I want a material and these are all the properties that it should have, and these are the properties it should not have. So you give this as a query. And then you could also tell it, if you have prior knowledge about how you want this particular search to happen, you could tell the system maybe use these tools and maybe sequence it in this way. So you can also give it some instructions.

然后它会走一个分步流程。第一步是它会检索自己的数据库。我们有一个非常大的材料数据库，我们把所有内容都导入进去了。其中很多是从大型出版社获得的独家授权。然后它会开始检索所有这些文献，看外面是否存在具备这些性质的东西，或者接近具备这些性质的东西。

And then it goes through a process of steps. So the first step is it will look through its database. So we have a very large database of materials which we have all ingested into this database. Lots of it is sort of exclusive licenses from the big publishing houses. And then it will start to look through all of this literature to see whether something exists out there that has these properties or which is close to having these properties.

如果没有，那么它就得进入一个新阶段。你可以和这个智能体对话，讨论它。所以它已经相当有用了。但通常下一步就是转向生成模型。在这个场景里，就是生成图像或视频的同一个生成模型。但在这里它会为你生成分子。

And so if it doesn't, then it will have to go into a new phase. So you can hold a conversation with this agent and talk about it. So it's already quite useful. But typically then the next step is that you go to a generative model. So in this case it's the same generative model that generates images or video. But in this case it will generate molecules for you.

你告诉它我要具备这些非常具体性质的分子——这叫做条件语句。然后它就会开始生成这些分子，往往是几十万个，因为在计算机里这很便宜。

So you tell it I want molecules with these very specific properties — the conditioning statement, it's called. And then it will start to generate these molecules, often hundreds of thousands of them, because it's quite cheap in a computer.

然后就是下一个阶段：从这些生成的分子中，我们现在必须筛出那些看起来非常有希望的，而这可能是一个非常昂贵的步骤。某种意义上，你为你真正希望这些分子运行于其中的过程，构建了一个多尺度的数字孪生。

And then comes the next phase, which is out of these generated molecules we now have to sieve out the ones which look very promising, and this can be a very expensive step. In some sense you build a multiscale digital twin of the process that you really want these molecules to operate in.

所以第一步基本上是把分子弛豫到基态，确保它处于最佳能量状态。然后你做一系列检查，比如它带电吗？我晃一晃它会散架吗？内部孔隙有多大，如果这重要的话——各种容易快速计算的东西，用来扔掉一大堆看起来没希望的东西。

And so the first step is basically you relax the molecule to its ground state to make sure that it's in its best energy state. Then you do a bunch of checks, like is it charged? If I shake it, will it fall apart? How big are the pores inside, if that's important — all sorts of things that are easy to compute fast to throw away a whole bunch of things that do not look promising.

所以就是把它筛下来。

So filtering it down.

对，这绝对是一个过滤步骤。是的。然后进入下一步。我们有一条流水线，为那种特定材料微调或蒸馏机器学习力场。这个过程是：我们取来关于那种材料的全部数据。我们有一个在广泛得多的数据上训练过的基础模型，然后我们把力场蒸馏成一个对这类特定问题非常高效的力场。

Yeah, it's definitely a filtering step. Yes. And then go to the next step. So we have a pipeline that fine-tunes or distills machine learning force fields for that particular material. So this is a process by which we take all the data there is about that material. We have a foundation model that's trained on a much wider range of data, and then we distill this force field into a very efficient force field for this particular class of problem.

我们把它用在 MD 循环里，通常是分子动力学循环，来模拟分子扭动、移动的过程，从中你往往能计算出那种特定分子的非常关键的性质。这些性质随后常常进入更高尺度的偏微分方程，或者进入一个真正对你想让这种材料运行于其中的器件建模的过程。

And we use that in an MD loop, typically the molecular dynamics loop, to simulate the molecule as it wiggles around and moves around, from which you can often compute very key properties of that particular molecule. Those properties then often go into a partial differential equation at a higher scale, or in a process that actually models the device in which you want this material to operate.

这又更昂贵了，所以你希望用越来越少的候选材料来做。最后你进入实验，对吧？现在我真的做实验——那更昂贵、也更慢，按老办法你最多只应该对大约 10 种材料做这件事。

And so that's again more expensive, and so again you want to do this with fewer and fewer candidate materials. And then at the very end you go to an experiment, right? Now I actually do the experiment — now that's more expensive and even slower, and you should do this only with order 10 materials at most in the old way.

然后你得到一个候选。我想说，这个领域当前正在发生的革命是自驱动实验室，你能做的实验数量要快得多。也许一天能做一百个实验。然后游戏规则更像是：智能体弄清楚实验的设置应该是什么。实验做完，数据回流，然后你有来自实验的数据和来自计算的数据。你把它们结合起来，设定下一阶段的实验。所以对那些来说，实验循环要快得多，而这是一个非常有趣的发展，我们现在正把平台与之整合。

And then you get a candidate. What I would say is the current revolution happening in this space is self-driving labs, where the amount of experiments you can do is much much faster. So you can do maybe a hundred experiments a day. And then the game is more like the agent figures out what the settings of the experiment should be. The experiments are done, the data comes back in, and then you have the data from the experiment and the data from your computations. You combine them to set the next stage for the experiments. And so the experimental loop is much much faster for those, and that's a very interesting development that we are now integrating our platform with.

明白了。那么你们有多少种分子类别和单个分子，已经完整走完了这个循环？

Got it. And so how many molecule classes and individual molecules have you gone all the way through this cycle with?

是的，我们正在做其中几个，但这个问题有点——你说的“完整走完”是什么意思？不同材料处于不同的成熟阶段。其中一个我们已经一路走到了真正做实验室实验。另一个关于半导体的正在推进，大概几个月后我们会开始做实验。

Yeah, we are engaged in a few of those, but the question is a little bit — what do you mean by all the way? The different materials are at different stages of maturity. So one of them we went all the way to actually doing the lab experiments. Another one on semiconductors is on its way, and probably in a few months we'll start doing the experiments.

我指的是完整走到实验室实验。我很好奇，你是否有足够的数据说明你们用这个流程的命中率更高还是更低，或者关于你们产出的候选相对传统方法，你能定性地或定量地说些什么。

Meant all the way to lab experiments. And I was curious if you have enough data to say that your hit rate with this process is higher versus lower, or if there's anything you can say qualitatively or quantitatively about the candidates that you produce relative to the traditional approach.

是的。我绝对认为有明确证据表明这些东西效率高得多。我们的一些科学家说过，我们现在几天能做的事，以前要一个博士生来做——但这更多是在数字领域。这些其实是更偏模拟工作的科学家。

Yeah. So I definitely think that there's definite evidence that these things are much more efficient. Some of our scientists have said things like what we can do now in a few days took a PhD before — but that's more in the digital domain. These are actually scientists that are more simulation-based work.

我们有两个进行中的项目带有实验部分。但我们与亚洲一家大型国家实验室签了合同，细节我还不太能说，我们在那里计划了更多这类实验。基本上安排是：我们带来一些客户，我们可以在那个实验室里和他们的实验室科学家一起做实验，他们也可以带来他们的合作伙伴，他们可以使用我们的平台，我们就能以这种方式收集数据。

We have two projects going which have experimental pieces to it. But we have a contract with a big national lab in Asia, which I cannot quite say the details of yet, where we have many more of these experiments planned out. So basically the setup is that we bring some customers and we can do experiments in that lab with their lab scientists, and they can also bring their partners, and they can use our platform, and we can collect data that way.

然后还有另一个我们目前在阿姆斯特丹做的关于钙钛矿的项目，现在正在运行。所以那也是一个实验室合作。

And then there is one other one that we're currently doing in Amsterdam on perovskites, which is running right now. So that's also a lab engagement.

钙钛矿是什么？抱歉。

Perovskites is what? Sorry.

好的。钙钛矿是一类材料，是一种半导体晶体结构，你会把它放在硅上，通常放在普通太阳能电池的上面，它可以帮助你滤出，或者基本上把可见光中更大比例的能量转化为能量。

Yeah. Perovskite is a material class that's a semiconductor crystal structure that you would put on silicon, typically on top of the normal solar cells, and that can help you filter out or basically convert a much larger amount of the energy in visible light to energy.

那你的实验室，或者说 CUSP，会发表论文吗？你现在还活跃在材料科学相关的学术出版中吗？

And does your lab, or does CUSP, publish? Are you active, are you still active in academic publishing around materials science now?

我自己每周有一天在大学，那天当然还是和学生一起工作，我们在那里什么都发表。我对所有与 AI for science 相关的事情都非常感兴趣，所以绝对是活跃的。不过 CUSP 其实也会发表论文。我们有实习生和我们的科学家一起工作，我们会把结果发表出来。比如我们最近就有关于带不确定性预测的机器学习力场、性质预测器的成果。而我认为最重要的、我们最近开源并发了博客文章的，是我们新的分子动力学框架，名字也叫 CUSP，容易混淆。它是通用粒子模拟器。我们和英伟达一起做这个的原因是，随着我刚才提到的这些机器学习力场的新进展——这些神经网络取代了力的量子力学计算——你没法在现有的 MD 模拟器里高效地运行它们，也就是那些让材料或分子随时间演化的模拟器，因为你需要在 GPU 上运行这些神经网络。而通常这些模拟器不在 GPU 上跑，更多是在 CPU 上跑。而且你还想并行运行。所以我们团队构建的是一个框架，你可以把这些力场编译成 JAX，它是基于 Python 的，然后它就能非常高效地运行这些 MD 模拟器。你还可以并行运行，这样就能用上你的 GPU，GPU 利用率很高。我们几天前、其实是一周前在 ICML 上开源发布了这个。反响很热烈。

So myself, I have one day at the university, and of course I still work in that day with students, and there we publish everything. I'm very interested in all sorts of things which have to do with AI for science. So definitely yes. But also CUSP actually publishes. So we have interns that work with our scientists where we publish the results. So we've, for instance, recently had results on machine learning force fields with uncertainty prediction, property predictors. And the most important thing, I think, that we've recently released open source and published a blog post about is our new molecular dynamics framework called CUSP, confusingly. So that's universal particle simulation simulator. And the reason why we did this, together with Nvidia, is that now with this new development of these machine learning force fields that I talked about, these neural networks that replace quantum mechanical calculations for the forces, you cannot run them very efficiently in the current MD simulators, the simulators that evolve a material or a molecule forward in time, because you need to run these neural networks on GPUs. And typically these simulators don't run on GPUs, more on CPUs. And you want to run things in parallel. And so what was built by our team is a framework where you can compile these force fields into JAX, which is Python based, and then it will actually very efficiently run these MD simulators. And you can run them in parallel as well, so that you use your GPUs, the utilization of your GPUs is high. And so we released this open source a few days ago actually, or a week ago, at ICML. And yeah, we got a lot of excited responses to that.

那分子的表示方式，这个领域的人是不是已经有标准的表示格式，然后你们的工具直接在这些现有格式上工作？

Are the representations of the molecules, like, are there standard formats for representing these things that someone working in the space would already have, and then your machinery just works on those existing ones?

这些表示，你可以把它想成化学领域的基础模型。你要做的是——这也是我们和 Meta 一起做的工作——你自己训练一个这样的力场。你取一个分子结构，基本上就是原子的位置和它们的类别，比如是氧还是氢。当然你希望它是等变的，因为旋转和平移无关紧要。然后你把它映射到一个潜空间，在那里它们被一些有意义的编码表示出来。如果你做的是机器学习力场，你就会从那里预测力和能量。但你也可以停在那个中间层，这样你就得到了那个特定材料的表示，你可以在非常广泛的材料和化学范围上训练它，从这个表示出发你可以构建性质预测器，可以用它来条件化你的生成模型。这个基础模型有各种各样的用途。但它也是为特定材料类别蒸馏模型的起点。

The representations, you can think of that as maybe a foundation model for chemistry. So what you do is, and this is also work that we did together with Meta, you train yourself one of these force fields. So you take a molecular structure, that is basically the position of the atoms and their class, like if it's oxygen or hydrogen. And of course you want this to be equivariant, because rotations and translations don't matter. And then you map this into a latent space where they get represented by some code that is meaningful. And if you do a machine learning force field, you will then actually predict the forces and the energies from that. But you can stop at that intermediate level, and then you have a representation for that particular material, and you can train this on a very wide range of materials and chemistry, and from that representation you can build property predictors, you can use it to condition your generative models. There's all sorts of uses for that foundation model. But it's also the starting point for distilling, let's say, models for specific material classes.

那说清楚一点，这些基础模型是按分子训练的，还是按类别训练的，还是范围非常广？

And to be clear, are these foundation models trained per molecule, or per class, or are they very broad?

这是个好问题。基础模型几乎按定义就是在非常广泛的材料类别上训练的。最好的数据集是 Materials Project 数据集和 Meta 的 Mol。他们生成了大量的 DFT 计算，也就是这些量子力学计算，来创建那个数据集，而那些计算被用来训练我刚才说的这些力场，或者说那些表示、那些基础模型。然后如果你说，我其实对某个特定材料感兴趣，那你可以做的是从那个非常广泛的表示、这个基础模型出发，然后针对那个特定材料类别微调这个模型。这样它就专门化于那个材料类别。但这样它也很快，因为你需要这些东西非常快，因为在分子动力学模拟里你必须调用它们很多很多次。因为 MD 模拟器里的一小步是飞秒，是非常小的一步，所以你必须调用很多次才能有任何进展。

Yeah, so that's a great question. So a foundation model almost by definition is trained on a very broad class of materials. The best data set for that is the Materials Project data set and Mol from Meta. So they have generated a very large number of DFT calculations, these quantum mechanical calculations, to create that data set, and those calculations are being used to train these force fields that I talked about, or those representations, those foundation models. And then if you say, but I'm actually interested in this particular material, what you can then do is start from that very broad representation, this foundation model, and then fine-tune that model for that particular material class. That way it is specialized for that material class. But that way it's also very fast, because you need these things to be very fast, because in a molecular dynamics simulation you have to call them many, many times. Because one little step in an MD simulator is a femtosecond. It's a tiny step, and so you have to call them many times to make any progress.

你前面提到了蒸馏。蒸馏就是在这里起作用的吗？你是想得到一个更小、更聚焦于你关心的分子的模型。

And you mentioned distillation earlier. Is that where distillation comes in? You're trying to get to a smaller model that's more focused on the molecules that you care about.

没错。蒸馏就是这个意思。你把大模型蒸馏成小模型。

Absolutely. That's what distillation is called. You distill the big model into a small model.

那在这种情况下，微调的使用和微调 LLM 的过程有多类似？比如用轨迹、基于语言的轨迹做强化微调。你们这里微调用的数据集是什么，微调的过程又是什么？

In this case, the use of fine-tuning, how analogous is it to the process of fine-tuning an LLM? You know, reinforcement fine-tuning with traces, language-based traces. Like what is your data set that you're fine-tuning and your process that you're fine-tuning with here?

其实它和 LLM 关系很近。事实上我们用的模型和 LLM 非常相关。你可以基本上把序列、你拥有的信息序列化，然后从那里映射到这个潜空间。通常你可以用图神经网络，它实际看的是三维结构，或者你可以把它当作 LLM，用序列版本。分子经常被表示成一个叫 SMILES 字符串的字符串，这样它就变成一维字符串，你也可以用它来表示分子。所以你可以同时用 LLM 风格的模型和图神经网络风格的模型来做这件事。

So actually it's quite related to an LLM. In fact the models we use are very related to an LLM. So you can basically think of the sequence, the information that you have, you can sequentialize it, and then from there you can actually map it into this latent space. So there's either, typically you can do like a graph neural network, which actually looks at the three-dimensional structure, or you can use it as an LLM, like a sequential version. Often molecules are represented as a string called a SMILES string, so then it becomes a one-dimensional string and you can use that to represent the molecule as well. So you can use both LLM style models as well as graph neural network style models for this.

所以你的基础模型可能和 LLM 类似，但在表示分子方面你有一个非常独特的分词器。

So your base model might be similar to an LLM, but you've got a very unique tokenizer in the case of representing molecules.

还不止如此，因为你基本上还可以看语言、图和分子的组合。

It goes even further because you can basically also look at combinations of language and graphs and molecules.

所以你可以从有大量文本的文献中学习，然后每当提到一个分子时，你实际上可以使用分子表示，也就是一个表示它的小图神经网络，这样它就成了一个 token。你用图神经网络把它变成一堆 token，这样你就真的可以开始把这些东西结合起来。你认为你在 CUSP 所做的工作未来会走向何方？近期、中期和更长远来看，你对这项工作最兴奋的是什么？

So you can sort of learn from the literature where there's a lot of text, and then every time a molecule is mentioned, you can actually use the molecular representation, which is a little graph neural network that represents it, so that becomes a token. You turn this into a bunch of tokens using your graph neural network, and so you can really start to combine these things. Where do you see the work that you're doing at CUSP going in the future? What are the near, mid, and longer term things that you're most excited about with that work?

好的。我们一直在扩展不同材料类别所需的工具。我们知道如何训练这些工具，但每新增一个材料类别都必须重新训练。我们也在往上游走，进入更粗粒度的表示，比如更大尺度的数字孪生，一直到对材料所在的实际反应器或设备进行建模。目前对我们最重要的努力是将平台与自驱动实验室连接起来，真正从自驱动实验室生成大量数据。这很重要。而且我认为最终非常重要的是走完整个流程：预测分子、在实验室中制造、放大材料，然后真正把它放进实际设备中，让客户对那种材料感到兴奋并愿意为之付费。对吧？所以这基本上意味着 scaling 的最后这部分仍然需要完成。但尽可能快地达到那里，我认为绝对是关键。所以我非常期待发现一种独特的材料——它不可能由 AI 完成——并把它做成实际的半导体器件或类似的东西，或者一种新的太阳能电池。我们可以说，AI 确实帮助了，是这种特定材料设计中的一个非常重要的部分。我认为这是一个非常令人兴奋的证明点，将推动整个行业前进。

Okay. So we are constantly expanding the tools that we need for different material classes. We know how to train these tools, but every new material class has to be retrained. We're also going up the stack, so we're going into more coarse-grained representations, like larger-scale digital twins, all the way up to modeling the actual reactor or device in which this material is sitting. The effort that's most important for us right now is connecting the platform to self-driving labs, to really generate large amounts of data from the self-driving lab. That's important. And I think ultimately it's very important to go through the entire process of predicting the molecule, making it in the lab, scaling the material, and then actually putting it in an actual device and getting a customer excited about that particular material and willing to pay money for it. Right? So this basically means that this last part of scaling is something that still needs to be done. But getting there as quickly as possible is, I think, absolutely key. So I'm really looking forward to discovering a material that is unique—it could not have been done by AI—and that makes it into an actual semiconductor device or something like this, or a new solar cell. And we can say, you know, AI actually helped, was a very important piece of the design of this particular material. I think that's a very exciting proof point that will help the entire industry forward.

你还在写一本书，书名很有趣。它讲的是生成式 AI 和随机热力学。那是什么？生成式 AI 和热力学之间有什么联系？请告诉我们一些你写这本书的方向。

You also have a book that you're working on that has a really interesting title. It's talking about generative AI and stochastic thermodynamics. What is that? What's the connection between generative AI and thermodynamics? And tell us a little bit about the direction you're taking with the book.

好的。也许先讲一点历史。我大约两年前开始这个，甚至在 CUSP 之前。所以在我停止为微软工作和创办初创公司之间有一段空闲。中间大约有六个月。当时我和妻子在意大利度假，而不是——你知道，我完全不能什么都不做，所以早上我……

Right. So maybe first a bit of history on this. I started this almost two years ago now, even before CUSP. So I had a bit of a lull between when I stopped working for Microsoft and the startup I started. So there was about six months between that. And we were on vacation—my wife and I were on vacation in Italy—and instead of, you know, I cannot do absolutely nothing, so in the mornings I...

我就写本书吧。

I'm just going to write a book.

是的。我在阳光下喝着卡布奇诺，然后开始写书。太棒了——这是你能有的最好的假期。下午我们会徒步穿山越岭，享受生活。然后我在一个叫 Maenberg 的小镇教课——这是荷兰语词，但人们可能说 Musenberg——它靠近南非的开普敦。那里有非洲数学科学研究所，所以我有幸在那里待了几周，讲授我正在写书的一些想法。然后我招募了两名学生帮我实际完成它，因为开始容易，完成难，你需要一些帮助。所以又花了两年才真正完成。这是巨大的工作量。但这本书是关于什么的？这本书是关于描述现代生成式 AI 的数学，包括概率模型、扩散模型和许多其他东西。这种数学结果等价于描述现代非平衡统计力学或热力学的数学。

Yeah. I had my cappuccino in the sun and I would start a book. So wonderful—the best vacation you can have. And in the afternoon we would hike through the mountains and enjoy life. And then I was teaching a course in a town called Maenberg—which is a Dutch word but probably people say Musenberg—but it's close to Cape Town in South Africa. There is the African Institute for Mathematical Sciences, and so I had the pleasure of spending a couple of weeks there teaching from these ideas that I was writing a book about. And then I recruited two students to help me out actually finish it, because starting is easy, finishing is hard, and you need some help. And so it took two more years to actually finish it. It's a huge amount of work. But what is the book about? So the book is about the mathematics that describes modern generative AI, including probabilistic models, including diffusion models and many other things. That mathematics turns out to be equivalent to the mathematics that describes modern non-equilibrium statistical mechanics, or thermodynamics.

这个说法有多具体？这个说法是，你知道，归根结底一切都是概率密度函数，还是这个说法更具体和实在？

And how specific is that statement? Is that statement, you know, everything is a PDF at the end of the day, or is that statement more specific and concrete?

它出奇地具体，我说出奇是因为我认为可能没有多少人把它当作类比。所以我认为这是一个非常好的问题，因为我认为这要深刻得多。在我看来，这不仅仅是类比。我认为有一个非常非常深的联系，这与我们最终谈论的是信息论有关。对吧？物理学的核心是信息论，机器学习的核心也是信息论。两者都被描述为——所以如果你谈论有信息损失的信息论，换句话说，随着时间演化你会丢失信息的过程。有一个观察者试图描述一个过程，但有太多自由度需要跟踪，你做不到。所以发生的是你在丢失信息，你需要用概率来捕捉它。这种数学是热力学和机器学习的核心。我认为这就是核心陈述。还有熵这个概念，在这两个理论中可能都很有趣，所以也许可以稍微谈一谈。熵基本上是你对某个特定状态感到的惊讶。你知道精确状态，还是你只有那个特定状态的一个非常模糊的概率分布？这是真的。但在物理学中，你认为熵可能是系统的一种可测量的东西，因为你实际上必须从能量中减去熵才能得到自由能，也就是你可以自由地用来在世界上做有用功的能量。所以它感觉像一个真实的东西，而不是——我不知道——关于世界的东西。在贝叶斯统计中，这正是它的样子：你谈论主观统计，因为它描述了我对世界不知道的东西，我需要对我不知道的世界上的事物赋予概率。

It's surprisingly specific, and I say surprisingly because I don't think many people think of it as an analogy. So that's why I think it's a really good question, because I think this runs much deeper. It's more than an analogy in my mind. I think there's a very, very deep connection, and this has to do with the fact that we are talking about information theory in the end. Right? At the core of physics is information theory, and at the core of machine learning is information theory. And both of these are described—so if you talk about information theory with loss of information, in other words, processes where you lose information as you are evolving over time. So there's an observer that tries to describe a process, but there are so many degrees of freedom to keep track of, you can't. And so what happens is that you're losing information, and you need to capture that by probabilities. And that mathematics is the core of both thermodynamics as well as machine learning. And I think that's the core statement. And there's this concept of entropy, which is maybe interesting in both of these theories, and so maybe it's good to talk a little bit about that. So entropy is basically the surprise that you find for a particular state. Do you know the precise state, or do you only have a very vague probability distribution for that particular state? And that's true. But in physics you think of entropy as maybe a thing you can measure of the system, because you have to subtract entropy from energy actually to get free energy, which is the energy you're free to use to do useful work in the world. And so it feels like a real thing, not like something—I don't know—about the world. In Bayesian statistics this is exactly what it is like: you talk about subjective statistics because it describes what I do not know about the world, and I need to ascribe probabilities to the things that I do not know about the world.

但在我看来，以及一些物理学家，特别是 E.T. Jaynes 是著名的一位，物理学中的熵也精确地描述了你对世界所缺失的所有信息。所以这两个领域之间有着非常深刻的联系。关于这一点，我想说的最后一件事是：有一个著名的 Landauer 定理，他基本上说，如果你从设备中擦除一个比特，你必须向环境辐射 k ln 2 的能量，这只是一个数字，作为热量。所以这里你可以看到直接的关系。我从芯片中删除一个比特，这是一块香农信息，他确定当你这样做时，你必须向环境辐射热量。所以你可以看到直接的某种信息守恒，类似于质量守恒，这在物理学中是非常基础的。

But in my view, and a number of physicists as well, in particular E.T. Jaynes is a famous one, entropy in physics also precisely describes all the information you're missing about the world. So there's this very deep connection between these two fields. Maybe the last thing I want to say about this: there is this famous theorem by Landauer who basically said that if you erase a bit from a device you must radiate k ln 2, which is just a number, amount of energy as heat to the environment. So here you can see the direct relationship. I'm deleting a bit, which is a piece of Shannon information, from a chip, and he determined that when you do that you must radiate heat to the environment. So you can see the direct kind of a conservation of information along the lines of conservation of mass, which is very foundational in physics.

是的。

Yeah.

所以事实上，热力学第二定律的表述是熵必须总是增加，这意味着你要么保持信息水平不变，即熵保持不变，要么在过程中丢失信息，这时熵就上升。所以当然，宇宙中的信息不会消失，而是从系统转移到热浴中，在那里你完全不可恢复地丢失了这些信息。所以，总之，这有点技术性，但想法是信息论是这两个理论的基础，这基本上使得数学非常相似，一个领域中开发的许多工具在另一个领域中有完全对应的类似物。这本书很大程度上是关于寻找这些不同的，这种两个领域之间的词典。对，在机器学习中有一个叫做随机归一化流的东西，然后在另一个领域有随机自由能估计，结果它们是完全相同的方法。

So in fact, the statement is the second law of thermodynamics says that entropy must always increase, which means you either keep the level of information the same, which is when the entropy stays the same, or you lose information in the process, and that's when the entropy goes up. So of course what happens is the information in the universe doesn't go away but it's transferred from the system to the heat bath where you've completely lost this information unrecoverably. And so yeah, anyway it's a bit technical but the idea is that information theory is behind both of these theories, which basically makes the math very similar and many of the tools which have been developed in one field have their exact analog in the other one. And this book is a lot about finding these different, this kind of dictionary between the two fields. Right, there's something called stochastic normalizing flows in machine learning, and then there is stochastic free energy estimation in the other field, and there turn out to be exactly the same methods.

那么最终这本书是给谁的，你认为这本书将如何改变他们看待控制的方式或他们能够做事的方式？

And so ultimately who is the book for and how do you see the book changing the way they view the control or the way they're able to do the things they do?

实际上对双方都是。这本书是为那些有兴趣了解一点热力学和非平衡热力学的机器学习者写的，我认为反过来也一样，它也是为那些想进入机器学习并建立在已有知识上的物理学家写的。我只想指出，关于扩散模型的第一篇论文的标题中实际上有“非平衡热力学”这个词。所以那篇论文的作者实际上已经知道这种联系。所以扩散模型确实是一个你取结构并破坏它的过程，这通常是世界上发生的事情。熵增加，然后我们试图在时间上逆转它，即从噪声开始并创建结构，这就是我们的生成模型。我认为它未来演变的方式，或者它可以被有效使用的方式，首先是它可以帮助物理学中的人使用这些工具进行机器学习。让我给你一个例子。计算自由能差是一个非常重要的问题，比如一个未结合的系统，就像一个蛋白质和一种你想要中和蛋白质的药物，所以你希望这种药物附着或结合到特定的蛋白质上，所以你想弄清楚它有多大意愿结合到蛋白质上的这个特定口袋。所以你需要计算未结合状态（这两个东西相距很远）和结合状态（它们在一起）之间的自由能差。这是一个非常重要的问题。所以这是化学家通常想要做的事情。但现在你可以使用现代机器学习方法，如扩散模型，来加速这些特定的计算。所以这是一个清晰的方式，你可以从机器学习方法开始，帮助化学家更好地进行计算。但反过来也是真的，因为随机热力学中已经开发出了一些东西，比如一个叫做反绝热驱动的概念，他们已经弄清楚了如何对特定物理系统进行非常有效的控制。这些概念现在可以用于扩散模型，以便更好地生成图像，因为事实证明，这对于降低扩散过程中的统计误差或噪声非常有帮助。所以它实际上可以帮助你构建更好的扩散模型。所以这是这两个领域之间的交叉 fertilization。

For actually both sides. It's written for the machine learner who is interested to learn a little bit about thermodynamics and non-equilibrium thermodynamics, and I think in reverse, right, it's also for the physicist who wants to get into machine learning and build on the things they already know. And I just want to point out that the first paper that was written about diffusion models actually had the word non-equilibrium thermodynamics in the title. So the authors of that paper actually already knew about this connection. So diffusion models really are a process by which you take structure and you destroy it, which is typically what happens in the world. The entropy goes up, and then we try to reverse that backward in time, which is to start with noise and create structure, which is our generative models. And the way I think it will evolve into the future is of course, or the way it can be used fruitfully is, first of all, it can help people in physics to use these tools for machine learning. And let me give you one example. So it's a very important problem to compute the free energy difference between, let's say, an unbound system, which is like a protein and a drug that you want to neutralize the protein, and so you want this drug to attach or bind to the particular protein, and so you want to figure out how much does it want to bind to this particular pocket on the protein. And so you need to compute the free energy difference between the unbound state where these two things are far apart and the bound state where they're together. That's a very important problem. And so it's something that a chemist would typically want to do. But now you can use modern machine learning methods like diffusion models in order to accelerate those particular calculations. So that's a clear way in which you can start from a machine learning method and help the chemist do their calculations better. But in the other direction is also true, right, because there are things which have been developed in stochastic thermodynamics like something like a concept called counterdiabatic driving is something that they have figured out on how to do very efficient control of particular physical systems. And those concepts can now be used in diffusion models in order to do a better job at generating images because it turns out that that's very helpful in bringing down statistical error or noise in the diffusion process. And so it can actually help you build better diffusion models. So this is a cross-fertilization between these two fields.

在机器学习方面，你认为这种类比或关系的主要目标是扩散模型，还是更基础的东西？

And do you see on the machine learning side the primary target of the analogy or relationship as being diffusion models or is it more fundamental than that?

这比那更基础。事实上，这本书深入到了变分自编码器。它深入到了 MCMC 方法，马尔可夫链蒙特卡洛方法，这些方法用于从特定分布中采样。它深入到了自由能估计。有许多不同的应用和交叉 fertilization 可以发生。

It's more fundamental than that. In fact, the book goes into variational autoencoders. It goes into MCMC methods, Markov chain Monte Carlo methods, which are used to sample from particular distributions. It goes into free energy estimation. There are many different applications and cross-fertilizations that can happen.

所以我认为作为一个领域，我们有点挣扎，你知道，我们正在创建这些越来越复杂的基础模型。我们对它们的理解，你知道，在机制方面仍然有些表面。你知道，热力学作为一个领域要成熟得多。你认为我们能够利用这种关系来更好地理解模型的核心吗？

So I think as a field we struggle a little bit with, you know, we're creating these ever more complex foundation models. Our understanding of them, you know, is still somewhat surface level like in terms of their mechanics. You know, thermodynamics as a field is much more mature. Do you see us being able to use this relationship to better understand the models kind of at their core?

绝对可以。至少它提供了一种不同的视角来尝试理解事物。所以我会说，随机热力学领域本身实际上相当新，有趣的是。所以它仍然是一个非常活跃的研究领域。热力学是平衡系统。随机热力学是非平衡系统。所以这实际上是一个相当新的领域。但它给你一种不同的方式来尝试理解扩散模型如何工作，因为你知道，像热、功和熵产生这些概念，我们在谈论机器学习模型时并不使用，但我认为它们给你一个非常新的和有趣的视角来思考发生了什么以及如何改进它们。

Absolutely. At least it gives a different perspective in trying to understand things. So I would say the field of stochastic thermodynamics is actually itself quite new, interestingly. So it is still very actively researched. So thermodynamics is systems in equilibrium. Stochastic thermodynamics is systems out of equilibrium. So that's actually quite a new field. But it gives you a different way to try and understand how diffusion models work because, you know, concepts like heat and work and entropy production, these things we don't use when we talk about machine learning models, but I think they give you a very new and interesting perspective on how to think about what's going on and how to also improve them.

反过来也一样，对吧？机器学习者有自己思考如何改进模型的方式，这也能帮助物理学家思考问题。我发现双方其实都对彼此的工作非常感兴趣，这很好。也许再进一步、更抽象一点：你最近在 ICLR 做了一场主题演讲，除了讲你在 CUSP 的工作和那本书，你还提出了另一个类比——我们姑且称之为从物理世界学习的机会——那就是研究波，并把它应用到机器学习和随机系统上。谈谈那项工作吧。

And in reverse it's also true, right? Machine learners have their own way of thinking about how to improve models, and that could also help physicists think about problems. I found both parties actually to be very interested in the other party's work, so that's good. Maybe to take the next step and get even more abstract: you recently did a keynote at ICLR, and in addition to talking about your work at CUSP and the book, you put forth another analogy—let's call it an opportunity to learn from the physical world—and that is in looking at waves and applying that to machine learning and stochastic systems. Talk a little bit about that work.

是的，这非常令人兴奋。我们认为需要在科学中更多地思考波，有两个原因。首先，现在在大脑里确实能观察到波，因为我们已经从单电极测量发展到几百甚至几千个电极的测量。当然，用单个电极你看不到移动的波、行波，但如果你放入几百或几千个电极，你就能真正看到这些行波，而且人们现在到处都观察到了它们。那么问题就变成：这是有功能的，还是只是某种副作用？这是第一个原因。另一个原因是，在神经网络中有一种叫过度平滑的现象，意思是当你从一条信息出发、有几千层时，这条信息在经过这些层时基本上会被指数级抑制。于是神经网络的输入和输出变得彼此独立。当然，这是你绝对不想要的，因为你希望输出能说明输入的一些信息——比如这个特定输入属于哪一类。但要确保信息穿过几千层一路从输入层传到输出层，一直有点困难。人们用了很多技巧来做到这一点，但我觉得我们在深度学习里开发的许多技巧，正是为了做到这一点。另一个应用是在推理中。如果你要推理一个问题，有时你必须把东西保持在记忆里。当然，我们可能把它写到短期记忆里，然后从这种记忆中读写。它是一个更稳定的部分，东西不会那么快混在一起。所以你需要某种记忆。基本上问题就变成：你如何向时间深处传递信息，或向时间深处推理，或者如何与你大脑中很远的东西通信，或者从具有很长程相互作用的材料中获取信息。而且在材料中，材料利用波来进行长程相互作用，这些波叫声子。所以声子是晶格振动，它们在材料中远距离携带信息。事实上，如果你想想宇宙，我们能看得很远、看进宇宙深处的唯一原因，就是光波击中了我们的仪器和眼睛，对吧？那又是波在远距离进行通信。所以现在有趣的是，在神经网络中，我们根本不用这个工具。我们只是创建映射，映射把数字映射到新的数字。于是问题就变成：我们能不能真正开始利用这个现象，这个我们在大脑中看到的波现象——我们能不能也开始在神经网络中使用它？这就是我们一直在开发的工作，尤其是——我们基本上设计了能自然产生这些波的神经网络，在我们称之为通道或记忆通道（你也可以叫它们胶囊）中移动这些信息，信息在其中被移动并保持稳定。我们在例如需要记忆的任务上非常成功。比如一个神经网络或 RNN 的任务——基本上任务是：你拿一个数字，把它保持在记忆里，然后在某个随机的其他时刻又来了一个数字，你把它保持在记忆里，然后把它们相加，然后在另一个随机时刻要求你给出和，而那个时刻可能在很远的未来。所以你必须做计算，还必须把东西保持在记忆里。而这些自然利用这些波的模型，它们在这些测试上实际上能比 RNN 做得好得多。

Yeah, so this is very exciting. There are two reasons why we thought we needed to think more about waves in science. First of all, waves are actually seen in the brain now, because we've gone from single-electrode measurements to maybe hundreds or thousands of electrode measurements. Of course, with a single electrode you cannot see a moving wave, a traveling wave, but if you put hundreds or thousands in, you can actually see these traveling waves, and people have now observed them everywhere. Then the question becomes: is this functional, or is it just a side effect of something? That's the first reason. The other reason is that in neural networks you have this phenomenon called over-smoothing, which means that you start with a piece of information and you have thousands of layers, and this information is basically exponentially suppressed as you go through it. So the input and the output of the neural network become independent of each other. Of course, that's something you really do not want, because you want the output to say something about the input—like maybe what's the class of this particular input. But it's always been a little hard to make sure that the information travels all the way from the input to the output layers through thousands of layers. People have used many tricks to do that, but I feel many of the tricks we have developed in deep learning are exactly to try to do that. And the other application is in reasoning. If you have to reason about a problem, you sometimes have to keep things in memory. Of course, we kind of write it maybe to short-term memory and then we read and write from this kind of memory. It's a more stable piece where things don't get mixed up so fast. So you need some kind of memory. And basically the question becomes: how do you communicate very deep into time, or reason very deep into time, or how do you communicate with things that are very far away in your brain, or take information from a material that has very long-range interactions. And also in materials, materials use waves for long-range interactions, which are called phonons. So phonons are lattice vibrations, and they carry information over long distances in a material. In fact, if you think about the universe, the only reason we can see very far and deep into the universe is because light waves hit our instruments and our eyes, right? And that is again waves doing the communication over these long distances. And so what's now interesting is that in neural networks, we don't use this tool at all. We just create maps, and the maps map numbers to new numbers. And so the question becomes: can we actually start using this phenomenon, this wave phenomenon that we see in the brain—can we also start to use it in neural networks? And so this has been the work that we've been developing, especially for—so we've basically designed neural networks that naturally create these waves, move this information around in what we call channels or memory channels, or you can call them capsules, where this information gets moved around and stays stable. And we've been very successful in, for instance, tasks that require memory. So a task where a neural network or an RNN—basically the task is you take a number, you hold it in memory, and then at some random other point another number arrives, you hold it in memory, and then you add them, and then at another random point you're asked to produce the sum, and that point could be very far into the future. So you have to do the computation and you have to hold things in memory. And these models that naturally work with these waves, they can actually do these tests much, much better than RNNs.

当我听到 RNN 和长期记忆时，我想到的是梯度爆炸之类的挑战，然后我想到波的振荡性质，它有一些内在的阻尼特性，让你能够访问更远未来的记忆，而不必面对这些梯度爆炸的挑战。这是这里起作用的机制的一部分吗？

When I hear RNNs and long-running memory, I think about challenges like exploding gradients, and then I think about the oscillatory nature of waves, and that has some inherent dampening properties that allow you to access memory further into the future without these exploding gradient challenges. Is that part of the mechanism at work here?

这是一个很好的观点。如果你把这个神经网络看作一个动力系统，你基本上把层看作时间，你试图从起点得到一个信号，并随着时间把它传播穿过各层。那么会发生几种情况。动力系统有不同的类型。第一种是稳定系统，基本上，如果你取两个不同的输入，它们映射到——过一段时间它们会坍缩到同一个点，然后从那一点开始向前移动。所以基本上一切都收敛到所谓的固定点，然后从那一点开始传播。那不是你想要的，因为信息会丢失，因为一切都被映射到一个点。如果你愿意，那是一种信息流失的梯度。那是信息丢失的情况。另一个极端是，你取两个输入点——比如两张相似的图像——然后它们开始疯狂地四处移动，那叫混沌。所以如果它是一个非线性系统，它开始疯狂地四处移动，你也会丢失信息，但原因不同。信息还在那里，但你无法在数值上追踪它。你的数值精度太小了。而这实际上就是我们说熵在增加的意思：它基本上变得随机了。它是一个随机过程。我们都知道马尔可夫链是一个依赖于前一步的随机过程。它会丢失关于输入的信息，因为我们无法追踪它。那也不是你想要的。所以你真正想要的是位于中间的某个地方，一个不太稳定也不太不稳定的地方。那叫做混沌边缘。

That's a very good point. If you think of this neural network as a dynamical system, you basically think of the layers as time, and you're trying to get a signal from the beginning and propagate it through the layers over time. So a couple of things can happen. There are different types of dynamical systems. The first one is a stable system where, basically, if you take two different inputs, they map to—after a while they collapse onto the same point, and then from there on that one point moves forward. So basically everything converges onto what's called a fixed point, and then from there on that's what propagates. That's not what you want, because information gets lost because everything gets mapped to a point. That's an inloating gradient, if you wish. That's something where the information gets lost. There's another extreme where you take two input points—say two images that are similar—and then they start to wildly move around, and that's called chaos. So if it's a nonlinear system, it starts to wildly move around, and you also lose the information, but for a different reason. The information is still there, but you cannot track it numerically. Your numerical precision is too small. And that's actually what we mean when we say entropy is increasing: it basically becomes random. It's a random process. And we all know that a Markov chain is a random process that depends on the previous step. It will lose information about the input because we can't track it. That's also not what you want. So what you really want is sitting somewhere in the middle, somewhere that's not too stable and not too unstable. And that's called the edge of chaos.

人们发现，实际上表现最好的神经网络运行在混沌边缘。人们也发现，不出所料，大脑也运行在混沌边缘。我们发现，如果在神经网络中包含这些波，你就会非常自然地处于这种混沌边缘的状态，而不需要微调系统让它处于那里。所以很容易达到这个混沌边缘的状态。然后，也许如果有时间，我们可以谈谈我们如何做到这一点，这是通过自发对称性破缺实现的，但我觉得那是一个相当技术性的讨论。

And people have found that in fact neural networks that perform best operate at this edge of chaos. And people have also found that unsurprisingly the brain operates at the edge of chaos. And we found that if you include these waves in your neural network, you very naturally operate in this regime of edge of chaos and you don't have to fine-tune the system to be there. So it's very easy to get to this edge of chaos regime. And then maybe if we have time we can talk about how we do this, and this is done by spontaneous symmetry breaking, but that's a rather technical discussion I think.

那么什么是自发对称性破缺？

And so what is spontaneous symmetry breaking?

好的，这是物理学中一个非常有趣的现象。这是另一个来自物理学的相当深刻的例子，我们可以开始用它来理解神经网络，也就是你之前的问题之一。自发对称性破缺是指一个具有特定对称性的系统，比如完美的平移对称性。想想气体具有完美的平移对称性，或者液体更好。然后液体在较低温度下实际上会凝结成固体。你可以从水变成冰，对吧？如果你看冰，它有晶体结构。所以它现在对称性更少，因为连续的平移群现在变成了离散的平移群，你只能在晶格间距上平移。所以你实际上把对称性破缺成了更小的东西。物理学中有一个非常深刻的理论说，当这种情况发生时，当一个连续对称性破缺成离散对称性，或者破缺成更小的对称性时，你就会有了新的波状模式，它们可以在不使用任何信息或使用非常非常少信息的情况下传播。这些是声子或光，或者这些是可以传播很长距离的波。它们不衰减。它们不色散。它们保持形状。如果你从某个形状开始，那么这个形状可以只是平移和移动。所以我们认为这是这些行波的非常好的候选者。所以我们所做的是，我们创建了一个具有大对称性的神经网络。我们只是把它内置进去。它不是我们通常使用的旋转或平移。我们只是给每个神经元一些额外的维度，我们说存在某种对称性，然后我们生成随机权重，这会产生随机激活，如果你让权重足够大，你从中采样权重的分布足够大，原始对称性就会被破缺。这解释起来很复杂，但对称性被破缺了，现在你处于这个破缺对称相，你有这些戈德斯通模式或这些波，它们可以在不消耗任何能量的情况下传播。它们确实如此。所以当你创建那个时，你会发现它们，你得到这些波，你可以看到东西非常自然地振荡。然后想法是，你现在可以训练神经网络利用这些振荡，这些振荡基本上是免费的。它们在那里运行而不消耗任何能量。它们非常稳定。所以它们从神经网络的开头一直振荡到神经网络的结尾。你现在可以设计系统，使它具有这些波，并具有这种混沌边缘行为。所以从开头，神经网络的输入，通过这种称为自发对称性破缺的现象产生的波状模式，一直传播到神经网络的输出。所以在这里我们使用物理学的一个深刻结果作为神经网络的设计原则。

Okay, this is a very interesting phenomenon in physics. So here's another example of something quite deep from physics that we can start to use to understand neural networks, which is one of your previous questions. And so spontaneous symmetry breaking is where a system that has a particular symmetry, let's say perfect translation symmetry. So think of a gas as perfect translation symmetry, or a liquid better. And then the liquid at lower temperature will actually condense into, let's say, a solid. You can go from water to ice, right? And if you look at ice, it has a crystal structure. So it has less symmetry now because the continuous translation group has now been turned into a discrete translation group where you can only translate over the lattice spacing. And so you've actually broken the symmetry into something smaller. And there is a very deep theory from physics that says when that happens, when a continuous symmetry breaks into a discrete symmetry or breaks into a smaller symmetry, then you'll have new wavelike modes which can propagate without using any information or with very, very little information. And these are these phonons or light or these are the waves that can travel very long distances. They don't decay. They don't disperse. They hold a shape. If you start with a certain shape, then the shape can just translate and just move. And so we thought that's a really good candidate for these traveling waves. And so what we did is we created a neural network with a large symmetry. We just bake it in. It's not like rotation or translation that we typically use. We just give every neuron some extra dimensions and we say there is some symmetry, and then we generate random weights which generates random activations, and if you make the weights large enough, the distribution from which you sample the weights large enough, the original symmetry gets broken. And that's complex to explain, but the symmetry gets broken, and now you're in this broken symmetry phase where you have these Goldstone modes or these waves which can travel without using any energy. And they do. So you find them when you create that, you get these waves where you can just see things oscillate around very naturally. And then the idea is you can now train the neural network to make use of these oscillations which are basically for free. They're running there without costing any energy. They're very stable. So they oscillate all the way from the beginning of the neural network to the end of the neural network. And you can now design the system so that it has these waves and it has this edge of chaos behavior. So information from the beginning, the input of the neural network, propagates all the way to the output of the neural network through these wavelike patterns that are created by this phenomenon called spontaneous symmetry breaking. And so here we use something, a deep result from physics, as a design principle for neural networks.

嗯，我很喜欢你能够运用所有这些来自物理学的工具，把它们带入机器学习，并试图找到其中的关联以及我们可以利用它们的方式。

Well, I love that you're getting to play with all of these tools from physics and bringing them into machine learning and trying to find the correlations and ways that we can take advantage of them.

还有更多即将到来。我认为数学和物理学有一个非常丰富的领域，都可以被机器学习利用。但这很美，因为交叉 fertilization 是双向的，对吧？因为通过我们的平台，我们使用 AI 工具帮助科学家发现新材料，并通过使用 AI 加速他们的模拟工具。所以这真的是这两个社区之间健康的交叉 fertilization，我觉得非常令人兴奋。

There's many more to come. I think there's a very rich field of mathematics and physics that can all be leveraged by machine learning. But it's beautiful because the cross-fertilization goes in two directions, right? Because with our platform we are using AI tools to help the scientists discover new materials and accelerate their simulation tools by using AI. And so it's really a healthy cross-fertilization between these two communities which I find very exciting.

嗯，Max，很高兴和你叙旧。你有很多事情要做。你在很多不同的角度上工作。了解这些很有趣。期待保持联系。

Well, Max, it's been great catching up with you. You have a lot going on. You're working on a lot of different angles. It's been enjoyable to learn about them. Looking forward to keeping in touch.

嗯，谢谢你，Sam。很高兴再次和你交谈。

Well, thank you, Sam. It was a pleasure again to talk to you.

非常感谢。

Thanks so much.

我想，下次见。

See you next time, I guess.

下次见。

See you next time.