使用结构化状态空间高效建模长序列

好的，大家好，欢迎来到第 41 期五月 AI 小组交流。本周我们有斯坦福的 Albert Gu 来介绍他关于用结构化状态空间高效建模长序列的研究。Albert 是斯坦福大学计算机科学系的最后一年博士生，导师是 Chris Re。他主要研究用于机器学习和深度学习模型高级信号能力的结构化表示，重点包括结构化线性代数、非欧几里得表示和序列模型理论。非常感谢 Albert 今天加入我们。开始之前，你对提问方式有什么偏好吗？

Okay, hi everyone, welcome to our 41st session of the May AI Group Exchange. This week we have Albert Gu from Stanford here with us to present his research on efficiently modeling long sequences with structured state spaces. Albert is a final year PhD candidate in the Computer Science department at Stanford University, advised by Chris Re. He's probably interested in studying structural representations for advanced signal capabilities of machine learning and deep learning models, with focuses on structured linear algebra, non-Euclidean representations, and theory of sequence models. Thank you so much, Albert, for joining us today. Before we start, do you have any preference on how you want to take questions?

谢谢介绍。对于这次演讲，我不太确定正式程度，但我很乐意让对话和提问保持随意。时间上，这不是一个完整的讲座，所以我非常欢迎中途提问，我会注意时间，如果太长会暂停，也会在几个部分停下来留出提问时间。

Yeah, thank you for the introduction. For this talk, I think I'm not sure about the level of formality, but I'm very happy to have it casual in terms of the conversation and the questions. I think there's some time, it's not going to be a full hour talk, so I'm more than happy to take questions during it, and I'll watch the time in case it gets too long, and also pause a few times for potential questions during some sections.

好的，听起来不错。我们尽量让这次交流互动性强一些。话不多说，交给 Albert。

Okay, sounds good. Let's try to make this session as interactive as possible. Without further ado, let me hand it over to Albert.

谢谢。好的，这次演讲将介绍一种新的序列模型，称为 S4，即结构化状态空间。在本次演讲中，当我提到序列模型时，我们将它们视为由基本层组成的黑盒序列映射，其中每一层简单地接受一个输入序列并返回一个相同形状的序列。目前，我们将它们视为一维到一维的映射，但这可以很容易地转换为更高维的特征。许多序列模型已经开发出来，满足这个接口，特别是在深度学习的背景下。这些包括许多经典的深度学习模型，如循环神经网络（RNN）和卷积神经网络（CNN），以及许多更现代的模型，如 Transformer 或神经常微分方程。所有这些模型都满足相同的接口：它们将序列映射到相同形状的序列，即相同的长度和特征维度。然后，你可以很容易地将其中任何一个整合到深度学习模型中，只需使用标准架构，其中可以包含归一化层、其他线性或非线性激活函数以及残差连接。所以，所有这些的核心组件是核心序列模型，这就是我们将重点关注的。基于序列模型的通用深度神经网络架构可用于解决许多类型的问题，涉及多种类型的序列数据，从文本和音频等模态到图像和视频，再到一般的时间序列数据或生物信号，例如这里所描绘的。在本次演讲中，我将对不同类型的序列数据进行粗略区分。现在，机器学习中的许多现代序列建模侧重于文本等数据，我粗略地将其归类为离散序列，因为输入以离散令牌的形式出现。其他类似的数据包括图或 DNA 碱基对。相比之下，本次演讲关注的是大致更连续的数据，例如视频、时间序列或音频。所有这些的共同点是存在一个底层的时间概念，数据从中采样。因此，我将广泛地将这类数据称为信号数据，以区别于序列数据。粗略地说，信号可以定义为由底层连续物理过程生成的数据，包括这里的所有例子。本次演讲由两部分组成：第一部分介绍一种称为 HiPPO 的方法，它是 S4 的前身，是一种用于信号在线记忆的新概念框架，并导致了一种建模信号和序列的新方法。第二部分将是 S4，它直接建立在 HiPPO 之上，并具有许多重要属性，这些属性对于解决某些类型的序列建模问题非常有效。在进入技术细节之前，我将快速预览实验结果，以突出我们将看到的改进类型以及它的优势。这将说明我们希望用这些新模型解决的挑战类型。第一个总体挑战就是信号或一般时间数据，这些数据无处不在。一些例子包括音频波形、时空数据如视频、生物信号如心电图（在医学中有重要应用）、市场和金融数据、每个主要行业生成的所有时间序列日志，以及许多其他类型的科学建模问题。我们稍后将回到这些实验，特别关注一些生物信号数据，但现在我将只用一个例子来说明，即音频。音频实际上是最常见的数据类型之一，因为它只是原始声音，无处不在。为了说明，机器学习现在真的都是关于文本的，最近许多头条结果都是关于人们收集所有能得到的原始文本数据，创建大规模模型，这导致了非常令人印象深刻的结果，如 GPT-3。我不知道听众，但希望你们中很多人都听说过这个模型。相比之下，音频实际上比文本多出几个数量级的数据。例如，一个标记数据集的数据量就比用于训练那些大型语言模型的所有数据还要多，但你很少听到这个领域的基准测试。我认为部分原因只是音频非常具有挑战性，当前模型的表现比文本差得多。这里有一个具体的例子：我们考虑了一个非常通用且困难的音频生成设置，即使用完全无条件的自回归模型生成口语数字 0 到 9。这里的黄金标准是称为 WaveNet 的基线，这是它试图说出这些数字的声音。效果不太好。这是 S4 的结果，这些结果只是来自大约两个月前：一、二、三、四。所以这是一个非常具体的例子。在本次演讲中，我们将看到像 S4 这样的模型是如何为信号设计的，并且可以对此类数据具有显著优势。第二个运行中的挑战可以通过更仔细地检查音频来激发。音频如此困难的一个原因是它的采样率非常高。

Thank you. All right, so this talk will be about a new sequence model called S4, or structured state spaces. For the purposes of this talk, when I mention sequence models, we will think of them as a black box sequence map composed of primitive layers, where each layer simply takes an input sequence and returns a sequence of the same shape. For our purposes right now, we'll think of them as just being a one-dimensional to one-dimensional map, but this can be easily converted to higher dimensional features. Many sequence models have been developed that satisfy this interface, particularly in the context of deep learning. These include many classical deep learning models such as recurrent neural networks (RNNs) and convolutional neural networks (CNNs), as well as many more modern models such as Transformers or neural ODEs. All of these models satisfy the same interface: they map a sequence to a sequence of the same shape, meaning the same length and feature dimension. Then you can incorporate any of these into a deep learning model fairly easily, just by using standard architectures where you can include normalization layers, other linear or non-linear activations, as well as residual connections. So the core component of all this is the core sequence model, and that's what we'll focus on. This generic deep neural network architecture based on sequence models can be used to solve many types of problems with many types of sequence data, from modalities such as text and audio to images and videos, to general time series data or biosignals, for example, which is depicted here. In this talk, I'm going to draw a very rough distinction between different types of sequence data. Now, much of modern sequence modeling in the context of machine learning focuses on data such as text, and very roughly I'll classify this as being a discrete sequence because the input comes in the form of discrete tokens. Other types of data like this include things like graphs or DNA base pairs. In contrast, what this talk focuses on is data that's roughly more continuous, things such as video, time series, or audio. What's common to all of these is that there's an underlying notion of time from which the data is sampled. So I'm going to very broadly call this type of data signal data as opposed to sequence data. Roughly speaking, signals can be defined as data that's generated from an underlying continuous physical process, including all these examples here. This talk can be composed of two parts: the first part covers a method called HiPPO, which was the predecessor to S4, and it's a new conceptual framework for the online memorization of signals and led to a new method for modeling signals and sequences. The second part will be S4, which built right on top of HiPPO and has a lot of important properties that have been very effective for addressing some types of sequence modeling problems. Before I get into the technical stuff, I'll give a quick preview of experimental results to highlight the types of improvements we will see and what it's good at. This will illustrate the types of challenges that we'll hope to address with these new models. The first challenge overall is just going to be signal or general temporal data that I just defined, and this data is really everywhere. Some examples include audio waveforms, spatial temporal data like videos, biosignals like electrocardiograms, which have important applications in medicine, market and financial data, all time series logs being generated by every major industry, and many other types of scientific modeling problems. We'll return to these experiments later with a particular focus on some biosignal data, but for now I will just use one example to illustrate, which is audio. Audio is actually one of the most common types of data because it's just raw sound, it's everywhere. To illustrate, machine learning right now is really all about text, and many headline results recently have been about people scraping together all the raw text data they can get, creating massive models on them, and that's led to very impressive results like GPT-3. I don't know the audience, but hopefully many of you have heard of this model. In contrast, audio actually has orders of magnitude more data than text. For example, a single labeled dataset has more data than all of the data used to train those massive language models, but you don't hear about benchmarks in this domain nearly as much. I think part of the reason is just that audio is very challenging, and current models perform much worse compared to text. Here's a concrete example: we considered a very general and hard audio generation setting of generating spoken digits zero to nine using a completely unconditional autoregressive model. The gold standard here is a baseline called WaveNet, and here's what it sounds like trying to say these numbers. It's not very good. Here are results for S4, which were just from the past two months or so ago: one, two, three, four. So that's a pretty concrete example. In this talk, we'll see how models like S4 are kind of designed for signals in a way and can have significant advantages for this type of data. The second example of a running challenge can be motivated by examining audio more closely. One reason why audio is so hard is because it's sampled at such a high rate.

极高采样率下，每秒有 16000 个或更多样本。相比之下，大多数序列模型处理不了超过一千个左右的样本。为了说明这一点，过去一年有一个名为 Long Range Arena 的基准测试，衡量模型在一系列长程任务上的表现。目前最流行的序列模型——Transformer 是主要焦点，但尽管它们取得了许多成功，在长上下文上表现并不好。人们尝试了几十种变体，性能都差不多，实际上只比随机猜测好一点。相比之下，S4 是专门为长上下文设计的，因此在这个基准测试上取得了巨大进步，并且是第一个在一些非常困难的长序列任务上取得进展的模型。所以这里有个大问题，Albert。

Extremely high rate where a single second has 16,000 or more samples. Well, in contrast, most sequence models can't deal with more than a thousand or so. And to illustrate, there was a benchmark in the past year called Long Range Arena that measured the performance of models on a suite of long-range tasks. The most popular sequence models these days, Transformers, were the main focus, but despite their many successes, they don't do so well on long context. There were dozens of variants that were tried, and they all get to around the same performance, which is actually not much above random guessing. In contrast, S4 is explicitly designed to be effective on long context, which leads to a huge improvement on this benchmark, and it's the first model to ever make progress on some really difficult long sequence tasks. So big question here, Albert.

是的，之前的任务是生成过程，而在这个长上下文挑战中，它是分类任务还是什么类型的任务？这些都是分类问题，数据包括多种模态，比如文本、图像，还有一些符号处理之类的东西。

Yeah, so in the previous task that was a generative process, and in this long context challenge, is it a classification or what kind of task is it? These are all classification problems, and they're on data that includes several data modalities such as text, images, some sort of symbolic processing, stuff like that.

我明白了。那么你可以把 S4 既用作生成模型来实际生成序列？

I see. And so you can use S4 both as a generative model to actually generate sequences?

是的，很多序列模型——我再次定义序列模型为一个黑盒接口，它只是一个序列到序列的映射——其中许多可以以多种方式使用，既可以用于分类也可以用于生成。例如，Transformer 或 RNN 是类似的东西，满足相同的接口，也可以以多种方式使用。

Yeah, so a lot of sequence models—again, a sequence model I'm defining as a black box interface that's just a sequence-to-sequence map—many of these can be used in many ways, both for classification and generation. For example, Transformers or RNNs are similar things that satisfy the same interface and can be used in many ways as well.

明白了，谢谢。好的，现在进入技术部分。第一部分是关于 HiPPO。为了说明 HiPPO 的目标，我举了一些机器学习模型目前难以处理的数据例子，特别是时间序列。为了强调为什么这很困难，我将用一个运行示例来说明现代模型难以具备的一个基本能力，那就是移动平均，它可能是现代时间序列分析中最基本的方法。这张图描绘了指数移动平均（EMA），即蓝线。它的使用方式是作为一个固定的、不可学习的特征，通常是任何时间序列分析流程中的第一个预处理步骤。现在，本着机器学习和深度学习的精神，我们不想手动创建这些特征，而是希望从数据中自动学习这类东西。所以具体来说，这里有一个非常简单的具体任务：假设你有一个模型，你给它输入这个黑色信号，你能让模型预测 EMA（蓝线）作为输出吗？不幸的是，标准序列模型如注意力机制和卷积完全无法做到这一点。原因本质上是因为 EMA 具有无界上下文；它实际上是信号历史的加权平均，权重呈指数衰减，延伸到无穷远。相比之下，大多数现代模型如注意力机制或卷积具有有限的上下文窗口。有些人会想到 RNN 等其他模型，简短的回答是 RNN 在这方面比注意力和卷积好，但由于优化等经验问题，它们仍然不够好。我们会看到，本次演讲中介绍的方法将非常自然地适合这个任务，并且是更强的版本。回到 EMA，一种思考方式是它是信号整个历史的一个非常简单的总结。换句话说，它是一个状态 x，一个单一数字，总结了输入 u 的整个历史。它之所以有用，是因为计算简单：如果你得到新数据，你可以用这个加权平均在常数时间内更新 EMA。除了这个简单的例子，我认为这两个属性在概念上非常重要。例如，它们正是任何实时决策问题所需的属性。抽象地说，你甚至可以想象你的大脑是一个状态，总结了你生活的整个上下文，并在你获取新信息时不断更新。所以我认为这是一个相当普遍的重要问题，这也是 HiPPO 的直接灵感来源。在机器学习背景下，这个问题对我们的模型有直接影响，因为正如我提到的，它们在长上下文上表现不佳。例如，文本模型通常只有大约 100 到最多 2000 个 token 的上下文范围，而如果你想处理语音和音频数据，一个单词的序列长度就超过 10000，并且可以无限延伸。所以这就是我一直在思考的问题。我试图将长程记忆这个模糊的目标转化为一个更正式的数学问题。概念上的想法是，如果你能将过去压缩成一个更小的状态，并准确地记住它，那么你应该能够重建过去，然后我们可以尝试将其转化为一个技术问题。所以想法是，我们将在线观察输入信号，并尝试维护一个良好的表示，以便能够重建它。因此，在本节中，我首先将形式化这个想法，然后定义 HiPPO 并可视化它，最后讨论一些推广。首先，形式化这个想法。HiPPO 的想法是，我们再次尝试在线观察输入信号，并尝试在给定记忆预算的情况下尽可能好地编码它。具体来说，你可以这样想：假设在某个初始时间 t0，我们已经看到了部分输入，我们将尝试压缩这个输入。你可以做的是存储到目前为止看到的最佳近似。例如，我们可以创建最佳多项式近似，并写下该多项式的系数。那么多项式的次数或系数的数量就是记忆预算，我们希望在所有时间连续地这样做。因此，随着我们在稍后的时间 t1 不断看到更多数据，我们必须更新我们的最佳近似并写下新的系数。现在核心问题是：首先，你如何实际找到这些最优近似？其次，随着你不断看到更多信息，你如何高效地更新这个表示？这就是主要的概念性想法。

Gotcha. Thank you. Yeah, okay, so now I'll get into the technical portions. The first part will be about HiPPO. To motivate what HiPPO's goal was, I gave a bunch of examples of data that machine learning models currently struggle with, particularly things like time series. To highlight why this is hard, I'm going to use a running example to illustrate a very basic capability that's difficult for modern models, and that's the moving average, which is perhaps the most basic method in modern time series analysis. This figure depicts the exponential moving average (EMA), which is the blue line. The way it's used is that it's a fixed, non-learnable feature that's often the first pre-processing step performed in any sort of time series analysis pipeline. Now, in the spirit of machine learning and deep learning, instead of doing manual processing like creating these features, we really would like to be able to learn these sorts of things automatically from the data. So in particular, here's a very simple concrete task: suppose you have a model and you're feeding it this black input signal, can you predict the EMA or the blue signal as the output? Unfortunately, it turns out that standard sequence models such as attention and convolutions cannot do this at all. The reason is essentially because the EMA has unbounded context; it's actually just a weighted average of the history of the signal with an exponentially decaying weight that stretches back infinitely. In contrast, most modern models such as attention or convolutions have finite context windows. Some people wonder about other things like RNNs, and the short answer is that RNNs are better than attention and convolutions here, but they still aren't that good due to empirical problems with optimization and other things. So we'll see that the methods introduced in this talk will be very naturally suited for this and are much stronger versions. But going back to the EMA, one way to think about it is that it's a very simple summary of the entire history of your signal. In other words, it's a state x, which is a single number that summarizes the entire history of the input u. The reason why it's useful is that it's easy to compute because if you get new data, you can update the EMA in constant time using this weighted average. Beyond the simple example, I think these two properties are actually conceptually really important. For example, they're exactly the properties that you need in any sort of real-time decision-making problem. And really abstractly, you can even imagine that your brain is a state that's summarizing the entire context of your life and is constantly updating as you require new information. So I think that's actually a pretty general important question, and this was a direct inspiration for HiPPO. In the context of machine learning, this question has a lot of direct impact on our models because, as I mentioned, they struggle with long context. For example, text models have been shown to typically have a context range of about 100 to at most 2000 tokens, whereas if you want to deal with data such as speech and audio, a single word in speech is a sequence of length more than 10,000, and this can really stretch to unboundedly. So this is the question that I was thinking about. What I did was I tried to convert this vague goal of long-range memory into a more formal mathematical question. The conceptual idea is that if you can compress the past into a smaller state that's accurately remembering it, then you should be able to reconstruct the past, and we can then attempt to turn this into a technical problem. So the idea is that we're going to observe an input signal online and try to maintain a good representation of it that allows us to reconstruct it. So first in this section, I'm going to formalize this idea, then I'll define HiPPO and visualize it, and then talk about a couple of generalizations. So the first thing is to formalize this idea. The idea of HiPPO is that again we're trying to observe an input signal online and we're going to try to encode it as well as possible given a memory budget. Concretely, you might think of it like this: suppose at some initial time t0, we've seen part of the input and we're going to try to compress this input. What you can do is store the best approximation to what we've seen so far. For example, we can create the best polynomial approximation and write down the coefficients of that polynomial. So now the degree of the polynomial or the number of coefficients is the memory budget, and we want to do this continuously at all times. So as we keep seeing more data at some later time t1, we'll have to update our best approximation and write down the new coefficients. Now the central question is: first of all, how do you actually find these optimal approximations, and moreover, how can you update this representation efficiently as you keep seeing more information? So this is the main conceptual idea.

还需要做一些工作来进一步形式化。具体来说，我之前一直在讨论最优近似，但这实际上并没有明确定义。所以我们需要找到一个度量来指定近似的质量。例如，我们可以选择指数衰减度量，它表示我们更关心近似输入的近期历史而非遥远过去。这将与 EMA 相关联。有了这个度量，问题就或多或少是明确定义的。基本上，我们暂时需要将度量作为一个超参数或先验来选择。稍后我会讨论如何实际学习它，但现在我们先选定一个度量，比如指数衰减度量。然后我们需要选择一个多项式基，这样问题就完全定义了。你可以写出系数的闭式解，并弄清楚它们如何随时间演化。我将跳过推导细节，但最终你会得到一个闭式方法。我想强调的是，推导过程有一些技术上有趣的新想法，但我觉得最有趣和最重要的部分，就是这个关于在线压缩与重构的简单概念，以及如何用数学形式化它。这就是要点。

A little bit of work to formalize a little more. In particular, I've been talking about optimal approximations, but that's actually not well defined. So what we need is to find a measure that specifies the quality of approximation. For example, we can choose the exponentially decaying measure, which says that we care about approximating the recent past of the input more than the far past. This will relate back to the EMA. Given this, the problem is more or less well defined. Basically, we have to pick the measure as a hyperparameter or a prior for now. Later, I'll talk about how you can actually learn it, but for now, let's pick a measure up front, say the exponential decaying measure. Then we need to choose a polynomial basis, and then the problem is completely defined. You can write down the coefficients in closed form and figure out how they evolve through time. I'm going to skip the details of the derivation, but you end up with a closed form method. What I want to emphasize is that the derivation has some technically interesting new ideas, but the most interesting and important part, I think, is just this simple conceptual idea of online compression and reconstruction and how to formalize that mathematically. So that's the main point.

现在定义已经讲清楚了，通过一些可视化展示，事情会变得更加清晰。首先，让我非常正式地定义什么是 Hippo。我提到过问题在于我们正在编码。所以 x(t)将始终表示我们的系数向量。问题是，随着我们在输入 u 中看到更多数据，这个向量如何随时间演化。结果发现，它只是满足一个简单的微分方程。通过推导，你可以写出这个微分方程的闭式解，并给出其中涉及的转移矩阵的闭式公式。具体来说，这个常微分方程被称为 Hippo 算子，其中的矩阵被称为 Hippo 矩阵，它们有闭式公式。实际上，这个矩阵就是这样一个矩阵。它是一个极其简单的矩阵，属于一种特殊类型的结构化矩阵。所以它只是一个简单的公式。然后我们写出这个微分方程的闭式公式，这就是我们的系数随时间演化的方式。这个方程被称为 Hippo 算子，或高阶多项式投影算子，因为我们是在向高次多项式基函数上投影。

Now with the definitions out of the way, things will become a lot more clear with some visualizations of what it does. First of all, let me be really formal about defining what Hippo is. I mentioned the problem was that we're encoding. So x of t is going to represent a vector of our coefficients at all times. The question is how this evolves through time as we see more data in the input u. It turns out that it just satisfies a simple differential equation. By going through the derivation, you can write down the differential equation in closed form and write down closed form formulas for the transition matrix involved here. To be concrete, the ODE is called the Hippo operator, and the matrices in the operator are called Hippo matrices, which have closed form formulas. In fact, the actual matrix is this matrix. It's an extremely simple matrix, which is a special type of structured matrix. So it's just a simple formula. Then we write down a closed form formula for this differential equation, and that's how our coefficients evolve over time. This equation is called the Hippo operator, or the High Order Polynomial Projection Operator, because we're projecting onto high degree polynomial basis functions.

从视觉上看，可以这样理解。我之所以称它为算子，是因为它将一个函数映射到另一个函数。所以它是一个算子，将黑色的输入信号 u 映射到蓝色的系数集 x，其中每个 x_t 都压缩了输入信号 u 的历史。你可以在线计算 x，每次看到一个输入 u。所以黑线代表当前时间步。我们逐渐看到更多的输入，并更新我们的系数向量，这里用蓝色表示。我仅可视化了最佳多项式近似的最低阶四个系数。现在来看重构的样子。随着我随时间移动并更新系数，这些系数所定义的多项式实际上就是这条红线。所以它正如我们所愿地重构了输入。注意，在可视化中我只用了四个系数，但实际上我用了 64 个系数，而整个函数的长度是 10,000，所以我进行了大量压缩。这里有一张静态图展示了重构的效果。由于我在压缩，我无法完美重构输入。那么重构的质量如何？这取决于度量。图中的绿线是我们投影所基于的指数衰减度量。直观上，你可以看到红色重构线在近期非常准确，而在更远的历史中逐渐退化，但仍然保留了关于整个信号的一些粗略信息。这就是 Hippo。

Visually, the way to think about it is like this. The reason I call it an operator is because it maps a function to a function. So it's an operator that maps this black input signal u to these sets of coefficients x in blue, where every time x sub t compresses the history of the input signal u. You can compute x online as you see u, one input at a time. So the black line represents our current time step. We're gradually seeing more of the input, and we are updating our coefficient vector, which is depicted in blue here. I've visualized just the lowest order four coefficients of the best polynomial approximation. Now here is what the reconstruction looks like. As I move along through time and update my coefficients, the polynomial that those coefficients define, in a sense, is actually just this red line. So it is reconstructing the input just like we wanted. Note that we are using only four coefficients here in the visualization, but I'm actually using 64 coefficients, while the whole function was length 10,000, so I'm compressing it a lot. Here's a static image that illustrates the effect of the reconstruction. Because I'm compressing it, I can't perfectly reconstruct the input. So how good is the reconstruction? It depends on the measure. The green line in this figure was the exponentially decaying measure that we are basically projecting onto. Intuitively, you can see that the red reconstruction line is really accurate for the recent past and degrades farther out in history, but still maintains some rough information about the whole signal. So that is Hippo.

哦，有一个澄清。是否可以将 x 视为每个时间点的状态，然后红线本质上是试图根据当前状态重构信号？还是你也使用所有过去的状态来恢复？

Oh, one clarification. Is it fair to think about x as being the state at each time point, and then essentially the red line is trying to reconstruct the signal given the current state? Or do you also use all the past states to recover?

完全正确。所以重构仅使用当前黑线处的系数向量。每次我都只使用当前向量。在可视化中，我展示了整条蓝线，但在任何给定时间点，我只记住当前向量，其长度为 64。

That's exactly right. So the reconstruction is happening using only the coefficient vector at the current black line. Every single time, I'm using only the current vector. In the visualization, I'm showing the whole blue line, but at any given point in time, I'm remembering only the current vector, which has length 64.

这里我只可视化了四个分量，但总共有 64 个。利用这 64 个数字，我用红色重建了到目前为止看到的内容。

Here I'm only visualizing four of the components, but it has length 64. Using those 64 numbers, I'm reconstructing what I've seen so far in red.

在之前的图中，如果我只取一条蓝线——实际上是最低阶系数——叠加在函数上，你会发现它恰好与 EMA（指数移动平均）吻合。所以移动平均可以看作是零阶或低阶投影。

In that previous figure, if I take just one of the blue lines—actually the lowest order coefficient—and overlay it on the function, you can see that it turns out to exactly match the EMA. So moving averages can be viewed as order-zero or low-order projections.

另一方面，HIPPO 本质上是对此的一个非常强的推广，它解决了一个自然的数学问题，并免费得到了 EMA 之类的东西。这就是 HIPPO。

On the other hand, HIPPO is essentially a very strong generalization of this that solves a natural mathematical question and gets back things like the EMA for free. So that's what HIPPO is.

现在我来谈谈它的一些扩展。首先，一个自然的问题：我一直用指数测度的例子，但其他情况呢？事实证明，HIPPO 可以针对任何测度推导出来。

Now I'll talk a little bit about some extensions of it. First, a natural question: I've been using this example of an exponential measure, but what about other cases? It turns out HIPPO can be derived for any measure.

例如，这里有一个很自然的情况：如果我想沿着均匀测度重建呢？换句话说，我只关心在滑动窗口中记住函数的近期历史。这是可行的，你会得到一个不同的常微分方程。这是一个重建效果：仅用 64 个内存数字，我试图均匀地重建这个函数的最后 2000 个时间步，它做得相当准确。

For example, here's a pretty natural case: what if I want to reconstruct along a uniform measure? In other words, I only care about remembering the recent past in sliding windows of my function. This is possible, and you get a different ODE. Here's a reconstruction effect: using just 64 numbers in memory, I'm trying to reconstruct the last 2000 time steps of this function uniformly, and it's doing this quite accurately.

你甚至可以进一步推广，例如当测度随时间变化而不是仅仅滑动时。所以这里有一个非常通用的框架，可以做很多事情。其中很多内容是他们期刊论文的后续工作。我们证明了对于任何测度，都存在一个对应的 HIPPO 算子，其中 HIPPO 矩阵 A 和 B 依赖于该测度，并且你可以写出它们的闭式表达式。

You can generalize even further, for example, when the measure is changing over time instead of just sliding along. So there's a very general framework here that can do lots of things. A lot of this was in follow-up work to their journal paper. We showed that for any measure, there exists a corresponding HIPPO operator where the HIPPO matrices A and B depend on the measure, and you can write them down in closed form.

这很重要，因为它在测度和这些常微分方程之间建立了等价关系。这意味着我们不必预先选择测度作为先验，比如指数衰减的情况。实际上，仅仅通过学习这些矩阵 A 和 B，在某种意义上就等同于学习测度。

This is important because it draws an equivalence between measures and these ODEs. This means we don't have to choose the measure upfront as a prior, like the exponentially decaying case. Actually, just by learning these matrices A and B, it's in some sense the same as learning the measure.

更好的是，这些算子不仅总是存在，而且矩阵总是结构化的。之前我们看到，对于指数衰减情况，矩阵非常简单。一般来说，它们会更复杂，但它们满足我在更早工作中引入的一种结构。它们都以某种方式结构化，这意味着你可以在近乎最优的时间内计算这些随时间更新的量。

Even better, not only do these operators always exist, but it turns out that the matrices are always structured. Previously we saw that for the exponential decaying case, the matrix was extremely simple. In general, they're going to be more complicated, but they satisfy a structure that I introduced in much earlier work. They are all structured in some way, which means you can calculate these updates through time in nearly optimal time.

以上就是 HIPPO 的主要要点。回顾一下：我们受到这些非常基本但重要的性质的启发，即试图维护一个总结整个上下文的状态。我们将其形式化为一个相当直观的数学问题，并能够解析地求解。这产生了一类很好的方法，用于处理信号中的长上下文。

So those were the main takeaways from HIPPO. To recap: we were inspired by these very basic but important properties of trying to maintain a state that summarizes the entire context. We formalized this into a mathematical problem that was pretty intuitive, and we were able to solve it analytically. This resulted in a nice class of methods for addressing long contexts in signals.

我看到聊天里有个问题：这些算子可以用 z 变换表示吗？

I see a question in the chat: can these operators be expressed in terms of z-transforms?

我不太确定你在这里指的是什么。据我所知，z 变换类似于拉普拉斯变换的离散版本。我不确定你指的是这个还是另一个概念。

I'm not quite sure what you mean here. To my understanding, z-transforms are like the discrete version of Laplace transforms. I'm not sure if that's the one you're referring to or another notion.

是的，我就是在想这个。似乎你可以用函数的变换来表示指数衰减，但似乎很可能存在联系。

Yes, that's what I was thinking about. It seems like you can express exponential decay in terms of transforms of the functions, but there seems likely to be a link.

是的，我认为所有这些都有紧密的联系，并且相互关联。事实证明，在下一部分我讲 S4 时，会有一些困难的计算问题。为了计算某些东西，我基本上会通过拉普拉斯空间或频率空间。所以我实际上对这个方程进行 z 变换，在几个值上计算该变换，然后反变换得到 HIPPO 矩阵或某个东西。

Yeah, I think all these things have a tight link and are connected to each other. It turns out that in the next part when I talk about S4, there will be some difficult computational issues. To compute certain things, I essentially go through Laplace space or frequency space. So I actually take the z-transform of this equation and calculate that transform at several values, then invert it to get the HIPPO matrices back or to get a certain thing back.

听起来不错，有道理。好问题。

Sounds good, that makes sense. Great question.

我想在这里停下来，看看有没有其他问题。如果没有，那很好，因为这在数学上通常是一个相当复杂的框架，但希望可视化能帮助解释很多。

I wanted to stop here at the summary for any other questions. If there are none, that's great, because this is usually a pretty complicated framework mathematically, but hopefully the visualizations help explain it a lot.

我将进入下一部分。本节中我没有包括任何实验。我们评估的方式就是重建效果有多好。实际上，在机器学习模型中直接使用这个方法效果还不错，但真正有效的是以特定方式将其融入模型。这就是 S4。

I'll move on to the next part. One thing I didn't include in this section was any experiments. The way we evaluated this was just how good the reconstruction is. Actually, using this method in machine learning models did pretty well naively, but where it became really effective was when incorporated into a model in a particular way. That's what S4 will be.

首先，我将通过 HIPPO 来定义 S4，这是最初的动机。这里的动机非常简单。对于记忆，HIPPO 将输入信号（我们认为是 1D）映射到更高维的信号。问题在于我们将维度从 1 扩大到了 N，其中 N 是我们的内存预算或系数数量。通常这至少是 100 左右。

First, I'm going to define S4 through HIPPO, which was the original motivation. The motivation here is very simple. For memory, HIPPO maps an input signal (which we think of as 1D) to a higher-dimensional signal. The problem is that we've blown up the dimension from one to N, where N is our memory budget or the number of coefficients. Typically this is at least 100 or so.

我从事深度学习，我想把 HIPPO 融入深度学习模型，但这是个问题，因为你不能简单地堆叠层——维度会不断增加。解决这个问题的一个非常简单的动机就是再次降低维度。方法是采用一个非常简单的线性投影。

I work in deep learning and I wanted to incorporate HIPPO into a deep learning model, but this is a problem because you can't just stack layers of this—you keep increasing the dimension. A very simple motivation to fix this is to just decrease the dimension again. The way to do this is to take a very simple linear projection.

我们有一个状态 x，它是一个 100 维的向量，我们用一个点积（可学习）作用它，得到单个数字。这本质上是取蓝线的线性组合得到最终输出，即红线。然后我们加上原始输入的倍数，这可以看作是一个跳跃连接。这就是 S4 的完整定义：这两个方程。

We have a state x, which is a 100-dimensional vector, and we'll hit it with a dot product (which can be learnable) to get back a single number. This is essentially taking a linear combination of the blue lines to get the final output, which is the red line. Then we'll add a multiple of the original input, which can be seen as a skip connection. That is the entire definition of S4: these two equations.

第一个方程是 HIPPO 方程，它将输入映射到一个记忆它的状态。第二个方程只是将状态线性组合回单个输出。对于有工程背景的人来说，这个定义可能看起来很熟悉，因为这是一个著名的模型，称为状态空间模型（SSM），有时用这个简单的控制图表示。它们已经存在很长时间了。

The first equation is the HIPPO equation, which takes the input to a state that memorizes it. The second equation just combines the state linearly back into a single output. For those with an engineering background, this definition may look really familiar, because this is a well-known model called a state-space model (SSM), sometimes depicted with this simple control diagram. They've been around for a long time.

几十年来，比如著名的通用滤波器，在许多科学领域都有应用。我认为除了控制和统计学，它们在计算神经科学和许多医学问题中也很常用。这部分的核心主题是，我们将看到 SSM 是一个非常优雅且自然的模型，但以前在深度学习中并没有以这种方式使用过，原因我们稍后会看到，而 S4 将解决这些问题。现在，为了定义 S4，我们将其视为 SSM 的一个实例：这两个方程中我们会代入具体的矩阵值。尽管这个模型定义简单，但实际计算却很困难，需要新的思路和算法。因此，我在这部分的目标是让你相信这是一个非常优雅且基础的模型。首先，我会讨论 SSM 的一些通用属性，这些属性在机器学习和深度学习中具有很多好处，且独立于 S4。然后，我会展示这些属性如何带来相关的权衡，使它们在深度学习中表现不佳，而 S4 将解决这些问题。最后，我会展示几个真实世界的实验，证明 S4 在各种场景下的有效性。

Decades such as the famous common filter and used in many scientific fields. I think outside of controls and statistics, they're also pretty commonly used in perhaps computational neuroscience and many medical problems as well. Now what the theme of this part will be is that we'll see that SSMs are a really elegant and natural model, but they haven't been used in deep learning before in this way, and for underlying reasons that we'll see and that S4 will address. But for now, just to define S4 in terms of this model, the way that we'll define it is that it's just an instantiation of an SSM: these two equations where we'll plug in specific values of matrices. And although it turns out that although this model is simple to define, actually computing with it turns out to be difficult and will require new ideas and algorithms. So my goal in this section is to convince you that this is a really elegant and fundamental model. So first of all, I will talk about some general properties of SSMs that would have a lot of benefits in machine learning and deep learning that are independent of S4. Then I will show how those come with associated trade-offs that prevent them from being really good in deep learning, and S4 will solve those problems. Finally, I'll show several real-world experiments that show S4's effectiveness in a bunch of settings.

在第一部分，我将描述三种不同的思考 SSM 的方式，这些方式赋予了它们许多优良特性。这是 S4 前身工作中发展的理论，并具有具体的实证优势。第一个特性是 SSM 本质上操作于连续时间信号，而非离散时间序列。我的理解是：在机器学习中，我们通常使用序列模型，即从输入序列到输出序列的参数化映射。如果我们不是映射序列到序列，我提出了一个术语“信号模型”，表示从函数到函数或信号到信号的参数化映射。给定这样一个映射，你可以通过离散化输入和输出得到序列。因此，信号模型在某种意义上是序列模型的泛化，它们映射函数到函数，但通过离散化得到序列模型。思考 SSM 的第一种方式是，它们只是一个简单的参数化信号模型，参数是矩阵 A、B、C 和 D，将输入函数映射到输出函数。这就是模型的接口或 API。这个特性之所以重要，是因为即使我们在离散时间中工作，模型在某种意义上也理解底层的连续域。稍后我会通过实验具体说明这一点。

In this first part, I'm actually going to describe three different ways to think about SSMs which gave them a lot of nice properties. This was theory developed in the predecessor work to S4, and will have empirical concrete empirical benefits. The first property is that SSMs inherently operate on continuous time signals instead of discrete time sequences. Here's how I think about it: in machine learning, we usually work with sequence models, which I defined as a parametrized map from an input sequence to an output sequence. Now what if instead of mapping a sequence to a sequence, I coined this term 'signal model' to denote a parametrized map that maps a function to a function, or a signal to a signal. Given one of these maps, you can essentially discretize the inputs and outputs however you want to get back a sequence. So essentially, the upshot is that signal models are in some sense a generalization of sequence models where they actually map functions to functions, but by discretizing them you get back a sequence model. The first way to think about SSMs is that they are just a simple parametrized signal model where the parameters are the matrices A, B, C, and D, and they map an input function to an output function. That's it in terms of the interface or the API of the model. The reason this property is important is because even when we're working in discrete time, the model in some sense understands the underlying continuous domain. I'll show what I mean concretely by this later empirically.

下一个视角与 HiPPO 的原始动机相关，即在线计算。我们如何实际计算这个 SSM 的输出？一种方法是逐个处理输入，就像 HiPPO 在在线设置中所做的那样。这是一种循环计算，因为每次更新都可以从前一次高效计算得出。为了说明为什么这并非易事：想象我们正在处理一个很长的输入，当前时间步由竖线表示，我们只得到一个新的数据点，即一个输入数字，然后想计算下一个输出。这个输出依赖于整个输入历史，因此你可能会认为下一次计算会随着序列长度而扩展。但实际上，我们可以在常数时间内完成计算。这是大多数序列模型不具备的重要特性。例如，在 Transformer 或卷积中，如果以在线或自回归方式计算从输入到输出的映射，每次计算都会随上下文窗口的整个长度而扩展。SSM 之所以能如此高效，是因为它们是有状态的。在内存中，我们维护一个状态（蓝色部分），它是一个单一向量，总结了历史信息，并且可以非常高效地更新。这使得它们在各种在线设置中都非常高效。我们稍后会再次看到为什么这很重要，但有一个主要缺点：如果你不在在线设置中，这种方法是顺序的，因此很慢。那么，如果你实际上知道所有未来的输入呢？理想情况下，你不会一步一步地做，而是可以做一些更快、可并行的事情。这基本上是 RNN 的主要问题，也是它们最近在机器学习中失宠的原因，因为当一次性看到大量数据时，它们是顺序的且不可并行。

The next perspective relates back to the original motivation of HiPPO, which was about online computation. How do we actually compute the output of this SSM? One way is to process the input one at a time, just like HiPPO did in an online setting. This is a recurrent computation because each update can be computed efficiently from the previous one. To unpack why this is non-trivial: imagine we're processing a very long input and we're at the current time step, denoted by the vertical line, and we get just one more data point, a single number for the input, and we want to compute the next output. This output depends on the entire history of the input, so you'd expect the computation of the next one to scale with the length of the sequence. But actually we can compute it in constant time. This is a non-trivial property that most sequence models don't have. For example, in a transformer or a convolution, if you were to do this in an online or autoregressive fashion, computing the mapping from one input to one output, each computation will scale with the entire length of the context window. The reason SSMs can do this so efficiently is because they are stateful. In memory, we maintain a state, which is the blue thing, a single vector that summarizes the history and can be updated very efficiently. This makes them really efficient in any sort of online setting. We'll see again why this matters, but there's one main drawback: if you're not in an online setting, this is slow because it's sequential. So what if you actually know all the future inputs? Then you ideally wouldn't do this step by step and could do something faster and parallelizable. That was basically the main problem with RNNs and why they've recently fallen out of favor in machine learning, because they are sequential and not parallelizable when you see a lot of data at once.

这引出了最终的表示，即卷积表示，它允许并行化。思路是，不是从输入到状态再到输出，而是直接从输入到输出，绕过状态，在整个序列长度上并行完成计算。原因是 SSM 等价于卷积，其中从输入 u 到输出 y 的映射相当于将输入与一个特定的卷积滤波器（图中绿色所示）进行卷积。因此，要计算这个映射，只需做 y = u 与卷积核 k 的卷积。这可以使用已知技术非常高效地完成。对于实践者来说，我认为思考 SSM 最有用的方式可能是将其视为一个非常花哨的 CNN，只是以不同的方式参数化卷积核。值得注意的是，这个核可以是无限长的，这再次说明了为什么它非常擅长长程依赖。回到之前的例子，EMA 实际上就是一个单一的卷积，将输入与一个指数衰减的卷积核进行卷积。正如我提到的，尽管 CNN 也是卷积，但它们无法表示 EMA，因为 CNN 的窗口是有限的，而 EMA 的窗口是无限的。另一方面，SSM 确实可以表示无限长的卷积。事实上，有一种非常简单的方法可以直接将 EMA 写成一个 SSM。Chris 之前也提到过这一点。

That motivates the final representation, which is the convolutional representation, allowing them to be parallelized. The idea is that instead of going from input to state to output, you can go straight from input to output, bypassing the state and doing the entire computation in parallel over the sequence length. The reason is that SSMs turn out to be equivalent to convolutions, where computing the map from input u to output y is equivalent to convolving the input by a particular convolution filter, depicted in green here. So to compute this map, you just do y = u convolved with k for this convolutional kernel. This can be done very efficiently using known techniques. For the practitioner, I think the most useful way to think about SSMs potentially is as essentially a very fancy CNN, where you are parametrizing the convolution kernel in a different way. Notably, this kernel can be infinitely long, which again points to one reason why this is very good at long-range dependencies. Just to call back to this example again, the EMA is actually literally just a single convolution, where you convolve the input by an exponentially decaying convolution kernel. As I mentioned, although things like CNNs are also literally convolutions, they cannot represent the EMA because CNNs have a finite window and the EMA has an infinite window. On the other hand, SSMs do represent infinitely long convolutions. In fact, there is a very simple way to write down the EMA directly as an SSM. Chris kind of pointed to that earlier.

以上就是我想提到的 SSM 的三个特性。总结一下：首先，我们将它们视为操作于连续时间信号的映射。

So those were the three properties of SSMs that I wanted to mention. Just to recap: first of all, we're going to think of them as maps that operate on continuous time signals.

信号不仅仅是序列。如果你的模型部署在某个环境中，或者它实时或在线地看到输入，它可以高效地以循环方式计算这些。而如果你一次性看到整个输入，比如通常在训练时，你可以更高效地并行计算。我有个问题，Albert。这非常酷。我只是想知道，如果目标实际上是获得信号的表示，以便执行不同的下游任务，那么拥有状态空间表示而不是直接输出结果不是更好吗？在这种情况下，我们是否必须坚持使用 HiPPO 而不是转向 S4？

Signals not just sequences. If your model is deployed in a setting or it sees inputs in real time or online, it can compute these efficiently recurrently. And if you see an entire input at once, such as usually during training time, you can compute it even more efficiently and in parallel. I have a quick question here, Albert. This is super cool. I was just wondering, if the goal is actually to get a representation of your signal so that you can perform different downstream tasks, isn't it better to actually have the state space representation rather than directly going to the outputs? In that case, would we have to stick with HiPPO instead of going to S4?

好问题。实际上，没人问过这个。这是个很棒的问题。我的理解是，本质上我们有一个非常有意义的状态，而 SSM 中投影状态的第二部分就像是可学习的东西，它负责从状态中提取正确的特征。我提到过，到目前为止我所做的一切都是可学习部分，它在某种意义上使用了整个状态。而且我提到目前我只考虑一维输入和一维输出的情况。但实际上，在我们实际的深度学习模型中，会有多维输入和输出，我们会在每个维度上运行一个 SSM，每个 SSM 都会学习如何以不同的方式使用状态。所以本质上我们会有很多——你可以想象我们有一个单一的状态，但有很多很多可学习的输出，它们会从该状态中提取不同的特征。因此，我们会得到许多不同的特征，它们以各自的方式利用状态。

Great question. Actually, no one's asking that. That's a great question. So the way I think about this is that what's happening is that essentially we have this nice state which is very meaningful, and then the second part of the SSM that projects it is kind of like the learnable thing that's figuring out how to extract the right features from this state. Now, I mentioned that everything I've done so far is the learnable part that's actually using the entire state in a sense. And I mentioned that I'm only considering the one-dimensional case so far with 1D inputs and outputs. But actually, what's going to happen in practice in our actual deep learning models is that we'll have multi-dimensional inputs and outputs, and we'll essentially run an SSM on each one of them, and each one of these will learn how to use the state in a different way. So we'll have essentially many—you can think of it as maybe we'll have a single state but many, many possible outputs that are all learnable, and we'll extract different features from that state. So we are going to get a lot of different features that utilize the state in however they want.

我明白了。好的，谢谢。但抱歉，这些维度之间不是也有相关性吗？如果你独立地学习每个空间，难道不想也保留相关性吗？

I see. Okay, thank you. But sorry, but isn't it like all these dimensions also have a correlation? So do you also—if you learn the space independently, don't you want to also preserve the correlation?

我认为很多处理时间序列的人都会关心这个问题。在深度学习中，我们通常不考虑这个方面，而是直接扔进一个非常大的模型和许多独立的层。我认为在实践中通常发生的是，模型会学习为最终预测任务所需的一切，这通常确实涉及——我认为它最终确实会去相关，但具体动态并不十分清楚。这更像是深度学习理论中一个更广泛的问题，目前还没有很好的理解。我能说的是，我们已经在许多类型的噪声数据上使用了它，这些数据通常涉及——我稍后会讲到实验，但我们已经在许多类型的时间序列和其他噪声数据（如脑电图）上尝试过。但有一天，一维的，对吧？嗯，它可以在多个维度上工作，我刚刚提到了这一点，稍后我会再次提到我们是如何做到的。但你可以简单地在多个维度上使用它，它就能直接工作。

This is something that I think a lot of people working with time series are concerned with. And somehow in deep learning we don't normally consider that aspect, and we kind of just throw in a really big model and a lot of these independent layers. I think in practice what usually happens is that the model learns whatever it needs to do for the final prediction task, and this often does involve—I think it does end up decorrelating things, but it's not super clear exactly the dynamics of what happens. This is kind of a more broad question for deep learning theory in general that's not well understood right now. What I can say is that we've used this on many types of noisy data that usually involve—I'm going to get to experiments later, but we have tried this on many types of time series and other noisy data like EEG. But one day, 1D, right? Well, it can work on multiple dimensions, which I kind of just pointed to, and I'll mention again later how we do that. But yeah, you can just kind of do it naively on multiple dimensions and it just works out of the box.

好的，好的。在我们进入实验之前，我有一点关于 S4 如何建立在 SSM 之上的内容。为了提醒你，S4 只是一个 SSM，我们插入了一些基于记忆理论的公式，并且有特殊的算法来计算它。那么首先，为什么需要这些矩阵？嗯，SSM 最重要的部分是状态，正如 Nandita 敏锐地指出的那样。HiPPO 所做的本质上是计算一个非常特殊的状态，这个状态在数学上有意义，并以捕捉长程依赖的方式压缩输入的历史。所以基本上，只需将该公式插入到 SSM 中，它就能学习到一个更有意义的状态，从而使 SSM 更好地处理长程依赖。为了用实验说明这一点，这里有一个在序列模型标准基准上的简单实验。实际任务并不重要，但它是经过充分研究的。标准序列模型如 Transformer、CNN 和 LSTM 都能达到大约 60% 左右的准确率。那么如果我们使用 SSM 会发生什么？如果你天真地随机初始化所有参数（这是深度学习中通常的做法），它实际上表现得很糟糕。但如果我们只是插入这个公式呢？插入这个公式，甚至不需要训练矩阵，就能给 SSM 带来巨大的提升，从远低于基线到显著高于基线。实际上，我在这个消融实验中使用了非常小的模型，但完整的 S4 模型在这个数据集上达到了 90% 以上，这比所有其他序列模型高出 20 多个百分点。这说明了为什么 HiPPO 如此有用。

Okay, okay. So before we get to the experiments, I just have a little bit on how S4 builds on top of SSMs. Just to refresh your memory, S4 is just an SSM where we plug in certain formulas that were based on the theory of memorization, and we have special algorithms to compute it. So first of all, why are these matrices needed? Well, the most important part of the SSM is the state, as Nandita keeps insightfully bringing up. So what HiPPO did was essentially compute a very particular state that was mathematically meaningful and compresses the history of the input in a way that captures long-range dependencies. So basically, just by plugging in that formula into this SSM, it learns a more meaningful state that allows the SSM to address long dependencies better. So just to illustrate this empirically, here's a simple experiment on a very standard benchmark for sequence models. The actual task doesn't matter, but it's well studied. Standard sequence models such as Transformers, CNNs, and LSTMs all get to around the same accuracy of about 60-ish percent. Now, what happens if we use an SSM? If you use it naively by randomly initializing all the parameters, which is what you would typically do in deep learning, it actually does terribly. But what happens if we just plug in this formula? Plugging this in and not even needing to train the matrix gives a massive boost to the SSM, going from much below the baselines to substantially above the baselines. Actually, I used very small models for this ablation, but the full model S4 on this dataset gets over 90, which is something like 20-plus points better than all other sequence models. So that kind of illustrates why HiPPO is so useful.

快速问题：在这个例子中，我是同时插入 A 和 B 矩阵，还是只插入 A？

Quick question: in this example, do I plug in both A and B matrices, or just A alone?

A 是更重要的矩阵，但实际上，插入 A 和 B 本质上——它们都是固定的矩阵，由 HiPPO 算子指定。我只展示了 A，因为它更重要，但这个特定的实验特别冻结了这两个矩阵。

A is the more important matrix, but actually, yeah, plugging in A and B essentially—they're both fixed matrices which the HiPPO operator specifies. I've only illustrated A because it's the more important one, but this particular experiment froze both of these matrices specifically.

好的。嗯，人们有一个问题：我们总是冻结这些吗？实际上，我们也可以训练它们。这是为了说明即使冻结它们，它也表现得非常好。但在实践中，我们确实会训练它们，这会让它做得更好一点。

Okay. Yeah, one question people have is: do we always freeze these? And actually, we can train them as well. This was to illustrate that even freezing them, it does super well. But in practice, we do train them, and it makes it do a little bit better.

所以这是一点，它指向了——我提到过 SSM 以前没有以这种方式在深度学习中用过，这是一个问题：如果你天真地使用它，它不起作用，所以你需要这个新理论。第二个原因实际上是它们在计算上很难处理。所以这里有一个说明。再次提醒你，我们将 SSM 视为从输入信号到输出信号的参数化映射。现在假设我们的输入长度为 L，那么输入就是 L 个数字的序列。那么整个输出的也是一个长度为 L 的序列。理想情况下，计算这个映射需要 O(L) 时间或稍多一点。但问题是：SSM 通过状态将输入映射到输出，这个状态给了它们很多好的性质，但它也高了一百维。因此，通过状态计算端到端映射将需要比计算最终答案多一百倍的计算和内存。这实际上是一个真正的问题。

So that was one thing, and that kind of points to—I mentioned that SSMs have not been used in deep learning before in this way, and that's kind of one problem: if you do it naively, it doesn't work, so you need this new theory. The second reason is actually that they're computationally pretty difficult to work with. So here's how to illustrate. Again, to remind you, we're thinking of an SSM as a parameterized map from an input signal to an output signal. Now suppose that our input had length L, so our input would just give us a sequence of L numbers. Then the output of this whole thing is also a sequence of L numbers. Computing this map ideally takes around O(L) time or not too much more. But here's the problem: SSMs map the input to the output through this state, and that state gave them a lot of nice properties, but it's also a hundred dimensions higher. So computing the end-to-end mapping through the state will take a hundred times more computation and memory than what's needed to compute the final answer. And this is actually a real problem.

实际上你不需要计算状态；你可以用卷积来计算。但问题是，在计算卷积之前，我必须先计算卷积核或卷积滤波器（图中绿色部分），而计算这个核跟计算状态一样慢。这其实说得通，因为问题的计算难度并没有改变。所以无论你怎么做，它都会很慢且内存效率低。S4 的主要贡献在于，当 SSM 是结构化的时候，可以大幅减少计算。例如，使用 HiPPO 矩阵而非非结构化矩阵时，我们可以节省 100 倍的计算量，使 S4 整体上非常高效。这是通过一个特定算法实现的，我简单展示一下。基本上，我们处理这个 SSM，但只需要处理特定的结构化情况，比如某些特定的 HiPPO 矩阵。利用一些算法思想，我们发现可以非常高效地计算之前提到的卷积核，然后用卷积完成整个计算。这里我就不详细讲了。另外，最近我们也在开发模型的简化版本，可以绕过所有这些，用更简单的方式实现。所以希望几周后我们能发布一些成果，那时你就不需要担心这个复杂的算法了。好了，这就是我想讲的关于 S4 的技术部分。我在这里停下来，欢迎大家提问。

You don't actually have to compute the state; you can compute it using a convolution instead. But what happens is that before computing the convolution, I have to compute the kernel or the convolution filter in green, and computing that is just as slow as computing the state. This sort of makes sense because that hasn't changed the computational hardness of the problem. So essentially, no matter how you do it, it's going to be slow and memory inefficient. The main point of S4 was showing that you could substantially reduce this computation when the SSM is structured. For example, when using the HiPPO matrix instead of an unstructured matrix, we can save a factor of 100 and make S4 overall extremely efficient. This is done through a particular algorithm, which I'll just flash up. Basically, we're trying to work with this SSM, but we only need to work with specific structured cases, such as some particular HiPPO matrices. Using some algorithmic ideas, it turns out there is a way to compute the convolution kernel, which was depicted in green before, very efficiently, and then compute the whole thing using a convolution. I won't go into details here. I will also mention that recently we've been developing simplifications of the model that allow you to bypass all of this and do things much more simply. So hopefully in a few weeks we'll have some stuff out where you don't need to worry about this really complicated algorithm. All right, so that was the technical portion I wanted to mention for S4. I'll stop here for questions as well.

那么，如果你确实想要获取状态，S4H3 能恢复真实状态吗？或者它只是...我不知道这在任何用例中是否有用，但让我们看看。

So if in any case you want to actually get the state, can S4H3 recover real state? Or is it like... I don't know if that would be in any use case, but let's see.

哦，你是说如果确实需要这个状态，我可以用状态计算而不是卷积？是的，可以。实际上，这在一些实验中会用到。我想我之前没有明确说明，但你可以用两种方式计算：通过卷积或者通过状态。卷积在训练时有利于并行化，而状态在推理或部署场景中更有用，比如在线情况下，你可以逐步展开状态而不是用卷积。所以两种方式都可以，这很酷。

Oh, you're saying if there's a case where I actually want this state, can I do that instead of the convolution? Yes, you can. In fact, that will be used in some experiments. I guess I didn't mention explicitly, but you can compute it in either way: either through the convolution or through the state. The convolution is useful during training time for parallelizability, but the state is useful in some inference or deployment settings where perhaps you might be online and then you would actually be going through the state instead of the convolution, unrolling things one step at a time. So you can do either way, which is pretty cool.

我有个更偏思考的问题。假设我对两种不同的度量感兴趣，比如我想看指数平均如何工作，但我也想要... 这是否意味着我只需要创建一个新的度量，在把它接入 SSM 之前高效地组合它们？还是说 S4 基本上可以... 有没有两个独立的模块我可以...？

I had more of a thought question. Let's say I'm interested in two different measures, like I want to see how the exponential average works but I also want... So does it basically mean that I just have to create a new measure that combines this efficiently before I plug it into SSM? Or can S4 basically kind of... are there two independent blocks that I can basically...

我正要讲到实验部分。实际上，我现在就展示那张幻灯片。首先，实验会基于这类信号数据。我之前提到过几次，我们实际做的是定义了一个一维到一维的映射，但我会取一个多维输入，然后堆叠多个这样的副本。与此并行的是，你可以对这些副本做很多事情。所以回答你的问题，我最近开始尝试的一个做法是，对每个副本使用不同的度量，也就是不同的 A 和 B 矩阵。这可以解释为使用了多种度量。

I'm just about to get to the experiment. Actually, I'll get that slide right now. First of all, the experiments will be on this type of signal data. As I mentioned a couple times, what we actually do is that I've defined this 1D to 1D map, but I'm actually going to just take a multi-dimensional input and stack a bunch of copies of this. Now, as a parallel to that, you can do many things with these copies. So to answer your question, one thing that I've been starting to experiment with is just using different measures, or essentially different A and B matrices, for every copy. That sort of has an interpretation of using multiple measures.

我明白了。因为当 Iman 谈到不同维度之间的相关性时，比如你有一张图像，两个不同的像素实际上是相关的。所以我在想，你可以有一个度量来捕捉这种相关性，但也可以有另一个度量来捕捉时间上的相关性。另外，既然你提到了，我不知道你是否在图像空间上尝试过。我很好奇这种长卷积是否真的对图像任务有影响，因为通常在做图像分析时，理论上思考起来，局部特征和全局特征都很重要。但我不知道仅仅使用这种长表示是否会丢失一些局部特征。

I see. Because when Iman actually talked about the correlations between different dimensions, like let's say you have an image, two different pixels are actually correlated. So I was thinking you can have a measure that captures this correlation, but you can have another measure that captures it over time. Another thing, since you mentioned that, I don't know if you tried that on image space. I would be curious if this kind of long convolution actually makes any difference with image-based tasks, because usually when we do image analysis, theoretically when we start thinking about it, it seems that local features as well as global features are important. But I don't know if we are missing any local features by just using this kind of long representation.

好问题。实际上，我们已经在图像上做了更多实验，不过这次演讲没有包含。局部来看，我们发现卷积的局部偏置似乎效果不错。很难量化缺失的特征，但我认为在某些设置下，我们只是与标准局部 CNN 持平或略差。这很难说。不过我要提一下，你可以通过改变度量来强制引入局部性。例如，如果你选择一个短窗口的均匀度量，那就相当于你只想要一个局部卷积核。

That's a good question. Actually, we have started doing more experiments on images, which I didn't include in this talk. Locally, we do find that the local bias of convolutions does seem pretty good. I don't know, it's hard to quantify missing features, but I think there are settings where we are only on par or maybe a little bit worse than a standard local CNN. It is hard to say. I will mention though that you can forcibly incorporate locality into this just by changing the measure. For example, if you choose a uniform measure that has a short window, that's the same as saying you just want a local convolution kernel.

因为我想象对于这种特定用例，比如处理非常高分辨率的图像数据，例如乳腺 X 光片，对吧？我们至少要处理千乘千的尺寸。所以对于这些，长卷积可能很有用，因为我们实际上想做缩放但不能，因为可能会丢失中间很多特征。但长卷积可以... 这是一个完美的问题，我本来不打算提，但现在我会在实验中也提到这一点。

Because I would imagine for this particular use case, like when we have to work with very high resolution image data, for example, imagine like mammograms, right? We have to go with like thousand by thousand minimum dimension. So for these, probably this would be useful because we actually want to do rescaling but we cannot because we'll lose probably a lot of features in the middle. But this kind of long convolution could... This is a perfect problem that I will... Actually, I wasn't going to, but now I will mention this in the experiments as well.

好的，这其实是我们考虑过的事情，基本上就是卷积的缩放和使用... 好吧，我马上讲到。在那之前，我想先讲实验。基本上，我只定义了一个简单的线性一维到一维映射，但你可以并行地处理很多特征，然后将其接入标准神经网络进行序列建模。我要展示的第一类数据是生物信号数据。这是一个真实世界的数据集，旨在从原始生物信号数据（如 EKG 和 PPG）预测生命体征（如心率）。这里可视化了。这些数据对深度学习模型来说很有挑战性，因为你可以看到它非常长；这是一个长度为 4000 的序列。如果你放大很多，它实际上会很平滑，但如果你缩小，它会显示出很多周期性和尖峰等特征。很多方法都在这个数据集上尝试过，包括标准机器学习技术如 XGBoost，以及许多非常现代的深度学习方法。

Okay, it's actually something that we have thought about, basically rescaling of convolutions and using... Right, okay. I'll get to that. Before that, I want to get to the experiments. Basically, I only defined a simple linear 1D to 1D map, but you can just do it in parallel across a lot of features and then plug it into a standard neural network to do sequence modeling. The first type of data I'll show is biosignal data. Here is a real-world dataset of trying to predict vital signs such as heart rate from raw biosignal data such as EKG and PPG. That's visualized here. This data is pretty challenging for deep learning models because you can see it's very long; this is a sequence of length 4000. If you zoom in a lot, it would be pretty smooth actually, but if you zoom out, it displays a lot of periodicity and spikes and other things. A lot of methods have been tried on this dataset, including standard machine learning techniques like XGBoost as well as many very modern deep learning methods.

序列模型在所有指标上都有显著提升。仅用我展示的那个通用深度学习模型，就能将均方根误差降低至少三分之二。实际上，这些是旧数据；最近我重新运行了实验，还能进一步降低误差。有一点我要指出：注意力机制和 Transformer 在这类数据上表现非常差，这是我反复验证的结果。所以数据类型存在某种偏好。S4 擅长处理信号，而注意力机制不擅长；反过来，注意力机制擅长处理某些离散数据，而 S4 则不那么擅长。好了，这是第一个实验。

Sequence models substantially improve over all of these. I think cutting the root mean squared error by at least two thirds on all of these targets just with that generic deep learning model that I showed. Actually, these were older numbers; recently I've been rerunning these again and you can drop this down even more. One thing I will note is that attention and transformers do really poorly on this type of data, and that's something I've found pretty consistently. So there's some sort of bias toward what type of data you have. S4 is really good at signals, and attention is not. Conversely, attention is good at some other types of discrete data that S4 is not as good at. Okay, so that was one experiment.

下一个实验是关于时间序列数据的预测任务：给定一个上下文窗口，预测未来值。我会讲得快一些，因为时间有限，我想多谈谈生物应用以及你们提到的问题。这里的模型非常复杂，而 S4 的设置极其简单：掩码预测。我们取整个序列，掩码掉要预测的范围，然后通过这个通用深度模型预测掩码部分。所以这是一个非常简单的应用。我不详细解读数据，但有很多基线模型，包括时间序列模型、LSTM、各种 Transformer，而 S4 在真实时间序列数据集（包括天气和能源数据）上表现优于所有模型，且专业化程度低得多。这些模型都是为时间序列设计的，而我们只是用了通用模型，甚至没有调整窗口大小。实际上，调整窗口大小后还能进一步降低误差。

The next one is on time series data, where we did a forecasting task: given a context window, predict future values. I'm going to go through this kind of fast because I don't have much time and I want to get to more bio applications and the things you guys brought up. The models here are very complicated, whereas for S4 we're doing an extremely simple setup: mask prediction. We take the entire sequence, mask out the desired forecast range, and then predict what's in the mask by passing it through this generic deep model. So it's an extremely simple application. I won't unpack the numbers too much, but there are many baselines here, including time series models, LSTMs, lots of transformers, and S4 does better than all of them on these real time series datasets, including weather and energy data, with much less specialization. These models were all designed for time series, and we were just using our generic model. We didn't even tune the window size. Actually, by tuning the window size, you can get the numbers down even more.

接下来是关于 Iman 提出的重采样问题。我会用音频来展示。音频采样率极高，序列很长。这是一个将一秒长的语音片段（长度 16000）分类为单词的数据集。大多数序列模型如 Transformer 和 RNN 在这里表现很差，唯一有效的是 CNN。红线指向一个语音 CNN 基线，效果还行。但如果以不同频率重采样信号会怎样？这在音频中很常见，因为信号可以以任何速率采样，听起来大致相同。例如，橙色序列是采样序列，但实际上与原始蓝色序列是同一个底层信号，只是频率不同。所以理想情况下，同一个模型应该能处理两者。但标准模型如 CNN 做不到，主要是因为之前提到的局部偏差。如果使用标准局部 CNN，在不同频率下会失效。然而，使用像 S4 这样的信号模型，它能理解底层连续域或连续函数，因此无需修改就能工作。这完全是零样本设置：在一个分辨率上训练，在另一个分辨率上测试。CNN 会失败，但 S4 开箱即用。这是因为 S4 的第一个特性：它是一个连续时间模型。

The next one points to Iman's question about resampling. I'll display this for audio. Audio is sampled at extremely high rates and is extremely long. This is a dataset of classifying one-second speech clips, length 16,000, into words. Most sequence models like transformers and RNNs are really bad here. The only thing that works is CNNs. The red line points to a speech CNN baseline, and these work okay. But what happens if you resample the signal at different frequencies? This happens commonly in audio because your signal can be sampled at any rate and sound more or less the same. For example, this orange sequence is a sequence of samples but it's actually the same underlying signal as the original blue sequence, just at a different frequency. So it's ideal if the same model works on both. But standard models like CNNs cannot do this, essentially because of the local bias brought up earlier. If you use a standard local CNN, it will break at a different frequency. However, by using a signal model such as S4, which understands the underlying continuous domain or continuous function, it can work here without modification. This is all in a zero-shot setting: trained at one resolution, tested on a different resolution. This breaks a CNN, but S4 can do it out of the box. That's because of this first property of being a continuous-time model.

最后我要展示的两点回顾了最开始的实验。我展示了一些自动生成的音频片段，那是自回归设置，我们一次生成一个样本。尽管上下文窗口非常大，使得生成效果更好、更连贯，但我们仍然可以像其他自回归模型一样快速采样。这是因为快速在线或自回归表示：每次计算状态并更新。最后，我展示了长程建模的基准测试，S4 在一系列不同任务上显著优于其他模型。这个基准测试也用于衡量训练时的模型速度，S4 与所有高效的 Transformer 变体一样快。这是因为我们引入了高效的并行化视角和新算法。所以，正如我所承诺的，所有这些特性都有具体的实证收益。

Now the last two things I'll show are calling back to the experiments at the very beginning. I showed some auto-generated audio clips, and that was an autoregressive setting where we generate things one sample at a time. Despite having an extremely large context window, which made it do better and more coherent, we could still sample things autoregressively just as fast as other autoregressive models. That's because of the fast online or autoregressive representation where you compute the state and update it every time. Finally, I showed this benchmark of long-range modeling where S4 substantially outperforms other models on a range of different tasks. This benchmark was also used to benchmark the speed of models during training, where S4 is just as fast as all these efficient transformer variants. That's because of the efficient parallelizable view along with the new algorithms we introduced. So all these properties, as I promised, have concrete empirical benefits.

现在时间不多了，我只想再讲几点。最后，针对在座的各位，我想指出我希望这个模型能发挥作用的领域：作为生物信号的深度学习通用工具。我已经举了一个例子：从心电图信号预测心率。但 Ce 还做了另一个工作。她和另一位实验室伙伴一直在尝试将 S4 用于原始脑电图信号数据集，这些信号非常嘈杂且长，难以处理。最先进的模型是最近才出现的；Ce 几个月前的模型在其中一个脑电图数据集上表现不错，但非常复杂，需要大量领域知识，比如电极放置位置和许多不同组件。所以我希望 S4 能作为一个通用工具或构建块，用于处理这类信号数据，而无需太多设计模型的领域专业知识。Ce 和 Khalid 一直在用 S4 进行初步实验，我们甚至不需要用 FFT 特征进行预处理，也不需要做其他很多事情。只需通过一个通用深度模型，组合最好的四层，他们发现了一些非常初步的结果：在某些设置下，S4 优于基线。这仍然非常初步，所以还有其他我们关心的设置，比如结合自监督学习，目前还不太理想。但我确实认为它在这个领域有很大潜力。

Now I'm running out of time, so I just want to get to a couple more things. For the last part, for this audience, I wanted to point to where I hope this model will be useful: as a general tool for deep learning for biosignals. I've pointed out one example already: predicting heart rate from EKG signals. But this was another one that Ce worked on. She and another lab mate have been trying to test S4 on a dataset of raw EEG signals that are difficult to process because they are so noisy and long. The state-of-the-art models are very recent; Ce's model from a couple months ago was there on one of these EEG datasets, but it was quite involved and required a lot of domain knowledge, such as the placement of electrodes and many different components. So I hope S4 could be useful as a generic tool or building block for addressing these types of signal data without as much domain expertise in designing the model. Ce and Khalid have been running some preliminary experiments using S4 on this data, where we don't even need to pre-process it with FFT features or do a lot of other things. Just run it through a generic deep model, compose the best four layers, and they found some very preliminary results where it is improving over the baselines in some settings. This is still very preliminary, so there are other settings we care about, such as incorporating self-supervision, where it's not quite there yet. But I do think it has a lot of potential in this type of domain.

另一个已发表的例子是与斯坦福医学院的合作，提交到胃肠病学期刊，关于从阻抗传感器数据检测胃酸反流。同样，S4 在这类预测任务上表现非常好。这就是我要讲的全部内容。总结一下：S4 是一个状态空间模型，由两个方程组成，我们代入特定公式并采用特殊算法来……

Another example that was published was a recent collaboration with Stanford Medicine, submitted to a gastroenterology journal, on detecting acid reflux from impedance sensor data. Again, S4 was really good on that type of prediction task. So that is all I was going to talk about. So just to review: S4 is an SSM, which are these two equations where we plug in certain formulas and have special algorithms to...

计算模型和整体 SSM，特别是 S4，具有许多非常好的特性，并带来了具体的实证收益，正如我们所看到的。我认为它可以成为未来建模多种序列数据的非常有效的构建模块。感谢大家的聆听，也感谢所有合作者的辛勤工作。这张幻灯片列出了一些资源，比如博客文章和相关论文，以及一篇正在提交中的论文的音频结果。如果有问题请随时联系，谢谢。这是我的最后一张幻灯片，但因为我在问答环节，我猜时间已经超了。所以当然，大家想走可以走，但如果想留下，我可以展示一下关于高分辨率图像的内容，这是之前提到的。让我找到那张幻灯片。是的，如果大家有冲突可以离开，我们稍后会在 YouTube 频道上传演讲录像。如果大家想留下，那我就准备分享幻灯片。我很快过一下，医学成像我们认为可能是 S4 的一个潜在强应用场景，因为它具有高分辨率特性。这张幻灯片是关于……我本来是从另一个角度讲的，但我想说的重点是，嗯，你能切换到演示模式吗？我想，嗯，抱歉，哎呀，我是在共享整个屏幕吗？不，我们看到的是你的屏幕而不是演示。好的，我以为我设置对了。你能切换视图吗？你知道，我以为我选对了视图。这个还能显示那个……嗯，哎呀，好的，太好了。所以，我想说的重点是，通常的图像数据集如 ImageNet 实际上分辨率非常低，而其他数据比如医学成像中，图像可能高达一千乘十万像素。这显然对于当前只能处理小块的模型来说太大了。我不知道如何解决这个问题，但这让我很着迷。但只是提一点，这是我一个更长的求职演讲的一部分，我指出了一些潜在的未来方向。我要提到的这个方向与之前讨论的一些事情有关，就像我展示的语音实验一样，我相信 S4 应该能在不同分辨率的图像上训练。所以你可以尝试在低维版本的图像（低分辨率版本）上训练，然后将同一个模型迁移到高维上，这与我展示的音频语音例子非常相似。所以，我认为这有可能行得通，关键在于像 S4 这样的信号模型可以在不同分辨率下工作，因为你可以以不同的采样率采样。所以，你需要一个理解连续域的信号模型，就像我展示的例子一样，这又指向了这个特性。这是我们还没有尝试过的，我不知道是否可行，但我内心觉得这可能是解决这类问题的正确方法或一种潜在的好方法。

Compute the model and overall SSMs and in particular S4 have a number of very nice properties with concrete empirical benefits as we saw and I think can become a very effective building block for modeling many types of sequential data in the future. Thanks for listening and thanks for all the collaborators for the hard work. This slide lists a couple of resources such as blog posts and related papers as well as the audio results from a paper that's under submission right now. Feel free to reach out if you have questions and thanks. This was my last slide but because I'm on ask I will, I guess I'm technically out of time. So of course people feel free to leave but if you want to stay I can show one thing about the high resolution images that was brought up. Let me find that slide. Yeah, if people have conflicts feel free to leave and we will put up the recording of the talk later in our YouTube channel. Otherwise if you would like to stay then yeah I'll be prepared to share the slide. So I'll just really quickly go over this where medical imaging is something that we think could be a potential strong use case for S4 because of this high resolution feature. So this slide was about, I was moving from a different way but the point I wanted to make was that, yeah, can you go to presentation mode? I think, um, yeah sorry, oops, am I showing my whole screen? No, we are seeing your screen rather than the presentation. Okay, I thought I had it on. Can you swap the view? You know, I thought I had it on the right view. Does this one still show the, um, oops, yeah okay great. So yeah, so the point I was making is that normally image datasets are things like ImageNet which are actually extremely low resolution compared to other data that we might find such as in medical imaging where apparently the images can be up to a thousand by a hundred thousand pixels. And this is obviously like way too big for current models which can only operate on small patches at a time. So I don't know how to address this really but it's something that fascinates me. But just a point, this is part of a longer job talk where I point to some potential future directions. The one that I'll mention here relates to some things that we brought up which is that just like the speech experiment that I showed, I believe that S4 should work training on images at different resolutions. So what you can do is essentially try to train on lower dimensional versions of the image, low resolution versions, and then transfer the same model to work on high dimensions, which is a very similar thing that I showed for the audio speech example. So yeah, I think that's potentially something that could work and the point is that a signal model like S4 will work at different resolutions because you can sample at different rates essentially. And yeah, so what you need is a signal model that understands the continuous domain just like the example I showed, and that points to this property again. So this is something where we haven't tried it and I don't know if it works but it's some part of me feels like it might be the right way to or one potential good way to approach this type of problem.

我会有相反的想法，你知道，我们并不是真的想从低分辨率生成高分辨率，但我想既然你有这种基于状态的表示，最终得到这个信号，我想在某些情况下，就像我们不得不处理这种情况：我们有一张非常高分辨率的图像，在运行卷积之前，由于内存计算、计算复杂度、内存复杂度等等，你必须将图像重新缩放到低得多的维度。是的，我们有可能丢失很多特征，特别是组织病理学，就是你展示的例子，或者乳腺 X 光片那种图像，你知道。

I would think in the opposite way, you know, it's not really we want to generate the high resolution from the low resolution, but I would imagine since you have this kind of like state based representation and finally you're getting this signal, I would imagine like in some case always like we had to deal with this kind of situation that we had a very high resolution image and before running through the convolution because of the memory computations, computational complexity, memory complexity and all this kind of thing, so you have to rescale the image into a much lower dimension. Yeah, and we had a chance of losing a lot of features specifically histopathology exactly the example that you showed or the mammogram those kind of images, you know.

我明白了。所以我在想，也许你可以迭代地增加分辨率，逐步捕捉越来越高的分辨率特征，但好处是，也许你可以先捕捉粗粒度特征，然后当你放大图像并重新缩放核时，它已经在做大家都知道的核心屏幕特征，然后随着你继续训练，它只需要学习更高频的特征。再说一次，我不知道这是否合理或有前景，但听起来确实很有趣。

I see. So I was thinking that perhaps like what you can do is kind of like iteratively increase the resolution and pick up higher and higher resolution features as you go, but the benefit is that perhaps you can pick up the coarser grain things and then right as you upsize the image and then you rescale your kernel essentially, then it's already going to be doing is everybody knows the core screen features but then it as you keep training it only has to learn the higher frequency features as you go. And again, I have no idea if this makes sense or is promising, but it sounds yeah it sounds pretty interesting.

我认为 Elbow 的想法实际上与病理学家分析图像的方式非常相似，他们通常查看不同的缩放级别，先看低分辨率图像，定位潜在的肿瘤区域，然后放大到更高分辨率，对吧？是的，我明白了。所以我不知道这是否会比 CNN 或其他方法更好，但它确实具有不同且有趣的特性。

I think Elbow's idea is actually very similar to how pathologists analyze images, so they usually look at different zoom differences, they usually look at low resolution image first and localize like the potential areas where the tumor are and then they zoom in to higher resolution, right? Yeah, I see. Yeah, and so I don't know if this will be better than CNNs though or other things, but it definitely has different and interesting properties.

好的，所以，是的，我想我的材料就这些了。我还可以再说几分钟，如果大家还有问题的话。

Okay, so that was, yeah, I think that's the end of the material I have. I can say around a few more minutes that people still have questions.

好的，太好了。非常感谢，Albert。我只有一个问题。好的，谢谢你的演讲，太棒了。你有没有发现任何场景下使用 Transformer 比 S4 更好？

All right, great. Thanks so much, Albert. I have just one question. Okay, yeah, thank you for the presentation, that was awesome. Did you find any scenario where it's better to use transformer than S4?

是的，另一个好问题。让我再共享一下屏幕，我有一张幻灯片准备了这个问题。基本上在开头我区分了连续时间、连续和离散数据，我认为 S4，或者说其中涉及的思想，可能是处理信号的最佳选择。但对于更高层次的概念或更离散的概念，比如语言或其他东西，我认为 Transformer 才是真正大放异彩的地方，可能更好。所以，这是……我不知道是不是又选对了屏幕，抱歉。总之，这是一张关于语言建模的幻灯片，我们拿了一个 Transformer（目前当然是文本和 NLP 的最佳模型），用 S4 替换了注意力机制，发现它表现不如 Transformer，但仍然优于所有非 Transformer 模型，并且显著优于所有非 Transformer 模型。它还有其他好处，比如由于快速循环视图，你可以更快地进行语言生成。这是主要观点，但这也表明，我个人直觉是 Transformer 非常适合密集和离散数据，而 S4 更适合噪声和原始数据。是的，我认为这里的加速非常有趣。

Yes, another great question. So let me just share my screen again, I have one slide prepared for that. Basically at the beginning I drew this distinction between continuous time, kind of continuous and discrete data, and I think that S4 will be the best, or like the ideas involved are potentially going to be the best thing to do for signals. But for kind of higher level concepts or more discrete concepts such as language or some other things, transformers I think that's where transformers really shine and are probably going to be better. So here's, I don't know if this is the right screen again, sorry. Anyways, here's the one slide on language modeling where we took a transformer which are currently of course the best models for text and NLP, and we replaced the attention with S4 and found that it doesn't do quite as well, but it is still better than all non-transformer models, and it is significantly better than all non-transformer models. It also has some other benefits like you can do language generation much faster because of the fast recurrent view. That was the main point of this, but this also does point to the fact that personally my intuition is that transformers are really good for dense and discrete data whereas S4 is really good for more like noisy and raw data. Yeah, and I mean the speed up here I think is very interesting.

你知道你进行语言建模时使用的窗口大小是多少吗？

Do you know what was the window you would take for language modeling like?

你考虑了多少个词元或单词？

How many tokens or words did you consider?

这个实验使用了相当标准的长度，512 或 1024 个词元。实际上，你可以不断增加 S4 的窗口长度，这只会稍微减慢速度，而且还会略微提升性能。但我是事后才发现这一点的，所以没有重新训练。

This experiment was done using a pretty standard length of either 512 or 1024 tokens. You can actually keep increasing the window length for S4, which only slows it down a little bit and actually improves the performance a little bit as well. But I found that out after the fact and I didn't feel like retraining this.

好的，谢谢。所以这张幻灯片的结果是加速，对吧？非常显著。这就是我们做这个实验的目的。不过，确实有很大的加速。但回到最初的问题，在建模数据的原始性能方面，Transformer 目前表现稍好一些。

Okay, cool. Thanks. So the findings of this slide is speed up, right? It's massive. That was the point we did this experiment for. But yeah, so there's a lot of speed up. In terms of the original question though, in terms of the raw performance of modeling the data, transformers are currently doing a little bit better here.

好的，谢谢。

Cool, thank you.

好了，还有其他问题吗？让我们给 Albert 虚拟鼓掌。感谢你对状态空间模型非常全面的介绍。

All right, is there any other questions? Let's all give Albert a round of virtual applause. Thank you for the very comprehensive presentation of state space models.

谢谢邀请。

Yeah, thanks for having me.

谢谢，谢谢，谢谢。感谢大家的参与。我们稍后会将讲座视频的录制上传到 YouTube 频道。下周见。谢谢。说再见吧。

Thank you, thank you, thank you. Thank you everyone for joining us. We will put up the recording of the video talk later to our YouTube channel. And yeah, we'll see you sometime next week. Thank you. Say bye.