Albert Gu from Stanford presents S4, a new sequence model for continuous signal data, built on the HIPPO framework.
要点 · TL;DR
S4 利用状态空间模型高效处理长序列，在连续信号上优于 Transformer。 S4 uses state space models to efficiently handle long sequences, outperforming Transformers on continuous signals.
HiPPO 提供了一种将历史在线压缩为状态的有原则的记忆方法。 HiPPO provides a principled online compression of history into a state for memorization.
S4 可递归用于推理或卷积用于并行训练。 S4 can be used recurrently for inference or convolutionally for parallel training.
核心观点 · Key points
S4 是一种状态空间模型，专为音频和生物信号等连续信号中的长程依赖而设计。 S4 is a state-space model designed for long-range dependencies in continuous signals like audio and biosignals.
Hippo 提供了一种将历史压缩为状态以进行在线记忆的原则性方法。 Hippo provides a principled way to compress history into a state for online memorization.
S4 可以递归计算用于在线推理，或卷积计算用于并行训练。 S4 can be computed recurrently for online inference or convolutionally for parallel training.
S4 在 Long Range Arena 等长程序列基准测试上优于 Transformer 和 CNN。 S4 outperforms Transformers and CNNs on long-range sequence benchmarks like Long Range Arena.
S4 在原始生物信号数据上表现良好，无需大量领域特定的预处理。 S4 works well on raw biosignal data without extensive domain-specific preprocessing.
反共识 · Contrarian takes
S4 可以建模无限长的卷积，不同于具有有限窗口的标准 CNN。 S4 can model infinitely long convolutions, unlike standard CNNs with finite windows.
S4 零样本泛化到不同采样率，打破了 CNN 的局部偏差。 S4 generalizes zero-shot to different sampling rates, breaking the local bias of CNNs.
尽管 S4 有速度优势，但 Transformer 在语言等密集离散数据上仍优于 S4。 Transformers still outperform S4 on dense discrete data like language, despite S4's speed advantage.
S4 可以在低分辨率图像上训练，并迁移到高分辨率图像而无需重新训练。 S4 can be trained on low-resolution images and transferred to high-resolution ones without retraining.
S4 中的状态是有意义的，可以提取用于可解释性，不同于黑盒模型。 The state in S4 is meaningful and can be extracted for interpretability, unlike black-box models.
本期章节 · Chapters（共 20）
引言与格式Introduction and Format
序列模型与 S4 概述Overview of Sequence Models and S4
长上下文挑战Long Context Challenge
形式化 Hippo：在线压缩与重建Formalizing Hippo: Online Compression and Reconstruction
HIPPO：用少量系数重建信号HIPPO: Reconstructing signals with few coefficients
从 HIPPO 到 S4：状态空间模型From HIPPO to S4: State Space Models
SSM 及其属性介绍Introduction to SSMs and their properties
SSM 的三个视角Three perspectives on SSMs
SSM 属性回顾Recap of SSM properties
状态空间表示与直接输出State space representation vs direct output
跨维度相关性Correlation across dimensions
S4 如何基于 SSM 和 HiPPO 构建How S4 builds on SSMs and HiPPO
S4 卷积效率S4 convolution efficiency
状态与卷积使用State vs convolution usage
S4 中的多重度量Multiple measures in S4
S4 在时间序列和音频上优于 TransformerS4 outperforms transformers on time series and audio