A researcher argues that improvements in video models are driven by language models, not video-specific architectures, and shares his experience building video models at xAI.
要点 · TL;DR
视频模型的大部分改进来自语言模型，而非扩散技术。 Most video model improvements now come from language models, not diffusion technology.
使用语言模型作为编排器的视频代理将成为下一个大趋势。 Video agents using language models as orchestrators will be the next big trend.
世界模型需要实时、交互、长程视频生成，这是一个长期目标。 World models require real-time, interactive, long-horizon video generation, a long-term goal.
核心观点 · Key points
大多数视频模型的改进现在来自语言模型，而非扩散技术。 Most video model improvements now come from language models, not diffusion technology.
训练视频模型需要合成文本-视频对，因为在线自然对齐很差。 Training video models requires synthetic text-video pairs due to poor natural alignment online.
视频模型训练成本与中等规模语言模型相当，且存储需求巨大。 Video model training cost is comparable to medium-scale language models, with huge storage needs.
步骤蒸馏通过从教师模型学习，将推理步骤从 100 步减少到 4-8 步。 Step distillation reduces inference steps from 100 to 4-8 by learning from a teacher model.
世界模型需要实时、交互、长时域的视频生成，这是一个长期目标。 World models require real-time, interactive, long-horizon video generation, a long-term goal.
使用语言模型作为编排器的视频智能体将是生成式媒体的下一个大趋势。 Video agents using language models as orchestrators will be the next big trend in generative media.
反共识 · Contrarian takes
视觉智能的提升主要来自语言模型，而非视频模型本身。 Visual intelligence gains come mostly from language models, not video models themselves.
机器人技术可能通过视频模型在虚拟仿真中解决，无需真实世界交互。 Robotics may be solved by video models without real-world interaction, via virtual simulation.
未来用户界面将直接从用户意图生成像素，绕过代码和渲染。 Future UIs will be generative pixels directly from user intent, bypassing code and rendering.
语言模型应自行管理上下文长度，而非依赖启发式修剪。 Language models should manage their own context length, not rely on heuristic pruning.
模型可以在测试时通过修改自己的智能体框架代码来自我编程。 Models can program themselves at test time by modifying their own agent harness code.
视频模型的瓶颈是语言和智能体能力，而非视觉架构。 The bottleneck for video models is language and agent capability, not vision architecture.
本期章节 · Chapters（共 23）
语言模型的视觉智能Visual intelligence from language models
赞助消息与介绍Sponsor message and introduction
从英伟达到xAI的转变Transition from Nvidia to xAI
小团队大影响：调试胜于算法Small Team, Big Impact: Debugging Over Algorithms
训练图像分词器与潜空间Training image tokenizers and latent space
训练图像与视频扩散TransformerTraining diffusion transformers for images and videos
翻页书演示与生成式UIFlipbook Demo and Generative UI
神经操作系统与世界模型Neuro OS and World Models
扩散模型的步骤蒸馏Step Distillation for Diffusion Models
音视频联合生成挑战Audio-Video Joint Generation Challenges
LLM与世界模型的时间感知Time awareness in LLMs vs world models
长时视频生成的挑战Challenges in Long-Horizon Video Generation
效率与替代方法Efficiency and Alternative Approaches
视频冗余与参考方案Redundancy in Video and Reference-Based Solution
视频与语言模型的上下文管理Context Management in Video and Language Models
语音模式体验Voice Mode Experience
视频生成的安全与水印Safety and Watermarking in Video Generation
扩散模型与世界模型的推理Reasoning in Diffusion Models vs World Models
多模态模型中图像生成的不同方法Different approaches to image generation in multimodal models
视频智能体与迭代创作Video agents and iterative creation
视频智能体与纯模型训练Video Agents vs. Pure Model Training
离开xAI与研究焦点Leaving XAI and Research Focus
扩展视频模型与切换领域Scaling video models and switching domains