A discussion on using JLM52 to auto-write GPU kernels and the process of handling long queries in inference engines.
要点 · TL;DR
KV 缓存和投机解码是无损优化，而量化是主要的有损优化。 KV caching and speculative decoding are lossless optimizations, while quantization is the main lossy one.
推理工程正与训练融合，需要量化感知训练和蒸馏等技术。 Inference engineering is merging with training, requiring techniques like quantization-aware training and distillation.
像 Rubin 这样的未来 GPU 正变得专用化，将推理工程从内核编写转向基础设施编排。 Future GPUs like Rubin are becoming specialized, shifting inference engineering from kernel writing to infrastructure orchestration.
核心观点 · Key points
KV 缓存和投机解码等推理优化是无损的，而量化是主要的有损优化。 Inference optimizations like KV caching and speculative decoding are lossless, while quantization is the main lossy optimization.
为生产环境支持一个新模型需要大量工作，包括量化、投机解码器训练和基础设施搭建。 Supporting a new model for production requires significant work including quantization, speculative decoder training, and infrastructure setup.
行业正朝着预填充和解码分离的方向发展，KV 缓存管理变得越来越重要。 The industry is moving towards disaggregated prefill and decode, with KV cache management becoming increasingly important.
推理工程正与训练融合，量化感知训练和蒸馏等技术变得必要。 Inference engineering is merging with training, as techniques like quantization-aware training and distillation become necessary.
像 Rubin 这样的未来 GPU 正变得更加专用化，推理工程从内核编写转向基础设施编排。 Future GPUs like Rubin are becoming more specialized, shifting inference engineering from kernel writing to infrastructure orchestration.
视频生成模型因大量 token 上的二次注意力成本而面临挑战，促使向自回归方法发展。 Video generation models face challenges due to the quadratic attention cost over many tokens, pushing towards autoregressive approaches.
反共识 · Contrarian takes
如果量化误差相互抵消，量化更多层反而能提升模型质量，这与量化越多越差的观点相反。 Quantizing more layers can improve model quality if quantization errors cancel out, contrary to the belief that more quantization is always worse.
Mega-kernel 在生产中往往不被采用，因为模块化内核的独立启动可能更快且更优化。 Mega-kernels are often not used in production because modular kernels with separate launches can be faster and more optimized.
GPU 正趋向于成为 ASIC，使得内核级编程变得不那么重要，而基础设施问题变得更加核心。 The GPU is trending towards becoming an ASIC, making kernel-level programming less relevant and infrastructure problems more central.
持续学习可能更适合通过 KV 缓存压缩来解决，而不是更新模型权重，后者可能导致推理不一致。 Continual learning may be better solved via KV cache compaction rather than updating model weights, which can cause reasoning inconsistencies.
投机解码的投机解码（spec-spec）是可能的，但训练复杂性使其往往不值得投入。 Speculative decoding on speculative decoding (spec-spec) is possible but training complexity makes it often not worth the effort.
本地 AI 推理技术如动态量化因约束不同，不能直接应用于数据中心推理。 Local AI inference techniques like dynamic quantization are not directly applicable to data center inference due to different constraints.
本期章节 · Chapters（共 27）
JLM52 编写GPU内核JLM52 Writing GPU Kernels
草稿模型与端点定制Draft Model and Endpoint Customization
结构化输出与推理工程Structured Outputs and Inference Engineering
LLM中的视觉能力Vision in LLMs
非确定性推理问题Non-deterministic inference issues
优化中保持模型质量Maintaining model quality across optimizations