March 16, 2026, Moonshot AI’s Kimi team officially open‑sourced the technical report and code repository for "Attention Residuals" (Attention Residuals, abbreviated AttnRes) on GitHub. The innovation was quickly hailed in the industry as a breakthrough that "rotates attention 90 degrees": it fundamentally reconstructs the Transformer architecture’s most basic yet long‑neglected residual connection, upgrading the traditional "simple addition" into a "depth‑wise attention selective aggregation." Experimental results show that, with the same compute budget, AttnRes can achieve a 1.25× compute advantage (equivalent to saving about 20% compute to reach baseline performance) and produces significant gains on key benchmarks such as MMLU and GPQA‑Diamond. This advance has been applied directly to the Kimi Linear series, and was called by former OpenAI researcher Jerry Tworek the start of a "deep learning 2.0" era.
Inherent issues with residual connections: a shared pain from ResNet to Transformer
Residual connections were first introduced in ResNet (proposed by He et al. in 2015) for CNNs to address gradient vanishing in deep networks, with the simple formula y = x + f(x). Transformer inherited this design, but at scales of tens of billions to trillions of parameters it exposes fatal flaws: information dilution and hidden‑state explosion.
Traditional residuals perform "unconditional addition"—each layer uniformly adds the previous layer’s output. As depth increases (L layers), shallow features are diluted by later layers and early critical information (e.g., key tokens in long sequences) becomes buried. To maintain signal strength, models must increase hidden‑state activations, causing unstable training and soaring memory costs. This is a core bottleneck for deep MoE models (e.g., Kimi Linear, DeepSeek‑V3 style) on long‑context and complex reasoning tasks.
Kimi’s AttnRes overturns this mechanism: each layer no longer blindly adds; instead it emits a learnable Query that attends to Keys from all previous layers, using Softmax weights to dynamically select and aggregate the "useful" historical representations. The paper figuratively describes this as "rotating attention 90 degrees"—attention that was applied across the sequence (time) dimension is extended to the depth (layer) dimension. This lets layer 50 directly attend to layer 2 with weight 0.8, rather than relying on layer‑by‑layer decay.
Technical core: dual designs of AttnRes and Block AttnRes
The core of AttnRes is "input‑dependent depth‑wise attention aggregation":
- Each layer produces a Query and computes attention against Keys from all previous layers.
- Softmax weights determine aggregation ratios, implementing "intelligent filtering."
- Fully compatible with existing Transformer/MoE frameworks as a drop‑in replacement without changing other components.
To address the memory complexity O(L·d) of full inter‑layer attention (L = number of layers, d = dimension), the team proposed Block AttnRes: partition the model into N≈8 blocks, apply AttnRes within each block while retaining traditional residuals between blocks. Experiments show N=8 captures over 95% of the gains while reducing memory complexity to O(N·d) and increasing inference latency by less than 2%.
This design was validated on Kimi Linear (a mixture‑of‑experts MoE Transformer with activated parameter scales of 3B–48B): after training on 1T tokens, AttnRes variants consistently achieved lower validation loss than the baseline, produced more uniform gradient distributions, and transformed hidden‑state growth into a "periodic" rather than "monotonic exploding" pattern.
Empirical results: 1.25× compute leverage and broad performance uplift
The paper compares three variants (PreNorm baseline, Full AttnRes, Block AttnRes) across five model scales:
- Compute efficiency: At 5.6 PFLOP/s‑days of compute, Block AttnRes reports loss 1.692 (baseline 1.714), with fitted curves indicating a 1.25× compute advantage. To reach identical performance, AttnRes requires roughly 80% of the compute.
- Benchmark improvements (under equal compute budget):
- MMLU: +1.1
- TriviaQA: +1.9
- GPQA‑Diamond (hard scientific reasoning): +7.5
- Minerva Math: +3.6
- HumanEval (code generation): +3.1
- Long‑sequence and multi‑step reasoning tasks benefit particularly, demonstrating AttnRes effectively mitigates information dilution.
As model scale increases, the gap between Full and Block variants narrows; Block AttnRes becomes the engineering‑optimal solution. The team emphasizes: "Even for models with hundreds of layers, dividing into eight blocks captures the vast majority of benefits."
Open sourcing and industry impact: from Kimi to the whole field
Moonshot AI has integrated AttnRes into the Kimi Linear series and open‑sourced the full code and paper (GitHub: MoonshotAI/Attention‑Residuals). Developers can replace existing Transformer residuals at zero cost and immediately gain both efficiency and performance improvements.
The innovation has sparked global discussion:
- Former OpenAI researcher Jerry Tworek commented on X: "The deep learning 2.0 era is coming—we need to rethink everything."
- The community regards AttnRes as "the start of the post‑residual Transformer era," and together with prior Moonshot contributions (Kimi Linear linear‑attention hybrid architecture and the Mooncake KV caching system) forms Moonshot’s "three‑horse" approach to raising inference efficiency.
- Compared with contemporaneous architecture innovations (DeepSeek mHC, Meta, etc.), AttnRes’s standout feature is its "minimal yet universal" change—no modification to MoE routing or attention kernels is required; only the residual replacement yields benefits.
Far‑reaching implications for AI training and inference
AttnRes directly addresses two longstanding issues in deep networks and opens new paths for trillion‑parameter models:
- Training: more stable gradients, lower loss, and reduced compute requirements, cutting training costs.
- Inference: combined with the Mooncake KV cache, AttnRes can further reduce memory and increase throughput, especially for long‑context agentic tasks.
- Future directions: the paper suggests AttnRes can be combined with linear attention and MoE to potentially yield a new class of "native depth attention" architectures.
Industry analysts note this marks a shift for Chinese AI teams from "following compute arms races" to "architectural innovation for asymmetric advantage." With compute costs still high in 2026, a 1.25× efficiency advantage means models can deliver a generational performance uplift on the same budget.
Conclusion: the rebirth of residuals and an AI architecture revolution
Kimi’s Attention Residuals are not a simple optimization but a philosophical‑level reconstruction of Transformer’s most fundamental module. They demonstrate that even after the Transformer era, there remain low‑hanging fruits to harvest. With the code open‑sourced and Kimi models evolving, this technique is likely to spread rapidly across the industry.
Moonshot AI’s AttnRes again proves that Chinese AI now leads not only in parameter scaling but also in foundational architecture innovation. 2026 may be the year when "deep learning 2.0" transitions from concept to reality.
▶️ Enjoyed this article? Creating content is hard work!
Follow our YouTube channel:
AI Rosetta
Your likes, subscriptions, and shares are our greatest support! 🙏
More AI news at: https://www.airosetta.com