On New Year's Day 2026, Chinese AI startup DeepSeek released a groundbreaking paper titled mHC: Manifold‑Constrained Hyper‑Connections. The paper proposes a novel architecture called mHC, co‑authored by Zhenda Xie, Yixuan Wei, and Huanqi Cao as first authors, with DeepSeek founder and CEO Liang Wenfeng also listed among the authors. Published on arXiv (preprint ID: 2512.24880), this work addresses the instability issues of traditional Hyper‑Connections (HC) in large‑scale model training while preserving their performance advantages. Building on DeepSeek's 2025 open‑source model DeepSeek‑V3, the mHC architecture restores the identity mapping property of residual connections through manifold constraints and incorporates infrastructure optimizations to achieve more stable training and better scalability. This New Year paper marks another major breakthrough by Liang Wenfeng's team in foundational AI architecture and points to a new direction for the future of large language models (LLMs). This article provides a detailed interpretation of the mHC architecture's technical details, experimental results, and significance, based on the original arXiv paper, IT Home reports, and the Hugging Face paper page.
Background and Motivation: Evolution and Challenges of Residual Connections
Residual connections are a cornerstone of deep learning models such as ResNet and Transformer, first proposed by Kaiming He and colleagues in 2015. Their key lies in the "identity mapping" property, which ensures stable signal propagation in deep networks and prevents gradient vanishing or explosion. In recent years, researchers have extended residual connections to boost model performance, with Hyper‑Connections (HC) being a representative approach. HC widens the residual stream (typically with an expansion ratio n of 4–8) and diversifies connection patterns (e.g., full or random connections), delivering significant performance gains without increasing computational complexity (FLOPs). For instance, in Transformers, HC enables information exchange across residual branches, enhancing representation capacity.
However, the diversified connections in HC disrupt the original residual's identity mapping, leading to signal amplification/attenuation, training instability (loss spikes, gradient norm explosions), and increased memory access overhead. These issues are particularly pronounced in large‑scale training: wider loss gaps, anomalous gradient fluctuations, and restrictions on applying HC to trillion‑parameter models. The DeepSeek team identified unconstrained mapping spaces as the root cause and proposed the mHC framework, which projects residual connections onto a specific manifold to restore identity mapping while retaining HC's diversity benefits. This motivation stems from deep insights into model topological design, aiming to provide a new path for foundational model evolution.
Technical Details of mHC: Manifold Constraints and Projection Mechanisms
The core of mHC is a general framework that constrains HC's residual connections within the Birkhoff polytope of doubly stochastic matrices. Doubly stochastic matrices are non‑negative matrices with row and column sums equal to 1, ensuring residual mappings as convex combinations of features, thereby achieving signal mean conservation and bounded norms.
Due to the compositional closure of doubly stochastic matrices, the product remains doubly stochastic, ensuring end‑to‑end stability. mHC's convex combination interpretation enables feature fusion while bounding the spectral norm ≤1, preventing signal explosions.
Additionally, mHC incorporates infrastructure optimizations:
- Kernel Fusion: Using TileLang to implement custom mixed‑precision kernels (bfloat16 for activations, float32 for parameters), reducing memory traffic by fusing operations.
- Selective Recomputing: To save memory, mHC stores the main residual only every certain layers
- , where n is the expansion ratio and L is the total layers), selectively recomputing unstaged parts during backpropagation, aligned with pipeline stages.
- DualPipe Extension: Extending DeepSeek‑V3's dual‑pipeline parallelism with high‑priority compute streams to fully overlap communication and computation.
These optimizations result in only a ~6.7% increase in training time overhead for n=4, with peak memory close to the baseline (1.05×).
Experimental Setup and Results: Large‑Scale Validation of Performance Gains
The paper validates mHC on Mixture‑of‑Experts (MoE) models with scales of 3B, 9B, and 27B parameters and expansion ratio n=4. Datasets are scaled proportionally (e.g., 262B tokens for 27B), using AdamW optimizer, RMSNorm, and RoPE positional encoding. Benchmarks include BBH, DROP, GSM8K, HellaSwag, MATH, MMLU, PIQA, and TriviaQA.
Results show mHC significantly improves stability: eliminating HC's loss spikes and gradient explosions (Figure 5). Performance‑wise, mHC outperforms baselines and HC across all 8 benchmarks—for example, reducing final loss by 0.021 on the 27B model, with +2.1% on BBH and +2.3% on DROP (Table 4). Compute scaling experiments (Figure 6a) confirm sustained advantages from 3B to 27B; token scaling (Figure 6b) shows further gains with more data. Propagation stability analysis (Figure 7) reveals mHC reduces max gain magnitude from ~3000 (HC) to ~1.6. Mapping visualizations (Figure 8) confirm mHC avoids HC's extreme values.
Compared to baselines, mHC averages 1‑2% downstream gains; versus HC, it avoids instability for better convergence. Memory analysis (Table 2) shows peak memory at only 1.05× baseline via recomputing.
Limitations and Future Directions
Despite its innovations, mHC has limitations: limited Sinkhorn‑Knopp iterations cause minor deviations; higher n increases memory access; recompute blocks require pipeline alignment; only doubly stochastic manifolds are explored. Future work could extend to other manifolds or trillion‑parameter models.
Impact and Significance: Insights for AI Architecture Design
This paper co‑authored by Liang Wenfeng is another contribution from DeepSeek to the open‑source ecosystem and provides a practical framework for large model training. As an extension of DeepSeek‑V3, mHC is likely to be integrated into future models, enhancing stability and efficiency. IT Home reports describe it as a "bold optimization of AI architecture," achieving significant loss reduction on 27B models with just 6.7% overhead. In the global AI race, this innovation highlights the rise of China's independent technology.
Overall, mHC deepens understanding of topological design through manifold constraints, shifting foundational models from scale competition to architectural optimization. This New Year paper heralds a more stable and scalable era for AI in 2026.
▶️ Enjoyed this article? Creating great content takes effort!
Subscribe to our YouTube channel:
AI Rosetta
Your likes, subscriptions and shares mean the world to us! 🙏
For more AI news, visit:
https://www.airosetta.com