On January 15, 2026, Wenfeng Liang, founder and CEO of Chinese AI startup DeepSeek, coauthored a paper titled "Conditional Memory via Scalable Lookup: A New Axis of Sparsity for Large Language Models" in the DeepSeek‑AI/Engram repository on GitHub. The paper, a joint effort by the DeepSeek AI team and researchers from Peking University, is co‑signed by Wenfeng Liang and researchers Xin Cheng, Wangding Zeng, Damai Dai, Qinyu Chen, Bingxuan Wang, Zhenda Xie, Kezhao Huang, Xingkai Yu, Zhewen Hao, Yukun Li, Han Zhang, Huishuai Zhang and Dongyan Zhao, following the team's earlier mHC paper. According to DeepSeek's GitHub update on January 15, 2026, the module introduces conditional memory as a new axis of sparsity for large language models (LLMs) to enable efficient knowledge lookup and reasoning optimization. TechCrunch reported the innovation will reinforce DeepSeek's leadership in open‑source AI and promote edge AI deployments in 2026. Reuters analyzed that Liang's successive publications reflect rapid iteration by Chinese AI firms and suggest long‑range reasoning will be a key battleground in the AGI race.
Background: The Need for a New Axis of Sparsity
LLMs face dual bottlenecks in compute and memory when handling language tasks. Traditional Mixture‑of‑Experts (MoE) expands capacity via conditional computation, but the Transformer architecture lacks a native mechanism for knowledge lookup and instead simulates retrieval through computation, which is inefficient. The paper argues that language modeling comprises two subproblems—compositional reasoning and knowledge retrieval—where the former requires dynamic computation and the latter can be optimized via static lookup. The authors observe that classic N‑gram embeddings can be modernized into efficient O(1) lookup tools, especially useful for on‑device inference to relieve memory pressure. Building on their mHC work that addressed training stability, this paper focuses on a new axis of sparsity—conditional memory—as a complement to MoE.
The paper emphasizes that linguistic heterogeneity calls for structural optimization: static patterns (e.g., named entities) can be stored in memory, freeing compute for complex reasoning. VentureBeat notes this motivation stems from the 2025 surge in edge AI, with DeepSeek aiming to solve long‑context bottlenecks on devices.
Technical Details of the Engram Module: Modernizing Classic N‑grams
The core contribution is the Engram module, an instantiation of conditional memory that modernizes classic N‑gram embeddings via scalable lookup. In simple terms: while MoE resembles a "team of experts" for conditional computation, Engram functions like an "intelligent repository," providing fast O(1) retrieval of static knowledge.
Key innovations:
- Tokenizer compression: Classic N‑grams rely on token frequency, but subword tokenizers (e.g., BPE) produce non‑contiguous IDs. Engram precomputes a projection function φ to map the original vocabulary V to a canonical ID space V', reducing the effective vocabulary size by ~23%.
- Multi‑head hash retrieval: Instead of directly parameterizing the exponentially large N‑gram space, Engram uses K hash heads to map contexts to embedding tables E_{n,k} (table sizes M_{n,k} chosen as primes). Formula: z_{t,n,k} = φ_{n,k}(g_{t,n}), e_{t,n,k} = E_{n,k}[z_{t,n,k}]. The final memory vector e_t is the concatenation of all retrieved embeddings.
- Context‑aware gating: Static memories can collide; Engram uses the current hidden state h_t as Query and retrieved memory as Key/Value, applying RMSNorm and a sigmoid to compute gate α_t = σ((RMSNorm(h_t)^T RMSNorm(k_t)) / √d). The gated output ~v_t = α_t · v_t ensures semantic alignment.
- Convolutional augmentation: A deep causal convolution (kernel size w=4, dilation δ = max N‑gram order) with SiLU activation expands the receptive field: Y = SiLU(Conv1D(RMSNorm(~V))) + ~V.
- Multi‑branch integration: Combined with mHC, Engram shares the Value projection matrix while each branch has an independent Key projection, supporting Manifold‑Constrained Hyper‑Connections (M=4).
The paper includes diagrams showing all‑to‑all communication during training and host prefetch at inference, emphasizing infrastructure‑aware efficiency: deterministic addressing enables runtime prefetch with near‑zero overhead.
Sparsity Allocation and the U‑shaped Scaling Law
The authors formalize the sparsity allocation problem: given fixed parameters and FLOPs, how should capacity be split between MoE experts and Engram memory? They identify a U‑shaped scaling law: pure MoE (ρ=1) is suboptimal, and allocating ~20–25% of capacity to Engram yields best performance. In 27B‑parameter experiments, Engram outperformed the MoE baseline: MMLU +3.4, CMMLU +4.0, BBH +5.0, ARC‑Challenge +3.7, HumanEval +3.0, MATH +2.4.
Mechanism analysis shows Engram frees early layers from static reconstruction, deepening network reasoning; offloading local dependencies reduces attention load and significantly improves long‑context retrieval (Multi‑Query NIAH 97.0 vs 84.2).
Experimental Results: Large‑Scale Pretraining Validation
The paper trains four models: Dense‑4B, MoE‑27B, Engram‑27B and Engram‑27B‑Hybrid. Trained on a 1T‑token dataset, Engram‑27B led across knowledge retrieval, reasoning and code benchmarks. "Infinite memory" experiments show embedding count growth follows a log‑linear trend, demonstrating Engram's scalability.
Industry Impact: Accelerating Edge AI and Global Competition
The paper's implications are multi‑faceted. First, it advances edge AI deployment: with a projected $100B edge AI market in 2026, Engram's low‑memory design could lower deployment barriers. Second, it reinforces DeepSeek's open‑source leadership: public code will accelerate community innovation. Environmentally, device‑side efficiency reduces energy use; an IEA report estimates AI electricity consumption reaching 1000 TWh in 2026, and Engram could save ~20%.
On China–U.S. AI competition, DeepSeek's rapid iteration challenges U.S. incumbents. Reuters notes that Chinese open‑source models could capture ~30% of global market share in 2026.
Challenges remain: compatibility across diverse hardware, optimization burdens for smaller companies, and growing regulatory scrutiny—e.g., EU AI Act requirements for edge AI transparency.
Conclusion: Liang's Innovative Legacy and a New Era for AI
Wenfeng Liang's successive publications cement DeepSeek's open‑source leadership. mHC and Engram respectively address training stability and on‑device inference, and their combination could enable efficient deployment of trillion‑parameter systems. The back‑to‑back papers suggest DeepSeek V3.4 or 4.0 may be forthcoming. This pace not only showcases rapid iteration in China's AI sector but also signals that open‑source AI in 2026 will reshape the global landscape. Liang's innovations are accelerating AI's transition from lab research to real‑world impact.
▶️ Like this article? Creating it takes a lot of work!
Subscribe to our YouTube channel:
AI Rosetta
Your likes, subscriptions and shares are our greatest support! 🙏
For more AI news, visit:
https://www.airosetta.com