On May 11, 2026, Kaiming He and his team at MIT posted the paper "ELF: Embedded Language Flows" on arXiv, proposing a new continuous diffusion language model. The core model ELF‑B uses only 105M parameters, was trained on 45B training tokens, and with 32 sampling steps reduced generative perplexity on the OpenWebText benchmark to 24, significantly outperforming a number of mainstream discrete and continuous diffusion language models (DLMs), including MDLM, Duo, FLM and LangFlow. This result quickly attracted industry attention as an important breakthrough for diffusion models in language, and provides a new route to challenge the GPT‑style autoregressive dominance.
Kaiming He, known for ResNet and MAE (Masked Autoencoders) in computer vision, successfully transferred mature diffusion and flow‑matching techniques from vision to language generation, demonstrating strong cross‑domain innovation.
Core innovation of ELF: from "discrete" to "continuous embedding space"
Traditional diffusion language models (DLMs) often operate in the discrete token space, facing issues such as discrete gradients and many sampling steps. ELF's core innovation is to place the diffusion process primarily in the continuous embedding space, mapping back to discrete tokens via a shared‑weight network only at the final step.
Design details include:
- Continuous‑time Flow Matching: Smooth flow transformations in embedding space to avoid the complexity of discrete noise prediction.
- Shared‑weight denoiser‑decoder: Greatly reduces parameter count (ELF‑B reduced from a potential 148M to 105M).
- In‑context conditioning: Efficiently integrates timestep and CFG scale conditioning without extra adaLN‑Zero modules.
- Classifier‑free guidance (CFG): Directly inherits mature techniques from image diffusion to improve quality and controllability.
Experiments show ELF‑B achieves better generation quality and diversity than larger baseline models under the same compute budget. The team also evaluated ELF‑M (342M) and ELF‑L (652M), demonstrating good scalability—larger models further reduce generative perplexity and improve text diversity.
Training data analysis: composition and efficient use of 45B tokens
The ELF models were trained on 45.2B tokens, far lower than the trillions used by mainstream large language models. The data mainly come from the classic OpenWebText (OWT) dataset.
Specific composition:
- The original OpenWebText contains about 9.04B tokens, mainly web pages linked from highly upvoted Reddit posts, including news, forum discussions, blogs and Wikipedia‑like content—relatively high‑quality general web text.
- The team trained on this dataset for 5 epochs (five full passes), resulting in an effective training volume of 45.2B tokens.
Why can ELF achieve leading results with only 45B tokens while mainstream LLMs require hundreds of billions or even trillions of tokens?
This is one of the most notable points of the ELF paper, mainly for the following reasons:
- Task objective differences: ELF focuses on unconditional text generation (generating coherent natural text from noise) rather than pursuing general intelligence, instruction following and complex reasoning, so its data needs are much lower than autoregressive LLMs that must capture vast world knowledge.
- Architectural efficiency: The continuous embedding space + flow matching design greatly improves data efficiency. Traditional discrete diffusion models typically require 500B+ tokens for good performance, while ELF achieves superior results with roughly one‑tenth the data.
- Small‑model characteristics: ELF‑B has only 105M parameters; small models inherently need less data, and optimizations like shared weights further amplify data efficiency.
- Focused experimental setup: The paper aims to demonstrate the feasibility and efficiency of continuous diffusion for language, so it uses standard DLM benchmarks rather than comprehensive general capability tests.
Notably, the team observed diminishing marginal returns from adding more data after 5 epochs on OWT, suggesting ELF's architecture is near its data‑efficiency limit under the current configuration.
Can training corpora be increased in the future?
The answer is clearly yes with great potential. The paper's ELF‑M and ELF‑L results show good scaling. Switching to larger and more diverse high‑quality corpora (e.g., The Pile, RefinedWeb, books, code, scientific literature mixtures) could significantly improve knowledge coverage, long‑text coherence, and instruction‑following. The open‑source code on GitHub allows the community to continue pretraining on larger corpora, leaving ample room for future development.
Performance and comparison with autoregressive models
On the OpenWebText benchmark, ELF‑B outperformed prior mainstream diffusion language models by a wide margin. It also shows strong diversity metrics, avoiding the mode‑collapse issues common in diffusion models.
Compared with GPT‑style autoregressive models, ELF uses parallel diffusion generation rather than left‑to‑right autoregressive prediction. This offers theoretical advantages in global consistency and potential for parallel computation, which may reduce accumulated errors and exposure bias. However, ELF is still at the research prototype stage, primarily validating unconditional generation; capabilities like instruction following and multi‑turn dialogue require further development.
Broader impact on AI research and industry
1. Diversifying generative modeling paradigms
Language generation has long been dominated by autoregressive Transformers. ELF's success proves diffusion models are viable and competitive in language, likely stimulating more research into non‑autoregressive and hybrid architectures. This will enrich the "species diversity" of AI models and provide better choices for different applications.
2. Significantly lowering the AI barrier and enabling technology democratization
With only 105M parameters and 45B tokens, ELF's training cost is far lower than mainstream large models. This offers high‑performance foundations for academia, SMEs and the open‑source community, promoting broader access to AI. Developers can reproduce and fine‑tune similar models on ordinary hardware, accelerating innovation.
3. Promoting cross‑domain technology transfer
ELF successfully transfers diffusion techniques from computer vision to language, reinforcing the trend toward "unified generative models." Future multimodal large models may more naturally integrate text, image and video generation; Kaiming He's team's cross‑domain experience is an important example.
4. Positive implications for compute and green AI
With training energy consumption a major concern today, ELF demonstrates a feasible path of "small models + efficient architectures." This may encourage the industry to shift focus from raw parameter scale to architectural and data efficiency, promoting more sustainable AI development.
5. Commercial and application prospects
For the open‑source community, ELF provides an efficient generation base; for industry it offers new tools for creative writing, parallel content generation, personalized education and other scenarios. In the long run, if diffusion language models surpass autoregressive models in long‑text coherence, creative diversity and controllability, they could reshape the content generation market.
Challenges and future outlook
Despite impressive results, ELF has limitations: it mainly validates unconditional generation; instruction following, long‑context understanding and multi‑turn interaction need strengthening; sampling steps have been optimized to 32 but remain far from real‑time interaction; comprehensive comparisons with GPT‑4, Claude and other very large models are still insufficient.
With expanded training data, instruction finetuning and multimodal integration, the ELF series could evolve from an academic prototype into a practical tool. Kaiming He's team's work again demonstrates that innovation does not necessarily come from larger scale, but often from smarter design.
▶️ Enjoyed this article? Creating content is hard!
Subscribe to our YouTube channel:
AI Rosetta
Your likes, subscriptions and shares are our greatest support! 🙏
More AI news at: https://www.airosetta.com