On May 11, 2026, Kaiming He and his team at MIT posted the paper "ELF: Embedded Language Flows" on arXiv, proposing a new continuous diffusion language model. The core model ELF-B uses only 105M parameters, was trained on 45B training tokens, and with 32 sampling steps reduced generative perplexity on the OpenWebText benchmark to 24, significantly outperforming a number of mainstream discrete and continuous diffusion language models (DLMs), including MDLM, Duo, FLM and LangFlow. This result quickly attracted industry attention as an important breakthrough for diffusion models in language, and provides a new route to challenge the GPT-style autoregressive dominance.

Kaiming He, known for ResNet and MAE (Masked Autoencoders) in computer vision, successfully transferred mature diffusion and flow-matching techniques from vision to language generation, demonstrating strong cross-domain innovation.

Core innovation of ELF: from "discrete" to "continuous embedding space"

Traditional diffusion language models (DLMs) often operate in the discrete token space, facing issues such as discrete gradients and many sampling steps. ELF's core innovation is to place the diffusion process primarily in the continuous embedding space, mapping back to discrete tokens via a shared-weight network only at the final step.

Design details include:

Continuous-time Flow Matching: Smooth flow transformations in embedding space to avoid the complexity of discrete noise prediction.
Shared-weight denoiser-decoder: Greatly reduces parameter count (ELF-B reduced from a potential 148M to 105M).
In-context conditioning: Efficiently integrates timestep and CFG scale conditioning without extra adaLN-Zero modules.
Classifier-free guidance (CFG): Directly inherits mature techniques from image diffusion to improve quality and controllability.

Experiments show ELF-B achieves better generation quality and diversity than larger baseline models under the same compute budget. The team also evaluated ELF-M (342M) and ELF-L (652M), demonstrating good scalability—larger models further reduce generative perplexity and improve text diversity.

Training data analysis: composition and efficient use of 45B tokens

The ELF models were trained on 45.2B tokens, far lower than the trillions used by mainstream large language models. The data mainly come from the classic OpenWebText (OWT) dataset.

Specific composition:

The original OpenWebText contains about 9.04B tokens, mainly web pages linked from highly upvoted Reddit posts, including news, forum discussions, blogs and Wikipedia-like content—relatively high-quality general web text.
The team trained on this dataset for 5 epochs (five full passes), resulting in an effective training volume of 45.2B tokens.

Why can ELF achieve leading results with only 45B tokens while mainstream LLMs require hundreds of billions or even trillions of tokens?

This is one of the most notable points of the ELF paper, mainly for the following reasons:

Task objective differences: ELF focuses on unconditional text generation (generating coherent natural text from noise) rather than pursuing general intelligence, instruction following and complex reasoning, so its data needs are much lower than autoregressive LLMs that must capture vast world knowledge.
Architectural efficiency: The continuous embedding space + flow matching design greatly improves data efficiency. Traditional discrete diffusion models typica