On December 20, 2025, NVIDIA researchers together with the MineDojo team officially released NitroGen, an open‑source vision‑to‑action foundation model. Designed for general‑purpose game agents, the model processes raw game pixels and directly outputs controller inputs, enabling zero‑shot play in over 1,000 games across genres such as action RPGs, platformers and racing. The work traces back to NVIDIA's April Fools' 2017 video "G‑Assist" — a joke concept of an AI that plays for you — which has now been realized through large‑scale behavior cloning. NitroGen demonstrates general intelligence in games and provides an extensible pathway for embodied AI and robotics, marking a new era of internet‑scale, data‑driven agent training.
NitroGen's development addresses a core bottleneck: traditional game AI depends on manual labels or bespoke simulators, leading to scarce data and poor generalization. The team used publicly available gameplay videos totaling 40,000 hours across 1,000+ games to automatically extract player input labels and build the first internet‑scale video‑to‑action dataset. The dataset is hosted on Hugging Face (nvidia/NitroGen) and contains 30,000 annotated clips and roughly 15 billion frames. Actions and metadata are stored in Parquet per 20‑second chunk (actions: 17 button booleans + 2 joystick [‑1,1] vectors; metadata includes game title, controller type and source URL). 846 games have over 1 hour of data, 91 exceed 100 hours, and 15 exceed 1,000 hours; action RPGs comprise 34.9% and platformers 18.4%.
The action‑extraction pipeline is a key innovation. In a fully automated flow, videos are sampled at 25 fps and SIFT/XFeat keypoint matching locates controller overlay layers (supporting Xbox/PlayStation and other types). The overlay is cropped and a hybrid classification‑segmentation network predicts joystick positions and button states. To prevent "cheating", the network infers only from the overlay image, masking the game screen. Quality control shows joystick correlation R² of 0.84 and per‑frame button accuracy of 0.96 against ground‑truth data, demonstrating reliable extraction. The pipeline requires no human intervention and is robust to compression artifacts and transparency variations common in real videos, greatly reducing annotation cost.
The model architecture builds on flow‑matching GR00T (originating from NVIDIA's robotics work) and has 493 million parameters. Input RGB frames are encoded by a pretrained SigLip2 visual Transformer, and outputs are produced by a diffusion‑matching Transformer (DiT) to generate action sequences (shape 21x16: two continuous joysticks + 17 buttons). Training uses pure behavior cloning (imitation learning) with large‑scale noisy data, requiring no rewards or task objectives, mirroring the "scale is capability" paradigm of language models. Inference is optimized for NVIDIA Hopper/Blackwell GPUs, supports Linux and Windows, and was tested on H100 hardware. The project also provides a general simulator: a Gymnasium API wrapper for Windows game binaries that supports one‑click control and rollout recording.
Performance benchmarks are impressive. The researchers built multi‑game benchmarks evaluating three task types (five rollouts per task) across visual styles including 3D action and 2D top‑down/side‑scrolling. The pretrained model shows nontrivial zero‑shot abilities such as high‑precision control in 3D combat and exploration in procedurally generated worlds. Transfer experiments are more striking: pretrained plus fine‑tuning on a new game yields up to a 52% increase in task success rate compared with training from scratch; low‑data settings (30 hours) see an average relative improvement of 10%, showing that scale injects transferable skills like navigation, combat and puzzle‑solving. Demo videos show the model playing The Witcher 3, Cyberpunk 2077 and Rocket League with fluid, human‑like inputs.
Open‑source assets are comprehensive: Hugging Face model weights (nvidia/NitroGen, NVIDIA non‑commercial license), dataset, evaluation suite and code are released. The project page (nitrogen.minedojo.org) includes a PDF of the paper detailing methods and results. Collaborators include NVIDIA (Loïc Magne, Linxi "Jim" Fan and others), Stanford (Anas Awadalla), and Caltech (Georgia Gkioxari). Limitations are clearly stated: focus on controller‑based games (weaker on mouse/keyboard genres like RTS/MOBA); noisy training data may limit generalization; users must provide their own game binaries.
The release has sparked discussion. VideoCardz called it "making G‑Assist real", Tom's Hardware highlighted robotics implications: games serve as rich simulators for training general embodied agents. Discussions on Reddit and X focus on potential uses: next‑generation NPCs, automated game QA and even browser/OS agents. Jim Fan posted about the release on LinkedIn and received broad engagement. Compared with OpenAI SIMA and DeepMind VPT, NitroGen pioneers an internet video pipeline with low cost and wide coverage.
NitroGen's impact extends beyond games. It validates an extensible path of "web scraping + automatic annotation + large‑scale cloning" that moves general agents from labs toward real‑world applications. Future directions include combining with reinforcement learning or multimodal inputs and extending to robot‑arm control. NVIDIA's release also reinforces its open‑source leadership and helps developers build the next generation of AI‑powered game ecosystems.
▶️ Enjoy this article? Creating it takes effort!
Follow our YouTube channel:
AI Rosetta
Your likes, subscriptions and shares are our biggest support! 🙏
More AI news at: https://www.airosetta.com