In AI, many systems excel at recognizing image content, performing classification, and detecting objects, but truly understanding the underlying physical principles of object motion, collision, and occlusion is rarer. Recently, a model from the Meta team, V‑JEPA (Video Joint Embedding Predictive Architecture), demonstrated it can 'learn' physical intuition from ordinary video and exhibit 'surprise' responses to events that violate physical laws. The model does not assume any physical rules in advance; it builds this intuition by observing video data.
Below we first introduce V‑JEPA's basic architecture, then discuss its performance on physical‑intuition tasks and the later upgraded V‑JEPA2 version.
V‑JEPA architecture and training strategy
From pixel space to latent representation
Traditional video‑understanding models often predict missing pixels directly in pixel space—that is, they try to fill in or infer occluded or absent pixel values. This approach can easily get bogged down by too many low‑level details and noise (for example, wind‑blown leaves or lighting changes).
The key idea of V‑JEPA is that it does not reconstruct at the pixel level but predicts in a latent space. In other words, the model first encodes video frames into more abstract, compact representations that retain only the 'important' information, and then makes predictions in this abstract space.
Specifically, the V‑JEPA architecture can be divided into three parts: Encoder1, Encoder2, and Predictor.
- During training, a set of video frames is first masked. The masked frames are fed into Encoder1, producing their latent representations.
- At the same time, the unmasked frames (original frames) are fed into Encoder2, yielding another set of latent representations.
- Finally, the Predictor takes the latent representations from Encoder1 as input and predicts the latent representations produced by Encoder2. In other words, the model does not try to reconstruct masked regions at the pixel level but aligns predictions in the latent space. In this way, it learns to use temporal coherence and dynamic information between frames to infer future or hidden content. Quanta Magazine
Because the model only needs to learn more 'salient' abstract information (such as object positions, velocities, and occlusion relationships), it can 'ignore' many irrelevant textures and lighting details. As one researcher put it, the model can 'discard unnecessary information and focus on the more important parts of the video.'
After this pretraining phase, the model can be fine‑tuned with labeled data for specific tasks (e.g., action classification, object recognition). Compared with training end‑to‑end from scratch, V‑JEPA's approach usually requires fewer annotated examples.
Testing physical intuition: surprise signals and the IntPhys benchmark
One question for V‑JEPA is whether it can show a 'surprise' response when observing scenarios that defy physical intuition. For example, if an object that should reappear after occlusion does not appear, can the model detect the anomaly?
The Meta team evaluated the model on the IntPhys benchmark (Intuitive Physics benchmark). In this test, the model must judge whether events in a video are physically plausible or implausible. V‑JEPA achieved close to 98% accuracy on this benchmark. By contrast, a model that predicts in pixel space performed barely better than random guessing.
The team also defined a 'surprise' metric: they measure the error between the model's predicted latent representation for future frames and the actually observed latent representation as a 'surprise signal.' Large errors indicate the model was 'confounded' by unexpected events. Experiments showed that when physically impossible phenomena occur in a video (for example, an object failing to reappear after occlusion or a sudden change in trajectory), the model's prediction error spikes sharply—similar to the human experience of surprise when witnessing abnormal physical behavior.
For example, if a ball rolls behind an occluder and disappears, the model expects to see the ball reappear in future frames; if it does not, the model produces a large error and registers 'surprise.'
This anomaly‑detection capability is one way the model manifests a learned physical intuition.
Because of this surprise mechanism, V‑JEPA shows some generality in judging physical plausibility and can detect physical anomalies across different types of videos.
V‑JEPA2: extended model and robot applications
Building on V‑JEPA, the Meta team released V‑JEPA2 in June 2025. V‑JEPA2 is a larger and more applied version: the model has 1.2 billion (1.2B) parameters and was pretrained on about 22 million videos.
They also explored applying V‑JEPA2 to robotic tasks. Concretely, by fine‑tuning a new predictor network with roughly 60 hours of robot operation data, the network was used to predict a robot's next action in an environment. Combining video with robot action‑environment interaction recordings, V‑JEPA2 was used to plan future robot behavior and succeeded on some simple manipulation tasks.
However, on a more challenging physical‑intuition benchmark (referred to as IntPhys 2), V‑JEPA2 and other models performed only slightly above chance. A major bottleneck is that the model currently handles only a few seconds of video input and predicts frames only a few seconds ahead; longer‑horizon prediction remains weak. The Meta team jokingly described this 'memory' as a 'goldfish memory'—it quickly forgets the distant past.
In addition, the model currently lacks an effective encoding of predictive uncertainty. If historical frames do not provide enough information for precise prediction, that uncertainty is not well quantified yet.
Summary and outlook
V‑JEPA's demonstrated abilities have sparked questions about whether AI can form physical intuition 'bottom‑up.' Instead of relying on preset physical formulas, it learns certain physical models at the latent level through spatiotemporal correlations in video. Its 'surprise' mechanism also offers a new approach to anomaly detection and physical‑plausibility judgment.
Nevertheless, it has many limitations: short‑term memory, weak handling of uncertainty, and challenges in reasoning about complex scenes and longer time spans. Extending from video understanding to robot control will require integration with action planning, sensor feedback, and real‑time control modules.
Future research directions may include extending temporal horizons, introducing probabilistic/uncertainty modeling, strengthening ties with causal inference, and embedding such physical‑intuition models into robotic systems to build agents with general physical common sense.
Reference papers (V‑JEPA / V‑JEPA2)
- V‑JEPA paper (Meta team, Video Joint Embedding Predictive Architecture):
Revisiting Feature Prediction for Learning Visual Representations from Video – arXiv:2404.08471
- V‑JEPA2 paper (1.2B parameter version + robot applications):