Dec. 5, 2025 — Ars Technica published a report noting that while mobile NPUs' peak TOPS, energy efficiency and support for dedicated operators are steadily improving, these hardware advances have not translated directly into large jumps in on‑device AI capability.
The article lists several specific impediments. First is model size and memory: even if an NPU executes matrix multiplications very efficiently, main memory to NPU bandwidth and on‑chip SRAM capacity remain limited. Most modern Transformer models generate O(n²) intermediates in attention layers, which causes memory usage to spike during inference. Engineering teams often must prune, shard, or offload parts of the model to the cloud, sacrificing latency, privacy or offline availability.
Second are quantization and precision trade‑offs. To fit mobile constraints, researchers and vendors use 8‑bit or even 4‑bit quantization, distillation and pruning. However, the article notes that on some tasks these techniques can produce noticeable quality degradation, especially for long contexts, generative text or vision tasks that require fine‑grained inference. While NPU instruction sets are optimized for low‑bitwidth operations, algorithmic errors cannot be remedied by hardware alone.
Third is software and operator compatibility. NPU vendors (such as Apple, Qualcomm and Google) provide different SDKs and drivers, and migrating a model from a research prototype to a real device involves extensive manual adaptation, operator fusion and custom kernel development. Ars Technica notes that compilers still cannot reliably map complex operators in high‑level frameworks (PyTorch, TensorFlow) into efficient NPU pipelines, so much potential performance is lost in implementation details.
Fourth are power and thermal limits. A phone's power budget is far lower than a data center's, and sustained high load triggers thermal throttling. Even if NPU peak performance is high, sustained throughput is constrained; vendors therefore design trade‑offs between short bursts and long‑term stability, which directly affect experiences like continuous conversation and real‑time translation.
The report also highlights several approaches under exploration: structured pruning and use of sparse matrices, mixed‑precision dynamic inference, hierarchical execution (part on edge, part in cloud), and more advanced compiler technologies (for example, MLIR‑based operator fusion and pipeline scheduling). Additionally, model designers are increasingly favoring hardware‑aware co‑design, creating architectures specifically suited for mobile inference.
Ars Technica's key point is that NPU progress is necessary but not sufficient. Improving on‑device AI meaningfully requires coordination across model architecture, quantization strategies, memory management, compilers and operating system layers. For users, this means that in the short term more perceptible improvements are likely to come from software optimizations and product‑level feature integration (such as faster on‑device speech recognition or image post‑processing) rather than from raw peak TOPS numbers alone.
Looking ahead, the article argues that only when hardware, software and model design truly work in concert and achieve unified optimization across on‑chip storage, bandwidth, instruction sets and development toolchains will the AI experience on phones undergo a qualitative change.