The NPU on Your Phone Is Mostly Idle — and That Is the Story

01 How TOPS Became the Headline Spec

Every computing generation picks one number to fight over, and the number is almost never the one that predicts user experience. Desktop PCs fought over megahertz until clock speed stopped translating into felt performance. Phones fought over megapixels, then core counts, then gigahertz again. The pattern holds because a single big number is easy to print on a box, and the box is where the sale happens.

The NPU inherited that role in the 2026 cycle. TOPS, trillions of operations per second, sounds like speed and photographs like progress: 35 on one flagship chip, 45 on the next. What the number actually measures is narrower. It is peak theoretical throughput at a chosen precision, usually INT8, on paper, at clock and voltage levels a phone cannot hold for long.

That distinction, ceiling versus sustained reality, is the entire subject of this analysis. Nothing on the spec sheet is false. It is a measurement of the best possible moment, marketed as though it were the average one.

02 What an NPU Physically Is

An NPU is a block of silicon specialized for one shape of math. Its core is an array of multiply-accumulate units wired so matrices flow through in a pipeline, which is exactly the operation behind every layer of a neural network. Around the array sits on-chip SRAM, small and fast, so intermediate results can be reused without round trips to main memory.

The trick is precision. NPUs do their work in quantized form, INT8 or even INT4 rather than 32-bit floats. Fewer bits mean less energy per operation and more operations per unit of area. A CPU is built for general branching code and a GPU for parallel throughput, but an NPU trades generality for operations per watt, the currency that matters inside a phone.

This is why the silicon itself is not the problem. For inference workloads that fit its constraints, an NPU is genuinely the right tool, often several times more efficient than the alternatives. The question is not whether the hardware works. It is how much of it the software ever asks to work.

03 The Utilization Gap

Set the two numbers side by side and the gap appears. Flagship phone chips now claim 35 to 45 TOPS of NPU throughput. The models actually deployed on those chips are documented at one to four billion parameters, quantized to INT4 or INT8, workloads that need only a few effective TOPS to run at interactive speed. The spec measures the ceiling; the software lives on the floor.

The first chart shows the claim side: vendor-disclosed peak INT8 TOPS, from the Apple A15 at 15.8 in 2021 to the Snapdragon 8 Elite at 45 in 2024. The second shows the demand side: parameter counts of models whose own documentation says they run on phones.

Both sets of figures are measured and sourced. The utilization gap between them is arithmetic, not opinion. What the gap means is interpretation, and a later section takes that up honestly, because the narrow truth is more interesting than the marketing version.

04 Why the Models Stayed Small

The constraint list starts with memory. A quantized 3-billion-parameter model needs a few gigabytes of live working set, and mobile DRAM bandwidth is a fraction of what a desktop GPU reads. Token generation is bandwidth-bound: every generated token streams weights past the compute array, so throughput tracks memory speed, not TOPS.

Then the body itself objects. Sustained matrix arithmetic heats a system-on-chip quickly, and phones manage heat by throttling rather than with fans. Battery budgets cap how long the NPU can run hard at all. Quantization softens the load but costs accuracy at the edges, and pushing below INT4 shows the damage. Each constraint alone is manageable; together they define a ceiling that frontier-scale models do not fit under.

That is why the frontier stayed in the cloud: datacenter memory bandwidth and power budgets are orders of magnitude larger. On-device models are real products, but they occupy the small, fast, private end of the range, and the physics that keeps them there is not scheduled to improve on a marketing calendar.

05 What Actually Runs on the NPU Today

The honest inventory is narrower than the keynotes but not empty. Photo and video pipelines use the NPU for segmentation, denoising, and semantic search over camera libraries. Speech runs locally: live transcription, translation, and voice-command recognition, where latency and privacy both favor staying on the device. Assistants do routing, deciding which requests are handled on the phone and which escalate to the cloud. Short micro-tasks, summarizing a notification, extracting a date, drafting a reply, run entirely on the chip.

Measured utilization is real but narrow: bursty workloads, short duty cycles, and models a fraction of the silicon's advertised size. Most claimed TOPS sit idle most of the time, because no shipped model asks for them.

Interpretation, clearly marked: the gap is not fraud, because peak specifications have always been ceilings. But when a marketing claim leans on a number the software never approaches, buyers are paying for headroom they cannot feel. That is a promise, not a spec, and it should be labeled as one.

06 The Memory Wall Nobody Puts on the Box

The binding constraint for larger on-device models is not compute; it is the memory path. Token generation streams the model's weights from DRAM for every single token, so end-to-end speed tracks bytes per second, not operations per second. A chip can post 45 TOPS and still spend most of its time waiting on a memory bus that feeds it a fraction of that.

On-chip SRAM helps, but only for the parts of a model that fit. A few tens of megabytes holds attention buffers and hot layers, not multi-gigabyte weight sets. That is why TOPS overstates real speed for exactly the workloads the TOPS number advertises: the bigger the model, the more time is spent moving data rather than multiplying it.

The idle silicon has a specific address, too. It is not that the array is broken; it is everything the array is waiting on. Headroom waits on bandwidth, on thermal limits, and on models sized to fit. None of those appears on the box, which is why the box and the experience disagree.

07 Conditions for Silicon to Earn Its Headline

Closing the gap has a concrete engineering list. Better quantization: 4-bit and mixed-precision schemes that keep accuracy while shrinking weight traffic. NPU-aware distillation: training small models for the array's shape rather than trimming down cloud giants. Aggressive speculative decoding: a tiny draft model proposing tokens that the larger model verifies, stretching the memory budget. And memory-path investment: wider buses, more SRAM, smarter caching.

The 2027 chip generations will show whether any of this lands. The signals to watch are bandwidth and SRAM capacity in the disclosures and, above all, the size of the models actually shipped on-device, which is the number that reveals what the silicon is really for.

Until then, the gap is a roadmap, not a scandal. The silicon is ahead of the software the way fast hardware usually is. TOPS became the headline because the real story, how much of the chip a phone actually uses, cannot be printed as one number. That is the number worth asking for.