Do We Really Need NPUs? The Honest Case for Smartphone Neural Accelerators

01 What an NPU actually is

A neural processing unit is a block of silicon inside a system-on-chip that is built to do one thing well: multiply huge grids of numbers at low power. Modern AI models, including the convolutional networks behind camera features and the transformer models behind language tools, ultimately reduce to chains of matrix multiplications. A CPU can perform these operations, and a GPU can perform them faster still, but both are general-purpose machines carrying overhead in instruction decoding, cache management, and scheduling that a dedicated circuit does not need.

An NPU strips that overhead away. It is built from dense arrays of multiply-accumulate units that operate on many values in parallel, wired to memory in a pattern that matches how neural networks read data. The result is that arithmetic which would drain a battery if run on the CPU can execute on the NPU at a fraction of the energy. Industry figures from Qualcomm, Apple, and MediaTek consistently claim an order-of-magnitude efficiency advantage for neural workloads on the dedicated block compared with the CPU performing the same task.

It is worth being precise about what the NPU is not. It is not a small brain, it does not learn on its own in any meaningful sense while you use it, and it does not contain the intelligence of the features it powers. The intelligence lives in the model weights, which are trained in data centers and shipped to the device as files. The NPU is the engine that runs those files. That distinction matters because it explains a recurring pattern in the industry: chips advertise enormous neural compute, but the experiences users notice depend more on the quality and size of the models shipped than on raw accelerator throughput.

The category names differ by vendor. Apple calls its block the Neural Engine, Qualcomm builds its into the Hexagon processor family, and MediaTek labels its version an APU or NPU. The engineering philosophies differ in detail, but the underlying idea is shared: a fixed-function, arithmetic-dense, power-efficient coprocessor for tensor workloads. Once that framing is clear, the honest question is not whether the hardware is real, but whether the workloads justify it. That is the question this analysis takes up in the sections that follow.

02 The workloads that really use it

The strongest argument for NPUs is not hypothetical future AI. It is a set of features that already run hundreds of times a day on ordinary phones. The most demanding is computational photography. Every time you press the shutter, the phone captures a burst of frames and runs them through neural networks that fuse exposures, reduce noise, fix faces, and correct colors. This happens in well under a second, which means the neural arithmetic must complete on-device and on battery, exactly the regime where the NPU's efficiency matters.

Speech is the second pillar. Voice-to-text transcription, wake-word detection, and real-time dictation all lean on acoustic models. Keeping these on-device is what allows a phone to respond to spoken commands without streaming your microphone to a remote server. Live translation follows the same pattern: the modern implementations that translate conversations with a second or two of delay run compact models locally, with the network path reserved for harder phrases.

The third cluster is more visual and more continuous. Portrait-mode depth estimation, video background blur for calls, face unlocking, and the subject segmentation behind photo editing tools all depend on neural inference that must run frame after frame, in real time, while the device is unplugged. These are precisely the workloads that would be punishing on a CPU, because the sustained throughput requirement collides with a strict thermal budget in a passively cooled slab of glass.

What is notable by 2026 is how much of this predates the current AI hype cycle. These features shipped years before chatbots arrived on phones, and they shipped because silicon vendors had already solved the efficiency problem. The NPU earned its place on the die through camera and speech workloads, not through generative demos. That history is the strongest rebuttal to the claim that NPUs are pure marketing, and it is also the reason the newest wave of on-device assistants can exist at all: the substrate was already there, waiting for models large enough to use it.

03 The TOPS race and what it hides

Since roughly 2018, the number every chip vendor puts at the center of its launch is TOPS, tera-operations per second, a measure of peak neural arithmetic throughput. The trajectory is steep. Qualcomm's Snapdragon 855 era carried single-digit TOPS figures; by the Snapdragon 8 Gen 3 generation, commonly cited claims sat around 45 TOPS; the Snapdragon 8 Elite is presented with roughly 80 TOPS of claimed NPU capability. Apple's Neural Engine has climbed from about 5 TOPS in the A12 to roughly 35 TOPS by the A17 Pro generation, with commonly cited figures for later parts remaining in that band. MediaTek's flagship APUs follow the same slope.

The problem is that TOPS is a peak laboratory number, measured on dense, well-behaved workloads with the memory system cooperating. A phone in a pocket rarely reproduces those conditions. Real neural workloads are frequently limited not by arithmetic units but by memory bandwidth, by how fast weights and activations can be moved into the accelerator. A network that reads a lot of data relative to the computation it performs may leave most of those advertised TOPS idle. Sustained performance is further limited by heat: a ten-second burst can run near peak, but a minute-long transcription task must throttle back.

The gap between advertised and realized throughput is not a scandal, but it is a measurement problem that favors marketing over engineering. Two chips with identical TOPS ratings can deliver meaningfully different real-world results depending on memory subsystems, software stacks, and how well the shipped models map onto the hardware. Vendors also count operations differently, some counting a multiply-add as one operation and others as two, which can double a headline figure without changing the silicon at all.

The fairest reading is that TOPS growth is real but its interpretation has drifted. The numbers track genuine architectural expansion, as the chart below shows, yet they say little about the experience of holding the phone. Utilization, not capacity, is the number that would actually inform a buying decision, and no vendor publishes it.

04 Local versus cloud: the economics of inference

The second honest argument for NPUs is economic. Every AI query has to run somewhere, and both options carry costs. Running inference in a cloud data center means the vendor pays for servers, electricity, and network transit, often for every single user interaction with a free feature. Running it on the phone shifts that cost onto silicon the customer has already purchased. This is not a subtle accounting point: it is the structural reason every major platform operator spent the years leading into 2026 pushing models onto devices. A feature that costs a fraction of a cent per query in the cloud can cost effectively nothing when the phone does the arithmetic.

The energy ledger cuts the same way from the user's perspective. Wireless transmission is expensive per bit, and radio wake-ups are a major battery drain. If a task can complete without the network, the phone avoids powering the modem, holding a connection, and waiting on a round trip. Published research on large language model inference suggests a single cloud query for a large model can consume on the order of a few watt-hours in the data center, before counting the radio on the phone. A compact model executed locally by an efficient NPU works with orders of magnitude less energy at the point of use, since it touches no network at all.

Latency favors local execution even more plainly. A cloud round trip means the phone must transmit, wait for a data center to run the model, and receive the answer; total response times for conversational queries are commonly measured in hundreds of milliseconds to seconds, and they vary with network conditions. A local NPU running a compact model can return results in tens of milliseconds, and it can do so in a basement or an airplane. Deterministic, immediate response is also what makes real-time features like translation and background blur feasible at all, since no network path can sustain frame-rate inference.

The honest caveat is that local models are smaller models. The quality gap between what fits on a phone and what runs in a hyperscale data center remains large, and the practical strategy most platforms have converged on is hybrid: keep routine, latency-sensitive, and private workloads local, and send genuinely hard problems to the cloud. The NPU is what makes the local half of that bargain worth having.

05 Privacy, latency, and battery: the steelman case

Strip away the TOPS arms race and the NPU case rests on three pillars: privacy, latency, and battery. The privacy argument is the most concrete. When inference happens locally, the raw data never leaves the device. Your keystrokes, your microphone input, your photographs, and your messages are processed by silicon in your hand and are not transmitted to anyone's servers. This is a structural guarantee, not a policy promise, and the difference matters. A privacy policy can be rewritten; a computation that never leaves the phone cannot be intercepted, subpoenaed, or monetized by a third party.

Latency is the second pillar, and it compounds. Every cloud interaction stacks transmission time, queueing, model execution, and the return trip; each hop adds variance. Local inference has a fixed, predictable cost, and its consistency is what makes interactive features feel instant rather than merely fast. Anyone who has used both a local transcription tool and a cloud-based one on a weak connection understands this immediately: the local tool is not just quicker on average, it is reliable in a way a networked tool cannot be.

Battery is the third pillar and the least appreciated. The NPU's efficiency is not primarily about doing more AI. It is about doing the AI the phone was already doing at lower cost. Computational photography, always-on listening, and video effects ran on phones before NPUs matured, and they ran hotter and drained faster. Each generation of accelerator reduces the energy per operation, which means the same features consume less of your charge. The benefit shows up not as a new feature but as the absence of a battery hit at the end of a photo-heavy day.

Taken together, these three pillars form a genuine engineering argument that would hold up even if no generative AI feature had ever shipped. The NPU is the hardware expression of a simple principle: workloads that are continuous, latency-sensitive, and personal should run as close to the user as possible. The steelman case does not need futuristic demos. It rests on the camera app you already use.

06 The honest limits nobody advertises

The case for NPUs is real, but the honest ledger needs a spending side too. The first cost is silicon area and money. An NPU occupies a meaningful fraction of a flagship chip's die, and that area is paid for by every buyer, including those who never run anything more neural than face unlock. Fixed-function silicon is a bet on a workload pattern; if the software ecosystem fails to exploit the block, the transistors sit idle. There is no way to repurpose a neural accelerator for something else.

The second limit is software, and it is the industry's quiet bottleneck. An NPU delivers nothing until a model is converted, quantized, and mapped onto its architecture, and that work is vendor-specific. A model optimized for one vendor's NPU may run poorly on another's, which fragments the ecosystem and channels effort away from features and toward porting. The vendor toolchains have improved substantially, but the fragmentation cost remains real, and it is borne across the entire industry rather than by any single vendor.

The third limit is the quality ceiling. Phone NPUs run compact models, and compact models are weaker models. On-device generative features in 2026 are impressive for their size, but they are not interchangeable with what the cloud can offer, and the gap is obvious to anyone who compares them side by side. Vendors close the gap with hybrid approaches that quietly hand hard queries to servers, which softens the local-first privacy story in the process. The marketing presents the on-device number; the experience frequently includes both paths.

Finally, the TOPS race itself has a distorting effect on product decisions. When the headline metric is accelerator throughput, silicon teams have an incentive to grow the block even past the point of useful workload coverage, and marketing teams have an incentive to claim the capabilities of cloud models while crediting the phone. The result is a widening spread between what the specification sheet promises and what the device does. Recognizing that spread, without denying the genuine progress underneath it, is the entire discipline of reading NPU claims honestly.

07 So do we need them?

The fairest verdict is that NPUs earned their place on the die before the AI hype arrived, and the hype has extended their mandate beyond what current software reliably exploits. The camera-and-speech case is settled: those workloads are continuous, battery-constrained, and latency-sensitive, and no alternative execution site competes. For those features, the answer to the question in the headline is unambiguously yes, and it has been for several hardware generations.

The generative case is younger and less settled. On-device assistants, summarization, and image editing are genuinely useful, and the NPU is what makes them viable offline and affordable at scale. But their quality depends on model size, model size depends on memory, and memory is the constraint that TOPS figures quietly ignore. A chip with double the neural throughput and the same memory bandwidth often cannot run a meaningfully better model. For this class of work, the NPU is necessary but not sufficient, and buyers should weight RAM and sustained performance at least as heavily as accelerator arithmetic.

For a buyer in 2026, the practical guidance is straightforward. Ignore peak TOPS as a differentiator at the flagship level, where every current chip has far more neural capacity than typical workloads demand. Instead, look at sustained performance, memory, and the software the vendor actually ships, because those determine what the accelerator does for you. Among mid-range chips, an NPU still matters, because it is what lets cheaper phones run the same camera and speech pipelines without melting or draining.

The skeptical framing survives contact with the evidence, but only in a qualified form. NPUs are not a scam, and they are not optional anymore; they are the reason a modern smartphone's most-used AI features are fast, private, and battery-affordable. What deserves skepticism is not the hardware but the measurement culture around it. The accelerators are real, the workloads are real, and the peak TOPS race is a poor proxy for both. Buy the phone for what it sustains, not for the number on the slide.