Your Next Processor Upgrade Is an NPU: What On-Device AI Actually Does on a 2026 Phone

01 The features that actually shipped

The AI features most people actually touch on a 2026 phone are not open-ended chatbots. They are Circle to Search, which reroutes a long-press on the screen into a visual search query; Live Interpreter, which translates spoken conversation with a split-screen display; and Note Assist, which summarizes or restructures text inside the notes app. Samsung demonstrated all three in its Galaxy AI walkthroughs, and the pattern holds across the industry: search, translation, and summarization are the workloads that shipped.

That pattern is not an accident of taste. These tasks share a shape. Each takes a bounded input — a cropped image region, a sentence of speech, a block of text — and runs it through a compact neural network that produces a bounded output. They are measured in fractions of a second and tens of megabytes, not the billions of tokens a cloud chatbot streams. They are also the features phone makers can run partially or entirely on the device itself, which is where the silicon story begins. Samsung's own description of Galaxy AI notes that it is built on a hybrid of on-device and cloud processing, with the routine work kept local.

02 What an NPU actually is

An NPU — neural processing unit — is a chip block specialized for one thing: multiplying matrices, over and over, at very low precision. Neural network inference is, at its mathematical core, a long sequence of matrix multiplications. A CPU executes those operations on general-purpose logic and a GPU executes them in wide parallel batches, but an NPU hardens the multiply-accumulate step directly into silicon and connects it to memory paths sized for streaming model weights.

The low-precision part matters as much as the parallelism. Where a CPU may compute in 16- or 32-bit floating point, NPUs are tuned for INT8 and even INT4 integer math — a quantized representation in which each weight is stored as a small integer instead of a full floating-point number. Quantization shrinks model size by a factor of four or more and trades a small amount of accuracy for a large gain in speed and power. The result is that a quantized model can run on an NPU at a fraction of the energy the same work would burn on the CPU or GPU.

Architecturally, the NPU is one core inside the system-on-a-chip (SoC), alongside the CPU cores, the GPU, the DSP, the modem, and the image processor. ARM, whose processor IP underlies most phone SoCs, sells an NPU family (Ethos) alongside its CPU and GPU designs for exactly this purpose. The NPU is not a coprocessor bolted on later; it sits on the same die and shares the same memory system.

03 The TOPS race

Every phone launch now includes an AI throughput figure expressed in TOPS — trillions of operations per second. The marketing numbers have roughly doubled in two generations of silicon: Apple's A18 Pro is claimed at 35 TOPS of NPU throughput, Qualcomm's Snapdragon 8 Gen 3 at 45, MediaTek's Dimensity 9400 at 67, and the Snapdragon 8 Elite at 80.

These figures deserve a specific caveat: they are vendor-claimed peaks, measured under each vendor's own assumptions about precision, sparsity, and clock speed, and there is no neutral cross-vendor benchmark behind them. TOPS is peak math — what the engine does at full utilization, at the claimed precision, in a short burst.

What actually decides real-world throughput is sustained performance: how long the SoC can hold its NPU clocks before thermals force a slowdown, and how fast data can move between memory and the compute array. A phone chassis has no fan. Push an NPU at its peak number for more than a few seconds and the power draw will push it back down. The gap between the peak figure and the sustained figure is real, it varies by vendor, and it is rarely advertised.

04 Why memory bandwidth is the bottleneck

For the large end of on-device models — language models that generate text — the bottleneck is usually not the NPU's compute at all. It is memory. Text generation is sequential: the model produces one token, then feeds it back in to produce the next. Every one of those steps requires streaming the model's weights — often billions of parameters — from RAM into the compute array.

That makes inference on device memory-bound: the compute engine spends much of its time waiting for data to arrive, and the ceiling on speed is the memory bus, not the multiplier count. This is why two chips with similar TOPS can feel very different running the same local model, and why the memory generation on a phone is as consequential an AI spec as the NPU behind it.

The JEDEC standards body has roughly tripled peak LPDDR bandwidth across the relevant generations: about 34 GB/s for LPDDR4X (standardized around 2018), 51 GB/s for LPDDR5 (2020), 68 GB/s for LPDDR5X (2022), and 93 GB/s for LPDDR6 (2025). Measured against the growth in claimed TOPS, that curve is the quieter but arguably more decisive one for how large and how fast an on-device model can be.

05 On-device, hybrid, and cloud

Running AI on the device buys three things. Latency: no network round trip, so features like Interpreter feel immediate. Privacy: the raw text, image, or audio never leaves the phone. And offline function: translation and summarization keep working in airplane mode or on a congested network. The cost is capability — a model small enough to fit in a phone's memory is categorically less powerful than a datacenter model.

The industry's answer is the hybrid design Samsung describes for Galaxy AI: keep a small model resident on the phone for the routine, latency-sensitive work, and route harder prompts to large cloud models. The phone decides, per request, where the work runs. Users see one feature; underneath, two very different computers are involved.

The honest framing is that the on-device share of these systems is growing at the edge of what hardware allows, not because local models match cloud models. They do not. What the local model provides is the always-available layer — the parts of the feature that must work in half a second, with no signal, on battery power.

06 The costs nobody markets

Sustained inference is expensive in the currency phones care about. Running a language model locally for any length of time draws measurable power, and it does so from the same battery that has to survive the day. Short interactions — a summary here, a translation there — are fine. Extended sessions show up in battery drain in a way casual demo use never reveals.

Thermal limits follow from the same physics. A phone has no active cooling, so a long inference run heats the SoC until the governor throttles clocks — NPU and memory bandwidth included. Sustained throughput is therefore well below the launch-announcement peak, and the throttling point varies with case material, ambient temperature, and what else the phone is doing.

Then there is the resource claim that never appears on a spec sheet. Model weights consume storage — several gigabytes for a capable local language model — and the models are kept resident in RAM for fast response. Memory reserved for AI is memory apps cannot use, which is one quiet reason RAM baselines have climbed. None of this appears as a line item next to the TOPS figure.

07 Limits and trajectory

The current limits are concrete. Agent-style tasks — where the model plans and executes a multi-step chain: look up a flight, compare options, draft a message — multiply the memory and energy cost, because each step requires fresh inference over a growing context. On-device NPUs can do short chains today; long autonomous runs remain cloud territory until memory capacity and energy efficiency improve.

Meanwhile the NPU's job description keeps widening beyond chat-adjacent features. The same matrix engines increasingly drive the camera pipeline — computational photography runs noise reduction, HDR merging, and semantic segmentation through the NPU on modern phones — and modem-side modulation and signal processing are moving onto AI accelerators as well. The block that arrived to serve one feature set is becoming the general-purpose neural substrate of the device.

The reasonable read of the trajectory: through 2026 and 2027, NPU capability — TOPS, memory bandwidth, model residency — becomes the default benchmark conversation of phone launches, the way camera megapixels and core counts once were. Buyers would do well to treat claimed peaks as what they are, and to weigh the memory spec alongside them.