The NPU Explained: How a Silent Chip Upgrade Put AI on Every Device

01 The Problem: AI Everywhere, Hardware Nowhere Ready

When AI features began landing in consumer software a few years ago, they arrived on hardware that was never designed for them. Running a neural network means performing billions of repetitive arithmetic operations per second, and the general-purpose CPU handled that work the way a hatchback handles towing: technically possible, thermally embarrassing. The quick fix was the GPU -- a massively parallel processor borrowed from gaming -- and for cloud data centers, stuffed with accelerator cards and wall power, the GPU fix worked.

On a laptop or a phone, the physics fell apart. A discrete-class GPU workload draws tens of watts; a phone has a power budget closer to a few. Every on-device AI feature that ran through the GPU arrived with a triple cost: battery drain, heat, and fans spinning up to run something as trivial as background blur on a video call. The other alternative -- shipping every request to the cloud -- carries its own costs: latency, privacy exposure, subscription fees, and the simple fact that a data center cannot blur your webcam when your train goes through a tunnel.

The industry's answer was not a better version of the existing processors. It was a new component, purpose-built for one class of arithmetic, sitting quietly next to the CPU and GPU. Almost every flagship phone and premium laptop sold today contains one, and most buyers have never heard its name.

02 What an NPU Actually Is

The standard definition is broader than the marketing. A neural processing unit, also known as an AI accelerator or deep learning processor, is a class of specialized hardware accelerator designed to speed up AI and machine-learning applications, including neural networks and computer vision. NPUs can appear as standalone chips, as blocks inside a CPU package, or as regions of a GPU -- the category describes the function, not a single form factor.

What distinguishes an NPU from the processors around it is specialization. A CPU is optimized for control flow: few operations, unpredictable branches, low latency per instruction. A GPU is optimized for throughput across many parallel threads, but remains general-purpose enough to render graphics. An NPU throws away generality entirely. It implements, directly in silicon, the exact operation that dominates neural network inference -- the multiply-accumulate -- and organizes the whole chip around doing nothing else, at enormous scale, for the least possible energy.

The names vary by vendor: Intel markets its NPU as the "neural engine" inside its AI PC platforms, Qualcomm's Snapdragon chips carry the Hexagon NPU, Apple calls its block the Neural Engine, and AMD ships an XDNA-based NPU under the Ryzen AI brand. Different labels, same design philosophy: fixed-function, massively parallel, miserly with watts.

03 The Efficiency Gap, Visualized

The honest way to compare CPU, GPU, and NPU for AI workloads is energy: operations per joule. The precise numbers vary by part, precision, and workload, so the defensible claim is about the shape of the gap rather than exact ratios -- but the shape is dramatic. For the low-precision arithmetic that dominates inference, an NPU typically delivers an order of magnitude or more additional operations per watt compared with a general-purpose CPU, and a substantial multiple over a GPU running the same quantized workload.

The mechanism behind that gap is worth understanding, because it is the same engineering trick underneath almost every NPU on the market: strip the processor down to the one operation neural networks actually need, and spend every transistor and every joule on that operation alone.

04 Inside the Machine: Multiply, Accumulate, Repeat

Neural network inference is, at the arithmetic level, mostly one thing: multiplying pairs of numbers and adding the results together -- the multiply-accumulate, or MAC, operation. A single layer's computation can be expressed as thousands of independent MACs, which is why the work parallelizes so beautifully. The heart of an NPU is a MAC array: a large grid of processing elements, each of which contains dedicated multiply-add logic, all fed simultaneously and working in lockstep.

Two further tricks multiply the array's advantage. The first is quantization: instead of the 32-bit or 16-bit floating-point numbers used in training, inference can usually run on 8-bit integers with negligible accuracy loss for most consumer tasks. An int8 multiply consumes a fraction of the power and silicon area of a float multiply, and NPUs are built to exploit that directly -- their headline throughput figure, measured in TOPS (trillions of operations per second), is quoted for low-precision arithmetic. The second is on-chip memory: an NPU places a large bank of fast SRAM adjacent to the MAC array, so activations and weights do not have to make repeated round trips to main memory. Moving data costs more energy than computing on it, and NPUs are engineered around keeping the data close to the arithmetic. A real production array contains thousands to tens of thousands of processing elements, not the nine in the diagram -- the scale is the product.

05 What Changed for Your Devices

The practical consequence of that silicon is a new class of always-on AI features that batteries can actually afford. Phone NPUs quietly handle face recognition, photo enhancement, live speech transcription and translation, and real-time background blur, without shipping your camera feed to a server. Laptop NPUs do the same for video call effects, local transcription, and productivity features, while offloading the CPU and GPU entirely -- which is why an AI video feature can now run for hours on a charge instead of draining the battery flat in one meeting.

The bigger shift is local large language models. A quantized model of a few billion parameters fits into the memory of a premium phone or laptop, and an NPU plus a fast GPU make inference on it usable -- this is precisely what Alex Ziskind's video demonstrates, running LLMs locally on a modern laptop faster than many desktops manage. Local inference buys privacy (the prompt never leaves the device), zero marginal cost, and offline availability, at the price of running smaller models than the frontier cloud systems. "AI PC" as a marketing category is essentially a contract about this capability: vendors now advertise NPU throughput figures in TOPS as a headline specification, the way they once advertised clock speeds.

The branding is ahead of the software, as branding usually is. But the direction is clear and, unusually for a marketing category, physically grounded: the NPU made on-device AI cheap in the one currency that matters on mobile hardware -- energy.

06 Where the Ceiling Sits

The NPU's limits are as structural as its advantages. The first ceiling is model size. A language model must sit entirely in memory to run, and consumer devices ship with 8 to 16 gigabytes of RAM shared between the OS, applications, and the accelerator. That caps practical local models at the few-billion-parameter class; frontier-scale models do not fit and will not fit soon, because the problem is physical memory capacity, not NPU throughput. The chip cannot accelerate data it cannot hold.

The second ceiling is memory bandwidth. Once a model fits, inference speed is largely governed by how fast weights can stream from memory to the arithmetic units -- a quantity NPUs only partially control, since they share the same memory subsystem as everything else in the device. This is why TOPS figures alone are a poor guide to real LLM performance, and why a laptop with modest NPU throughput but fast memory can outperform a nominally faster machine. Marketing quotes the MAC array; users feel the bandwidth.

The third ceiling is workload fit. The NPU's specialization is its weakness: it excels at dense, low-precision, repetitive arithmetic and gains little on the sparse, irregular, or high-precision work that some models require. The fixed-function gamble also ages faster than general-purpose silicon -- an NPU designed around the inference patterns of one model generation may be poorly matched to the next. On-device AI is therefore best understood as the tier for responsive, private, always-on features on smaller models, with large and frontier workloads remaining a cloud tier. The NPU did not replace the data center; it redrew the boundary between what stays on your desk and what leaves.