NPU vs CPU vs GPU vs TPU: The Silicon Division of Labor Behind AI

01 Four chips, four jobs

Ask what hardware runs AI and the honest answer is: all of it, in different roles. The central processing unit runs the operating system and decides what happens next. The graphics processing unit turns models from research papers into products by doing arithmetic at enormous scale. The neural processing unit inside a phone sips battery while it enhances your photos and transcribes your voice. The tensor processing unit sits in warehouse-scale data centers and serves millions of inference requests for pennies.

None of these is simply "faster" than another. Each one is an argument about which bottleneck matters most. A CPU spends its transistors on caches, branch predictors and control logic so that a single stream of instructions finishes as quickly as possible. A GPU spends its transistors on thousands of small arithmetic lanes that all march in lockstep. An NPU spends them on a narrow set of matrix operations done with the least energy per operation. A TPU spends them on a fixed grid of multipliers big enough that entire neural network layers flow through it like water through a pipe.

The numbers in this piece are drawn from vendor-published specifications and public architectural papers, and we label them as approximate throughout. Peak figures on a datasheet describe what a chip can do in its best case, not what it does in your hand. That gap is itself one of the most important stories in AI hardware, and we return to it near the end.

02 The CPU: latency and logic

A CPU is a latency machine. Its whole design exists to make the next instruction dependent on nothing — to keep a single thread of control moving at the highest possible clock speed. Deep pipelines speculate on which way a branch will go, large caches keep the data a program just touched close at hand, and out-of-order execution reshuffles work to hide the time it takes memory to respond. All of that control machinery is what lets a CPU run a database, a browser, an operating system and a physics simulation on the same device.

That generality is exactly why a CPU is a poor engine for neural networks. Machine learning is, at computational bottom, repeated matrix multiplication: dense, regular, and with almost no branches at all. A modern CPU core can issue a handful of arithmetic operations per cycle and has at most a few dozen cores, so feeding a workload that wants billions of multiply-accumulates per second is like commuting a stadium crowd through a front door built for a family home. Vector extensions help, but the ratio remains roughly an order of magnitude or more against dedicated silicon.

What the CPU keeps is judgment. It launches the work, prepares the data, hands tensors to the accelerators, and processes the results. In every serious AI system the CPU is the conductor; it is just no longer the orchestra.

03 The GPU: throughput by brute force

The GPU was never designed for AI, which is part of why it conquered AI so completely. Graphics demanded the same thing neural networks demand: apply one small computation to millions of vertices or pixels at once. The processing model that resulted — thousands of threads executing the same instruction on different data, a scheme called SIMT — maps almost perfectly onto matrix math. Researchers noticed this in the late 2000s, general-purpose GPU computing became a discipline, and tensor cores later hardened the pattern into silicon.

The chart below compares representative peak throughput figures across chip classes, and it needs a warning label that we will put right on it: TOPS and TFLOPS are not the same unit, and neither are INT8 and FP16 the same precision. We therefore split the comparison into two panels, one per precision-and-unit combination, so that only like figures sit on the same axis.

Read the two panels together and a pattern appears. In raw arithmetic volume a datacenter accelerator dwarfs everything else, and a consumer GPU sits far above a phone NPU. But the phone NPU is not trying to win that race. It is trying to do its work inside a thermal envelope of a few watts, which is a different sport entirely — one measured in operations per joule rather than operations per second.

04 The NPU: why your phone needed a new brain

By the mid-2010s phone makers hit a wall that no amount of CPU or GPU speed could solve. Voice assistants listened continuously, cameras ran computational photography on every frame, and translation sat ready at any moment. Each of these is a neural network inference, and each must happen without draining the battery or scorching the chassis. A general-purpose core burns a large amount of control overhead to do math that is, from its point of view, numbingly repetitive. So the industry built a new block whose only job is that repetition: the neural processing unit.

Qualcomm's Hexagon NPU and Apple's Neural Engine are the two most widely shipped examples, and their design philosophy is the same: strip out branch prediction, cut precision to 8-bit and 16-bit where the network tolerates it, and lay out memory so the weights and activations flow without stalls. The payoff is efficiency. Vendor-published figures put a modern phone NPU in the tens of TOPS at single-digit watts, roughly an order of magnitude more operations per watt than a discrete GPU and two orders more than a CPU core at comparable precision.

There is a second dividend that does not show up in any efficiency figure: privacy and latency. When the inference happens on the device, your voice clip, your face and your documents never need to leave it, and the answer arrives without a network round trip. That is why on-device AI is not merely a battery optimization but a product strategy — and why laptop makers began marketing NPU throughput as a headline specification once operating systems started shipping local model features that demanded it.

05 The TPU and the ASIC logic of hyperscale AI

At the top of the scale the logic inverts. A company serving billions of inference requests per day knows exactly which workloads it runs, and when you know the workload you can stop being general on purpose. Google's Tensor Processing Unit is the canonical example: an application-specific integrated circuit whose heart is a systolic array — a large two-dimensional grid of multipliers where data pulses through in rhythm, weights flowing in one direction and activations in another, with each cell passing its result to its neighbor like a bucket brigade.

Systolic design trades flexibility for density. A CPU core moves data to the computation; a systolic array arranges the computation so the data barely has to move at all. Chained by the thousands into pods, these chips train and serve some of the largest models in existence. The cost is that an ASIC does one thing superbly and everything else not at all — which is fine when the one thing is your entire business, and ruinous when it is not.

The economics deserve emphasis because they explain the whole category. Designing an ASIC costs an enormous fixed sum, and it only pays off above a certain volume. Below that line, a GPU's generality wins because it amortizes its development cost across every customer's workloads. This is why the TPU exists inside one of the world's largest cloud operators and not inside your laptop.

06 How they share one device

Here is the part the acronym wars obscure: in practice these chips are not rivals but colleagues, frequently on the same piece of silicon. A modern phone system-on-chip contains a CPU cluster, a GPU and an NPU separated by a few hundred micrometers, and the operating system decides where each computation runs. Camera frames go to the NPU, game rendering to the GPU, and everything that is neither of those to the CPU. Frameworks such as Core ML and Android's neural network APIs exist precisely to route workloads to the right engine.

The data center tells the same story at higher volume. A single user request might involve a CPU service parsing and authorizing, a TPU or GPU array running the model itself, and a fleet of accelerators standing behind it. The laptop is becoming a third tier: local NPU-handled inference for small always-on models, cloud accelerators for frontier reasoning, and the CPU orchestrating both. The question "which chip runs AI?" has quietly become "which chip runs this layer of AI, and who decides?"

The decider, increasingly, is software. Compilers and runtime schedulers inspect each network layer and place it where the arithmetic, memory and power budget line up best. Heterogeneous computing is not a hardware trend so much as a scheduling problem that someone has to solve on every device, every second.

07 The memory wall that rules them all

Every chart in this article is a lie in one specific way: it shows what the compute units can do, and the compute units are rarely the bottleneck. In a classical processor architecture, memory and arithmetic live on separate chips connected by a comparatively narrow pipe, and moving a number across that pipe costs far more energy — and often more time — than doing the math on it once it arrives. Researchers call this the memory wall, and it has governed accelerator design for a decade.

The consequences are practical. A GPU with terabytes-per-second of bandwidth through stacked high-bandwidth memory can still idle its arithmetic lanes if a model's weights do not fit and must stream in mid-computation. A phone NPU with generous peak TOPS can be throttled by the modest memory of a handset. This is why so much published research is less about faster multipliers and more about squeezing more accuracy out of fewer bits: the cheapest byte to move is the one you never load.

It is also why measured performance routinely falls short of datasheet performance, often by a wide margin. Peak figures assume perfect data movement that no real workload achieves. When you compare two accelerators, the honest comparison is end-to-end time on the actual model at the actual batch size — a number no vendor puts on the box.

08 What the next node changes

Three currents are reshaping this division of labor. Manufacturing processes keep shrinking, so the same architecture yields more throughput per watt each generation, and advanced packaging lets designers stack memory on top of logic or stitch chiplets into composites that behave like one large die. Both attacks on the memory wall are physical: shorten the distance the data must travel and widen the road it travels on.

The second current is precision. Networks that once demanded 32-bit floats now run respectably in 8-bit integers, and research continues to push toward 4-bit formats. Every halving of numeric width doubles the effective arithmetic rate of the silicon you already own — the rare free lunch in computing, paid for by a small and carefully measured loss of accuracy.

The third current is the slow fading of the boundaries themselves. CPUs are growing matrix extensions; NPUs are absorbing more of the software stack; GPUs are adding dedicated AI paths; and the compilers keep getting better at hiding the seams. The four-letter taxonomy will not disappear — the physics it represents is too different — but the choice is moving out of the developer's hands and into the toolchain. That is the sign of a technology maturing: the division of labor stops being a decision and becomes an assumption.