How AI Chips Work: Inside the Neural Engine in Your Phone

01Why phones needed a new kind of chip

For most of computing history, processors got faster by doing the same work more quickly. That model broke down when machine learning arrived on phones. Neural networks do not need a fast general-purpose core; they need the same small arithmetic operation, repeated billions of times, on huge blocks of numbers. A conventional CPU spends most of its energy fetching instructions and moving data around, not on the math itself.

A GPU does better, because it is built to grind through thousands of similar calculations in parallel. But a GPU is designed for graphics, where flexibility matters more than efficiency. Running a language model on a phone with the GPU alone drains the battery and heats the case. The industry's answer was a third processor: a neural processing unit, tuned for exactly one job — the linear algebra that neural networks run on.

02The NPU: matrix math as a hardware circuit

An NPU, sometimes called a neural engine or AI accelerator, is a chip block whose data path is shaped like the math it performs. Where a CPU has branch predictors and caches optimized for unpredictable code, an NPU has wide buses, large on-chip memories, and thousands of small arithmetic units wired together. The workload is predictable: layers of matrix multiplications followed by simple nonlinear functions, in a fixed order.

That predictability is the whole trick. Because the chip knows what is coming, it can stream data continuously, keep weights close to the arithmetic units, and skip the overhead that makes general processors flexible. Apple has shipped a Neural Engine in its A-series chips since 2017, and Qualcomm, MediaTek, and Samsung all now devote significant die area to similar blocks. The result is that a modern phone can run image models, transcription, and small language models without waking the CPU's big cores.

03MAC arrays: multiply-accumulate at massive scale

The heart of nearly every NPU is the multiply-accumulate, or MAC, operation: multiply two numbers, add the product to a running total. A single neuron in a neural network is essentially a long chain of multiply-accumulates — each input times its weight, summed into one output. Nearly all of a network's computation reduces to this one primitive, repeated.

NPUs therefore build enormous grids of MAC units, sometimes thousands operating in lockstep. Input values flow in from one side, weights from another, and each intersection performs a multiply and passes the result onward, with adder trees collapsing partial sums into final outputs. The grid's regularity means data flows like water through pipes rather than being fetched piecemeal. This is also why the design of the on-chip memory matters as much as the arithmetic: keeping thousands of units fed with activations and weights, without stalling, is the real engineering challenge.

04Quantization: shrinking models to fit on silicon

A neural network trained in the lab usually stores every weight as a 32-bit floating-point number. That is precision the hardware rarely needs for inference. Quantization compresses those weights down to 8-bit integers, or even 4 bits, by mapping the original range onto a smaller set of levels. The model shrinks by four to eight times, and integer math is far cheaper than floating-point on mobile silicon.

The cost is a small accuracy loss, which engineers claw back with careful retraining, called quantization-aware training, or by keeping a few sensitive layers in higher precision. The payoff is enormous: a model that would have needed several gigabytes and a data center GPU can fit in a phone's memory and run at interactive speeds. Nearly every on-device AI feature shipped today — photo segmentation, live transcription, message summaries — is a quantized model running on an NPU.

05On-device vs cloud inference

Running AI locally and running it in a data center are two different engineering worlds. The cloud gives you room-sized accelerators, water cooling, and effectively unlimited model size — which is why the frontier of capability still lives on servers. Its weaknesses are latency, cost, and connectivity: every word you speak has to make a round trip, and providers bill for every token.

On-device inference inverts the trade. Bandwidth is effectively infinite because the data never leaves the phone, latency is measured in milliseconds, and marginal cost per query is zero once the model is shipped. The limits are memory, thermal headroom, and raw compute, so local models are smaller and less capable than the biggest cloud models. The practical pattern in 2026 is hybrid: small, private, always-on tasks run on the NPU, and heavy reasoning gets delegated to the cloud when the network is available.

06TOPS: how AI chip performance is measured

Chipmakers advertise neural engines in TOPS — trillions of operations per second, counting each multiply and each add as separate operations. A flagship phone NPU in 2026 advertises numbers in the tens of TOPS, and laptop-class NPUs claim more. Because a MAC is two operations, a chip performing X trillion multiplies per second reports 2X TOPS, which is worth remembering when comparing marketing slides.

TOPS is a peak figure, like a car's top speed. It assumes every arithmetic unit is fed perfectly, which never happens with real models. Two chips with identical TOPS ratings can deliver very different real-world throughput depending on memory bandwidth, quantization support, and software. The honest comparisons come from running the same actual networks on both chips — the number on the spec sheet is a ceiling, not a promise.

07What on-device AI means for privacy and battery

Privacy is the quiet superpower of the neural engine. Data that never leaves the phone cannot be intercepted in transit, logged on a server, or folded into a training set. It is the reason face unlocking runs entirely on local silicon, and why features like live call transcription could ever ship at all. Regulators have noticed: keeping sensitive processing local is often the simplest way to satisfy data-residency rules.

The second dividend is battery. For a given amount of AI work, a dedicated NPU draws a fraction of the energy the CPU would need, which means always-on features — listening models, camera enhancements, keyboard prediction — cost almost nothing. The neural engine has quietly become the reason phones can offer assistant features that would have been unthinkable as cloud round-trips, both because the network is not always there and because users would not tolerate the latency.