How the chips that power AI actually work

01The problem with general-purpose chips

A CPU is a remarkably flexible decision-maker. It handles branching code, operating-system work, and a changing mix of tasks with a small number of sophisticated cores. Machine learning presents a different shape: enormous arrays of numbers must be multiplied, added, and updated over and over.

That repetition exposes the CPU's trade-off. Its instruction machinery and cache hierarchy are valuable for irregular work, but much of the silicon sits idle when the job is a uniform block of multiply-and-accumulate operations. A neural network does not need one brilliant worker as much as it needs thousands of workers doing the same small job.

02What makes an AI accelerator different

Accelerators spend more of their area on parallel arithmetic and less on the control logic needed for every possible program. A GPU uses many relatively simple lanes; an NPU or tensor processor goes further by adding fixed-function paths for matrix multiplication, convolution, attention, and low-precision formats.

The advantage is conditional, not magical. A model must be translated into kernels that fit the accelerator's data types and memory layout. When the software maps well, throughput rises and energy per prediction falls. When it does not, the unused hardware becomes expensive decoration.

03How tensor cores process matrices

Consider a layer that computes C = A × B. Rather than fetching one number at a time, a tensor unit loads small tiles of A and B, multiplies corresponding values, and accumulates the partial results in registers. The same input can be reused across many operations before it is evicted.

This is why the word tensor matters: the hardware is arranged around blocks of dimensions, not isolated scalar instructions. Specialized paths can also fuse operations, such as scaling and activation, reducing trips through memory. Lower precision makes each value smaller, allowing wider pipes while a higher-precision accumulator protects the result from excessive rounding.

04The memory bottleneck and bandwidth solutions

Arithmetic is only useful when operands arrive on time. Large models quickly outrun the capacity and bandwidth of on-chip caches, so accelerator designers stack high-bandwidth memory beside the compute die, widen links, and build networks that move data between chips.

There is a hierarchy of compromises: registers are fast but tiny, on-chip SRAM is close but limited, HBM is capacious and wide but costly, and ordinary system memory is larger yet slower. Compilers try to tile and schedule work so that values are reused. In practice, the best accelerator is often the one that spends fewer joules moving a number than multiplying it.

05Training vs inference: different chip needs

Training is an optimization loop. It stores activations, computes gradients, and updates weights across many passes, which demands high throughput, large memory, and fast communication between devices. Checkpointing and distributed parallelism add their own traffic.

Inference is the forward pass used to answer a request. Latency, cost per token, and serving many users at once matter more. Quantization can shrink weights to eight, four, or even fewer bits; sparsity can skip zeros; and smaller edge NPUs can run a compact model without sending private data to a cloud.

06The manufacturing race: TSMC, Samsung, and node sizes

Process nodes are not a simple speedometer, but denser transistors can provide more arithmetic and cache in a given area. The challenge is turning that density into a working, affordable package. Huge accelerator dies have lower yield risk when designers split them into chiplets or use advanced packaging.

Foundries, memory suppliers, substrate makers, and packaging lines therefore form one bottlenecked chain. A leading node may be available on paper while capacity for the required interposer or HBM stack is scarce. The AI-chip contest is as much about manufacturing coordination as it is about circuit design.

07What comes next: photonic and neuromorphic chips

Photonic accelerators use properties of light to perform some linear-algebra operations with very low movement of electrical charge. Neuromorphic designs instead imitate aspects of biological signaling, sending sparse events rather than clocking dense arrays continuously. Both approaches are promising, but neither is a drop-in replacement for today's programmable accelerators.

The near-term future is likely heterogeneous. CPUs will orchestrate, GPUs and NPUs will carry dense kernels, and specialized devices will take narrow workloads where their physical advantage is clear. The winning systems will hide that complexity behind compilers and portable model runtimes.