Tensor Processing Units and the 2026 Custom-Silicon Moment

The Tensor Processing Unit is a neural processing unit application-specific integrated circuit developed by Google for neural network machine learning. Google began using TPUs internally in 2015 and opened them to third parties in 2018 through its cloud, alongside a smaller edge version sold as a component. In the years since, the chip has quietly moved from internal trivia to the center of cloud economics: Google Cloud now generates product revenues primarily from the sale of TPU systems, and TensorFlows, JAX and PyTorch workloads all run on the architecture.

That trajectory is why TPUs are worth explaining in 2026 specifically. Every major hyperscaler now fields or funds custom AI silicon, and each program is, in part, an argument about the TPU thesis: that a processor designed around one workload, tied to one software stack, beats a general-purpose design on cost and power. Understanding the thesis requires understanding the array at its core.

01 The Chip That Treats Math as Dataflow

A conventional processor is built around latency: fetch an instruction, fetch the operands, compute, write back, repeat. A TPU is built around throughput. Its dominant feature is a large two-dimensional grid of multiply-accumulate units in which operands enter at one edge and flow rhythmically through neighboring cells, each performing a multiplication and accumulation on every cycle. Nothing fetches anything from main memory in the inner loop; the data moves the way pressure moves through a fluid.

The first-generation TPU put a 256 by 256 grid of such units on the die, 65,536 multipliers executing in parallel per clock, fed by a large block of on-chip weight memory. The design choice that made it work was numeric: the chip computed in 8-bit integer arithmetic, accepting reduced precision in exchange for roughly four times the compute density and a fraction of the energy per operation compared with 32-bit floating point. For inference workloads, where weights are fixed and tolerance is statistical, the trade was close to free.

02 Inside the Systolic Array: Why the Grid Beats the Cache

The grid is called a systolic array, by analogy with the pulse of blood through tissue: inputs pulse in, partial sums propagate outward, and every cell does useful work on every beat. The reason this beats a cache-centric design is arithmetic intensity. In matrix multiplication, each operand fetched from memory can be reused across an entire row or column of the output. A systolic array maximizes that reuse by construction, so the memory system is relieved of traffic that would otherwise dominate power and time.

The schematic below shows the flow in miniature: activations enter from the left, weights enter from the top, and each cell multiplies, accumulates and passes both along. Scale the drawing from six cells to 65,536 and the economic argument comes into focus.

03 TPU versus GPU: Two Answers to the Same Question

A GPU is a generalist that won on accident: its graphics pipeline happened to map onto neural network math, and generality made it the default accelerator for a field that changes shape every year. A TPU is a specialist that wins on purpose: strip out rasterization, texture units and much of the memory hierarchy, and the remaining die area is matrix multipliers. The trade is flexibility for efficiency. A GPU absorbs any new architecture a researcher invents; a TPU makes the operations it was built for dramatically cheaper per unit of energy.

The efficiency gap is not marginal. The original Google analysis of the first TPU generation reported on the order of thirty to eighty times better performance per watt than contemporaneous server CPUs and GPUs on its target inference workloads. The figure below renders that reported range as an approximate index; it describes 2015-era hardware and should not be read as a current-generation comparison.

04 Precision as a Design Lever

The integer bet of the first generation did not survive training. Gradient accumulation needs range as well as resolution, and later TPU generations moved to floating point, adopting the bfloat16 format, which keeps the exponent width of 32-bit floats at half the bit cost. Precision is therefore not a purity question but a lever: every bit removed from an operand multiplies the arithmetic a fixed die can perform and divides the energy each operation consumes. The chart below compares operand widths across the common formats.

05 Why Hyperscalers Build Their Own Silicon

The first reason is the power wall. A datacenter does not buy compute; it buys compute inside an electricity and cooling budget, and performance per watt is the binding constraint on how large a fleet can grow in a given building. The second reason is utilization. A merchant GPU serves every customer's workload, so most of its die serves no single buyer efficiently; a custom accelerator serves one company's dominant workload, and internal telemetry can drive the roadmap in a way no merchant vendor will.

The third reason is supply. When demand for accelerators exceeds foundry output, owning your own design in someone's fabrication queue is leverage: it cannot be rationed away at the margin in favor of a better-paying rival. The fourth reason is margin. Selling access to your own silicon turns a cost center into the product itself, which is the quiet headline in the Wikipedia summary line that Google Cloud now derives product revenues primarily from TPU systems.

06 The Economics of Tying Silicon to a Model Stack

A TPU is not a chip in isolation; it is a chip fused to a compiler, a framework and a model architecture roadmap. TensorFlow and JAX were built with the array in mind, PyTorch support closes the gap for everyone else, and the fusion cuts both ways. The software stack can assume a systolic matrix and schedule accordingly, extracting utilization a generic stack cannot. The same coupling raises switching costs: a model optimized for one vendor's primitives is not free to migrate, and the cloud that owns the silicon becomes the path of least resistance for training the next generation.

This is the part of the custom-silicon story that pure hardware comparisons miss. The moat is not transistor count; it is the feedback loop between the people who design the model and the people who design the die. When one organization controls both, architecture decisions, such as which numeric formats the next model family will emphasize, propagate into silicon with a lead a merchant vendor cannot match.

07 The 2026 Custom-Silicon Moment

What makes TPUs newly newsworthy in 2026 is not the chip but the imitation. Every hyperscaler of scale now fields its own accelerator program or funds one through a silicon partner, because the four pressures in section 05, power, utilization, supply and margin, bind on all of them equally. The TPU is the longest-running existence proof that the strategy works: it is the only custom AI accelerator with a decade of production history inside a fleet at this scale, and it has graduated from serving one company's inference to being sold as infrastructure to outsiders.

The competitive geometry that follows is unfamiliar. Merchant GPUs remain the universal lingua franca that any laboratory can rent by the hour, while custom arrays compete on total delivered economics for tenants willing to write against their stacks. Inference demand at 2026 scale has made that second lane big enough to matter, which is why a chip explained in an explainer video is, in economic terms, one of the most consequential objects in the industry.

08 What TPUs Do Not Change

Specialization has limits, and honest accounting marks them. Data movement still costs more than arithmetic at every level of the hierarchy, and no array geometry abolishes the memory wall; it only moves it. Workloads that fall outside the pattern the array was built for, sparse models, exotic attention variants, non-neural computation, give back much of the advantage, which is precisely why GPUs retain their role. And a fused hardware-software stack concentrates risk: a design miss is a design miss for an entire fleet, not one product line.

The durable lesson of the TPU is therefore not that custom silicon beats general purpose silicon everywhere. It is that when one workload becomes important enough and stable enough, the economics tip toward building the machine the workload wants. Neural network computation has been the most important such workload of the decade, and the systolic array, a 1980s idea that waited for a problem worthy of it, is the shape the tip took.