Build a TPU From Scratch: the Systolic Array Logic That Runs Modern AI

01Why matrix multiply is the workload that defines an accelerator

Strip a modern neural network to its arithmetic and one operation remains: the multiplication of two big dense matrices, repeated billions of times. Every convolution is a matrix multiply in disguise, every attention head is a pair of them, and every fully connected layer is the operation at its most naked. Training and inference differ in direction and bookkeeping, not in the underlying kernel. This is why accelerator design begins with a single question — how fast can we multiply matrices — rather than a general-purpose notion of computing power.

Matrix multiply rewards exactly what fixed hardware is good at. The work is dense, meaning no branches and no wasted slots. It is regular, meaning the same instruction pattern repeats across the whole computation. And it is large, meaning the setup cost of any specialized circuit gets amortized across trillions of multiply-accumulate operations. A workload with that shape does not need flexibility; it needs throughput and discipline.

That is the design brief the tensor processing unit answers. Everything distinctive about a TPU — the grid of processing elements, the pulse of data through it, the oversized on-chip memory — follows from taking one operation seriously and building silicon that performs it with almost no overhead. The hobbyist build examined in this article starts from the same brief, scaled to a workbench.

02The von Neumann tax: what pure CPUs and GPUs leave on the table

A conventional CPU spends a startling fraction of its effort not computing but organizing computation. Fetching instructions, decoding them, predicting branches, routing operands through a cache hierarchy — every one of those steps is overhead from the perspective of pure arithmetic. For workloads with irregular control flow, that overhead buys irreplaceable flexibility. Matrix multiply is the opposite workload: perfectly predictable, so the machinery of prediction becomes pure waste. This is the von Neumann tax, and dense math pays it on every single operation.

GPUs reduced the tax by repurposing graphics hardware: thousands of simple lanes executing the same instruction on different data, with matrix work tiled to fit. It is a strong design, and it dominates the installed base. But a GPU still fetches weights and operands from memory through caches on the way to the arithmetic units, and every fetch costs energy and latency that multiply no numbers.

The chart below gives the resulting gap as an illustrative, relative index: a CPU core at one, GPU-style SIMT execution around two orders of magnitude higher, and a systolic array higher still — all at the same process node. The ranking, if not the exact ratios, is the reason dedicated matrix engines exist at all.

03Systolic arrays: data that moves instead of memory that serves

The systolic array answers a question the memory wall poses: what if, instead of fetching data to the compute units, you push the data through them? H. T. Kung and Charles Leiserson formalized the design, and the name borrows from physiology — data pulses through the array the way blood pulses through tissue, each cell receiving from its neighbor, computing, and passing the result downstream. In the classic matrix-multiply arrangement, weights sit stationary in the cells while activations flow past, and partial sums accumulate as they travel.

The consequences compound. Because each cell talks only to its nearest neighbors, wires stay short, clock speeds stay high, and the design scales cleanly with silicon area. Because data is reused in place rather than re-fetched, the array converts a memory-bandwidth problem into a pipeline problem. And because every cell performs the same multiply-accumulate step, the control logic is almost trivially simple — a welcome property when you are laying out thousands of identical units.

The Wikipedia record notes that systolic-like data flow predates the formal theory, appearing in Colossus during the Second World War before Kung and Leiserson's formulation. The modern TPU is that old idea grown to wafer scale: a grid of cells pulsing matrices through themselves, one heartbeat per clock cycle.

04Inside the from-scratch build: MACs, control, and the fill/drain cycle

The anchor video for this article — a meticulously documented hobby project with roughly 202,000 views — builds a working TPU from individual components, and its real lesson is how few ideas the array actually requires. Each processing element is a multiply-accumulate unit: two input registers, a multiplier, an adder and an accumulator. Wire those cells into a grid, feed weights in from one edge, stream activations through the other, and the grid itself becomes the multiplier for whole matrices.

The subtle engineering is in the fill and drain cycle. Before the first useful result emerges, the array must be primed — activations and weights travel down their respective pipelines until every cell holds valid data, a latency the builder pays on every shape change. Results then drain from the far edge in a staggered wave, which is why the control logic counts cycles as carefully as it routes data. Clocking discipline, not arithmetic, is where the build concentrates its difficulty.

What the project demonstrates is the design's elegance under constraint. With modest parts and no fabrication beyond careful wiring, the array produces correct results at a rate that scales with cell count. Scale that structure up, add proper memory bandwidth, and the distance from workbench to datacenter becomes a difference of degree rather than kind.

05Throughput math: utilization, batching, and why TPUs love big matrices

An N-by-N systolic array performs up to N-squared multiply-accumulates per clock cycle when fully occupied, and the phrase that matters is fully occupied. Utilization is the whole game. A matrix that matches the array dimensions fills every cell; a smaller matrix leaves cells idle while they still burn clock cycles; a much larger matrix must be tiled into passes, paying the fill-and-drain latency on every tile. This is why TPUs love big, regular, batched work — batching stacks many examples into tall matrices that keep the array saturated.

The edge effects are not a footnote; they are the difference between headline teraflops and delivered teraflops. Serve the array one small request at a time and utilization collapses, which is why inference servers batch aggressively and why latency-sensitive single-request inference often runs on smaller array blocks. Published peak numbers describe an array that is perfectly fed; real deployments spend their engineering effort getting close to that fiction.

The same arithmetic explains the shape of the industry. Arrays grow because utilization rewards area, memory bandwidth grows to feed the larger arrays, and compilers that tile matrices to fit become the quiet determinant of how much silicon actually earns its power budget. Peak throughput is a claim; utilization is the product.

06Precision and memory: bfloat16, on-chip SRAM, and HBM bandwidth

Two resource decisions complete the accelerator design. The first is precision. The bfloat16 format keeps the eight-bit exponent of 32-bit floating point and discards most of the mantissa, which means values neither overflow nor underflow the way narrow 16-bit formats do, while multiply hardware shrinks dramatically. Neural networks tolerate the lost precision because their parameters are statistical rather than exact — training converges through noise, not around it. Cheaper multipliers mean more cells per die, and more cells mean more throughput per watt.

The second decision is memory. The chart below gives the illustrative ordering: a register access costs roughly one unit, on-chip SRAM several times that, and off-chip HBM DRAM two orders of magnitude more. Those ratios dictate architecture. SRAM in front of the array holds activations and accumulations; HBM streams weights and batches; and the systolic design exists precisely to minimize how often the expensive level is touched, since each cell reuses its operands for many cycles.

The from-scratch build makes the trade visible at hobby scale — its buffer chips and register files are the same hierarchy, just small enough to see. At datacenter scale, that hierarchy is what liquid-cooled plumbing and tall HBM stacks exist to serve. Compute is the easy part; feeding it is the product.

07What one built-by-hand array explains about Google's warehouse-scale TPUs

Google's tensor processing units have run in production since 2015, per the timeline recorded on Wikipedia, and the warehouse-scale descendants of that first chip now arrive in racks cooled like industrial equipment. The through-line from hobbyist workbench to pod-scale deployment is the persistence of the same three ideas: a grid of identical multiply-accumulate cells, data pulsed through the grid to avoid re-fetching it, and a memory hierarchy sized to keep the grid fed. What changes with scale is everything surrounding the array — interconnect, thermals, fault tolerance, and the compiler stack that tiles real models onto real silicon.

That perspective is what the build project delivers better than any spec sheet. Once you have filled and drained a small array by hand, the mysterious acronyms collapse into pipeline timing you have personally debugged. The gaps between generations of commercial TPUs stop looking like magic and start looking like utilization engineering: bigger grids, wider memory, better tiling.

It also explains why matrix engines keep proliferating across vendors. The systolic idea is old, published and fundamentally simple, which means every serious silicon team can build one. The differentiation lives in the surroundings — and the hobbyist array, built from scratch on a desk, is the cleanest possible demonstration of where the core actually sits.