How Google Builds TPUs: The Custom Chip Behind Gemini and Apple's AI

01Why Google Builds Its Own AI Chip

In 2013, Google faced a forecast that would terrify any infrastructure planner. If people kept talking to voice search at the rate they were adopting it, Google would have needed to roughly double its data center fleet just to keep up with speech recognition alone — the conventional processors of the era would have made that workload economically absurd. A small group of engineers was quietly chartered to find another way, and what emerged from that skunkworks was not another x86 purchase order but a chip of Google's own design.

The reasons went beyond raw cost. Buying general-purpose CPUs meant renting someone else's idea of what computation should look like, and neural network inference is not general purpose at all — it is overwhelmingly multiply-accumulate arithmetic on large matrices. Supply was a second worry: Google's appetite for compute was growing faster than the semiconductor market's willingness to build for it. Owning the design gave Google a measure of independence from the GPU allocation lotteries that everyone else in the industry now complains about.

A third reason was workload specialization. A chip tuned for tensor math can strip out everything a neural network does not need and pour the saved silicon area into more arithmetic units. Google bet early that machine learning would eat its product line — search ranking, translation, photos, ads — and that a chip purpose-built for it would compound advantages across everything. That bet has aged well.

02What a TPU Actually Is: Systolic Arrays in Plain Terms

Strip away the acronyms and a TPU is a machine for multiplying enormous matrices very fast while moving data as little as possible. Its signature structure is the systolic array, a grid of processing elements that behave like an assembly line. Numbers enter from one edge, march through the grid in lockstep, and each station they pass multiplies and accumulates as it goes. The term comes from systole — the contraction of a heart pumping blood — because data pulses through the chip in rhythmic waves rather than flooding it all at once.

The key insight is locality. In a conventional processor, shuttling data between memory and arithmetic units burns more time and energy than the arithmetic itself. In a systolic array, each datum that arrives at a processing element is immediately reused by its neighbors, so one memory fetch feeds many operations. In Google's design this grid lives inside a block called the Matrix Multiply Unit — on the first-generation chip, a 256 by 256 arrangement capable of 65,536 multiply-accumulates per cycle. That is what a chip looks like when you delete everything except what neural networks actually do.

The trade-off is flexibility. A GPU can run a physics simulation in the morning and a diffusion model in the afternoon; a TPU does tensor math and effectively nothing else. For a company whose business is overwhelmingly tensor math, that is not a limitation — it is the entire point.

03From Research Project to Ironwood: A Decade of Generations

The first TPU, deployed inside Google's data centers in 2015, was an inference chip — quiet, fast, and already doing real work behind Search and Translate while the rest of the world assumed AI compute meant GPUs. The second generation, announced at Google I/O in 2017, added training capability and proved the concept was not a one-off. The third generation in 2018 brought the pods: racks of chips wired into a single liquid-cooled machine that developers could book like a resource, and a public cloud offering so outside researchers could rent the silicon.

The fourth generation in 2021 pushed interconnect bandwidth hard, because by then the bottleneck in large-model training was as much chip-to-chip communication as it was raw arithmetic. The v5e in 2023 was a different kind of statement: a deliberately cheaper, smaller, more efficient chip aimed squarely at inference economics, reflecting where the industry's volume actually was. The v5p, also in 2023, was the upscale sibling — roughly 459 peak bf16 TFLOP per chip for the largest training jobs.

In April 2025 Google announced the seventh generation, Ironwood: a chip rated at roughly 4,614 peak FP8 TFLOP, roughly 3,600 of which are interconnect-capable, packaged into a 9,216-chip pod. It is Google's first fully liquid-cooled TPU, designed specifically for inference of reasoning-class models — the kind that think in long chains of thought before answering. Through 2026, Ironwood has been rolling out at scale across Google Cloud, and it is the generation now serving the heaviest Gemini traffic.

04How TPUs Power Gemini's Training and Serving

Gemini, Google's flagship model family, is the clearest public demonstration of what the TPU program was built for. Google has said that its largest Gemini models — including the Ultra tier — are trained on TPU pods spanning multiple data center sites, and that its most capable model at the time of the Gemini Ultra announcement was trained across a substantial fleet of TPU v4 and v5e chips. Training a frontier model is a bandwidth and synchronization problem as much as a compute problem, and the TPU's custom interconnect is designed precisely for the all-reduce chatter that dominates that workload.

Serving is the other half, and arguably the more commercially important one. Every time a Gemini or assistant query is answered, a TPU runs inference, and because Google owns the chip end to end it can shape the silicon to the serving pattern — v5e for cost-efficient volume traffic, Ironwood for the long reasoning chains of thinking models. In late 2025, Google disclosed that it planned to deploy more than a million TPUs across its fleet by the end of 2026, more than triple the roughly 300,000 it reported a year earlier. That is the quiet fact behind every "capacity constrained" earnings call: the constraint is being addressed with home-grown silicon.

The economics matter here. Serving chat-style queries is a marginal-cost business, and a chip purpose-built for inference can deliver tokens per dollar that general-purpose hardware struggles to match. Google's DeepMind chief has argued that TPUs hold a per-token cost advantage for serving Google's models, and that NVIDIA's latest hardware carries a price premium the workload does not require. Rivals dispute the comparison, but the direction is clear: Google designs for Google's serving reality.

05The Apple Connection: The iPhone's Cloud Brain Runs on Google Silicon

Here is the twist that makes this more than a Google story. In July 2024, Apple's machine learning research group published a paper on its server foundation model — the roughly 3-billion-parameter model behind Apple Intelligence's cloud features — and disclosed the training infrastructure: 2,048 TPU v5 chips for pretraining and 1,968 TPUv5 chips for instruction tuning, plus 2,048 TPUv4 chips for more specialized jobs. The company that spends years making its own silicon, the company famous for never depending on a rival's platform, trained part of its cloud intelligence on Google's.

The reasoning is mundane rather than scandalous: TPUs were available, capable, and priced for exactly this kind of large-model training job. Apple has since scaled its own infrastructure aggressively — its publicly reported spending on AI compute runs into the billions per year, and its capex guidance has climbed steeply since 2024. But the disclosure punctured a convenient myth: that the AI accelerator market has exactly one vendor. When a company with Apple's silicon pride rents Google's chips for frontier-scale training, the custom-ASIC option is clearly real.

Apple is far from alone. Anthropic has trained and served models on Google Cloud TPUs, and the CNBC documentary that frames this article opens with exactly this point — the quiet infrastructure fact that much of the AI a consumer touches, including Apple's, may be running on the same chip family behind Gemini.

06TPUs vs GPUs: The Economics of Inference and Training

NVIDIA's GPUs dominate the market for a structural reason: a GPU is programmable enough to serve every customer's workload, and a single software ecosystem — CUDA — lets any lab move fast. That generality costs silicon area and power, but when your workloads are diverse, that flexibility is worth paying for. For most of the industry, the choice is rational.

Google's position is different because its workloads are not diverse at the chip level. When nearly everything you run is tensor math at enormous scale, the overhead of generality is waste, and the fixed cost of designing your own ASIC is amortized across a fleet measured in the millions of chips. Custom silicon only pays above a certain volume threshold — and Google crossed that threshold long before almost anyone else tried.

The competitive pressure is now visible in NVIDIA's own disclosures: the company has begun breaking out a custom-silicon category when it describes the accelerator market, acknowledging that a meaningful slice of the market is no longer shopping for GPUs. That slice includes Amazon's Trainium and Inferentia chips, Microsoft's Maia, Meta's MTIA, and OpenAI's partnership with Broadcom on its own accelerator — a pattern that reads like the largest AI consumers concluding that at their scale, designing for their own workloads beats buying generality.

07What Custom Silicon Means for the AI Chip Market

The market NVIDIA now faces is bifurcating. On one side: everyone who buys accelerators in quantities too small or workloads too varied to justify designing a chip — labs, enterprises, most of the world. On the other: a handful of hyperscalers and frontier labs for whom the math has flipped. Google proved the model works, and the proof has been copied.

The copycats matter because they erode the assumption that AI compute is a single-vendor market. Amazon builds Trainium for training and Inferentia for inference; Microsoft has Maia; Meta has MTIA; OpenAI is co-designing with Broadcom. Each one that moves a workload in-house shrinks the addressable market for general-purpose GPUs at exactly the moment demand for those GPUs is exploding elsewhere — which is why NVIDIA's business keeps growing even as its share is nibbled at the edges.

The TPU is the template for all of it: a decade of evidence that a big enough consumer of AI compute can build its own silicon, keep its costs and roadmap private, and — in the twist nobody at Google planned for — quietly end up powering a rival's flagship product. Google began using TPUs internally in 2015, opened them to cloud customers in 2018, and today TPU systems are the primary product revenue line of Google Cloud. The secret project started to make voice search cheap ended up as one of the most consequential chips in the industry, and most consumers have never heard its name — they just use it every day.