Behind the Build: What AI Data Center Infrastructure Actually Requires

01The Scale of AI Compute Demand

The conversation around artificial intelligence tends to fixate on models and algorithms, but the physical substrate enabling those models is undergoing a transformation orders of magnitude larger than most people appreciate. A single large language model training run can require thousands of GPUs running in parallel for weeks or months, drawing megawatts of power continuously. The data centers housing this equipment are not incremental upgrades to existing facilities; they are fundamentally different beasts, engineered from the ground up for a workload profile that barely existed five years ago.

Hyperscale operators have publicly disclosed cluster sizes exceeding 100,000 GPUs for individual training runs. Meta's Llama 3 training cluster, for instance, was built around 16,000 H100 GPUs and consumed roughly 150 megawatts at peak. That is a single workload in a single facility. When you multiply across the dozens of simultaneous training and inference workloads running at companies like Google, Microsoft, Amazon, and Meta, aggregate demand reaches into the gigawatts. The International Energy Agency has estimated that global data center electricity consumption could double between 2022 and 2026, driven primarily by AI workloads.

The scale challenge is not merely about racking more servers. Every additional GPU brings cascading requirements: more power delivery, more heat rejection, more network bandwidth, more physical space, and more redundancy. A facility designed for conventional cloud workloads at 10 kilowatts per rack cannot simply accept AI training racks drawing 60 kilowatts each. The electrical distribution, cooling capacity, and structural load tolerances were never sized for this. The result is that AI compute demand is forcing a ground-up rethinking of data center design, from the substation down to the rack.

02Power Density: Why AI Racks Run Hot

The defining physical characteristic of AI infrastructure is power density. A traditional enterprise data center rack might draw 5 to 10 kilowatts. An AI training rack populated with GPUs can draw 40 to 80 kilowatts or more, depending on the generation of accelerator and the configuration. NVIDIA's DGX H100 systems, for example, each draw approximately 10.2 kilowatts, and a standard 42U rack can hold several of them alongside networking gear. The upcoming generation of accelerators pushes densities even higher, with some designs approaching 100 kilowatts per rack.

This density is a direct consequence of the silicon itself. Modern AI GPUs are built on advanced packaging with thousands of compute cores, high-bandwidth memory stacks, and interconnects running at extraordinary data rates. The H100 SXM module has a thermal design power of 700 watts per GPU. A single server with eight GPUs draws over 5.6 kilowatts for the GPUs alone, before counting the host CPU, memory, networking interfaces, and power supply losses. Concentrate dozens of such servers into a rack and the power profile becomes unlike anything the data center industry has previously handled at scale.

The thermal physics are unforgiving. Every watt of electrical power entering a server becomes a watt of heat that must be removed. At 60 kilowatts per rack, conventional air cooling reaches its practical limit. Air has low heat capacity and poor thermal conductivity compared to liquid, and the fan power required to move enough air through a high-density rack becomes a significant fraction of total consumption. This is why power density is not just an electrical problem but a thermal one, and why the two disciplines must be co-designed rather than treated independently.

03Cooling Architectures Compared

Cooling is where AI data center engineering gets most contentious. The industry is in a period of rapid transition from air-based cooling to liquid-based systems, and the choice of architecture has cascading implications for capital cost, operational complexity, and facility design. The three primary approaches in production today are raised-floor air cooling, direct-to-chip liquid cooling, and immersive cooling, each with distinct trade-offs.

Air cooling remains the dominant approach in legacy facilities and is still used for lower-density inference workloads. It relies on computer-room air conditioning units to push chilled air through perforated floor tiles and into the intakes of server racks. The approach is well understood, relatively simple to maintain, and compatible with virtually all existing server hardware. Its fundamental limitation is thermodynamic: air simply cannot carry enough heat away from a 60-kilowatt rack without prohibitive fan energy. At high densities, the air itself becomes a bottleneck, and hot-spots develop where airflow cannot keep pace with heat generation.

Direct-to-chip liquid cooling, which circulates coolant through cold plates mounted directly on GPUs and CPUs, is becoming the de facto standard for AI training. Liquid has roughly 3,000 times the heat transfer capacity of air by volume, meaning a small flow of coolant can remove heat that would require massive volumes of air. The trade-off is complexity: every server needs plumbed connections, the coolant distribution system adds a second fluid loop to the facility, and leaks can be catastrophic. Immersion cooling, where entire servers are submerged in dielectric fluid, offers the highest heat removal capacity but introduces the most operational complexity, as maintenance requires removing hardware from fluid baths. Most hyperscalers are standardizing on direct-to-chip for next-generation AI facilities, treating immersion as a longer-term bet.

04The Fiber and Networking Backbone

Power and cooling dominate the headlines, but the networking infrastructure inside an AI data center is equally critical and arguably less understood. AI training is not a collection of independent servers doing separate work; it is a tightly coupled distributed system where thousands of GPUs must exchange intermediate results every iteration of the training loop. The network connecting them must deliver extraordinary bandwidth with sub-microsecond latency, and it must do so without becoming the bottleneck that idles expensive GPUs.

The dominant interconnect in AI training clusters today is InfiniBand, specifically NVIDIA's Quantum and Quantum-2 InfiniBand platforms. A single Quantum-2 switch provides 64 ports of 400 gigabits per second, and large clusters are built with fat-tree topologies that can require thousands of switches and tens of thousands of optical transceivers. The optical fiber running through an AI data center is staggering in quantity. A 16,000-GPU cluster might use over 3,000 kilometers of fiber optic cabling internally, and the transceiver cost alone can run into the tens of millions of dollars.

Ethernet-based alternatives, particularly the Ultra Ethernet Consortium's emerging standards, are being pursued as a way to reduce dependence on NVIDIA's networking ecosystem. The challenge is that Ethernet, even at 400 and 800 gigabit speeds, has historically struggled with the congestion management and lossless behavior that distributed training requires. The AI networking backbone is also where the cost economics get particularly painful. In a large training cluster, networking can account for 10 percent or more of total infrastructure cost, and the transceivers and fiber are consumable expenses that get refreshed with every generation of interconnect.

05Site Selection and Grid Interconnection

You cannot build an AI data center anywhere. The site selection process for hyperscale AI facilities has become one of the most complex real estate and infrastructure problems in the modern economy, driven by three primary constraints: power availability, fiber connectivity, and water access. Finding a location that satisfies all three simultaneously, at the scale required, is increasingly difficult.

Power is the binding constraint. A 500-megawatt AI data center requires the electrical output of a mid-size power plant, and it needs that power reliably 24 hours a day, 365 days a year. Not every grid can deliver this. Regions with abundant renewable energy, such as parts of Texas with wind power or the Pacific Northwest with hydroelectric capacity, have become attractive, but grid interconnection queues in many markets now stretch years. Some operators are bypassing the grid entirely, contracting directly with natural gas producers or nuclear facilities for dedicated generation. Microsoft's agreement with Constellation Energy to reopen the Three Mile Island nuclear plant is a signal of how far operators will go to secure dedicated power.

Water access is the second constraint. Evaporative cooling towers, which are the most cost-effective way to reject heat at scale, consume millions of gallons of water per day in a large facility. In water-stressed regions, this creates direct competition with agricultural and municipal users. The site selection calculus also includes climate: facilities in cooler climates can use free cooling for a larger fraction of the year, reducing both energy and water consumption. This is why Scandinavia, Iceland, and parts of Canada have seen significant data center investment despite their distance from major population centers.

06The Cost Economics of AI Infrastructure

The capital required to build a hyperscale AI data center is staggering. Industry estimates place the cost of a large AI facility at between $1 billion and $4 billion, depending on size and whether the GPU servers are included in the building cost or accounted separately. When you include the servers, the numbers become even more dramatic. A cluster of 16,000 H100 GPUs alone costs roughly $500 million at current pricing, and that is just the accelerators, before networking, storage, power, cooling, and the building itself.

The cost breakdown reveals where the money goes. Power infrastructure, including substations, transformers, switchgear, and power distribution units, typically represents the largest single category at roughly 35 percent of total facility cost. Cooling systems, whether CRAC units, coolant distribution units, or immersion tanks, account for another 25 percent. The servers and accelerators themselves, if included in the facility budget rather than treated as a separate IT expense, represent about 20 percent. Networking, land, and building structure round out the remainder. The key insight is that the GPUs, which dominate the conversation, are actually a minority of the total infrastructure investment when you account for the facility built to house them.

Operating costs are equally consequential. A 500-megawatt facility running at full load consumes roughly 4.4 terawatt-hours per year, and at commercial electricity rates of $0.05 to $0.07 per kilowatt-hour, that translates to $220 million to $308 million in annual energy costs alone. When you factor in maintenance, staffing, water, and the amortization of capital equipment, the total cost of ownership for a large AI data center over a ten-year period can exceed $10 billion. These numbers explain why only a handful of companies in the world can afford to build at this scale, and why the economics of AI infrastructure are becoming a moat as significant as the algorithms themselves.

07What Hyperscalers and Colocation Providers Are Building

The build-out is happening at a pace that strains credibility. Microsoft, Google, Amazon, Meta, and Oracle are collectively committing hundreds of billions of dollars to AI infrastructure over multi-year horizons. Microsoft alone has indicated plans to spend over $80 billion on AI data center infrastructure in fiscal year 2025. Google and Amazon are at comparable scales. These are not aspirational numbers; they are capital expenditure commitments reflected in quarterly filings, and they represent physical construction of facilities, purchase of land, and procurement of accelerators.

The colocation market is adapting in parallel. Providers like Applied Digital, CoreWeave, Lambda, and others are building AI-specific facilities that cater to customers who need GPU capacity but cannot justify their own hyperscale build. These providers often locate in non-traditional markets where power is cheaper and land is abundant, and they differentiate on speed of deployment and willingness to support the high-density rack configurations that AI workloads require. The colocation model is particularly important for AI startups and research institutions that need access to large-scale GPU clusters without the capital outlay of building a facility.

The most significant trend shaping the next generation of AI data centers is the move toward standardized, modular designs. Rather than engineering each facility as a one-off, hyperscalers are developing repeatable blueprints that can be deployed across multiple sites. Microsoft's pre-fabricated data center modules, which arrive on site largely assembled, are one example. The goal is to compress construction timelines from years to months, because the demand curve for AI compute is so steep that every month of delay represents lost revenue and lost competitive position. The infrastructure build is, in a real sense, the frontier of AI competition, and the companies that can build fastest and most efficiently will have a structural advantage that compounds over time.