GPU Architecture: The Silicon Engine Behind the AI Revolution

Figure 1: NVIDIA data center revenue by fiscal year. FY25 (ending January 2025) revenue reflects full-year AI infrastructure demand. Source: NVIDIA annual reports.

01 From Pixels to Parallelism: The GPU Origin Story

The graphics processing unit was not designed for artificial intelligence. It was designed to render 3D graphics, a task that happens to share a fundamental mathematical structure with neural network computation: massive parallelism. A screen displaying a 3D scene at 60 frames per second with a resolution of 1920 by 1080 pixels must compute the color of over 124 million pixels per second, and each pixel requires multiple arithmetic operations involving vertex transformations, texture sampling, and lighting calculations. These operations are independent, meaning they can be executed simultaneously rather than sequentially.

A CPU, optimized for low latency and complex sequential logic, typically has between 4 and 32 high-performance cores. A GPU, optimized for high throughput on independent operations, has thousands of simpler cores. The NVIDIA H100 GPU, released in 2023 and still the workhorse of most AI data centers in 2026, contains 80 streaming multiprocessors with a total of 16,896 CUDA cores. The CPU excels at getting from instruction A to instruction B as quickly as possible. The GPU excels at getting from instruction A to instruction A-thousand as quickly as possible, even if each individual instruction is slower.

This architectural distinction is the foundation of the AI revolution. Neural network training consists almost entirely of matrix multiplication operations, where each element of the output matrix is computed independently. A GPU can distribute these operations across its thousands of cores, achieving throughput that a CPU cannot match. The discovery that GPUs, originally built for gaming, were ideally suited for deep learning computation was made by researchers including Andrew Ng and the team at NVIDIA around 2008-2009, but it took until 2012, when AlexNet used GPUs to win the ImageNet competition by a dramatic margin, for the insight to become conventional wisdom.

02 The Streaming Multiprocessor Architecture

The fundamental building block of a modern NVIDIA GPU is the streaming multiprocessor (SM). Each SM contains dozens of CUDA cores for integer and floating-point arithmetic, special function units for transcendental operations, and tensor cores for matrix multiplication. The SM also includes registers, shared memory, and an L1 cache, all managed by a hardware scheduler that can context-switch between threads with zero overhead.

The H100 GPU's 80 SMs are organized into 8 graphics processing clusters, each containing 4 texture processing clusters, each containing 2 SMs. This hierarchical organization allows the GPU to partition work across its resources efficiently. The scheduler can launch thousands of threads simultaneously, grouping them into warps of 32 threads that execute the same instruction on different data, a model called single instruction, multiple threads (SIMT).

Memory bandwidth is as important as compute capacity. The H100 SXM5 variant uses 80 gigabytes of HBM3 memory with a bandwidth of 3.35 terabytes per second. For comparison, a high-end CPU with DDR5 memory achieves approximately 100 gigabytes per second. This 30x bandwidth advantage is critical because neural network operations are often memory-bound: the time to load weights from memory exceeds the time to perform the arithmetic. Without HBM, the GPU's compute cores would spend most of their time waiting for data.

03 Tensor Cores: The AI Accelerator Within

Tensor cores are the component that transformed the GPU from a general-purpose parallel processor into a dedicated AI engine. Introduced with the Volta architecture in 2017, tensor cores perform mixed-precision matrix multiplication in a single clock cycle. A single tensor core can multiply two 4-by-4 matrices and add the result to a third 4-by-4 matrix in one cycle. The H100 contains 528 tensor cores across its 80 SMs, delivering 3,958 teraFLOPS of FP16 matrix multiplication throughput.

The progression of tensor core capabilities across NVIDIA architectures illustrates the pace of AI-specific hardware development. Volta (2017) supported FP16. Turing (2018) added INT8 and INT4. Ampere (2020) introduced FP64 and structured sparsity, which doubles throughput by skipping zero-valued operations. Hopper (2022) added FP8, a format that halves memory usage and doubles throughput compared to FP16 with minimal accuracy loss for inference. Blackwell (2024) introduced second-generation FP8 and a new transformer engine that automatically manages precision selection on a per-layer basis.

The transformer engine is particularly significant. It monitors the activation range of each layer in a transformer model and dynamically switches between FP8 and FP16 to maintain numerical stability while maximizing throughput. This hardware-level optimization means that framework developers do not need to manually tune precision settings for each model architecture. The GPU handles it transparently, and the performance gain is substantial: FP8 training of large language models on Hopper achieves approximately 2.5 times the throughput of FP16 training on the same hardware.

Figure 2: Peak FP16 dense tensor core performance across four NVIDIA architectures. Values from official NVIDIA whitepapers. B200 uses FP4 for 9 PFLOPS; shown here at FP16 dense for architectural comparison.

04 Memory Hierarchy and HBM

The GPU memory hierarchy is engineered around a single principle: getting data to the compute cores as fast as possible. At the top are registers, which are fastest but limited in capacity. Below that is shared memory, a programmer-managed cache that sits on the same chip as the SMs and provides approximately 19 terabytes per second of bandwidth. Below that is L2 cache, and below that is HBM, the large but slower off-chip memory.

High Bandwidth Memory (HBM) is the technology that makes large-model AI training feasible. HBM uses stacked DRAM dies connected to the GPU through a silicon interposer, with thousands of microscopic connections running in parallel. This is fundamentally different from the traditional CPU memory model, where a few high-speed lanes connect the processor to a PCB-mounted DIMM. HBM3, used in the H100, stacks 8 DRAM dies vertically, each die providing 10 gigabytes of capacity and 419 gigabytes per second of bandwidth, for a total of 80 gigabytes and 3.35 terabytes per second.

The Blackwell B200, NVIDIA's 2024 flagship, uses HBM3e memory with 192 gigabytes of capacity and 8 terabytes per second of bandwidth. This 2.4x bandwidth improvement over Hopper is what allows the B200 to train models with over 200 billion parameters on a single GPU, a task that required multi-GPU tensor parallelism on Hopper. The memory wall, the point where memory bandwidth rather than compute capacity becomes the bottleneck, has been the dominant challenge in AI hardware design since 2020, and HBM is the primary tool for pushing that wall back.

05 The CUDA Software Ecosystem

Hardware alone does not make a platform. NVIDIA's most durable competitive advantage is not its silicon but its software stack, particularly the CUDA programming model. CUDA, introduced in 2007, was the first general-purpose parallel computing platform for GPUs. Before CUDA, using a GPU for non-graphics computation required mapping scientific problems onto graphics APIs like OpenGL, a technique called GPGPU computing that was powerful but extremely difficult to use.

CUDA provides a C++-like programming model where developers write kernel functions that execute on the GPU, and the runtime manages thread scheduling, memory transfers, and synchronization. Over 18 years, NVIDIA has built a comprehensive ecosystem on top of CUDA: cuBLAS for linear algebra, cuDNN for deep learning primitives, NCCL for multi-GPU communication, TensorRT for inference optimization, and Triton for kernel development. Every major deep learning framework, PyTorch, TensorFlow, JAX, depends on CUDA as its primary backend.

The switching cost of moving from CUDA to an alternative is enormous. AMD's ROCm platform and Intel's oneAPI offer functional alternatives, but they lack the mature libraries, the optimized kernels for every common operation, and the community knowledge that CUDA has accumulated. The challenge is not technical but ecological: even if ROCm matched CUDA feature-for-feature today, the millions of lines of CUDA code in production, the thousands of optimized kernels, and the decade of Stack Overflow answers would still make migration painful. This is why NVIDIA's data center revenue grew from 3 billion to 115 billion dollars in five years despite technically capable competitors.

06 The AI Inference Economy

Training gets the headlines, but inference is where the volume is. Every query to ChatGPT, Claude, or Gemini triggers a forward pass through a transformer model on GPUs in a data center. The computational cost of inference exceeds the cost of training for any model that serves millions of users, typically within months of deployment. NVIDIA's inference revenue, reported separately from training revenue since 2024, is growing faster than training revenue and is expected to surpass it by 2027.

Inference has different hardware requirements than training. Training benefits from high precision (FP16 or FP8) and maximum throughput. Inference can use lower precision (INT8, FP4, or even binary) with quantization techniques that reduce model size and memory bandwidth requirements by 4 to 16 times with minimal quality loss. The H100's FP8 tensor cores deliver 3,958 teraFLOPS for training but can achieve 30,000 teraFLOPS for INT8 inference, a 7.6x throughput advantage that directly translates to lower cost per query.

The economics of inference are driving a new class of AI-specific chips. Google's Tensor Processing Units (TPUs), Amazon's Trainium and Inferentia accelerators, and a wave of startup inference chips from Cerebras, Groq, and SambaNova all target the inference market specifically. These chips sacrifice the graphics pipeline that GPUs retain and focus entirely on matrix multiplication and memory bandwidth for transformer models. Whether any of these alternatives can erode NVIDIA's dominance depends on the CUDA switching cost, the pace of NVIDIA's own inference optimization, and the willingness of hyperscale cloud providers to vertically integrate their silicon.

07 The Silicon Supply Chain

GPU manufacturing is a story of global interdependence. NVIDIA designs its chips in Santa Clara, California, but does not manufacture them. The physical fabrication is done by TSMC in Taiwan, using its N4 and N5 process nodes for Hopper and N4P for Blackwell. The HBM memory is manufactured by SK Hynix in South Korea, with Samsung and Micron as secondary suppliers. The advanced packaging that connects the GPU die to the HBM stacks is performed by TSMC using its CoWoS (Chip on Wafer on Substrate) technology, a process whose limited capacity was the primary constraint on H100 supply in 2023 and 2024.

The Blackwell B200 uses a dual-die design connected by a 10 terabyte per second NV-HBI (NVIDIA High Bandwidth Interface) link, effectively creating a single GPU from two reticle-limited dies. This dual-die approach was necessary because the maximum chip size that current lithography can produce, approximately 858 square millimeters, was insufficient for Blackwell's transistor budget. The packaging complexity of bonding two dies with 10,000 signal connections at high yield is a manufacturing achievement that took TSMC and NVIDIA over two years to perfect.

Geopolitically, this supply chain is a source of strategic concern. The United States has imposed export controls restricting the sale of advanced GPUs to China, citing national security. NVIDIA has produced modified versions of the H100 and B200 with reduced interconnect bandwidth to comply with these controls, but the modifications reduce performance and have cost the company an estimated 5 to 10 billion dollars in lost revenue. The long-term question is whether TSMC's manufacturing monopoly on leading-edge chips is sustainable, and whether the CHIPS Act investments in US and European fab capacity can create a more distributed supply chain without sacrificing the yield and cost advantages of concentration.