GPU Architecture: The Silicon Powering AI

01 From Pixels to Tensors: The GPU Origin

The graphics processing unit was invented to solve a specific problem: rendering 3D graphics in real time. A CPU processes tasks sequentially — one instruction at a time per core — which is fine for spreadsheet calculations but painfully slow when you need to compute the color of two million pixels sixty times per second. The GPU takes a fundamentally different approach: it packs thousands of simple cores onto a single chip and processes many pixels simultaneously. This architectural difference — throughput over latency — is the foundation of everything that followed.

The insight that changed computing came around 2007, when NVIDIA introduced CUDA, a programming model that allowed general-purpose computation on GPUs. Researchers realized that the same parallelism that made GPUs good at rendering pixels also made them good at matrix multiplication — the core operation in neural networks. A GPU can multiply large matrices far faster than a CPU because it can distribute the computation across thousands of cores. This discovery is what made modern deep learning practical. Without GPUs, training a model like GPT-3 would take years instead of weeks; training a frontier model would be simply infeasible.

02 The Core Architecture: SIMT and Parallel Processing

Modern GPUs use a single-instruction, multiple-thread (SIMT) execution model. Unlike a CPU, which might have 8 to 16 complex cores each running independent instructions, a GPU like the NVIDIA H100 has thousands of simpler cores organized into streaming multiprocessors (SMs). Each SM schedules threads in groups called warps (NVIDIA) or wavefronts (AMD). All threads in a warp execute the same instruction simultaneously, but operate on different data. This is efficient when the workload is uniform — every pixel needs the same shading calculation, or every matrix element needs the same multiply-accumulate.

The tradeoff is branch divergence. If threads within a warp take different paths through an if-else statement, the GPU must serialize both paths, reducing effective parallelism. This is why GPU code is optimized to minimize branching and why neural network operations — which are predominantly dense matrix multiplications with no conditional logic — map so well to the architecture. The SIMT model is not general-purpose in the way a CPU is, but for the workloads it suits, it achieves throughput that a CPU cannot match at any reasonable power budget.

Figure 1: Comparison of processing core counts across CPU and GPU architectures. The GPU's advantage is not per-core speed but sheer parallel throughput.

03 Memory Hierarchy: Bandwidth Is Everything

A GPU's compute cores are useless without data to process, and moving data is often the actual bottleneck. GPU memory hierarchies are designed for bandwidth — the rate at which data can be fed to the cores — rather than latency. At the top of the hierarchy are registers, which are fast but limited. Below that is shared memory, a small scratchpad that programmer-managed kernels use for inter-thread communication. Below that is L2 cache, and below that is the main GPU memory — typically HBM (high-bandwidth memory) on data-center GPUs or GDDR on consumer cards.

The numbers are striking. An NVIDIA H100 has 80 GB of HBM3 memory with a bandwidth of 3.35 terabytes per second. A consumer RTX 4090 has 24 GB of GDDR6X at 1.01 TB/s. For comparison, a high-end CPU DDR5 system delivers around 80 GB/s. This 40x bandwidth advantage is what allows GPUs to process the massive matrices that neural networks require. When an LLM generates a token, it must load model weights from memory for every forward pass — making memory bandwidth, not compute, the limiting factor for inference on large models.

04 Tensor Cores: Silicon Built for AI

The most significant architectural addition for AI workloads is the tensor core. Introduced with NVIDIA's Volta architecture in 2017, tensor cores are specialized units that perform mixed-precision matrix multiply-accumulate operations in a single clock cycle. A standard CUDA core can do one multiply-accumulate per cycle; a tensor core can do an entire 4x4 matrix multiplication per cycle. The H100's tensor cores handle FP16, BF16, INT8, and even FP8 precisions, with throughput measured in hundreds of teraflops.

This specialization matters because matrix multiplication is the dominant operation in neural networks. Every attention head, every feed-forward layer, every convolution is ultimately a series of matrix multiplies. By dedicating silicon to this operation, tensor cores achieve an order-of-magnitude speedup over general-purpose CUDA cores for AI workloads. Competitors have followed suit: AMD's Matrix Core technology and Google's TPU (tensor processing unit) take similar approaches. The result is that modern AI accelerators are no longer general-purpose GPUs with AI as a side effect — they are AI-specific processors that happen to share heritage with graphics hardware.

Figure 2: Tensor core performance across NVIDIA GPU generations. FP8 precision (faded bars) roughly doubles throughput. Values are vendor-published specifications for dense operations.

05 From Training to Inference: Different Workloads, Different Chips

AI workloads split into two categories with very different hardware requirements. Training requires massive compute and memory bandwidth to process batches of data through forward and backward passes. It is dominated by large matrix multiplications and benefits from high-precision arithmetic (FP16 or BF16). Training a frontier model requires clusters of thousands of GPUs connected by high-speed interconnects like NVIDIA's NVLink or InfiniBand, because the model's parameters must be synchronized across devices at each step.

Inference — running a trained model to generate predictions — has different constraints. It is often latency-sensitive (a chatbot must respond quickly) and memory-bound (loading model weights for each forward pass). Inference can use lower precision (INT8, FP8) to reduce memory and compute requirements. This has driven a market bifurcation: training clusters use the most powerful GPUs available, while inference can run on cheaper, more efficient hardware. Companies like Groq and Cerebras have built specialized inference chips that strip out graphics functionality entirely, optimizing purely for transformer inference throughput.

06 The Interconnect Problem: Scaling Beyond One Chip

Modern AI models are too large to fit on a single GPU. A 70-billion-parameter model in FP16 requires roughly 140 GB of memory — more than a single H100's 80 GB. This necessitates model parallelism: splitting the model across multiple GPUs. The challenge is communication. Each forward pass requires transferring activations between GPUs, and each backward pass requires synchronizing gradients. The interconnect bandwidth between GPUs determines how much of the total runtime is spent on computation versus communication.

NVIDIA's NVLink provides 900 GB/s between paired GPUs — roughly 7x faster than PCIe Gen5. For larger clusters, InfiniBand or RoCE (RDMA over Converged Ethernet) provides high-bandwidth, low-latency communication across hundreds of nodes. The software stack — NVIDIA's NCCL, Megatron-LM, DeepSpeed — orchestrates this communication automatically, but the physical interconnect remains a critical and expensive component. A large training cluster's cost is often dominated by networking, not by the GPUs themselves. This is why companies like Google design custom interconnects (TPU pod architecture) and why optical interconnects are an active area of research.

07 The Road Ahead: Custom Silicon and New Architectures

The GPU's dominance in AI is not guaranteed. Google's TPUs have been training and serving production models since 2015, and the TPU v5 generation offers competitive performance for both training and inference. Startups like Groq, Cerebras, SambaNova, and Tenstorrent are building chips with radically different architectures — wafer-scale engines, dataflow architectures, and spatial computing approaches that abandon the SIMT model entirely. The question is whether the GPU's software ecosystem (CUDA, cuDNN, and the vast body of GPU-optimized AI code) creates enough lock-in to keep it dominant even if alternative hardware is technically superior.

NVIDIA is not standing still. The Blackwell architecture, announced in 2024 and shipping in 2025-2026, introduces second-generation transformer engines, FP4 precision support, and a dual-die design that doubles the effective chip size. The Hopper-to-Blackwell transition represents a roughly 2.5x improvement in training performance and up to 30x improvement in inference for large models. Meanwhile, the open-source Triton compiler and efforts like OpenAI's Triton language are making it easier to target non-NVIDIA hardware, gradually eroding CUDA's moat. The next five years will determine whether AI compute remains a GPU business or fragments into specialized accelerators for each workload type. Either way, the silicon that powers AI is one of the most consequential technology categories of the decade.