Computer memory explained: the hierarchy that makes computers fast

01 THE GAP THAT SHAPED COMPUTING

Here is the central awkward fact of computer design: processors think in nanoseconds, and everything else makes them wait. A modern core can execute an instruction in roughly a third of a nanosecond - and then spend about 80 nanoseconds retrieving data from main memory. For every trip to DRAM, the processor could have done on the order of two hundred useful things. For a trip to an SSD, three hundred thousand.

That gap is not an engineering oversight; it is physics and economics pulling in opposite directions. Fast storage is small and expensive. Big storage is slow and cheap. No single technology wins on both axes, so no computer is built from just one.

Everything about how computers feel - why a program flies one moment and crawls the next, why adding RAM sometimes helps more than a faster chip - follows from how the industry manages that gap. The machine's answer is a hierarchy, and the hierarchy is the real memory system.

02 A LADDER BUILT FROM LATENCY

Climb the ladder from the top: registers inside the CPU, accessed in a fraction of a nanosecond. Then the L1 cache, tens of kilobytes per core, near 1 ns. Then L2 at roughly 4 ns, and a larger L3 shared among cores at roughly 12 ns. Below that, DRAM at around 80 ns, an NVMe SSD near 100,000 ns, and a spinning hard disk at about 10,000,000 ns. Each rung down buys orders of magnitude more capacity for orders of magnitude more waiting.

The ladder works because of locality. Programs tend to reuse what they just used, and to touch data sitting next to what they just touched. Caches exploit both tendencies automatically, keeping recently and predictably needed data close to the core without any instruction asking for it.

When it works, it works startlingly well: typical workloads post cache hit rates above 90 percent, so the processor mostly samples the fast end of the ladder while the terabytes below sit quietly. The average access time of the whole hierarchy can land within a small factor of the fastest tier - which is the entire trick.

03 SRAM VERSUS DRAM

Two technologies split the fast end of the ladder. SRAM - static random-access memory - stores each bit in a ring of six transistors that hold their state as long as power flows. No refresh, no settling time, access in about a nanosecond. It is also bulky and costly: six transistors per bit means that building main memory from SRAM would swallow an entire die for a few hundred megabytes.

DRAM - dynamic random-access memory - stores each bit as charge in a tiny capacitor guarded by a single access transistor. One transistor plus one capacitor per bit makes it roughly an order of magnitude denser and far cheaper, which is why main memory is measured in tens of gigabytes. The charge leaks away in milliseconds, so every row must be periodically read out and rewritten - hence dynamic.

The division of labor follows directly: SRAM for caches measured in megabytes close to the core, DRAM for main memory measured in gigabytes a bus ride away. Each technology is used exactly where its physics is the right trade.

04 CACHE: PREDICTING THE FUTURE

A cache is a bet that the near future resembles the recent past. Data moves between tiers in fixed blocks - typically 64-byte cache lines - organized so the hardware can find any line in a few gate delays. When requested data is present, it is a hit; when not, it is a miss, and the request falls down the ladder, stalling the core for tens or hundreds of cycles.

To keep the miss rate low, modern caches run prediction machinery around the clock. Prefetchers watch access patterns and fetch lines before they are asked for - they are strikingly good at spotting a sequential scan. Replacement policies guess which resident line is least likely to be needed and evict it. None of this is visible to the program; the hardware is quietly forecasting the future.

For programmers, the lesson is that memory access patterns are performance. Walking an array sequentially can be dozens of times faster than chasing the same amount of data through pointers, and restructuring code to touch memory in blocks can matter more than switching algorithms. The compiler helps, but the hierarchy rewards those who think about layout.

05 THE FALLING PRICE OF A BIT

The hierarchy's reach is a story about cost curves. In 1994, DRAM cost roughly 450 dollars per gigabyte - memory for a serious workstation was a bigger line item than the processor. By 2000 it was around a dollar per gigabyte; by 2020, about three tenths of a cent. That is a fall of roughly five orders of magnitude in twenty-six years.

The drivers compounded: lithographic shrinks packed more bits per wafer, gigabit-scale fabrication concentrated in a handful of East Asian plants pushed yields up and costs down, and standardized interface generations from SDRAM through DDR5 let the whole industry ride the same curve. Learning-curve economics at their purest.

The irony is that as capacity got cheap, distance got expensive. Capacity per DRAM chip grew far faster than latency improved - DRAM latency is only modestly better than two decades ago - so the gap the hierarchy must hide has widened even as filling the ladder costs less every year.

06 MEMORY IN THE AI ERA

AI workloads turn memory from a supporting character into the lead constraint. A model's weights must live close enough to compute to be streamed at enormous bandwidth, and they are large: a 7-billion-parameter model at 16-bit precision needs around 14 gigabytes before a single activation is stored. If the weights do not fit beside the accelerator, the accelerator waits.

The industry's answer is a GPU-side hierarchy topped by HBM - high bandwidth memory - stacks of DRAM dies sitting millimeters from the processor and delivering several terabytes per second, versus roughly 50 GB/s for a DDR5 channel on a desktop. Below that sits the familiar ladder: VRAM to system DRAM to NVMe, with offloading schemes trading speed for capacity.

Inference adds a second memory consumer: the KV cache, a running record of attention state that grows with context length and batch size. Much of practical AI engineering - quantization, batching, paged attention, memory mapping - is memory management under another name, and whether a workload is compute-bound or memory-bound decides which hardware is even worth buying.

07 LIMITS AND WHAT COMES NEXT

The honest limits: DRAM scaling has slowed sharply - bit density no longer doubles every generation or two - and SRAM cells scale worse than logic, so caches grow more slowly than everything around them. The capacitor at the heart of DRAM is already thousands of atoms wide; physics is no longer a distant advisor.

The counter-moves are architectural rather than material: 3D-stacked cache bolted onto processors, CXL fabrics that let machines pool memory across a rack, and processing-in-memory research that computes where the bits sit. History supplies a cautionary tale in Intel's Optane - genuinely novel technology that still lost to the economics of NAND and DRAM.

The safe prediction is continuity. The hierarchy has survived sixty years of revolutions elsewhere in the machine because it is not a technology but a bargain: trade capacity for distance, then hide the distance with prediction. Whatever the technologies on each rung become, the bargain will outlive them.