The Local-AI Hardware Ladder: What Each Tier of Silicon Actually Buys

01The Market That Buyer's Guides Built

Local inference crossed from hobbyist project to product category faster than the coverage around it adapted. A shelf of buyer's guides now ranks laptops, desktops, and single-board machines against one another, usually by benchmark wins and spec-sheet arithmetic. That framing quietly misreads the market. What actually sorts this hardware into tiers is not processing power but memory, and the market is better read as a ladder of memory classes than as a race.

The productive question is not which chip wins a benchmark but which class of model a machine can hold, and how quickly it can feed that model while answering. Capacity decides what fits at all; bandwidth decides whether tokens arrive at a usable pace. Set against those two numbers, raw compute is the abundant ingredient.

Read as a ladder, the market starts making sense: why neural engines are marketed in trillions of operations per second, why Apple machines punch above their price class, why do-it-yourself builders converge on one specific card, and why every ladder eventually meets a rental counter.

02Rung One: NPUs and What They Cannot Do

The bottom rung is the neural processing unit now built into mainstream laptop and desktop processors, sold on a single number: TOPS, trillions of operations per second. The figure is real but narrow. It measures throughput for low-precision matrix arithmetic, which is exactly the work done by small convolutional and encoder-style networks, and almost none of the work done by a large language model.

What NPUs actually run is a fixed entourage: camera framing, background blur, live captions, transcription front-ends, and other quantized encoders measured in hundreds of millions of parameters. A chat-scale model is out of reach twice over. The engine has no path to gigabytes of weights, and the dual-channel DDR5 that feeds the rest of the system, at roughly 100 GB/s of shared bandwidth, would starve text generation to a crawl.

That gap between the marketing number and the actual workload explains much of the category's confusion. TOPS implies on-device assistants; the delivery mechanism for anything conversational quietly remains the cloud, or a different piece of silicon. Rung one buys efficiency and always-on features, not model ownership.

03Rung Two: Unified Memory and the Mac Exception

The second rung belongs to unified memory, and in practice it belongs to one vendor. Apple's M4 Max can be configured with up to 128 GB of unified memory addressable by the GPU, fed at a manufacturer-stated 546 GB/s. No Windows notebook ships an equivalent pool of GPU-visible memory at consumer prices, and that asymmetry defines the rung.

The advantage is structural rather than cosmetic. Text generation is bandwidth-bound, so a machine whose GPU can address the whole memory pool holds genuinely large quantized models, in the 30B to 70B class, on a laptop that also runs all day unplugged. The conventional x86 layout splits the same job: modest system RAM beside the processor and a small, separate pool of VRAM on a discrete card.

The exception exists because Apple owns the memory subsystem end to end, while PC makers assemble parts chosen by others. That is also the rung's ceiling: 546 GB/s is generous for a laptop and still well short of what a top desktop graphics card moves.

04Rung Three: VRAM Stacks and the DIY Route

The third rung is the discrete VRAM stack, and its reference point is NVIDIA's GeForce RTX 5090: 32 GB of GDDR7 moving a manufacturer-stated 1,792 GB/s. Capacity and bandwidth lead the consumer market simultaneously, which makes the card the fastest token-generation machine an individual can simply buy.

The DIY route assembles around that ceiling. Builders pair the card with open-source runtimes, tune quantization by hand, and accept a hot, expensive machine in exchange for inference no laptop rung can match. But 32 GB is a hard wall. A 70B-class model at 4-bit occupies roughly 42 GB, so the most-discussed open weights do not fit on one card, and multi-card setups trade money and complexity for capacity.

This rung therefore marks the practical border of consumer local AI. Everything whose weights fit under 32 GB runs comfortably; everything past it belongs to specialists willing to engineer around the wall with a second card or a used server.

05The Cloud Line: Where Local Stops Making Sense

Above the 70B class the ladder simply ends. A 405B-parameter model at 4-bit needs on the order of 230 GB before a single token is produced, more memory than any consumer machine addresses, and the frontier keeps climbing the same curve. Past that line, local stops being a purchase decision and becomes a physics problem.

Cloud rental resolves the problem by converting capital into operating expense: per-token pricing, elastic capacity, and no machine humming in the study. For frontier-class work the comparison is not close, and it is not really about privacy or monthly cost. The large end of the open-weight world is locally unavailable, whatever a buyer is willing to spend.

The line does drift. Better quantization and aggressive distillation pull each model generation down roughly one size class, which is how systems that once demanded a data center now idle on the second rung. But in every cycle so far, the top of the capability distribution has stayed ahead of what consumer memory pools can hold.

06Who Each Rung Is Actually For

The rungs map cleanly onto user classes once the marketing is set aside. Rung one is not a platform anyone chooses; it ships inside ordinary laptops and serves everyone implicitly. Rung two belongs to privacy-conscious professionals and application developers who want a capable model on a machine they also carry, which is why the Mac keeps winning the category without appearing to campaign for it.

Rung three is the territory of tinkerers, researchers, and independent developers, people for whom experimentation velocity or data custody outweighs a 600-watt power budget and an open chassis. Above the cloud line sit the teams whose products need frontier quality on demand; they rent, and the hardware decision dissolves into a monthly bill.

Priced per unit of capability, the middle rungs are the most expensive, while the cloud is cheapest per token and owns the least. Each rung is ultimately a different trade between ownership and ceiling, not a step on one quality scale.

07What Redraws the Ladder Next

Three forces could redraw the ladder, and they are not symmetric. The deepest would be cheap high-bandwidth memory reaching consumer platforms, because capacity has grown affordable for years while bandwidth remains the scarce, proprietary trait. The Mac exception is fundamentally a bandwidth exception, and it narrows the day a competing memory stack ships.

Quantization is the second force: every gain that preserves quality at lower precision slips each model beneath a smaller rung's ceiling. Distillation is the third, converting frontier behaviors into compact students that consumer pools can hold. Neither raises the ladder; both lower the models toward it.

Watch memory pricing rather than TOPS. Until bandwidth democratizes or distillation stalls, the market stays a ladder of memory tiers, and each rung keeps selling a different relationship to the same models: features, ownership, craftsmanship, or rent.