Running LLMs Locally: A Complete Guide to Ollama, vLLM, and llama.cpp

01 Three Approaches

llama.cpp is the foundational project: a C++ implementation of LLM inference that runs on any CPU and most GPUs. It pioneered quantization (running models in 4-bit, 8-bit, and 16-bit precision) to fit large models in small memory. Ollama wraps llama.cpp with a simple CLI and API — it's the easiest way to run models locally. vLLM is built for production: it uses PagedAttention for high-throughput serving on GPUs and is the standard for API-compatible local deployments.

02 The Quantization Trade-Off

Quantization reduces model precision from 16-bit to 4-bit or 8-bit, shrinking memory usage by 2-4x with minimal quality loss. Q4 (4-bit) quantization reduces a 7B model from 14GB to 4GB — runnable on a laptop. The quality loss is 1-3% on most benchmarks. Q8 (8-bit) halves the model size with essentially zero quality loss but is slower because it processes more data. The sweet spot for most users: Q4 for speed, Q8 for quality, Q5_K_M as a compromise.

03 What Actually Works

On an M2 MacBook, llama.cpp runs a 3B model at 85 tokens/sec (Q4) — fast enough for real-time interaction. A 7B model runs at 25 tokens/sec — acceptable for interactive use. A 13B model runs at 12 tokens/sec — usable but slow. On a GPU (A100), vLLM serves a 70B model at 50+ tokens/sec to multiple users. The key insight: you don't need a data center to run useful AI. A $1,500 laptop can run a 3B model that handles most tasks as well as GPT-3.5. The era of AI as a cloud-only service is ending.