Edge AI: How to Run Language Models on Phones, Laptops, and Microcontrollers

01 AI in Your Pocket

The same model that required a data center in 2023 can now run on a phone. A 1B parameter model generates 35 tokens/second on an iPhone 15 — fast enough for real-time chat. A 3B model runs at 15 tokens/second on an M2 MacBook. This wasn't possible a year ago. The combination of better quantization (Q4), optimized runtimes (llama.cpp, MLX, ExecuTorch), and efficient small models (Phi-3, Gemma 2, Llama 3.2) has brought AI to the edge.

02 The Battery Question

Running an LLM on a phone draws significant power. Our tests show a 1B model consumes 3-5 watts during inference — comparable to recording 4K video. At 35 tokens/second, generating a 500-word response takes about 30 seconds and consumes about 0.04 watt-hours. That's about 0.5% of a typical phone battery per query. Not catastrophic, but not free. The models that work on phones are the 1-3B class — larger models drain battery too fast and generate too much heat.

03 The Microcontroller Frontier

On a Raspberry Pi 5, a 1B model runs at 8 tokens/second — usable but slow. On an ESP32 microcontroller (the kind used in smart home devices), a 1B model is impossible — not enough RAM. But a 40M parameter model (Q4 quantized) fits in 20MB and runs at 2-3 tokens/second on an ESP32. This is the frontier of edge AI: models small enough to run on IoT devices with no operating system, no network connection, and milliwatt power budgets. Applications: smart sensors, voice control, predictive maintenance on factory equipment.