Mixture of Experts: How Models Get Smarter Without Getting Bigger

01 The Routing Problem

A Mixture of Experts (MoE) model has multiple 'expert' sub-networks, each specialized in different types of inputs. For each token, a 'router' decides which 2-4 experts to send it to. Only the selected experts are activated, so the model uses far fewer parameters per token than a dense model of the same total size. DeepSeek V3 has 671B total parameters but activates only 37B per token — making it as fast to run as a 37B model while having the knowledge capacity of a 671B model.

02 The Efficiency Gain

MoE models get 'more bang for the buck' because different experts specialize. One expert might handle math, another language, another code. This specialization is more efficient than having all parameters participate in every token. The efficiency gain is roughly 5-10x: a 671B MoE model with 37B active can match a 300-400B dense model on quality while being 8-10x faster for inference. This is why DeepSeek V3, trained for $5.6M, matches models that cost $100M+ to train.

03 The Challenges

MoE models have unique challenges. Load balancing — if all tokens go to the same expert, you lose the efficiency gain. Routing instability — small changes in the router can send tokens to different experts, creating inconsistent behavior. And training complexity — MoE models are harder to train than dense models, requiring careful tuning of the router loss function. But the efficiency advantages are so large that most frontier models (GPT-4, Mixtral, DeepSeek V3, rumored Llama 4) are now MoE architectures. The future of large models is likely sparse, not dense.