The Sparse Brain: How Mixture of Experts Scales LLMs Without Scaling Cost

MoE models (amber/green) show a large gap between total and active parameters. Dense models (blue) use all parameters for every token. The active parameter count determines inference cost.

01 The Problem With Dense Models

A dense language model uses every one of its parameters to process every token. A 70-billion-parameter model like Llama 3 70B activates all 70 billion parameters for every word it reads and every word it generates. This is architecturally simple and easy to optimize, but it creates a hard coupling between model quality and inference cost. To make the model smarter, you add parameters. To add parameters, you increase the compute required for every forward pass. A model twice as large costs twice as much to run, regardless of the input.

This coupling is the fundamental scaling problem for dense models. The relationship between parameter count and model quality is roughly logarithmic: doubling parameters yields a predictable but diminishing improvement in loss. The cost, however, is linear. Double the parameters, double the inference compute. The economics are brutal. Each incremental improvement in model quality costs as much as the last one, but delivers less. Dense scaling eventually produces a model that is marginally better than its predecessor but costs twice as much to serve.

02 The Mixture of Experts Insight

Mixture of experts, or MoE, breaks the coupling between total parameters and active compute. The idea is older than large language models. The technique was introduced in the 1990s as a form of ensemble learning, where multiple specialized models, called experts, each learn to handle a subset of the input space. A gating network, trained alongside the experts, learns to route each input to the expert best suited to handle it. The result is a system that has the capacity of many models but only runs the relevant ones for any given input.

Applied to language models, the architecture replaces the dense feed-forward network in each transformer layer with a set of expert networks and a router. When a token enters a layer, the router evaluates it and selects a small number of experts, typically two out of eight, or two out of sixty-four. Only the selected experts process the token. The rest sit idle. The total parameter count is the sum of all experts, but the active parameter count is only the selected few. The model has the capacity of a large network but the inference cost of a small one.

03 How Routing Works

The router is a small neural network, typically a single linear layer followed by a softmax, that takes the token's hidden representation as input and outputs a probability distribution over the available experts. The top-k experts, usually the two highest-probability experts, are selected. The token's representation is then processed by those experts, and their outputs are combined using the router's probabilities as weights.

The routing decision is made independently for each token at each MoE layer. A single sentence may route different tokens to different experts. A token that is part of a code block may go to one expert, while a token in the surrounding prose may go to another. The model learns which experts specialize in which types of input during training, without explicit supervision. The specialization emerges from the gradient signal: if an expert consistently receives tokens it processes well, the router learns to send more of those tokens to it. If an expert performs poorly, the router learns to avoid it.

04 The Sparse Activation Advantage

The key insight is that sparse activation decouples total capacity from per-token compute. DeepSeek V3 has 671 billion total parameters but activates only 37 billion per token. That means the model has the knowledge capacity of a 671-billion-parameter network, but the inference cost of a 37-billion-parameter model. The ratio matters. A dense 671-billion-parameter model would require roughly 18 times more compute per token than DeepSeek V3, while having the same total knowledge capacity.

This is why MoE has become the default architecture for frontier open-weights models. The training cost is higher because all parameters must be trained, but the inference cost is dramatically lower. For an open-weights model that users will self-host, inference cost is the primary constraint. A model that is cheap to run but has frontier-level knowledge capacity is exactly what the self-hosting ecosystem needs. MoE delivers that.

At equal total parameter count, MoE models require dramatically fewer FLOPs per token. A 64-expert MoE uses roughly 6 percent of the compute of a dense model with the same total capacity.

05 The Memory Overhead Tradeoff

MoE is not free. The total parameter count determines memory requirements, not the active parameter count. A 671-billion-parameter MoE model must load all 671 billion parameters into GPU memory, even though only 37 billion are active for any given token. This means an MoE model requires the same memory as a dense model of the same total size, while delivering less compute per token. The tradeoff is memory for compute.

This tradeoff has practical consequences. A dense 70-billion-parameter model fits on a single 80-gigabyte GPU. A 671-billion-parameter MoE model requires multiple GPUs just to hold the weights, even though its active compute is comparable to the 70-billion dense model. The MoE model is cheaper to run per token but more expensive to provision. This favors cloud providers and well-funded labs that can afford the memory overhead, while making MoE models harder to self-host for smaller organizations. The efficiency gain is real, but it shifts the cost from compute to memory.

06 Load Balancing and Routing Instability

The router faces a fundamental optimization challenge. If it sends all tokens to the same few experts, those experts become overloaded while the rest sit idle. If it distributes tokens evenly regardless of expertise, the model loses the benefit of specialization. The training process must balance these forces, and it does so through auxiliary loss functions that penalize uneven expert utilization. The auxiliary loss encourages the router to distribute tokens across experts, while the primary task loss encourages it to route tokens to the experts that process them best.

This balance is fragile. During training, routers can collapse, concentrating on a small number of experts and rendering the rest useless. They can also oscillate, shifting tokens between experts without converging on a stable specialization. These failure modes are well-documented and are the subject of active research. Techniques like expert choice routing, where experts select tokens rather than tokens selecting experts, and capacity constraints, which limit the number of tokens an expert can process, have been proposed to address these issues. The routing problem is not solved. It is managed.

07 Why MoE Is Becoming the Default

The trend is clear. DeepSeek V3, Mixtral, Grok 2, and reportedly GPT-5 all use some form of mixture of experts. The architectural pattern is converging because the economic pressure is overwhelming. Frontier models need more capacity than dense scaling can economically provide. The cost of training a dense trillion-parameter model, and the cost of serving it, is prohibitive for all but the largest organizations. MoE offers a path to that capacity at a fraction of the inference cost.

The implication for the model layer is significant. If frontier capability requires MoE, and MoE requires significant memory infrastructure, then the minimum viable cluster for frontier AI is defined by memory capacity, not compute. This favors organizations with large GPU fleets and the ability to dedicate them to holding model weights. It is a structural barrier that consolidates frontier AI among a small number of players, even as open-weights models make the trained models themselves freely available. The architecture of the model shapes the economics of the industry, and MoE is reshaping both.