Mixture of experts is the architectural answer to a specific tension: capability scales with parameters, but so does the cost of using them. MoE breaks that link.
The structure
In a dense model every parameter participates in every token. In an MoE model certain layers are replaced by a set of parallel experts, each a subnetwork, together with a small router that decides which experts handle each token.
Typically only a couple of experts activate per token. The rest sit idle for that token and may activate for the next one.
The number that matters
This is where published parameter counts become misleading. A model advertised at several hundred billion parameters may activate only tens of billions per token. Compute cost tracks the active count, not the total.
So an MoE model can carry the knowledge capacity of something enormous while costing roughly what a much smaller dense model costs to run. When comparing models, active parameters is the figure that predicts serving cost, and it is frequently not the one in the headline.
What it does not save
Memory. Every expert has to be resident, because the router may select any of them for the next token. Memory requirements track total parameters, not active ones.
This is the central tradeoff. MoE converts a compute problem into a memory problem. That is a good trade in a datacentre with abundant memory and a bad one on constrained hardware, which is why MoE and edge deployment sit awkwardly together.
Routing, and why it is difficult
The router is trained alongside everything else, and training it well is the hard part of MoE.
Left alone, routers collapse: a few experts get chosen for nearly everything while the rest are never selected, never receive gradient, and never become useful. The model then has the memory footprint of many experts and the capability of a few. Preventing this requires explicit load-balancing pressure during training, and getting that balance right is where much of the engineering effort goes.
MoE against the other techniques
MoE is not compression. It is an architectural choice made before training, and it cannot be applied to a finished dense model the way quantization or pruning can.
It composes with them, though. MoE models are routinely quantised, and they can be distilled into dense students, which is often attractive precisely because it converts the memory burden back into a smaller resident model.
Reading a spec sheet
- Total parameters tells you memory required
- Active parameters tells you compute per token and roughly what it costs to serve
- Expert count and top-k tell you how sparse it is
- A dense model and an MoE model with the same headline number are not comparable in any useful sense
Common questions
What is a mixture of experts model?
A mixture of experts model replaces some dense layers with many parallel expert subnetworks plus a router that selects a small number of them per token. Total parameter count is very large, but only the selected experts run, so compute per token stays low.
What is the difference between total and active parameters?
Total parameters are everything the model contains and must be held in memory. Active parameters are the subset actually used for a given token. In an MoE model active parameters can be a small fraction of the total, which is why comparing headline parameter counts across dense and sparse models is misleading.
Are MoE models cheaper to run?
Cheaper in compute, not in memory. You still have to hold every expert available, so memory requirements track total parameters. The saving is in arithmetic per token, which is why MoE suits high-throughput serving with plenty of memory rather than constrained hardware.