What Is a Mixture-of-Experts (MoE) Model? A Plain-English Guide
A mixture of experts (MoE) model is a neural network built from many specialized sub-networks, called experts, plus a small router that decides which experts handle each piece of input. Instead of running every parameter on every token, as Wikipedia’s entry on the architecture describes it, an MoE model activates only a handful of experts per token — why MoE has become the default architecture for the largest language models being trained today.
The idea isn’t new, but it took a decade of scaling work to become practical. In this guide from bota.chat: what MoE means, how the router decides, why “sparse” is the operative word, and how leading models put the idea to work.
What “Mixture of Experts” Actually Means
At its core, an MoE model replaces a single large feed-forward block inside a transformer with several smaller feed-forward sub-networks sitting side by side. Each sub-network is called an expert, and despite the name, no expert is trained to “know” a specific subject like grammar or math — experts specialize on patterns that emerge naturally during training, such as certain token types or reasoning steps.
The concept traces back to 1991, when Jacobs, Jordan, Nowlan, and Hinton published “Adaptive Mixtures of Local Experts” in the journal Neural Computation, describing a system that split a task among several networks combined through a separate gating mechanism. That design is a direct ancestor of the mixture-of-experts architecture used in today’s large language models, though the scale has grown by orders of magnitude.
A useful mental model: think of a hospital intake desk. It doesn’t treat every patient itself — it reads the symptoms and sends the patient to the two or three specialists best suited to the case, while the rest of the staff stays free for others. That intake desk is the router, the second piece of any MoE model.
Every mixture-of-experts layer is built from the same pieces:
- A bank of experts — smaller feed-forward sub-networks in parallel.
- A router (gating network) — scores each expert and picks which run.
- A combination step — merges the selected experts’ outputs.
- Optionally, a shared expert that stays active for every token.
How the Router Picks Experts
The router, sometimes called the gating network, is a small trainable component sitting in front of the bank of experts. For every token, it computes a score for each expert and selects the top few — a method known as top-k routing. Typical values of k range from one or two experts in earlier designs up to several dozen in newer sparse architectures.
This routing happens per token, not per sequence. A single sentence can send its first word to one set of experts and its fifth word to a completely different set, all within the same forward pass. The router’s scores are learned jointly with the rest of the network, so the split of labor emerges from training data rather than being hand-designed. Keeping that division balanced — so a handful of experts don’t absorb most of the traffic while others sit idle — is one of the harder engineering problems in building an MoE model, usually addressed with an auxiliary load-balancing loss during training.
Only the selected experts run their computation for that token, while the rest of the network’s expert weights stay dormant. That selective activation is what separates MoE from a conventional dense architecture and is the whole reason the approach scales so efficiently.
A single routing decision looks like this:
- The router receives the token’s hidden representation.
- It scores every expert in the layer.
- It ranks the scores and keeps the top-k experts.
- Each selected expert processes the token independently.
- The router combines the outputs, weighted by their scores.
Active vs Total Parameters (Why MoE Is “Sparse”)
MoE models are described using two parameter counts instead of one: total parameters and active parameters. Total parameters count every weight across every expert, including ones unused for a given token. Active parameters count only the weights touched during that token’s forward pass — the router plus the small subset of experts it selected.
This gap between the two numbers is why the term “sparse” applies. A model can carry hundreds of billions of total parameters for storing broad, varied capacity, while spending compute closer to that of a much smaller dense model on every single token. The table below illustrates the pattern with representative figures for context.
| Model type | Total parameters | Active parameters per token | Compute per token |
|---|---|---|---|
| Dense model | 70B | 70B (100%) | High, fixed |
| Sparse MoE model | 400B+ | ~30-40B (under 10%) | Low, variable |
Because only a fraction of the network fires per token, engineers can grow total capacity without a proportional rise in inference cost — a tradeoff that matters directly for the kind of long-context reasoning covered in our breakdown of the 1M token context window, where cost per token has an outsized effect on what’s usable in production.
MoE vs Dense Models
A dense model activates every parameter for every token, no exceptions. That uniformity makes dense models simpler to reason about, but cost scales linearly with size: doubling the parameter count roughly doubles the compute per token.
Where Dense Models Still Win
Dense architectures avoid the routing overhead and load-balancing headaches of a gating network. Smaller dense models are also easier to fine-tune predictably, since every parameter sees every training example.
Where MoE Pulls Ahead
MoE models decouple capacity from cost: a well-tuned sparse model can match or beat a dense model several times its active-parameter size, since the total parameter pool stores far more knowledge without paying for it every pass. The tradeoff is added complexity — routing logic, load balancing, and communication overhead across hardware.
| Factor | Dense model | MoE model |
|---|---|---|
| Compute per token | Fixed, scales with full size | Lower, scales with active experts only |
| Training complexity | Simpler, no routing | Higher, needs load balancing |
| Capacity scaling | Costly to grow | Cheaper to grow total parameters |
| Best fit | Small, latency-sensitive tasks | Largest flagship models |
As Noam Shazeer and his co-authors put it in the paper that revived the architecture for deep learning:
We introduce a sparsely-gated mixture-of-experts layer (MoE), consisting of up to thousands of feed-forward sub-networks. A trainable gating network determines a sparse combination of these experts to use for each example.Shazeer et al., “Outrageously Large Neural Networks” (arXiv:1701.06538)
Real-World MoE Models
The modern lineage of mixture-of-experts language models starts with the 2017 paper “Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer,” which showed that inserting an MoE layer between LSTM layers could scale model capacity over 1,000x with only minor increases in computation — a pattern nearly every subsequent MoE language model has followed.
A short list of milestones shows how quickly the architecture matured:
- 1991 — Adaptive Mixtures of Local Experts introduces the gating concept.
- 2017 — The Sparsely-Gated MoE Layer applies it inside deep LSTM networks at scale.
- 2021 — Switch Transformer routes to a single expert per token, training a model past a trillion parameters.
- 2023-2024 — Mixtral popularizes open-weight sparse MoE models; DeepSeek’s MoE line separately introduces a shared expert that stays active alongside the routed ones.
- 2024-2026 — Frontier labs adopt MoE as the default for their largest models.
Moonshot AI’s flagship release is a current example of that trajectory. Its LatentMoE architecture routes each token to just 16 of 896 available experts, keeping active compute low while total capacity stays enormous — a design detailed further in the guide to Moonshot’s Kimi K3. The company behind that architecture, profiled in our overview of Moonshot AI, built its research program around pushing sparse routing further than earlier open releases.
Not every large model follows this path — some labs still prefer dense architectures for smaller, latency-sensitive deployments, reserving MoE for flagship releases. Common threads across MoE releases include:
- A router trained jointly with the experts.
- A load-balancing mechanism to prevent expert collapse.
- A meaningful gap between total and active parameter counts.
- At least one shared or always-active expert in newer designs.
FAQ
What is a mixture of experts in simple terms?
A neural network made of smaller sub-networks (experts) plus a router that picks a few relevant experts per input, instead of using the whole network every time.
How does the router in an MoE model work?
It scores every available expert for each token and selects the top few (top-k routing). Selected experts process the token; the rest stay inactive.
What is the difference between active and total parameters?
Total parameters count every weight in the model, including unused experts. Active parameters count only the weights used for one token — usually a small fraction of the total.
Is MoE better than a dense model?
MoE offers more capacity per unit of compute, since only a subset of experts runs per token. Dense models remain simpler to train, which still suits smaller or latency-sensitive use cases.
What are examples of mixture-of-experts models?
Switch Transformer, Mixtral, and Moonshot AI’s flagship LatentMoE model (routing to 16 of 896 experts) are widely cited examples in production language models.