Mixture of Experts (MoE)
A mixture of experts is a model architecture with many specialized sub-networks, or "experts," and a gating router that activates only a few per input. This sparse activation lets total parameter count grow large while the compute used for any single token stays modest, improving efficiency at scale.
Definition
Mixture of Experts (MoE)In a dense model, every parameter participates in processing every token. A mixture-of-experts layer instead holds multiple parallel expert networks and a learned router that selects a small subset — often one or two — for each token. Only the chosen experts run, so the model can have far more total parameters than it activates per token.
This conditional computation lets MoE models scale capacity without a proportional increase in inference cost. The trade-offs include more complex training, load-balancing across experts, and higher memory to hold all experts even though only a few fire at a time.
Frequently asked questions
What is Mixture of Experts (MoE)?
A mixture of experts is a model architecture with many specialized sub-networks, or "experts," and a gating router that activates only a few per input. This sparse activation lets total parameter count grow large while the compute used for any single token stays modest, improving efficiency at scale.
Sources
Related terms
← Full AI & prompt engineering glossary
Put Mixture of Experts (MoE) to work with Prompeteer, the Agentic Contextual AI Platform →