---
title: Mixture of Experts (MoE) — AI Glossary
description: A mixture of experts is a model architecture with many specialized sub-networks, or "experts," and a gating router that activates only a few per input. This sparse activation lets total parameter count grow large while the compute used for any single token stays modest, improving efficiency at scale.
canonical: https://prompeteer.ai/glossary/mixture-of-experts
---

# Mixture of Experts (MoE)

A mixture of experts is a model architecture with many specialized sub-networks, or "experts," and a gating router that activates only a few per input. This sparse activation lets total parameter count grow large while the compute used for any single token stays modest, improving efficiency at scale.

## Definition

In a dense model, every parameter participates in processing every token. A mixture-of-experts layer instead holds multiple parallel expert networks and a learned router that selects a small subset — often one or two — for each token. Only the chosen experts run, so the model can have far more total parameters than it activates per token.

This conditional computation lets MoE models scale capacity without a proportional increase in inference cost. The trade-offs include more complex training, load-balancing across experts, and higher memory to hold all experts even though only a few fire at a time.

## Sources

- [Shazeer et al. (2017), Outrageously Large Neural Networks: The Sparsely-Gated MoE Layer](https://arxiv.org/abs/1701.06538)
- [Fedus et al. (2021), Switch Transformers](https://arxiv.org/abs/2101.03961)

## Related terms

- [inference](https://prompeteer.ai/glossary/inference)
- [quantization](https://prompeteer.ai/glossary/quantization)
- [throughput](https://prompeteer.ai/glossary/throughput)