How does a 'Mixture of Experts' model achieve this?

A Mixture of Experts model divides its network into many smaller specialized sub-networks, called "experts," and uses a routing mechanism to decide which handful should process each piece of input. Rather than activating every parameter in the entire model for every request, only a small, relevant subset of experts switches on for any given prompt, letting the overall model hold an enormous total parameter count while keeping the cost of answering any one question much lower than using every parameter every time.

Deleting unused parameters permanently would remove the model's ability to call on them for different inputs later, defeating the purpose of having many specialized experts on hand. Running every parameter twice would simply double the cost rather than reduce it, the opposite of what this design achieves.

This architecture is part of why today's most capable models can hold parameter counts in the hundreds of billions yet still answer everyday queries at a speed and cost impractical if every parameter were used for every answer.