Why Every Frontier Model Is Now a Mixture-of-Experts

By Thorsten Meyer

Buying for a business?Offer from Amazon

Get business pricing on tech for your team

• Business-only prices and quantity discounts
• Tax-exempt purchasing
• Multiple users, one account, clear invoices

Create a free Business account →Free to register

As an affiliate, we earn on qualifying purchases.Hide

If you have read almost anything I have written this year about running models locally, you have run into the same two numbers side by side — a total parameter count and a much smaller “active” one. Kimi K3: 2.8 trillion total, roughly 104 billion active. DeepSeek’s flash line: hundreds of billions total, a low-double-digit-billion active. Qwen3 235B: 235 total, a fraction firing per token. I keep invoking that split because it governs almost every practical decision in local inference — and I keep not stopping to explain it properly.

This is that explanation. Because the reason every serious open model in 2026 quotes two parameter counts instead of one is that they are all now mixtures of experts, and understanding what that means is the single highest-leverage concept for reasoning about how these models run, what they cost, and why they behave the way they do.

The dense model, and the wall it hit

Start with what came before, because MoE is a solution to a specific problem.

A traditional "dense" transformer uses every one of its parameters for every token it processes. A 70-billion-parameter dense model does roughly 70 billion parameters' worth of computation to produce each token — all of it, every time. This is simple and it works, and it has one fatal property at scale: the cost of running the model and the cost of making it smarter are welded together. To make a dense model more capable, you make it bigger, and every parameter you add gets paid for on every single token, forever, in both compute and memory bandwidth. Capability and running-cost rise in lockstep.

That lockstep is the wall. Somewhere past a few hundred billion dense parameters, the per-token cost of using all of them for everything becomes economically absurd — you are paying to run the entire brain to answer "what's the capital of France?" The industry needed a way to keep growing total knowledge without growing the per-token bill at the same rate. Mixture-of-experts is that way.

The core idea: a big brain, but only part of it fires

Here is the whole concept in one image. Instead of one monolithic network where everything activates for every token, an MoE model splits much of its capacity into many parallel sub-networks called experts — dozens, hundreds, sometimes many hundreds of them. For each token, a small, fast component called the router looks at what is being processed and selects only a handful of experts to actually run. The rest stay dormant for that token.

So the model contains an enormous amount of total knowledge spread across all its experts, but any given token only pays to activate a small slice of it. K3's 2.8 trillion parameters are all there, holding learned knowledge — but only about 104 billion of them do any work on each token. You get the breadth of a giant model at roughly the per-token compute of a much smaller one. That is the trick, and it is why the technique took over: it breaks the lockstep. You can grow total capability by adding experts without growing the per-token compute proportionally.

The "experts" are not neatly specialized the way the name suggests — this is the one place the metaphor misleads. They do not divide up cleanly into "the law expert" and "the Python expert." What each expert learns is emergent and mostly not human-interpretable; the router learns, during training, which combinations of experts to consult for which patterns, and the specialization is statistical rather than labeled. Think of it less as a panel of named specialists and more as a large set of learned sub-circuits that the router has learned to mix.

The two numbers, and why each governs a different cost

Now the payoff, because this is the part that actually matters when you run these models, and it is the source of the most common expensive mistake I see.

Total parameters set your memory requirement. Active parameters set your speed. These are two different costs, paid to two different parts of your hardware, and MoE splits them apart in a way that dense models never forced you to think about.

Memory is governed by the total, because every expert must be resident in memory to be selectable. The router might choose any of them for the next token, so all of them have to be loaded and ready, even though most sit idle at any given moment. This is why K3 needs hundreds of gigabytes of memory even though it only "uses" 104 billion parameters per token — all 2.8 trillion have to be in the room, waiting to be called. You pay for the whole brain in RAM whether or not it all fires.

Speed is governed by the active count, because only the selected experts do computation. Generating a token costs roughly what a dense model of the active size would cost — so K3 generates at something closer to the speed of a 100-billion-class model than a 2.8-trillion one. This is the gift: frontier-scale knowledge at mid-scale generation speed.

The expensive mistake follows directly from missing this split. People see "104 billion active" and provision hardware for a 104-billion model, then discover the thing will not load because the memory bill was written by the 2.8-trillion total. Or they see "2.8 trillion" and assume it will crawl, then are surprised by how fast it generates. On my own fleet, this is the first calculation for any new model: total parameters against my memory ceiling to see if it fits at all, then active parameters against my bandwidth to estimate how fast it will run. Two numbers, two questions, two different pieces of hardware.

Why this shape is everywhere in 2026

Once you see the split, it becomes obvious why every lab converged here, and why the whole open-weight frontier looks the way it does.

MoE is what makes trillion-parameter open models runnable at all. A 2.8-trillion dense model would be economically impossible to serve — every token would pay the full freight. As a mixture of experts, the same total knowledge becomes serveable, because the per-token compute is set by the active slice. The technique is the reason the open frontier could scale total capability into the trillions without the serving cost scaling with it. Every headline open model this year — the Kimi line, DeepSeek, Qwen's large models, the big Chinese releases broadly — is MoE, and they quote two parameter counts for exactly this reason.

It also reshapes the hardware underneath, which is why this concept sits under everything I have written about inference silicon. An MoE's memory-heavy, compute-light profile — vast weights that must all be resident, but only a fraction active per token — is a different hardware problem than a dense model's, and it pushes directly toward the pooled-memory, high-bandwidth, cluster-as-one-brain direction the whole industry is moving. The experts can even be distributed across machines, each holding a share, which is precisely how enormous MoE models get served across clusters. When I run a large MoE across unified-memory Apple silicon, I am relying on exactly this: the total has to fit in the pooled memory, and the active slice is what actually moves per token.

And it interacts with quantization in a way worth flagging, since it ties these threads together. The router is one of the parts of the network most sensitive to aggressive quantization — squeeze it too hard and it starts selecting the wrong experts, and the model degrades in ways a simple accuracy number can miss. This is exactly why calibrated, mixed-precision quantization protects the router while compressing the bulk. The MoE structure is why dynamic quantization has to be selective rather than uniform. The concepts are the same story told from different angles.

The one honest caveat

MoE is not free, and the tradeoff is worth stating plainly rather than selling past.

You pay in memory for capability you are not using on any given token. A dense model of the active size would need far less memory; an MoE demands that you house the entire set of experts to get the benefit of the few. For a datacenter with pooled memory this is a fine trade — memory is comparatively cheap, and the compute savings dominate. For a memory-constrained local setup it is the binding constraint, the thing that decides whether a model runs at all. That is the tension at the center of local inference in 2026: the models got dramatically more capable and more compute-efficient per token, and in exchange they got hungrier for total memory. Which is exactly why the quantization work matters so much — it is how you fit the whole brain into a room that is too small for it at full precision.

The one-line version

Every frontier open model is now a mixture of experts because it is the only way to keep growing total capability without growing the per-token cost in lockstep. The router fires a small slice of a huge brain for each token, which splits your costs cleanly in two: total parameters decide whether it fits in memory, active parameters decide how fast it runs. Learn to read those two numbers as two separate questions and you can predict, before downloading anything, both whether a model will run on your hardware and how it will feel when it does. That single skill is most of what practical local inference actually requires.

Analysis from the perspective of a builder and founder running a local-first inference operation. Architecture descriptions (total-vs-active parameters, routing, expert distribution, quantization sensitivity of the router) reflect the established design of current mixture-of-experts models; specific parameter counts for named models are as published by their developers and are point-in-time. Fleet-specific reasoning reflects the author's own DojoClaw setup. Point-in-time as of 7 August 2026.