Sparse arithmetic is not sparse time

A sparse mixture-of-experts layer replaces one dense feed-forward transformation with a router and a set of feed-forward experts. For each token representation x, the router scores the experts, chooses a small set, and combines their outputs. Early sparsely gated MoE work established the attractive scaling idea: grow model capacity far faster than per-example computation by activating only a small part of the parameter set.¹ GShard and Switch Transformers then showed how this idea could be distributed across many devices, with Switch simplifying routing to one expert per token in part to reduce routing and communication complexity.²³

The model-level arithmetic is easy to state. If there are E experts and each token selects k, then only a fraction roughly k/E of expert parameters is active for that token. This is a capacity statement. It says a model can hold many alternative parameter sets without evaluating all of them for every token. It does not say the routed layer will run E/k times faster than a dense alternative.

Three reasons break that inference. First, the dense comparison is usually not “activate all experts”. A credible baseline is a dense feed-forward block sized and trained to reach comparable quality under a comparable budget. Its kernels may be larger, regular and highly efficient. Second, the router adds scoring, selection, permutation and bookkeeping. Third, expert parallelism moves activations to the devices that own the selected weights and then moves outputs back. Sparse work can therefore reduce mathematical operations while increasing bytes, launches and synchronisation.

The decisive term is not the sum of expert work over the cluster. It is the exposed critical path. Let a_i be the processed assignments for expert i, let d(i) identify its device, and let c_i translate its batch into time. A first approximation is:

Active parameter count predicts how much expert arithmetic is requested; peak device completion time predicts when the synchronised layer can continue. Recent systems work makes the distinction explicit. Fine-grained MoE configurations can lower the computation-to-communication ratio because smaller expert matrices are less efficient and more expert choices create more traffic. Parallel mappings that work for attention can also place expert communication on slower links. In one reported large-scale training study, communication exceeded seventy per cent of MoE latency for particular fine-grained, inter-node configurations.¹¹ That number belongs to the reported setup, but the mechanism is general: a sparse layer can become a network-bound collection of small matrix multiplications.

Thought experiment: move only the experts

Freeze a trained model, a batch and every token-to-expert decision. Now change only expert placement. In layout A, the four hottest experts occupy four different devices. In layout B, they occupy one device while the other devices hold cold experts. The model output is identical because each token reaches the same weights. Total active arithmetic is identical. The communication volume may even be identical if both layouts cross the same number of rank boundaries.

Yet layout B finishes later because its hot device performs more serial-equivalent work. The intervention isolates placement as a cause. It also shows why a model-side load-balancing metric is insufficient. An expert histogram can look acceptable while the sum of several moderately hot experts creates a device hotspot. Conversely, unequal token counts can produce equal completion times if faster devices carry more work or if experts have different shapes.

The router creates a distributed queue

It is tempting to picture routing as a switch: each token chooses an expert, the expert computes, and the result returns. A distributed implementation is closer to a queueing network. Tokens begin on ranks determined by data, tensor or context parallelism. Router scores are computed locally or collectively. Tokens are grouped by destination expert, packed into contiguous buffers, exchanged across an expert-parallel group, processed in variable expert batches, exchanged back and restored to their original order.

The two network legs matter. Switch Transformer pseudocode contains an all-to-all for dispatch and another for the return path.³ Modern implementations add specialised dispatchers, fused permutation kernels, hierarchical collectives and overlap. These can reduce exposed cost, but they do not remove the causal chain. The router selects destinations; destinations induce traffic; traffic and expert batches determine the layer schedule.

Capacity is a policy, not a free buffer

Top-k routing does not guarantee equal expert batches. Switch Transformers defined an expert capacity proportional to tokens per batch divided by the number of experts, multiplied by a capacity factor. Assignments beyond capacity could skip expert computation and continue through the residual path; underfilled experts still occupied fixed-capacity slots in the tensor representation.³ A higher capacity factor drops fewer assignments but increases memory and padded work. A lower factor bounds buffers and work but changes the model computation for overflowed tokens. Capacity changes where the cost appears: in overflow, padding, memory or tail.

Dropless execution chooses another bargain. It processes every assignment, so quality is not altered by capacity overflow, but the system must handle unequal expert batch sizes. MegaBlocks replaces fixed padding with block-sparse operations and reported higher throughput than Tutel and a dense baseline in its studied training configurations.⁸ The result demonstrates that implementation can reclaim sparse work. It does not mean dropless kernels eliminate stragglers under every routing distribution.

Thought experiment: shuffle only time

Create twelve microbatches whose aggregate routing histogram is perfectly uniform. In sequence A, every microbatch is itself balanced. In sequence B, each microbatch concentrates on a different pair of experts, so the hot pair moves over time. At the end of the batch, both sequences report the same total count per expert. A monitoring dashboard that aggregates at batch level declares them equivalent.

They are not equivalent to the execution engine. Sequence B produces a straggler in every microbatch, but the identity of the straggler changes. Static expert reordering based on the aggregate cannot remove the within-batch storms. This exact temporal distinction has become important in reinforcement-learning training, where recent work reports that hot experts can shift rapidly between microbatches and uses routing replay to place coarse reordering and fine replication at different timescales.¹⁴

Load is a distribution over experts, devices, layers and time. An average over any one of those axes can conceal the variable that determines the barrier. The minimum useful trace therefore records routing per layer and microbatch, maps logical experts to physical devices, and joins that state to kernel and collective intervals.

Communication depends on placement and topology

For an activation vector of b bytes, each remote assignment sends roughly one vector to the expert and one result vector back. Ignoring protocol overhead and compression, remote payload is approximately twice the number of remote processed assignments times b. Latency also depends on message granularity, peers, topology, collective implementation and contention. A large byte count on a high-bandwidth intra-node fabric can be cheaper than many small transfers across nodes.

This is why expert parallelism cannot be tuned independently of tensor, context, data and pipeline parallelism. MoE Parallel Folding decouples attention and expert mappings so each layer type can use a topology more suitable to its communication pattern.¹¹ NVIDIA’s current Megatron Core guidance likewise treats token dispatch, expert parallel overlap and parallel folding as explicit configuration problems rather than automatic consequences of sparsity.¹²

flowchart TD
  A["The layer that waits for one<br/>device"] --> B["Sparse arithmetic is not<br/>sparse time"]
  B --> C["The router creates a<br/>distributed queue"]
  C --> D["Three objectives that do not<br/>coincide"]
  D -. "re-test" .-> A
  classDef first fill:#0a1424,stroke:#5B5CF6,stroke-width:2px,color:#e6f3ff
  classDef step fill:#0a1424,stroke:#3f7da0,stroke-width:1px,color:#e6f3ff
  class A first
  class B,C,D step

Three objectives that do not coincide

Routing discussions often collapse three questions into one. Did tokens reach experts that improve the model objective? Did the assignments use experts evenly enough to train and execute? Did the resulting batches run efficiently on the available system? These are related, but no one metric answers all three. Routing quality, load balance and hardware efficiency must be measured as separate objectives.

Routing quality

A learned router should send a token to parameters useful for transforming that token. Specialisation may therefore be intentionally non-uniform. Legal language, source code and multilingual morphology need not use experts at equal rates. An overly strong balancing pressure can redirect tokens away from the experts preferred by the language-modelling objective. Loss-Free Balancing addresses this tension by adjusting expert-specific routing biases from recent load rather than applying the balancing signal as an auxiliary gradient to the main objective.⁹ Its reported experiments support the method in the studied models; they do not remove the need to test routing quality after workload shift.

Load balance

One convenient measure is maximum load violation:

Expert Choice routing reverses the assignment direction: experts select top-scoring tokens up to a fixed bucket size. This gives each expert a controlled batch while allowing tokens to receive variable numbers of experts.⁶ The mechanism is attractive for training balance, yet its batchwise selection can conflict with causal generation because the choice can depend on other tokens in the batch. The Loss-Free Balancing analysis explicitly identifies future-token leakage for Expert Choice in causal language modelling.⁹ A balanced batch is not automatically an admissible autoregressive mechanism.

Hardware efficiency

Even perfectly equal expert counts can produce poor kernels. Each expert may receive too few tokens for efficient matrix multiplication. Top-two routing doubles assignments and can increase communication or make the most loaded expert a tighter bottleneck. In one Expert Choice experiment, a top-two variant had the same per-token expert count as its comparison but a slower step because the most loaded expert governed execution.⁶ The result is a reminder to trace the critical path rather than infer it from average work.

Hardware efficiency also depends on placement, memory residency, precision, fusion, local versus remote routes, topology and device speed. Tutel adapts parallelism and pipelines all-to-all traffic; MegaBlocks changes the sparse computation representation; MoE Parallel Folding changes the mapping; disaggregated serving systems independently scale attention and experts and attempt to hide their communication.⁷⁸¹⁰ These are systems interventions around the same learned model idea.

Negative control one: fixed hash routing

A learned router can appear indispensable because it is evaluated only against another learned router. Hash Layers supplies a serious negative control. It routes tokens with a fixed hash and reported competitive or better results than learned-routing approaches in the authors’ experiments, without router parameters, an auxiliary balancing loss or a sophisticated assignment algorithm.⁵ This does not prove hashing is generally superior. It proves that some gains attributed to learned specialisation may instead come from conditional parameter allocation, stable partitioning or additional capacity.

Any routing claim should survive a balanced, deterministic sparse baseline. The comparison separates the value of learning the route from the value of having multiple parameter sets. Hash routing may be easier to cache, plan and balance, while failing when semantic adaptation of the route is genuinely useful. That trade is empirical.

Negative control two: the dense layer

The other control is a strong dense model. Dense execution has no expert traffic, no overflow policy and no routing collapse. It can use large regular matrix multiplications and simpler parallel plans. At low concurrency, with slow interconnects or very fine-grained experts, those advantages can outweigh sparse arithmetic. MegaScale-Infer describes a serving regime in which MoE feed-forward execution becomes memory-intensive rather than compute-intensive, motivating disaggregation and request consolidation.¹⁰ The dense baseline is therefore not an obsolete straw model. It is the mechanism that tests whether conditional capacity has paid for its system.

Measure the break-even boundary

The right unit of evaluation is a quality-constrained operating point. Choose a workload distribution, sequence lengths, batch policy, latency target, hardware topology and model-quality threshold. Then compare end-to-end alternatives. A sparse model wins only if it meets the quality threshold and improves the chosen operational objective, such as tokens per second per device, cost per accepted output, or tail latency at fixed throughput.

A useful first-order break-even condition is:

The break-even inequality must hold at the percentile that governs the service. This formulation prevents three accounting errors. Summed kernel time is not necessarily elapsed time because kernels may run concurrently. Summed communication is not exposed communication because some intervals may overlap. Average device work is not critical work because synchronised execution waits for the slowest participant.

Training and serving need separate denominators

A training step and an autoregressive serving step expose different versions of the same route. Training usually supplies many tokens, runs both forward and backward passes, and can amortise expert kernels over a large global batch. The balancing mechanism also changes which experts receive gradients. A useful denominator is therefore time or cost per non-padding training token at a fixed learning outcome, with dropped assignments, backward collectives and optimiser state included.

Serving has at least two regimes. Prefill processes many input tokens together and can resemble a forward-only training batch. Decode may contain one new token per active sequence, so the number and composition of assignments change as requests arrive and finish. Expert weights can then be limited by memory bandwidth, while low concurrency creates small per-expert batches. A model that trains efficiently can consequently serve poorly without any contradiction.

The serving denominator should be chosen from the service objective: time per output token at a stated concurrency, sustainable tokens per second while meeting a tail-latency target, or cost per completed request at a stated quality threshold. Report prefill and decode separately. Mixing them into one average can make a long prompt hide slow generation, or make a short benchmark exaggerate fixed dispatch overhead.

Causal validity also belongs in this split. A batchwise routing method that chooses tokens for experts using information from the whole batch may be usable in training yet inadmissible for left-to-right generation. Expert Choice illustrates the issue: controlled expert buckets solve one balance problem, but later analysis shows how batchwise selection can reveal future-token information in a causal language model.⁹ Systems balance cannot be purchased by changing the computation being evaluated.

Use one break-even sheet for training, one for prefill and one for decode. They may select different expert placements, replication policies, parallel groups or even different model variants. The relevant comparison is not a checkpoint-level label such as “MoE is faster”. It is a workload-level claim whose denominator, quality constraint and hardware boundary are explicit.

Worked example: sixteen tokens, four experts, two devices

Consider top-one routing for sixteen tokens. Experts E1 and E3 sit on device A; E2 and E4 sit on device B. Each processed assignment costs one abstract compute unit, and we ignore network cost for the first comparison.

The total is sixteen in every row. Active arithmetic has not changed. The critical compute rises from eight to ten and then twelve units. Placement alone adds twenty per cent over the skewed spread case and fifty per cent over the balanced case. Add remote token movement and the difference can widen.

Now apply a capacity of five assignments per expert. The skewed E1 load of eight overflows by three. A fixed-capacity implementation can bound its compute, but three tokens no longer receive E1’s transformation. Increasing capacity to eight removes overflow while padding the other experts. Dropless execution preserves all assignments but returns to a critical load set by the hot device. There is no capacity setting that makes the quality, memory and timing trade disappear.

Worked scenario: a serving cluster under two traffic mixes

Take a synthetic serving design with sixty-four devices and two hundred and fifty-six logical experts. Every token selects two experts. Each activation vector occupies sixteen kibibytes including the chosen precision and representation. Experts are distributed across devices, and the serving scheduler forms microbatches from concurrent requests.

In traffic mix A, requests are diverse, routing is moderately dispersed and popular experts are spread across high-bandwidth groups. Large microbatches create useful expert GEMMs. Dispatch and return overlap with part of the expert work. The routed layer stays below its measured dense baseline at the p95 operating point.

In traffic mix B, a domain-specific burst concentrates routes on a small expert subset. The scheduler has lower concurrency, so expert batches shrink. Several hot experts share a slower device and many assignments cross the inter-node boundary. The same model now misses its per-token latency objective even though active parameters per token have not changed.

This is a worked scenario, not a deployment report. It exposes the variables that a real trace must replace: per-layer route counts, origin and destination ranks, expert-to-device mapping, device-specific kernel curves, bytes per assignment, collective topology, overlap and dense baseline. Recent serving systems respond to this variability with expert replication, placement, disaggregation and independent scaling. Janus, for example, treats attention and MoE resources separately and schedules expert activations under dynamic workloads; ViBE uses device-specific execution models; MegaScale-Infer consolidates expert work and pipelines communication.¹⁰¹³¹⁵ Their reported gains remain tied to their systems and evaluations.

What to instrument

Start with trace identities that survive aggregation: request, sequence, layer, microbatch, token origin rank, selected expert, expert host and model version. Add the following measurements:

• assignment counts per expert and per device, including p50, p95 and maximum across microbatches;
• capacity, overflow, dropped assignments, padded slots and tokens receiving zero, one or multiple experts;
• permutation, dispatch, expert kernel, return and restore intervals on a shared clock;
• remote assignment fraction, payload bytes, message-size distribution and links crossed;
• expert batch-size histograms and achieved kernel throughput rather than nominal peak FLOP/s;
• device-specific completion curves, thermal or power throttling indicators and slowest-rank identity;
• overlap measured as interval intersection, not assumed from enabled configuration flags; and
• quality and calibration by route policy, including domain and temporal slices.

The minimum release metric is not expert utilisation; it is quality-constrained end-to-end performance with the causal timing decomposition attached. Better load balance that increases loss is not a free systems win. A faster MoE kernel that moves cost into communication is not an end-to-end win. Higher average throughput while violating a p99 latency objective may be an operating loss.

Run the routing simulator

The following artefact is a first-order routing laboratory. It generates deterministic synthetic token routes, maps experts to devices, applies either dropless or fixed-capacity execution, estimates remote payload, models one slower device, and compares the resulting critical path with a configurable dense baseline. It is intentionally small enough to inspect.

Its assumptions are explicit. Each device’s local expert work is represented as serial-equivalent assignment time. One activation-sized payload travels out and back for each remote processed assignment. Collective latency scales with a simple logarithmic stage count. A user-set overlap fraction hides part of the smaller of compute and communication. Router and packing are fixed at 0.015 microseconds per assignment, and device zero receives the configured slowdown. Real collectives, kernels and schedulers require measured curves.

function estimate({loads, host, deviceSlowdown, expertUs,
                   remoteAssignments, activationBytes,
                   bandwidthGBs, latencyUs, overlap, denseUs, tokens}) {
  const devices = 1 + Math.max(...host);
  const work = Array(devices).fill(0);
  loads.forEach((n, expert) => work[host[expert]] += n);

  const deviceTime = work.map((n, d) =>
    n * expertUs * (deviceSlowdown[d] ?? 1));
  const expert = Math.max(...deviceTime);

  const bytesOutAndBack = 2 * remoteAssignments * activationBytes;
  const stages = devices === 1 ? 0 : Math.ceil(Math.log2(devices));
  const network = stages === 0 ? 0 : 2 * latencyUs * stages
                + bytesOutAndBack / (bandwidthGBs * 1000);
  const hidden = overlap * Math.min(expert, network);
  const moe = expert + network - hidden;
  const dense = tokens / devices * denseUs;

  return {moe, dense, speedup: dense / moe, deviceTime};
}

const positive = estimate({
  loads:[270,250,260,244], host:[0,1,2,3],
  deviceSlowdown:[1.02,1,1,1], expertUs:.65,
  remoteAssignments:220, activationBytes:8192,
  bandwidthGBs:200, latencyUs:8, overlap:.75,
  denseUs:1.4, tokens:1024
});

const negative = estimate({
  loads:[380,300,190,154], host:[0,0,2,3],
  deviceSlowdown:[1.12,1,1,1], expertUs:.65,
  remoteAssignments:700, activationBytes:8192,
  bandwidthGBs:25, latencyUs:25, overlap:.10,
  denseUs:1.4, tokens:1024
});

console.assert(positive.speedup > 1, 'positive case should win');
console.assert(negative.speedup < 1, 'negative case should lose');
console.log({positive, negative});

stateDiagram-v2
  [*] --> Claim
  state "Figure 1. Equal active<br/>arithmetic can produce unequal<br/>completion time" as Claim
  state "Thought experiment, move only<br/>the experts" as Test
  state "Figure 2. Expert-load weather map" as Revision
  Claim --> Test: expose mechanism
  Test --> Decision: evidence survives
  Test --> Revision: rival survives
  Revision --> Claim: narrow or rebuild
  Decision --> [*]

The decision instrument

An MoE design should pass through four gates. A “yes” at an earlier gate cannot compensate for a “no” later. The aim is not to eliminate sparsity; it is to make the capacity benefit pay its full routing bill.

Release tests that change the decision

Run an ablation ladder rather than one headline benchmark. First compare the dense baseline with the trained MoE. Then replace the learned router with a fixed balanced route where the model permits it. Next replay identical routes under alternative expert placements. Repeat with communication disabled or emulated locally to estimate the network tax. Change batch ordering without changing its aggregate histogram. Inject one slower device. Finally, rerun on domain-concentrated and low-concurrency traffic.

Each intervention removes a competing explanation. If learned routing does not beat hashing at matched quality, semantic routing has not earned its complexity. If placement changes latency while routes remain fixed, the bottleneck is physical. If local emulation is fast but the distributed run is slow, the network is exposed. If only the average batch works, the design lacks temporal resilience.

Operating sentinels

After release, watch for changes in the route distribution, not just model latency. A new domain mix can make different experts hot. A router update can change cross-node traffic without changing validation loss. A thermal or power condition can turn a previously acceptable device into a persistent straggler. A scheduler change can reduce microbatch size and collapse expert kernel efficiency.

Useful alarms combine cause and consequence: route divergence plus device-tail growth; remote fraction plus collective exposure; overflow plus quality shift; expert-batch shrinkage plus kernel-throughput loss. Each alarm should carry the model version, expert placement, parallel configuration and workload slice needed to reproduce the condition. Without that state, “MoE latency regressed” is an observation without an experiment.

The operational object is the routed system, not the checkpoint alone. Its release manifest must therefore version router parameters or rules, balancing settings, capacity policy, expert placement, dispatcher, kernels, parallel groups, precision, scheduler and fallback. A weight rollback that leaves a changed placement or dispatcher in place is not a reconstruction of the previous system.

Sparse capacity has to earn its route

A mixture-of-experts model offers a powerful bargain: more parameter capacity without evaluating every parameter for every token. The bargain is real, but it is incomplete. The router turns model choice into physical work. It determines which experts learn, which devices wait, which links carry activations and which buffers overflow.

The changed architecture decision is precise. Do not approve an MoE from active FLOPs or parameter counts. Approve it from a quality-constrained break-even trace that includes the slowest device, the two-way token traffic, actual kernel efficiency and exposed overlap. Require dense and deterministic sparse controls. Intervene on placement and temporal order. Test domain bursts and slower hardware. Version the routed system rather than the weights alone.

When those tests pass, sparsity has survived contact with the system and the added capacity has earned its route. When they fail, the right response may be better placement, a different dispatcher, fewer or larger experts, lower top-k, a fixed route, a dense fallback or no MoE at all. The point is not to defend an architecture. It is to identify the boundary at which conditional computation becomes an operational advantage.