The quiet half of the transformer block

Attention moves evidence; the FFN changes the token

Attention and the feed-forward network operate along different axes. For a sequence matrix with one row per token, self-attention mixes information across rows. The position-wise network then applies one learned function to each row separately. Its weights are shared across positions, but its activation pattern depends on the token vector entering that position. This distinction prevents a common conceptual error: attention is not the whole computation, and the feed-forward network is not another attention mechanism.

The parameter ledger explains why this quiet sublayer is often large. Ignoring biases, an ordinary two-matrix FFN owns about 2dh parameters. A standard attention projection set is roughly 4d² when query, key, value and output maps all have width d. If the FFN hidden width is four times the model width, its two matrices own about 8d² parameters. The exact balance varies with grouped queries, gating and architecture, but the arithmetic makes one point unavoidable: the post-attention transform can carry more block parameters than attention itself.

The residual connection also changes the interpretation. The FFN output is normally added to an existing stream rather than replacing the token representation. It can write a strong update, a small correction or approximately zero. Width creates candidate directions for that update; the non-linearity decides which directions are active for the present token. A later layer can read the combined stream, so a feature produced locally can influence future cross-token attention.

This sometimes motivates the metaphor that FFNs are “memories”. The metaphor is useful only at a narrow structural level: inputs can activate learned response patterns. It fails if it suggests an addressable database, an attributable source or a guaranteed factual store. A hidden unit is distributed across weights and examples, and its activation does not by itself identify what caused the learned association. For architecture decisions, “token-local nonlinear transform” is the safer mechanism.

A gate makes the hidden feature conditional

A gated linear unit computes two projections from the same token. One projection proposes candidate values; the other controls how much of each candidate passes. SwiGLU uses the SiLU, also called Swish, function for the gate. In Shazeer's controlled T5 experiments, parameter-matched GEGLU and SwiGLU variants improved the pretraining objective relative to ordinary ReLU or GELU FFNs. Downstream results often favoured the gated variants but were noisy, and the paper did not establish a universal mechanism.[2]

Parameter matching matters when judging the gate. Comparing a three-matrix SwiGLU with the same hidden width as a two-matrix ReLU FFN gives the gated model roughly half again as many dominant FFN parameters and matrix products. The usual two-thirds width adjustment makes the comparison closer, though activation memory, kernel fusion and numerical behaviour can still differ. A claimed gain should therefore name what was matched: parameters, theoretical operations, wall-clock, memory or training budget.

SwiGLU is conditional computation inside one shared FFN. Every token still reads the same matrices, but different hidden channels are amplified or suppressed by the token's state. This feature-level gate is cheap to batch because it remains dense matrix multiplication. It also supplies the bridge to mixture-of-experts: an MoE router applies conditionality at the scale of entire parameter sets.

Conditionality at two scales

From gated features to gated parameter fields

Suppose one shared feed-forward network is asked to transform legal citations, Python identifiers, conversational pronouns and floating-point numerals. Its hidden width can hold many useful regions, but every token reads the same weights. An MoE layer creates several FFNs with separate parameters and trains a router to select a small subset for each token. The selected outputs are weighted and added.

The seminal sparsely gated MoE work made this capacity argument explicit and also named the obstacles: accelerator branching, shrinking per-expert batches, network bandwidth and self-reinforcing load imbalance.[3] GShard later replaced every other Transformer FFN with top-2 experts and scaled multilingual translation on 2,048 TPU v3 cores, demonstrating that conditional capacity could survive a carefully co-designed model, compiler and interconnect.[4] Switch Transformer simplified routing to top-1 and reported favourable speed-quality trade-offs against dense T5 variants in its setting.[5]

Total parameters are not active parameters

For a SwiGLU expert with model width d and gated hidden width h, the dominant expert parameter count is approximately 3dh: two input projections and one output projection. With E experts, the layer owns roughly 3Edh expert parameters. A top-k router activates roughly 3kdh expert parameters per token, plus router and dispatch work. The ratio E/k looks like the capacity multiplier. It is not an efficiency guarantee.

A model may advertise 64 experts and activate two. That does not mean it behaves like a dense model with one thirty-second of the cost. All expert weights must be stored somewhere. During training, optimizer state and gradients enlarge that footprint. During inference, cold or remote experts may require memory movement. The router scores candidates, and distributed expert parallelism commonly performs all-to-all communication to place token vectors beside their selected weights.

The route is layer-local. A router normally sees the current token representation and chooses experts for that layer; it does not assign a permanent occupation to the word. The token bank may route differently in a river sentence, a finance sentence and a code sample because attention and earlier layers have changed its vector. Conversely, a router that keys mostly on token identity may create stable-looking expert labels without learning the task-dependent partition an observer imagines.

Top-1 and top-2 routing also express different causal bargains. Top-1 makes the expert choice sharp and limits expert matrix work, but a mistaken or overloaded selection has no second expert contribution. Top-2 can interpolate between two functions and offer more gradient paths, while roughly doubling selected-expert work relative to top-1. Switch found top-1 favourable in its scale and implementation; ST-MoE later recommended top-2 in its own design study and reported only small fixed-capacity gains over top-1.[5][6] The disagreement is evidence that routing policy is an empirical systems choice, not a universal architectural constant.

Specialisation is a hypothesis with a control

Routing visualisations often show tokens associated with particular domains, vocabulary or syntax. That is evidence of differentiated traffic, not automatically evidence that the selected expert is causally better for the token. The route could correlate with superficial token identity; experts could remain redundant; or the gain could come from stochastic regularisation and extra parameters rather than semantic specialisation.

The THOR study provides a useful warning. In its two-expert multilingual translation experiments, learned Switch-style routing performed similarly to random routing, and the authors found conventional sparse models parameter-inefficient in that setting.[8] The scope matters: it does not refute larger MoEs, and the same paper observed improvements as expert count increased. It does establish the correct negative control. A content router should be compared with a balanced random or static router at matched compute and load.

A stronger causal test goes beyond traffic counts. For sampled tokens, evaluate the selected expert set and several equal-compute alternatives while freezing every other model component. Then ask whether the router's preferred set produces lower loss, better calibrated probabilities or more robust task behaviour. Swap expert outputs between token groups, ablate one expert and test paraphrases that preserve meaning while changing surface form. Specialisation earns a mechanistic claim only when these interventions change behaviour in the predicted direction.

More recent open evidence is stronger but still bounded. OLMoE trained a 6.9-billion-total, 1.3-billion-active model from scratch, reported favourable training speed against active-parameter-matched dense models, and observed domain and vocabulary specialisation across released checkpoints.[9] Yet the same work found that some plausible MoE choices, including a shared expert in its tested configuration, did not help. The durable lesson is not that routing always discovers clean experts; it is that expert behaviour must be measured against causal alternatives.

flowchart TD
  A["The token that had already<br/>looked around"] --> B["The quiet half of the<br/>transformer block"]
  B --> C["Conditionality at two scales"]
  C --> D["The bill arrives in the system"]
  D -. "re-test" .-> A
  classDef first fill:#0a1424,stroke:#5B5CF6,stroke-width:2px,color:#e6f3ff
  classDef step fill:#0a1424,stroke:#3f7da0,stroke-width:1px,color:#e6f3ff
  class A first
  class B,C,D step

The bill arrives in the system

Expert FLOPs are only one line item

A dense FFN is regular. Large matrix multiplications use contiguous tensors and are friendly to accelerator kernels. An MoE preserves dense computation inside each expert but surrounds it with irregular work. The end-to-end time of one sparse layer can be decomposed without pretending the terms are independent:

Training and inference stress different terms. Pretraining can aggregate large token batches, giving each expert enough rows for efficient matrix multiplication and amortising collective communication. Autoregressive decoding produces one new token per active sequence at each step. Unless many sequences are served together, each expert sees a small, fluctuating micro-batch. The system may become weight-memory or network bound even though its per-token expert FLOPs look attractive. Prompt prefill and token-by-token decode should therefore be measured separately.

Latency distributions matter as much as averages. All outputs in a synchronous MoE layer wait for the slowest required expert path. A mean-balanced router can still create bursty hot experts for particular languages, code fragments or repeated prompts. The relevant observables include per-layer p95 and p99 load, queue time and expert completion time, not merely the aggregate fraction assigned to each expert over an entire corpus.

Switch Transformer showed why expert capacity is central. If a batch creates A token-expert assignments across E experts, a fixed implementation often allocates each expert a capacity near cA/E, where c is a capacity factor. A larger factor reduces overflow but increases reserved slots, activation memory and communication. A small factor risks dropping or bypassing tokens when popular experts overflow.[5]

Load-balancing losses try to prevent collapse. The original sparse MoE paper reported a self-reinforcing loop in which favoured experts received more updates, improved faster and attracted still more traffic. In one reported ablation, removing balancing produced a maximum-to-mean load ratio of 17.8 and worse perplexity than configurations with importance or load losses.[3] Switch used a differentiable auxiliary loss combining the observed fraction of tokens sent to each expert with the router's mean probability mass.[5]

Balancing is not free or conceptually neutral. It asks the router to optimise model loss and resource allocation together. Perfectly uniform traffic may block useful asymmetry if some functions genuinely occur more often. Weak balancing may create dead experts and tail latency. ST-MoE found that a router z-loss, which discourages very large router logits, stabilised its tested sparse models without the severe quality loss caused by tight update clipping. It also showed that sparse and dense models preferred different fine-tuning hyperparameters, so a dense-tuned protocol could conceal sparse gains.[6]

A production-shaped batch

Systems can change the bargain

The capacity trade-off is partly an implementation artefact, not a law of MoE. MegaBlocks reformulated expert computation as block-sparse operations so imbalanced expert batches could be processed without token dropping or dense padding. On its tested A100 setup, it reported up to 40 per cent faster end-to-end training than the compared Tutel MoE implementation. It also reported 1.8 to 2.4 times faster training to matched validation loss than dense Megatron-LM baselines.[7] Those figures are hardware-, model- and software-specific. Their general significance is that better kernels can move the sparse frontier.

They do not erase total weight memory or distributed communication. A dropless kernel solves the local shape problem; it does not prove that remote expert traffic is cheap, that low-concurrency decoding will batch well, or that a sparse model transfers better after fine-tuning. OLMoE likewise reports that its 1.3-billion-active model has inference computation comparable to roughly one-billion-parameter dense models but requires memory for 6.9 billion total parameters.[9]

A falsifiable SwiGLU and MoE laboratory

The included Python artefact is deliberately small. It does not train a language model or claim to predict production throughput. It isolates three pieces of logic that prose tends to blur. A feature gate can change an output while holding the candidate value fixed. A useful content router can beat both one shared function and random routing when the data truly contain different regimes. A skewed routing distribution can waste fixed capacity even when nominal expert computation is unchanged.

Experiment 1: hold the value, vary the gate

Two three-dimensional inputs share the same value projection, [1, 0]. Only a context coordinate used by the gate changes sign. In the favouring context, the first SiLU gate is 1.981 and the scalar output is 1.981. In the suppressing context, the same candidate feature is multiplied by -0.255. This is a causal demonstration of feature-level conditionality, not evidence that a trained model uses the gate in this particular semantic way.

Experiment 2: make specialisation necessary, then remove it

The positive case generates two separated regions of a two-dimensional input space. Each region follows a different linear target function. One shared linear expert cannot fit both perfectly; two specialist experts can, provided the router selects the correct region. On 200 held-out points, the dense baseline records mean squared error 0.908, the content router is numerically zero and the balanced random router records 10.139.

A router earns a causal explanation only when conditional functions are necessary. The negative control changes one causal feature: both regions now obey the same target function. Dense, content-routed and random-routed systems all reach numerical zero. The preferred explanation correctly disappears. More experts and smarter routing add no value when the function is shared.

The example also exposes a counterexample to sparse enthusiasm. A sufficiently expressive shared network can represent several regimes without routing, especially when those regimes reuse features. MoE is attractive when conditional separation buys more than shared composition, not whenever data contain categories. Extending the lab with a nonlinear dense baseline is therefore an important next test: increase shared width until its quality matches the experts, then compare full training and serving cost.

Experiment 3: expose the fixed-capacity loss

The capacity simulator compares four experts receiving 256 top-1 assignments. Balanced loads use every slot. A skewed distribution of [205, 17, 17, 17] overflows 141 assignments and leaves 141 slots idle at capacity factor 1.0. A moderately skewed distribution with factor 1.25 reduces overflow to five but reserves 69 unused slots. The simulator does not decide whether an implementation drops, reroutes or processes overflow dynamically. It makes the unresolved assignments visible.

Run the companion file with Python 3.10 or later and NumPy:

Inspect the implementation in this paper · Review the recorded evidence in this paper

python paper61_swiglu_moe_lab.py
python paper61_swiglu_moe_lab.py --json paper61_lab_results.json

Expected final line: Assertions: PASS. The assertions fail if the feature gate stops changing the fixed candidate or if content routing no longer beats the shared model in the positive case. They also fail if random routing unexpectedly wins, routing helps the negative control, or skew creates no overflow.

#!/usr/bin/env python3
"""Paper 61 laboratory: SwiGLU gating and sparse-expert economics.

Requires: Python 3.10+ and NumPy.
Run:
    python paper61_swiglu_moe_lab.py
    python paper61_swiglu_moe_lab.py --json paper61_lab_results.json

The data are synthetic. The tests are designed to distinguish conditional
specialisation from mere parameter multiplication and to expose routing load.
"""
from __future__ import annotations

import argparse
import json
import math
from dataclasses import asdict, dataclass
from pathlib import Path
from typing import Any

import numpy as np

SEED = 61


def silu(x: np.ndarray) -> np.ndarray:
    """SiLU(x) = x * sigmoid(x), evaluated stably enough for this small lab."""
    return x / (1.0 + np.exp(-x))


@dataclass(frozen=True)
class GateProbe:
    context: str
    value_projection: list[float]
    gate_logits: list[float]
    gated_features: list[float]
    output: float


def swiglu_probe() -> list[GateProbe]:
    """Hold the candidate value fixed while changing only the gate context."""
    # The first two coordinates carry the candidate value. The third coordinate
    # affects only the gate, which lets us vary one causal feature at a time.
    inputs = {
        "gate-favouring context": np.array([1.0, 0.0, 1.0]),
        "gate-suppressing context": np.array([1.0, 0.0, -1.0]),
    }
    w_value = np.array([[1.0, 0.0], [0.0, 1.0], [0.0, 0.0]])
    w_gate = np.array([[0.2, -0.2], [0.0, 0.0], [2.0, -2.0]])
    w_out = np.array([1.0, -1.0])

    probes: list[GateProbe] = []
    for label, x in inputs.items():
        value = x @ w_value
        logits = x @ w_gate
        gated = silu(logits) * value
        probes.append(
            GateProbe(
                context=label,
                value_projection=np.round(value, 6).tolist(),
                gate_logits=np.round(logits, 6).tolist(),
                gated_features=np.round(gated, 6).tolist(),
                output=float(np.round(gated @ w_out, 6)),
            )
        )
    return probes


def fit_linear(design: np.ndarray, target: np.ndarray) -> np.ndarray:
    weights, *_ = np.linalg.lstsq(design, target, rcond=None)
    return weights


def mse(target: np.ndarray, prediction: np.ndarray) -> float:
    return float(np.mean((target - prediction) ** 2))


@dataclass(frozen=True)
class RoutingResult:
    case: str
    dense_mse: float
    content_router_mse: float
    random_router_mse: float


def routing_experiments() -> list[RoutingResult]:
    """Compare a shared expert, a useful router, and a balanced random router."""
    rng = np.random.default_rng(SEED)
    n = 800
    regime = rng.integers(0, 2, size=n)
    x = rng.normal(0.0, 0.55, size=(n, 2))
    x[:, 0] += np.where(regime == 0, -1.5, 1.5)
    design = np.c_[x, np.ones(n)]

    order = rng.permutation(n)
    train, test = order[:600], order[600:]
    content_route = (x[test, 0] > 0).astype(int)
    # Balanced random control: exactly half of the held-out examples visit each expert.
    random_route = np.array([0, 1] * (len(test) // 2))
    rng.shuffle(random_route)

    def evaluate(case: str, target: np.ndarray) -> RoutingResult:
        dense = fit_linear(design[train], target[train])
        experts = [
            fit_linear(design[train][regime[train] == i], target[train][regime[train] == i])
            for i in (0, 1)
        ]
        dense_pred = design[test] @ dense
        content_pred = np.array(
            [design[row] @ experts[content_route[j]] for j, row in enumerate(test)]
        )
        random_pred = np.array(
            [design[row] @ experts[random_route[j]] for j, row in enumerate(test)]
        )
        return RoutingResult(
            case=case,
            dense_mse=mse(target[test], dense_pred),
            content_router_mse=mse(target[test], content_pred),
            random_router_mse=mse(target[test], random_pred),
        )

    # Positive case: the two regions truly require different functions.
    w_left = np.array([2.0, -0.5, 0.3])
    w_right = np.array([-0.7, 1.8, -0.2])
    specialised_target = np.where(regime == 0, design @ w_left, design @ w_right)

    # Negative control: both regions obey the same function, so routing is unnecessary.
    w_common = np.array([1.2, -0.4, 0.1])
    common_target = design @ w_common

    return [
        evaluate("different functions by region", specialised_target),
        evaluate("same function in both regions", common_target),
    ]


@dataclass(frozen=True)
class CapacityResult:
    case: str
    tokens: int
    experts: int
    top_k: int
    capacity_factor: float
    capacity_per_expert: int
    loads: list[int]
    max_to_mean_load: float
    coefficient_of_variation: float
    overflow_assignments: int
    padded_slots: int
    useful_slot_utilisation: float


def capacity_report(
    case: str,
    loads: list[int],
    tokens: int,
    top_k: int,
    capacity_factor: float,
) -> CapacityResult:
    experts = len(loads)
    assignments = tokens * top_k
    if sum(loads) != assignments:
        raise ValueError(f"loads sum to {sum(loads)}, expected {assignments}")
    capacity = math.ceil((assignments / experts) * capacity_factor)
    processed = [min(load, capacity) for load in loads]
    overflow = sum(max(0, load - capacity) for load in loads)
    padded = experts * capacity - sum(processed)
    mean_load = assignments / experts
    cv = float(np.std(loads) / mean_load)
    return CapacityResult(
        case=case,
        tokens=tokens,
        experts=experts,
        top_k=top_k,
        capacity_factor=capacity_factor,
        capacity_per_expert=capacity,
        loads=loads,
        max_to_mean_load=float(max(loads) / mean_load),
        coefficient_of_variation=cv,
        overflow_assignments=overflow,
        padded_slots=padded,
        useful_slot_utilisation=float(sum(processed) / (experts * capacity)),
    )


def capacity_experiments() -> list[CapacityResult]:
    return [
        capacity_report(
            "balanced top-1 batch",
            loads=[64, 64, 64, 64],
            tokens=256,
            top_k=1,
            capacity_factor=1.0,
        ),
        capacity_report(
            "skewed top-1 batch",
            loads=[205, 17, 17, 17],
            tokens=256,
            top_k=1,
            capacity_factor=1.0,
        ),
        capacity_report(
            "moderately skewed with buffer",
            loads=[85, 65, 55, 51],
            tokens=256,
            top_k=1,
            capacity_factor=1.25,
        ),
    ]


def run_assertions(
    probes: list[GateProbe],
    routing: list[RoutingResult],
    capacity: list[CapacityResult],
) -> None:
    assert probes[0].value_projection == probes[1].value_projection
    assert abs(probes[0].output - probes[1].output) > 1.0

    positive, negative = routing
    assert positive.content_router_mse < positive.dense_mse * 0.01
    assert positive.random_router_mse > positive.dense_mse
    assert max(
        negative.dense_mse,
        negative.content_router_mse,
        negative.random_router_mse,
    ) < 1e-20

    balanced, skewed, buffered = capacity
    assert balanced.overflow_assignments == 0
    assert skewed.overflow_assignments > 0
    assert skewed.useful_slot_utilisation < balanced.useful_slot_utilisation
    assert buffered.capacity_per_expert > balanced.capacity_per_expert


def payload() -> dict[str, Any]:
    probes = swiglu_probe()
    routing = routing_experiments()
    capacity = capacity_experiments()
    run_assertions(probes, routing, capacity)
    return {
        "assumptions": {
            "data": "synthetic two-dimensional regression",
            "router": "sign of first feature for content routing; seeded uniform random control",
            "capacity": "fixed expert buffers; overflow is counted rather than silently repaired",
            "seed": SEED,
        },
        "swiglu_gate_probe": [asdict(item) for item in probes],
        "routing_experiments": [asdict(item) for item in routing],
        "capacity_experiments": [asdict(item) for item in capacity],
        "assertions": "PASS",
    }


def print_report(results: dict[str, Any]) -> None:
    print("Paper 61: SwiGLU and MoE laboratory")
    print("=" * 41)
    print("\n1. Feature-level gate probe")
    for row in results["swiglu_gate_probe"]:
        print(
            f"  {row['context']:<26} value={row['value_projection']} "
            f"gate={row['gate_logits']} output={row['output']:.6f}"
        )

    print("\n2. Router discrimination and negative control")
    for row in results["routing_experiments"]:
        print(
            f"  {row['case']:<34} dense={row['dense_mse']:.6f} "
            f"content={row['content_router_mse']:.6f} "
            f"random={row['random_router_mse']:.6f}"
        )

    print("\n3. Capacity and load")
    for row in results["capacity_experiments"]:
        print(
            f"  {row['case']:<34} capacity={row['capacity_per_expert']:>3} "
            f"max/mean={row['max_to_mean_load']:.3f} "
            f"overflow={row['overflow_assignments']:>3} "
            f"padding={row['padded_slots']:>3} "
            f"utilisation={100 * row['useful_slot_utilisation']:.1f}%"
        )
    print("\nAssertions: PASS")


def main() -> None:
    parser = argparse.ArgumentParser()
    parser.add_argument("--json", type=Path, help="optional path for machine-readable results")
    args = parser.parse_args()

    results = payload()
    print_report(results)
    if args.json:
        args.json.write_text(json.dumps(results, indent=2), encoding="utf-8")
        print(f"Results written to {args.json}")


if __name__ == "__main__":
    main()

The sparse adoption record

A model team can make MoE look favourable by choosing the wrong denominator. Comparing a sparse model's active parameters with a dense model's total parameters hides memory. Comparing theoretical FLOPs hides communication. Comparing pretraining loss hides fine-tuning. Comparing a learned router with no routing baseline hides whether selection matters. The decision instrument below forces all four ledgers into one record.

Write acceptance thresholds before running the final comparison. Name the target quality metric, maximum total memory, latency percentiles, overflow policy, allowed communication share and minimum router advantage over the random control. Precommitment prevents a team from celebrating whichever denominator happens to favour the sparse run. Threshold values are use-case decisions; the requirement to state them is general.

The system boundary must include the scheduler, expert placement, collective library, serving batcher and fallback path. Changing any of these can move the same weights to another region of the phase diagram. Record the boundary with the result so that a later hardware migration triggers re-evaluation rather than inheriting an obsolete “MoE is cheaper” claim.

The decision rule is deliberately asymmetric. Dense SwiGLU needs to prove ordinary model quality and operating cost. Distributed MoE must additionally prove routing causality, balance, topology fit and recovery because it introduces new failure surfaces. This is not hostility to sparsity. It is a requirement that extra moving parts purchase a measured capability.

When MoE passes, preserve the evidence as a release artefact: model and router checkpoints, expert placement, load distributions, capacity policy, system traces, downstream results and the random-router comparison. Without that record, a later hardware change or traffic shift can invalidate the original bargain while the architecture name remains unchanged.