Part I

What the staircase changes

Imagine replacing a ruler with infinitely many marks by one with only a small number of notches. Every value must move to a nearby notch, unless it lies beyond the ruler and is forced to an endpoint. Quantisation is this operation applied to tensors, with engineering choices about where the ruler starts, how wide it is and which values share it.

A common affine quantiser maps a real value x to an integer code q, then reconstructs an approximation x̂:

s is the scale, z the zero point, and the integer limits follow from the chosen bit width. Symmetric schemes often use z = 0; asymmetric schemes spend the code range around the observed minimum and maximum.

Two errors appear. Rounding error moves an in-range value to the nearest level. For a uniform grid, its absolute magnitude is at most half a step, except at tie and saturation conventions. Clipping error occurs when a value exceeds the chosen range. It can be much larger because all values beyond the endpoint collapse to the same code. Reducing bits while holding the range fixed widens the step. Narrowing the range preserves smaller steps for ordinary values but clips more extremes. Calibration is therefore a choice about which error to accept.

Jacob and colleagues formalised an integer-only inference path in which scales and zero points connect quantised operators while accumulations retain wider precision.[1] Modern language-model methods use the same basic ingredients, then change the format, granularity or transformation around them. A “four-bit model” might use uniform signed integers, a non-uniform codebook such as NormalFloat, a floating format with exponent bits, or a block-scaled format in which many narrow values share a scale. These representations have the same storage width and different spacing, dynamic range and hardware behaviour.[7][15][16]

Equal numerical error can have unequal consequence

Consider a linear transformation y = Wx. Quantising the weights gives Ŵ = W + ΔW, so the output change is Δy = ΔWx. Two candidate quantisers can have the same weight mean-squared error, ||ΔW||², yet produce very different output error. One perturbation may fall mostly in directions the input never uses. The other may align with a high-variance input direction or a feature on which the decision boundary has a narrow margin.

This is why measuring only tensor reconstruction can be misleading. GPTQ uses approximate second-order information to choose and compensate weight rounding in a way that reflects input sensitivity.[3] AWQ identifies salient weight channels through activation statistics and protects them with an equivalent scaling transformation.[5] The methods differ, but both reject the idea that every unit of weight error has equal behavioural cost.

Thought experiment 1: rotate the error, keep its size

Take a classifier whose inputs occupy a thin two-dimensional sheet inside a thousand-dimensional space. Add a quantisation perturbation of fixed norm to its weights. In the first trial, orient the perturbation perpendicular to the sheet. Predictions barely move. In the second, rotate the same perturbation into the classifier’s narrow-margin direction. Many labels flip. The bit width, weight error norm and number of altered parameters are held constant. Only alignment changes.

The experiment identifies the missing variable: sensitivity is relational. It belongs to the combination of parameter, input distribution and downstream decision. A calibration set helps estimate that relation, but it can also hide a failure if the deployment distribution contains different languages, sequence lengths, tool outputs or rare token patterns.

A serious negative control

Now construct weights and activations that already lie exactly on a three-bit grid. Give the classifier large margins. Quantise, dequantise and run the same inputs. The maximum numerical error is zero, so every output is identical. The accompanying benchmark includes this case and asserts the result.

This control matters because it prevents a vague conclusion that lower precision is inherently damaging. Quantisation harms behaviour only through approximation that reaches a consequential computation. Some tensors, models and tasks provide ample slack. A useful method must distinguish that benign case from one in which outliers, narrow margins or repeated state reuse turn a small representation into a weak system.

Part II

flowchart LR
  A["One model, three four-bit<br/>conversions"] --> B["On this page"]
  B --> C["What the staircase changes"]
  C --> D["Three tensor roles, three<br/>error shapes"]
  D -. "re-test" .-> A
  classDef first fill:#0a1424,stroke:#087A98,stroke-width:2px,color:#e6f3ff
  classDef step fill:#0a1424,stroke:#3f7da0,stroke-width:1px,color:#e6f3ff
  class A first
  class B,C,D step

Three tensor roles, three error shapes

Weights: a persistent approximation to the learned transformation

Weights are static during ordinary inference. Quantise them once and the same approximation participates in every token, prompt and request. This makes them convenient to calibrate and pack. It also means a poorly quantised channel creates a persistent bias rather than a transient event.

Weight-only schemes usually keep activations and accumulations at higher precision. They can reduce model storage and memory traffic without taking on the full difficulty of low-bit activation arithmetic. GPTQ uses a layer-wise second-order approximation to select rounding and update remaining weights.[3] AWQ uses activation statistics to identify salient weight channels, then rescales an equivalent computation so that uniform low-bit quantisation preserves them better.[5] The broader empirical case for four-bit weight precision is strong but bounded: Dettmers and Zettlemoyer found that, across their tested model families and bit budgets, four bits often gave a favourable size-quality trade, while three-bit behaviour was less predictable.[6]

The phrase “weight-only” still hides choices. One scale for an entire tensor gives low metadata overhead and lets a few extremes set the step for millions of ordinary values. Per-output-channel scales localise that influence. Group-wise scales divide each channel into blocks, often improving reconstruction at the cost of more scale data and more demanding kernels. Non-uniform codebooks place levels where the weight distribution is dense. QLoRA’s NormalFloat is an influential example designed around approximately normal weights, although its original purpose is memory-efficient fine-tuning rather than a universal inference format.[7]

Worked example 1: one matrix, two rulers

Read the benchmark through the quantiser rather than through the final score. A signed four-bit grid has seven positive non-zero levels in this implementation. The largest injected weight is 1.25, so one tensor-wide scale becomes 1.25 ÷ 7, or about 0.179 per step. The median absolute weight is only 0.040. Under that global ruler, 85.7 per cent of weights reconstruct to zero. The quantiser has not randomly damaged the matrix. It has spent most of its codes representing the range needed by two extremes.

Now give each output row a separate ruler. The median row scale falls to 0.024, while only the two rows containing the injected extremes retain coarse scales. The proportion of reconstructed zeros falls to 18.5 per cent. Both candidates still use four bits per weight; the difference is the domain over which one scale is authoritative. Output NRMSE improves because most rows recover locally useful resolution.

This example also exposes a boundary. Per-row scaling does not repair the outlier’s own row. It localises its tax. If that row carries a critical feature, a smaller group, protected channel, rotation or extra precision may still be necessary. Granularity trades global contamination for local metadata and kernel work. The best ruler is therefore neither the finest one possible nor the one with the smallest reconstruction score. It is the coarsest regular grouping that preserves the computations the workload actually uses.

Activations: range loss conditioned on the current input

Activations are produced at runtime. Their ranges can vary by token, batch, language, prompt structure and layer. A calibration set can estimate typical behaviour, but an unseen outlier can still stretch a static scale. Dynamic quantisation can adapt the range for each token or group, though scale calculation, conversion and kernel fusion become part of the latency path.

Large language models made activation outliers impossible to ignore. LLM.int8() observed a small set of emergent outlier dimensions and handled them in higher precision while performing most matrix multiplication in eight bits.[2] SmoothQuant instead applies a mathematically equivalent channel scaling that moves some quantisation difficulty from activations into weights, enabling efficient W8A8 inference in the tested models.[4] Later work found “massive activations” that can be orders of magnitude larger than ordinary values and may play a functional role resembling implicit bias terms.[8] Treating every extreme as disposable noise is therefore unsafe.

The characteristic activation failure is resolution theft. Suppose one token contains the value 120 while most values lie between −3 and 3. A global four-bit symmetric scale must cover 120. The nearest grid around zero becomes so coarse that many ordinary values round to zero. The extreme token may retain its sign and rough size while every other token loses detail. Per-token or per-group scales contain the damage, but they alter metadata, arithmetic and hardware access patterns.

Thought experiment 2: one rogue token

Take a fixed matrix multiplication over 64 tokens. In trial A, every activation lies in the same moderate range. In trial B, change one value in one token to a large magnitude. Keep the model, all other tokens and the bit width fixed. With one scale for the complete activation matrix, the second trial changes the grid for every token. With one scale per token, only the rogue token receives the coarse ruler.

The benchmark implements this intervention. Under its synthetic four-bit stress case, per-tensor activation quantisation gives ordinary tokens an NRMSE of approximately 1.000, because many values collapse to zero. Per-token scaling reduces ordinary-token NRMSE to 0.116. The magnitude is constructed, but the causal lesson is general: activation error can be triggered by an input that changes the range rather than by a change to the model.

KV cache: stored attention state with a growing horizon

The key-value cache is a decode-time computational cache, not enterprise knowledge and not a durable semantic memory. For each transformer layer, it stores keys and values from prior tokens so the model does not recompute the full prefix during every generation step. Its size grows with sequence length, batch size, layer count, key-value heads and head dimension. During later tokens, the attention operation repeatedly reads this stored state.

Keys and values participate differently. A query compares with keys to produce attention logits; small directional changes can alter which token receives attention. Values are then mixed using those probabilities; their error changes the retrieved content. KIVI measured different distribution structures and found that keys in its tested models were better served by per-channel quantisation, while values were better served by per-token quantisation.[11] This is a precise example of the paper’s thesis: two tensors inside the same cache, at the same bit width, prefer different rulers.

Cache error is also horizon-dependent. An old quantised key may be consulted thousands of times. Its approximation can shift a later attention choice; that changed output influences the next generated token and therefore the state added to the cache. Error does not simply add one fixed amount per step, but it can propagate through the continuation. Long-context retrieval, multi-turn consistency and exact copying therefore deserve dedicated tests. KVQuant’s separate treatment of key channels, pre-rotary keys, non-uniform levels and sparse outliers is further evidence that cache compression is a distribution-design problem, not a storage flag.[12] Short-prompt perplexity is a weak proxy for this failure surface.

Part III

Equivalent full-precision models can quantise differently

A model’s input-output function does not uniquely determine the distribution of its intermediate tensors. Equivalent rescalings and rotations can preserve the full-precision result while changing which values become outliers. Quantisability is partly a property of representation, not only of function.

SmoothQuant exploits a scaling invariance. If one channel of an activation is divided by a factor and the corresponding weight channel is multiplied by the same factor, the full-precision matrix product is unchanged. The activation range becomes easier to represent while the weight range becomes harder. A tuning parameter decides where the difficulty should live.[4]

Rotations push the idea further. An orthogonal transformation can spread concentrated energy across dimensions while preserving dot products when paired correctly. QuaRot uses Hadamard rotations to remove activation and KV-cache outliers and reported end-to-end four-bit inference with limited quality loss on its tested Llama 2 models.[9] SpinQuant found that full-precision-equivalent rotations did not all quantise equally: random choices produced up to a thirteen-point range in average downstream zero-shot reasoning accuracy in its experiments, motivating learned rotations.[10]

This result is unusually revealing. The full-precision network can remain numerically identical while the low-bit network changes materially. Therefore neither layer identity nor raw bit width fully explains performance. The placement of information relative to the quantisation grid matters. It also supplies a negative control for causal reasoning: when a full-precision-equivalent transformation changes only quantised performance, the changed distribution is responsible, not a new learned capability.

Formats decide whether range or local resolution wins

Integer formats place uniformly spaced levels after scaling. They are simple and often map well to integer matrix-multiply hardware. Floating formats spend bits on an exponent, giving wider dynamic range and non-uniform spacing. FP8 E4M3 and E5M2 choose different exponent-mantissa balances; neither is accurately described as “eight-bit accuracy” without naming the format.[15] Microscaling formats attach a shared scale to a small block and then store narrow elements within that block, combining local adaptation with regular hardware structure.[16]

A non-uniform codebook can place more representable levels near common values. It helps when the assumed distribution fits. It can fail when calibration is wrong or tails matter. A smaller group supplies more local scales, but scale storage and lookup consume bytes and operations. A mathematically attractive format can lose in service if kernels unpack it inefficiently.

Mixed precision is not an admission of failure. It is often the rational result of a sensitivity landscape. A small number of layers, channels, outliers or cache regions may justify extra bits while the majority remain compressed. The design burden is to make the exception regular enough for efficient kernels and explicit enough for release evidence.

The failure boundary

Below four bits, codebook and optimisation choices become increasingly important, and some model families prove much harder than others. End-to-end W4A4 or lower can be viable with transformations, quantisation-aware training or mixed precision, but it should not be inferred from a successful W4A16 result. Similarly, two-bit KV-cache results depend on distribution-aware granularity, residual high-precision windows, outlier handling and kernel details. The burden rises as the grid narrows.

Part IV

When fewer bits become a faster service

Compression and acceleration are related but distinct. Packed weights reduce bytes moved from high-bandwidth memory. That can help token-by-token decoding, which often behaves like a memory-bandwidth problem at modest batch sizes. Quantising activations allows low-precision matrix multiplication and can matter more during large-batch or long-prompt prefill. Compressing the KV cache reduces the bytes read during attention and may permit larger batches. Each gain needs a kernel that performs quantisation, packing, multiplication, accumulation and dequantisation efficiently.

Atom combines low-bit weight-activation quantisation, outlier handling, group quantisation and KV-cache compression with custom kernels, illustrating algorithm-system co-design.[13] QServe similarly couples W4A8KV4 arithmetic to progressive quantisation and serving kernels rather than assuming stored bits automatically become speed.[14] Current TensorRT documentation exposes several low-precision modes, but the available path remains model, operator and hardware dependent.[18]

A broad online characterisation across several quantisation methods, model sizes and A100/H100 systems found task-, workload-, parallelism- and hardware-dependent trade-offs.[17] This is the right caution. A method can reduce model bytes yet slow a small request because unpacking dominates. It can improve throughput at batch 32 and leave time to first token unchanged. It can fit one more replica and thereby improve queueing without making a single kernel faster. These are valuable outcomes, but they should be named correctly.

Worked example 2: a long-context assistant under one memory budget

At sixteen bits, the weights occupy about 26.0 GB. Four-bit weights reduce the theoretical payload to 6.5 GB. For the stated architecture and workload, a sixteen-bit KV cache occupies about 21.5 GB, while a four-bit cache occupies about 5.37 GB. The arithmetic is simple: batch × tokens × layers × two tensors × KV heads × head dimension × bytes per element. The cache figure changes with every one of those workload and architecture terms; the weight figure does not.

Suppose a team begins with W4A16 because the model does not fit beside the serving workspace. That change removes roughly 19.5 GB of fixed weight payload. It may solve the first capacity problem. Once the service admits four simultaneous long-context requests, however, the uncompressed cache can exceed the compressed weights by more than three times. Further weight optimisation then addresses the smaller term. KV compression becomes the capacity lever, even if a benchmark table had ranked weight-only quantisation as the safer generic recipe.

The candidate profiles now require different evidence. W4A16 needs broad task regression and hardware-kernel measurements because its error is persistent across requests. KV4 needs context-length sweeps, exact retrieval, copying, multi-turn consistency and tests that separate keys from values because its error is request state. The combined W4KV4 profile may have the best memory geometry, but its quality cannot be inferred by adding the two isolated deltas. Their errors meet inside attention and generation.

The production decision follows the dominant byte flow and the consequence of its failure. On a short-context, low-batch endpoint, weight traffic may dominate and cache quantisation may add needless risk. On a long-context shared service, cache capacity can set admission control and queueing. A four-bit label does not reveal either case. A workload equation, an isolated quality test and an observed kernel timeline do.

A benchmark that exposes the mechanism

A useful quantisation benchmark begins with isolation. Quantise weights while leaving activations and cache in the reference format. Then restore weights and quantise activations. Then isolate keys and values. This produces a failure signature before interactions blur attribution. Add the combined candidate only after the component tests.

The executable artefact below does this on synthetic data. It is intentionally small enough to inspect. It includes an exact-grid negative control, a weight outlier case, an activation outlier case, a key-retrieval stress test and a value-mixture test. The point is not to predict a real model’s quality. It is to demonstrate that changing the scale axis can reverse which scheme succeeds while bit width remains fixed.

{
  "activations_4bit": {
    "per_tensor_all_tokens_nrmse": 0.5832864970096727,
    "per_tensor_ordinary_tokens_nrmse": 0.9999999999989091,
    "per_token_all_tokens_nrmse": 0.09714852847784164,
    "per_token_ordinary_tokens_nrmse": 0.11578760723134171
  },
  "assumptions": {
    "data": "synthetic stress cases; not model benchmark results",
    "quantiser": "signed uniform symmetric round-to-nearest",
    "seed": 82
  },
  "keys_2bit_retrieval_accuracy": {
    "16": {
      "full_precision": 1.0,
      "key_per_channel": 1.0,
      "key_per_token": 0.15
    },
    "256": {
      "full_precision": 1.0,
      "key_per_channel": 1.0,
      "key_per_token": 0.075
    },
    "64": {
      "full_precision": 1.0,
      "key_per_channel": 1.0,
      "key_per_token": 0.065
    }
  },
  "negative_control": {
    "max_abs_error": 0.0
  },
  "values_4bit": {
    "value_per_channel_output_nrmse": 0.32014949361205586,
    "value_per_token_output_nrmse": 0.10868871118065787
  },
  "weights_4bit": {
    "per_output_channel_output_nrmse": 0.15103615909352594,
    "per_tensor_output_nrmse": 0.7685499824052446
  }
}

#!/usr/bin/env python3
"""Synthetic benchmark for 'Quantisation and the Shape of Lost Precision'.

The benchmark isolates four mechanisms:
1. an exact-grid negative control;
2. weight quantisation with per-tensor versus per-output-channel scales;
3. activation quantisation under a single runtime outlier;
4. KV-cache quantisation where key and value tensors prefer different axes.

It uses only synthetic data and NumPy. Results are deterministic for seed 82.
"""

from __future__ import annotations

import json
from typing import Any

import numpy as np


def quantise_symmetric(
    x: np.ndarray,
    bits: int,
    *,
    axis: int | tuple[int, ...] | None = None,
) -> tuple[np.ndarray, np.ndarray]:
    """Round to a signed, uniform grid and dequantise back to float64.

    ``axis`` names the dimensions reduced when estimating the scale. For a
    matrix shaped [rows, columns], axis=1 therefore gives one scale per row.
    """
    if bits < 2:
        raise ValueError("bits must be at least 2 for this signed quantiser")

    x = np.asarray(x, dtype=np.float64)
    qmax = (2 ** (bits - 1)) - 1
    maximum = np.max(np.abs(x), axis=axis, keepdims=axis is not None)
    scale = np.where(maximum == 0.0, 1.0, maximum / qmax)
    integer = np.clip(np.rint(x / scale), -qmax, qmax)
    return integer * scale, scale


def nrmse(reference: np.ndarray, candidate: np.ndarray) -> float:
    """Root-mean-square error normalised by reference RMS."""
    numerator = np.sqrt(np.mean((reference - candidate) ** 2))
    denominator = np.sqrt(np.mean(reference**2)) + 1e-12
    return float(numerator / denominator)


def exact_grid_negative_control() -> dict[str, float]:
    """Values already on a 3-bit grid must survive exactly."""
    grid = np.arange(-3, 4, dtype=np.float64) / 3.0
    quantised, _ = quantise_symmetric(grid, 3)
    maximum_error = float(np.max(np.abs(grid - quantised)))
    assert maximum_error == 0.0
    return {"max_abs_error": maximum_error}


def weight_experiment(rng: np.random.Generator) -> dict[str, float]:
    """Two rare weights stretch a global scale but not row-wise scales."""
    inputs = rng.normal(0.0, 1.0, size=(512, 128))
    weights = rng.normal(0.0, 0.06, size=(96, 128))
    weights[3, 17] = 1.25
    weights[49, 88] = -0.90
    reference = inputs @ weights.T

    per_tensor, _ = quantise_symmetric(weights, 4)
    per_channel, _ = quantise_symmetric(weights, 4, axis=1)

    result = {
        "per_tensor_output_nrmse": nrmse(reference, inputs @ per_tensor.T),
        "per_output_channel_output_nrmse": nrmse(
            reference, inputs @ per_channel.T
        ),
    }
    assert result["per_output_channel_output_nrmse"] < result[
        "per_tensor_output_nrmse"
    ]
    return result


def activation_experiment(rng: np.random.Generator) -> dict[str, float]:
    """One token-level outlier can erase ordinary tokens under a global scale."""
    weights = rng.normal(0.0, 0.08, size=(96, 128))
    activations = rng.normal(0.0, 1.0, size=(64, 128))
    outlier_token = 31
    activations[outlier_token, 7] = 120.0
    reference = activations @ weights.T

    per_tensor, _ = quantise_symmetric(activations, 4)
    per_token, _ = quantise_symmetric(activations, 4, axis=1)
    tensor_output = per_tensor @ weights.T
    token_output = per_token @ weights.T
    ordinary = np.arange(len(activations)) != outlier_token

    result = {
        "per_tensor_all_tokens_nrmse": nrmse(reference, tensor_output),
        "per_tensor_ordinary_tokens_nrmse": nrmse(
            reference[ordinary], tensor_output[ordinary]
        ),
        "per_token_all_tokens_nrmse": nrmse(reference, token_output),
        "per_token_ordinary_tokens_nrmse": nrmse(
            reference[ordinary], token_output[ordinary]
        ),
    }
    assert result["per_token_ordinary_tokens_nrmse"] < result[
        "per_tensor_ordinary_tokens_nrmse"
    ]
    return result


def key_retrieval_accuracy(
    context_length: int,
    *,
    trials: int = 200,
    bits: int = 2,
    seed: int = 82,
) -> dict[str, float]:
    """Stress key quantisation with a large, query-irrelevant channel.

    A target key shares a low-amplitude direction with the query. One unrelated
    key channel has a much larger range. Per-token quantisation lets that rogue
    channel set each token's scale; per-channel quantisation isolates it.
    """
    rng = np.random.default_rng(seed + context_length)
    dimension = 64
    full_precision_hits = 0
    per_token_hits = 0
    per_channel_hits = 0

    for _ in range(trials):
        target = int(rng.integers(context_length))
        signal = rng.normal(size=dimension)
        signal[0] = 0.0
        signal /= np.linalg.norm(signal)

        keys = rng.normal(0.0, 0.08, size=(context_length, dimension))
        keys[:, 0] = rng.normal(0.0, 10.0, size=context_length)
        keys[target, 1:] += signal[1:]
        query = signal.copy()

        per_token, _ = quantise_symmetric(keys, bits, axis=1)
        per_channel, _ = quantise_symmetric(keys, bits, axis=0)

        full_precision_hits += int(np.argmax(keys @ query) == target)
        per_token_hits += int(np.argmax(per_token @ query) == target)
        per_channel_hits += int(np.argmax(per_channel @ query) == target)

    result = {
        "full_precision": full_precision_hits / trials,
        "key_per_token": per_token_hits / trials,
        "key_per_channel": per_channel_hits / trials,
    }
    assert result["key_per_channel"] > result["key_per_token"]
    return result


def value_mixture_experiment(
    *, trials: int = 200, bits: int = 4, seed: int = 82
) -> dict[str, float]:
    """Stress value quantisation with token-dependent vector magnitudes."""
    rng = np.random.default_rng(seed)
    per_token_errors: list[float] = []
    per_channel_errors: list[float] = []

    for _ in range(trials):
        values = rng.normal(0.0, 1.0, size=(64, 64))
        token_scales = np.exp(rng.normal(0.0, 1.0, size=64))
        values *= token_scales[:, None]

        logits = rng.normal(size=64) - 0.5 * np.log1p(token_scales)
        attention = np.exp(logits - np.max(logits))
        attention /= np.sum(attention)
        reference = attention @ values

        per_token, _ = quantise_symmetric(values, bits, axis=1)
        per_channel, _ = quantise_symmetric(values, bits, axis=0)
        per_token_errors.append(nrmse(reference, attention @ per_token))
        per_channel_errors.append(nrmse(reference, attention @ per_channel))

    result = {
        "value_per_token_output_nrmse": float(np.mean(per_token_errors)),
        "value_per_channel_output_nrmse": float(np.mean(per_channel_errors)),
    }
    assert result["value_per_token_output_nrmse"] < result[
        "value_per_channel_output_nrmse"
    ]
    return result


def run() -> dict[str, Any]:
    rng = np.random.default_rng(82)
    return {
        "assumptions": {
            "seed": 82,
            "quantiser": "signed uniform symmetric round-to-nearest",
            "data": "synthetic stress cases; not model benchmark results",
        },
        "negative_control": exact_grid_negative_control(),
        "weights_4bit": weight_experiment(rng),
        "activations_4bit": activation_experiment(rng),
        "keys_2bit_retrieval_accuracy": {
            str(length): key_retrieval_accuracy(length)
            for length in (16, 64, 256)
        },
        "values_4bit": value_mixture_experiment(),
    }


if __name__ == "__main__":
    print(json.dumps(run(), indent=2, sort_keys=True))

From numerical fidelity to behavioural evidence

Tensor metrics remain useful. Record clipping rate, scale distribution, signal-to-quantisation-noise ratio, cosine similarity and output NRMSE by layer and group. They explain where approximation enters. They cannot establish whether the system remains acceptable. Behavioural evidence must include the target task, difficult slices and operating conditions.

For a language model, use perplexity as one signal, then test exact retrieval, arithmetic, code, multilingual input, structured output, calibration and the domain decision the service actually supports. Quantisation can preserve average utility while changing a small safety or fairness slice, so those properties need their own release tests rather than inheritance from the full-precision model.[19] Fix sampling seeds or use greedy decoding when the purpose is numerical attribution, then repeat under the production decoding policy to measure distributional effects.

The precision decision instrument

The following instrument turns the article’s distinctions into an architecture decision. It is deliberately sequential. Later evidence cannot repair a failure discovered earlier, although a poor numerical metric may be accepted if task behaviour and operating value remain strong.

For a memory-bound, low-batch decode service, W4A16 may be the first candidate because it reduces weight movement while retaining dynamic activations. For compute-heavy prefill with supported kernels, W8A8 or a carefully validated W4A8 profile may yield more value. For long-context batching, KV compression deserves its own workstream because its capacity benefit can exceed another reduction in fixed weight memory. For edge deployment, a fully low-bit path may be necessary, which raises the case for quantisation-aware training and hardware-specific operator design.

The decision should be made at matched quality. Compare each profile at the same task acceptance threshold, including the cost of retries, escalation and reduced batch capacity. “Tokens per second” without quality can reward a damaged model. “Perplexity unchanged” without systems value can reward a compressed file that is slower to serve.

A release manifest for low precision

Record the source model and revision, quantised tensor classes, format, bit width, group size, scale method, clipping policy, calibration corpus and hash, protected layers, transformation method, kernel library, hardware target and evaluation suite. Store the full-precision baseline and a reversible route to it. A profile without this lineage is difficult to reproduce and harder to investigate after an incident.

Monitor the operating envelope rather than attempting to observe “quantisation error” directly in every request. Useful sentinels include context-length distributions, language mix, clipping statistics from sampled traces, changes in exact retrieval, structured-output repair rates and escalation by task slice. A change in model, tokenizer, prompt template or kernel can invalidate earlier calibration even when the nominal bit width stays the same.