Part I

flowchart TD
  A["One request, two incompatible<br/>stories"] --> B["On this page"]
  B --> C["A request is a timeline, not a<br/>benchmark score"]
  C --> D["Prefill and decode change the<br/>machine"]
  D -. "re-test" .-> A
  classDef first fill:#0a1424,stroke:#D95545,stroke-width:2px,color:#e6f3ff
  classDef step fill:#0a1424,stroke:#3f7da0,stroke-width:1px,color:#e6f3ff
  class A first
  class B,C,D step

A request is a timeline, not a benchmark score

Begin at the user boundary. A client sends a request, waits for the first visible token, receives a stream, and eventually sees completion. Inside that interval, work crosses systems with different owners and clocks. The client connection may traverse a content-delivery edge, an API gateway, authentication, policy checks and a model router. The serving layer admits the request, schedules it, processes the prompt, generates a first token, then repeats a decode step until stopping. A response proxy may buffer or coalesce chunks before the client receives them.

A useful first decomposition is:

Here, T_TTFT,client is the time from the client issuing the request to receiving the first content-bearing token. The other terms are inbound transit, gateway and request preparation, wait before execution, prompt processing, the first token-generation step, and return transit. The equation is an accounting identity only when the segments do not overlap. In an asynchronous system, the trace must instead find the longest dependency chain and record overlapped work separately.

After the first token, the request enters another measure:

O is the number of output tokens, Δ_i is the observed gap before token i, and T_tail covers final buffering, detokenisation or post-processing after the last generated token. Average inter-token latency compresses the sum into one value, but the distribution of gaps still matters. A stream with a 25 millisecond average and occasional one-second stalls feels different from a steady 40 millisecond rhythm.

Official and implementation tooling increasingly exposes TTFT, inter-token latency, end-to-end request latency, queue time, prefill time and decode time as separate measures. MLPerf collects TTFT and time per output token for LLM benchmarks; NVIDIA's current benchmarking tools separate first-token, inter-token and request latency; vLLM exposes request-level histograms for queue, prefill, decode and end-to-end time.[11][12][13] The important point is not the particular metric name. It is the preservation of boundaries.

Thought experiment: equal tokens, unequal waiting

Take two requests with 4,128 total tokens. Request A has 4,096 input tokens and asks for 32 output tokens. Request B has 32 input tokens and asks for 4,096 output tokens. If “tokens processed” were a sufficient latency predictor, the requests would be similar. They are not. Request A concentrates work before the first token and can process many prompt positions in parallel. Request B asks the system to traverse thousands of sequential decode iterations, each dependent on the token before it.

This thought experiment gives the first practical rule: never benchmark a latency-sensitive service with a single average sequence length. Preserve at least the joint distribution of input and output lengths, because a long prompt and a long response stress different parts of the system. For multimodal inputs, record image, audio or video work separately rather than converting everything into an opaque token total.

Percentiles describe populations, not detachable parts

A p95 TTFT of two seconds means that 95 per cent of observations at the named boundary were no greater than two seconds during the measurement interval. It does not identify one request called “the p95 request”, and it does not permit phase percentiles to be added. The request at the p95 of queue time may not be the request at the p95 of prefill or network time. Adding p95 queue, p95 prefill and p95 return transit can therefore create a total that no request experienced.

Preserve the per-request timeline, calculate the total for each request, and only then take the total's percentile. To diagnose that tail, inspect the requests in the exceedance set, such as all requests above the p95 TTFT threshold. Compare their queue age, token shape, cache state, route and client boundary with the rest of the population. This keeps correlations attached to real requests and reveals whether one phase repeatedly appears on the slow path.

Mixtures require equal care. Suppose 90 per cent of traffic is short interactive chat and 10 per cent is long document analysis. An aggregate p95 may describe the document class almost exclusively. A major chat improvement could leave that number unchanged, while a degradation in chat p90 remains hidden. Report the overall service objective, then condition by decision-relevant classes such as priority, prompt band, cached-prefix status, region and workflow type. Do not create so many slices that every cell becomes statistically empty.

A latency budget is an architectural allocation, not percentile arithmetic. Teams may assign targets to gateway, queue, model and network spans, but compliance must still be tested on joined request traces at the client boundary. When phase distributions are correlated, a local budget can pass while the user target fails. The trace is the unit that preserves those interactions.

Part II

Prefill and decode change the machine

A decoder-only language model appears to run the same stack of layers for every token. Operationally, however, the prompt and the response create two different workloads. During prefill, the system reads the prompt, computes the hidden states for all prompt positions and writes the key-value entries that later attention steps will reuse. During decode, it proposes one new token per active sequence, appends that token's key and value, then repeats. The second step cannot begin until the first step has chosen a token.

This dependence changes how much parallel work is available. Prefill exposes many prompt positions at once and can usually make effective use of matrix multiplication throughput. Decode exposes one new position for each active sequence at an iteration. With a small batch, the device may spend more time moving model weights and cached keys and values than performing arithmetic. Batching more sequences creates reuse, but it also couples users through the scheduler and consumes more cache. The familiar statement that prefill is compute-bound and decode is memory-bound is therefore a useful starting tendency, not a law. Model architecture, context length, batch shape, quantisation, cache layout, interconnect and kernels can move either phase.[2][6]

Memory movement and the KV cache

The key-value cache avoids recomputing attention keys and values for every previous token. It exchanges arithmetic for persistent state. For a simple dense representation, its approximate size across active requests is:

Here, 2 represents keys and values, b is bytes per stored element, L is the number of transformer layers, H_KV is the number of key-value heads, d is the head dimension, and S_r is the cached sequence length for active request r. The expression omits allocator metadata, alignment, replicas, temporary workspaces, cache compression and architecture-specific state. It is a capacity lens, not a universal memory invoice.

Three consequences follow. First, long prompts occupy cache before they emit useful output. Second, long responses grow their cache while remaining resident for many iterations. Third, fragmented allocation can prevent admission even when the sum of free bytes looks sufficient. PagedAttention attacks the third problem by allocating cache in blocks rather than reserving one contiguous worst-case region per sequence. Its authors report markedly higher throughput at comparable latency in their tested workloads, but the durable mechanism is more important than the reported multiplier: cache allocation policy changes which requests can coexist, and coexistence changes queueing and decode efficiency.[5]

Worked example: three requests on an idle server

Consider a synthetic, warm-server experiment with batch size one. All three requests use the same model, engine and endpoint. The values below are illustrative, not measurements from a named system. They vary prompt and response shape while keeping other conditions fixed.

The long prompt makes the first visible token late, but leaves streaming broadly similar. The long response preserves fast TTFT and dominates completion time through sequential decode. A prompt compression change should be judged against the second row. An output policy, decode kernel, cache or batching change should be judged against the third. One end-to-end average would erase the experimental contrast that tells the engineer what to change.

Part III

When the queue becomes the model

Service time is what a request consumes after it receives the relevant resource. Response time includes what it waits before and between service intervals. The distinction is small under light load and decisive near saturation. A server that needs 300 milliseconds of accelerator work can still produce a five-second TTFT if work arrives faster than the scheduler can retire it, if one long prefill blocks short requests, or if cache pressure prevents admission.

Little's law gives a first consistency check for a stable system:

L is the average number of items in the system, λ is the average completed arrival rate, and W is average time in the system. If a service completes 20 requests per second and requests spend two seconds inside the measured boundary, about 40 requests should be present on average. A dashboard showing four suggests mismatched boundaries, dropped observations or a non-steady interval. Little's law does not identify the cause, but it detects stories that cannot all be true.[1]

An M/M/1 queue offers a second, deliberately simplified intuition. If arrivals are Poisson, service times are exponential, and one server works at rate μ while requests arrive at rate λ, utilisation is ρ = λ/μ and mean queue wait is ρ/(μ − λ). Real language-model servers violate these assumptions through batching, token-level scheduling, variable lengths, pre-emption and cache constraints. The formula is not a sizing model. Its useful warning is the denominator: as offered load approaches service capacity, waiting grows nonlinearly.

Thought experiment: preserve the mean, change the arrivals

Server A receives one request every 100 milliseconds. Server B receives ten requests together, then none for one second. Both average ten requests per second. Give every request identical prompt and output lengths. If the service can complete one request every 80 milliseconds, Server A may remain almost queue-free. Server B starts each burst with nine requests waiting. The mean arrival rate is unchanged; the arrival process is not.

Now introduce one 32,000-token prompt at the front of each burst. A request-level batch scheduler may hold later requests behind it. A token-level scheduler can interleave work, but the long prefill may still disturb ongoing decode. Split the prefill into chunks and the disturbance shrinks, although the long request now takes more scheduling turns. Isolate prefill on another pool and interference falls, but model replicas and key-value state must be transferred. Scheduling does not remove work. It chooses which request experiences the work, which resource holds it, and which new overhead pays for isolation.

Scheduling choices and their debts

Early serving designs formed request batches and waited for the whole batch to finish. Orca showed why autoregressive generation benefits from iteration-level scheduling and selective batching: completed sequences can leave and new work can enter between iterations.[3] Modern continuous batching generalises this operating idea, but it creates a shared, changing batch whose composition determines token time.

Chunked prefill divides a long prompt into bounded pieces that can be interleaved with decode iterations. Sarathi-Serve uses this mechanism to reduce prefill interference and form more efficient hybrid batches in its evaluated workloads.[8] Phase-disaggregated systems such as Splitwise and DistServe place prompt and token-generation work on separate resources, allowing different parallelism and capacity choices for TTFT and token latency.[6][7] They must then pay for duplicated model state, routing and key-value transfer. Dynamic systems investigate moving between co-located and disaggregated execution as the workload changes.[9]

No topology wins independently of workload. A 2026 study of one production-style disaggregated configuration found that prefill execution itself accounted for only 2 to 23 per cent of p95 TTFT under the tested bursty workloads; queueing and inter-node cache transfer accounted for the remainder.[10] That bounded result is a useful counterexample to a common inference: isolating prefill does not guarantee that prefill compute remains the dominant first-token cost.

A better operating objective is goodput under explicit latency constraints: accepted work that meets both first-token and inter-token targets, rather than raw tokens per second produced while users miss their service levels. Report throughput beside the latency distribution and rejected or deferred load. Otherwise an optimisation can look efficient by allowing a growing queue to absorb its debt.

Part IV

stateDiagram-v2
  [*] --> Claim
  state "Thought experiment: equal<br/>tokens, unequal waiting" as Claim
  state "Percentiles describe<br/>populations, not detachable<br/>parts" as Test
  state "Memory movement and the KV cache" as Revision
  Claim --> Test: expose mechanism
  Test --> Decision: evidence survives
  Test --> Revision: rival survives
  Revision --> Claim: narrow or rebuild
  Decision --> [*]

The compound request

A model server sees a request after some work has already happened and declares it complete before the user necessarily sees the last byte. The client may first resolve DNS, establish or reuse a connection, traverse an edge and gateway, authenticate, retrieve context and serialise a prompt. On the return path, a reverse proxy can buffer chunks, a client library can coalesce events, and a user interface can delay rendering. In an agent, one model call may propose a tool, wait for its effect, then invoke another model. The model call is a component span inside the product's critical path, not a synonym for product latency.

This is why TTFT needs an observation boundary. Server-side TTFT might begin when the engine accepts a request and end when it produces a token. Client-side TTFT begins when the client sends and ends when it receives. “Time to first useful content” may end later, after a preamble or structured field becomes meaningful. All are legitimate measures. They answer different questions.

Thought experiment: move only the observer

Place Client A in the same data centre as the server and Client B on a distant mobile network. Send an identical request through the same gateway. If engine TTFT is equal but client TTFT differs, model optimisation cannot recover the whole gap. Next, log each server token and compare it with each received stream event. If server token times are smooth while client events arrive in clusters, the likely mechanism lies in transport, buffering or client handling. The model and prompt were held constant; only the observation path changed.

Distributed timestamps introduce their own trap. A monotonic clock is reliable for intervals inside one process, but its origin is local. Current vLLM metric guidance explicitly warns that monotonic timestamps from different processes cannot be directly compared.[12] Use a trace context to join spans, measure durations locally, and record clock uncertainty when absolute wall times must cross hosts. OpenTelemetry's generative-AI semantic conventions provide a useful starting vocabulary, but phase-specific fields still need a versioned local contract as the conventions evolve.[14] An apparent negative network duration is evidence of a broken measurement, not a fast network.

Worked scenario: a governed adviser response

The first visible token arrives after 70 milliseconds of inbound transit, 45 milliseconds of gateway and policy work, 430 milliseconds on the retrieval critical path, 25 milliseconds of prompt assembly, 610 milliseconds in the model queue, 285 milliseconds of prefill, and 85 milliseconds for first decode and return. Client TTFT is 1.55 seconds. The first model then emits an 18-token tool proposal over 396 milliseconds. The calculation tool takes 760 milliseconds. A second model call needs 640 milliseconds to queue, prefill and return its first token, followed by a 110-token answer over 2.64 seconds. The complete interaction is about 5.99 seconds.

A 25 per cent reduction in the first prefill saves about 71 milliseconds. Removing half the first queue saves 305 milliseconds. Cutting tool time by a third saves about 253 milliseconds. Reducing the final response by 30 tokens at 24 milliseconds per token saves roughly 720 milliseconds, provided the shorter answer remains sufficient. The point is not that output should always be shortened. It is that the critical path supplies a common currency for comparing changes that occur in different components.

Parallel work should be represented as a dependency graph rather than added blindly. If policy retrieval takes 430 milliseconds while account retrieval takes 280 milliseconds and they run concurrently, their contribution is 430 milliseconds plus coordination overhead, not 710. If the second cannot begin until the first identifies an account, both remain on the path. This is the same discipline used in distributed tracing and project networks: optimise a branch only when it is critical or likely to become critical after another branch improves.

A latency diagnosis record

Before changing hardware or code, write one page that freezes the claim and the discriminating test. Name the user-visible objective, such as p95 client TTFT below 1.2 seconds and p95 inter-token gap below 80 milliseconds at the expected burst envelope. Then capture the five coordinates, configuration manifest, trace boundaries, workload distribution and suspected phase. The table below turns common symptoms into experiments.

The record should also define a stop rule. A phase is worth optimising when it contributes materially to the target percentile, the proposed change moves that phase under representative load, and quality, cost or fairness do not regress beyond an agreed boundary. This avoids a common sequence: optimise the component with the best profiler, discover that user latency barely moves, then reinterpret the local improvement as success.

Executable latency profiler

The accompanying Python artefact analyses timestamped request traces rather than calling a model. Each record contains client send, gateway arrival, queue entry, prefill start and end, first server token, first client token and last client token. The profiler validates timestamp order, computes TTFT, inter-token and end-to-end distributions, averages named phases, and recommends the next phase-specific test. Its assumptions are visible: one aligned timeline per record, positive token counts and no causal attribution for unobserved work.

The built-in positive case holds gateway, prefill, first decode and per-token service constant while increasing queue delay across a synthetic burst. It should identify queueing as the largest mean TTFT phase and streaming as the largest end-to-end phase. The negative case makes prefill end before prefill start. It must fail rather than turn a clock defect into a performance claim. Rejecting an invalid trace is part of the profiler's result.

#!/usr/bin/env python3
"""Phase-aware profiler for synthetic or exported LLM inference traces.

The profiler does not call a model. It analyses timestamped request traces and
keeps unobserved time as a residual instead of assigning it to a guessed phase.
Run with no arguments for the built-in positive and negative cases, or pass a
JSONL file containing the fields documented in TraceRecord.
"""

from __future__ import annotations

import argparse
import json
import math
from dataclasses import asdict, dataclass
from pathlib import Path
from statistics import mean
from typing import Iterable, Sequence


@dataclass(frozen=True)
class TraceRecord:
    request_id: str
    prompt_tokens: int
    output_tokens: int
    client_send_ms: float
    gateway_arrival_ms: float
    queue_enter_ms: float
    prefill_start_ms: float
    prefill_end_ms: float
    first_token_server_ms: float
    first_token_client_ms: float
    last_token_client_ms: float

    @classmethod
    def from_dict(cls, raw: dict) -> "TraceRecord":
        missing = [name for name in cls.__dataclass_fields__ if name not in raw]
        if missing:
            raise ValueError(f"missing fields: {', '.join(missing)}")
        record = cls(**{name: raw[name] for name in cls.__dataclass_fields__})
        record.validate()
        return record

    def validate(self) -> None:
        if self.prompt_tokens < 1 or self.output_tokens < 1:
            raise ValueError(f"{self.request_id}: token counts must be positive")
        points = [
            self.client_send_ms,
            self.gateway_arrival_ms,
            self.queue_enter_ms,
            self.prefill_start_ms,
            self.prefill_end_ms,
            self.first_token_server_ms,
            self.first_token_client_ms,
            self.last_token_client_ms,
        ]
        if any(b < a for a, b in zip(points, points[1:])):
            raise ValueError(
                f"{self.request_id}: timestamps are not monotonic; "
                "cross-process clocks must not be compared without alignment"
            )

    def phases(self) -> dict[str, float]:
        return {
            "network_in": self.gateway_arrival_ms - self.client_send_ms,
            "gateway": self.queue_enter_ms - self.gateway_arrival_ms,
            "queue": self.prefill_start_ms - self.queue_enter_ms,
            "prefill": self.prefill_end_ms - self.prefill_start_ms,
            "first_decode": self.first_token_server_ms - self.prefill_end_ms,
            "network_to_first": self.first_token_client_ms
            - self.first_token_server_ms,
            "stream": self.last_token_client_ms - self.first_token_client_ms,
        }

    def metrics(self) -> dict[str, float]:
        stream = self.last_token_client_ms - self.first_token_client_ms
        itl = stream / (self.output_tokens - 1) if self.output_tokens > 1 else math.nan
        return {
            "ttft": self.first_token_client_ms - self.client_send_ms,
            "itl": itl,
            "e2e": self.last_token_client_ms - self.client_send_ms,
        }


def percentile(values: Sequence[float], q: float) -> float:
    if not values:
        return math.nan
    ordered = sorted(values)
    if len(ordered) == 1:
        return ordered[0]
    rank = (len(ordered) - 1) * q
    lo = math.floor(rank)
    hi = math.ceil(rank)
    if lo == hi:
        return ordered[lo]
    return ordered[lo] + (ordered[hi] - ordered[lo]) * (rank - lo)


def analyse(
    records: Sequence[TraceRecord], label: str, workload: str = "trace replay"
) -> dict:
    if not records:
        raise ValueError("at least one trace is required")
    metric_rows = [record.metrics() for record in records]
    phase_rows = [record.phases() for record in records]
    phase_means = {
        phase: mean(row[phase] for row in phase_rows) for phase in phase_rows[0]
    }
    ttft_phases = ("network_in", "gateway", "queue", "prefill", "first_decode", "network_to_first")
    dominant_ttft_phase = max(ttft_phases, key=lambda phase: phase_means[phase])
    dominant_e2e_phase = max(phase_means, key=phase_means.get)
    p95_ttft = percentile([row["ttft"] for row in metric_rows], 0.95)
    p95_e2e = percentile([row["e2e"] for row in metric_rows], 0.95)
    valid_itl = [row["itl"] for row in metric_rows if math.isfinite(row["itl"])]
    p95_itl = percentile(valid_itl, 0.95)
    prompt_p50 = percentile([record.prompt_tokens for record in records], 0.50)
    prompt_p95 = percentile([record.prompt_tokens for record in records], 0.95)
    output_p50 = percentile([record.output_tokens for record in records], 0.50)
    output_p95 = percentile([record.output_tokens for record in records], 0.95)

    remedies = {
        "network_in": "move the endpoint closer, reuse connections, and inspect payload size",
        "gateway": "profile authentication, rate limiting, tokenisation, and request shaping",
        "queue": "shape admission, add phase-appropriate capacity, or change scheduling",
        "prefill": "reduce uncached context, improve kernels, or partition prompt processing",
        "first_decode": "inspect the first decode step, sampler, and host-device synchronisation",
        "network_to_first": "flush streaming chunks and inspect proxy buffering",
        "stream": "improve decode bandwidth, batching, cache locality, or output length policy",
    }

    return {
        "label": label,
        "workload": workload,
        "requests": len(records),
        "prompt_p50": prompt_p50,
        "prompt_p95": prompt_p95,
        "output_p50": output_p50,
        "output_p95": output_p95,
        "p50_ttft_ms": percentile([row["ttft"] for row in metric_rows], 0.50),
        "p95_ttft_ms": p95_ttft,
        "p95_itl_ms": p95_itl,
        "p95_e2e_ms": p95_e2e,
        "dominant_ttft_phase": dominant_ttft_phase,
        "dominant_ttft_phase_mean_ms": phase_means[dominant_ttft_phase],
        "dominant_e2e_phase": dominant_e2e_phase,
        "dominant_e2e_phase_mean_ms": phase_means[dominant_e2e_phase],
        "recommended_ttft_test": remedies[dominant_ttft_phase],
        "recommended_e2e_test": remedies[dominant_e2e_phase],
        "phase_means_ms": phase_means,
    }


def load_jsonl(path: Path) -> list[TraceRecord]:
    records: list[TraceRecord] = []
    with path.open("r", encoding="utf-8") as handle:
        for line_number, line in enumerate(handle, 1):
            if not line.strip():
                continue
            try:
                records.append(TraceRecord.from_dict(json.loads(line)))
            except (json.JSONDecodeError, TypeError, ValueError) as exc:
                raise ValueError(f"{path}:{line_number}: {exc}") from exc
    return records


def demo_records() -> list[TraceRecord]:
    """Positive case: identical service time, rising queue delay during a burst."""
    records: list[TraceRecord] = []
    for index, queue_ms in enumerate([4, 8, 12, 18, 45, 90, 170, 260, 390, 520], 1):
        start = index * 1_000.0
        records.append(
            TraceRecord(
                request_id=f"burst-{index:02d}",
                prompt_tokens=2048,
                output_tokens=65,
                client_send_ms=start,
                gateway_arrival_ms=start + 18,
                queue_enter_ms=start + 24,
                prefill_start_ms=start + 24 + queue_ms,
                prefill_end_ms=start + 24 + queue_ms + 145,
                first_token_server_ms=start + 24 + queue_ms + 156,
                first_token_client_ms=start + 24 + queue_ms + 174,
                last_token_client_ms=start + 24 + queue_ms + 174 + 64 * 22,
            )
        )
    return records


def invalid_demo() -> TraceRecord:
    """Negative case: a server timestamp appears before an earlier client event."""
    raw = asdict(demo_records()[0])
    raw["prefill_end_ms"] = raw["prefill_start_ms"] - 5
    return TraceRecord.from_dict(raw)


def print_report(report: dict) -> None:
    print(f"Scenario: {report['label']} ({report['requests']} requests)")
    if (
        report["prompt_p50"] == report["prompt_p95"]
        and report["output_p50"] == report["output_p95"]
    ):
        shape = (
            f"{report['prompt_p50']:.0f} input / "
            f"{report['output_p50']:.0f} output tokens"
        )
    else:
        shape = (
            f"input p50/p95 {report['prompt_p50']:.0f}/{report['prompt_p95']:.0f} | "
            f"output p50/p95 {report['output_p50']:.0f}/{report['output_p95']:.0f} tokens"
        )
    print(
        "Latency coordinates: client boundary | TTFT/ITL/E2E | "
        f"p50+p95 | {report['workload']} | {shape}"
    )
    print(f"p50 TTFT: {report['p50_ttft_ms']:.1f} ms")
    print(f"p95 TTFT: {report['p95_ttft_ms']:.1f} ms")
    print(f"p95 ITL:  {report['p95_itl_ms']:.1f} ms/token")
    print(f"p95 E2E:  {report['p95_e2e_ms']:.1f} ms")
    print(
        f"Dominant TTFT phase: {report['dominant_ttft_phase']} "
        f"({report['dominant_ttft_phase_mean_ms']:.1f} ms mean)"
    )
    print(f"TTFT first test: {report['recommended_ttft_test']}")
    print(
        f"Dominant E2E phase: {report['dominant_e2e_phase']} "
        f"({report['dominant_e2e_phase_mean_ms']:.1f} ms mean)"
    )
    print(f"E2E first test: {report['recommended_e2e_test']}")


def main() -> int:
    parser = argparse.ArgumentParser(description=__doc__)
    parser.add_argument("trace", nargs="?", type=Path, help="optional JSONL trace file")
    args = parser.parse_args()

    if args.trace:
        print_report(analyse(load_jsonl(args.trace), args.trace.name, "trace replay"))
        return 0

    print_report(analyse(demo_records(), "bursty positive case", "replayed burst"))
    print("\nNegative control:")
    try:
        invalid_demo()
    except ValueError as exc:
        print(f"rejected as expected: {exc}")
        return 0
    raise AssertionError("negative control should have failed")


if __name__ == "__main__":
    raise SystemExit(main())

Expected built-in output

Compact glossary

TTFT

Time from a stated request boundary to the first observed token or stream event.

Inter-token latency

The gaps between successive observed output tokens. Some tools report time per output token using a related aggregate.

Prefill

Prompt processing that constructs hidden state and the initial key-value cache before generation.

Decode

Autoregressive generation after prefill, normally one new token per active sequence and iteration.

KV cache

Stored attention keys and values from prior positions, reused so decode does not recompute the full prefix.

Offered load

The arrival process presented to the service, including rate, burst shape, priorities and request-length distribution.

Goodput

Useful completed work that satisfies stated quality and latency conditions, rather than all raw work performed.

Critical path

The chain of dependent spans that determines when the observed outcome can complete.

Optimise the phase you can name

Inference latency becomes tractable when the single number is opened into a timeline. The client waits across admission, queueing, prompt work, first decode, token iterations, memory transfers, network delivery and, increasingly, tools and repeated model calls. These phases respond to different levers. More compute cannot flush a proxy. A faster attention kernel cannot drain an admission queue. Phase disaggregation cannot erase the cost of moving state. A shorter prompt cannot repair a long tool dependency.

The changed architecture decision is simple but demanding: do not approve a latency optimisation without a phase-aware trace and a representative offered-load replay. Require every target to name its observation boundary, metric, percentile, arrival process and request or workflow shape. Preserve input and output distributions. Use the idle microbenchmark as a negative control, not as the service claim. Compare candidate changes by their expected reduction on the user-visible critical path, then verify the effect at the target percentile.

The changed research programme is equally concrete. Benchmark schedulers across prompt-output mixtures and burst processes, report cache occupancy and rejected load, keep server and client token times distinct, and include the overhead of isolation or transfer. For compound agents, evaluate workflow completion and slack as well as individual calls. The first defensible latency claim is not “the model is fast”. It is “under this workload and boundary, this phase dominates, this intervention should move it, and this trace can prove whether it did”.