Part I · The number that loses the failure

Generator evaluation begins with an inconvenient mismatch. Researchers and product teams want a compact comparison, but the object being compared is expansive. A text-to-image system maps a prompt, seed, decoding configuration, safety policy and sometimes a reference image into a distribution of possible outputs. The evaluation then samples only a minute fraction of that space. A score is therefore not a property floating inside the model. It is an estimate produced by a particular prompt distribution, sampler, feature extractor, reference set and judgement rule.

The earliest popular automated scores made real progress. Inception Score rewarded samples that an ImageNet classifier labelled confidently while also rewarding diversity across predicted classes. Fréchet Inception Distance, introduced with the two-time-scale GAN work, compared the mean and covariance of real and generated features and became far more useful because it compared generated samples with a reference distribution.¹² These instruments helped replace hand-picked galleries with reproducible measurements.

FID treats real features as approximately Gaussian with mean μ_r and covariance Σ_r, and generated features as Gaussian with μ_g and Σ_g:

The first term grows when the centres differ. The trace term grows when the spread and correlations differ. Lower is better. The equation gives a principled distributional comparison, but it also performs severe compression. Two complicated clouds become two means, two covariance matrices and then one number. The score cannot tell whether the mismatch came from ugly samples, missing modes, overrepresented modes or a feature extractor that ignored the decisive distinction.

That loss is not merely theoretical. Precision and recall methods were proposed because a one-dimensional divergence could not separate sample quality from coverage.³ Improved manifold-based versions, followed by density and coverage, made the diagnostic split more operational while also revealing new estimator sensitivities.⁴⁵ Finite-sample bias can change FID comparisons; image resizing and compression choices can move scores; and the usual Inception feature space can reward alignment with ImageNet categories rather than the visual property a user cares about.⁶⁷⁸

The practical conclusion is not that FID is useless. It remains a valuable, standardised signal when the reference distribution, preprocessing, sample size and feature extractor fit the task. The conclusion is narrower and more demanding: every metric sees through a representation, and every scalar forgets why it moved. Stein and colleagues compared seventeen metrics across several encoders and found that no existing metric strongly tracked their large human realism study across the evaluated models; their results also showed how rankings could depend on the feature network.⁹ Later work on FID has continued to question the Gaussian assumption, sample efficiency and suitability of Inception features for modern text-to-image output.¹⁰

The museum copier

Imagine two sealed machines asked to generate portraits in the style of a public-domain collection. Both receive equally high realism ratings. Machine A has learned a broad portrait distribution and creates unfamiliar sitters, poses and lighting. Machine B retrieves a training portrait and changes a few pixels. Now vary only the copying distance. Make the alteration smaller and smaller while keeping human beauty ratings constant.

If the evaluation uses only realism, the machines remain tied. If it uses FID on a large set, the copier may look excellent because its samples sit directly on the reference distribution. Yet the machines support different claims. A is evidence of generative coverage. B may be a high-quality reproduction system, a privacy problem or a rights-management problem depending on the source and licence. Memorisation is logically independent of beauty and can even improve distributional similarity.

flowchart LR
  A["The catalogue model that won<br/>the review"] --> B["On this page"]
  B --> C["Part I · The number that loses<br/>the failure"]
  C --> D["Part II · Five questions, five<br/>kinds of evidence"]
  D -. "re-test" .-> A
  classDef first fill:#0a1424,stroke:#007D68,stroke-width:2px,color:#e6f3ff
  classDef step fill:#0a1424,stroke:#3f7da0,stroke-width:1px,color:#e6f3ff
  class A first
  class B,C,D step

Part II · Five questions, five kinds of evidence

The five dimensions are not a universal ontology of everything that matters in generative systems. They are a practical minimum for the claim “this generator produces acceptable outputs for this use”. Each dimension asks a different counterfactual question. If it were changed while the others stayed fixed, would the release decision change? If yes, it deserves its own observation rather than a hidden weight inside a total.

1. Fidelity: does the output belong where the claim says it belongs?

Fidelity is often paraphrased as realism, but the word must be anchored. A photorealistic dragon can have high visual realism and zero fidelity to a prompt requesting a red bicycle. A medical synthetic image may look plausible to a lay reviewer while violating anatomy. A stylised icon may intentionally depart from photographs. Fidelity therefore has at least two reference relations: visual or domain plausibility, and conditional faithfulness to the prompt or control signal.

Distributional measures such as FID, KID and precision-like metrics can estimate how generated features relate to a reference set. KID uses a kernel discrepancy and has an unbiased estimator under its formulation, making it useful alongside FID rather than a cosmetic alternative.¹¹ For text-conditioned systems, image-text compatibility measures add another view. CLIPScore showed that a learned cross-modal representation could provide a useful reference-free compatibility signal, but its own paper also documented domains where that signal was weaker.¹²

Fine-grained tests ask questions a holistic similarity score can blur. TIFA turns a prompt into visual questions and checks whether a VQA system can answer them from the image. GenEval uses object detection and related models to inspect count, colour, position and co-occurrence.¹³¹⁴ These tests are more interpretable because a failure can be named: the second object is absent, the colour is wrong, or the spatial relation reversed. They also inherit the errors of their question generator, detector and VQA model. A metric stack should report those dependencies, not call them ground truth.

Fidelity is always fidelity under a declared representation and condition. A model comparison should therefore pin the encoder, preprocessing, sample count and prompt set. Where the content is specialised, use a domain-relevant representation or human expert review. Where prompt adherence matters, test attributes separately instead of assuming an overall embedding similarity sees them.

2. Diversity: does the generator cover the intended variation?

Diversity has several scales. Across prompts, the model should respond to distinct requests. Within one prompt, it may need to offer meaningfully different candidates rather than camera-angle jitter. Across a reference population, it may need to cover rare modes, identities, contexts and compositions. A single “diversity” statistic can be as lossy as the beauty score it replaces unless the scale is named.

Precision-recall methods for generative models provide a durable intuition. Precision asks how much generated mass lies in plausible regions; recall asks how much of the reference distribution receives generated coverage. Density and coverage refine the local geometry and try to reduce some pathologies of earlier estimators.³⁵ The estimates remain sensitive to representation, neighbourhood size, dimensionality and outliers. Their real value is diagnostic separation: one model can be sharp but narrow, another broad but implausible.

For conditional generation, coverage should also be computed by prompt slice. The full set may look balanced because common prompts dominate. Minimum-slice coverage, macro-averaged coverage or a consequence-weighted distribution can reveal a rare category that the overall estimate hides. The appropriate aggregation depends on the claim. A creative ideation tool may value set diversity because users browse several alternatives; work on set-based text-to-image evaluation shows why relevance, diversity and arrangement should be connected to how people actually explore candidates rather than to distributional similarity alone.¹⁵

The prompt lottery

Two generators each achieve 90 per cent prompt success across one thousand cases. In System Even, every one of ten prompt families scores about 90 per cent. In System Lottery, nine common families score 99 per cent and the tenth scores 9 per cent. Now vary only which family carries consequence. If the tenth family is decorative texture, the average may support release with a warning. If it contains disability-related layouts, safety diagrams or minority-language instructions, the same average becomes indefensible.

The example is deliberately numerical rather than moral. It shows that a population average has no meaning outside the sampling distribution and consequence model that produced it. The release record needs denominators, slice estimates and an explicit response to a failed slice.

3. Memorisation: is the generator producing or retrieving?

Memorisation cannot be inferred from a low FID, a high originality rating or a casual nearest-neighbour gallery. It requires a comparison with the training set, or at least with a defensible proxy, and a notion of similarity appropriate to the source. Pixel distance catches exact copies but misses crops and transformations. Generic perceptual embeddings may merge legitimately similar images or ignore a copied distinctive element. A serious audit uses several views, manually inspects the tail and records the training-data access boundary.

Sample-level work on authenticity treats copying as a dimension separate from fidelity and diversity.¹⁶ Extraction research has shown that diffusion models can emit memorised training images under generate-and-filter attacks, including examples with people and logos.¹⁷ These findings establish possibility under tested models and procedures. They do not mean every near neighbour is unlawful, private or harmful. Licence, consent, public availability, transformation and jurisdiction remain separate questions.

The operational test should therefore speak precisely: “fraction of generated samples within threshold ε of a training item under encoders E₁ and E₂, with manual adjudication of the closest k cases”. If the training set is unavailable, the result should be “not measured”, not a comforting zero. Unknown memorisation risk is not evidence of originality.

4. Usefulness: does the output improve the real task?

Usefulness is where evaluation stops admiring the artefact and asks what it changes. For ideation, useful output may increase relevant variety before the user finds a direction. For synthetic data, usefulness may be measured by downstream performance on a real held-out set. For product imagery, it may be the fraction of drafts accepted after bounded editing time. For a medical simulator, it may be whether experts can perform the intended training task without learning false cues.

Downstream tests can reverse a visual ranking. Azizi and colleagues showed that, in their ImageNet configuration, synthetic diffusion data improved classifier training and reported gains under specified models and datasets.¹⁸ That result supports the possibility of useful synthetic augmentation. It does not turn FID into a universal utility proxy, nor does it guarantee transfer to another domain. The outcome measure should match the use: classification accuracy on real data, task completion, edit time, retrieval success or expert judgement.

Usefulness also includes cost and workflow. A model that produces one excellent image after forty attempts may score highly on curated samples and poorly in service. Record attempts per accepted output, generation latency, human review time, failure recovery and abandonment. The generator is part of a human-machine loop; output quality without interaction cost can overstate value.

5. Harm: what can go wrong for whom, and under which interaction?

Harm is not the complement of beauty. A polished image can contain hateful content, expose a memorised person, reinforce a stereotype, misrepresent a product, bypass a policy or facilitate deception. The evaluation must distinguish the model’s capability from the way a user elicits it and from the wider system that distributes, labels and acts on the output.

HEIM made this multidimensional burden concrete by evaluating text-to-image systems across twelve aspects, including alignment, quality, originality, bias, toxicity, fairness, robustness, multilinguality and efficiency; no single model led every aspect in that study.¹⁹ Sociotechnical safety work similarly separates model capability, human interaction and systemic impact because context determines whether a capability becomes harm.²⁰ The NIST Generative AI Profile frames risk management across the lifecycle and includes content, privacy, information integrity, intellectual-property and human-AI configuration concerns among its cross-sector considerations.²¹

A release suite should include ordinary prompts, ambiguous prompts, adversarial prompts and realistic multi-turn or editing paths. It should measure false refusals as well as unsafe generations, because a safety intervention can remove legitimate utility unevenly. Harm rates need slices, severity and exposure assumptions. A one-in-ten-thousand event cannot be estimated from two hundred prompts. Rare severe events may require targeted red-team discovery, stress testing and conservative controls rather than a prevalence claim.

Part III · An evaluation designed to be wrong

This is the generator-specific layer of the broader operating argument in The Eval Pipeline is the Product. Here the narrower question is which independent evidence prevents attractive outputs from masking a generator’s failure.

A strong evaluation is not a collection of fashionable metrics. It is a testable contract connecting a release claim to observations that could defeat it. The contract should be written before model selection, because a metric chosen after seeing outputs becomes part of the optimisation loop. Teams naturally favour the instrument that flatters the preferred model.

The contract starts with the use. “Generates good images” cannot be falsified. “Produces non-duplicative draft catalogue scenes for eight declared prompt families, with a human approving every external use” can. The narrower claim identifies the prompt population, the reference material, the required downstream action, the role of the reviewer and the boundaries that the generator must not cross.

Build a portfolio of imperfect instruments

Distributional metrics are efficient for broad comparison. Sample-level detectors make failures inspectable. Human review captures judgements no fixed encoder can settle. Downstream tasks measure actual utility. Red-team prompts search for failures that representative sampling may almost never encounter. The portfolio works because the instruments fail differently.

Human evaluation needs the same discipline as model evaluation. Otani and colleagues reviewed thirty-seven text-to-image papers and found many human studies insufficiently specified for reliable repetition; their protocol emphasises design choices that make judgements verifiable.²² A defensible study randomises order, blinds model identity, separates criteria, provides anchor examples, records rater expertise, preserves disagreement and reports denominators. “Which image is better?” invites the rater to substitute a personal criterion. “Which image contains exactly two red cups to the left of a blue bowl?” is more reproducible.

Automated metrics also need controls. Pin library and model versions. Use the same resizing and compression pipeline for every candidate. Report sample count and random seeds. Repeat stochastic generation. Compare more than one suitable feature space when ranking is consequential. Run bootstrap or repeated-sample intervals. If two intervals overlap heavily, call the result uncertain rather than inventing a winner from the third decimal place.

Then test the test. A duplicate-injection mutation should raise the memorisation signal. Removing a rare mode should reduce slice coverage. Shuffling prompt-image pairs should collapse alignment. Adding an adversarial visual token should trigger the relevant safety check. If the evaluation remains green, the harness has demonstrated blindness. An evaluation earns trust by catching controlled defects, not by producing stable dashboards.

Where the portfolio still breaks

The method has hard boundaries. Feature-space metrics can miss attributes their encoders do not represent. Training-data comparisons are incomplete when the corpus is unavailable or deduplicated differently. Human panels can be culturally narrow, fatigued or anchored. Safety taxonomies can lag new abuse. Downstream studies can overfit one workflow. Offline tests do not establish long-term social impact. A confidence interval quantifies sampling variation under assumptions; it does not measure systematic blind spots.

These limits should appear in the decision receipt. A missing cell may require narrower scope, additional evidence or a control outside the model. The objective is not to claim total measurement. It is to make the known observation boundary harder to confuse with the world.

A minimal worked example

The executable harness below creates a synthetic two-dimensional reference distribution with three modes. It then evaluates three generators. Balanced approximates all modes. Glossy Copier concentrates on common modes and places many samples almost exactly on training points. Wild Explorer covers the space broadly but produces many off-manifold samples. Each generated item also receives synthetic usefulness, harm and aesthetic outcomes.

The harness estimates fidelity as the fraction of generated points within a held-out support radius. Diversity is the worst prompt-slice coverage, not the overall average. Originality is the fraction outside a tight training-neighbour threshold. Usefulness and safety are sample-level rates. Every estimate receives a bootstrap interval. The release gate uses the conservative lower bound and adds a rare-slice usefulness threshold.

All values are synthetic teaching fixtures. Glossy Copier wins beauty and raw fidelity because it sits close to the reference, partly by copying. It fails the originality gate, the minimum diversity slice and rare-slice usefulness. Wild Explorer wins diversity and originality but fails fidelity, usefulness and the conservative safety bound. The balanced generator passes without winning every column. That is the point: release fitness is a conjunction of material claims, not a beauty contest or a total-area score.

"""Synthetic generator-evaluation harness for 'Evaluating Generators Beyond Beauty'.

Python 3.10+ with NumPy. All data, thresholds and model names are synthetic.
The harness demonstrates why a release decision needs separate estimates of:
    fidelity, diversity, originality, usefulness and safety.

Run:
    python3 generator_evaluation_harness.py

Expected result:
    balanced passes the broad release contract;
    glossy_copier fails diversity, originality and rare-slice usefulness;
    wild_explorer fails fidelity, usefulness, safety and rare-slice usefulness;
    licensed_template passes a narrow reproduction contract where diversity and
    originality are deliberately marked not material.
"""
from __future__ import annotations

from dataclasses import dataclass
from typing import Callable
import json
import numpy as np

SEED = 7
N = 700
BOOTSTRAPS = 800
CENTRES = np.array([[-2.0, 0.0], [2.0, 0.0], [0.0, 2.5]])
PROBS = np.array([0.45, 0.35, 0.20])
STDS = np.array([0.45, 0.55, 0.40])


@dataclass(frozen=True)
class Interval:
    estimate: float
    low: float
    high: float


@dataclass(frozen=True)
class Report:
    name: str
    metrics: dict[str, Interval]
    rare_slice_usefulness: Interval
    aesthetic: Interval
    failed_gates: tuple[str, ...]
    decision: str


def sample_reference(rng: np.random.Generator, n: int) -> tuple[np.ndarray, np.ndarray]:
    labels = rng.choice(3, size=n, p=PROBS)
    points = CENTRES[labels] + rng.normal(size=(n, 2)) * STDS[labels, None]
    return points, labels


def nearest_distance(a: np.ndarray, b: np.ndarray, chunk: int = 200) -> np.ndarray:
    """Euclidean distance from each row of a to its nearest row in b."""
    result: list[np.ndarray] = []
    for start in range(0, len(a), chunk):
        block = a[start : start + chunk]
        squared = ((block[:, None, :] - b[None, :, :]) ** 2).sum(axis=2)
        result.append(np.sqrt(squared.min(axis=1)))
    return np.concatenate(result)


def interval(values: np.ndarray, seed: int) -> Interval:
    values = np.asarray(values, dtype=float)
    rng = np.random.default_rng(seed)
    means = np.empty(BOOTSTRAPS)
    for i in range(BOOTSTRAPS):
        means[i] = values[rng.integers(0, len(values), len(values))].mean()
    return Interval(
        estimate=float(values.mean()),
        low=float(np.quantile(means, 0.025)),
        high=float(np.quantile(means, 0.975)),
    )


def generate(
    kind: str,
    train: np.ndarray,
    train_labels: np.ndarray,
    n: int,
    seed: int,
) -> tuple[np.ndarray, np.ndarray, np.ndarray, np.ndarray, np.ndarray]:
    rng = np.random.default_rng(seed)

    if kind == "balanced":
        labels = rng.choice(3, n, p=[0.45, 0.35, 0.20])
        shift = np.array([[0.05, 0.02], [-0.04, 0.03], [0.02, -0.04]])
        points = CENTRES[labels] + shift[labels]
        points += rng.normal(size=(n, 2)) * STDS[labels, None] * 1.03
        copied = rng.choice(n, size=int(0.01 * n), replace=False)
        sources = rng.choice(len(train), size=len(copied), replace=False)
        points[copied] = train[sources] + rng.normal(scale=0.01, size=(len(copied), 2))
        usefulness_p = np.array([0.92, 0.88, 0.84])[labels]
        harm_p = np.array([0.010, 0.015, 0.025])[labels]
        aesthetics = np.clip(rng.normal(0.84, 0.06, n), 0.0, 1.0)

    elif kind == "glossy_copier":
        labels = rng.choice(3, n, p=[0.78, 0.18, 0.04])
        points = CENTRES[labels] + rng.normal(size=(n, 2)) * STDS[labels, None] * 0.55
        copied = rng.choice(n, size=int(0.35 * n), replace=False)
        sources = []
        for item in copied:
            pool = np.where(train_labels == labels[item])[0]
            sources.append(rng.choice(pool))
        points[copied] = train[np.asarray(sources)] + rng.normal(
            scale=0.006, size=(len(copied), 2)
        )
        usefulness_p = np.array([0.95, 0.78, 0.48])[labels]
        harm_p = np.array([0.005, 0.025, 0.080])[labels]
        aesthetics = np.clip(rng.normal(0.94, 0.035, n), 0.0, 1.0)

    elif kind == "wild_explorer":
        labels = rng.choice(3, n, p=[0.38, 0.34, 0.28])
        points = CENTRES[labels] + rng.normal(size=(n, 2)) * STDS[labels, None] * 1.75
        off_manifold = rng.choice(n, size=int(0.15 * n), replace=False)
        points[off_manifold] += rng.normal(scale=2.5, size=(len(off_manifold), 2))
        usefulness_p = np.array([0.75, 0.69, 0.63])[labels]
        harm_p = np.array([0.02, 0.04, 0.06])[labels]
        aesthetics = np.clip(rng.normal(0.73, 0.10, n), 0.0, 1.0)

    else:
        raise ValueError(f"unknown generator: {kind}")

    useful = rng.random(n) < usefulness_p
    harmful = rng.random(n) < harm_p
    return points, labels, useful, harmful, aesthetics


def assess(
    name: str,
    train: np.ndarray,
    train_labels: np.ndarray,
    test: np.ndarray,
    test_labels: np.ndarray,
    manifold_radius: float,
    copy_radius: float,
    seed: int,
) -> Report:
    points, labels, useful, harmful, aesthetics = generate(
        name, train, train_labels, N, seed
    )
    distance_to_train = nearest_distance(points, train)
    distance_test_to_generated = nearest_distance(test, points)

    fidelity_samples = distance_to_train <= manifold_radius
    coverage_samples = distance_test_to_generated <= manifold_radius
    coverage_by_slice = [
        interval(coverage_samples[test_labels == label], seed + 10 + label)
        for label in range(3)
    ]
    diversity = min(coverage_by_slice, key=lambda item: item.estimate)

    metrics = {
        "fidelity": interval(fidelity_samples, seed + 20),
        "diversity": diversity,
        "originality": interval(distance_to_train >= copy_radius, seed + 21),
        "usefulness": interval(useful, seed + 22),
        "safety": interval(~harmful, seed + 23),
    }
    rare = interval(useful[labels == 2], seed + 24)
    aesthetic = interval(aesthetics, seed + 25)

    # Benefit and safety claims must clear gates with the conservative bound.
    thresholds = {
        "fidelity": 0.90,
        "diversity": 0.85,
        "originality": 0.93,
        "usefulness": 0.85,
        "safety": 0.96,
    }
    failed = [
        metric for metric, threshold in thresholds.items() if metrics[metric].low < threshold
    ]
    if rare.low < 0.70:
        failed.append("rare_slice_usefulness")

    return Report(
        name=name,
        metrics=metrics,
        rare_slice_usefulness=rare,
        aesthetic=aesthetic,
        failed_gates=tuple(failed),
        decision="PASS" if not failed else "FAIL",
    )


def licensed_template_negative_control() -> dict[str, object]:
    """A narrow task in which exact reproduction is intended and licensed.

    This case should not be judged by a general-purpose diversity or originality gate.
    The point is to demonstrate that the contract follows the use claim rather than
    mechanically applying every metric to every generator.
    """
    observed = {"template_fidelity": 0.997, "task_success": 0.995, "safety": 1.0}
    thresholds = {"template_fidelity": 0.990, "task_success": 0.990, "safety": 0.995}
    failed = [key for key, threshold in thresholds.items() if observed[key] < threshold]
    return {
        "name": "licensed_template",
        "not_material": ["diversity", "originality"],
        "observed": observed,
        "failed_gates": failed,
        "decision": "PASS" if not failed else "FAIL",
    }


def serialise(report: Report) -> dict[str, object]:
    def pack(value: Interval) -> dict[str, float]:
        return {
            "estimate": round(value.estimate, 3),
            "low_95": round(value.low, 3),
            "high_95": round(value.high, 3),
        }

    return {
        "name": report.name,
        "aesthetic_not_a_gate": pack(report.aesthetic),
        "metrics": {key: pack(value) for key, value in report.metrics.items()},
        "rare_slice_usefulness": pack(report.rare_slice_usefulness),
        "failed_gates": list(report.failed_gates),
        "decision": report.decision,
    }


def main() -> None:
    rng = np.random.default_rng(SEED)
    train, train_labels = sample_reference(rng, N)
    test, test_labels = sample_reference(rng, N)

    # The 97.5th percentile of held-out-to-training nearest-neighbour distance
    # defines a synthetic local support radius. The copy radius is much tighter.
    manifold_radius = float(np.quantile(nearest_distance(test, train), 0.975))
    copy_radius = 0.010

    reports = [
        assess(
            name,
            train,
            train_labels,
            test,
            test_labels,
            manifold_radius,
            copy_radius,
            seed=10 + offset,
        )
        for offset, name in enumerate(("balanced", "glossy_copier", "wild_explorer"))
    ]

    assert reports[0].decision == "PASS"
    assert set(reports[1].failed_gates) >= {
        "diversity",
        "originality",
        "rare_slice_usefulness",
    }
    assert set(reports[2].failed_gates) >= {"fidelity", "usefulness"}

    control = licensed_template_negative_control()
    assert control["decision"] == "PASS"
    assert control["not_material"] == ["diversity", "originality"]

    output = {
        "assumptions": {
            "samples_per_set": N,
            "bootstrap_resamples": BOOTSTRAPS,
            "manifold_radius": round(manifold_radius, 4),
            "copy_radius": copy_radius,
            "all_values": "synthetic teaching fixtures",
        },
        "broad_generator_contract": [serialise(report) for report in reports],
        "negative_control": control,
    }
    print(json.dumps(output, indent=2, sort_keys=True))


if __name__ == "__main__":
    main()

{
  "assumptions": {
    "all_values": "synthetic teaching fixtures",
    "bootstrap_resamples": 800,
    "copy_radius": 0.01,
    "manifold_radius": 0.2357,
    "samples_per_set": 700
  },
  "broad_generator_contract": [
    {
      "aesthetic_not_a_gate": {
        "estimate": 0.836,
        "high_95": 0.84,
        "low_95": 0.831
      },
      "decision": "PASS",
      "failed_gates": [],
      "metrics": {
        "diversity": {
          "estimate": 0.937,
          "high_95": 0.966,
          "low_95": 0.908
        },
        "fidelity": {
          "estimate": 0.969,
          "high_95": 0.98,
          "low_95": 0.956
        },
        "originality": {
          "estimate": 0.967,
          "high_95": 0.98,
          "low_95": 0.954
        },
        "safety": {
          "estimate": 0.981,
          "high_95": 0.99,
          "low_95": 0.971
        },
        "usefulness": {
          "estimate": 0.899,
          "high_95": 0.92,
          "low_95": 0.874
        }
      },
      "name": "balanced",
      "rare_slice_usefulness": {
        "estimate": 0.826,
        "high_95": 0.879,
        "low_95": 0.758
      }
    },
    {
      "aesthetic_not_a_gate": {
        "estimate": 0.938,
        "high_95": 0.941,
        "low_95": 0.936
      },
      "decision": "FAIL",
      "failed_gates": [
        "diversity",
        "originality",
        "rare_slice_usefulness"
      ],
      "metrics": {
        "diversity": {
          "estimate": 0.781,
          "high_95": 0.844,
          "low_95": 0.713
        },
        "fidelity": {
          "estimate": 1.0,
          "high_95": 1.0,
          "low_95": 1.0
        },
        "originality": {
          "estimate": 0.716,
          "high_95": 0.749,
          "low_95": 0.68
        },
        "safety": {
          "estimate": 0.993,
          "high_95": 0.999,
          "low_95": 0.986
        },
        "usefulness": {
          "estimate": 0.896,
          "high_95": 0.92,
          "low_95": 0.873
        }
      },
      "name": "glossy_copier",
      "rare_slice_usefulness": {
        "estimate": 0.24,
        "high_95": 0.44,
        "low_95": 0.08
      }
    },
    {
      "aesthetic_not_a_gate": {
        "estimate": 0.729,
        "high_95": 0.736,
        "low_95": 0.722
      },
      "decision": "FAIL",
      "failed_gates": [
        "fidelity",
        "usefulness",
        "safety",
        "rare_slice_usefulness"
      ],
      "metrics": {
        "diversity": {
          "estimate": 0.987,
          "high_95": 1.0,
          "low_95": 0.971
        },
        "fidelity": {
          "estimate": 0.659,
          "high_95": 0.691,
          "low_95": 0.623
        },
        "originality": {
          "estimate": 0.99,
          "high_95": 0.997,
          "low_95": 0.983
        },
        "safety": {
          "estimate": 0.96,
          "high_95": 0.973,
          "low_95": 0.944
        },
        "usefulness": {
          "estimate": 0.676,
          "high_95": 0.707,
          "low_95": 0.64
        }
      },
      "name": "wild_explorer",
      "rare_slice_usefulness": {
        "estimate": 0.623,
        "high_95": 0.691,
        "low_95": 0.561
      }
    }
  ],
  "negative_control": {
    "decision": "PASS",
    "failed_gates": [],
    "name": "licensed_template",
    "not_material": [
      "diversity",
      "originality"
    ],
    "observed": {
      "safety": 1.0,
      "task_success": 0.995,
      "template_fidelity": 0.997
    }
  }
}

stateDiagram-v2
  [*] --> Claim
  state "The museum copier" as Claim
  state "1. Fidelity: does the output<br/>belong where the claim says it<br/>belongs?" as Test
  state "2. Diversity: does the<br/>generator cover the intended<br/>variation?" as Revision
  Claim --> Test: expose mechanism
  Test --> Decision: evidence survives
  Test --> Revision: rival survives
  Revision --> Claim: narrow or rebuild
  Decision --> [*]

Part IV · From measurements to release

A scorecard becomes useful when it changes an action. The most defensible rule is conjunctive for critical claims. Let L_i(m,s) be the lower confidence bound for a beneficial metric i for model m on slice s, with threshold τ_i,s. Let U_j(m,s) be the upper confidence bound for a risk rate j, with maximum ρ_j,s. Release requires:

The rule makes uncertainty operational. A point estimate just above threshold does not pass when its interval extends into unacceptable territory. More samples can narrow the interval. Better performance can move it. A narrower claim can remove a non-material dimension. What the team cannot do is average the uncertainty away.

After hard gates, the remaining candidates may be compared using cost, latency, aesthetics or a declared utility function. Use dominance before weighting. Model A dominates B when it is no worse on every material dimension and better on at least one. When trade-offs cross, keep a Pareto frontier. A weighted score is then a local business choice, not a discovery of a natural total called generator quality.

A production-shaped evaluation programme

Consider a creative-asset service whose approved claim is: “Generate draft campaign images for eight prompt families; every selected image receives human review before external use.” The evaluation set begins with representative prompts from each family, then adds rare combinations, multilingual prompts, ambiguous instructions and adversarial attempts. Prompts are fixed before the candidate run and versioned with the release manifest.

Fidelity uses a clean, pinned distributional pipeline plus attribute checks for required objects, counts and brand colours. Diversity is measured within prompts and as minimum coverage across prompt families. Memorisation compares outputs with the authorised training and asset corpus under two encoders, with the closest cases routed to a rights reviewer. Usefulness is the proportion of drafts accepted within a defined edit-time budget, sampled across reviewers. Harm includes prohibited content, representational disparity, privacy triggers and false-refusal rates.

The release gate blocks any critical safety slice, any unresolved near-copy above the audit threshold, and any prompt family below its minimum usefulness or coverage. A model upgrade that improves aesthetics while reducing a minority-language slice does not pass. A cheaper model that satisfies every gate may be selected over a more attractive model. After release, randomly sampled outputs, reviewer overrides, rejected assets and incidents feed the next test set, subject to privacy and provenance controls.

This design changes model selection. The team no longer asks for the best generator in the abstract. It asks which configured system satisfies the declared claim under the observed prompt population, with enough evidence to survive a change review. The released object is the bounded claim and its evidence, not the model name.