What a test actually preserves

A migration cannot preserve a test until the team states which parts of the test carry meaning and which parts are replaceable scaffolding.

A test is an observation programme

A conventional unit test looks like a small script, but its semantics are distributed. The fixture chooses an initial world. The stimulus invokes behaviour. The oracle decides which observations matter. Mocks restrict allowed interactions. Timeouts and ordering rules constrain when effects may occur. The runner supplies lifecycle, isolation, discovery and reporting conventions. A line-by-line rewrite can preserve the visible assertions while changing any of the other obligations.

Consider a class whose setUp method creates a fresh in-memory repository before each test. A migration turns that setup into a session-scoped fixture because the fixture appears expensive. Every assertion remains textually equivalent. The suite is faster and green. Yet state now leaks between tests. The target suite does not denote the same experiment, because the initial world has changed.

Test semantics therefore live in a bundle: initial state, stimulus, observations, tolerated variation, side-effect constraints, temporal conditions and execution environment. The migration case should name that bundle explicitly. Otherwise, the factory is validating whatever happens to be easy to measure.

Three levels of equivalence

The word “equivalent” is too strong unless its observation boundary is stated. For test migration, three nested claims are useful.

1. Execution equivalence: source and target are discovered, run and terminate under matched environments.
2. Observation equivalence: they agree on selected outcomes such as pass state, exception class, returned value, external calls, state changes and trace predicates.
3. Discrimination equivalence: they continue to reject the important faulty alternatives that the source rejected.

Compilation and green status sit below all three. Coverage is useful for showing whether code was reached, but it does not show whether the reached behaviour was checked. The 2026 study of unittest-to-pytest migration reported that 51.5 per cent of 480 model-generated variants failed to execute; successful variants retained the same measured coverage.⁸ That is valuable evidence about executability, yet unchanged coverage alone cannot establish preserved fault discrimination.

Preservation is not obedience to the source

The source suite is evidence of intended behaviour, not an infallible authority. It may contain assertions written around an old defect, broad exception checks, sleeps that conceal a race, duplicated tests, or behaviour that policy now requires the product to change. This is one form of the wider test-oracle problem: knowing which observed result should count as correct is often harder than executing the system.⁷ A factory that preserves every source observation perfectly can therefore reproduce technical debt with greater speed and confidence.

Consider an old test that expects any exception from a five-step payment operation. The source framework’s annotation treats an exception from setup, validation or the final call as success. The target framework permits a scoped assertion around the one operation expected to fail. The narrower target is deliberately not observation-equivalent to the source on several faulty implementations. It is a better test, but only if the changed oracle reflects reviewed intent rather than an agent’s stylistic preference.

This creates two distinct change types. A translation change claims that the experiment remains the same while its expression changes. A semantic repair claims that the experiment itself should change. Combining them in one unlabelled patch destroys reviewability: when source and target disagree, nobody can tell whether the factory found a defect, introduced one, or implemented an approved correction.

The case record should therefore allow three dispositions. Preserve requires bounded agreement and retained fault sensitivity. Repair requires an explicit intended divergence, a domain owner, a replacement oracle and challenge cases showing why the new distinction is preferable. Retire requires evidence that the test is obsolete or redundant and that its risk is covered elsewhere. The validator may detect the divergence, but it cannot grant the policy authority that makes the divergence acceptable.

A practical implementation creates two linked candidates when repair is necessary. Candidate one is the closest defensible translation and provides a preservation baseline. Candidate two contains the semantic change and carries a separate decision receipt. Reviewers can then compare the old experiment, the faithful translation and the proposed repair against the same programme variants. This makes an intentional improvement visible without weakening the migration gate for every other test.

The governing rule is to reject unexplained divergence, not all divergence. Exact sameness is inappropriate when the source oracle is wrong, but silent “improvement” is equally unsafe. The factory earns trust by forcing every difference into one of three states: preserved by evidence, authorised as a change, or unresolved and deferred.

flowchart TD
  A["The suite that went green too<br/>easily"] --> B["What a test actually preserves"]
  B --> C["Translation validation, not<br/>generation"]
  C --> D["Where the agent earns its place"]
  D -. "re-test" .-> A
  classDef first fill:#0a1424,stroke:#D95545,stroke-width:2px,color:#e6f3ff
  classDef step fill:#0a1424,stroke:#3f7da0,stroke-width:1px,color:#e6f3ff
  class A first
  class B,C,D step

Translation validation, not generation

The proposal mechanism may be stochastic. The acceptance mechanism should be explicit, deterministic where possible, and independent enough to detect plausible mistakes.

Trust the individual result only after checking it

Compiler verification attempts to prove a translator correct for all admissible inputs. Translation validation takes a different route: it checks each produced result. CompCert demonstrates how strong a mechanically checked semantic-preservation claim can be for a specified compiler and source language.² A migration agent does not come with that kind of theorem. Its prompts, model weights, context assembly and repository state all change the output distribution.

That does not make model assistance unusable. It changes where trust belongs. The model may search, infer local intent and draft transformations. The factory accepts only candidates that survive independent checks. Google’s reported large-scale migration workflow used this generate-and-validate pattern: candidate changes were subjected to completion, whitespace, parsing, model “punt”, build and test checks before human review.⁹ The transferable principle is not that those six checks are universally sufficient. It is that proposal and admission are different authorities.

A model proposes a migrated test; a deterministic authority kernel decides whether the proposal may advance. The kernel owns schema validation, environment selection, execution, comparison, thresholds, provenance and state transition. It should not ask the proposing model whether its own rewrite is semantically correct and treat that answer as proof.

The challenge set is the real specification

A source test run on the current programme provides one positive case. A stronger challenge set adds negative and transformed cases:

• known historical defects and regression fixtures;
• mutation operators targeted to the changed assertions or branches;
• metamorphic relations, such as invariance under reordered independent inputs;
• fuzzed inputs near boundaries and across dependency contracts;
• repeated runs under perturbed order, clock, locale, seed and concurrency;
• source and target execution traces normalised to the same observation vocabulary.

Differential testing has long exploited disagreement among implementations to expose errors. Csmith generated valid C programmes and compared compiler outcomes, reporting hundreds of previously unknown bugs.³ Metamorphic testing addresses cases where a complete oracle is unavailable by checking relations between executions rather than one absolute answer.⁴ Mutation testing deliberately introduces small faults to ask whether a test suite notices.⁵ These techniques become complementary gates in migration because each attacks a different blind spot.

Recent work reinforces the need. A 2026 differential-fuzzing study found that LLM-generated refactorings were functionally non-equivalent in 19 to 35 per cent of cases across its evaluated settings, and about 21 per cent of those non-equivalent refactorings escaped the existing test suites.¹⁰ Kaizen similarly reported semantic divergences in translated high-performance computing programmes that compiled and passed developer-written tests, but failed under metamorphic fuzzing and differential execution.¹¹ DiffTestGen takes a complementary route by directing generated tests towards a particular code change so that behavioural differences are more likely to become observable.¹² These are bounded studies, not universal rates. Their causal lesson is robust: static plausibility and inherited tests share blind spots, so the validator needs generated challenges.

from __future__ import annotations

from dataclasses import dataclass, asdict
from hashlib import sha256
import json
from typing import Callable

Implementation = Callable[[int, bool], int]
Test = Callable[[Implementation], None]


def correct(amount: int, urgent: bool = False) -> int:
    if amount < 0:
        raise ValueError("amount must be non-negative")
    rate = 2 if urgent else 1
    return amount * rate // 100


def mutant_normal_rate(amount: int, urgent: bool = False) -> int:
    if amount < 0:
        raise ValueError("amount must be non-negative")
    return amount * 2 // 100


def mutant_urgent_ignored(amount: int, urgent: bool = False) -> int:
    if amount < 0:
        raise ValueError("amount must be non-negative")
    return amount // 100


def mutant_negative_abs(amount: int, urgent: bool = False) -> int:
    rate = 2 if urgent else 1
    return abs(amount) * rate // 100


def mutant_zero_fee(amount: int, urgent: bool = False) -> int:
    if amount < 0:
        raise ValueError("amount must be non-negative")
    if amount == 0:
        return 1
    rate = 2 if urgent else 1
    return amount * rate // 100


def source_test(impl: Implementation) -> None:
    assert impl(1_000, False) == 10
    assert impl(1_000, True) == 20
    assert impl(0, False) == 0
    try:
        impl(-1, False)
    except ValueError:
        pass
    else:
        raise AssertionError("negative amount was accepted")


def good_target_test(impl: Implementation) -> None:
    # Different style, same distinctions.
    cases = [(1_000, False, 10), (1_000, True, 20), (0, False, 0)]
    for amount, urgent, expected in cases:
        assert impl(amount, urgent) == expected
    try:
        impl(-1, False)
    except ValueError:
        return
    raise AssertionError("negative amount was accepted")


def weak_target_test(impl: Implementation) -> None:
    # Green on the correct implementation, but no precise oracle.
    assert impl(1_000, False) >= 0
    assert impl(1_000, True) >= 0


MUTANTS: dict[str, Implementation] = {
    "normal_rate": mutant_normal_rate,
    "urgent_ignored": mutant_urgent_ignored,
    "negative_abs": mutant_negative_abs,
    "zero_fee": mutant_zero_fee,
}


@dataclass(frozen=True)
class Result:
    test: str
    baseline_passed: bool
    killed: tuple[str, ...]
    source_kill_retention: float


def passes(test: Test, impl: Implementation) -> bool:
    try:
        test(impl)
        return True
    except Exception:
        return False


def evaluate(name: str, test: Test, source_kills: set[str]) -> Result:
    killed = {name for name, mutant in MUTANTS.items() if not passes(test, mutant)}
    retention = len(killed & source_kills) / len(source_kills)
    return Result(name, passes(test, correct), tuple(sorted(killed)), retention)


def main() -> None:
    source_kills = {name for name, mutant in MUTANTS.items()
                    if not passes(source_test, mutant)}
    results = [
        evaluate("source", source_test, source_kills),
        evaluate("good_target", good_target_test, source_kills),
        evaluate("weak_target", weak_target_test, source_kills),
    ]

    assert all(result.baseline_passed for result in results)
    assert results[1].source_kill_retention == 1.0      # positive case
    assert results[2].source_kill_retention < 1.0       # negative case

    payload = {"source_kills": sorted(source_kills),
               "results": [asdict(result) for result in results]}
    canonical = json.dumps(payload, sort_keys=True).encode()
    print(json.dumps(payload, indent=2))
    print("receipt_sha256:", sha256(canonical).hexdigest())


if __name__ == "__main__":
    main()

The expected result is that all three tests pass on correct; the source and good target kill all four mutants; the weak target kills none and therefore receives zero retention. The hash is not a security proof. It is a compact receipt binding the reported result to the exact serialised payload.

Order the gates by cost and information

A practical validator spends cheap checks first: file integrity, parse, schema, discovery and build. It then spends execution budget on matched positive cases, trace comparison, mutants, metamorphic transformations, fuzzing and repeated runs. Expensive human review should receive a small set of candidates with concentrated uncertainty, not every syntactic rewrite.

That order is economic, not epistemic. Passing a cheap gate does not make a later gate optional. It merely prevents spending scarce capacity on obviously invalid candidates. The factory should record the first failing gate and preserve the candidate for diagnosis. Silent discard destroys the data needed to improve recipes, prompts and challenge sets.

Where the agent earns its place

An agent is justified when repository-specific interpretation exceeds a deterministic recipe’s practical reach, and when independent validation can still constrain the result.

Start with the strongest non-agent baseline

Many framework changes are explicit mappings. JUnit documents a gradual path from JUnit 4 to Jupiter, including coexistence through the Vintage engine and concrete replacements for lifecycle annotations, rules, runners and expected exceptions.¹³ OpenRewrite publishes a composite migration recipe that encodes many of these transformations as source-aware rules.¹⁴ For those cases, a deterministic codemod is cheaper to reproduce, easier to review and less variable than a language model.

This negative control matters because “agent factory” can become a decorative label for a batch codemod. The factory should route by semantic distance. Known one-to-one mappings go to recipes. Local transformations with clear examples may go to a constrained model. Cases involving ambiguous intent, altered lifecycle or missing source validity should be escalated to a person.

The semantic long tail

The agent earns its place where mappings are conditional on surrounding code: converting inherited setup into fixtures, preserving custom runner behaviour, replacing hand-built mocks with extension-managed lifecycles, repairing imports across modules, or translating order-dependent integration tests. These cases require context discovery and local synthesis. They also produce the failures seen in the 2026 framework-migration study: missing fixtures, signature drift, structural mismatch, syntax errors and type errors.⁸

A good factory does not send the whole repository to a model and ask for a migration. It builds a typed case package: source test, dependency slice, framework mapping, relevant examples, build target, fixture graph, allowed files, forbidden effects, expected discovery identity and challenge policy. The model returns a patch plus a structured account of assumptions. That account is not trusted as explanation; it is material for targeted validation.

Agent roles without an agent swarm

“Factory” describes repeated governed production, not necessarily multiple conversational agents. A minimal implementation can use one model behind deterministic services. Separate logical roles are still useful:

• Inventory role: extracts tests, build targets, owners, dependencies, fixtures and historical stability.
• Classifier role: assigns migration class and selects deterministic, model or human route.
• Proposal role: produces a patch within a file and capability boundary.
• Validation role: invokes deterministic gates and independent challenge generation.
• Review role: presents the smallest evidence packet needed for an accountable decision.

These roles may be implemented as ordinary functions, queues and workers. Splitting them into agents is justified only when specialisation or independent error checking exceeds the coordination tax. The authority boundary matters more than the number of agents.

Receipts, replay and release

At scale, the product is not only migrated code. It is a replayable decision record showing why each test was accepted, rejected, deferred or superseded.

The migration case as a state machine

Each test begins as an identified source artefact with a stable hash and owner. It advances through classification, proposal, execution, challenge, review and release. Every transition has preconditions and produces an immutable event. A failed build is not “still migrating”; it is a specific rejected candidate linked to a toolchain snapshot. A human edit does not overwrite model provenance; it creates a new candidate with a parent.

This event-sourced design prevents two common errors. First, the programme does not confuse the latest patch with the accepted patch. Second, it can reconstruct what evidence was available when a decision was made. SLSA defines provenance as verifiable information about where, when and how an artefact was produced.¹⁵ In-toto similarly models authorised steps and artefact integrity across a software supply chain.¹⁶ A migration ledger can borrow those ideas without pretending that provenance proves semantic correctness.

What belongs in the decision receipt

The receipt should distinguish source evidence from factory judgement. A passing command is source evidence. “Equivalent enough for this low-consequence module” is a judgement with an owner and policy basis. A model-generated summary is neither unless independently checked.

A production-shaped migration

The factory first inventories tests, rules, runners, owners, historical flake rate and execution cost. It marks 19,400 tests as recipe-compatible, 6,100 as context-sensitive and 2,500 as unresolved because they use custom runners, shared test bases or unavailable services. The numbers are illustrative, but the partitioning logic is real: do not spend model inference on standard annotation changes.

For recipe-compatible tests, an AST transformation creates candidates. For context-sensitive tests, a model receives the source file, superclass slice, fixture graph, approved migration examples and a file-level capability. It cannot edit production code, build policy or neighbouring modules. Unresolved tests enter a human queue with a generated evidence pack, not an automatic patch.

Every candidate passes ordered gates. The runner confirms test identity and discovery count. Source and target run against the same subject commit and container image. A normaliser compares status, exceptions, selected database writes, outbound mock calls and domain events. Critical modules receive targeted mutants for amount boundaries, duplicate authorisation, timeout handling and idempotency. Each test is rerun under shuffled order and two worker counts. Reviewer packets show only unexplained differences and lost mutant kills.

Release occurs by build target, not repository-wide. A target can migrate while JUnit 4 and Jupiter coexist, using the platform’s gradual path.¹³ The factory observes post-merge failures and can roll back the target independently. The programme measures accepted tests per reviewer hour, lost-kill rate, residual manual cases, introduced flake rate and escaped regressions. Raw generation volume is an internal throughput measure, never the success criterion.

Unknown outcome is a first-class state

Distributed test environments fail ambiguously. A runner may time out after applying a database write. A remote browser may lose its connection after an assertion was evaluated. A build service may accept a job without returning the result. The factory must not translate “no receipt” into “failed” or “passed”. It records an unknown outcome, performs readback where possible, and retries only when the action is idempotent.

This distinction matters during migration because duplicated execution can create misleading state. Integration tests may seed data, publish events or reserve ports. The acceptance kernel should know which operations are read-only, reversible, idempotent or externally consequential. A migration system that cannot observe postconditions should reduce autonomy rather than increase retry count.

stateDiagram-v2
  [*] --> Claim
  state "A test is an observation programme" as Claim
  state "Three levels of equivalence" as Test
  state "Preservation is not obedience<br/>to the source" as Revision
  Claim --> Test: expose mechanism
  Test --> Decision: evidence survives
  Test --> Revision: rival survives
  Revision --> Claim: narrow or rebuild
  Decision --> [*]

A decision instrument for migration programmes

Before choosing tools, classify the migration by semantic distance, consequence, observability and the quality of the source tests.

Seven questions before the first batch

1. What behaviour must remain invariant? Name values, effects, traces, timing classes and tolerated differences.
2. Which source tests are already untrustworthy? Identify ignored, flaky, quarantined, duplicated and oracle-free tests before migration.
3. What is the deterministic baseline? Run source-aware recipes before assigning any semantic long tail to a model.
4. Which faults form the challenge set? Use historical regressions, targeted mutants, boundaries and metamorphic relations tied to the migration.
5. Can the validator observe the risky dimension independently? If fixture scope changes, observe state isolation; if mocks change, observe interactions.
6. Who owns exceptions and expiry? Every waived gate needs a named owner, reason, compensating control and review date.
7. How will release be reversed? Prefer coexistence, build-target canaries and behavioural rollback over one irreversible cutover.

Measures that resist dashboard theatre

Programme reporting should separate throughput from assurance. Useful throughput measures include candidates generated, deterministic coverage, median cycle time and reviewer minutes. Assurance measures include source-kill retention, unexplained observation differences, introduced flake rate, unknown outcomes, exception age and escaped migration regressions. The two groups should never be collapsed into one “automation rate”.

A high automation rate can be achieved by weakening tests or routing difficult cases out of scope. A high pass rate can be achieved by generating assertions that rarely fail. A high coverage rate can coexist with an empty oracle. The headline measure should be accepted semantic burden per unit of accountable review, with loss indicators reported separately.

A discriminating pilot

Select three strata: a purely syntactic batch, a fixture-heavy batch and a small set with custom lifecycle semantics. Establish the deterministic baseline first. For each stratum, compare manual migration, deterministic recipe and bounded model assistance under the same review and validation policy. Measure accepted completion, reviewer effort, observation divergence, source-kill retention and introduced flakiness.

The pilot should be able to falsify the agent thesis. If the model reduces typing but increases review time, it has not improved the factory. If deterministic recipes cover nearly all cases, the model is unnecessary. If the validator cannot distinguish weak from preserved tests, the challenge design is inadequate. If high-risk cases remain unobservable, the programme must narrow scope.

This experimental posture also protects against a common category error. A model can be highly capable at producing idiomatic target-framework code while the migration system remains unreliable. Capability belongs to the proposal stage. Reliability belongs to the compound process of context, authority, validation, ownership, release and recovery.

Compact glossary

Translation validation

Checking each produced transformation for required preservation properties rather than assuming the translator is always correct.

Observation vector

The selected outcomes compared across source and target executions, such as status, values, exceptions, effects and trace predicates.

Fault-sensitivity retention

The share of important faulty alternatives rejected by the source test that the target test continues to reject.

Challenge set

The finite collection of correct versions, known faults, mutants, transformed inputs and environment perturbations used for bounded validation.

Decision receipt

A replayable record binding source identity, proposal lineage, execution evidence, exceptions, reviewer judgement and release outcome.

sequenceDiagram
  participant Q as Question
  participant M as Mechanism
  participant E as Evidence
  participant D as Decision
  Q->>M: The suite that went green too easily
  M->>E: A test is an observation programme
  E-->>M: compare rival explanations
  E->>D: Migrate the evidence, not merely the syntax
  D-->>Q: revise scope or act

Migrate the evidence, not merely the syntax

Large test estates tempt teams to optimise for converted files. That is the wrong unit. A test is an executable distinction between behaviour the system accepts and behaviour it rejects. Migration succeeds when the target preserves that distinction under a challenge set appropriate to the consequence.

The architecture follows from that claim. Use deterministic transformations for known mappings, model proposals for bounded semantic variation, and human reconstruction where intent or observability fails. Place all three behind the same authority kernel. Compare source and target observations, challenge their fault sensitivity, perturb the environment, preserve rejected candidates, and issue a decision receipt for every release.

This does not turn finite testing into proof. It makes the remaining uncertainty visible and governable. The programme decision changes accordingly: do not fund an autonomous migration agent as the product. Fund a migration factory whose proposals are replaceable, whose evidence is replayable and whose release rights are earned one case at a time.