An Agentic Test Migration Factory with Repository Gates

Migration is a semantic recovery problem

A legacy test suite contains more than steps. It encodes navigation conventions, data dependencies, business rules, environment assumptions, shared components, exception paths and years of local workarounds. Translating the visible syntax into a modern framework can produce code that runs while losing the behaviour the suite was meant to protect.

The factory should therefore treat every conversion as a change proposal against a repository, not a document translation job. The source test may itself be wrong, duplicated or obsolete. The target application may have changed. The new test needs maintainable selectors, fixtures, data isolation and failure diagnostics. Passing once in a migration environment does not settle any of those questions.

The unit of success is an accepted, maintainable test that protects an approved behaviour in the target repository. Generated lines, converted files and model confidence are not outcomes.

Inventory the estate before assigning agents

Migration begins with a read-only census. The inventory records source suite, test identifier, business journey, owner, execution history, last pass, dependencies, reusable components, data needs, criticality and duplication candidates. It also records whether the source test has a trusted oracle.

The cohort decision should occur before model use. Obsolete tests are retired with evidence. Duplicates are consolidated. Tests whose behaviour changed are redesigned with a product owner. Only suitable tests enter conversion. This avoids paying an agent to preserve accidental complexity.

Recover intent into a canonical test model

The canonical model is the boundary between legacy tooling and target code. It contains preconditions, actors, data, actions, expected observations, cleanup, tolerances and traceability to a requirement or accepted behaviour. It does not retain tool-specific control flow unless that flow has business meaning.

A language model can help interpret opaque step names and connect them to repository symbols. It must not invent missing expectations. If the source clicks “Submit” and then sleeps, the canonical model needs an approved postcondition. A screenshot similarity check may be useful evidence, but it should not become the only oracle for a consequential transaction.

Ambiguity is a routing state, not an invitation to produce plausible test code.

Use specialist stages, not a free-running coding agent

The factory benefits from specialised services because the stages use different context and permissions. A discovery stage reads source and target metadata. A mapping stage creates the canonical model. A generator proposes target code. A verifier executes in a sandbox. A quality stage checks repository conventions. None can merge.

Agent boundaries should be typed. The generator receives the canonical test model, allowed target libraries, repository conventions and a bounded impact map. It does not receive production credentials or unrestricted network access. The verifier executes the exact patch hash in an ephemeral environment. The quality service cannot change the patch while scoring it.

The latest research argues for this caution. SWE-Cycle reports a sharp capability drop when environment reconstruction, implementation and verification are combined end to end. RepoReason identifies integration width as a major repository-level bottleneck. SWE-Skills-Bench finds that many packaged skills add no pass-rate improvement and can add substantial token overhead or harm performance when guidance mismatches context. Those results concern research benchmarks, not a banking migration factory, but they warn against assuming that more agent scaffolding automatically improves engineering outcomes.

Retrieval needs several repository views

Similarity search alone is weak for migration. A shared component may have a generic name. A fixture may live in a distant module. A selector standard may be documented outside the test folder. The context builder should combine symbol search, dependency graphs, ownership metadata, build configuration, execution history and approved examples.

The packet identifies why each file is present. This improves review and reduces prompt volume. Repository content remains untrusted data. A comment or fixture cannot instruct the agent to reveal secrets, disable tests or widen network access. The controller enforces the task envelope outside model context.

Context is pinned to a source commit. If the target branch changes, the patch must be rebased and affected checks rerun. The generator should not work against a moving branch and then report success on a stale snapshot.

Verification must defeat test-passing shortcuts

The verification ladder should include syntax, static analysis, focused execution, mutation or negative testing, integration tests, repository policy and independent review. A generated test can pass because it never reaches the intended path, because selectors match the wrong element or because the assertion is too weak.

The test should demonstrate failure before the target behaviour or implementation is fixed where possible. For a migration, a controlled mutation can prove that the assertion distinguishes correct from incorrect behaviour. This is stronger than watching both old and new tests pass on the same healthy application.

Flakiness is tracked separately. Retrying until green hides unstable selectors, timing or data. The system records every attempt and classifies failure. A quarantine decision has an owner and expiry. It does not count as a completed conversion.

The pull request is an evidence package

The reviewer should receive the behaviour trace, source test, canonical model, files consulted, generated patch, changed dependencies, execution environment, checks, mutations, known gaps and proposed ownership. A verbose model rationale is less useful than these artefacts.

The patch identity binds every result. If the model makes another edit after tests, prior evidence is invalid for affected checks. Generated tests are first-class code changes. Their requirements and oracles receive the same review as production code.

The NIST SSDF and SLSA offer useful controls for protected builds, dependencies and provenance. OpenSSF Scorecard can contribute repository risk signals. None establishes business correctness, so the factory maps them to the relevant gate without claiming certification.

Design for scale without creating a central bottleneck

A central platform team can provide inventory, sandboxes, context building, generation adapters, evaluation and evidence manifests. Domain and repository teams own behaviour, target conventions, acceptance and maintenance. Migration-product owners manage cohorts and dependencies. Security owns cross-cutting policy. This keeps accountability near the code.

Capacity planning should use accepted conversions by complexity class, not a blanket throughput target. A simple field validation differs from a multi-system journey with asynchronous events. Cohorts need separate pass rates, review effort and defect escape measures.

Release the factory in stages

Begin with one application, one target framework and a bounded set of stable tests. Build the inventory and canonical model first. Run generated patches without creating pull requests. Compare intent recovery and oracle quality with experienced engineers. Then enable draft pull requests under strict repository gates.

Expand by cohort only after accepted conversion, review effort, flake rate and defect detection meet the agreed range. High-consequence journeys remain under stronger review. Merge automation, if ever used, should be limited to mechanical classes with deterministic evidence and a proven rollback path.

Follow one conversion from inventory to retirement

Consider a regression suite for a commercial-servicing journey. The legacy test signs in through a shared account, searches for a customer, changes a servicing instruction and checks for a success banner. It embeds two environment URLs, depends on a seeded customer and waits for a fixed number of seconds after submission. The target suite uses service-level setup, role-specific identities, stable selectors, event-based waits and a repository fixture library.

The inventory stage does not merely label the file “UI test.” It identifies the journey, actors, preconditions, protected data, external dependencies, actions, observable outcomes and cleanup obligations. Repository retrieval finds the target application route, API contract, existing fixture patterns and recent tests for the same service. A domain reviewer confirms that the final success banner is insufficient: the durable postcondition is a versioned servicing instruction plus an audit event.

The generation stage creates a small patch rather than a new framework. It imports the repository's existing authenticated client, calls an approved fixture builder and uses the observable audit event as the oracle. The sandbox executes the healthy path, then mutates the submission so that the audit event is absent. If the candidate still passes, its oracle is rejected. It also runs the test repeatedly across isolated workers to detect state leakage.

During dual run, the old and new tests execute against equivalent releases. Differences are classified as legacy defect, candidate defect, environment issue or intentional coverage change. Retirement occurs only after the repository owner accepts the evidence, the replacement has been stable for the agreed period and any unique legacy coverage is preserved elsewhere. A conversion is complete when trusted coverage has moved, not when new code has been written.

Define the factory's typed contracts

Loose natural-language handoffs make a multi-stage factory difficult to govern. Each stage should consume and produce a schema with validation, version and provenance. The model may propose a value, but the controller decides whether that value is admitted to the next stage.

Schemas should represent uncertainty. An inferred precondition carries its source and confidence. A missing business oracle is a named gap, not an empty string. Conflicting repository examples remain separate until a repository owner selects the current convention. Version conversion is explicit so that an old in-flight patch cannot be interpreted under a new schema silently.

Every handoff also has an idempotency key. A retried generator call cannot create multiple branches. A timed-out pull-request request is reconciled through the repository before retry. Sandbox jobs record the exact source and dependency digests. Typed state is what allows the factory to resume safely after interruption without asking a language model to reconstruct what happened.

Engineer test data and environments as products

Migration programmes often blame flaky tests on code generation when the deeper problem is unstable data and shared environments. The factory needs an explicit test-data strategy: synthetic by default, masked only under controlled need, deterministic seeds, bounded retention and cleanup receipts. A fixture describes the business state it creates, not just a sequence of API calls.

An environment broker allocates isolated capacity or a named shared slot. It checks dependency health before execution and records service versions. If a required dependency is degraded, the result is infrastructure-unknown, not a failed candidate. Network capture, browser logs and service traces are collected under redaction policy. Secrets are injected through short-lived workload identity and never appear in prompts, patches or build logs.

Flake analysis separates timing, ordering, shared state, external dependency and genuinely nondeterministic product behaviour. Repeating a test until it passes hides these classes. Quarantine is time-bound, owner-assigned and visible in coverage reporting. The factory may recommend an event wait or fixture isolation, but a repository owner accepts changes that alter test semantics.

Performance and accessibility checks require their own environmental guarantees. A functional conversion should not quietly drop them because the target framework exposes a different API. The canonical intent model records non-functional assertions and routes them to approved runners. This makes gaps visible before the old suite is retired.

Prevent generated-code supply-chain drift

An agent capable of editing tests can also introduce dependencies, shell steps and outbound calls. The patch policy should constrain file scope, package changes, executable permissions, workflow files and network destinations. Dependency additions require a separate approval path with licence, vulnerability and provenance checks.

The generation runtime should have read-only access to source by default and write access only to an isolated worktree. It cannot merge, alter branch protection or change CI secrets. Model-suggested commands pass through an allow-listed executor with argument validation, budgets and working-directory confinement. Output is treated as untrusted until scanners and repository gates pass.

Provenance links the patch to its base commit, retrieved context, approved model route, prompt or policy version, tools and reviewer decisions. The aim is not to archive private reasoning. It is to reproduce the material inputs and actions that affected the repository. A clean test run does not compensate for an unreviewed dependency or an expanded execution boundary.

Make rollback and retirement symmetrical

Factories need a reversal path for both generated changes and programme decisions. A faulty converted test can be reverted through the normal repository process. A cohort can pause when defect escape, flake or review burden crosses a threshold. The legacy suite should remain runnable during the agreed proving window, but it must not remain forever as an unowned safety blanket.

Retirement requires a manifest that maps each removed asset to accepted replacement coverage, owner approval, execution evidence and the commit that removed it. If multiple old tests collapse into one better journey, the manifest records that many-to-one relationship. If a legacy assertion is intentionally discontinued, the business rationale is explicit.

The migration service should publish a cohort ledger: discovered, classified, blocked, proposed, accepted, dual-run, retired and reopened. Counts without complexity and coverage context are not meaningful. A small number of retired high-maintenance journeys may be more valuable than many trivial unit-test rewrites.

Use architecture review to protect the product boundary

Before scaling, the review board should be able to answer:

• Which repository artefacts and execution histories formed the intent record?
• How are business outcomes distinguished from incidental UI behaviour?
• Which stage can write code, open a pull request, run commands or merge?
• How are target patterns selected and invalidated when conventions change?
• Which mutation tests prove that a candidate detects the defect it claims to cover?
• How are test data, identities, secrets and execution logs protected?
• What happens when repository, CI or pull-request state is unknown after a timeout?
• Which measures trigger cohort pause, rollback or legacy-suite retention?
• Who accepts a coverage change, and where is that authority recorded?

Evaluate the factory as a system, not a code generator

An offline benchmark of syntactically valid tests is a weak release signal. The evaluation needs representative repositories, journey complexity, dependency patterns, data constraints and failure cases. Each case has an expert-reviewed intent model and controlled defects that the replacement should detect.

Intent recovery is scored at proposition level: actors, preconditions, actions, outcomes, error paths and cleanup. Patch quality measures scope, repository conventions, dependency discipline and explanation. Oracle evaluation plants controlled defects and verifies that the test fails for the intended reason. Security cases include malicious repository text, unexpected scripts, secret references and attempts to alter protected workflows.

Human review measures correction type and effort, not just acceptance. A reviewer who spends an hour rewriting a plausible patch is evidence of factory cost. Corrections should be attributed to inventory, retrieval, canonical modelling, generation, environment or verification. That diagnosis directs investment far better than a single completion rate.

Production telemetry should distinguish agent effort from system outcomes. Useful measures include accepted coverage moved, defect-detection strength, reviewer minutes, post-merge failure, flake, rollback, security blocks and retirement completion. Token use and generation speed inform cost but do not establish value.

Release comparison should hold repositories and cases stable where possible. A new model, prompt, retrieval strategy or runner version is a component change. The team replays affected cohorts and records the delta. A model upgrade earns no additional repository authority until system-level evidence supports it.

The factory also needs sentinel cases in production. These known patterns exercise critical context, mutation and policy paths without creating real pull requests. A sudden change can reveal repository-index staleness, provider behaviour or runner drift before it affects a migration cohort. Operational alerts should identify the failing component and preserve the relevant manifests for diagnosis.

Architecture governance can then make bounded decisions: approve one framework, increase a cohort size, allow draft pull requests, or pause a repository family. There is no single “agent approved” state. Authority remains granular and reversible.

Record the decisions that make a cohort safe

Each migration cohort should carry a small set of architecture decision records. They name the selected target framework, context sources, permitted file scope, test-data method, verification strategy, pull-request authority and retirement rule. The record includes alternatives and the evidence that justified the choice. It is linked to generated patches so a reviewer can see which assumptions governed the work.

A decision may allow retrieval from code, test history and approved internal patterns while excluding production incidents or customer records. Another may require contract-level assertions for payment state changes. A third may prohibit package additions. The controller turns these decisions into enforceable constraints rather than relying on a project wiki.

Exceptions are bounded by repository, file, reason, approver and expiry. If a legacy journey cannot use synthetic data, the accepted alternative is documented with handling and deletion controls. If one target pattern is temporarily unavailable, the cohort pauses or uses an approved fallback. An agent is not permitted to invent an exception because delivery is late.

The decision set is reviewed when repository ownership, framework version, security posture, environment topology or defect evidence changes. Superseded records remain attached to prior patches. This produces an auditable explanation for why one class of migration was allowed and another remained manual.

Programme governance can compare decisions across cohorts and consolidate only where ownership and constraints genuinely align. Common evidence schemas and runners create scale. Local test meaning and acceptance remain with repository teams. That balance prevents the factory from becoming either an ungoverned global agent or a set of disconnected experiments.

The operating review should examine blocked cohorts as carefully as completed ones. A block caused by missing ownership, unknown test data or an absent business oracle is useful portfolio evidence. It identifies engineering debt that automated conversion cannot solve. Owners can invest in the dependency, accept a bounded manual path or retire the test for a documented reason. Hiding blocks behind a low-confidence patch would merely move the problem into production maintenance.

Cost attribution follows the same stages. Inventory, retrieval, generation, sandbox execution, reviewer correction and dual run are measured separately. This shows whether a stronger model reduces total effort or only shifts cost into context and verification. It also makes build capacity and reviewer demand visible before a larger cohort is approved.

The most mature factory will appear deliberately uneventful. It will produce small patches, explain exclusions, stop on missing intent, fail controlled mutations and retire code only with evidence. That behaviour demonstrates engineering mastery more convincingly than a stream of large automated conversions.

A successful migration factory turns opaque legacy assets into owned, verifiable repository changes. Its value comes from recovering intent, strengthening test oracles and reducing repeated engineering work. The agent is useful because it can navigate and draft at scale. The factory is trustworthy because it never lets drafting become proof.