TLDR

1. A passing patch is a local observation. The operational unit of value is an accepted repository state transition that survives environment reconstruction, review, integration and later change.
2. The hard mechanism is hidden coupling: requirements, build state, generated artefacts, dependencies, ownership and future commits can invalidate a patch that looked correct in an isolated task.
3. Evaluate agents through a lifecycle benchmark with typed inputs, patch-bound evidence, independent gates and a post-merge sentinel, while reporting review effort and cost separately.
4. This machinery is unnecessary for hermetic, low-coupling tasks. There, a simple localisation-and-repair pipeline is the stronger baseline and often the cheaper design.
5. Choose agent scope and authority from measured repository outcomes, not leaderboard rank. Permit broader autonomy only for task cohorts that remain accepted, reviewable and regression-free.

The patch that passed too early

A coding agent receives a small ticket in a payments software development kit. Retries should use a configurable timeout rather than a hard-coded five seconds. The issue names the relevant module. The agent finds the line, changes the constant, adds a focused unit test and runs the available regression suite. Every test is green. The diff is short, readable and apparently complete.

The repository carries more state than the agent has observed. Public clients are generated from an interface description, and the generated files have not been refreshed. A lockfile still selects a dependency version whose retry type lacks the new field. The local container uses a newer compiler than the continuous integration image. The changed directory has a code owner because its behaviour affects backward compatibility. None of those facts appears in the ticket. None is exercised by the focused test.

The pull request therefore fails in stages. The build matrix rejects one supported runtime. A reviewer asks why the public generated client still exposes the old behaviour. After a manual repair, the change merges. Two weeks later, the scheduled generator runs and restores the old constant from the unchanged interface description. The original demonstration counted success at the first green test. The repository never completed the intended change.

This distinction is causal rather than ceremonial. A patch is text. A repository state includes source, generated artefacts, dependency resolution, build configuration, tests, ownership rules, release metadata and assumptions carried by future work. The same text can be correct in one state and wrong in another. A green test result can belong to an earlier commit. A reviewer can accept a change whose effect later disappears. Measurement has to bind the requirement, environment, patch, evidence, authority and follow-up to the same transition.

Let S₀ be the authorised base state and Δ the proposed change. The agent produces a candidate state S₁ = T(S₀, Δ). The useful question is not simply whether one test function returns true. It is whether the transition has passed the non-compensating conditions that make S₁ a legitimate successor.

The conjunction matters. Ten minutes saved in review cannot compensate for evidence attached to the wrong commit. A high focused-test score cannot compensate for an unauthorised merge. Low model cost cannot compensate for a generated artefact that will overwrite the fix. Cost, latency and review minutes remain useful, but they are soft operating measures observed after the hard acceptance gates.

The boundary is also explicit. This article follows a change through a repository, merge and defined post-merge sentinel. It does not prove that customers received the intended business outcome or that an unseen runtime condition is safe. Those claims require separate production observations. The purpose here is narrower: make the software lifecycle, including its hidden state, the measured environment of the agent.

Part I: The unit of work is a repository transition

A repository is often pictured as a tree of files. That view is convenient for navigation and incomplete for evaluation. A working repository is a bundle of coupled states. Some are committed, such as source files, build scripts and test fixtures. Some are derived, such as generated clients, compiled schemas and lockfiles. Some live outside the tree, such as container images, package registries, protected-branch rules and reviewer assignments. A task succeeds only relative to the parts that its change can disturb.

The benchmark unit should therefore be a transition contract, not an issue prompt. The contract records the authorised base commit, the requirement evidence, the environment recipe, permitted actions, expected postconditions, required reviewers, test oracles and follow-up horizon. The agent may discover additional dependencies, but it cannot silently redefine the contract merely because a convenient test passed.

The distinction between source and state prevents three common attribution errors. First, it prevents the model from receiving credit for scaffolding supplied by the evaluator. A pre-built container, hand-selected file set and hidden human repair are part of the configured system. Second, it prevents evidence drift. Test results must identify the exact patch and environment that produced them. Third, it prevents temporal amnesia. A change that passes today and is automatically reversed by a generator tomorrow did not establish the intended successor state.

Thought experiment: the identical diff

Create two copies of the same repository at the same base commit. In Copy A, the lockfile and generated client are authoritative. The build job verifies both. In Copy B, a nightly process regenerates the client from an interface description stored elsewhere, and the local test command does not run that process. Give the same ticket and the same final diff to the same agent.

In both copies, focused tests pass. In Copy A, the diff is a stable transition. In Copy B, the next regeneration removes it. We varied one causal feature: whether the edited file is authoritative or derived. Patch text, model, prompt and test result stayed fixed. The different outcome shows why correctness cannot be a property of the diff alone.

Now vary a second feature. Keep Copy A unchanged, but run its tests against commit c₁. The agent then adds one more edit and submits commit c₂ without rerunning them. A dashboard that says “tests passed” preserves a true statement and attaches it to the wrong state. The error is not model hallucination. It is a broken evidence chain.

This view does not require every task to carry every gate. It requires the evaluator to justify the profile. A pure function with a complete specification, pinned dependencies and no side effects may need only a small hermetic contract. A public client library, database migration or authentication change needs a wider one. Evaluation depth should scale with hidden coupling and consequence, not with how impressive the agent interface looks.

flowchart LR
  A["The patch that passed too early"] --> B["Part I, The unit of work is a<br/>repository transition"]
  B --> C["Part II, What the benchmark<br/>frame leaves outside"]
  C --> D["Part III, Build a lifecycle<br/>benchmark"]
  D -. "re-test" .-> A
  classDef first fill:#0a1424,stroke:#007D68,stroke-width:2px,color:#e6f3ff
  classDef step fill:#0a1424,stroke:#3f7da0,stroke-width:1px,color:#e6f3ff
  class A first
  class B,C,D step

Part II: What the benchmark frame leaves outside

SWE-bench changed software-agent evaluation for the better by grounding tasks in real repository issues and executable tests. It made a full repository, rather than a standalone function, part of the problem. That remains important evidence. Its success also created a temptation to treat issue-to-patch resolution as the complete software lifecycle.

SWE-Lancer extends realism through paid freelance tasks and managerial choices, with independent tasks graded by end-to-end tests. It is useful evidence about economically framed work. Its defined task endpoints still differ from asking whether a change remains maintainable through later repository evolution.

Several results show why the configured environment matters. SWE-agent found that an agent-computer interface for navigation, editing and execution materially changed performance. The score therefore measures a model, interface and evaluator together. Agentless then supplied a useful rival: a simple three-stage localisation, repair and validation pipeline performed competitively on its benchmark setting at low reported cost. More autonomy was not automatically the cause of success.

Those studies support a disciplined baseline rule. Before adding planning loops, sub-agents, persistent memory or broad shell access, test whether a constrained pipeline can solve the cohort. Complexity earns its place only when it resolves a measured dependency that the simpler system cannot.

From a prepared task to a complete cycle

Recent benchmark designs expose stages that an issue-to-patch task often supplies for free. SWE-Cycle separates environment reconstruction, code implementation and verification-test generation, then composes them into a full-cycle task from a bare repository. Its reported drop from isolated stages to the combined task is evidence of cross-phase dependency, not merely of weaker code generation.

SWE-RPG adds validated intermediate references for requirement clarification and implementation planning. In its evaluated runs, implicit requirement recovery accounted for a substantial share of failures. That result should not be promoted into a universal percentage, but it identifies a failure class that final patch pass or fail cannot locate. RepoReason similarly treats repository reasoning as a diagnostic problem, measuring reading load, simulation depth and integration width. The key lesson is that discovering many facts is different from integrating the right facts into one change.

Review is another independent stage. c-CRAB derives executable tests from human review concerns and reports that the evaluated review agents collectively addressed only part of the benchmark. A patch generator and a review agent can also share blind spots, especially when both rely on similar models or context. Independent review evidence therefore cannot be assumed merely because a second model produced comments.

Time changes what maintainable means

A snapshot cannot reveal whether a design remains easy to change. SWE-EVO evaluates release-sized evolution rather than one isolated issue. SWE-CI follows sequences of historical commits and argues that maintainability becomes visible through successive modifications. SWE-Milestone represents continuous evolution as dependent milestones, so an early design decision can help or obstruct later work. SWE-Marathon extends the horizon further and reports failure modes such as premature termination, poor self-verification and attempts to exploit the verifier in its setting.

These are mostly recent preprints, and their particular datasets, agents and scoring choices need replication. Their collective contribution is methodological: they make requirement recovery, environment setup, review, dependency propagation and future change observable. The article’s argument does not depend on any one leaderboard.

Conditional on a patch passing its immediate tests, the probability of lifecycle acceptance should decline as hidden coupling and follow-up horizon increase. Requirement, environment, integrity, authority and survival receipts should explain additional outcome variance beyond patch pass alone. A paired benchmark can falsify this by showing no incremental predictive value from those receipts.

Why productivity studies can disagree without contradiction

A controlled experiment on an HTTP server task reported that developers with GitHub Copilot completed the task 55.8 per cent faster. A later randomised study by METR, involving experienced maintainers working on their own mature open-source repositories, reported a 19 per cent slowdown in its particular setting. METR’s subsequent design update described task-selection, completion and time-accounting problems that made its newer estimate a weak proxy for overall value.

The honest conclusion is neither “coding agents accelerate work” nor “coding agents slow work”. The unit, repository familiarity, task selection, interaction style, review standard and outcome measure differ. A short, well-specified construction task can benefit from fast generation. A maintainer in a mature repository may spend time checking suggestions, correcting local conventions or waiting for an agent while doing other work. Productivity is a property of a human-agent-repository workflow under a task distribution.

Thought experiment: the frozen repository

Take a benchmark task and freeze the repository forever after the submitted patch. No dependency updates arrive. No other contributor changes the same interface. No generator runs. No code owner asks for a different abstraction. Under that world, passing the hidden tests is a strong endpoint because there is no later state in which brittleness can appear.

Now change only one condition: the repository receives the next three authentic changes from its history. A hard-coded branch that passed the original tests blocks the second change. A slightly more modular patch passes all four states. The first benchmark was not wrong. It measured local functional repair. The second measured evolutionary compatibility. The claim must match the horizon.

Part III: Build a lifecycle benchmark

A lifecycle benchmark starts with a repository transition contract and ends with an evidence package. It does not need to recreate every organisational process. It needs enough structure to expose the causal stages that determine whether the change becomes a legitimate repository state.

Gate R: recover the requirement before optimising the diff

The ticket is an observation about desired change, not necessarily a complete specification. The benchmark should record explicit acceptance criteria, exclusions, compatibility constraints and unresolved questions. The agent may propose inferred requirements, but the evaluator should distinguish stated facts from assumptions. A requirement receipt can be a small typed document containing claim, source, confidence and test implication.

A useful intervention hides one repository convention at a time. Remove a compatibility test, an architecture note or a neighbouring implementation. If performance collapses only when the compatibility note disappears, the agent was using that evidence. If it remains unchanged while the final patch violates the rule, the prompt may have been ignored or the test oracle may be incomplete.

Gate E: reconstruct rather than inherit the environment

Environment reconstruction measures whether the agent can identify and instantiate the repository’s actual execution conditions. The receipt should include a container or toolchain digest, resolved dependency graph, service fixtures and the commands used to establish the base state. The first test is not the candidate patch. It is whether the untouched base repository behaves as expected.

This prevents a common confound. If an evaluator privately repairs the environment, installs an undocumented package or selects the correct runtime, the agent has not demonstrated that capability. Conversely, when the decision is only about code repair inside a managed platform, environment setup may be intentionally supplied. The system boundary must say so.

Gate I: bind every observation to the candidate state

Each receipt should identify the base commit, candidate commit, patch digest, environment digest, test suite version and policy version. A result without those identities is a floating claim. This gate also detects a subtle race: an agent can run tests, modify code and then present the earlier results as if they covered the later patch.

Supply-chain provenance standards such as SLSA provide useful vocabulary for artefact provenance and verified properties. The benchmark need not claim a SLSA level. It can borrow the core discipline that evidence should say what was built, from which source, by which process and under which identity.

Gate V: verify the transition from several angles

Verification should begin with focused tests derived from the requirement, then preserve previously passing behaviour. It should also check generated artefact freshness, schema compatibility, lint or type constraints and at least one adversarial or metamorphic case where appropriate. Hidden tests remain valuable, but secrecy is not independence. A hidden test can encode the same mistaken assumption as the agent.

Failure injection makes the gate more discriminating. Give the agent a stale lockfile, an irrelevant failing test, a misleading comment or a generated file that should not be edited directly. Record whether it finds the authoritative source and whether the repair survives regeneration. The goal is not to create puzzle tricks. It is to vary the causal feature that distinguishes robust repository work from patch mimicry.

Gate A: keep repository authority distinct from model proposal

A coding agent may inspect, propose, test and respond to review. It does not acquire merge authority merely because its checks pass. Repository controls make that distinction executable. Protected-branch rules can require status checks and reviews. CODEOWNERS can require approval from owners of changed files. These mechanisms are not proof of correctness, but they prevent a proposal from silently becoming an authorised state.

Review evidence should include time, number of substantive rounds, concerns raised, changes after review and final disposition. A low review time is good only when acceptance quality remains stable. An agent that floods reviewers with plausible but unnecessary code can increase throughput at the generation stage while reducing system throughput.

Gate S: observe whether the change survives

The post-merge sentinel is a bounded follow-up, not a promise of permanent correctness. Its form depends on the task. It may rerun generation, apply the next historical change, build against the release matrix, monitor a canary or verify that rollback remains possible. The receipt records the observation window and what could still remain unknown.

For a benchmark, future historical commits provide a practical counterfactual. Apply the candidate patch, then replay one or more subsequent changes. Measure conflicts, regressions and extra repair effort. This does not recreate the original team’s intentions perfectly, but it reveals design debt that a one-shot test cannot.

Score cohorts, not theatre

Report hard-gate acceptance by task cohort, with uncertainty across repeated runs. Also report partial progress: environment reconstructed, requirement recovered, focused tests passed, regression tests passed, review accepted and sentinel passed. These stage results locate the bottleneck without allowing one stage to compensate for another.

Keep soft measures beside the hard result: model and tool cost, elapsed time, human review minutes, number of retries, changed lines and rollback effort. Avoid one weighted master score unless the decision utility is explicit. A procurement team may legitimately trade cost and speed among systems that already pass the hard gates. It should not trade five dollars of saving against an unknown post-merge outcome.

Sample the task distribution before comparing systems

A benchmark can be lifecycle-shaped and still mislead if its tasks come from one convenient corner. Stratify by change type, repository age, language, environment maturity, integration width, ownership sensitivity and follow-up horizon. Include bug repair, feature work, refactoring and dependency change. Preserve tasks where the environment initially fails, provided the evaluator can distinguish a reconstructable repository from a broken oracle.

Record selection rules before running the systems. Otherwise an agent may appear to improve because people route it the tickets that already resemble its successful demonstrations. That routing can be operationally sensible, but it changes the claim from general capability to useful specialisation. The result should name the cohort and the rejected population.

Every run also needs a benchmark manifest: model and scaffold versions, tool rights, token and time budgets, network policy, human interventions, random seeds where meaningful, base-state checks and evaluator version. Repeat stochastic runs on a representative subset. Report unresolved environment failures separately from incorrect patches. A system should not receive zero for an evaluator defect, and an evaluator should not quietly repair a capability that the system was meant to demonstrate.

Finally, retain abandoned and rejected attempts. Accepted pull requests alone create survivorship bias. The review burden, dead-end exploration and recovery work in failed trajectories are part of the cost of the configured workflow. A repository benchmark becomes decision-grade only when it records the denominator that produced the successes.

Two worked examples and a negative control

Minimal case: a pure parser function

A repository contains a single function that parses a bounded configuration string. The requirement enumerates valid tokens and expected errors. Dependencies are pinned. The function has no I/O, generated artefacts, shared schema or release-specific behaviour. The benchmark provides the environment and a deterministic test suite, including malformed inputs.

A simple localisation-repair-validation pipeline changes twelve lines and passes focused and regression tests. Requiring code-owner simulation, multi-agent debate and a thirty-day sentinel would add no useful discrimination. The right negative control is deliberately easy to govern. It tests whether the lifecycle proposal has become ritual.

Result: use the hermetic profile. Bind the patch to its test evidence, verify the deterministic contract and stop. The simple baseline wins on cost and inspectability. A complex autonomous agent has not earned extra scope.

Production-shaped case: the retry policy

Return to the payments SDK. The transition contract identifies the issue, supported runtime matrix, public compatibility rule, interface description, generator command, dependency policy and code owner. The base check proves that the untouched repository builds in the pinned environment. The agent may edit source and tests in an isolated worktree, but it cannot push to the protected branch.

The agent first proposes changing only the implementation. A requirement check reveals that public clients are generated. It updates the authoritative interface description, regenerates the clients, refreshes snapshots and adjusts a migration note. Focused tests cover timeout selection and failure behaviour. Regression checks run across supported runtimes. A policy check verifies that the lockfile changed only through the package manager. The patch digest and all test receipts identify the same candidate commit.

A code owner raises a backward-compatibility concern: zero had previously meant “use the library default”, while the new schema treats it as immediate timeout. The agent does not defend its first patch rhetorically. It updates the requirement map, adds a compatibility case and changes the implementation. Earlier approvals and tests become stale because the patch identity changed. The full gate reruns.

After merge, the sentinel executes the generator from a clean checkout and compares outputs. It then applies the next public-client regeneration from repository history. The change remains present and the build passes. The transition is accepted with a residual boundary: no claim has yet been made about customer latency under a live payment incident.

The comparison changes the research question. Asking “Which agent has the highest solve rate?” mixes tasks that need different system boundaries. A better question is “For which coupling profile does this configured workflow produce accepted transitions at lower total cost than the strongest simple baseline?” That framing can support model selection, architecture and governance without turning every repository into a maximal-autonomy experiment.

stateDiagram-v2
  [*] --> Claim
  state "Figure 1. A green patch is a<br/>thin slice through a living<br/>repository" as Claim
  state "Thought experiment, the<br/>identical diff" as Test
  state "Figure 2. What the demo<br/>observes, what the repository<br/>can reveal" as Revision
  Claim --> Test: expose mechanism
  Test --> Decision: evidence survives
  Test --> Revision: rival survives
  Revision --> Claim: narrow or rebuild
  Decision --> [*]

Part IV: Make the agent a bounded repository participant

The lifecycle benchmark implies an operating architecture. The model proposes actions inside an isolated workspace. Deterministic components enforce identity, permissions and effect limits. Independent checks observe the resulting repository state. Human or repository authority decides whether the proposal may merge. The system can expand the agent’s scope only after cohort evidence shows that the extra authority produces better accepted outcomes.

Separate proposal, execution, verification and authorisation. The model can choose a test command from an allowlisted contract; a runner executes it and records the environment. The model can propose a patch; a version-control service binds it to a commit and digest. The model can explain why the change should merge; protected-branch rules and authorised reviewers decide whether it may.

Independence is a property of the evidence path

Adding a second agent does not automatically create independent verification. Two agents using the same model family, repository summary and generated tests can reproduce the same mistaken assumption. Independence comes from changing the evidence path: deterministic compilation, separately derived tests, static policy, runtime observation, a reviewer with different context or a later change that the patch did not anticipate.

A model reviewer can still be useful. It can search for missing tests, inspect API compatibility and explain a suspicious diff. Its result should be marked as a model proposal unless an external oracle verifies the concern. The c-CRAB design is instructive because it tries to turn review claims into executable consequences rather than judging comments by verbal similarity alone.

Security should also remain inside the lifecycle rather than arrive as a final scan. The NIST Secure Software Development Framework treats secure practices as part of the software development lifecycle. For an agent benchmark, that can mean least-privilege tool access, secret isolation, dependency policy, generated artefact provenance and a record of which process produced the candidate. The benchmark need not pretend that these checks prove security. It should reveal whether the agent’s apparent success depended on bypassing them.

This architecture also improves incident analysis. If a bad change merges, investigators can ask which chain broke: the requirement was wrong, the environment was not reproduced, evidence belonged to another patch, verification missed a coupling, required authority was bypassed or the sentinel failed to observe the effect. “The agent made a mistake” is too coarse to support repair.

Executable artefact: a repository transition gate

The following standard-library Python script implements the hard-gate idea. It accepts two profiles. The lifecycle profile requires generated artefact freshness, code-owner approval, resolved review, post-merge observation and a release receipt. The hermetic profile deliberately omits those conditions. Soft measures are reported and never used to override a failed gate.

The script includes three synthetic cases: an accepted lifecycle transition, a deceptive green patch whose evidence belongs to an earlier commit, and the hermetic parser negative control. Assertions make the expected behaviour executable.

#!/usr/bin/env python3
"""Evaluate whether a coding-agent change completed a repository transition.

The gate deliberately separates hard acceptance conditions from soft operating
metrics. It uses only the Python standard library and synthetic evidence.
"""

from __future__ import annotations

from dataclasses import asdict, dataclass
from enum import Enum
import hashlib
import json
from typing import Iterable


class Profile(str, Enum):
    """Controls how much lifecycle evidence a task requires."""

    HERMETIC = "hermetic"
    LIFECYCLE = "lifecycle"


@dataclass(frozen=True)
class TransitionEvidence:
    task_id: str
    profile: Profile
    base_commit: str
    patch_commit: str
    evidence_commit: str
    submitted_patch_hash: str
    verified_patch_hash: str
    requirements_recovered: bool
    environment_reproduced: bool
    focused_tests_passed: bool
    regression_tests_passed: bool
    dependency_policy_passed: bool
    generated_artifacts_fresh: bool = False
    code_owner_approved: bool = False
    review_threads_resolved: bool = False
    post_merge_sentinel_passed: bool = False
    release_receipt_present: bool = False
    review_minutes: int = 0
    agent_cost_usd: float = 0.0


@dataclass(frozen=True)
class GateDecision:
    task_id: str
    accepted: bool
    failed_gates: tuple[str, ...]
    soft_metrics: dict[str, float | int]


def patch_digest(lines: Iterable[str]) -> str:
    """Return a stable digest for the exact submitted patch text."""

    payload = "\n".join(line.rstrip("\n") for line in lines).encode("utf-8")
    return hashlib.sha256(payload).hexdigest()


def evaluate(evidence: TransitionEvidence) -> GateDecision:
    """Apply non-compensating gates and return explicit failure reasons."""

    checks: list[tuple[str, bool]] = [
        ("base_commit_recorded", bool(evidence.base_commit)),
        ("patch_is_a_new_state", evidence.patch_commit != evidence.base_commit),
        ("evidence_bound_to_patch", evidence.evidence_commit == evidence.patch_commit),
        (
            "patch_digest_verified",
            evidence.submitted_patch_hash == evidence.verified_patch_hash,
        ),
        ("requirements_recovered", evidence.requirements_recovered),
        ("environment_reproduced", evidence.environment_reproduced),
        ("focused_tests_passed", evidence.focused_tests_passed),
        ("regression_tests_passed", evidence.regression_tests_passed),
        ("dependency_policy_passed", evidence.dependency_policy_passed),
    ]

    if evidence.profile is Profile.LIFECYCLE:
        checks.extend(
            [
                ("generated_artifacts_fresh", evidence.generated_artifacts_fresh),
                ("code_owner_approved", evidence.code_owner_approved),
                ("review_threads_resolved", evidence.review_threads_resolved),
                ("post_merge_sentinel_passed", evidence.post_merge_sentinel_passed),
                ("release_receipt_present", evidence.release_receipt_present),
            ]
        )

    failed = tuple(name for name, passed in checks if not passed)
    return GateDecision(
        task_id=evidence.task_id,
        accepted=not failed,
        failed_gates=failed,
        soft_metrics={
            "review_minutes": evidence.review_minutes,
            "agent_cost_usd": round(evidence.agent_cost_usd, 2),
        },
    )


def main() -> None:
    patch = ["- timeout = 5", "+ timeout = retry_policy.timeout_seconds"]
    digest = patch_digest(patch)

    positive = TransitionEvidence(
        task_id="payments-sdk-retry",
        profile=Profile.LIFECYCLE,
        base_commit="a81f2c1",
        patch_commit="bc94a76",
        evidence_commit="bc94a76",
        submitted_patch_hash=digest,
        verified_patch_hash=digest,
        requirements_recovered=True,
        environment_reproduced=True,
        focused_tests_passed=True,
        regression_tests_passed=True,
        dependency_policy_passed=True,
        generated_artifacts_fresh=True,
        code_owner_approved=True,
        review_threads_resolved=True,
        post_merge_sentinel_passed=True,
        release_receipt_present=True,
        review_minutes=18,
        agent_cost_usd=2.40,
    )

    deceptive_green_patch = TransitionEvidence(
        task_id="payments-sdk-retry-stale",
        profile=Profile.LIFECYCLE,
        base_commit="a81f2c1",
        patch_commit="d31ed02",
        evidence_commit="bc94a76",  # Results belong to an earlier patch.
        submitted_patch_hash=digest,
        verified_patch_hash="0" * 64,
        requirements_recovered=True,
        environment_reproduced=True,
        focused_tests_passed=True,
        regression_tests_passed=True,
        dependency_policy_passed=True,
        generated_artifacts_fresh=False,
        code_owner_approved=False,
        review_threads_resolved=False,
        post_merge_sentinel_passed=False,
        release_receipt_present=False,
        review_minutes=7,
        agent_cost_usd=1.15,
    )

    negative_control = TransitionEvidence(
        task_id="pure-parser-function",
        profile=Profile.HERMETIC,
        base_commit="1",
        patch_commit="2",
        evidence_commit="2",
        submitted_patch_hash=digest,
        verified_patch_hash=digest,
        requirements_recovered=True,
        environment_reproduced=True,
        focused_tests_passed=True,
        regression_tests_passed=True,
        dependency_policy_passed=True,
        # Lifecycle-only fields are intentionally false for this low-coupling task.
        review_minutes=2,
        agent_cost_usd=0.08,
    )

    decisions = [evaluate(case) for case in (positive, deceptive_green_patch, negative_control)]

    assert decisions[0].accepted
    assert not decisions[1].accepted
    assert "evidence_bound_to_patch" in decisions[1].failed_gates
    assert "generated_artifacts_fresh" in decisions[1].failed_gates
    assert decisions[2].accepted

    print(json.dumps([asdict(decision) for decision in decisions], indent=2))


if __name__ == "__main__":
    main()

payments-sdk-retry          accepted: true
payments-sdk-retry-stale    accepted: false
  failed: evidence_bound_to_patch, patch_digest_verified,
          generated_artifacts_fresh, code_owner_approved,
          review_threads_resolved, post_merge_sentinel_passed,
          release_receipt_present
pure-parser-function        accepted: true

Decision instrument: earn autonomy by cohort

Do not choose one autonomy level for an entire engineering organisation. Classify task cohorts by hidden coupling and consequence. Begin with the least powerful workflow that can satisfy the transition contract. Increase autonomy only when controlled evidence shows lower total effort or better acceptance without loss of reviewability, security or recovery.

A practical rollout begins in shadow mode. Run the agent on historical tasks, then on live tasks without merge authority. Compare its transitions with the simple baseline and ordinary team process. Sample reviewer effort, defects caught after the agent stopped, reversions and follow-up conflicts. Grant broader write actions only to cohorts whose evidence improves. Keep merge authority separately assigned.

A specialised application of this pattern already appears in An Agentic Test Migration Factory with Repository Gates. That article focuses on semantic preservation during test migration. The present argument is broader: every coding-agent workflow needs a declared unit of repository change, a profile of hidden coupling and evidence that remains bound across the lifecycle.

Make the repository, not the patch, the product

Coding demonstrations are persuasive because the visible loop is short. Read an issue, inspect files, write code, run tests. That loop proves a real capability. It does not establish that the repository reached an accepted successor state.

The architecture decision changes when the unit of value changes. Instead of buying the agent with the highest isolated solve rate, evaluate configured workflows against repository transition contracts. Keep a simple localisation-and-repair baseline. Sample tasks across hidden coupling and consequence. Bind tests to the exact patch and environment. Preserve code-owner and merge authority. Observe at least one later state in which a brittle fix could fail.

This approach is deliberately unfriendly to sweeping claims. A system may be excellent at hermetic repairs and weak at requirement recovery. Another may reconstruct environments well and create excessive review burden. A third may pass everything before merge and produce code that obstructs the next change. The lifecycle evidence vector preserves those shapes until a particular engineering decision justifies compression.

The resulting research programme is concrete. Measure how much each gate improves prediction of accepted, regression-free outcomes. Remove gates that add no information for a cohort. Inject failures that distinguish authoritative from derived state. Compare autonomous loops with constrained pipelines. Test whether review and later-change evidence remain independent after model and interface improvements.

The goal is not to make coding agents imitate every human ceremony. It is to place them inside the causal structure that gives software work value. A useful software-engineering agent does more than produce a plausible patch. It helps a repository change state, leaves evidence that the change was legitimate, and stops where its evidence ends.