Code Agents and the Repository as an Environment

A code agent earns repository-scale competence only when edits become controlled interventions in a pinned, executable software world and acceptance comes from independent postconditions, not plausible code.

TLDR

Repository-scale coding is not text completion over a large prompt. It is active diagnosis in a partially observed software world whose files, dependencies, build state, tests and conventions jointly determine whether a change is correct.
The useful mechanism is a closed intervention loop: inspect, form a hypothesis, make a bounded edit, execute the repository, read the resulting evidence, and revise until independent postconditions hold.
Model choice matters less than most leaderboards imply when the interface cannot expose relevant state or the verifier cannot distinguish a real repair from a test-passing shortcut.
For a local, well-specified defect with a strong test, a simple localisation-repair-validation pipeline may equal or beat an autonomous loop; agency must earn its coordination cost.
Architects should grant repository authority only after the environment is reproducible, observations are decision-relevant, commands are scoped, and acceptance is tied to the exact patch through a layered evidence receipt.

The patch that passed and still failed

Worked scenario A small Python library called RetryKit has a precise-looking issue: setting attempts=0 should disable retries, while omitting the value should retain the default of three. The bug appears obvious. In config.py, the function returns value or DEFAULT_ATTEMPTS, so zero is treated as absence.

A model proposes the conventional fix: return the default only when the value is None. The focused unit test now passes. The diff is one line. The explanation is accurate. A reviewer scanning the issue and changed file could reasonably approve it.

Yet the repository still violates the contract. A worker module copied the same fallback expression months earlier. Jobs configured with zero attempts continue to retry in the execution path that matters. The first patch fixed the function named in the test, not the behaviour named in the issue. A second proposal repairs both files but also removes a security scan from the continuous-integration pipeline to shorten feedback. It is functionally correct and still inadmissible.

The puzzle is not that language models sometimes miss code. Human engineers do too. The puzzle is why a fluent patch, a passing test and an accurate local explanation can jointly provide so little evidence about the repository outcome.

The answer is that a repository is an environment, not a document. Its current state includes source, generated artefacts, dependency versions, build rules, test data, configuration, branch history, ownership and executable behaviour. An issue reveals only part of that state. Files shown to a model are observations of it. Edits and commands are interventions. Test results, type errors, traces and diffs are consequences. Correct repair requires a loop that changes hypotheses in response to those consequences.

This framing changes the architecture decision. The central component is no longer “the coding model”. It is the configured system that decides what state can be observed, which actions are permitted, how the exact patch is executed, what counts as independent evidence, and who may accept the result. The model proposes; the repository answers back.

Figure 1. The signature repository dependency terrain. Authored conceptual figure. The issue and focused test sit close in text space, while the behavioural duplicate and protected pipeline policy are farther away. The geometry is illustrative; it represents causal and authority distance, not measured file distance.

Part I

A repository is a partially observed world

A prompt can contain thousands of lines and still omit the state that decides the task. A dependency lock file may select behaviour different from the documentation. A generated client may overwrite a hand edit. A test may be skipped on one platform. A fixture may establish state through a service that the local module never names. A repository is therefore more than source text, just as a chess position is more than a list of legal moves.

The distinction that matters is between world state and context. World state is the configuration of the software world at a particular commit and execution moment. Context is the bounded observation compiled for the agent: issue text, selected files, symbol results, command output, test failures and policy. Increasing the context window does not enlarge the world, and it does not guarantee that the chosen observation contains the causal dependency.

The original SWE-bench made this difference operational by pairing real issues with executable repository snapshots and fail-to-pass tests. Its initial 2,294 instances from twelve Python repositories moved evaluation beyond isolated functions. The important step was not merely more code. It was a world in which a proposed patch could be applied and observed.

SWE-agent then isolated another causal feature: the interface through which a model navigates, edits and runs programs. Its historical result showed a large gain over non-interactive baselines with the same broad model class. That does not prove that an open-ended agent is always necessary. It establishes that the agent-computer interface can be part of the measured capability rather than neutral plumbing.

Thought experiment: the frozen archive and the live worktree

Give two identical models the RetryKit issue. Both receive the same repository commit, token budget and initial files. The first receives a frozen archive rendered into text. It may request no further information. The second receives a read-only search interface, an isolated worktree, a bounded command runner and the same final patch budget.

Vary no other feature. The first system must guess which omitted relations matter before it sees any consequence. The second can search for the configuration symbol, inspect call sites, run the focused test, observe that an integration test still fails and revise its hypothesis. The difference is not extra intelligence hidden in the model. It is an experimental channel that converts uncertainty into evidence.

Now make the issue completely local: one pure function, one unambiguous specification and one exhaustive test. The live worktree may add little. This reversal is important. It predicts that interactive agency should help most when uncertainty can be reduced by cheap, discriminating actions. It should help less when the necessary state is already fully observed.

Figure 2. Interface as a causal intervention. Authored thought experiment. The right-hand system can select new observations after seeing consequences. The figure does not claim that looping always wins; it predicts an advantage only when interaction exposes decision-relevant state.

A useful state formalism

A repository task can be represented as a partially observed controlled process without pretending that software engineering is exactly a textbook POMDP. Let sₜ denote the repository state at step t: commit, files, dependency graph, environment, generated artefacts and relevant external fixtures. Let aₜ be a bounded action such as a search, edit or test command. Let oₜ be the resulting observation.

sₜ₊₁ = T(sₜ, aₜ) and oₜ₊₁ = O(sₜ₊₁, qₜ₊₁)

T is the transition caused by an action. O is the observation process, parameterised by query q. A file read, test run and trace are different observation operators over the same underlying state.

The repair is acceptable only when an independently defined predicate V holds for the final state and declared intent I:

accept ⇔ V(s_final, I, patch_hash, environment_digest) = true

The patch hash and environment digest prevent evidence from silently moving to a different state. If the agent edits after testing, the earlier receipt no longer proves the new patch.

This formalism exposes three failure classes. An observation failure hides the relevant dependency. A transition failure applies an unintended change or executes in the wrong environment. A verifier failure accepts a state that does not satisfy the intended behaviour. Model error is only one path into each class.

SWE-Explore strengthens the observation claim. Across 848 issues, ten languages and 203 repositories, its exploration metrics tracked downstream repair, while line-level coverage and efficient ranking remained differentiating capabilities. A 2026 diagnostic preprint, RepoReason, similarly treats execution as a semantic oracle and reports degradation as integration width and execution depth increase. These results do not establish a universal law, but they make repository exploration measurable rather than treating it as hidden prelude.

Figure 3. Partial observability cutaway. Authored conceptual figure. The inner context is a selected observation. Tooling can reveal more of the repository, but some intended behaviour or external state remains outside autonomous reach and must become an explicit gap.

Horizon is accumulated state, not elapsed time

A ten-minute task can be long-horizon if each step changes what must be remembered and verified. A two-hour task can be short-horizon if it is one deterministic build. The relevant quantity is the number of dependent decisions whose errors survive into later state.

A simple illustration assumes independent per-step error e. After n consequential steps, the probability of an entirely clean trajectory is (1-e)ⁿ. At one per cent error, a hundred steps leave only about 36.6 per cent clean trajectories. Real code-agent errors are not independent, so the equation is not a forecast. A wrong early hypothesis can correlate many later edits, making the decline steeper. Independent tests and checkpoints matter because they can detect and reset accumulated error before it becomes repository history.

METR’s time-horizon work provides a complementary empirical lens by fitting task success against the time a human expert needs. SWE-EVO and SWE-Marathon extend the repository side: broader multi-file evolution and project-scale tasks remain substantially harder than isolated issue repair, with failure analyses pointing to incomplete integration, weak self-verification and premature termination. These benchmarks differ in construction and should not be collapsed into one trend line. Together they warn that success on a short patch benchmark is weak evidence for sustained repository work.

Figure 4. Horizon as unresolved error accumulation. Synthetic relationship, not benchmark data. Curves use the independence formula shown. The saw-tooth checkpoint line is conceptual: a real checkpoint only resets risk when it can independently detect the relevant error class.

Part II

The argument in one passChart · flow

flowchart TD
  A["The patch that passed and<br/>still failed"] --> B["A repository is a partially<br/>observed world"]
  B --> C["Repair is hypothesis testing<br/>by intervention"]
  C --> D["Agency is a conditional design<br/>choice"]
  D -. "re-test" .-> A
  classDef first fill:#0a1424,stroke:#B96D00,stroke-width:2px,color:#e6f3ff
  classDef step fill:#0a1424,stroke:#3f7da0,stroke-width:1px,color:#e6f3ff
  class A first
  class B,C,D step

A structural route through Code Agents and the Repository as an Environment, from The patch that passed and still failed to Sources. Reading use: follow the relation, then test it against the surrounding evidence and limits.

Repair is hypothesis testing by intervention

The most useful code-agent action is not “write code”. It is an action that separates live hypotheses. Suppose a failing test could be caused by a parser, a cache key or a stale generated schema. Reading one more random file adds tokens. Running the failing path with a trace may eliminate two causes. The difference is decision relevance.

A disciplined loop has six movements:

Orient. Pin the base commit, reproduce the failure, inventory build and test entry points, and record what cannot be reconstructed.
Localise. Search symbols, call sites, dependency edges, ownership and recent changes under a context budget.
Hypothesise. State the suspected mechanism and a predicted observation that would distinguish it from a rival.
Intervene. Make the smallest reversible edit or run the cheapest command that can produce that observation.
Verify. Execute focused and broader checks tied to the exact patch, including a negative or mutation case where feasible.
Reconcile. Compare intended, observed and remaining state; stop, revise, or escalate an explicit specification gap.

This is closer to debugging than to prose generation. The agent maintains a temporary belief about the repository, but the repository state remains authoritative. A confident explanation cannot override an execution result. Conversely, a failing command is not automatically evidence against the patch if the environment is unhealthy. The observation needs a typed status such as candidate-failed, infrastructure-unknown or verifier-invalid.

SWE-Cycle makes the phase dependency visible. It evaluates environment reconstruction, implementation and verification test generation separately, then combines them in a bare-repository FullCycle task. Solve rates drop sharply in the combined setting. The result supports a practical inference: strong isolated components do not guarantee a coherent state transition across the whole repair cycle.

Minimal worked example: search before edit

Return to RetryKit. The issue names normalise_attempts, so a direct patcher edits that function. A repository agent first searches for DEFAULT_ATTEMPTS and the phrase or DEFAULT_ATTEMPTS. It finds the worker duplicate. That search is useful because the issue concerns a semantic contract, not a file.

The agent then predicts two outcomes. If the defect is local, the focused unit test is sufficient. If the contract is duplicated, a worker-level case with zero attempts will still fail. It runs both. The second failure changes the patch scope for a reason that can be inspected. The final edit centralises fallback semantics in one function and makes the worker call it.

The causal gain comes from choosing an observation that could prove the first patch incomplete. Merely reading more of the repository would not guarantee that distinction. This is why a context compiler should retain the reason each file or command entered the trajectory.

Thought experiment: the same patch under two oracles

Take the local-only RetryKit patch and hold it fixed. Oracle A runs the new unit test in config.py. Oracle B runs that test, a worker integration test and a mutation that reintroduces each fallback defect. No model, prompt or patch changes. Only the verifier changes.

Oracle A accepts. Oracle B rejects. Therefore “the agent solved the issue” is not a property of the patch alone. It is a relation among patch, intent, environment and acceptance instrument.

This is not a theoretical corner. An empirical study of plausible SWE-bench Verified patches found that running all developer tests exposed an average 7.8 per cent of previously plausible patches as incorrect. Its broader analysis estimated about 11 per cent incorrectness among plausible patches and an average 6.4 percentage-point inflation in reported resolution rates. The authors also found cases that passed all developer tests yet differed semantically from the human patch. The study is bounded by its sampled tools and manual analysis, but it directly demonstrates verifier-induced overstatement.

A separate OpenAI audit shows the opposite failure: tests can reject correct implementations because they are too narrow, or demand unspecified behaviour because they are too wide. The audited set was deliberately selected from 138 problems that one model did not consistently solve, so the 59.4 per cent issue rate must not be generalised to the full benchmark. The lesson is still decisive. More tests are not automatically a better oracle; tests must correspond to stated behaviour.

Independent verification means the evidence can disagree with the generator for a reason the generator did not choose. Hidden tests, differential behaviour, mutation, static analysis, compatibility checks and human semantic review each cover different failure classes. A model that writes its own acceptance test from the same mistaken interpretation creates correlated reassurance.

Figure 5. Verification cone. Authored conceptual figure. Each outer layer can reject a patch that passes the inner layer. The ordering is not a claim that human review is always strongest; it marks a widening semantic claim.

Executable repository repair lab

The companion artefact turns the opening case into a deterministic benchmark. It creates a temporary synthetic repository, applies three candidate patches, compiles the tree, runs visible and hidden tests, injects two known mutations, checks changed-file scope and emits a patch-bound JSON receipt.

The candidates vary one feature at a time. local_only fixes the named function but leaves the worker contract broken. repository_fix repairs both paths and survives mutation. scope_escape is functionally equivalent to the accepted repair but also removes a protected CI step. This separates correctness from authority.

Candidate	Scope	Compile	Visible test	Hidden contract	Mutation	Accepted
`local_only`	Pass	Pass	Pass	Fail	Fail	No
`repository_fix`	Pass	Pass	Pass	Pass	Pass	Yes
`scope_escape`	Fail	Pass	Pass	Pass	Pass	No

The table is measured output from the included script, not a model benchmark. Its purpose is logical: a visible green test is neither repository correctness nor permission to change every file.

Figure 6. Synthetic measured gate pattern. Values come from the runnable standard-library artefact bundled with this article. Teal circles are passed gates; coral hatching marks rejection. The matrix is intentionally small enough to inspect line by line.

Complete runnable Python lab

Assumptions. Python 3.10 or later; no third-party packages; no network access. Save as repository_repair_lab.py and run python3 repository_repair_lab.py. Expected output matches the table above and ends with three assertions.

"""Repository repair lab for 'Code Agents and the Repository as an Environment'.

Python 3.10+. Standard library only. All repositories, issues, patches and tests are
synthetic. The lab demonstrates that visible-test success, repository correctness,
and authorised change scope are different acceptance conditions.
"""
from __future__ import annotations

from dataclasses import asdict, dataclass
from hashlib import sha256
from pathlib import Path
from tempfile import TemporaryDirectory
from typing import Mapping
import compileall
import json
import os
import subprocess
import sys

BASE_FILES: dict[str, str] = {
    "retrykit/__init__.py": "from .config import normalise_attempts\nfrom .worker import effective_attempts\n",
    "retrykit/config.py": '''DEFAULT_ATTEMPTS = 3\n\n\ndef normalise_attempts(value: int | None) -> int:\n    """None means use the default; zero means disable retries."""\n    return value or DEFAULT_ATTEMPTS\n''',
    "retrykit/worker.py": '''from .config import DEFAULT_ATTEMPTS\n\n\ndef effective_attempts(job: dict[str, int | None]) -> int:\n    # Historical duplicate of configuration fallback logic.\n    return job.get("attempts") or DEFAULT_ATTEMPTS\n''',
    "tests/test_config.py": '''import unittest\n\nfrom retrykit.config import normalise_attempts\n\n\nclass ConfigTests(unittest.TestCase):\n    def test_zero_disables_retries(self):\n        self.assertEqual(normalise_attempts(0), 0)\n\n    def test_none_uses_default(self):\n        self.assertEqual(normalise_attempts(None), 3)\n\n\nif __name__ == "__main__":\n    unittest.main()\n''',
    "tests/test_unrelated.py": '''import unittest\n\nfrom retrykit.config import normalise_attempts\n\n\nclass RegressionTests(unittest.TestCase):\n    def test_positive_value_is_preserved(self):\n        self.assertEqual(normalise_attempts(5), 5)\n\n\nif __name__ == "__main__":\n    unittest.main()\n''',
    "ci/pipeline.yml": "steps:\n  - unit-tests\n  - security-scan\n",
}

HIDDEN_TEST = '''import unittest\n\nfrom retrykit.worker import effective_attempts\n\n\nclass RepositoryContractTests(unittest.TestCase):\n    def test_worker_honours_zero_retry_contract(self):\n        self.assertEqual(effective_attempts({"attempts": 0}), 0)\n\n    def test_worker_uses_default_only_for_none(self):\n        self.assertEqual(effective_attempts({"attempts": None}), 3)\n\n\nif __name__ == "__main__":\n    unittest.main()\n'''

FIXED_CONFIG = '''DEFAULT_ATTEMPTS = 3\n\n\ndef normalise_attempts(value: int | None) -> int:\n    """None means use the default; zero means disable retries."""\n    return DEFAULT_ATTEMPTS if value is None else value\n'''

FIXED_WORKER = '''from .config import normalise_attempts\n\n\ndef effective_attempts(job: dict[str, int | None]) -> int:\n    return normalise_attempts(job.get("attempts"))\n'''

CANDIDATES: dict[str, dict[str, str]] = {
    "local_only": {
        "retrykit/config.py": FIXED_CONFIG,
    },
    "repository_fix": {
        "retrykit/config.py": FIXED_CONFIG,
        "retrykit/worker.py": FIXED_WORKER,
    },
    "scope_escape": {
        "retrykit/config.py": FIXED_CONFIG,
        "retrykit/worker.py": FIXED_WORKER,
        "ci/pipeline.yml": "steps:\n  - unit-tests\n  # security scan removed to shorten the feedback loop\n",
    },
}

ALLOWED_PATHS = {"retrykit/config.py", "retrykit/worker.py", "tests/test_config.py"}
TEST_TIMEOUT_SECONDS = 10


@dataclass(frozen=True)
class Result:
    candidate: str
    changed_files: tuple[str, ...]
    patch_digest: str
    scope_ok: bool
    compiles: bool
    visible_tests: bool
    hidden_contract: bool
    mutation_detected: bool
    accepted: bool


def write_files(root: Path, files: Mapping[str, str]) -> None:
    for relative, content in files.items():
        path = root / relative
        path.parent.mkdir(parents=True, exist_ok=True)
        path.write_text(content, encoding="utf-8")


def run_unittest(start_dir: Path, repo: Path) -> bool:
    env = os.environ.copy()
    env["PYTHONPATH"] = str(repo)
    try:
        completed = subprocess.run(
            [sys.executable, "-m", "unittest", "discover", "-s", str(start_dir), "-p", "test_*.py"],
            cwd=repo,
            env=env,
            stdout=subprocess.PIPE,
            stderr=subprocess.STDOUT,
            text=True,
            check=False,
            timeout=TEST_TIMEOUT_SECONDS,
        )
    except subprocess.TimeoutExpired:
        return False
    return completed.returncode == 0


def digest_patch(patch: Mapping[str, str]) -> str:
    payload = "".join(f"{path}\0{patch[path]}\0" for path in sorted(patch))
    return sha256(payload.encode("utf-8")).hexdigest()[:12]


def mutation_is_detected(repo: Path, visible_dir: Path, hidden_dir: Path) -> bool:
    """A strong oracle must reject reintroducing either known defect."""
    config_path = repo / "retrykit/config.py"
    worker_path = repo / "retrykit/worker.py"
    original_config = config_path.read_text(encoding="utf-8")
    original_worker = worker_path.read_text(encoding="utf-8")

    try:
        config_path.write_text(BASE_FILES["retrykit/config.py"], encoding="utf-8")
        config_mutant_rejected = not run_unittest(visible_dir, repo)
        config_path.write_text(original_config, encoding="utf-8")

        worker_path.write_text(BASE_FILES["retrykit/worker.py"], encoding="utf-8")
        worker_mutant_rejected = not run_unittest(hidden_dir, repo)
    finally:
        config_path.write_text(original_config, encoding="utf-8")
        worker_path.write_text(original_worker, encoding="utf-8")

    return config_mutant_rejected and worker_mutant_rejected


def evaluate(name: str, patch: Mapping[str, str]) -> Result:
    with TemporaryDirectory(prefix=f"repo-lab-{name}-") as temp:
        root = Path(temp)
        repo = root / "repo"
        hidden = root / "judge"
        write_files(repo, BASE_FILES)
        write_files(repo, patch)
        write_files(hidden, {"test_hidden.py": HIDDEN_TEST})

        changed = tuple(sorted(patch))
        scope_ok = set(changed).issubset(ALLOWED_PATHS)
        compiles = compileall.compile_dir(repo, quiet=1)
        visible = run_unittest(repo / "tests", repo) if compiles else False
        hidden_ok = run_unittest(hidden, repo) if visible else False
        mutation = mutation_is_detected(repo, repo / "tests", hidden) if hidden_ok else False
        accepted = all((scope_ok, compiles, visible, hidden_ok, mutation))

        return Result(
            candidate=name,
            changed_files=changed,
            patch_digest=digest_patch(patch),
            scope_ok=scope_ok,
            compiles=compiles,
            visible_tests=visible,
            hidden_contract=hidden_ok,
            mutation_detected=mutation,
            accepted=accepted,
        )


def mark(value: bool) -> str:
    return "PASS" if value else "FAIL"


def main() -> None:
    results = [evaluate(name, patch) for name, patch in CANDIDATES.items()]

    headers = ("candidate", "scope", "compile", "visible", "hidden", "mutation", "accepted")
    rows = [
        (
            r.candidate,
            mark(r.scope_ok),
            mark(r.compiles),
            mark(r.visible_tests),
            mark(r.hidden_contract),
            mark(r.mutation_detected),
            "YES" if r.accepted else "NO",
        )
        for r in results
    ]
    widths = [max(len(headers[i]), *(len(row[i]) for row in rows)) for i in range(len(headers))]
    print("  ".join(headers[i].ljust(widths[i]) for i in range(len(headers))))
    print("  ".join("-" * widths[i] for i in range(len(headers))))
    for row in rows:
        print("  ".join(row[i].ljust(widths[i]) for i in range(len(headers))))

    print("\nJSON receipt:")
    print(json.dumps([asdict(r) for r in results], indent=2))

    by_name = {r.candidate: r for r in results}
    assert by_name["local_only"].visible_tests and not by_name["local_only"].hidden_contract
    assert by_name["repository_fix"].accepted
    assert by_name["scope_escape"].hidden_contract and not by_name["scope_escape"].scope_ok


if __name__ == "__main__":
    main()

Part III

Agency is a conditional design choice

The repository-as-environment thesis can be overstated. It does not imply that every patch needs an autonomous explorer. Agentless is the decisive negative control. Its fixed localisation, repair and patch-validation pipeline reported 32 per cent on SWE-bench Lite in its original evaluation, outperforming then open-source agent baselines at low reported cost. The authors also identified problematic benchmark instances, reinforcing the need to scrutinise both task and verifier.

That result matters because it reproduces much of the visible success without allowing a model to choose an open-ended sequence of actions. For bounded issue repair, a well-engineered pipeline can concentrate context, reduce trajectory drift and make failure easier to diagnose. Agency should therefore be justified by the value of adaptive observation, not by the prestige of autonomy.

Use the least adaptive mechanism that can close the evidence gap. A direct patch generator is suitable when the relevant code and contract are already local and complete. A fixed localisation-repair-validation pipeline fits tasks with predictable phases. A bounded interactive agent fits tasks where each observation changes what should be inspected or edited next. Human-led work remains necessary when intended behaviour, acceptable trade-offs or external operational state cannot be derived from the repository.

A matched experiment that can weaken the thesis

A useful evaluation must separate the benefit of interaction from the benefit of simply spending more tokens or seeing more files. Hold the model, base commits, issue statements, wall-clock ceiling and independent verifier constant. Randomly assign matched tasks to three interfaces: a static bundle that permits one proposal, a prescribed localisation-repair-validation pipeline, and an interactive loop that may choose its next observation or edit. Give every arm the same command catalogue and total inference budget. The variable is whether evidence can redirect the trajectory.

Stratify tasks before running them. One stratum contains local defects whose relevant contract and failing test already sit together. Another contains cross-file defects with duplicated rules, generated artefacts or build dependencies. A third contains specification gaps that execution cannot resolve. This prevents an average score from hiding the mechanism. The thesis predicts little benefit, and possible harm, from interaction in the first stratum; a gain over the fixed pipeline in the second; and no defensible autonomous solution in the third.

Measure four outcomes. First, test functional correctness with pre-authored focused, regression and differentiating checks. Second, bind every result to the exact patch and environment digest so that later edits cannot inherit earlier evidence. Third, record scope violations, weakened tests and changes to protected configuration. Fourth, measure the cost of each accepted repair in model calls, executed commands, elapsed time and reviewer corrections. A pass rate without these denominators cannot show whether interaction bought information or merely bought more attempts.

The mechanism measure is more demanding. For each selected observation, require a compact pre-registered record: the live hypotheses, the predicted outcomes and the next decision under each outcome. Then ask whether the observation actually changed localisation, patch scope or verification. Repository browsing that never changes a decision is context consumption, not evidence-seeking. A failed command may still be valuable if it eliminates a hypothesis; a successful command may be waste if its result was already implied by known state.

The thesis weakens if adaptive interaction adds no concentrated advantage on tasks whose dependencies are initially hidden but executable feedback can reveal them. It also weakens if gains disappear when extra sampling, context volume and command count are controlled. In that case, the real explanation may be search budget rather than repository feedback. Conversely, a gain limited to cross-file, observation-sensitive tasks would support the narrower causal claim without licensing autonomy elsewhere.

This experiment also distinguishes model progress from environment progress. Repeat it after improving only symbol search, build reproducibility or verifier coverage. If all model arms improve, the environment was the bottleneck. Repeat it after changing only the model. If performance rises while scope violations and verifier disagreements remain, proposal quality improved without solving repository control. That distinction changes investment: better models, better tools and stronger acceptance instruments are separate interventions, even when one product bundles them.

Figure 7. Mechanism decision field. Practitioner synthesis, not measured boundaries. The diagonal represents the point at which adaptive observation is expected to repay trajectory cost. High product ambiguity remains human-led even when tools are excellent.

The explicit failure boundary

The intervention loop ceases to help when the environment cannot answer the question being asked. Five boundaries should trigger redesign or escalation.

Missing normative intent. Tests and code show what the system does, not which behaviour a product owner should prefer. An agent cannot recover a policy decision from execution.
Irreproducible state. If dependencies, data or services cannot be reconstructed, failures remain ambiguous. More retries can turn uncertainty into false confidence.
Weak or correlated oracles. A generator that writes the patch, test and explanation from one mistaken interpretation may create a coherent but wrong evidence bundle.
Unbounded authority. An action loop with shell, network, secrets and merge rights can optimise the task by weakening controls or changing the evaluator.
Moving base state. Evidence collected on one commit does not prove a rebased patch. The system must pin, reconcile or rerun.

More context is not a universal repair. The 2026 SWE-Skills-Bench preprint reports that most tested skill packages produced no pass-rate improvement, while some version-mismatched guidance degraded results. SWE-bench Science likewise reports mixed effects from scientific guidance: aligned knowledge can constrain repair, while poorly aligned guidance can anchor the agent. These are bounded benchmark findings, yet they support a practical rule: context should be selected as evidence, not injected as ceremony.

Figure 8. Repository repair failure surface. Practitioner synthesis. Reproducibility and oracle strength are separate gates. A stable environment with a weak oracle produces repeatable overfitting; a strong oracle in an unstable environment produces uninterpretable failures.

Part IV

How the claim changes under pressureChart · state

stateDiagram-v2
  [*] --> Claim
  state "Thought experiment: the frozen<br/>archive and the live worktree" as Claim
  state "A useful state formalism" as Test
  state "Horizon is accumulated state,<br/>not elapsed time" as Revision
  Claim --> Test: expose mechanism
  Test --> Decision: evidence survives
  Test --> Revision: rival survives
  Revision --> Claim: narrow or rebuild
  Decision --> [*]

The revision loop begins with Thought experiment, the frozen archive… and keeps counterexamples inside the reasoning path. Reading use: follow the relation, then test it against the surrounding evidence and limits.

Design the repository environment, not the demo

A production-shaped task exposes why the environment must include authority and evidence, not merely tools. Consider a synthetic payments monorepo. A requirement changes idempotency keys from “one key per request” to “one key per business operation”. The implementation spans an API validator, a worker, a database uniqueness constraint, an SDK, migration order, telemetry and a compatibility period for older clients.

A code agent receives a typed change request and a pinned commit. It may read the repository and issue history, but not production records. It works in an isolated branch with no merge permission. The command broker allows declared build, test and schema-inspection commands; outbound network is denied unless a dependency mirror is explicitly authorised. The agent proposes a plan, localises call paths, edits a bounded file set and runs focused checks.

The repository then supplies several surprises. A generated SDK overwrites one manual change. A compatibility test shows that old clients reuse keys differently. A database migration passes locally but violates the downgrade test. Each observation changes the plan. A context dump prepared before execution could not have guaranteed those discoveries because they arise from interventions against exact state.

The final pull request is an evidence package, not a transcript. It binds the base commit, patch hash, changed paths, commands, environment digest, focused and regression results, schema diff, known gaps and reviewer decision. If the agent edits after the last test, affected receipts expire. If the pull-request creation call times out, the controller reads repository state before retrying so it does not create duplicates.

Practitioner synthesis Evidence also has a state boundary. Suppose every check passes on base commit b₀, then a dependency update lands and the branch is rebased to b₁. The source hunk may be textually identical, yet the executable world has changed: a transitive parser version, generated client or migration order may now differ. Reusing receipts from b₀ would confuse “this patch passed there” with “this candidate satisfies the postcondition here”. The controller should compute an impact set from the changed base, patch and dependency graph. Unaffected evidence may remain valid only under an explicit reuse rule; affected checks expire and run again. The same rule applies after a formatter, code generator, conflict resolution or reviewer amendment changes the patch.

Long-horizon work creates many opportunities for this evidence drift. The agent may explore for hours, the branch may move, or a person may amend one file. Recovery should therefore reconstruct from authoritative state: current commit, applied diff, action receipts, open obligations and verifier results. The transcript can inform diagnosis, but it cannot resurrect expired proof. A green badge without patch and environment digests is historical evidence, not a current postcondition.

The deterministic shell around the model is what turns stochastic proposals into governed repository changes. It does not make the patch correct. It makes authority, state and evidence inspectable enough for correctness to be tested.

Ten fields of a repository control loop

Field	Repository meaning	Control question
Intent	Issue, acceptance criteria, exclusions and consequence	What behaviour is being changed, and who may clarify it?
Identity	Requester, agent runtime, reviewer and merge actor	Who requested, proposed, verified and accepted?
World state	Base commit, dependencies, generated artefacts, services and fixtures	Which exact software world produced the observation?
Context	Selected files, symbols, history, logs and policy	Why is each observation relevant and permitted?
Reasoning	Current hypotheses, predictions and unresolved gaps	Which observation could distinguish the live alternatives?
Authority	Allowed paths, commands, network, secrets and budgets	Which effects are mechanically impossible?
Action	Edits, commands, generated tests and pull-request proposal	Is each action typed, bounded and reversible?
Evidence	Patch-bound test, analysis, mutation and provenance receipts	Can evidence disagree with the generator?
Outcome	Verified behaviour, regression status and review disposition	Did the intended postcondition hold, not merely the command?
Release and recovery	Merge authority, canary, rollback, timeout reconciliation and incident path	What happens after acceptance or an unknown outcome?

These fields prevent unlike information from being called “memory”. Workflow state records where the repair is in its lifecycle. World state describes the current repository and environment. Context is the selected observation. An evidence ledger records what was tested against which patch. Repository knowledge such as conventions and ownership may persist, but it must remain attributable and supersedable.

Figure 9. Production authority flow. Authored architecture figure. Boxes and routed arrows are used because authority, execution and evidence boundaries are the subject. Repository content remains untrusted input; the controller, worktree and verifier enforce effects outside the model.

The repository is also untrusted input

Comments, issue text, build scripts and contributor files can contain instructions that are meaningful to a model but have no authority. A repository agent must treat them as data about the task, not commands to reveal secrets, disable tests or widen network access. GitHub’s security guidance warns that untrusted pull-request content can compromise workflows when privileged triggers and checkout are combined. Its coding-agent guidance similarly recommends limiting tools and ignoring untrusted event sources by default.

The command broker should therefore validate executable, arguments, working directory, network destination, time, output size and write scope. Secrets should enter only through short-lived workload identity into commands that require them, never through the model context. The agent should not merge, alter branch protection or rewrite its own verifier.

NIST’s SSDF provides the broader secure-development frame. SLSA provenance supplies a useful pattern for recording where, when and how an artefact was produced. Neither proves semantic correctness. Their value is narrower: they make the production and custody of the patch and build evidence harder to misattribute.

Repository authority decision record

The practical decision should not be “Which coding agent is approved?” Authority belongs to a task class inside a repository environment. The following gate record is deliberately non-additive. Failure of one hard gate is not compensated by a stronger model.

Six gates before granting interactive repository authority

Intent gate. Acceptance criteria name observable behaviour, exclusions and the person authorised to resolve ambiguity.
State gate. The base commit, dependencies, fixtures and required services are pinned or every unreconstructed element is typed as unknown.
Observation gate. Search, history, build and test interfaces expose the dependencies likely to decide the task within a declared budget.
Action gate. Paths, commands, network, credentials, package changes and pull-request rights are mechanically scoped and reversible.
Evidence gate. At least one verifier is independent of the patch-generation path, and all results bind to the patch and environment digests.
Recovery gate. Timeouts, duplicate effects, rebase, rollback, failed verification and incident escalation have deterministic handlers and owners.

Decision rule. If intent or evidence fails, keep the task human-led. If state or observation fails, improve the environment before upgrading the model. If action or recovery fails, allow read-only analysis or draft patches only. Grant bounded interactive authority only when all six gates hold for the named task class.

Evaluation should mirror the record. Report localisation coverage, environment reconstruction, patch correctness, regression preservation, mutation detection, scope violations, reviewer correction and unknown outcomes separately. A single resolved-rate number hides whether the system improved because it understood more, received a better interface, saw leaked context or encountered a weaker verifier.

SWE-bench-Live demonstrates why freshness belongs in this evidence. Its live-updatable tasks from newer issues show a performance gap relative to static benchmarks under controlled evaluation and are designed to reduce contamination risk. The implication is not that one benchmark replaces another. It is that a repository agent should be tested on changing, executable environments rather than granted durable authority from a frozen leaderboard.

Compact glossary

World state: The exact repository, dependency, environment and fixture state whose behaviour is being changed.
Context: A bounded observation compiled from the world state for a particular decision.
Intervention: An edit or command chosen because its consequence can separate competing hypotheses.
Plausible patch: A patch that passes the available validation but may still be semantically incorrect.
Decision receipt: The record of why a proposed action was admitted under identity, intent and policy.
Effect receipt: Evidence that an exact command or patch produced an observed state in a named environment.
Unknown outcome: A state in which a command may have taken effect but the system lacks reliable confirmation.
Repository authority: The mechanically enforced rights to read, edit, execute, open a pull request, merge or release.

Make the repository answer back

A code model can propose a convincing edit from text. A repository-scale agent has a harder burden: it must discover which state matters, make bounded interventions, interpret consequences, preserve unrelated behaviour and produce evidence that can reject its own proposal.

The design decision therefore moves away from model-first procurement. Ask four questions instead. What exact software world is the system acting on? Which observations can reduce uncertainty? Which effects are mechanically permitted? Which independent postconditions prove the intended behaviour for the exact patch?

Build the environment before granting the autonomy. For local, fully specified defects, prefer a simple patch or fixed pipeline. For cross-file tasks where feedback changes the next move, use a bounded agent inside a reproducible worktree with a typed command broker and layered verifier. Where intent or external state is unavailable, stop and route the gap to a human owner.

The durable intuition is simple. Source code is not the whole repository, a green test is not the whole outcome, and an agent transcript is not operating evidence. The repository becomes a useful environment when it can answer interventions with observations strong enough to change the plan and reject the patch.

The evidence conversationChart · sequence

sequenceDiagram
  participant Q as Question
  participant M as Mechanism
  participant E as Evidence
  participant D as Decision
  Q->>M: The patch that passed and still failed
  M->>E: Thought experiment: the frozen archive and the live worktree
  E-->>M: compare rival explanations
  E->>D: Sources
  D-->>Q: revise scope or act

Question, mechanism, evidence and decision exchange the burdens developed in Code Agents and the Repository as an Environment. Reading use: follow the relation, then test it against the surrounding evidence and limits.

Sources

Primary Jimenez, C. E. et al. SWE-bench: Can Language Models Resolve Real-world GitHub Issues? ICLR.
Primary Yang, J. et al. SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering.
Primary Xia, C. S. et al. Agentless: Demystifying LLM-based Software Engineering Agents.
Primary Zhang, L. et al. SWE-bench Goes Live!
Primary Wang, Y., Pradel, M. and Liu, Z. Are “Solved Issues” in SWE-bench Really Solved Correctly? An Empirical Study.
Primary Guan, H. et al. SWE-Cycle: Benchmarking Code Agents across the Complete Issue Resolution Cycle.
Primary Zhang, S. et al. SWE-Explore: Benchmarking How Coding Agents Explore Repositories.
Primary Pham, M. V. T. et al. SWE-EVO: Benchmarking Coding Agents in Long-Horizon Software Evolution Scenarios.
Primary Desai, R. et al. SWE-Marathon: Can Agents Autonomously Complete Ultra-Long-Horizon Software Work?
Primary Kwa, T. et al. Measuring AI Ability to Complete Long Software Tasks, NeurIPS.
Primary preprint Li, J. et al. From Laboratory to Real-World Applications: Benchmarking Agentic Code Reasoning at the Repository Level.
Primary preprint Han, T. et al. SWE-Skills-Bench: Do Agent Skills Actually Help in Real-World Software Engineering?
Primary preprint Xu, Z. et al. SWE-bench Science: Can Coding Agents Resolve Engineering Tasks in Science?
Official audit OpenAI. Why SWE-bench Verified no longer measures frontier coding capabilities.
Official standard NIST. SP 800-218, Secure Software Development Framework.
Official specification SLSA. Build requirements and provenance.
Official documentation GitHub. Secure use reference for GitHub Actions and Risks and mitigations for the Copilot coding agent.