Agents, world models and multi-agent systems

From model to system: scaffolding, memory, tools and agents

Rajesh Ranjan Mahapatra

An apparent agent capability should be attributed to the smallest configured boundary that preserves it under controlled component swaps, because end-to-end behaviour is often created by interactions among weights, runtime, context, orchestration and environment.

© 2026 Rajesh Ranjan Mahapatra. 34 min read

Agents, world models and multi-agent systems

From model to system: scaffolding, memory, tools and agents

An apparent agent capability should be attributed to the smallest configured boundary that preserves it under controlled component swaps, because end-to-end behaviour is often created by interactions among weights, runtime, context, orchestration and environment.

TLDR

  1. An agent score measures a configured system unless the rest of the stack is controlled.
  2. Context is present input; episodic records, knowledge, world state and evidence are different stores.
  3. A component owns a capability only when controlled substitutions support that attribution.
  4. Factorial swaps expose interactions that one-model or one-scaffold comparisons hide.
  5. Ship and compare agents with a versioned configured-system manifest, not a model name.

The model that passed on the left and failed on the right

Imagine two terminals on the same desk. Both call the same model release with the same user request: “Move tomorrow’s service visit to the earliest slot after 3 pm, but do not accept a different engineer.” On the left, the request succeeds. On the right, the assistant apologises, loops twice and proposes an impossible time.

You check the obvious facts. The model identifier is identical. Temperature is 0.2 in both terminals. The user wording is copied byte for byte. Yet the left terminal has a typed scheduling tool, a policy excerpt retrieved into context, an episodic record of the earlier engineer preference, one retry after a recoverable tool error and a final readback from the booking system. The right terminal has a generic calendar API, a flat chat history and no verification step.

Which terminal reveals what the model can do?

The tempting answer is that the successful terminal “unlocked” a capability already inside the weights. That answer may be partly true, but it skips the causal work. The successful outcome required a sequence: interpret the constraint, retain it, map it to a typed action, obtain available slots, reject invalid candidates, commit a change and verify the new world state. Some of that sequence was generated by the model. Some was supplied to it. Some was enforced around it. Some occurred outside it.

The same checkpoint can participate in two systems with different abilities. Conversely, two checkpoints can appear equally capable when a strong harness repairs both of their mistakes. A model name therefore tells us less about an agent than many reports imply.

Exploded specimen of a configured AI agent Five separated layers show model weights at the centre, then runtime, effective context, orchestration, and the external environment with tools and state. Environment tools · permissions · data · world state Orchestration loop · routing · memory policy · validation Effective context instructions · evidence · current observation Runtime Weights trained dispositions decoding + budget visible now process control external effects and authoritative state
An exploded agent specimenThe model is the innermost component, not the whole acting system. An end-to-end property may be preserved by one layer, composed across several, or enabled only by the environment. Illustrative.

The capability is in the configuration

A language model maps an input sequence and inference state to a distribution over possible continuations. An agent system turns some of those continuations into observations, decisions and effects. The difference is not cosmetic. Once a generated string can call a calculator, search a corpus, edit a file or alter a booking, the next input depends on what happened outside the model.

ReAct made this closed loop explicit by interleaving model-generated reasoning with actions and environmental observations.[1] Toolformer showed a different route: train a model to decide when and how to call external APIs.[2] These papers concern different mechanisms, but both break the fiction that a tool-using result is produced by weights alone.

The hidden assumption in many capability claims is simple: if the model emits the decisive token, the capability belongs to the model. That is like crediting a pilot’s engine for a safe landing because the engine supplied the final thrust. The engine matters. So do the control surfaces, instruments, runway, weather information and pilot actions. A component can be causally important without being the correct boundary for the whole claim.

An agent score is a system score until the experiment makes a narrower attribution possible.
Controlled substitutions across an agent stack Three panels compare a reference system, a model-only swap, and a model-plus-harness swap. Solid borders mark fixed components and dotted borders mark changed components. Reference Model-only swap System swap weights A runtime R context C orchestration O environment E 62% success weights B runtime R context C orchestration O environment E 70% success weights B runtime R₂ context C₂ orchestration O₂ environment E 84% success
Change one variable before changing the candidateThe middle comparison supports a conditional model claim. The right comparison supports a complete-system claim. Dotted outlines mark deliberate interventions. Illustrative values.

Closed loops amplify small differences

In a one-shot question, a slightly different token distribution may change only the wording. In an agent trajectory, the same difference can select another tool, alter world state and produce a wholly different next observation. The effect compounds because later inputs are endogenous: the system partly creates the evidence it will next receive.

This makes runtime details matter. A context limit can remove an earlier constraint. A call budget can stop a search before verification. A parser can turn malformed JSON into a valid action or a hard failure. A retry policy can rescue a transient error or repeat a harmful action. A safety gateway can reject a request before the model generates anything. Even identical open weights can diverge across serving engines or precision settings.

A successful answer is the end of a trajectoryNo single stage is equivalent to “the agent did it”. The verified outcome depends on proposal, authority, execution and readback remaining connected. Illustrative configured-system trajectory.
The argument in one pass
Chart · flow
flowchart LR
  A["The model that passed on the<br/>left and failed on the right"] --> B["The capability is in the<br/>configuration"]
  B --> C["Five places behaviour can change"]
  C --> D["Find the minimum preserving<br/>boundary"]
  D -. "re-test" .-> A
  classDef first fill:#0a1424,stroke:#087A98,stroke-width:2px,color:#e6f3ff
  classDef step fill:#0a1424,stroke:#3f7da0,stroke-width:1px,color:#e6f3ff
  class A first
  class B,C,D step
A structural route through From model to system: scaffolding, memory, tools and agents: from The model that passed on the left and… to Credit the narrowest boundary that…. Reading use: follow the relation, then return to the surrounding argument for evidence and limits.

Five places behaviour can change

It is useful to name the layers only after seeing why they matter. Consider the left terminal again. We can alter five things without changing the user’s sentence. Each intervention changes a different causal surface.

Weights: trained dispositions, not a completed agent

The innermost layer is the model package: checkpoint, tokenizer, adapters and any other learned parameters required to reproduce inference. It contains dispositions acquired during training. It may recognise a scheduling constraint, propose a tool call or notice a contradiction. It does not by itself possess a live booking, an authenticated identity or permission to change one.

For a closed API, “the weights” may not be observable. The narrowest reproducible candidate is then a declared model endpoint and version. Claims should follow that evidence. Saying “endpoint X under runtime configuration R” is less dramatic than saying “model X can act”, but it is more testable.

Runtime: how a model release becomes an inference event

The runtime includes the serving engine, numerical precision, decoding method, sampling parameters, context limit, reasoning budget, stop rules and provider-side interface behaviour. These choices can alter output before orchestration begins. A deterministic setting does not make the whole system deterministic because tool responses, concurrency and external state may still vary.

Runtime is not a footnote to weights. It is the machinery that turns a model package into a particular sequence of proposals. In model comparison, it should be fixed or semantically matched. In complete-system comparison, it may be part of the candidate.

Context: what the model can condition on now

Context is the effective input visible to the current inference call. It can include the user request, system instructions, demonstrations, retrieved documents, tool observations, summaries and selected earlier turns. The word “effective” matters because the stored record and the presented context are not identical. An orchestrator may omit, compress, reorder or label stored material before the model sees it.

Context is not memory. An episodic record may exist outside the model. A knowledge base may hold stable reference material. World state may live in an operational system. An evidence ledger may record what was used and what happened. A memory policy decides which parts of those stores enter present context. Calling all five things “memory” hides their authority, lifetime and failure modes.[3]

Context as a lit stage with several distinct stores backstage A central stage labelled effective context is illuminated, while separate backstage stores for episodic record, knowledge, world state and evidence ledger feed it through a selection gate. Effective context visible to this inference call Selection gate Episodicrecord Knowledgebase Worldstate Evidenceledger Current inference can use only what reaches the lit stage
Storage backstage, context on stageA fact may exist in storage yet remain causally absent from the current inference. Attribution requires the retrieval and presentation path, rather than only the existence of a “memory” feature. Illustrative.

Orchestration: how proposals become a process

Orchestration is the control layer around model calls. It decides when to call the model, which context to assemble, whether to plan, how to route tools, what to retain, how to parse outputs, when to retry and how to validate completion. ReAct is one orchestration pattern. A fixed localisation, repair and validation pipeline is another. A search tree, debate loop or multi-agent topology is yet another.

Complexity is not evidence of capability. Agentless used a simple three-phase workflow for software repair and, on its reported SWE-bench Lite evaluation, achieved competitive results without letting the model choose an open-ended sequence of actions.[5] The important lesson is not that fixed workflows always win. It is that an observed gain may come from task decomposition, context control or validation rather than a more “agentic” model.

Orchestration composes behaviour across time. It can expose a latent model skill, compensate for a weakness, introduce a new failure or create a capability that no single call exhibits.

The parser can change the verdict without changing the generationA tool-use score may partly measure output handling. The repair policy is legitimate system engineering, but it must not be silently credited to the weights. Illustrative edge case grounded in documented parser variance.

Environment: what the system can observe and change

The environment contains tools, APIs, permissions, external services, documents, clocks, users and world state. It defines the action repertoire and supplies feedback. Toolformer demonstrated that models can learn when and how to invoke APIs, but the API still performs the external computation.[2] ToolSandbox and τ-bench go further by evaluating stateful tool use, policy following and dynamic interaction rather than isolated function selection.[9][8]

Tools change the system’s action repertoire; they do not retroactively move their computation into the model. A calculator-enabled system can multiply accurately even when the checkpoint cannot. A database-enabled system can return a current balance even when the training data cannot contain it. A permission kernel can prevent an otherwise plausible action.

Failure path caused by an external API contract change A model proposal travels through an unchanged orchestrator to a changed API schema, causing a rejection and misleading model-failure label. A compatibility adapter creates a recovery path. Modelunchanged Orchestratorunchanged API schemafield renamed Compatibilityadapter Verifiedeffect rejection mislabelled “model failure”
An environmental fault can masquerade as a model regressionVersioned tool contracts and effect receipts let investigators locate the break. Without them, outcome-only evaluation attributes failure to the most visible component. Illustrative failure injection.

Find the minimum preserving boundary

Listing layers prevents one kind of confusion, but it does not yet tell us where a property belongs. For that, we need interventions. The central proposal of this paper is a practical test called the minimum preserving boundary.

Start with a property stated as an observable criterion, not a personality word. “Completes a reschedule with the original engineer and a verified booking ID” is testable. “Understands customer needs” is not yet precise enough. Fix a task distribution, evaluator, success threshold and resource budget. Then change components while holding the rest constant.

Mechanism

A boundary must pass two tests

Outside invariance: once the components inside a proposed boundary are fixed, the property survives declared substitutions outside it. Inside sensitivity: at least one admissible substitution inside the boundary changes the property. The first test prevents over-attribution. The second prevents a boundary from becoming so broad that it explains nothing.

The minimum preserving boundary is the smallest component set that passes both tests over the declared condition sweep. “Smallest” means minimal by inclusion, not cheapest or physically smallest. More than one incomparable minimum may exist. That result is informative because it reveals redundancy or alternative routes.

The smallest useful formalism

Represent the configured agent as a five-part tuple:

S = (W, R, C, O, E)

W is the model package. R is the inference runtime. C is effective context. O is orchestration. E is the external environment. Let D be the task distribution, V the evaluator and Y the measured outcome. A run can be written as:

Y = V(run(S, t, ξ)),   t ∼ D

The symbol ξ covers stochastic variation such as sampling, concurrency and simulator randomness. A property holds when expected performance exceeds a declared threshold τ. The threshold might be 0.80 verified success, but it must be justified by the decision, not chosen after seeing results.

For a proposed boundary B, keep its components at their candidate settings and vary compatible components outside it. If the worst tested condition remains above τ, B preserves the property over that sweep. Then test whether swapping something inside B can remove or materially weaken the property. Without that sensitivity, the property may belong to a narrower subset.

What counts as a legitimate substitution?

A substitution is useful only when it changes the intended layer while preserving the question. Replacing one scheduling API with another can test environmental portability if both expose the same action semantics, permissions and authoritative state. Replacing the scheduling API with a text file does not test the same property. It removes the possibility of committing a booking, so failure says nothing specific about the model or orchestrator.

Three checks keep substitutions honest. The first is interface compatibility. Candidate layers should receive equivalent fields, action types and error signals unless the interface itself is under study. The second is resource compatibility. A model given twice the call budget or a harness allowed unlimited retries is not facing the same condition. The third is construct compatibility. The evaluator must still measure the declared property rather than an easier proxy.

Substitutions can also be too gentle. Testing three prompts that differ only in punctuation may create the appearance of portability while leaving the same hidden assumptions intact. A useful sweep should include independently implemented alternatives: a strict and permissive parser, retrieval and summarisation policies, fixed and live environment snapshots, or two orchestrators that share the same action contract but organise control differently.

Interactive benchmarks such as AgentBench and GAIA helped move evaluation beyond static question answering by placing models in environments that require action, browsing or tool use.[7][6] Their broader lesson is not that one score reveals a context-free agent faculty. It is that the apparatus determines which capacities can be expressed and observed. A benchmark name therefore does not substitute for a candidate manifest.

The property itself must remain at the same grain. “Produces a valid tool-call object” may have a narrow boundary around model, runtime and schema presentation. “Reschedules a visit correctly” additionally requires policy, authority, live availability, execution and readback. Both are legitimate properties, but the second cannot inherit the first property’s boundary merely because the tool call appears inside the trajectory.

This is why there is no universal location of capability. The same configured system may show weight-level portability for grammatical correction, context-conditioned performance for policy questions and system-composed performance for state-changing work. The boundary follows the property being tested. Broad labels such as “planning”, “memory” or “agency” should be decomposed until an intervention can change one causal demand at a time.

A lattice of candidate boundaries for capability attribution Configurations are arranged from single components at the bottom to the complete system at the top. A highlighted path shows a minimal preserving boundary containing weights, orchestration and environment. Complete systemW R C O E W O Epreserves W C Opreserves R C O Epreserves W OW EC OO ER OC E WRCOE No single layer preserved the property in this condition sweep
Attribution is a search over boundaries, not a label attached in advanceThe highlighted example has several minimal preserving sets. That makes the property distributed or redundantly supported under the tested design. Illustrative lattice.

Why one-factor swaps are not enough

A component may help only when paired with another. Better weights might exploit a structured tool schema that weaker weights ignore. A retrieval policy might help a model that follows source labels but confuse one that does not. These are interactions, and they make causal ownership relational.

Interaction chart for two models and two orchestrators Two lines show success rates for models W0 and W1 under orchestrators O0 and O1. The lines diverge, revealing a positive interaction. 40%55%70%85%100% O₀ · simple loopO₁ · guarded loop 45%65% 60%90% W₁ benefits disproportionately from O₁ W₁ candidate weightsW₀ baseline weights
Diverging lines reveal a model–orchestration interactionThe model ranking does not reverse here, but the size of the model advantage depends on the orchestrator. A single score hides that dependence. Illustrative synthetic values.
Executable lab · browser

Try the 2 × 2 attribution calculation

Enter four verified success rates. The lab computes descriptive main effects and the difference-in-differences interaction. It does not calculate statistical significance. Use repeated matched tasks before treating small differences as evidence.

Model effect: +20.0 points. Orchestration effect: +25.0 points. Interaction: +10.0 points.

Move the boundary outward only when the evidence forces you

Begin with the narrow claim you hope to support. If you want a model claim, hold runtime, context construction, orchestration, environment, evaluator and budget fixed. Repeat across several reference harnesses to test portability. If the effect vanishes or reverses, lower the claim to a model–harness configuration.

If you want to choose a deployed agent, compare complete systems under common external rules. Let each system own its internal prompts, memory policy and parser, but standardise the task set, permissions, evaluator and resource accounting. If you want a robustness claim, fix the candidate and sweep declared conditions such as context pressure, tool latency or environment snapshots.

A 2026 position paper makes this candidate-boundary distinction explicit: model comparisons under a shared execution stack, complete-system comparisons and robustness studies license different conclusions.[11] The minimum preserving boundary adds a diagnostic step. It asks how far the boundary must expand before the observed property survives controlled substitutions.

How the claim changes under pressure
Chart · state
stateDiagram-v2
  [*] --> Claim
  state "Swap the core, keep the apparatus" as Claim
  state "Closed loops amplify small<br/>differences" as Test
  state "The two-line time-zone assistant" as Revision
  Claim --> Test: expose mechanism
  Test --> Decision: evidence survives
  Test --> Revision: rival survives
  Revision --> Claim: narrow or rebuild
  Decision --> [*]
The revision loop begins with Swap the core, keep the apparatus and keeps counterexamples inside the reasoning path. Reading use: follow the relation, then return to the surrounding argument for evidence and limits.

Build the configuration you can name

An attribution experiment fails before the first run when the tested system is not specified. “Model B with tools” is not a configuration. Which endpoint? Which decoding budget? Which system instructions? Which retrieval snapshot? Which parser? Which permissions? Which tool versions? Which evaluator? If any of these can change unnoticed, later investigators cannot reproduce the candidate or locate a regression.

A configured-system manifest is an experimental instrument. It gives every run a stable identity and marks which components are inside the candidate boundary. It should record identifiers, versions, hashes where possible, interfaces, budgets, state cut-offs and ownership. It should also keep the evaluator outside the candidate unless evaluation is itself the object of study.

The configured-system manifest is a causal mapA version list is necessary but insufficient. Each field should answer how an observed difference could have entered the run. Practitioner synthesis.

A protocol that produces licensed claims

Use the manifest with a staged intervention plan.

  1. Operationalise the property. Define verified success, prohibited shortcuts, resource limits and the task distribution before running candidates.
  2. Declare the candidate boundary. Mark every component as fixed, candidate-owned or deliberately varied. Unmarked variation is a confound.
  3. Run matched replacements. Swap one layer at a time against shared tasks and environment snapshots. Repeat stochastic runs.
  4. Test interactions. Use at least a small factorial design for component pairs likely to cooperate, such as weights × orchestration or context × environment.
  5. Search for the preserving boundary. Ask which candidate component sets remain above threshold across variations outside them.
  6. Write only the licensed conclusion. Name the candidate, conditions, evaluator, uncertainty and components not isolated.

Positive results permit a conditional attribution over the tested substitutions. A negative result is not a dead end. It tells you to expand the boundary, narrow the property or improve the system. A missing cell, drifting environment or unversioned evaluator means the attribution is not established.

Executable lab · Python 3.11+

Configured-system attribution lab

The script validates a manifest, reads repeated outcomes from a full two-level factorial design, estimates descriptive main effects and pairwise interactions, then searches for minimum preserving boundaries. It uses only the Python standard library and can generate a synthetic demonstration.

Shell · run the synthetic demonstration
python configured_system_attribution_lab.py --demo ./paper58_demo

A positive boundary result licenses attribution only within the manifest, task set and tested condition sweep. No boundary result means the property was not preserved, the threshold was too demanding, or the design omitted a needed configuration.

Open the complete runnable Python artefact
Python · configured_system_attribution_lab.py
#!/usr/bin/env python3
"""Configured-system attribution lab.

Uses a full two-level factorial design over five agent-system layers to estimate
main effects, pairwise interactions and minimum preserving boundaries.
Only Python's standard library is required.
"""
from __future__ import annotations

import argparse
import csv
import itertools
import json
import math
import random
import statistics
import sys
from pathlib import Path
from typing import Any, Iterable

LAYERS = ("weights", "runtime", "context", "orchestration", "environment")
REQUIRED_FIELDS: dict[str, tuple[str, ...]] = {
    "weights": ("id", "checkpoint", "tokenizer"),
    "runtime": ("id", "engine", "decoding", "budget"),
    "context": ("id", "assembly_policy", "sources", "cutoff"),
    "orchestration": (
        "id",
        "loop",
        "memory_policy",
        "parser",
        "retry_policy",
        "validator",
    ),
    "environment": ("id", "tools", "permissions", "state_snapshot"),
    "evaluator": ("id", "task_set", "metric", "threshold"),
}


def validate_manifest(manifest: dict[str, Any]) -> None:
    missing_sections = [section for section in REQUIRED_FIELDS if section not in manifest]
    if missing_sections:
        raise ValueError(f"Manifest missing sections: {', '.join(missing_sections)}")
    for section, fields in REQUIRED_FIELDS.items():
        value = manifest[section]
        if not isinstance(value, dict):
            raise ValueError(f"Manifest section {section!r} must be an object")
        missing = [field for field in fields if field not in value]
        if missing:
            raise ValueError(f"Manifest section {section!r} missing: {', '.join(missing)}")
    threshold = manifest["evaluator"]["threshold"]
    if not isinstance(threshold, (int, float)) or not 0 <= threshold <= 1:
        raise ValueError("evaluator.threshold must be a number between 0 and 1")


def read_runs(path: Path) -> list[dict[str, Any]]:
    rows: list[dict[str, Any]] = []
    with path.open(newline="", encoding="utf-8") as handle:
        reader = csv.DictReader(handle)
        required = {"task", "repeat", "success", *LAYERS}
        if reader.fieldnames is None or not required.issubset(reader.fieldnames):
            absent = sorted(required.difference(reader.fieldnames or []))
            raise ValueError(f"Run CSV missing columns: {', '.join(absent)}")
        for line_no, row in enumerate(reader, start=2):
            parsed: dict[str, Any] = {"task": row["task"], "repeat": int(row["repeat"])}
            for layer in LAYERS:
                value = int(row[layer])
                if value not in (0, 1):
                    raise ValueError(f"Line {line_no}: {layer} must be 0 or 1")
                parsed[layer] = value
            success = float(row["success"])
            if not 0 <= success <= 1:
                raise ValueError(f"Line {line_no}: success must be between 0 and 1")
            parsed["success"] = success
            parsed["cost"] = float(row.get("cost", 0) or 0)
            rows.append(parsed)
    if not rows:
        raise ValueError("Run CSV contains no observations")
    return rows


def configuration(row: dict[str, Any]) -> tuple[int, ...]:
    return tuple(int(row[layer]) for layer in LAYERS)


def aggregate(rows: Iterable[dict[str, Any]]) -> dict[tuple[int, ...], float]:
    buckets: dict[tuple[int, ...], list[float]] = {}
    for row in rows:
        buckets.setdefault(configuration(row), []).append(float(row["success"]))
    return {key: statistics.fmean(values) for key, values in buckets.items()}


def require_complete_factorial(means: dict[tuple[int, ...], float]) -> None:
    expected = set(itertools.product((0, 1), repeat=len(LAYERS)))
    missing = sorted(expected.difference(means))
    if missing:
        readable = ["".join(map(str, config)) for config in missing[:8]]
        suffix = " …" if len(missing) > 8 else ""
        raise ValueError(
            "A complete 2^5 factorial is required. Missing configurations: "
            + ", ".join(readable)
            + suffix
        )


def marginal_effect(means: dict[tuple[int, ...], float], index: int) -> float:
    high = [value for config, value in means.items() if config[index] == 1]
    low = [value for config, value in means.items() if config[index] == 0]
    return statistics.fmean(high) - statistics.fmean(low)


def pair_interaction(
    means: dict[tuple[int, ...], float], first: int, second: int
) -> float:
    others = [index for index in range(len(LAYERS)) if index not in (first, second)]
    differences: list[float] = []
    for other_values in itertools.product((0, 1), repeat=len(others)):
        base = [0] * len(LAYERS)
        for index, value in zip(others, other_values):
            base[index] = value

        def cell(a: int, b: int) -> float:
            config = base.copy()
            config[first] = a
            config[second] = b
            return means[tuple(config)]

        differences.append(cell(1, 1) - cell(1, 0) - cell(0, 1) + cell(0, 0))
    return statistics.fmean(differences)


def minimum_preserving_boundaries(
    means: dict[tuple[int, ...], float], threshold: float
) -> list[tuple[tuple[str, ...], float]]:
    """Find minimal candidate-layer sets that preserve success above threshold.

    A boundary B is preserving when every observed configuration with all layers
    in B set to candidate level 1 remains above threshold, regardless of the
    settings outside B. Minimality is by set inclusion.
    """
    preserving: list[tuple[frozenset[int], float]] = []
    for size in range(1, len(LAYERS) + 1):
        for indices in itertools.combinations(range(len(LAYERS)), size):
            index_set = frozenset(indices)
            if any(previous.issubset(index_set) for previous, _ in preserving):
                continue
            eligible = [
                value
                for config, value in means.items()
                if all(config[index] == 1 for index in index_set)
            ]
            worst_case = min(eligible)
            if worst_case >= threshold:
                preserving.append((index_set, worst_case))
    return [
        (tuple(LAYERS[index] for index in sorted(indices)), worst)
        for indices, worst in preserving
    ]


def licensed_claim(boundaries: list[tuple[tuple[str, ...], float]]) -> str:
    if not boundaries:
        return (
            "No tested boundary preserved the property. Lower the claim, improve the "
            "system, or expand the condition set before attributing capability."
        )
    names = [set(boundary) for boundary, _ in boundaries]
    if {"weights"} in names:
        return (
            "The property is model-portable across the tested runtime, context, "
            "orchestration and environment variants. This remains a conditional claim."
        )
    if all(len(name) == 1 for name in names):
        labels = ", ".join(sorted(next(iter(name)) for name in names))
        return f"Several single-layer boundaries preserve the property: {labels}."
    if any("weights" in name and len(name) > 1 for name in names):
        return (
            "The property is interaction-owned under the tested design. Attribute it "
            "to the listed model–system configuration, not to the weights alone."
        )
    return (
        "The property is system-composed under the tested design. Attribute it to the "
        "minimum preserving configuration and retain the tested condition scope."
    )


def analyse(manifest_path: Path, runs_path: Path) -> None:
    manifest = json.loads(manifest_path.read_text(encoding="utf-8"))
    validate_manifest(manifest)
    rows = read_runs(runs_path)
    means = aggregate(rows)
    require_complete_factorial(means)
    threshold = float(manifest["evaluator"]["threshold"])

    print("Configured-system attribution report")
    print("=" * 38)
    print(f"Observations: {len(rows)}")
    print(f"Factorial cells: {len(means)}")
    print(f"Property threshold: {threshold:.1%}\n")

    effects = [(layer, marginal_effect(means, index)) for index, layer in enumerate(LAYERS)]
    print("Descriptive marginal effects")
    for layer, effect in sorted(effects, key=lambda item: abs(item[1]), reverse=True):
        print(f"  {layer:14s} {effect:+.1%}")

    interactions: list[tuple[str, str, float]] = []
    for first, second in itertools.combinations(range(len(LAYERS)), 2):
        interactions.append(
            (LAYERS[first], LAYERS[second], pair_interaction(means, first, second))
        )
    print("\nLargest pairwise interactions")
    for first, second, effect in sorted(
        interactions, key=lambda item: abs(item[2]), reverse=True
    )[:5]:
        print(f"  {first:14s} × {second:14s} {effect:+.1%}")

    boundaries = minimum_preserving_boundaries(means, threshold)
    print("\nMinimum preserving boundaries")
    if not boundaries:
        print("  none")
    else:
        for boundary, worst in boundaries:
            print(f"  {', '.join(boundary):48s} worst observed cell {worst:.1%}")
    print("\nLicensed conclusion")
    print("  " + licensed_claim(boundaries))
    print(
        "\nLimit: this report describes the supplied task distribution, versions and "
        "condition sweep. It does not establish untested robustness, safety, general "
        "intelligence or consciousness."
    )


def demo(output_dir: Path, seed: int = 58) -> tuple[Path, Path]:
    output_dir.mkdir(parents=True, exist_ok=True)
    manifest = {
        "weights": {
            "id": "weights-candidate-v1",
            "checkpoint": "synthetic/model-candidate",
            "tokenizer": "synthetic-tokenizer-v1",
        },
        "runtime": {
            "id": "runtime-candidate-v1",
            "engine": "reference-engine",
            "decoding": {"temperature": 0.2, "top_p": 1.0},
            "budget": {"max_tokens": 2048, "max_calls": 8},
        },
        "context": {
            "id": "context-candidate-v1",
            "assembly_policy": "typed-fields-with-source-labels",
            "sources": ["task", "policy", "current-observation"],
            "cutoff": "snapshot-001",
        },
        "orchestration": {
            "id": "orchestration-candidate-v1",
            "loop": "propose-act-observe-verify",
            "memory_policy": "retain-decisions-and-effect-receipts",
            "parser": "strict-typed-action",
            "retry_policy": {"max_retries": 1, "same_error_retry": False},
            "validator": "readback-required",
        },
        "environment": {
            "id": "environment-candidate-v1",
            "tools": ["lookup", "calculate", "commit", "readback"],
            "permissions": "least-privilege-demo",
            "state_snapshot": "synthetic-world-v1",
        },
        "evaluator": {
            "id": "evaluator-v1",
            "task_set": "synthetic-configured-agent-tasks",
            "metric": "verified_task_success",
            "threshold": 0.70,
        },
    }
    manifest_path = output_dir / "demo_manifest.json"
    manifest_path.write_text(json.dumps(manifest, indent=2), encoding="utf-8")

    rng = random.Random(seed)
    runs_path = output_dir / "demo_runs.csv"
    with runs_path.open("w", newline="", encoding="utf-8") as handle:
        fieldnames = ["task", "repeat", *LAYERS, "success", "cost"]
        writer = csv.DictWriter(handle, fieldnames=fieldnames)
        writer.writeheader()
        for config in itertools.product((0, 1), repeat=len(LAYERS)):
            w, r, c, o, e = config
            # Synthetic data: orchestration and environment matter most, while the
            # weights–orchestration interaction makes the best system more than the
            # sum of two isolated improvements.
            logit = (
                -1.25
                + 0.50 * w
                + 0.20 * r
                + 0.42 * c
                + 0.76 * o
                + 0.68 * e
                + 0.55 * w * o
                + 0.30 * c * e
            )
            probability = 1 / (1 + math.exp(-logit))
            for task in range(1, 61):
                for repeat in range(1, 3):
                    writer.writerow(
                        {
                            "task": f"task-{task:03d}",
                            "repeat": repeat,
                            **{layer: value for layer, value in zip(LAYERS, config)},
                            "success": int(rng.random() < probability),
                            "cost": f"{0.03 + 0.01*r + 0.02*o + 0.01*e:.2f}",
                        }
                    )
    return manifest_path, runs_path


def build_parser() -> argparse.ArgumentParser:
    parser = argparse.ArgumentParser(description=__doc__)
    parser.add_argument("--manifest", type=Path, help="Configured-system manifest JSON")
    parser.add_argument("--runs", type=Path, help="Factorial run CSV")
    parser.add_argument(
        "--demo",
        type=Path,
        metavar="OUTPUT_DIR",
        help="Generate and analyse a reproducible synthetic demonstration",
    )
    return parser


def main() -> int:
    args = build_parser().parse_args()
    try:
        if args.demo:
            manifest, runs = demo(args.demo)
            print(f"Created {manifest}")
            print(f"Created {runs}\n")
            analyse(manifest, runs)
            return 0
        if not args.manifest or not args.runs:
            raise ValueError("Use --demo OUTPUT_DIR or provide both --manifest and --runs")
        analyse(args.manifest, args.runs)
        return 0
    except (OSError, ValueError, json.JSONDecodeError) as exc:
        print(f"error: {exc}", file=sys.stderr)
        return 2


if __name__ == "__main__":
    raise SystemExit(main())

What the artefact assumes

Each of the five layers has a baseline level and a candidate level. Every one of the 32 combinations is observed. The default boundary rule uses the worst observed cell outside a proposed boundary, so one hidden failure prevents a portability claim. The script reports descriptive effects, not causal certainty by itself. Randomisation, matched tasks, confidence intervals and failure inspection still matter.

What it cannot establish

The lab cannot prove safety, general intelligence, subjective experience or robustness outside the supplied condition set. It cannot rescue a bad evaluator or a contaminated task set. It also cannot distinguish two mechanisms that produce the same traces unless the experiment intervenes on their difference.

Optional depth: a compact experimental checklist
Before runsDuring runsAfter runs
Freeze task IDs, evaluator and threshold.Log inputs, outputs, tool calls, observations and versions.Inspect missing cells and environment mismatches.
Declare fixed, owned and varied components.Use paired tasks and repeated stochastic trials.Estimate main effects, interactions and uncertainty.
Specify admissible substitutions and budgets.Record effect receipts and final readback.State the minimum preserving boundary and prohibited inferences.
The evidence conversation
Chart · sequence
sequenceDiagram
  participant Q as Question
  participant M as Mechanism
  participant E as Evidence
  participant D as Decision
  Q->>M: The model that passed on the left and failed on the right
  M->>E: Swap the core, keep the apparatus
  E-->>M: compare rival explanations
  E->>D: Credit the narrowest boundary that survives substitution
  D-->>Q: revise scope or act
Question, mechanism, evidence and decision exchange the burdens developed in From model to system: scaffolding, memory, tools and agents. Reading use: follow the relation, then return to the surrounding argument for evidence and limits.

Credit the narrowest boundary that survives substitution

When a demonstration succeeds, stop asking only, “Which model did this?” Ask, “What is the smallest configured boundary that preserves this result, and what did the experiment hold fixed?” That question changes model selection, system design, incident review and public claims.

For model selection, compare checkpoints through a shared reference stack and repeat the comparison across more than one credible harness. For product selection, compare complete systems under common external rules. For release decisions, keep a configured-system manifest and rerun anchor tasks whenever any causal layer changes. For incident diagnosis, use trajectory and effect receipts before blaming the most visible component.

DecisionCandidate boundaryWhat must be controlledLicensed wording
Choose a checkpointModel releaseRuntime, context, orchestration, environment, evaluator and budget“Model B outperformed A under the declared reference configurations.”
Choose a productComplete agent systemTasks, permissions, external rules, evaluator and resource accounting“System B achieved higher verified success under common external rules.”
Claim robustnessFixed model or fixed systemPredeclared condition sweep and versioned evaluator“The candidate remained above threshold across the tested conditions.”
Diagnose a regressionLast known good configurationOne-layer swaps, anchor tasks, traces and effect receipts“The change is associated with layer X under matched replay.”
Let the decision determine the candidate boundaryModel, system and robustness claims are all useful, but their controls and wording differ. A common benchmark name does not make them interchangeable. Decision instrument synthesised from the paper’s attribution method.
Decision rule

Name the system at the boundary your evidence supports

Do not promote a system-level success into a model-level capability claim. Do not demote a real harness innovation because it is “only scaffolding”. Credit the narrowest boundary that survives intervention, preserve interactions when they matter and attach every conclusion to its tested conditions.

The practical consequence is modest but far-reaching. A model is no longer the default unit of explanation merely because it is the most expensive component. The unit of explanation is earned by substitution. Once teams adopt that discipline, benchmark gains become easier to interpret, regressions become easier to locate and agent designs become easier to reproduce.

The next time two identical checkpoints behave differently, the discrepancy is not noise to explain away. It is the beginning of the causal investigation.

Glossary

Agent system
An executable closed loop in which model proposals can guide actions, receive observations and influence later inputs.
Candidate boundary
The components treated as the entity being compared. Everything outside should be fixed, matched or deliberately varied.
Configured-system manifest
A versioned record of model, runtime, context, orchestration, environment and evaluator settings for a run.
Effective context
The information presented to the model for a particular inference call, after selection, formatting and truncation.
Effect receipt
A record from an authoritative system that an action changed, or failed to change, world state.
Minimum preserving boundary
The smallest tested component set that keeps a declared property above threshold across substitutions outside that set and remains sensitive to changes inside it.
Orchestration
The control logic for context assembly, model calls, tool routing, episodic retention, parsing, retries, validation and completion.
Runtime
The inference engine, numerical and decoding settings, context and call budgets, stop conditions and provider interface that realise a model invocation.
World state
Authoritative external facts that can change through action, such as a booking, balance, file or device state.

References

  1. Yao, S., Zhao, J., Yu, D., Du, N., Shafran, I., Narasimhan, K. and Cao, Y. (2023). “ReAct: Synergizing Reasoning and Acting in Language Models.” ICLR. arXiv camera-ready record.
  2. Schick, T., Dwivedi-Yu, J., Dessì, R., Raileanu, R., Lomeli, M., Hambro, E., Zettlemoyer, L., Cancedda, N. and Scialom, T. (2023). “Toolformer: Language Models Can Teach Themselves to Use Tools.” NeurIPS. NeurIPS paper.
  3. Packer, C., Wooders, S., Lin, K., Fang, V., Patil, S. G., Stoica, I. and Gonzalez, J. E. (2023). “MemGPT: Towards LLMs as Operating Systems.” arXiv:2310.08560.
  4. Yang, J., Jimenez, C. E., Wettig, A., Lieret, K., Yao, S., Narasimhan, K. and Press, O. (2024). “SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering.” NeurIPS. Conference paper.
  5. Xia, C. S., Deng, Y., Dunn, S. and Zhang, L. (2024). “Agentless: Demystifying LLM-based Software Engineering Agents.” arXiv:2407.01489.
  6. Mialon, G., Fourrier, C., Wolf, T., LeCun, Y. and Scialom, T. (2024). “GAIA: a benchmark for General AI Assistants.” ICLR. Conference paper.
  7. Liu, X. et al. (2024). “AgentBench: Evaluating LLMs as Agents.” ICLR. ICLR paper.
  8. Yao, S., Shinn, N., Razavi, P. and Narasimhan, K. (2024). “τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains.” arXiv:2406.12045.
  9. Lu, J. et al. (2025). “ToolSandbox: A Stateful, Conversational, Interactive Evaluation Benchmark for LLM Tool Use Capabilities.” Findings of NAACL. ACL Anthology paper.
  10. Ben Sghaier, O., Li, H., Adams, B. and Hassan, A. E. (2026). “Don’t Blame the Large Language Model: How Agent Harness Evolution Shapes Coding Agent Quality.” Preprint. arXiv:2607.03691.
  11. Zhu, P., Sun, L., Yu, P. S. and Su, S. (2026). “‘LLM Agent Performance’ Is Not a Single Evaluation Target.” Preprint. arXiv:2602.03238.
  12. Huang, Y. et al. (2026). “MemoHarness: Agent Harnesses That Learn from Experience.” Preprint. arXiv:2607.14159.