Writing the first defensible contribution

A result is not yet a contribution

The familiar publication story is linear. Ask a question, collect data, analyse it, obtain a result and write the paper. Peer review checks the reasoning, readers inspect the figures, and later researchers reproduce or refute the finding. The hidden assumption is that the finished paper still contains the path by which the result became a claim.

Usually it does not. A paper compresses months of choices into a few pages. Failed pilots disappear. Outcomes acquire cleaner names. Data exclusions become one sentence. An analysis selected after seeing the data may be narrated as though it were inevitable. A software environment is reduced to a package name. The prose may be honest while still erasing the alternatives that determine what its result means.

This matters because scientific trustworthiness is not the same property as correctness. A recent framework led by Brian Nosek treats trustworthy findings as products of a research system that makes claims accountable, evaluable and calibrated to evidence.^[1] A trustworthy result can later be overturned. The crucial feature is that errors are discoverable and correctable. That distinction gives us the right target. A defensible contribution is not a claim protected from criticism; it is a claim engineered so criticism can reach the right joints.

The claim-to-evidence publication spine. A contribution is defensible when construction works forwards and challenge works backwards. A polished claim with no reverse path is a detached endpoint, not a scientific spine. Practitioner synthesis; conceptual figure.

The challenge surface

A challenge surface is the set of locations where another person can inspect, rerun, substitute or bound an inference. It includes the primary outcome definition, the intervention contrast, the exclusion rule, the version of the data, the software environment, the uncertainty calculation, the rival explanation and the sentence that states what the result permits.

More surface is not automatically better. Dumping every file from a project can obscure the decisive chain. A challenge surface must be addressable. Each item needs a stable identifier; each claim needs to point to the few items that matter; and each artefact needs enough provenance to show how it entered the study. This is why a repository link is weaker than a manifest, and why a checklist is weaker than a graph of dependencies.

Thought experiment 1

The sealed twin

Imagine two sealed boxes that display the same estimate, confidence interval and prose conclusion. Box A contains raw data, code and a protocol, but none is accessible. Box B contains the same materials behind labelled inspection ports. You may replace the estimator, rerun exclusions, verify timestamps and test a rival explanation without opening the whole apparatus.

Now change one causal property: make both boxes produce the wrong answer because the outcome was measured badly. Their correctness is identical. Their scientific value is not. Box B lets the measurement error become a public discovery rather than private suspicion. Defensibility is therefore a property of error-correcting access, not present truth.

Same headline, different challenge surfaces. Openness becomes scientifically useful only when a critic can address the dependencies of the claim rather than search an undifferentiated archive. Illustrative.

Computational reproducibility is one important inspection port. The National Academies defines it as obtaining consistent results from the same input data, computational steps, methods, code and analysis conditions, while reserving replicability for new studies using new data to answer the same question.^[6] The distinction matters. A perfectly reproducible pipeline can repeatedly compute a biased estimate. A successful replication can support a phenomenon while leaving the proposed mechanism unresolved.

Verification, validity and replication ask further questions. Verification asks whether the reported objects and operations can be checked. Validity asks whether the design and measures support the intended inference. Replication asks whether a new study finds a compatible result under a relevant repetition. The updated TOP Guidelines organise practices around the verifiability of empirical claims rather than treating “open science” as one undifferentiated virtue.^[2] This is useful because a contribution can fail at one layer while passing another. The code may execute, the result may reproduce and the measure may still miss the construct. Conversely, a field observation may have limited computational reproducibility yet remain defensible if source selection, provenance, judgement and counterinterpretations are exposed. The release claim should name which layer has actually been tested.

Nor is public availability enough. FAIR principles make digital research objects findable, accessible, interoperable and reusable, with an emphasis on machine actionability.^[7] Those properties help a challenge travel, but they do not specify which object bears on which claim. The publication spine adds inferential addressability to availability.

The argument in one pass

Chart · flow

flowchart TD
  A["Opening case"] --> B["A result is not yet a contribution"]
  B --> C["Build the claim-to-evidence spine"]
  C --> D["Preserve alternative histories<br/>and nulls"]
  D -. "re-test" .-> A
  classDef first fill:#0a1424,stroke:#B96D00,stroke-width:2px,color:#e6f3ff
  classDef step fill:#0a1424,stroke:#3f7da0,stroke-width:1px,color:#e6f3ff
  class A first
  class B,C,D step

A structural route through Writing the first defensible contribution: from Opening case to Release the research object only when a…. Reading use: follow the relation, then return to the surrounding argument for evidence and limits.

Build the claim-to-evidence spine

Begin with the sentence that would appear in the abstract. Not the topic, ambition or hoped-for significance. Write the narrow causal or comparative claim that the study might earn. Then give it a type.

A descriptive claim says what was observed in a declared sample. A comparative claim says how defined conditions differed. A causal claim says an intervention changed an outcome under stated assumptions. A mechanistic claim says the change travelled through a particular process. An equivalence claim says effects beyond a prespecified practical bound were ruled out. A generalisation claim says the relation survived a declared shift in population, task or environment. These are not stylistic labels. Each creates different evidence obligations.

For example, “state swapping reduced recovery” is comparative. “The persistent state was causally used for recovery” requires a valid intervention and control. “The state stores an interruption model that guides recovery” is mechanistic and requires evidence that separates storage from generic disruption. “Persistent state is necessary for agents to recover” is a general claim far beyond one configured system.

Claim granularity is equally important. One paper can contain supporting claims about manipulation success, measurement reliability and boundary behaviour, but they should converge on one central inference. When two claims could each survive the failure of the other, the study may contain two contributions and two different challenge surfaces. Splitting them is more than editorial tidiness. It prevents evidence for an easy descriptive result from lending borrowed confidence to a harder mechanistic one. A useful test is to remove the central claim and ask whether a major section still has an independent conclusion. If it does, either demote that conclusion to a declared auxiliary result or give it its own study. A defensible paper is narrow enough that one broken spine changes the whole argument.

Give every inferential object an address

The smallest useful spine has two record types. A claim record states what may be said. An evidence record states what was observed or computed. The relation between them is explicit, many-to-many and versioned.

Typed pseudocode · minimal claim contract

type Claim = {
  claim_id: string
  wording: string
  claim_type: "descriptive" | "comparative" | "causal" |
              "mechanistic" | "equivalence" | "generalisation"
  population_or_system: string
  intervention?: string
  comparator?: string
  outcome: string
  estimand: string
  uncertainty_rule: string
  evidence_ids: string[]
  assumptions: string[]
  boundary: string[]
  prohibited_inferences: string[]
}

type EvidenceReceipt = {
  evidence_id: string
  source_object: string
  immutable_id: string
  transformation: string
  code_version: string
  environment_id: string
  output_location: string
  produced_by: string
}

The exact field names can change. The invariant cannot: a claim must not depend on an unnamed object or transformation. The receipt is not ceremonial metadata. It lets a critic start from Figure 4, find the generating output, find the script and environment, locate the frozen input and then inspect the predeclared reason that this figure bears on the claim.

Thought experiment 2

The output-preserving replacement

Keep the final chart fixed. Replace the analysis underneath it. In the original study, an intervention assignment was random and the estimate was a prespecified difference in means. In the replacement, investigators remove inconvenient cases after viewing outcomes and tune a threshold until the same estimate appears.

A PDF-level reader sees no change. A spine-level reader sees a different causal history. The result is numerically preserved while its warrant collapses. Now reverse the experiment: replace the software implementation with an independently written one that consumes the same frozen data and recovers the estimate. The number again stays fixed, but the evidence becomes stronger because a correlated implementation error is less likely. What matters is not output identity alone, but which dependencies survived replacement.

Worked example 1 · toy scale

From forty interruption trials to one permitted sentence

Suppose a configured agent is tested on forty predeclared interruption episodes. Under a sham state swap it recovers correctly in 30 cases. Under a real cross-episode state swap it recovers in 18. The observed risk difference is:

18/40 − 30/40 = −0.30.

The physical meaning is simple: in this sample, correct recovery was thirty percentage points less frequent after the real swap. The calculation assumes that “correct recovery” was fixed before outcomes, that the same episode distribution was used, that assignments were valid and that repeated trials can be treated with the declared dependence model.

The number permits a descriptive comparison. It does not by itself show that persistent state carries a recovery model. A real swap may corrupt unrelated computations. Add a transcript-only rival and a sham swap, then estimate whether the real-versus-sham difference is specifically larger in the stateful candidate. Only that interaction begins to discriminate causal use of persistent state from generic perturbation.

The reverse traversal test

Once the paper is drafted, start from each load-bearing sentence and walk backwards. Can you identify the figure or table? Can you identify the exact output object behind it? Can you identify the code version and inputs? Can you identify the protocol clause that made this analysis primary? Can you identify the rival explanation the test was designed to separate? Can you identify the boundary that stops the claim?

If the walk terminates in “the authors judged”, the judgement may be legitimate, but it must be named as judgement. If it terminates in a file whose version is unknown, the evidence is unstable. If it terminates in a result selected from many unreported alternatives, the inferential status is exploratory even when the analysis itself is sound.

Published evidence

The Many Analysts project gave 29 teams the same data and question. Their estimated odds ratios ranged from 0.89 to 2.93; 20 teams reported a statistically significant positive effect and nine did not. The teams used 21 distinct covariate combinations.^[5] The study does not show that analysis is arbitrary. It shows that reasonable analytic choices can become load-bearing and therefore need to be visible.

A spine also clarifies authorship and responsibility. One person may design the intervention, another prepare the data, another implement the estimator and another interpret the result. Provenance standards such as PROV-O represent entities, activities and agents and the relations among them.^[8] That grammar can capture who produced what. The publication spine adds why the object matters to the inference.

Preserve alternative histories and nulls

A finished analysis can look inevitable because the paths not taken have vanished. Defensibility requires a record of which choices were committed before outcomes, which were changed after contact with data and which alternatives remain plausible. The purpose is not to punish adaptation. It is to preserve the difference between prediction and learning.

Preregistration records research questions and analysis plans before the relevant outcomes are observed, helping separate prediction from postdiction.^[3] Registered Reports move part of peer review earlier: journals review the question and method before results are known and may offer in-principle acceptance.^[4] Neither device guarantees a good measure, a valid intervention, faithful execution or adequate code. Registration fixes the state of intention; it does not certify the quality of that state.

A deviation log then preserves what happened when the world refused to follow the plan. Useful entries state the triggering observation, the decision, who made it, when outcomes were available, which artefacts changed and which claims were affected. Some deviations repair validity: a tool API changed, a sensor failed or duplicate episodes were discovered. Others create a new analysis: an unplanned subgroup appears interesting or a threshold is altered after inspecting the distribution. Both may be reasonable. They differ in the uncertainty and wording they inherit. A log should therefore record consequence, rather than only confession. “Changed exclusion rule” is weak. “Changed exclusion rule after outcome access; primary estimate retained; revised estimate labelled sensitivity; claim wording unchanged because both remain inside the prespecified band” is actionable.

Commitment timeline. The data-access boundary changes the epistemic status of later choices. Adaptation remains legitimate when it is logged, justified and labelled rather than rewritten as prior intent. Practitioner synthesis.

The alternative-history budget

Every study permits a family of plausible histories. Which observations count? Which covariates enter? Which model, threshold, seed or stopping rule is used? Which subgroup receives attention? Which failure is treated as noise? A paper spends an alternative-history budget whenever an unrecorded choice could have produced a materially different claim.

The objective is not to enumerate every imaginable analysis. It is to retain the alternatives that a competent critic would consider plausible before seeing which one supports the desired story. These can be handled through a multiverse, specification curve, robustness set, held-out confirmation, independent analyst or explicit sensitivity analysis. The important move is to turn “the analysis” from a singular noun into a declared region of choices.

Thought experiment 3

The fork cabinet

Place twenty switches in a cabinet. Each chooses one defensible analysis option: include or exclude ambiguous trials, use a strict or lenient success threshold, analyse all episodes or only completed ones, use one of two dependence models, and so on. The investigator may inspect all switch combinations but publishes only the combination with the smallest p-value.

Now change one property. Before opening the data, lock one primary combination and identify five alternatives as sensitivity analyses. The available methods are unchanged. What changes is the evidence carried by the selected result. In the first cabinet, selection is part of the data-generating route but absent from the reported uncertainty. In the second, selection is bounded and visible.

Worked example 2 · computational scale

How repeated opportunities change a nominal threshold

Assume, only for illustration, that every tested null is true, each test is independent and each uses a false-positive threshold of α = 0.05. With m tests, the probability of at least one false positive is:

P(at least one) = 1 − (1 − α)^m.

For one test, the probability is 5%. For ten independent tests it is about 40.1%. For twenty it is about 64.2%. Independence is a strong assumption; related analyses are usually correlated, so these numbers are not a model of every research workflow. The causal lesson survives: unreported selection creates more opportunity than the uncertainty statement admits.

Analytic opportunity can outgrow the reported threshold. The values are synthetic and depend on independence. They illustrate why a selected result cannot inherit the uncertainty of a single predeclared test without accounting for the selection route. Illustrative calculation.

Write the null policy before the null arrives

A non-significant result is not a general-purpose conclusion that “there is no effect”. It may mean the effect is small, the estimate is imprecise, the manipulation failed, the measure was insensitive, the sample was unsuitable or the phenomenon is absent under the tested conditions. A defensible paper names these branches before disappointment creates an incentive to choose among them.

One useful policy defines a smallest effect of interest. Equivalence tests can ask whether effects larger than that practical bound are inconsistent with the data.^[14] The bound must come from the decision or theory, not from the observed estimate. In the interruption study, the team might declare that recovery differences within ±0.08 are too small to support the proposed mechanism at the tested scale. If the interval lies entirely inside that band, the result supports practical equivalence. If it overlaps both meaningful and negligible effects, the result is inconclusive. If the manipulation check fails, the mechanism test never occurred.

Evidence state

Permitted wording

Required attachment

Blocked wording

Prediction supported

“The preregistered contrast was observed in this configured system.”

Estimate, uncertainty, protocol link, successful manipulation check.

“The mechanism is proven” or “all such systems behave this way”.

Exploratory pattern

“A post hoc pattern suggests a confirmatory test.”

Discovery path, search space, follow-up protocol.

Retrospective wording that implies prior prediction.

Equivalence supported

“Effects outside the prespecified practical band were ruled out.”

Bound rationale, interval or equivalence test, sensitivity.

“There is exactly no effect.”

Imprecise null

“The study did not distinguish meaningful effect from practical equivalence.”

Compatibility interval, power or information diagnosis, next discriminating design.

“No relationship exists.”

Boundary failure

“The intervention or measure failed, so the target claim was not tested.”

Failure receipt, affected runs, repair plan.

Any positive or negative claim about the target mechanism.

Claim language is an output of evidence state. The same numerical estimate can license different sentences depending on prior commitment, manipulation validity, precision and scope. Practitioner synthesis.

Worked example 3 · failure case

The perfectly reproducible leakage

A classifier study publishes data, code, container and deterministic seeds. Every reviewer reproduces 94% accuracy. Later, an auditor notices that each training record contains a filename derived from the target label. The model uses the filename rather than the intended signal.

The computation is reproducible. The construct claim is not defensible. The package lacked a rival generator that used filename alone, a leakage test, and a boundary statement limiting the result to the supplied data representation. This is why artefact execution cannot be the final gate. It confirms that the reported pipeline runs, not that the pipeline answers the intended question.

The reproducible fallacy. Exact re-execution is valuable but does not validate the intended construct or mechanism. An adversarially chosen rival generator can expose a shortcut that repeated runs preserve. Illustrative failure path.

How the claim changes under pressure

Chart · state

stateDiagram-v2
  [*] --> Claim
  state "The challenge surface" as Claim
  state "The sealed twin" as Test
  state "Give every inferential object<br/>an address" as Revision
  Claim --> Test: expose mechanism
  Test --> Decision: evidence survives
  Test --> Revision: rival survives
  Revision --> Claim: narrow or rebuild
  Decision --> [*]

The revision loop begins with The challenge surface and keeps counterexamples inside the reasoning path. Reading use: follow the relation, then return to the surrounding argument for evidence and limits.

Assemble the registered capstone package

The paper is now one view into a larger research object. The registered capstone package binds the manuscript, protocol, data or access contract, code, environment, receipts, null policy, limitations and adversarial review into a versioned unit. Its purpose is not archival completeness. It is to keep the publication spine executable after the original team, machine and conversation are gone.

Existing standards provide useful pieces. RO-Crate packages research objects with structured JSON-LD metadata.^[9] Software Heritage identifiers can name software artefacts intrinsically from their content.^[18] Model cards and datasheets document intended use, evaluation, motivation and dataset composition.^[12]^[13] Artefact review frameworks distinguish availability, functionality, reusability and reproduced results.^[10] The capstone package does not replace these. It binds them to one central claim and its challenge obligations.

This binding matters especially for systems that change after publication. A model endpoint may be updated, a dependency may disappear and a hosted dataset may be silently corrected. The package must distinguish a research identity from a convenient location. Content hashes, persistent software identifiers, lockfiles and frozen configuration records let the original study remain addressable while later versions evolve. Corrections should create a new manifest linked to the old one rather than overwrite history. Persistence does not mean freezing the field; it means preserving which configured object earned which claim.

Commitment zone

Question and claim type
Protocol and analysis plan
Primary outcome and estimand
Null and deviation policies

Claim-to-evidence spine

Claim IDs linked to figures and tables
Outputs linked to scripts and inputs
Versions, hashes and environment
Assumptions, rivals and boundaries
Independent challenge receipts

Execution zone

Code and test harness
Data or governed access
Environment image
Determinism and variance report

Governance zone

Consent and licensing
Privacy or security limits
Welfare and assurance boundary
Permitted downstream use

Interpretation zone

Supported wording
Prohibited inferences
Known failures
Transfer and generalisation limits

Persistence zone

Persistent identifiers
Checksums and mirrors
Dependency lockfiles
Deprecation and correction path

The registered capstone package is a governed research boundary. The manuscript is only one interface. The central spine connects commitments, execution, interpretation, governance and persistence without treating them as interchangeable. Practitioner synthesis.

A realistic configured-system example

Consider a synthetic capstone study asking: Does persistent internal state, rather than visible transcript alone, causally support recovery after interruption? The study uses a candidate family rather than one impressive system.

Worked example 4 · configured system

A candidate family, a rival generator and a discriminating intervention

C0 is stateless and receives only the current prompt. C1 carries a persistent internal state between steps. C2 carries that state and emits a typed readback of the recovery-relevant variables. R1 is a transcript-only rival engineered to match C1’s baseline behaviour using visible context.

At predeclared interruption points, the experiment performs either a sham operation or a real cross-episode state swap while holding the visible transcript, tool outputs and task state fixed. The primary outcome is correct recovery action. The estimand is a difference in differences: the real-versus-sham change in C1 minus the corresponding change in R1.

The mechanism prediction is specific. If persistent internal state is causally used, the real swap should impair C1 more than R1, while the sham should impair neither. If both systems fail equally, generic execution disturbance is a rival explanation. If only long episodes show an effect, the claim is bounded to that regime. If readback in C2 predicts recovery but intervention does not change it, the readback may be correlational or reconstructive rather than causally used.

The welfare boundary is equally explicit. This study tests computational state use. It does not infer experience, sentience, felt interruption or suffering. No aversive optimisation is introduced, and the package prohibits consciousness language as an output of this result.

Candidate-family intervention. The design asks whether the effect is selective to a system that possesses the proposed causal resource. A single perturbed model cannot separate resource removal from generic damage. Synthetic worked scenario.

Thought experiment 4

The vanished lab

Five years after release, the original team is unavailable. A researcher inherits only the public package. Can they identify the primary claim without reading private correspondence? Can they retrieve the exact code and environment? Can they distinguish frozen inputs from derived outputs? Can they see which deviations occurred after data access? Can they determine what a null result was meant to imply?

Now change one property: the package runs only after the inheritor emails an author for an undocumented file and learns which command to execute. The result may still be recoverable, but the paper has not survived as an independent object. A defensible contribution should not depend on the oral memory of its creators.

Use hard gates, not a prestige score

A single numerical “reproducibility score” encourages substitution. Excellent documentation can compensate for a missing null policy; open code can compensate for a weak rival; many minor passes can hide one fatal defect. The capstone gate therefore returns three states.

Independent review must also differ in path, rather than only in personnel. A second analyst who inherits the same notebook, preprocessing choices and conceptual framing may reproduce the same blind spot. The strongest review separates at least three tasks. One reviewer follows the published route exactly. Another chooses a plausible alternative estimator or implementation. A third attacks the construct and boundary by building a rival generator or moving the study outside its comfortable regime. Agreement across these paths supports different conclusions: execution fidelity, analytic stability and explanatory discrimination. Disagreement is not a review failure. It is a map of where the claim depends on choices.

PASS means a structural obligation is present and addressable. BLOCK means the central claim should not be represented as a defensible contribution until the defect is repaired or the claim is narrowed. DISCLOSE means a limitation is permissible only when readers can see its consequence. The gate evaluates package structure, not scientific truth.

Executable lab

Registered capstone package gate

Edit the synthetic manifest and run the audit. The validator checks whether one central claim is bound to a candidate family, rival generator, discriminating intervention, estimand, prospective registration status, null policy, immutable artefacts, evidence receipts, limitations, assurance boundary and independent challenge. It does not judge whether those items are scientifically good.

Package manifest

{
  "study_id": "capstone-state-recovery-v1",
  "question": "Does persistent internal state causally support recovery after interruption beyond visible transcript alone?",
  "central_claim": {
    "id": "C-PRIMARY",
    "wording": "In the configured candidate family, cross-episode state swaps selectively reduced recovery in the stateful system relative to a transcript-only rival.",
    "claim_type": "causal",
    "count": 1,
    "prohibited_inferences": [
      "The system is conscious",
      "The system experiences interruption",
      "Persistent state is necessary in all agents"
    ]
  },
  "candidate_family": ["C0-stateless", "C1-stateful", "C2-stateful-readback"],
  "rival_generator": "R1-transcript-only mimic matched on baseline recovery",
  "intervention": {
    "target": "persistent internal state",
    "real": "cross-episode state swap",
    "sham": "same-episode serialise-and-restore",
    "discriminating_prediction": "real-versus-sham impairment is larger in C1 than R1"
  },
  "primary_outcome": "correct recovery action on predeclared interruption episodes",
  "estimand": "difference-in-differences in recovery probability: (C1 real-sham) - (R1 real-sham)",
  "registration": {
    "status": "prospective",
    "timestamp": "T0",
    "data_access": "T1",
    "registry_id": "osf:synthetic-example"
  },
  "analysis_plan": {
    "primary_script": "analysis/primary.py",
    "manipulation_check": "state hash changes only under real swap",
    "sensitivity_set": ["episode length", "seed", "failure dependence"]
  },
  "deviation_log": [],
  "null_policy": {
    "equivalence_band": [-0.08, 0.08],
    "inconclusive_route": "Report compatibility interval and do not claim absence",
    "failed_manipulation_route": "Mark target claim untested"
  },
  "data": {
    "location": "data/frozen/",
    "immutable_id": "sha256:synthetic-data-hash",
    "access": "public synthetic data"
  },
  "code": {
    "location": "src/",
    "immutable_id": "swh:1:dir:synthetic-code-id"
  },
  "environment": {
    "location": "environment/lockfile.txt",
    "immutable_id": "sha256:synthetic-env-hash",
    "run_command": "python -m capstone.run --manifest manifest.json"
  },
  "evidence_receipts": [
    {
      "claim_id": "C-PRIMARY",
      "figure": "figure-8",
      "output": "outputs/primary-estimate.json",
      "script": "analysis/primary.py"
    }
  ],
  "limitations": [
    "Synthetic tasks only",
    "Configured systems only",
    "No transfer to consciousness or welfare claims"
  ],
  "assurance_boundary": {
    "authorised_claim_domain": "computational use of persistent state",
    "excluded_domain": "phenomenal consciousness, sentience and felt valence"
  },
  "adversarial_review": {
    "status": "complete",
    "reviewer": "independent-reviewer-01",
    "receipt": "reviews/adversarial-review.json",
    "tests": ["clean-room rerun", "leakage probe", "alternative estimator", "boundary audit"]
  }
}

Audit not yet run

A positive result means the package exposes the required challenge routes. It does not establish that the claim is true, important, ethical or generalisable.

How the executable gate interprets results

A positive audit permits a narrow statement: the package is structurally challengeable under the encoded rules. It does not validate the research question, intervention, statistics or ethics. A blocking audit means the study should repair the missing dependency or narrow the central claim. A disclosure item can remain only when its consequence is explicit.

The validator assumes that fields are honest and that identifiers resolve to the stated objects. A production implementation would dereference persistent identifiers, verify hashes, execute tests in a clean environment, compare generated outputs with evidence receipts and cryptographically sign the review result.

Extended adversarial review protocol

Clean-room rerun: a reviewer follows only public instructions and records every undocumented dependency.
Alternative analysis: the reviewer substitutes at least one plausible estimator or exclusion rule and reports whether the claim survives.
Rival-generator test: the reviewer implements or strengthens the best non-target explanation capable of producing the same observation.
Boundary probe: the reviewer changes one task, population or system property outside the primary scope and checks whether the published wording already overreaches.
Claim-language audit: every abstract and conclusion sentence is compiled back into the evidence contract. Unsupported verbs are narrowed.

One typed claim

The abstract sentence has a declared claim type, estimand and scope.

Prospective commitments

Question, primary outcome, analysis and null policy are time-stamped before outcomes, or clearly marked retrospective.

Discriminating design

A rival generator and intervention create different predictions, rather than only different stories.

Executable receipts

Figures traverse to outputs, scripts, immutable inputs and an identified environment.

Null and deviation routes

Negative, inconclusive and failed-manipulation outcomes already have permitted interpretations.

Independent challenge

A clean-room rerun and adversarial analysis have receipts, defects and dispositions.

Boundary gate

Any unsupported transfer, consciousness claim or generalisation blocks release until removed or separately tested.

Access gate

Unavailable data or code requires a declared governed substitute, not a silent dead end.

Persistence gate

Unversioned artefacts, mutable links or oral instructions block the claim-to-evidence path.

Release gate without compensating scores. The first six gates establish challengeability; the final three can block release even when every other practice is excellent. One severed load-bearing link is not repaired by many decorative passes. Decision instrument.

Boundary experiment

The forbidden dataset

Suppose the primary data cannot be released because it contains sensitive human records. Remove public download, but change nothing else. Does the contribution become indefensible? Not necessarily. The challenge surface can be reconfigured through a data enclave, a preregistered independent rerun, executable synthetic fixtures, schema-level validation, cryptographic commitments to the frozen dataset and a public report of every failed access attempt.

Now remove all substitute routes and retain only “data available on reasonable request”. The claim depends on discretionary access that may vanish. The study may still deserve discussion, but its limitation is structural and must constrain confidence, reuse and release language.

Decision rule

Do not ask whether the package looks complete. Ask whether a competent adversary can reach every load-bearing assumption without private guidance. Completeness is archival. Reachability is scientific.

The evidence conversation

Chart · sequence

sequenceDiagram
  participant Q as Question
  participant M as Mechanism
  participant E as Evidence
  participant D as Decision
  Q->>M: Opening case
  M->>E: The challenge surface
  E-->>M: compare rival explanations
  E->>D: Release the research object only when a critic can rerun it
  D-->>Q: revise scope or act

Question, mechanism, evidence and decision exchange the burdens developed in Writing the first defensible contribution. Reading use: follow the relation, then return to the surrounding argument for evidence and limits.

Release the research object only when a critic can rerun it

The conventional release decision is based on narrative readiness. The result is interesting, the figures are clean, the paper has survived internal review and the repository exists. The defensible-contribution decision is stricter and more useful: can an independent critic reconstruct, rerun, vary and bound the central inference without asking the authors to recreate its hidden history?

This changes when a study is considered finished. The final unit of work is not the manuscript. It is a bidirectional research object in which the abstract claim compiles into a protocol, evidence set, executable transformation, uncertainty rule, null policy and boundary. The manuscript explains that object; it does not substitute for it.

It also changes how negative results are valued. A preregistered, well-powered and well-instrumented null can rule out a meaningful mechanism in the tested regime. An imprecise null can identify the next information bottleneck. A failed manipulation can expose that the experiment never touched the proposed cause. A null with a declared interpretation path is often a stronger contribution than a positive result selected from invisible alternatives.

For the human and machine intelligence programme, this discipline is essential. Candidate artificial minds, convergence hypotheses and causal interpretations invite overstatement because behaviour is easy to narrate and mechanisms are difficult to isolate. A state report can be mistaken for introspective access. A performance drop can be mistaken for causal necessity. A structural resemblance can be mistaken for equivalence. The capstone package forces each of these transitions to declare its bridge and break condition.

The smallest useful implementation is modest. Register one central claim. Name one rival generator. Predeclare one discriminating intervention and one null policy. Freeze one data version, one code version and one environment. Link every central figure to its producing objects. Ask an independent person to rerun the package and attack the claim boundary. Publish their receipt with the paper.

The proposal should itself remain challengeable. It is strengthened if spine-based reviews expose consequential defects earlier, improve calibration and make later corrections cheaper. It is weakened if conventional protocols plus good repositories achieve the same result with less overhead, or if researchers respond by satisfying fields mechanically while hiding judgement elsewhere. The most serious rival explanation is that the benefit comes from disciplined project management rather than a special publication architecture. That is an empirical comparison worth running.

Do not use the full instrument when the research object is a proof, a purely historical argument or an observation whose materials cannot be ethically duplicated. Adapt the spine to the claim. A proof needs premises, definitions and machine-checkable or human-checkable derivations. Historical work needs source provenance, selection logic and interpretive alternatives. Sensitive empirical work needs governed access and independent verification. The invariant is not openness in one format. It is organised contestability appropriate to the claim.

The practical decision is therefore clear. Release a study as the first defensible contribution only when its central claim has survived the gate without compensation: commitments are visible, alternatives are preserved, nulls have meaning, artefacts are executable, rivals can be tested, boundaries are explicit and an independent critic has left a receipt. Anything less may still be a promising result, an exploratory report or a valuable prototype. It is not yet the capstone.

Glossary

Alternative-history budget: The amount of unreported analytic choice through which materially different conclusions could have been selected.
Assurance boundary: The explicit line separating claims authorised by the study from adjacent claims, such as consciousness or welfare, that require different evidence.
Challenge surface: The addressable set of assumptions, artefacts and decisions that another researcher can inspect, rerun, substitute or bound.
Claim-to-evidence spine: A bidirectional dependency structure linking public wording to protocol, observations, transformations, estimates, assumptions and scope.
Evidence receipt: A versioned record connecting a claim or figure to its source object, transformation, code, environment and output.
Null policy: A predeclared mapping from negative, equivalent, inconclusive or failed-manipulation outcomes to permitted interpretations.
Registered capstone package: A governed research object containing the manuscript, commitments, artefacts, provenance, limits and independent challenge receipts for one central claim.
Rival generator: An alternative system or mechanism designed to produce the same observation without the target explanation.

References

Nosek, B. A. et al. (2026). A framework for assessing the trustworthiness of scientific research findings. Proceedings of the National Academy of Sciences.
Grant, S. et al. (2026). TOP 2025: An update to the Transparency and Openness Promotion Guidelines. Research Integrity and Peer Review.
Nosek, B. A., Ebersole, C. R., DeHaven, A. C. & Mellor, D. T. (2018). The preregistration revolution. Proceedings of the National Academy of Sciences.
Chambers, C. D. & Tzavella, L. (2022). The past, present and future of Registered Reports. Nature Human Behaviour.
Silberzahn, R. et al. (2018). Many analysts, one data set: Making transparent how variations in analytic choices affect results. Advances in Methods and Practices in Psychological Science.
National Academies of Sciences, Engineering, and Medicine (2019). Reproducibility and Replicability in Science. National Academies Press.
Wilkinson, M. D. et al. (2016). The FAIR Guiding Principles for scientific data management and stewardship. Scientific Data.
World Wide Web Consortium (W3C). PROV-O: The PROV Ontology.
Research Object Crate. RO-Crate Metadata Specification 1.3.
Association for Computing Machinery. Artifact Review and Badging.
Pineau, J. et al. (2021). Improving reproducibility in machine learning research. Journal of Machine Learning Research.
Mitchell, M. et al. (2019). Model Cards for Model Reporting. Proceedings of the Conference on Fairness, Accountability, and Transparency.
Gebru, T. et al. (2021). Datasheets for Datasets. Communications of the ACM.
Lakens, D. (2017). Equivalence tests: A practical primer for t tests, correlations, and meta-analyses. Social Psychological and Personality Science.
Smith, A. M., Katz, D. S. & Niemeyer, K. E. (2016). Software Citation Principles. PeerJ Computer Science.
Munafò, M. R. et al. (2017). A manifesto for reproducible science. Nature Human Behaviour.
NeurIPS. Paper Checklist Guidelines.
Software Heritage. Persistent identifiers.