A public theory test is usually narrated like a race. Two camps enter, a dataset appears, and one camp leaves with the trophy. This story is attractive because it turns a difficult research programme into a clean event. It is also usually false.

A scientific theory is not one sentence. It is a layered object. At the centre sits a claim about the world. Around it sit bridge claims linking that claim to an observable system, auxiliary assumptions about the task and apparatus, and analysis choices that turn measurements into reported outcomes. A result can challenge any of those layers. Calling the whole theory a winner or loser hides the location of the impact.

The unit that evidence strikes

Picture the theory as a four-layer commitment stack. The core claim says what organisation or process matters. The system bridge says that the studied brain, model or organism instantiates that organisation. The measurement bridge says that a recorded variable is sensitive to it. The decision rule says which pattern counts as support, challenge or failed access. Evidence reaches the core only by passing through the other layers.

This matters because the same numerical result can carry different meanings. A classifier at chance may contradict a claim that information must be linearly available in the sampled region. It does not automatically contradict a claim that the information exists in a non-linear, transient or unsampled form. Conversely, a successful classifier may show that information is decodable while leaving open whether the system actually uses it. Decodability, causal use and theoretical necessity are separate commitments.

A fair test therefore registers not only the expected signal but the path by which the signal bears on the theory. It also states the strongest permitted conclusion if one link fails. When the localiser fails, the honest status may be “uninformative”. When the localiser passes and the target event is absent under a sensitive analysis, the measurement bridge and theory commitment face more pressure. The protocol becomes useful when it prevents a team from deciding this hierarchy after seeing the data.

Where a prediction can retreat

Suppose a theory says conscious visual content is globally broadcast. To test it, researchers operationalise broadcast as decodable prefrontal activity in a particular task, measured with a particular imaging modality and analysed with a particular classifier. A null result may mean the global-broadcast claim is wrong. It may instead mean prefrontal decoding is an imperfect bridge, the task did not elicit the target process, the instrument missed the relevant scale, or the analysis was insensitive.

Those are not equally convenient excuses. Some were predictable before the study; others may be genuine discoveries. The point is to declare in advance which failure would place which commitment at risk. Without a risk map, every negative result arrives with an unlimited supply of escape routes.

There are domains where a winner is meaningful. If two bridge designs compete to carry a fixed load at fixed cost, and the deployment decision is precisely “which bridge should we build?”, a common metric can select one. Even then, the result need not establish the winning design’s general theory of structures. Decision victory and explanatory victory are different achievements.

flowchart LR
  A["The storm that proved nothing"] --> B["The podium is the wrong instrument"]
  B --> C["Turn disagreement into a<br/>measurable object"]
  C --> D["Govern the route from<br/>prediction to revision"]
  D -. "re-test" .-> A
  classDef first fill:#0a1424,stroke:#087A98,stroke-width:2px,color:#e6f3ff
  classDef step fill:#0a1424,stroke:#3f7da0,stroke-width:1px,color:#e6f3ff
  class A first
  class B,C,D step

Before choosing participants, scanners or statistical thresholds, ask a smaller question: what outcome does each theory expect from the same experiment? Not “what does each theory say in general?”, but “what probability does this version of the theory assign to each result we could actually observe?”

That move converts a verbal dispute into a prediction delta. For a finite set of mutually exclusive outcomes, the raw delta is the total variation distance between the theories’ predictive distributions:

Here, e is one proposed experiment, o is a possible registered outcome, and each theory assigns a probability to every outcome. The equation is valid only when the outcome categories are shared, mutually exclusive and collectively exhaustive. For a binary result, it becomes the absolute difference between the two probabilities.

Separability is not importance

A high prediction delta does not mean the question is profound. It means the proposed observation can separate the registered models. A trivial calibration check can have a high delta, while a philosophically central question can have no present measurement that distinguishes its rivals. The score therefore answers an experimental question, not a metaphysical one.

Consider three registered outcomes for a perturbation: an early response only, a late response only, or both. Theory A assigns probabilities 0.70, 0.10 and 0.20. Theory B assigns 0.10, 0.60 and 0.30. The absolute differences are 0.60, 0.50 and 0.10; halving their sum gives a delta of 0.60. The test is potentially discriminating even though neither theory predicts a single outcome with certainty. This is usually more realistic than forcing every account into a yes-or-no slogan.

The outcome set must be fixed before unblinding. If researchers add a fourth category only after noticing an unexpected pattern, the original probabilities no longer describe the realised test. The new category may be scientifically valuable, but it belongs to an exploratory branch or a new packet. A prediction delta is meaningful only for the outcome space to which the probabilities were committed.

Raw delta is not enough. A test may target a peripheral implication, rely on a fragile bridge or duplicate evidence already embedded in both theories. For experiment selection, the article proposes an explicitly heuristic quantity:

B is confidence that the observable genuinely tracks the proposed mechanism. C is how central the prediction is to the registered theory version. I discounts evidence that repeats the same dependency, participants, pipeline or auxiliary assumption. Each runs from zero to one. This product is a design score, not a posterior probability. It forces a team to expose why an apparently sharp test may still be weak.

Rate the design debt, not the preferred theory

Bridge reliability, centrality and independence are contestable judgements. Treating them as precise measurements would manufacture certainty. Their value comes from forcing a structured disagreement before the main result. Each rating should therefore include a short rationale, a named assessor, an uncertainty range and the condition that would change it.

For bridge reliability, ask whether the observable changes when the proposed mechanism is independently manipulated, whether the instrument resolves the relevant spatial and temporal scale, and whether a plausible rival generator can produce the same signal. For centrality, ask whether proponents would revise the theory if the prediction failed under valid conditions. For independence, trace shared participants, preprocessing, task demands, model assumptions and analytic code. Two modalities are not independent simply because one produces images and the other waveforms.

Disagreement among raters is itself useful. If proponents rate centrality at 0.9 and neutral reviewers rate it at 0.3, the project has found a governance problem before spending its full budget. The packet can preserve both ratings or require a scope negotiation. It should not average the conflict into a cosmetically precise 0.6. The purpose of the score is to reveal where adjudication depends on an unresolved judgement.

This also changes null interpretation. If every theory assigns a low probability to the observed outcome, the result may be evidence for a missing rival, a broken bridge or a misdescribed outcome space. A league table would still rank the least-wrong account. A portfolio protocol instead marks the shared miss and opens a model-development branch. The absence of a winner can be the most informative result when all registered accounts fail together.

Experiment selection then becomes sequential. The first test should split a consequential part of the current portfolio through a valid bridge. After its result, the next test should target the rivals that remain close. In the example, a second perturbation might be valuable if A predicts 0.25 and C predicts 0.80. The first result prunes B; the second addresses the surviving ambiguity. One enormous omnibus study is not automatically better than two smaller tests whose deltas are conditional on what has already been learned.

The practical remedy is a prediction matrix. Rows are registered outcomes; columns are versioned theories; each cell contains a probability and the commitment it represents. The matrix exposes observational equivalence classes: groups of theories that the proposed test cannot presently separate. It also reveals when two accounts disagree verbally but occupy nearly the same predictive column.

Real fields rarely contain only two live accounts. A pairwise test can strongly separate Theory A from Theory B while doing almost nothing to distinguish A from Theory C. Suppose A assigns an 0.80 probability to a positive result, B assigns 0.20 and C assigns 0.75. A positive result creates a large update against B. It offers little reason to prefer A over C. Reporting that “A won” would turn a local contrast into a field-wide claim.

From a duel to a theory portfolio

The commitment packet

A high-delta test still fails if “Theory A” quietly changes meaning between planning and interpretation. The smallest useful remedy is a versioned commitment packet signed before unblinding. It is not a manifesto. It is a compact, typed record that makes later disagreement locatable.

The packet is not a promise that every surprise has been anticipated. It is a record of what the planned test was allowed to establish. Exploratory findings can still generate new mechanisms, categories and theories. They simply do not retroactively increase the diagnostic force of the registered result.

A practical packet also separates validity gates from theory outcomes. A manipulation check, signal-quality threshold or behavioural-performance criterion may determine whether the target contrast was created and measured. If a gate fails, the outcome is not automatically evidence for the theory that predicted no effect. It may be evidence that the experiment never reached the state in which the rival predictions applied. This distinction prevents absence of access from being counted as absence of mechanism.

A commitment packet constrains interpretation, but it cannot make the path from data to conclusion neutral. Complex analyses contain many defensible choices. In a large neuroimaging study, 70 teams analysed the same dataset against the same nine hypotheses; no two teams used identical workflows, and their inferential conclusions varied substantially.[6] That result does not imply analysis is arbitrary. It shows that the pipeline is part of the experiment.

Registered Reports address one incentive problem by reviewing the question and method before outcomes are known, then granting in-principle acceptance independent of whether the result is positive.[7] Adversarial collaboration addresses another by making rivals jointly design the test. Neither practice alone tells us how a theory should change after a mixed result. That requires a governed hand-off.

Separate four kinds of authority

A workable collaboration distinguishes four roles. Theory stewards specify the strongest version they are prepared to risk. Design stewards translate those commitments into tasks, manipulations and validity gates. Analysis stewards implement the registered pipeline and labelled robustness analyses. An adjudication group maps outcomes back to commitments without owning either theory. One person may hold more than one role in a small project, but the role changes must remain visible.

This separation is not a ritual of neutrality. Every role carries judgement. Its purpose is to prevent one judgement from silently controlling the whole chain. A proponent should be able to reject a measurement bridge before data collection, but not after a disliked result without issuing a revision. An analyst should be able to report that a conclusion depends on a defensible preprocessing choice, but not select that choice because it favours one camp. An adjudicator should classify local evidence, not rewrite the theories.

The protocol also needs an analysis envelope. The primary pipeline answers the registered question. A small set of pre-declared alternatives tests whether the conclusion survives plausible analytic choices. Exploratory analyses remain welcome but are labelled as generators of later packets. This arrangement preserves discovery while protecting the evidential status of the original result. The many-analysts finding is then treated as a design fact: analytic flexibility is another causal path between the world and the headline.

The revision receipt

When results arrive, each theory version receives a receipt. It records the observed outcome, the validity-gate status, the local judgement against each prediction, and the update the proponents actually make. Possible local judgements include supported, challenged, bridge failure, under-informative and out of scope. They should not be collapsed into one total score unless an aggregation rule was registered and justified.

The receipt then identifies the changed object: core claim, system bridge, scope, auxiliary, parameter range or no change. It records the new version identifier and the next discriminating test. This creates a lineage rather than a rhetorical survival story.

A revision that merely adds words but leaves all future observations compatible has not increased empirical content. A revision that narrows the theory may still be progress if it removes a false generalisation. A changed measurement bridge may be progress if a new intervention validates it. The receipt does not rank these moves morally. It preserves enough lineage for other researchers to see what was surrendered, what was retained and what now becomes testable.

Theories should be allowed to learn. Prohibiting revision would reward vagueness and punish discovery. The problem is cost-free rescue, where every result is absorbed without changing future risk. A revision receipt makes the cost operational by requiring at least one of four consequences: a narrower scope, a changed bridge, a new parameter constraint or a new prediction that could fail.

A revision must pay an explanatory cost

Six ways a fair contest becomes ceremonial

The most common failure is not misconduct. It is asymmetry. One camp treats its prediction as central and the rival’s as peripheral. One camp can invoke measurement limits while the other cannot. One theory is represented by a formal model; the other by an evolving verbal family. One positive result is counted once for each modality even though the modalities share the same task and assumptions.

Other failures are structural. A test may select outcomes that both theories predict. The validity control may fail, turning a null into missing evidence. Analysts may tune the pipeline on part of the final data, then describe the remaining sample as wholly untouched. An aggregate “score” may hide that each theory failed a different high-value commitment.

stateDiagram-v2
  [*] --> Claim
  state "The unit that evidence strikes" as Claim
  state "Where a prediction can retreat" as Test
  state "Separability is not importance" as Revision
  Claim --> Test: expose mechanism
  Test --> Decision: evidence survives
  Test --> Revision: rival survives
  Revision --> Claim: narrow or rebuild
  Decision --> [*]

The Cogitate collaboration provides a rare worked case. Proponents of Integrated Information Theory and Global Neuronal Workspace Theory, together with a theory-neutral consortium, developed preregistered predictions for conscious visual content, its maintenance over time and interareal connectivity. The project used fMRI, MEG and intracranial EEG across a large multi-site sample.[1][2]

Three questions hidden inside one headline

The case contains three different inferential questions. Did the registered observations occur? Did those observations challenge the commitments written into the protocol? Do those commitments fairly represent the broader theory family that researchers now wish to defend? The first is mainly empirical, the second is contractual and the third is theoretical. A disagreement at the third level does not erase an answer at the first.

This distinction matters because scientific theories often exist as families of formulations. A biological implementation claim may be stricter than a computational principle. A task-specific prediction may be one proposed bridge rather than the only bridge. The Nature result can therefore challenge registered commitments without logically refuting every version that shares the same label. Equally, the continued existence of a broader version does not mean the tested version escaped unchanged.

A portfolio view records both facts. It reduces credence in the challenged local commitments, preserves compatible commitments at their warranted weight and raises the priority of experiments that separate the revised accounts. The proper output is a vector of local updates plus a next-test decision, not a scalar victory score.

The resulting Nature paper did not crown a winner. It reported a direct challenge to IIT from the absence of sustained posterior synchronisation, and challenges to GNWT from the general absence of the preregistered offset ignition and the limited prefrontal representation of some consciously perceived dimensions. It also reported findings compatible with parts of both accounts. The authors explicitly invited readers to weigh the evidence rather than accept a definitive verdict, and called for a quantitative framework that can integrate prediction centrality, measurement error and cross-sample consistency.[1]

The later GNWT response is especially instructive. Its authors argued that an offset ignition was not a central GNWT prediction unless the offset was consciously attended, and that the study’s clearly visible stimuli could not test the theory’s core contrast between conscious and non-conscious processing.[3] This does not by itself show evasion. It shows that “centrality”, “scope” and “bridge” remained live objects after a sophisticated preregistration.

A revision receipt would make the options explicit. One receipt might state that the preregistered biological implementation was challenged while the computational core remained untouched. Another might register a scope revision requiring attended offsets. A third might contest whether the original packet fairly represented GNWT at all. These are materially different scientific states. A headline saying “GNWT survived” or “GNWT failed” erases them.

The next step need not repeat the same contest at greater scale. A 2026 adversarial protocol extends the comparison to non-human primates and mice, using Neuropixels recordings and causal perturbations of prefrontal regions.[4] This moves the bridge from correlational human imaging towards finer-grained recording and intervention. It is valuable because it changes what the experiment can discriminate, not because animals provide a second ballot.

What the lab tests, and what it cannot

The lab tests whether a proposed binary experiment contains enough declared disagreement to justify an adjudication claim. A high raw delta permits a strong update if the observation is valid. A low delta means the result should be framed as joint model development or measurement work. A low effective yield points to the specific design debt: bridge, centrality or independence.

A positive result permits only the registered local update. A negative result may challenge a theory only if the validity gates pass. The tool cannot establish that either theory is true, cannot discover the right outcome categories, and cannot turn subjective centrality ratings into objective facts. It also assumes a binary outcome and equal priors for the displayed Bayesian update. The JSON packet is the durable contribution; the score is a planning aid.

The displayed recommendation uses transparent heuristic bands rather than a claim of universal optimality. A team should calibrate its own bands against study cost, reversibility and the harm of an overclaimed conclusion. In a cheap pilot, low yield may still justify measurement development. In an expensive multi-site adjudication, the same yield should trigger redesign. The downloadable packet preserves the inputs so another team can challenge the ratings without reconstructing them from prose.

{
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "title": "TheoryCommitmentPacket",
  "type": "object",
  "required": [
    "packet_id", "theories", "outcome", "bridge",
    "validity_gates", "revision_rules"
  ],
  "properties": {
    "packet_id": { "type": "string" },
    "theories": {
      "type": "array",
      "minItems": 2,
      "items": {
        "type": "object",
        "required": ["name", "version", "scope", "p_positive", "centrality"],
        "properties": {
          "name": { "type": "string" },
          "version": { "type": "string" },
          "scope": { "type": "string" },
          "p_positive": { "type": "number", "minimum": 0, "maximum": 1 },
          "centrality": { "type": "number", "minimum": 0, "maximum": 1 }
        }
      }
    },
    "outcome": {
      "type": "object",
      "required": ["positive_definition", "negative_definition"],
      "properties": {
        "positive_definition": { "type": "string" },
        "negative_definition": { "type": "string" }
      }
    },
    "bridge": {
      "type": "object",
      "required": ["claim", "reliability"],
      "properties": {
        "claim": { "type": "string" },
        "reliability": { "type": "number", "minimum": 0, "maximum": 1 }
      }
    },
    "validity_gates": {
      "type": "array",
      "minItems": 1,
      "items": { "type": "string" }
    },
    "revision_rules": {
      "type": "array",
      "minItems": 2,
      "items": { "type": "string" }
    }
  }
}

function assess(packet, observedPositive) {
  const [a, b] = packet.theories;
  const delta = Math.abs(a.p_positive - b.p_positive);
  const centrality = Math.min(a.centrality, b.centrality);
  const yieldScore =
    delta *
    packet.bridge.reliability *
    centrality *
    packet.evidence_independence;

  const likelihoodA = observedPositive
    ? a.p_positive
    : 1 - a.p_positive;
  const likelihoodB = observedPositive
    ? b.p_positive
    : 1 - b.p_positive;

  // Toy display only: equal priors and mutually exclusive theories.
  const posteriorA = likelihoodA / (likelihoodA + likelihoodB);

  return { delta, yieldScore, posteriorA };
}