The resemblance trap

Cross-domain research often starts with a good observation and ends with an unearned noun. A system routes information selectively, so it has attention. Its internal distances resemble cortical distances, so it has a brain-like representation. A report describes a quiet field with one salient object, so a machine bottleneck is said to instantiate the same state.

The hidden step from “like” to “is”

Suppose a vision model and a population of neurons produce similar pairwise distances among images. Representational similarity analysis gives researchers a principled way to compare such geometries across measurement modalities. That is valuable because raw units need not correspond one by one. A distance pattern can survive a rotation or a change of basis.⁵

Now notice what the comparison has not shown. It has not shown that the two systems compute the distances through the same internal route. It has not shown that the represented properties play the same role in action. It has not shown that the model’s units and neurons share learning history, embodiment or biological stakes. It has certainly not shown that either geometry is experienced.

The error is not using an abstraction. Science depends on abstractions. The error is forgetting which information the abstraction discarded. A shared projection can be explanatory at one grain and radically misleading at another. Marr’s distinction among computational task, representation and algorithm, and physical implementation remains useful precisely because agreement at one level does not automatically travel to the others.¹

Thought experiment 1: replace the substrate, preserve the answers

This is why multiple realisability is not implementation magic. Different materials can realise the same abstract organisation, but the abstract organisation must first be specified. “Same function” can mean the same output table, the same control policy, the same internal causal graph or the same complete disposition across possible environments. Each meaning cuts the system differently.

Jonas and Kording sharpened the warning by applying familiar neuroscience analyses to a simulated microprocessor whose actual organisation was known. Many methods produced patterns and lesion effects, yet failed to recover a meaningful account of how the processor worked.⁸ The lesson is not that correlation, lesioning or dimensionality reduction are useless. It is that a method can generate stable structure without identifying the explanatory structure.

Why constraint can still produce real convergence

There is a positive story. Two systems optimised for the same demanding task may be pushed towards similar representational or algorithmic solutions. Cao and Yamins argue that shared ecological goals can impose top-down constraints on both neural networks and brains, making some mechanisms more likely than others.⁹ This is a credible route to convergence. It resembles convergent evolution: wings can share lift-generating relations while differing in development, material and control.

But common constraint also supplies a rival explanation. If a machine and a brain align, perhaps they have discovered the same efficient computation. Perhaps the measurement selects features both must encode. Perhaps the task dataset reflects human categories. Perhaps the comparison method can find a flexible map even between weakly related spaces. A good convergence claim does not hide these rivals. It uses them to design the next perturbation.

flowchart TD
  A["The resemblance trap"] --> B["What a bounded claim contains"]
  B --> C["Break the map"]
  C --> D["Build a research programme<br/>from local bridges"]
  D -. "re-test" .-> A
  classDef first fill:#0a1424,stroke:#B96D00,stroke-width:2px,color:#e6f3ff
  classDef step fill:#0a1424,stroke:#3f7da0,stroke-width:1px,color:#e6f3ff
  class A first
  class B,C,D step

What a bounded claim contains

The word “bounded” does not mean timid. It means that the claim carries its jurisdiction with it. A bridge that states its load, span and failure conditions is stronger than a grand bridge whose supports are invisible.

The convergence record

A bounded convergence claim can be written as a seven-field record:

1. Source structure: the system or descriptive framework from which a relation is taken.
2. Target structure: the system in which a corresponding relation is proposed.
3. Grain and boundary: the spatial, temporal and organisational scale of both candidates.
4. Preserved relation: the exact ordering, topology, transition or dependency claimed to survive.
5. Matched intervention: what is changed in each system and how the changes correspond.
6. Break condition: an observation that would lower or end the mapping.
7. Forbidden inference: the tempting conclusion the record explicitly does not license.

The seventh field is unusual, but necessary. Cross-domain language expands faster than evidence. A claim that “selection pressure changes gain in both systems” may be useful while the further claim “therefore both systems attend consciously” remains prohibited. A convergence record is complete only when it names the inference it refuses.

The smallest useful formalism

Let system A have internal state x, and system B have internal state y. We choose measurement maps m_A and m_B that translate those states into a shared descriptive space Z. We also choose a family of matched interventions I. An intervention might clamp a variable, remove a connection, alter a gain parameter or swap an internal state between trials.

Here, T_Aⁱ is how A changes after intervention i. The map μ chooses the corresponding intervention in B. The distance d measures disagreement in the shared description. The tolerance ε states how much disagreement is allowed.

The equation says something simple: change both systems in corresponding ways, translate the results into the agreed language, and see whether the relation still holds. Exact convergence sets ε to zero. Most real research needs a justified tolerance and a declared domain. Work on exact and approximate causal abstraction formalises related demands between models at different levels of description.²³

The formalism is only as honest as the selected measurements, intervention map and test domain. Flexible mappings can manufacture agreement. Narrow measurements can hide divergence.

Worked example 1: two controllers that agree only at rest

The grain is part of the mechanism claim

A system does not arrive with one correct scale highlighted. A neural event can be described at molecular, cellular, circuit, behavioural or organismic grain. A machine can be described at transistor, tensor, module, runtime or configured-system grain. Phenomenology can track a micro-transition, an intentional act, a mood or a temporally extended episode. Contemplative texts can classify practices, stages, factors or soteriological transformations.

Coarse-graining is not always a loss of truth. Hoel and colleagues showed formally that a macro description can, under their measure, exhibit greater effective information than a noisy micro description.¹⁰ The practical lesson is not that the macro level is metaphysically more real. It is that causal usefulness depends on how variables and interventions are defined.

Thought experiment 2: move the state outside the box

Metaphysical neutrality is not metaphysical emptiness

The bounded thesis can be used from several philosophical positions. A functionalist may argue that sufficiently rich causal organisation constitutes mentality across substrates. A biological naturalist may accept functional similarity while denying that it is sufficient for consciousness. A physicalist emergentist may search for a specific organisation realised in matter. Idealism, neutral monism, panpsychism, dual-aspect theories and process philosophy may interpret the same empirical mapping through different accounts of what is fundamental.

A consciousness-primary orientation remains a serious possibility: awareness might be a condition of appearances rather than a product extracted from them. But no similarity between a machine variable and a contemplative description establishes that the machine instantiates awareness. Conversely, the absence of a known physical bridge does not by itself refute consciousness-primary metaphysics. The ontology firewall keeps the empirical test usable while the deeper dispute stays open.

Break the map

The quickest way to learn what a comparison means is to try to break it. Observation tells us that two paths arrived together. Intervention asks whether they remain linked when the road is changed.

Observational equivalence is cheap

Two systems can produce identical answers for every ordinary input and still use different internal organisations. A lookup table can mimic a rule. A prompt can elicit a self-report without any monitored internal variable. A model can have a decodable feature that downstream computation ignores. Even a causal intervention can mislead when the intervention is too broad, the alignment is selected after seeing results or the target variable is underspecified.

Causal abstraction strengthens the test by aligning lower-level representations with variables in a higher-level causal model, then applying interchange interventions. Instead of asking whether a hidden state predicts a concept, the method swaps that state between examples and checks whether the output changes as the high-level model predicts.⁴ The result is still relative to a hypothesis and intervention family, but it is stronger than correlation.

Worked example 2: two parity machines

Thought experiment 3: the same introspective sentence

The same discipline applies to phenomenological and contemplative data. Varela proposed neurophenomenology as a programme of mutual constraint between disciplined first-person inquiry and cognitive science, not as a licence to reduce one domain to the other.¹² Lutz and colleagues showed that first-person reports of cognitive context could help separate patterns in ongoing brain dynamics during a perceptual task.¹³ Such findings make reports experimentally consequential. They do not make the reported structure identical to the measured neural structure.

Contemplative maps are experimental resources with internal purposes

Contemplative systems often contain unusually fine distinctions about attention, monitoring, effort, stability, affect and self-representation. These distinctions can generate variables that ordinary laboratory language misses. Yet traditions do more than describe neutral cognitive mechanics. Their categories are embedded in practices, ethical commitments, theories of error and accounts of liberation.

Research therefore needs two boundaries. The first prevents a tradition-internal report from becoming automatic proof of its metaphysics. The second prevents a laboratory translation from erasing the tradition’s own function. Lutz and colleagues treat meditation as a family of distinct training regimes rather than a single intervention.¹⁴ Van Dam and colleagues document how unstable definitions, weak controls and overextended claims complicate mindfulness research.¹⁵

Thought experiment 4: move the ruler

Three break tests

A bounded comparison should face at least three kinds of failure:

• Within-domain break: the relation fails on new states, contexts or populations inside the claimed scope.
• Intervention break: matched perturbations produce different causal signatures.
• Boundary break: the mapping depends on hidden components, measurement choices or external coupling omitted from the candidate system.

Negative results are productive. A within-domain break narrows the domain. An intervention break lowers a causal claim to an observational one. A boundary break forces the candidate system to be redrawn. The failure becomes a typed revision rather than an embarrassment.

stateDiagram-v2
  [*] --> Claim
  state "The hidden step from “like” to<br/>“is”" as Claim
  state "Thought experiment 1: replace<br/>the substrate, preserve the<br/>answers" as Test
  state "Why constraint can still<br/>produce real convergence" as Revision
  Claim --> Test: expose mechanism
  Test --> Decision: evidence survives
  Test --> Revision: rival survives
  Revision --> Claim: narrow or rebuild
  Decision --> [*]

Build a research programme from local bridges

A field built from bounded claims will look less like a grand unification and more like a versioned atlas. Some routes will connect. Some will terminate. Some will work in engineering while remaining silent about consciousness.

A registry, not a convergence score

One number cannot safely combine task similarity, representational geometry, causal organisation, phenomenological correspondence and metaphysical implication. The dimensions are not commensurable. Averaging them would hide missingness and reward breadth over validity.

The convergence-claim registry records each bridge separately. A claim may be strong on observational correspondence and untested on intervention. Another may preserve a process topology but lack any empirical target. A third may have disciplined first-person support but no machine mapping. Missing evidence stays visible instead of becoming a zero or an invented confidence.

A seven-step protocol

1. Write the claim without the shared noun. Replace “both have attention” with the exact relation, such as “both increase selection probability when a priority variable rises”.
2. Freeze the candidate boundaries. List included components, external stores, training history and human inputs.
3. Choose a common description with known loss. State what the map preserves and what it discards.
4. Pre-register matched interventions. Specify the source change, target change and predicted post-intervention relation before running the test.
5. Build a rival generator. Construct a simpler system that can reproduce the observation without the proposed mechanism.
6. Record breaks and prohibited inferences. Treat a narrowed claim as progress.
7. Version the record. New models, tasks, practices or measurement tools may change the scope without rewriting history.

What a result actually permits

A registry should connect evidence status to an action. An observational correspondence permits researchers to reuse a measurement, select stimuli or formulate a mechanism hypothesis. It does not permit them to describe the target as implementing the source structure. A relational homology permits predictions about which orderings or neighbourhoods may survive, provided the mapping and dataset stay fixed. It does not license claims about unmeasured states.

An intervention-preserving result permits a local mechanism statement: within the declared boundary, domain and perturbation family, the target realises the tested causal role. That statement can justify a design transfer, a new perturbation or a controlled model substitution. It still does not establish that every causal property has been captured. A finite test can never enumerate an open environment without additional assumptions.

A failed result also carries permission. It permits the team to reject a specific bridge, narrow its scope or search for a different grain. It does not prove that the two domains have no useful relation. Failure at one measurement may reveal a better variable. Failure under one intervention may expose compensation elsewhere in the system. The registry converts both success and failure into disciplined next moves.

Worked example 3: a configured vision comparison

Neuroscience benefits when behaviour constrains the question rather than appearing only as a final readout. Krakauer and colleagues argue that increasingly precise neural tools do not remove the need for careful behavioural decomposition.¹⁶ In a convergence study, the same rule applies twice: the machine task and biological behaviour must each be independently understood before their internal measurements are aligned.

Worked example 4: when the comparison fails well

Executable lab: test the bridge before naming it

The lab below has two parts. The parity assay exhaustively tests whether a proposed hidden-variable mapping survives interchange interventions. The registry builder then turns a prose analogy into a typed record. All data are synthetic.

A positive parity result permits a narrow statement: the proposed variable has the tested causal role under all enumerated swaps. A negative result lowers the claim to observational equivalence or motivates a different alignment. Neither result establishes consciousness, semantic content beyond the task, or equivalence outside the finite circuit.

const xor = (x, y) => Number(Boolean(x) !== Boolean(y));

function machineA({a, b, c}, injectedH = null) {
  const h = injectedH ?? xor(a, b);
  return {h, y: xor(h, c)};
}

function machineB({a, b, c}, injectedH = null) {
  const h = injectedH ?? xor(b, c);
  return {h, y: xor(a, h)};
}

function highLevelSwap(base, source) {
  const sourceH = xor(source.a, source.b);
  return xor(sourceH, base.c);
}

// Invariant for the proposed mapping:
// for every base/source pair, lowLevelSwap.y === highLevelSwap(base, source)

An open hypothesis with a lowering condition

The strongest boundary

Even a perfect causal mapping can be local. Two systems may instantiate the same finite-state transition graph while differing in learning, development, embodiment, temporality and relation to a subject. Formal equivalence at one model level may coexist with ontological disagreement. Conversely, an imperfect mapping may still be scientifically fruitful if it identifies where biological, computational and experiential descriptions pull apart.

The bounded thesis therefore rejects two extremes. It rejects “everything is merely metaphor”, because relations and interventions can be tested. It also rejects “the equation is shared, therefore the phenomenon is shared”, because a formal object acquires explanatory meaning through its realisation, boundary and use.

sequenceDiagram
  participant Q as Question
  participant M as Mechanism
  participant E as Evidence
  participant D as Decision
  Q->>M: The resemblance trap
  M->>E: The hidden step from “like” to “is”
  E-->>M: compare rival explanations
  E->>D: Let convergence earn only the next experiment
  D-->>Q: revise scope or act

Let convergence earn only the next experiment

The next time a machine, brain process, lived structure and contemplative category appear to converge, do not decide first whether the analogy is profound or absurd. Decide what experiment the match earns.

Strip away the shared noun. Write the source and target structures. Freeze the boundary. Name one relation. Choose a matched intervention. State the break condition. Add the forbidden inference. Then run the smallest test that could lower the claim.

For engineers, this changes architecture transfer. A biological or contemplative idea should enter a design only through the relation it contributes and the failure it predicts. For neuroscientists, it changes model comparison. A high brain score or representational fit becomes a starting constraint, not a mechanism certificate. For phenomenologists and contemplative researchers, it protects first-person structure from both dismissal and premature reduction. For philosophers, it separates empirical bridge-building from the metaphysical arguments that remain necessary.

The four maps can stay on the light table. Their matched bays may reveal a general control principle, a useful measurement bridge or a new experiment. Their unmatched coasts are equally important. They show where history, embodiment, purpose, first-person appearance and metaphysical interpretation refuse to collapse.

This decision changes publication language too. A title should say “shared representational geometry under task X” rather than “machines think like brains” when that is all the evidence supports. A diagram should show omitted relations. A null result should remain visible. A product design should carry the same boundary into requirements, tests and user claims. The discipline is not cosmetic. It changes which systems are built, which experiments are funded and which risks are taken seriously.

The goal is not to make every map one map. It is to know exactly where a crossing can bear weight.