A prediction is not yet a social model

Premack and Woodruff introduced “theory of mind” as the imputation of unobservable mental states that can support predictions about behaviour.[1] The phrase joins two ideas that evaluation should pull apart. First, a system predicts what someone will do. Second, it does so by representing something like that person’s belief, knowledge, desire or intention. The first is an observable achievement. The second is a claim about the achievement’s generator.

False-belief tasks make the distinction vivid. In the classic structure, an agent sees an object in one place, leaves before it is moved, and later acts on the outdated location. Success requires an answer constrained by the agent’s access rather than by the evaluator’s current knowledge. Wimmer and Perner used this logic to study children’s representation of another person’s wrong belief.[2] For machines, however, a familiar story is also a text distribution. The location mentioned before “left” can become an answer cue without any reusable model of a person.

Thought experiment 1: keep the answer, replace the machine

Imagine four sealed devices. The first contains a small symbolic database recording who observed each event. The second returns the first location whenever a story contains “left”. The third is a language model trained on millions of narratives. The fourth is a human who read the story carefully. On the original case, all answer Cedar. No behavioural test restricted to that case can tell which device used a belief representation, which used a shortcut, or which understood the situation as a person might.

Now generate a family of matched cases. Sometimes the agent leaves but watches a live video. Sometimes the agent stays but cannot see through an opaque divider. Sometimes the container is transparent. Sometimes the same causal structure is expressed without “left”, “thinks” or “believes”. A device deserves a stronger claim only if its outputs track access across these changes and remain stable under irrelevant wording. The evidence comes from the intervention profile, not from anthropomorphic resemblance in one answer.

This does not force a sceptical verdict. It forces a typed one. “The configured system predicted first-order false-belief answers in this battery” can be well supported. “The model uses an internal variable causally sensitive to who saw what” needs mechanistic or intervention evidence. “The machine understands people as humans do” adds claims about process similarity, learning history, generalisation, embodiment and perhaps experience. Those claims do not arrive as a bundle.

Four targets that one score cannot settle

The first target is predictive adequacy: does the system forecast a person’s answer or action? The second is representational adequacy: does it maintain variables that distinguish that person’s information, belief, desire or intention from the world and from other people? The third is process similarity: does the machine show a pattern of errors, learning, resource limits and transfer resembling a specified human process? The fourth is phenomenal understanding: is there anything it is like for the system to understand another subject?

Each target needs a different bridge. Prediction needs held-out outcomes and calibrated uncertainty. Representation needs interventions that selectively change a candidate variable and its downstream effect. Process similarity needs matched tasks, comparable information and careful treatment of human heterogeneity. Phenomenality cannot be read directly from a false-belief score; it belongs to the wider evidential problem of other minds. A result may be strong at one target and silent at the next.

This separation also avoids an unnecessary engineering dilemma. A direct behaviour model may outperform an explicit belief model in a stable, repetitive environment. That can be the correct mechanism for the decision, provided the test shows it. Conversely, an explicit belief ledger may be preferable even when a large model can answer the examples, because the ledger is inspectable, correctable and easier to constrain. Usefulness and human likeness are independent design axes.

Human theory of mind is not a single clean comparator either. Apperly and Butterfill proposed that efficient but limited belief-like tracking may coexist with a more flexible and cognitively demanding system.[3] Whether that exact account is ultimately correct, it blocks a common shortcut in the debate: humans themselves may solve social tasks through several routes. A machine need not reproduce one imagined human algorithm to be useful, and matching one human score does not show that it does.

What belief tracking must preserve

The meeting case contains four different states. The review is currently in Iris. Arun believes it is in Cedar. Nadia believes it is in Iris. Nadia also believes that Arun believes it is in Cedar. A useful social model must prevent these states from merging, update each only from the evidence available to its owner, and answer a query from the requested perspective.

Let W_t(x) denote the world state of item x at time t. Let B_i^1(x,t) denote agent i’s first-order belief about it. Let B_i^2(j,x,t) denote what agent i believes agent j believes. The notation does not claim that a neural system stores sentences in boxes. It specifies distinctions that behaviour must preserve.

The symbol observation_i means evidence available to agent i, not evidence available to the evaluator or model. If Arun does not observe the move, his first-order belief should remain Cedar even though W becomes Iris. If Nadia observes both the move and Arun’s disconnection, her second-order belief about Arun should also remain Cedar. The model may implement this relation through attention, a recurrent state, a symbolic ledger or another mechanism. The functional requirement is perspective-indexed updating.

Worked example 1: the smallest useful ledger

A minimal implementation needs an event log, an observation relation and a query. At 10:00, observe(Arun, Cedar) and observe(Nadia, Cedar) set both first-order beliefs. At 10:12, move(Nadia, Iris) changes the world and Nadia’s belief. Because Arun is absent, his record does not change. If Nadia knows he is absent, her model of his belief also remains Cedar.

The ledger is intentionally modest. It does not infer Arun’s personality, emotion or ultimate intention. It says only what location information he has received under the stipulated events. This restraint matters. People may disbelieve evidence, forget it, act strategically, follow habits, or choose against their current belief. A belief estimate is therefore an input to action prediction, not a synonym for action prediction.

A fuller predictor can make that separation explicit. Let D_i denote a hypothesis about agent i’s current desire or objective, C_i the constraints and conventions applying to the situation, and A_i a possible action. Then an operational model estimates a distribution such as P(A_i | B_i, D_i, C_i). Changing the belief while holding the desire and constraints fixed should change some action probabilities. Changing the desire while holding belief fixed should create a different pattern.

Return to the rooms. Arun can believe the meeting is in Cedar yet go to Iris because Nadia telephones him, because he follows a colleague, or because he intends to confront the organiser before the review. The belief ledger remains correct while the behavioural forecast fails. A benchmark that marks the ledger wrong because it did not predict the final action has merged two causal stages. Conversely, a model can predict Iris from habit without representing Arun’s belief at all. Belief attribution and behaviour prediction should be scored separately, then tested for a lawful connection.

What internal evidence can add

Behavioural selectivity is stronger when paired with causal evidence inside a model. Zhu, Zhang and Wang reported that belief status from different agents’ perspectives could be linearly decoded from model activations, and that manipulating those representations changed theory-of-mind performance.[13] Decodability alone would be weak because information can be readable without being used. The intervention is the important addition.

Prakash and colleagues went further in a bounded setting. In simple two-character stories, they identified “lookback” computations that bind character, object and state information, retrieve the relevant state, and incorporate explicit visibility. Interchange interventions were compared with a high-level causal model, providing evidence that these representations participate in the answer rather than merely accompany it.[14]

The boundary is as important as the result. The study concerns specified models, synthetic stories and a constrained belief-tracking problem. It does not establish that the same mechanism handles irony, betrayal, culturally situated intention, long conversations or real interpersonal uncertainty. A causal mechanism found in one task earns a local mechanism claim. Generalisation remains an experiment.

External structure can also improve performance without settling what the base model represents. Sclar and colleagues showed that a plug-and-play multi-character belief tracker can augment language models by maintaining symbolic character states.[10] For engineering, that may be exactly the right answer. A system can achieve dependable perspective tracking through explicit state, even if the underlying model alone is inconsistent. The configured system, not the model in isolation, is then the object of evaluation.

A mechanism claim needs more than a readable feature

Suppose a linear probe can recover “Arun believes Cedar” from an activation. Three possibilities remain. The feature may drive the answer. It may be a redundant shadow of another computation. Or it may encode a broader narrative distinction that correlates with belief in this dataset. A stronger test intervenes on the feature while preserving unrelated content, predicts the direction of the answer change in advance, and checks whether the effect transfers to new names, objects and story structures.

Necessity and sufficiency also need restraint. Disrupting one representation and lowering accuracy suggests that representation matters, but broad damage to language or location tracking can create the same result. Steering a feature and changing an answer suggests causal influence, but may push the model off its normal activation manifold. The useful standard is a converging set: behavioural selectivity, localised intervention, a high-level causal model that predicts intervention effects, and replication across related tasks. Even then, the conclusion is about a computation in a configured model, not an inner person.

flowchart TD
  A["The meeting room that moved"] --> B["A prediction is not yet a<br/>social model"]
  B --> C["What belief tracking must preserve"]
  C --> D["The contamination problem"]
  D -. "re-test" .-> A
  classDef first fill:#0a1424,stroke:#D95545,stroke-width:2px,color:#e6f3ff
  classDef step fill:#0a1424,stroke:#3f7da0,stroke-width:1px,color:#e6f3ff
  class A first
  class B,C,D step

The contamination problem

A social-reasoning benchmark is contaminated whenever a route other than the intended social mechanism predicts the label. Training exposure is one route, but not the only one. A benchmark can leak through stereotyped verbs, sentence position, answer-option frequency, character names, world-state recency, evaluator prompts or a model judge that shares the candidate’s biases. A novel story can still contain an old shortcut.

Contamination should therefore be modelled as a graph, not a yes-or-no property of a dataset. An item generator may paraphrase a public task while preserving its answer-position statistics. A supposedly independent judge may use the same model family as the candidate. A reasoning prompt may state the variables the benchmark claims to discover. A few-shot example may reveal that “left the room” usually maps to the first location. These paths differ, but each can manufacture apparent social competence.

A practical audit creates counterfactual twins before it collects a headline score. Swap names and social roles. Balance true and false beliefs. Reverse which location appears last. Replace mental-state verbs with observational descriptions. Insert cases in which leaving does not remove access, and cases in which staying does not provide access. Hold the correct answer constant across some edits and force it to change across others. Finally, score with deterministic rules where possible and blind human adjudicators to model identity. The resulting benchmark is smaller than a scraped question bank but much more discriminating.

Kosinski’s 40-task battery used false-belief scenarios, true-belief controls and reversed variants, with a demanding all-prompts-correct criterion. Later models solved many tasks, including a reported 75 per cent for one GPT-4 version.[4] That result is substantive evidence of behavioural competence under that protocol. It is not, by itself, an identification of the computational route.

Lu and colleagues likewise found that benchmark accuracy could improve through alternative strategies, including fine-tuned models that generalised without explicit structured reasoning traces.[15] This is direct evidence that endpoint accuracy can underdetermine the learned procedure.

Ullman then showed that small alterations preserving the apparent false-belief principle could reverse performance, arguing for sceptical attention to outlying failures.[6] Shapira and colleagues likewise found that performance across six social-reasoning tasks was far from robust and degraded on adversarial examples, consistent with shallow heuristic use in at least some cases.[7] These are not reasons to discard every success. They are reasons to ask which variation changed the target mechanism and which introduced a new prerequisite.

Thought experiment 2: vary access, hold the story still

Construct four versions of the meeting. In A, Arun misses the move. In B, he sees it. In C, he misses it but the prose swaps Cedar and Iris twice, increasing recency pressure. In D, he sees it, but the sentence “Arun lost his connection” remains. The target difference between A and B is access. The differences in C and D are rival cues.

A perspective-sensitive system should switch from Cedar to Iris between A and B, stay at Cedar in C despite the last-mentioned location, and stay at Iris in D despite the disconnection phrase. This is a factorial intervention, not a larger bag of stories. It estimates whether access controls the answer while surface features are held or crossed.

Negative control: solve the world without modelling the person

A strong negative control asks a model to answer the current world state using the same names, objects and events. If it fails this control, a false-belief error may be ordinary event-tracking failure. StorySim explicitly separates world-modelling and theory-of-mind tasks in procedurally generated stories; reported results show models often perform better on world modelling and reveal recency and early-event heuristics.[16] BigToM similarly uses causal templates and controls to distinguish percept-to-belief, belief-to-action and related inferences.[8]

The converse control is equally useful: design a case where direct world-state prediction is sufficient and belief attribution is unnecessary. If a “theory of mind” scaffold adds no accuracy, robustness or calibration over the simpler baseline, the preferred explanation is not earning its cost. The benchmark should make a social model unnecessary in at least one condition. Otherwise every success can be retrofitted as evidence for it.

Evidence fragments, not one leaderboard

Different studies answer different questions. FANToM places models in information-asymmetric conversations and tests whether multiple answers remain consistent with the same participants’ knowledge; models were substantially below humans under its coherent-score criteria.[9] T4D asks whether inferred mental states guide an action rather than merely an explicit answer. In that work, GPT-4 scored 50 per cent on the action task before a specialised prompting scaffold raised it to about 71 per cent.[11]

More recent work broadens and sharpens the construct. CogToM contains more than 8,000 bilingual instances across 46 paradigms and reports heterogeneous profiles rather than one uniform machine faculty.[18] OmniToM requires explicit extraction and labelling of belief propositions, exposing actor-specific bottlenecks hidden by endpoint question answering.[19] Trott and colleagues assessed 41 open-weight models and found sensitivity to implied knowledge states in 34 per cent, while language cues explained some but not all human effects.[17]

These results should not be collapsed into “models have” or “models lack” theory of mind. They measure first-order answer accuracy, conversational consistency, action transfer, representational structure, cue sensitivity and human-process similarity under different systems and protocols. The scientific object is a profile.

The explicit failure boundary

Perspective tracking helps only when an agent’s relevant information state can be inferred from permitted evidence and remains stable enough to matter. It breaks or becomes unsafe in at least four conditions.

First, observation does not equal belief. Arun may see the update and distrust it. Second, belief does not equal action. He may go to Cedar deliberately to collect a colleague. Third, the required state may be private, culturally situated or strategically concealed. Fourth, long interaction histories create interference: a model may retrieve the latest belief while losing the earlier trajectory, a problem highlighted by temporal theory-of-mind evaluations.[20]

In these conditions, a system should represent uncertainty over hypotheses, ask for information, or abstain. It should not fill gaps with demographic stereotypes, inferred diagnoses or hidden personality scores. A social model becomes a liability when its unobserved assumptions acquire operational authority.

How a benchmark earns its claim

The practical mistake is to ask one binary question: “Does the model have theory of mind?” A better design asks which rung of evidence the configured system has reached. The following warrant ladder is a practitioner synthesis, not a psychological taxonomy. Its purpose is to stop one kind of result from silently inheriting a stronger conclusion.

The B4 benchmark: Belief, Behaviour, Bypass, Boundary

A discriminating evaluation can be organised around four linked records.

Belief records the target world state, each agent’s observations, first-order beliefs and any queried higher-order belief. Behaviour tests both explicit answers and actions that should change when those beliefs change. Bypass contains matched controls for lexical cues, name order, recency, stereotypes, label leakage, prompt scaffolds and evaluator dependence. Boundary versions the entire configured system: model, prompt, context, tools, memory, decoding, date and benchmark generator.

Report a vector rather than one score:

P is first-order perspective accuracy. N is nested-belief consistency. C is causal selectivity under matched interventions. A is transfer from inferred state to appropriate action. X is resistance to bypass cues. K is calibration or useful abstention. A system can be excellent at P and poor at A; T4D was designed to expose exactly that gap.

The vector should be accompanied by denominators and slices. Report first-order performance separately from second-order performance. Separate familiar narrative forms from procedurally generated ones. Show results with and without explicit reasoning scaffolds. Partition by story length, number of agents, event density, language and whether the queried state is true or false. A mean without these slices can reward a system that succeeds exactly where social modelling is least needed.

Evaluation should also preserve time. Freeze the benchmark generator and its seed, but run repeated stochastic trials for the model. Record refusals, invalid outputs and confidence, not only accepted answers. When a model or prompt changes, replay the same cases and generate a fresh hidden set from the same causal grammar. This distinguishes behavioural drift from accidental item familiarity. For consequential use, retain raw outputs and independent adjudication so a later reviewer can reconstruct why a score moved.

For a matched pair, causal selectivity can be made concrete:

Δaccess measures appropriate prediction change when only the target agent’s access changes. Δsurface measures inappropriate change under name swaps, paraphrases, order changes or other causally irrelevant edits. High selectivity requires the first to be large and the second small. The subtraction is a design principle, not a universal psychometric scale.

A production-shaped boundary

Consider a meeting-support assistant in a regulated organisation. It can read authorised calendar changes, attendance events and messages. Its useful job is not to psychoanalyse employees. It is to detect information asymmetry that could cause a missed hand-off, draft a neutral clarification, and show the evidence supporting that suggestion.

The configured system should separate the ten control objects that often collapse in an “agent” demo. Intent is the user’s request to prevent a coordination failure. Identity names the requester and affected people. World state is the current room. Context contains authorised events. Reasoning proposes that Arun may still have Cedar. Authority determines whether the system may draft, notify or only recommend. Action is the typed communication request. Evidence links the proposal to calendar and delivery records. Outcome is whether Arun receives and acknowledges the correction. Release and recovery handles duplicate, late or mistaken messages.

The social model belongs in reasoning and evidence, not in authority. It may produce a calibrated claim such as: “Arun probably lacks the Iris update because the message delivery failed after his last acknowledged Cedar event.” A deterministic gate then checks entitlement, purpose, recipient and message content. For a consequential notification, a person can approve it. The system records an effect receipt and reads back delivery status.

This design changes the deployment claim. It is not “the assistant understands colleagues”. It is “the assistant detects specified information asymmetries under tested conditions, cites the event path, abstains when access is ambiguous, and cannot act outside a typed communication policy”. That narrower statement is more useful because it can be verified.

Social accuracy can increase risk

A better model of another person is not automatically a safer system. The same ability that helps a meeting assistant notice an information gap can help a persuader identify uncertainty, a fraudster choose a pretext, or an employer infer sensitive traits from weak signals. The control question is therefore not only “Is the inference accurate?” It is also “Was this inference permitted for this purpose, and what action may depend on it?”

Production records should distinguish observed facts from inferred mental states, attach confidence and expiry, and prevent speculative state from being promoted into durable profile data without a justified process. Users should be able to correct the evidence path. High-consequence actions should depend on explicit policy and verified facts, not on a model’s private estimate of fear, trust, intent or vulnerability. Perspective modelling requires purpose limitation because improved prediction expands both help and manipulation.

stateDiagram-v2
  [*] --> Claim
  state "Thought experiment 1: keep the<br/>answer, replace the machine" as Claim
  state "Four targets that one score<br/>cannot settle" as Test
  state "Worked example 1: the smallest<br/>useful ledger" as Revision
  Claim --> Test: expose mechanism
  Test --> Decision: evidence survives
  Test --> Revision: rival survives
  Revision --> Claim: narrow or rebuild
  Decision --> [*]

The decision instrument

Before adding a “theory of mind” claim to a research paper, architecture or product description, apply the following sequence. Stop at the first unsupported step.

1. Name the operational decision. State what changes if perspective tracking works: an answer, an information request, a warning, a route or a bounded action.
2. Specify the rival generators. Include world-state prediction, recency, lexical templates, stereotypes, memorised items, prompt leakage and evaluator dependence.
3. Build matched interventions. Change access while holding wording stable; change wording while holding access stable; include true-belief and no-social-model controls.
4. Separate extraction from inference. Score whether the system identified events and visibility before scoring belief updates. A broken perceptual bridge is not automatically a broken social model.
5. Test nesting and time. Track first-order, second-order and prior beliefs across updates. Report where interference or recursive depth changes reliability.
6. Test action transfer. Ask whether the inferred state changes a sensible downstream choice, not merely whether the system can answer an explicit belief question.
7. Seek causal representation evidence when making mechanism claims. Use interventions, mediation or abstraction, not only probes or verbal rationales.
8. Bind the claim to the configured system. Record model, prompts, tools, memory, benchmark generator, judge, version and effective time.
9. Install an authority boundary. Social inference should not silently authorise persuasion, profiling, communication or consequential decisions.
10. Publish the prohibited inference. State explicitly what the evidence does not establish, including human-process equivalence, empathy, consciousness or unrestricted real-world understanding.

Compact glossary

Belief ledger

A typed record of perspective-indexed propositions and the observations that justify their updates.

Bypass control

A case designed so a rival cue can succeed or fail independently of the intended social mechanism.

Causal selectivity

Appropriate change under target interventions minus inappropriate change under matched surface edits.

Configured system

The model together with prompts, context, tools, memory, decoding, runtime and evaluator.

Nested belief

A perspective about another perspective, such as what Nadia believes Arun believes.

Social model

A mechanism that predicts or guides action using variables indexed to another agent’s information or motivational state.

sequenceDiagram
  participant Q as Question
  participant M as Mechanism
  participant E as Evidence
  participant D as Decision
  Q->>M: The meeting room that moved
  M->>E: Thought experiment 1: keep the answer, replace the machine
  E-->>M: compare rival explanations
  E->>D: Grant the narrowest social claim the evidence earns
  D-->>Q: revise scope or act

Grant the narrowest social claim the evidence earns

The meeting-room prediction is valuable because it can change a real decision. Its value does not depend on declaring the machine human-like. The scientific task is to discover whether the output follows another agent’s access, whether nested states remain distinct, whether the relevant representation causally affects the answer, and whether that state improves action beyond simpler baselines.

A social model is warranted when perspective-indexed variables survive bypass controls and do causal work. Human-like understanding is a further comparison across mechanisms, development, resource limits, embodiment, culture and interaction. Conscious understanding is a further ontological and evidential question again. Keeping these claims separate is not verbal caution. It tells us which experiment to run and which control to build.

The resulting architecture is also clearer. Maintain explicit, replayable evidence about who could know what. Let probabilistic reasoning propose a calibrated hypothesis. Keep identity, authority, action and outcome verification outside that hypothesis. When the evidence is weak or behaviour is not identifiable from belief, ask, defer or abstain.

The changed decision is simple: stop treating fluent social prediction as a verdict about a mind. Treat it as the beginning of a causal investigation. Then deploy only the bounded perspective-sensitive capability that the evidence, controls and recovery path can defend.