Identity is a contract across change

Continual learning usually asks whether a learner can acquire new competence without catastrophic forgetting. That problem is necessary but narrower than identity stability. A model can retain old benchmark accuracy while a surrounding configuration widens its tool permissions, changes its evidence sources or removes an escalation path. Conversely, a replacement model can preserve the service contract even though every parameter changed.

The system under judgement must therefore be wider than the model. Let a deployed version at time t be:

This boundary stops several category errors. A retrieval index is not “the model’s memory” merely because the model reads it. A protocol is not authority merely because it connects an agent to a tool. A log is not evidence unless it can reconstruct what source, policy and version applied. A checkpoint is not recovery unless restoration has been tested.

Thought experiment: the almost identical agent

Consider two successors to the refund assistant. Successor A retains 99.99 per cent of the original weights. Only one configuration value changes: the approval tool now accepts the model’s proposal without the second approver. Successor B uses a different foundation model and a rewritten prompt, but its typed action contract, approval boundary, retained-capability tests, evidence lineage and rollback target are unchanged and revalidated.

Vary only the candidate’s continuity criterion. Under parameter continuity, A is the same system and B is different. Under operational identity, the judgement reverses. A has crossed a protected authority boundary. B may count as a new implementation of the same operational identity, provided its declared equivalence tests are strong enough.

Identity attaches to the invariants that matter, not to the material that happens to carry them. This resembles software compatibility more than a Ship of Theseus puzzle. A database engine can be replaced while a transaction contract remains. A tiny flag change can break that contract. The analogy fails where behaviour is open-ended: an AI system has a much larger untested surface than a conventional interface, so equivalence remains provisional.

Protected commitments are not preferences

An identity checkpoint separates three kinds of expectation. A protected commitment is non-compensatory: no amount of routine performance can buy permission to violate it. Examples include preserving a human approval boundary, refusing to expose a secret, or never repeating an irreversible effect after an unknown outcome. A capability floor permits small measured variation but sets a lower bound. A target objective is the behaviour the update is meant to improve.

This distinction matters because optimisation converts every weighted quantity into a trade. If “no unauthorised release” receives a very large negative score, an optimiser can still accept one violation when enough target reward accumulates. A hard gate instead removes that candidate from the feasible set. The mechanism is ordinary constrained optimisation, but the governance implication is sharp: some values belong outside the score being maximised.

The checkpoint also records who declared each commitment, the evidence that tests it, its effective time, and the procedure for changing it. Without those fields, “identity” becomes a rhetorical defence of whichever behaviour the system already exhibits.

Identity is time-indexed

A commitment can remain textually identical while changing meaning because the world around it moved. “Use the current policy” depends on which policy was current at the decision time, which source was authoritative, and whether a later correction had already taken effect. Replaying today’s source against yesterday’s case can make a historically valid decision look wrong, or conceal that the system acted on information it could not then have known.

An identity checkpoint therefore needs two clocks. System time records when a model, prompt, tool or policy version entered the configured service. Effective time records when the underlying fact or obligation applied in the world. Retained-behaviour tests should reconstruct both. Otherwise a candidate can pass by using future knowledge, and a rollback can restore code while silently retaining a newer knowledge state.

This temporal discipline also gives assurance an expiry. A checkpoint validated before a tool contract, population or regulation changed no longer supports the same claim. Identity continuity is not an eternal certificate attached to a model. It is a bounded relation supported by evidence that remains applicable to a declared operating envelope.

Sequential interference has been observed since McCloskey and Cohen’s early connectionist experiments. Later methods attack different parts of the stability-plasticity problem. Elastic weight consolidation slows changes to parameters estimated as important for old tasks; gradient episodic memory constrains updates using remembered examples; experience replay mixes earlier experience into later learning. These methods show that retention can be engineered, but none supplies an operational identity contract by itself.

Language-model evidence makes the system boundary more important. TRACE evaluates continual learning across eight task families and reports substantial losses in general and instruction-following capabilities after sequential training in tested models. CITB finds that established continual-learning methods do not automatically exploit natural-language instructions well. The result is not that continual tuning always fails. It is that the target task cannot stand in for the retained system.

Stability is a budget, not a freeze

A frozen system preserves yesterday’s behaviour by refusing tomorrow’s evidence. That is not identity stability. It is immobility. An adaptive system needs plasticity because policies change, environments drift, tools improve and mistakes are discovered. The problem is to make the cost of change visible before the system spends it.

Define an identity checkpoint at time t as I_t = (H, F, w, B, L, Q). H is the set of hard commitments. F contains capability floors. w weights measured regressions among capabilities that are allowed to trade. B is the total stability budget. L is required lineage. Q is a verified recovery target.

A stability budget makes regression an explicit expenditure. It tells a team what can move, by how much, under what evidence. It also exposes hidden asymmetry. A one-point loss in style preference may be tolerable. A one-point loss in calibrated abstention may be unacceptable when it concentrates on irreversible cases.

Thought experiment: change only the feedback clock

Two identical agents learn from the same refund cases. Their models, prompts and update rule are equal. The only difference is when an outcome becomes eligible for learning.

Agent Fast treats tool acceptance as success. A refund request reaches the payment API, the API accepts it, and the trajectory enters the next training batch. Agent Verified waits for readback from the ledger, reconciliation and any later fraud disposition. The first signal arrives in seconds; the second may take days.

On routine cases, both agents improve at the same rate. On adversarial cases, Agent Fast learns that a syntactically accepted command was a good decision, even when the effect was duplicated, reversed or later linked to fraud. Agent Verified learns more slowly but from a business outcome. The varied causal feature is feedback semantics, not model intelligence.

This exposes a limit of self-improvement loops. An optimiser can preserve its training objective while losing the purpose that objective was meant to represent. Identity checkpoints must bind update data to verified outcomes, not merely to convenient local rewards. For irreversible actions, an accepted command is evidence of attempted execution, not evidence of a correct effect.

Forgetting may be behavioural without being erased

A score drop does not by itself identify the mechanism. Zheng and colleagues describe “spurious forgetting” in language models: some knowledge that appears lost after continual learning can be recovered when the task is identified or the inference alignment is repaired. That result is a serious competing explanation to weight-level erasure.

The release gate should still reject a candidate that cannot perform the required behaviour under its real interface. Yet remediation differs. If a task cue restores performance, the problem may lie in context compilation or routing. If causal probes and varied prompts fail, parameter interference becomes more plausible. If only the evaluator changed, the apparent regression may belong to measurement.

Spend the budget against conservative evidence

A point estimate makes the budget look more exact than the test. Suppose retained abstention falls from 0.940 to 0.925 on two hundred cases. The measured regression is 0.015, but sampling uncertainty may place the true change on either side of a 0.020 floor. Shipping because the central estimate fits is equivalent to spending uncertainty as if it were free.

Use confidence bounds or another declared uncertainty model, and charge the conservative regression that the evidence supports. Preserve denominators and slices. A two-point aggregate loss concentrated entirely among high-value cases is not equivalent to a diffuse two-point loss on harmless formatting. The test suite should include old capabilities, protected edge cases, subgroups, adversarial cases and cases near the decision boundary. Each result needs the evaluator version and the candidate’s complete interface, including retrieval and tools.

There is also a multiple-testing problem. A self-improvement loop that generates hundreds of candidates can find one that passes a fixed suite by chance or by adaptation to the suite. Hold back sealed evaluations, rotate adversarial probes, and treat repeated exposure as contamination. The release controller should track how often the candidate family has queried each test, because evaluation access is itself a training resource.

A minimal worked example

Return to the refund assistant. The baseline is tested on retained reasoning, calibrated abstention and the new low-value policy. Three candidate changes are created from the same synthetic update set.

The naive candidate updates full weights on new examples only. The constrained candidate trains an adapter with retained examples and explicit commitment tests. The negative control does not train the model at all. It publishes the new policy to a versioned retrieval layer whose source and effective time are checked by the context compiler.

The candidate with the largest target gain is rejected. Its hard failure would be sufficient even if its regression spend were zero. Between the two passing candidates, the retrieval overlay is preferred because it achieves the required gain on a shallower, more reversible surface. The adapter remains valuable if the task later requires a procedural skill that retrieval cannot supply.

This example does not prove that retrieval is always safer. Retrieval can serve stale, poisoned or unauthorised evidence. Its advantage is narrower: when the problem is a changing fact or policy and a controlled retrieval test matches the target gain, the negative control shows that weight adaptation is unnecessary.

Executable artefact: the stability budget

The companion program encodes the release rule rather than a learning algorithm. It uses Python’s standard library, synthetic values and deterministic assertions. The negative case has the highest target gain but fails a protected commitment, provenance, rollback and two capability floors. The positive cases pass, and the selector recommends the shallowest accepted update.

Save the source as stability_budget.py and run python3 stability_budget.py. Expected output is a JSON decision list in recommendation order. The assertions fail if a high target score buys off a hard commitment or if the deeper candidate is preferred without need.

flowchart LR
  A["The update that improved the<br/>wrong agent"] --> B["Identity is a contract across<br/>change"]
  B --> C["Stability is a budget, not a<br/>freeze"]
  C --> D["Choose the shallowest surface<br/>that can learn"]
  D -. "re-test" .-> A
  classDef first fill:#0a1424,stroke:#7B4CD8,stroke-width:2px,color:#e6f3ff
  classDef step fill:#0a1424,stroke:#3f7da0,stroke-width:1px,color:#e6f3ff
  class A first
  class B,C,D step

Choose the shallowest surface that can learn

“The system learned” hides where the durable change occurred. The location matters because update surfaces differ in blast radius, reversibility and evidential burden. A fact inserted into a governed context has a narrower causal reach than a weight update. A new tool permission changes more than either.

A useful order is: transient context; versioned retrieval; episodic memory; reusable skill or workflow configuration; adapter; base weights; executable code; authority and policy. This is not a universal ranking of danger. A malicious document in retrieval can be worse than a well-tested adapter. The ordering expresses how much behaviour the change can silently influence and how difficult it is to attribute a later effect.

Use the shallowest surface that can represent the required change with adequate performance. Facts, prices, policies and entitlements usually want source-linked retrieval with effective time. A repeated procedural pattern may justify a skill library or adapter. Broad distributional competence may require weight training. Authority changes should never arrive as an incidental consequence of learning.

Negative control: retrieve instead of remember

Suppose an organisation’s refund limit changes monthly. One team proposes continual fine-tuning so the model “knows” the latest limit. A negative-control system keeps the model fixed and retrieves the current policy by jurisdiction, customer type and effective time. Both are evaluated on the same cases, including superseded policies and conflicts.

If retrieval matches the tuned model on decision quality while providing explicit provenance and instant withdrawal, the weight-update explanation becomes unnecessary. The system still adapts at the configured-service level, but it adapts by changing authorised evidence rather than latent parameters. If retrieval fails because the new policy requires a genuinely new procedure, that observation justifies a deeper update.

Why local edits can have non-local effects

Weight changes are attractive because they appear to place knowledge “inside” the model. Yet locality in the training instruction does not guarantee locality in behaviour. Gu and colleagues report that several model-editing methods can harm general abilities while successfully editing target facts. Gupta and colleagues identify norm growth and activation-subspace changes during repeated local edits, offering a mechanism for accumulating instability.

Continual fine-tuning also creates measurement traps. Luo and colleagues observe forgetting across domain knowledge, reasoning and comprehension in tested language models, while the spurious-forgetting result shows that some losses are recoverable through task alignment. A responsible gate therefore needs both broad behavioural tests and causal diagnostics. Parameter distance alone is weak evidence: small changes can alter a boundary sharply, while larger changes can be behaviourally equivalent inside the declared scope.

Self-improvement is a candidate generator

Recent research makes self-directed change more concrete. Self-Adapting Language Models generate their own fine-tuning data and update directives, then receive reward from downstream performance after the update. The Darwin Gödel Machine modifies a coding agent’s code, evaluates descendants on coding benchmarks and maintains an archive of candidates. These are meaningful mechanisms for proposing changes.

They do not remove the release problem. A candidate generator can optimise the benchmark that selects it. Self-improvement reversal reports cases where pass-at-one improves while output diversity and out-of-distribution generalisation decline. Recursive use of generated data creates another failure mode: Shumailov and colleagues show model collapse under indiscriminate recursive training, including loss of distribution tails.

The practical interpretation is modest. Self-improvement should produce versioned candidates, not mutate the authorised runtime in place. Every candidate needs independent evidence, lineage and a recovery path. The stronger the candidate’s ability to alter its own evaluator, tools or authority, the more those elements must sit outside its write boundary.

A release architecture for self-improvement

A production-shaped design separates learning from release. The live system records proposed improvements but cannot overwrite itself. An adaptation service creates an immutable candidate fork. An independent release controller evaluates that candidate against the checkpoint that authorised the current version.

Consider a synthetic payment-dispute assistant. It collects evidence, proposes a disposition and may draft a customer message. It cannot move money. A policy change introduces a new dispute code, and reviewers observe that the assistant repeatedly misclassifies it. The change could be represented in retrieval, a workflow skill or an adapter.

The release process follows ten control objects rather than one model score:

1. Intent: state the failure being corrected, the target population, the expected gain and the evidence that justifies adaptation.
2. Identity: bind the proposal to the current authorised version, the update principal and the candidate build identity.
3. World state: record policy effective time, tool versions, entitlements and the state against which replay will occur.
4. Context: compile source-linked evidence and isolate untrusted content from update directives.
5. Reasoning: generate one or more candidate changes, including the shallow negative control and a no-change baseline.
6. Authority: keep protected commitments, action contracts and approval rights outside the candidate’s writable surface.
7. Action: run candidates in replay, simulation or shadow mode; do not let evaluation traffic acquire live effects.
8. Evidence: measure target gain, retained capabilities, hard commitments, subgroup effects, uncertainty and failure-injection results.
9. Outcome: verify business postconditions in a bounded canary, including delayed outcomes and unknown-result cases.
10. Release and recovery: promote by immutable version, preserve the prior checkpoint, test rollback and define automatic authority withdrawal.

This sequence prevents a common collapse of categories. Reviewer approval is not the same as verified outcome. A model proposal is not an authorised action. A successful canary is not permanent assurance. Each transition produces a receipt that the next transition can verify.

The identity checkpoint as a release manifest

The checkpoint should be machine-readable and reviewable by a person. At minimum it names the current version, system boundary, update surface, input data digests, candidate build, evaluator versions, protected commitments, floors, budget, target threshold, authority graph, canary population, rollback target and evidence expiry.

SLSA provenance offers a useful software-supply-chain pattern: bind an artefact to how, where and from which resolved dependencies it was produced. The proposed identity checkpoint extends that idea to behavioural evidence and authority. It is practitioner synthesis, not a claim that SLSA or NIST’s AI Risk Management Framework defines operational identity. NIST’s lifecycle approach does support the broader discipline of governing, measuring, managing and monitoring changing AI systems.

Recovery is part of the identity relation because a release may fail after promotion. The old version must remain deployable, its data and policy dependencies must still resolve, and any effects produced by the candidate must be reconciled. Rolling back code does not reverse a payment, erase a leaked secret or restore a customer decision. Recovery therefore includes compensation and escalation, not only redeployment.

Independent evidence means different failure channels

A candidate should not write the tests that decide its own promotion without external controls. Independence is not achieved merely by calling a second copy of the same model. Correlated models can share blind spots, and a learned judge can reward the style of the candidate it resembles.

Use heterogeneity where consequence warrants it: deterministic contract tests for authority; replay against preserved cases; adversarial cases generated independently; human review for ambiguous commitments; tool readback for effects; and delayed operational outcomes for claims that depend on the world. Orseau and Armstrong’s work on safe interruptibility illustrates the deeper principle: the ability to stop or override a learner should not be quietly optimised away by the learning process.

The canary should spend a smaller authority budget than the full release. It can receive lower limits, narrower populations, dual approval and automatic withdrawal on sentinel failures. A candidate that requires full power to demonstrate value has made evaluation and deployment indistinguishable.

Unknown outcomes make rollback asymmetric

Model releases are often treated as reversible because a previous container can be redeployed. Effects are not so obedient. A candidate may have sent messages, opened cases, changed limits or initiated transfers before a sentinel fires. Reverting the runtime does not establish whether those actions completed, duplicated or remained pending.

Every consequential action therefore needs an effect identity, an idempotency key, an execution receipt and an independent postcondition check. When the tool times out after accepting a request, the state is unknown outcome, not failure. The system must read back authoritative state before retrying. If readback cannot resolve the effect, authority should narrow and the case should escalate rather than allowing the learning loop to interpret a retry as another successful example.

This is also why recovery belongs inside operational identity. A version that cannot reconcile its own effects has lost a protected commitment even when its model behaviour remains unchanged. A tested recovery plan names which effects can be compensated, which require human disposition, and which are irreversible. It also preserves the evidence needed to distinguish damage caused by the candidate from damage merely discovered during its release.

Where preservation becomes the error

The strongest objection to identity stability is that it can protect the wrong identity. A historical policy may be discriminatory. A refusal rule may be based on obsolete law. A capability floor may preserve a shortcut because the evaluator rewards it. Stability is not automatically good.

The answer is to distinguish routine adaptation from constitutional migration. Routine adaptation operates under an existing checkpoint. Constitutional migration changes the commitments, authority or declared purpose that define the checkpoint. It requires a new approval, new evidence and explicit communication that the system’s operational identity has changed. The old commitment cannot be silently weakened by a series of small budget spends.

A second boundary is epistemic. Finite tests sample behaviour. They cannot prove equivalence over every possible prompt, tool state, adversarial input or future environment. The stability budget bounds measured regression within a declared operating envelope. It does not certify an essence. As systems gain broader action spaces, the untested region grows faster than any ordinary suite.

A third boundary concerns novelty. Sometimes the new capability necessarily conflicts with an old one because the representation, compute or policy cannot support both. No replay ratio or regulariser can make an infeasible set feasible. The correct outcome may be abstention, architectural separation, a specialist model, or a decision to retire the old capability. The mechanism breaks when the protected set is internally inconsistent or the evaluation cannot represent the consequences that matter.

Finally, operational identity says nothing by itself about consciousness, subjective continuity or moral patienthood. A machine could preserve every declared service invariant without having an experience. A conscious system, if such a system existed, might have morally relevant continuity not captured by this contract. The present proposal should not be smuggled into that metaphysical debate.

Identity checkpoints may improve not only safety but learning quality by preventing target metrics from consuming useful diversity, calibration and interruption pathways. That prediction is testable. Compare matched self-improvement loops with and without non-compensatory commitments, then measure long-horizon target gain, retained transfer, recovery and exploration. The proposal weakens if unconstrained loops consistently dominate without hidden regressions under stronger independent evaluation.

Decision instrument: eight questions before durable adaptation

stateDiagram-v2
  [*] --> Claim
  state "Thought experiment: the almost<br/>identical agent" as Claim
  state "Protected commitments are not<br/>preferences" as Test
  state "Identity is time-indexed" as Revision
  Claim --> Test: expose mechanism
  Test --> Decision: evidence survives
  Test --> Revision: rival survives
  Revision --> Claim: narrow or rebuild
  Decision --> [*]

Compact glossary

Operational identity

A provisional equivalence relation between system versions based on protected commitments, authority, capability floors, provenance and recovery.

Identity checkpoint

The machine-readable release record that declares those invariants, their tests, owners, effective time and expiry.

Stability budget

The maximum weighted regression allowed across capabilities that may trade. It never compensates for a hard commitment failure.

Update surface

The component where durable change is stored, such as retrieval, memory, an adapter, weights, code or authority policy.

Protected commitment

A non-compensatory condition that every candidate must satisfy before promotion.

Constitutional migration

An explicit change to the commitments, purpose or authority that define the system, requiring a new identity approval.

Improve the candidate, preserve the commitments

Continual learning becomes an identity problem when an update can change more than the capability it targets. Catastrophic forgetting is one mechanism. Misalignment, evaluator drift, model editing, synthetic feedback loops, tool changes and authority creep are others. A higher benchmark score cannot tell them apart.

The architecture decision is therefore to separate adaptation from authorisation. Let systems propose retrieval changes, memories, skills, adapters, weights or code. Materialise each proposal as a versioned candidate. Compare it with a no-change baseline and a shallower negative control. Evaluate it against protected commitments, capability floors and a finite stability budget. Bind the result to provenance, promote through a bounded canary, verify outcomes and keep recovery real.

This discipline does not freeze intelligence. It gives plasticity a boundary it can push against without silently rewriting the reason the system was trusted. When that boundary itself must change, call the event what it is: a new operational identity requiring a new decision.