Part one

The endpoint erases the path

Imagine a second pair of facilities. Both end with a limit of 500 and drawn amount of 200. In one, the customer drew 600, breached the limit, repaid 400 and received a later limit increase. In the other, the customer drew 200 once. Hold the endpoint constant while varying only the path. Current exposure is equal; control breaches, fees, notifications and explanations are not.

This thought experiment separates three questions that are often collapsed into “give me the customer state”. A current-state question asks what values hold now. An as-of question asks what state the system represented at an earlier frontier. A causal-history question asks which accepted changes produced that state and why. A latest row can answer the first. A system-versioned table can often answer the second. The third needs semantics that row versions may never have recorded.

The reconstruction contract

Let S₀ be an initial state, E = [e₁, e₂, …, eₙ] a change sequence, and rᵥ a reducer at version v. Reconstruction is the left fold:

The symbols are simple. order(E) selects the sequence appropriate to the question. The reducer rᵥ(S, e) returns the next state after event e. Increasing n moves the frontier forward. Changing the order, omitting one event or changing reducer version may change the answer even when all event payloads remain untouched.

The formula earns its keep only under a contract. The entity key must remain stable, or be linked through an explicit identity event. Capture must be closed over every state-relevant mutation inside the declared boundary. The order must preserve source or causal constraints that affect the result. Events must retain enough schema and operation meaning to interpret them. The reducer must be deterministic for the same inputs, or its external dependencies must be frozen and versioned.

I call this reconstruction closure: for a named entity, purpose and time frontier, every change capable of altering the requested state is represented, interpretable and applied by a reproducible transition rule. Closure is never universal. A facility stream can be closed for limit and utilisation while remaining open for customer identity if identity changes live elsewhere.

Identity is itself a state transition

Stable ordering is useless if the key stops naming the same thing. Suppose facility F-17 is migrated from a legacy account number to a new contract identifier. A consumer that treats the new key as a new entity loses the earlier drawdown history. A consumer that silently joins both keys risks combining two facilities that happened to reuse an old number. The history needs an explicit identity transition such as FacilityRekeyed, with old key, new key, effective frontier and authority for the linkage.

Merges and splits are harder. When two customer records are resolved as one person, the operation does not prove that every historical event should be rewritten under one key. Some decisions were made while the system still regarded the records as distinct. Conversely, a later split can invalidate a merged projection without making the original evidence disappear. A defensible model therefore keeps immutable event identity separate from mutable entity resolution. It can answer both “which record received this event?” and “which entity does current resolution associate with it?”

This distinction prevents a common reconstruction error: applying today’s identity graph to yesterday’s decision and claiming that the old system knew the same relationship. Identity lineage must be replayed or selected at the same temporal frontier as the state it qualifies.

A snapshot and a stream must meet at one frontier

Most CDC pipelines do not begin at the first transaction in the database’s life. They bootstrap from a snapshot and continue from the transaction log. Debezium connectors coordinate snapshot records with a recorded log position so that subsequent streaming can resume from a known point.^[3][4] The exact mechanism varies by connector, but the evidential requirement does not: the baseline and suffix must meet without a gap or overlap.

Imagine that the snapshot reads facility F-17 at limit 500. While the scan is still running, a transaction changes the limit to 250. If streaming starts after that transaction but the snapshot row was read before it, the update can be lost. If the snapshot already includes 250 and the same log event is replayed again without idempotence, it can be duplicated. A usable baseline therefore names the capture frontier against which its rows are consistent, and the suffix begins strictly after that frontier.

A checkpoint reduces later cost by replacing a proven prefix with a materialised state. It must carry the last included source position, event schema frontier, reducer version and a state checksum. Without those fields, replay from the checkpoint can duplicate a change, skip a change or combine state produced by one reducer with suffix events interpreted by another. A snapshot without a source frontier is inventory, not a replay base.

Part two

flowchart LR
  A["Two identical rows, two<br/>different histories"] --> B["The endpoint erases the path"]
  B --> C["CDC observes mutation, event<br/>sourcing defines history"]
  C --> D["Four clocks and a partial order"]
  D -. "re-test" .-> A
  classDef first fill:#0a1424,stroke:#087A98,stroke-width:2px,color:#e6f3ff
  classDef step fill:#0a1424,stroke:#3f7da0,stroke-width:1px,color:#e6f3ff
  class A first
  class B,C,D step

CDC observes mutation; event sourcing defines history

Change data capture and event sourcing both produce streams, which makes them easy to confuse. Their causal positions differ. CDC observes changes written to a database. PostgreSQL logical decoding, for example, streams SQL modifications through logical replication slots; availability of old row versions for updates and deletes depends on replica identity.^[2] SQL Server CDC likewise records insert, update and delete activity for enabled source tables.^[14] Debezium commonly wraps a row key, before and after images, source metadata and operation type in a change-event envelope.^[3]

Event sourcing makes the event sequence the principal application history. State changes are initiated through stored domain events, and current or past state is derived by applying them.^[1] A domain event can say that a limit was reduced because an annual review completed under policy version 6. A raw CDC event may say only that column limit changed from 500 to 250.

CDC gives mutation provenance; a well-designed domain event can give decision provenance. Mutation provenance answers which row changed, at which log position, inside which source transaction. Decision provenance answers which business occurrence, command or approved judgement the change represented. Neither should be fabricated from the other.

A thought experiment about identical updates

Hold the SQL update constant: limit = 250. In world A, the value follows an approved annual review and should trigger covenant and communication workflows. In world B, an operator corrects a keying error and downstream systems should reverse prior consequences. In world C, a batch repair rewrites the column after a migration and no customer-facing event should fire. The row mutation is identical. The legitimate reactions differ.

A classifier could infer likely intent from surrounding columns or application logs. That may be useful investigation support, but it does not turn inference into historical fact. Where downstream behaviour depends on why the write occurred, record a typed domain event or command receipt at the decision boundary. The transactional outbox pattern addresses the dual-write problem by storing application state and the message record within one local transaction, then relaying the outbox asynchronously.^[5][6]

CloudEvents standardises common event metadata for interoperability, including identifiers, sources and types.^[12] That is useful envelope discipline. It does not decide whether the payload names a durable domain occurrence, a technical notification or a flattened row image. A standard envelope cannot rescue weak event semantics.

Corrections are new evidence about old evidence

An append-only history does not forbid correction. It forbids pretending that correction never happened. In the facility case, the 10:07 correction can preserve a reference to the 10:03 limit reduction, state that the earlier event was applied to the wrong facility and supply the replacement value. The original occurrence remains visible for system-knowledge questions; the correction changes the valid-time interpretation for questions about the facility itself.

That produces two legitimate projections. The knowledge-time projection says that between 10:03 and 10:07 the operating system represented a limit of 250 and may have raised a breach. The corrected valid-time projection says the facility’s limit for that interval should be treated as 500 once the error is known. An investigator needs both. Overwriting the first projection hides an actual control path. Refusing to update the second forces every current calculation to repeat an obsolete error.

Reversal and compensation are related but distinct. A reversal asserts that an earlier change should no longer contribute to state. A compensating event records a new action whose effect offsets a prior one. Refunding a payment does not erase the payment; it adds a refund. Correcting a mistyped limit may supersede the erroneous value without generating an economic counter-transaction. Event types should preserve that distinction because downstream accounting, notification and assurance rules may react differently.

The serious negative control

Consider a small reference table containing the currently approved country code for each booking jurisdiction. Consumers need the current code and, occasionally, the row as recorded last quarter. The change path carries no independent business meaning. SQL Server system-versioned temporal tables maintain current and history tables and support point-in-time queries through FOR SYSTEM_TIME.^[11] That may answer the entire requirement.

Adding a domain event store here creates event schemas, reducers, checkpoints, replay tests and support procedures without changing a decision. This negative control matters because it defeats the claim that an append-only event history is automatically superior. Use events when the path is part of the product or evidence, not as a ceremony around every mutable table.

Part three

Four clocks and a partial order

A stream can be perfectly sorted by one clock and wrong for the question. The facility correction was valid from 10:03, committed at 10:07, captured at 10:07:01 and processed by a downstream view at 10:07:04. Asking what the source knew at 10:04 should exclude it. Asking for the best corrected account of the world at 10:04 should include its retroactive effect. One timestamp cannot represent both answers.

The Dataflow model made the distinction between event time and processing time explicit, including the reality that late data can arrive behind an estimated watermark and force a refinement.^[8] Temporal database work similarly distinguishes the time for which a fact is valid from the time a system recorded that version. In practice, a reconstruction envelope often needs at least four clocks.

Ordering also has a spatial boundary. Lamport showed that distributed events have a causal partial order: one event precedes another when it can affect it; independent events need not have a natural global order.^[7] A stream platform can impose a total sequence inside a partition, yet that sequence may be arbitrary across unrelated entities. Reconstruction should preserve the narrowest order that the reducer needs.

For a single aggregate, keying all state-changing events by stable aggregate ID is a common way to retain per-entity order. Cross-aggregate constraints are harder. A payment may update an account, a limit pool and a case in one database transaction. If those rows are emitted to separate topics or partitions, consumers can observe different arrival orders. Sorting everything by wall-clock time creates apparent certainty while ignoring clock skew and transaction structure.

Do not invent causality merely to obtain a tidy list. Preserve source positions and transaction IDs, then order only events whose relationship matters to the state or invariant. Concurrent changes can remain concurrent until a domain rule resolves them.

Atomic in the source, fractured in the stream

The fictional annual-review transaction changes both limit and status. The source commits them atomically. A row-level CDC pipeline may emit two records, and a downstream consumer may see the limit reduction before the status moves to review. Publishing the intermediate projection creates a state that never existed at a committed source boundary: limit 250, drawn 300, status active.

Debezium can emit transaction boundary events and enrich change records with transaction metadata, though such metadata is available only for transactions observed after the connector is deployed.^[4] A reconstruction consumer can buffer the records until the transaction end, apply them as one unit and expose only the committed projection. When that is impossible, the product must label the projection as partial rather than pretending each row event is an independent business state.

Exactly once has a boundary

Duplicate delivery is normal under retry. Consumers therefore need an immutable event ID or source position and an idempotent application rule. Kafka transactions can atomically write produced records and consumed offsets inside Kafka. Kafka’s own design documentation is explicit that exactly-once delivery to other destination systems generally requires cooperation from those systems.^[9] A message guarantee does not prove that an external side effect happened once.

Deletes need equal care. Debezium can emit a delete record followed by a tombstone so that compacted Kafka topics can eventually remove prior values for the key.^[4] Kafka compaction retains at least the latest value for each key and preserves the order of records it keeps, but it removes superseded records and eventually removes tombstones.^[10] A compacted topic may be excellent for rebuilding current keyed state. It is not a complete historical ledger after compaction.

Completeness is a frontier, not a feeling

A projection can be internally consistent and still be behind. “Processed at 10:10” says when the consumer ran, not which source changes it has incorporated. The receipt needs a frontier that the source and consumer can compare: a log sequence number, binlog coordinate, transaction sequence or set of per-partition offsets. For a multi-partition view, the frontier is usually a vector rather than one scalar timestamp.

This matters during backpressure and recovery. Suppose partitions A and B are current to source position 900, while partition C is current only to 740 because one schema change is quarantined. A dashboard assembled across all three partitions should not claim a globally current state. It can publish a bounded answer at the common complete frontier, expose the partial nature of the newer answer or defer the decision. Picking the largest observed timestamp hides the missing causal segment.

The same rule applies when a connector restarts. A resume position proves where capture intends to continue. It does not prove that every downstream projection has durably applied the prefix. Source offset, transport offset, reducer checkpoint and published-view version are separate receipts. End-to-end reconstruction is complete only at the lowest verified frontier shared by all required stages.

Part four

Replay becomes evidence only when it can fail

A replay demonstration that merely produces a plausible row proves little. The preferred mechanism needs an independent oracle or invariant. For the facility, the current projection can be compared with an authoritative source read at a declared frontier. Historical replays can be checked against retained decision receipts, source snapshots or known control events. Negative cases should drop an event, duplicate one, split a transaction, change an identity key and run an old event through a new reducer.

The reducer is part of the evidence chain. Suppose version 1 treats a missing currency as GBP, while version 2 rejects the event. Replaying the same bytes under version 2 changes historical state. Both behaviours may be reasonable for their releases. A decision receipt therefore records the reducer or projection version, schema IDs and source frontier used to produce the answer.

Three oracles for a replay claim

A strong replay test uses more than one kind of oracle. The state oracle compares a reconstruction at frontier K with an authoritative source snapshot, retained export or independently materialised view at the same frontier. It catches missing, duplicated and misordered events, but only where an equivalent state is available.

The invariant oracle checks properties that every admissible state transition must preserve. Drawn amount cannot be negative. A limit breach cannot remain in active status unless policy explicitly permits it. Aggregate versions must increase without gaps. Invariants can identify an impossible intermediate publication even when the final row later converges. They are especially valuable for transaction-fracture and out-of-order tests.

The consequence oracle checks whether reconstruction explains known downstream effects. If a breach alert was issued at 10:04, a replay that never crosses the threshold is incomplete or is using a corrected-valid-time view for the wrong question. Conversely, a replay that crosses the threshold but no alert exists may expose a control failure rather than a replay failure. The test must distinguish state reconstruction from effect verification.

No oracle is absolute. A current source can contain the same corruption as its CDC stream. An invariant can encode a mistaken policy. A downstream effect can fail independently. Agreement among differently derived oracles is stronger because their errors are less likely to be identical. A replay earns confidence by surviving independent ways of being wrong.

Determinism requires a sealed context

A reducer can look pure while depending on changing surroundings. Suppose a drawdown event stores EUR 100, while the projection computes GBP exposure by calling today’s exchange-rate service. Replaying next month changes yesterday’s exposure although the event bytes and reducer code are unchanged. The result is a new analytical interpretation, not a reproduction of the original state.

The same leak appears through current customer classifications, mutable calendars, time-zone libraries, rounding rules, feature flags and reference tables. There are three honest remedies. Store the decision-relevant value inside the event, such as the booked GBP amount and rate identifier. Record a separate versioned reference event that joins at the historical frontier. Or seal a reproducible dependency bundle and name its version in the replay receipt.

Not every projection needs historical determinism. A current risk dashboard may intentionally revalue all past cash flows under today’s market data. It should label that output as current revaluation and preserve the source events from which it was computed. A forensic replay, by contrast, must use the dependencies available to the original decision or state clearly which dependency could not be recovered.

System clocks and randomness need the same treatment. The reducer should receive event time as data rather than call the wall clock. Any stochastic allocation should carry its seed or, preferably, emit the chosen allocation as an event. Replayable state is determined by the complete input envelope, not by event payloads alone.

Schema compatibility is necessary, not sufficient

CDC consumers may process records created under several table schemas. Debezium maintains schema history for connectors that can read database DDL so older log records can be interpreted under the structure in force when they were written.^[3] Schema registries add compatibility checks as producer and consumer contracts evolve.^[13] These controls prevent many decoding failures.

They do not prove semantic continuity. Renaming risk_grade to risk_band can be wire-compatible while changing the scale. Adding a default may let old readers parse the event while changing the meaning of absence. A replay migration should include semantic fixtures: known histories with expected states, invariants and decisions under each supported reducer version.

There are three defensible strategies. Preserve the original reducer and query old projections under their historical version. Upcast old event representations into a canonical model through explicit, tested migrations. Or rebuild a new projection and label it as a current reinterpretation, not as the state the old system actually computed. The choice depends on whether the question is forensic, operational or analytical.

A production-shaped reconstruction path

For the synthetic facility service, the application writes current rows and a domain outbox record in one local transaction. CDC reads the database log into an immutable raw capture zone. A normaliser preserves the raw envelope, source position, transaction ID, schema ID and capture time while adding a stable aggregate key. A transaction assembler withholds publication until all expected records or an explicit end marker arrive.

A projection service then applies a versioned reducer. It produces current facility state, a valid-time view and a system-knowledge view. It writes checkpoints only after source reconciliation and stores the exact event frontier. A decision service receives a bounded state envelope rather than an unscoped stream. Its receipt names the projection version and frontier, so a later investigation can replay the same evidence state.

The raw capture remains distinct from the domain log. Raw CDC supports forensic recovery and new projections. The domain event records preserve application intent. Current source reads remain the authority for live effects unless the application is intentionally event-sourced. No layer is promoted to “truth” beyond the claim it can reconstruct and verify.

Worked example: replay the facility

The minimal sequence below contains an open, a drawdown, a two-row annual-review transaction, a repayment and a two-row correction transaction. A duplicate delivery of the drawdown is appended at the end. The verified path orders by source position, suppresses the repeated event ID, applies each transaction atomically and compares the result with the declared authoritative state.

The naive path orders by arrival, publishes each row immediately and applies the duplicate. It produces a plausible active facility, but drawn amount is 500 rather than 200. It also exposes an impossible intermediate state after the limit row arrives without the paired review status. The missing-correction path is internally consistent yet disagrees with the source. These failures are useful because they prevent fluent replay output from becoming its own oracle.

function applyEvent(state, event) {
  const next = { ...state };
  if (event.kind === "OPEN") {
    next.limit = event.limit;
    next.drawn = 0;
    next.status = event.status;
  }
  if (event.kind === "DRAW") next.drawn += event.amount;
  if (event.kind === "REPAY") next.drawn -= event.amount;
  if (event.kind === "SET_LIMIT") next.limit = event.value;
  if (event.kind === "SET_STATUS") next.status = event.value;
  next.headroom = next.limit - next.drawn;
  return next;
}

function invariantIssues(state) {
  const issues = [];
  if (state.drawn < 0) issues.push("drawn amount below zero");
  if (state.drawn > state.limit && state.status !== "REVIEW") {
    issues.push("excess draw without REVIEW status");
  }
  return issues;
}

// The page script adds:
// 1. source-position or arrival ordering;
// 2. event-ID deduplication;
// 3. transaction buffering;
// 4. source-state comparison;
// 5. positive and negative assertions.

The explicit failure boundary

Reconstruction cannot recover evidence that never entered the boundary. It fails when a source purges log segments before capture, an out-of-band writer bypasses the observed database, primary keys are reused without identity lineage, cross-system transactions have no shared causal marker, old schemas cannot be interpreted, or the reducer reads changing external services during replay.

It also fails more subtly when retention changes the question. A compacted topic may retain enough information for current state while deleting the path needed for an investigation. A system-time history can show when the database held a row while omitting when the fact was valid in the world. A domain event can preserve intent while missing an external side effect that failed after commit.

Part five

stateDiagram-v2
  [*] --> Claim
  state "The reconstruction contract" as Claim
  state "Identity is itself a state<br/>transition" as Test
  state "A snapshot and a stream must<br/>meet at one frontier" as Revision
  Claim --> Test: expose mechanism
  Test --> Decision: evidence survives
  Test --> Revision: rival survives
  Revision --> Claim: narrow or rebuild
  Decision --> [*]

Choose the smallest history that answers

Event architecture often begins with a tool choice and works backwards. Reverse the order. Name the question, its temporal frame and the consequence of an incorrect reconstruction. Then choose the least complicated mechanism that preserves the distinctions the decision needs.

The reconstruction readiness card

Before approving replay as an operational or audit capability, complete the following card. A missing answer is a design debt. A vague answer such as “Kafka keeps order” is not evidence until it names the key and partition boundary.

The strongest objection

Event histories create permanent compatibility obligations. Teams must operate retained logs, schema registries, checkpoints, replay jobs, dead-letter handling, transaction buffers and migration tests. Investigators can confuse captured row changes with business events. Consumers can multiply until the source cannot change safely. These costs are real, and they often exceed the value of historical reconstruction.

The answer is not to promise a universal event backbone. It is to make the evidential benefit explicit. Replay earns its cost when it supports a decision that current rows and simpler history mechanisms cannot: reproducing what was knowable, building independent projections, correcting a past interpretation, explaining a path-dependent outcome or recovering state after a projection loss. Otherwise, query the source.