Home · Work · System design
Operations and servicingTransactional saga with effect verificationevent spine

Standing-order failure recovery

A failed standing order may involve funds, limits, timing, destination state or a platform incident, each requiring a different remedy.

Approach

The workflow diagnoses the failure from authoritative events, explains the customer impact and offers only safe, policy-permitted recovery actions.

Organisations, systems and operating conditions are intentionally anonymised and recomposed. The design demonstrates engineering and banking-domain reasoning; it does not represent a named client estate, vendor product or measured production result.

Overview

System overview

Typed commands, state changes and receipts cross an event spine without surrendering domain ownership.

Standing-order failure recovery: event-spine architecture Twelve independently owned platform services connect through a typed event and evidence spine. The design separates request admission, event ordering, capability execution, systems of record, dead-letter recovery, and an immutable receipt archive. aSystem boundary and decision authoritySchematic architecture · illustrative scenario Request and event admissionCommand-side servicesAuthoritative systemsOperations rail Channel & case eventsHTTPS · batch · streamSchema & identity gateOIDC · event versionPartition & ordering serviceentity key · watermarkBackpressure gatequota · pause envelopeadmittedlag signal Evidence ContextCompilerContext & evidenceParty RelationshipResolverIdentity & relationshipsAuthority Rules EngineAuthority & policyJourney StateCoordinatorWorkflow & custodyTyped Action BrokerIntegration & effectsDecision ReceiptLedgerEvidence & auditReasoning RouteManagerModel executionEffective-DatedKnowledge HubKnowledge & rulesBehaviour EvaluationHarnessEvaluation & releaseOutcome RecoverySupervisorResilience & recoveryOperational Read ModelWorld state & read modelsDecision ReviewWorkbenchHuman decision & operationspublishpublishpublishpublishpublishpublishconsumeconsumeconsumeconsumeconsumeconsume EVENT + STATE SPINErequest.accepted · context.qualified · authority.decided · action.dispatched · effect.readback · custody.changed Servicing platformsource of recordproduct systemsource of recorddocument storesource of recordworkflowsource of record Dead-letter & replay queueDecision receipt archiveTrace, lag & cost telemetryIncident & custody queue
Figure 1. Typed events coordinate independently owned services without turning the event spine into a system of record. Ordering, backpressure, readback and replay stay explicit. This is an illustrative system model, not a named deployment.

Permitted workThe AI layer may diagnose, prepare and coordinate. Core servicing effects, remediation and complaint outcomes remain policy-controlled actions.

Consistency ruleKeep one durable case state and explicit custodian while source effects reconcile asynchronously.

Hard boundaryThe model is not a system of record, identity provider, policy authority or proof that an external effect occurred.

Deployment

Deployment

The journey can change account, payment, case or document state across systems that do not share one transaction boundary.

Standing-order failure recovery: deployment and trust-zone topology A production topology separates public edge, identity boundary, workload cluster, integration plane, data enclave and operating controls. Connections name transport and identity protocols, and a second region provides controlled failover. bDeployment topology and failure domainsLogical deployment · no measured estate data EDGE AND ACCESS PRIVATE WORKLOAD CLUSTER INTEGRATION AND DATA ENCLAVE OPERATIONS, EVIDENCE AND RECOVERY PLANE Web applicationfirewallTLS · bot · rate API ingressschema · quota Identity brokerOIDC · MFA Purpose gateclaims · consent HTTPStokensubject Regional traffic managerhealth · canary · failover Workload identity issuershort-lived service tokens Event & work queuepartition · DLQ · backpressure Secrets & key serviceenvelope encryption · rotation Authority RulesEnginezonal workloadJourney StateCoordinatorzonal workloadTyped ActionBrokerzonal workloadDecision ReceiptLedgerzonal workloadOutcome RecoverySupervisorzonal workloadOperational ReadModelzonal workloadReserved workloadslotdisabledReserved workloadslotdisabled HTTPS · scoped JWT workload token async eventkey lookup IntegrationgatewaymTLSServicing platformsystem of recordproduct systemsystem of recorddocument storesystem of recordworkflowsystem of record Model route enclavepolicy · schema · egress Qualified data storesSQL · graph · object Trace, metric & cost pipelineOpenTelemetry · redaction Decision receipt archiveappend-only · retention Reconciliation workersreadback · replay · DLQ On-call & review queuescustody · SLA · escalation Secondary-region standbycompatible manifest telemetryreceiptsexceptions
Figure 2. Workloads remain isolated from systems of record. Short-lived identity, typed integration contracts, append-only evidence and controlled failover define the deployable boundary; the topology is schematic.
Authority checkpoint

A new payment cannot be proposed until the original outcome and duplication risk are known.

Runtime

Runtime flow

A deterministic outer workflow contains model-led work inside typed, observable calls. Dashed messages remain proposals until policy or a human grants authority.

Standing-order failure recovery: runtime sequence and control branches Ten lifelines and eighteen protocol messages show authentication, evidence acquisition, policy authority, bounded reasoning, effect verification, human review and append-only evidence. A shaded branch contains timeout and unknown-outcome handling. cRuntime protocol and control pointsProtocol model · illustrative execution Channelcustomer or staffAccess gateidentity & purposeCoordinatorworkflow ownerContextqualified evidenceAuthoritypolicy decisionReasonerbounded inferenceAction brokertyped capabilitySource estateauthoritative stateReview deskhuman decisionReceiptledgerappend-only proof ALT · timeout, rejection, or unknown business effect No blind retry. Resolve source state, narrow the result, or retain custody. Material evidence missing → review queueUnknown effect → readback and reconciliationAuthority denied or expired → stop before dispatch HTTPS · request envelopeOIDC claims · purpose · deadlinetyped evidence requestREST/gRPC reads at qualified timefacts · validity · source versioncontext manifest · exclusionsauthority decision · conditionsminimal residual · closed schemacited proposal · confidence flagsexact capability and payload hashsigned grant · expiry · limitsidempotent command or readaccepted status · effect referenceindependent state readbackaction/effect receiptcase pack when review is requireddisposition · rationale · custodyqualified response · limitations authoritative callproposal / reviewevidence writePattern: Transactional saga with effect verification
Figure 3. The protocol separates request acceptance, permission, business effect and independent readback. A timeout first changes the knowledge state; it does not automatically justify another call. The paths are protocol semantics, not an observed production trace.
S

State model

A durable saga record owns step status, retries, deadlines, compensation and custody. Source systems retain legal business state.

I

Integration choice

Mutating calls pass through typed capability contracts. Events notify downstream services, but command acceptance and effect readback remain distinct.

C

Consistency semantics

Use idempotent local transactions and eventual cross-system consistency. Unknown outcome is a first-class state, not an exception hidden by retry.

Services

Services and interfaces

These roles are deliberately vendor-neutral. Each can be independently owned, versioned and replaced.

  1. COS
    Workflow and custody

    Journey State Coordinator

    Runs a versioned state machine with explicit waits, deadlines, retries, compensations, human tasks and terminal states.

    Degraded routePause at a known checkpoint, preserve custody and resume from the last verified state rather than restarting the request.
  2. CAG
    Integration and effects

    Typed Action Broker

    Exposes typed, allow-listed capabilities with schema validation, delegated identity, idempotency, timeout semantics and effect readback.

    Degraded routeDisable the affected capability, preserve the prepared action and transfer it to an authorised manual route.
  3. RRS
    Resilience and recovery

    Outcome Recovery Supervisor

    Classifies unknown outcomes, replays idempotent work, runs compensations and restores from the last verified checkpoint.

    Degraded routeStop new effects, continue safe reads where possible, and expose the recovery state and owner to customers and operators.
  4. OSR
    World state and read models

    Operational Read Model

    Builds time-qualified projections from source events and reconciliations without becoming the legal system of record.

    Degraded routeExpose source lag and fall back to direct reads for defined critical facts; never present missing feeds as zero.
  5. DES
    Evidence and audit

    Decision Receipt Ledger

    Appends request, versions, policy result, model proposal, approval, action receipt, readback, correction and custody events under one correlation key.

    Degraded routeBlock a consequential close when mandatory evidence is missing; keep the case open with an explicit evidence defect.
  6. PDS
    Authority and policy

    Authority Rules Engine

    Evaluates identity, purpose, capability, amount, risk tier and policy version; returns allow, deny, step-up or human-review with reasons.

    Degraded routeFail closed for effects and restricted data. Read-only explanation may continue only from approved public or customer-visible sources.
Figure 4. Each service exposes a typed contract and a defined degraded route. Coordination stays in the control plane while domain ownership remains outside it.
Data

Data and records

Durable records carry provenance, authority, effect and custody without turning a transcript into an uncontrolled memory store.

Standing-order failure recovery: state ownership and evidence lineage Authoritative source observations feed six versioned runtime records. Each record names a single write owner. An append-only event and evidence stream builds a disposable current-state projection and supports audit replay. dState lineage, ownership and temporal validityLogical records · fields are illustrative AUTHORITATIVE INPUTS · POINT-IN-TIME QUALIFIEDServicing platformsource version · observed atproduct systemsource version · observed atdocument storesource version · observed atworkflowsource version · observed atqualified inputqualified inputqualified inputqualified input WRITE OWNER · Evidence Context CompilerRequest enveloperequest_idpurposeactor_refsubject_refversioned · attributable · retainedWRITE OWNER · Authority Rules EngineContext manifestcontext_idsource_refsource_versionobserved_atversioned · attributable · retainedWRITE OWNER · Typed Action BrokerCase statecase_idstatestate_versioncurrent_custodianversioned · attributable · retainedWRITE OWNER · Reasoning Route ManagerAuthority decisiondecision_idpolicy_versionrequested_capabilityoutcomeversioned · attributable · retainedWRITE OWNER · Behaviour Evaluation HarnessAction and effect receiptaction_ididempotency_keyinput_hashdispatch_statusversioned · attributable · retainedWRITE OWNER · Operational Read ModelCustody eventtransfer_idfrom_ownerto_ownerreasonversioned · attributable · retained APPEND-ONLY CASE EVENTS + DECISION RECEIPTSrequest hash · source versions · policy version · proposal · authority · effect readback · custody Operational current-state viewrebuildable projection
Figure 5. The record chain preserves source version, write ownership, authority and custody. Solid records are required in this scenario; dashed records remain compatible but dormant.
Consistency contractDomain source rule

Keep one durable case state and explicit custodian while source effects reconcile asynchronously.