The observation has two histories

A time series is usually drawn as though each point were a small piece of reality. In practice, every point has two histories. One belongs to the process being measured: customers arriving, turbines rotating, packets crossing a network. The other belongs to the measurement path: sensors, counters, sampling clocks, joins, missing-value rules and export pipelines.

Conflating the two histories creates a basic category error. A business process can change while the sensor remains sound. A sensor can change while the business process remains sound. They can also fail together. The International Vocabulary of Metrology keeps measurement concepts explicit because an observed value and the quantity attributed to the measurand are not interchangeable. In software telemetry, the same discipline applies even when the “sensor” is a counter or log pipeline.

The equation is a discipline, not a claim that the components can always be uniquely recovered. It forces a monitor designer to state where a proposed explanation lives. Seasonality, regime change and rare events belong to the process model. Sensor bias, drop-out, frozen values and timestamp corruption belong to the measurement model. An extreme residual can reveal disagreement, but it cannot identify the layer that caused it without additional structure.

This model also clarifies the overloaded word drift. A gradual movement in x_t may be process drift. A changing relation between predictors and an outcome is concept drift. A changing sensor transfer function g_t is measurement drift. A shifting mix of observed cases is population drift. These changes may look similar in a chart while affecting different contracts. Webb and colleagues’ formal treatment of concept drift is valuable precisely because it asks which distribution changed, by how much, over what duration and where in the input space.

Thought experiment: move the clock, change nothing else

Imagine an hourly electricity series with a 20-megawatt peak. In world A, the peak occurs at 18:00 every weekday. In world B, the same amplitude and duration occur once at 03:00 on a mild Sunday. Keep the value, noise and sensor unchanged. Move only the event’s phase relative to the calendar.

A value-only detector treats the peaks alike. A temporal detector should not. The weekday peak has many matched-phase precedents and belongs in the expected distribution for that context. The Sunday peak has little support at its phase. Contextual abnormality arises from the relation between value and time, not from magnitude alone. The distinction is one reason the classical anomaly taxonomy separates point, contextual and collective anomalies, as reviewed by Chandola, Banerjee and Kumar.

Seasonality is broader than a sine wave. It can follow hour of day, day of week, month, holidays, billing cycles, school terms or combinations of them. High-frequency data often carry several cycles at once. MSTL extends STL-style decomposition to multiple seasonal patterns, which matters when a daily pattern sits inside a different weekend pattern and a slower annual pattern.

flowchart LR
  A["The 02,00 drop with four<br/>incompatible explanations"] --> B["The observation has two histories"]
  B --> C["Decompose before detecting"]
  C --> D["Four hypotheses need four<br/>kinds of evidence"]
  D -. "re-test" .-> A
  classDef first fill:#0a1424,stroke:#007D68,stroke-width:2px,color:#e6f3ff
  classDef step fill:#0a1424,stroke:#3f7da0,stroke-width:1px,color:#e6f3ff
  class A first
  class B,C,D step

Decompose before detecting

A raw threshold asks whether y_t is far from a centre. A temporal detector asks whether y_t is surprising under information that was legitimately available before time t. That conditional question is harder, but it is the one operations teams care about.

The forecast need not be a large neural model. A seasonal median, exponential smoother, state-space model or robust regression can provide the expected value. The right baseline is the simplest one that preserves the structures relevant to the decision. STL, introduced by Cleveland and colleagues, decomposes a series into trend, seasonal and remainder components using locally weighted smoothing. Its value here is conceptual as much as algorithmic: it makes the monitor state which variation it has declared expected.

Decomposition moves recurring structure out of the anomaly budget. If every Monday morning is high, the detector should compare this Monday with prior Monday mornings, not with the weekly average. If variance rises during market open, the uncertainty band should widen then rather than generating a daily burst of pages.

Minimal worked example: twenty-one days of hourly orders

Consider a synthetic café with 21 days of hourly orders. Demand rises at breakfast and lunch, falls on Sundays and otherwise varies by roughly three orders per hour. At 12:00 on the final Monday, the café receives 84 orders. The overall mean is 39 and the overall standard deviation is 18, so a global three-sigma rule may remain silent. A narrow business-hours threshold may alert.

Now compare the observation with prior Monday 12:00 values: 79 and 82. The new value is ordinary. At 03:00 on Sunday, however, 52 orders would be modest relative to the global distribution yet extraordinary relative to its matched phase, where the expected count is four. The detector’s judgement reverses because the reference class changed.

This example also exposes a common leakage error. If the seasonal template is estimated using the period being evaluated, the anomaly can pull its own baseline towards itself. A defensible benchmark freezes the reference interval, fits only on past data and evaluates future points. The rule matters more than the model name.

Point anomaly or change?

A residual spike and a changed process can start with the same first point. Their difference appears through persistence. A point anomaly spends its evidence quickly: it departs, then the process returns. A regime change keeps making the old model wrong in the same direction. Detecting that accumulation is a different statistical problem.

Page’s cumulative-sum idea makes the mechanism intuitive. Subtract a small allowance from each residual and accumulate what remains. Random positive and negative errors tend to cancel. A sustained positive shift keeps adding evidence until the cumulative statistic crosses a boundary. The NIST CUSUM guidance notes that cumulative charts are especially effective for smaller mean shifts that a pointwise chart may miss.

For retrospective segmentation, PELT finds multiple change points by optimising a penalised cost and, under stated conditions, can achieve linear computational cost. For streaming inference, Bayesian online change-point detection maintains a posterior over run length, the time since the most recent change. ADWIN takes another route by adapting the comparison window as evidence of distribution change arrives. These methods encode different assumptions. None removes the need to model seasonality first, because regular calendar movement can otherwise look like repeated change.

A detector must also decide whether a change is actionable. A slow growth in legitimate demand may be real and harmless. A 0.2 per cent shift in a safety-critical control variable may be material. Statistical evidence quantifies incompatibility with a reference model; consequence determines response.

Four hypotheses need four kinds of evidence

The most useful monitor does not pretend the four causes are mutually exclusive. A seasonal peak can coincide with a network event. A sensor can fail during a real process shift. The monitor should keep competing hypotheses visible, state the evidence for each and allow an unresolved result.

Hypothesis 1: expected seasonal or calendar structure

The key test is recurrence at matched phase. Compare the point with previous observations sharing the relevant calendar state: hour, weekday, holiday class, billing boundary or operating mode. Then ask whether the current residual remains unusual after those effects are removed.

Seasonality can evolve, so a matched phase is necessary rather than sufficient. A summer heatwave can amplify the normal afternoon electricity peak. A new product can change the shape of month-end traffic. The baseline needs versioning and an explicit refresh rule. Silent rolling baselines can chase slow degradation until the degraded state becomes “normal”.

Hypothesis 2: sustained process change

The tests are persistence and segmentation. Does the residual stay biased after the first alarm? Does a change-point method prefer two regimes over one? Do peer signals and business outcomes move in ways consistent with a changed process? A release marker, policy change or external covariate can strengthen the explanation, but temporal coincidence alone does not prove causation.

A regime-change alert should name the parameter believed to have changed. “Anomaly score 0.93” is weak. “Median authorisation latency increased by 18 milliseconds after release R42, persisted for six hours and appears in gateway and processor traces” is diagnosable.

Hypothesis 3: transient real event

A rare event is a real departure that does not establish a new stable regime. Evidence includes simultaneous movement in independent measurements, downstream consequences, external incident records and recovery after the event. Duration matters. A two-minute burst and a six-hour plateau may have equal maximum residuals but demand different controls.

For ranged anomalies, evaluation must reward timely detection without pretending one hit identifies the whole interval. Kim and colleagues showed that the common “point adjustment” protocol can greatly inflate results; under that protocol even random scores may appear competitive. The lesson is operational: score when the detector first knew enough, how long it stayed wrong and whether it identified the event boundary.

Hypothesis 4: measurement or sensor fault

A broken measurement often violates relationships that a real event would preserve. Redundant sensors disagree. A flow meter changes while upstream pressure and downstream inventory remain consistent. A request counter falls while completed transaction records do not. A temperature probe jumps faster than the physical system can change. These are residuals against a measurement model or observer, not merely against the series’ own past.

Observer-based fault detection compares measurements with outputs predicted from a system model. NIST-published work on sensor fault detection describes residuals as differences between sensor measurements and estimated system outputs, while also noting limitations when observers have not converged or critical sensor loss makes the system unobservable. That boundary matters: redundancy improves diagnosis, but it is not free proof.

Thought experiment: duplicate the sensor, change nothing else

Return to the 28 per cent throughput drop. In world C, three independently derived signals fall together: gateway acceptances, processor acknowledgements and ledger postings. In world D, only the dashboard exporter falls; the other two remain aligned with their forecast. Keep the primary chart identical. Vary only the corroborating evidence.

World C supports a real event or regime change. World D supports a measurement fault. The primary series alone cannot distinguish them. Cross-signal coherence supplies information that no more sophisticated univariate model can manufacture.

A serious counterexample: when one threshold is enough

Suppose a laboratory instrument produces one independent reading per batch. The process is stationary, has no calendar structure, the sensor is calibrated before every batch and any reading beyond a fixed safety limit requires the same shutdown response regardless of cause. A simple limit is superior to a decomposition stack. It is transparent, fast and aligned with the decision.

This negative control weakens any universal claim for typed anomaly diagnosis. The layered method earns its cost only when temporal context changes expected values, causes require different responses, or measurement faults are plausible. Complexity is justified by decision separation, not by the prestige of the detector.

Benchmark diagnosis, not alarm volume

Time-series anomaly research has repeatedly discovered that benchmark design can dominate apparent progress. Wu and Keogh documented flaws in widely used datasets and argued that unreliable exemplars can create an illusion of improvement. Later work showed that evaluation protocols themselves can over-reward weak scores. More recent benchmark systems such as TimeEval and TAB standardise larger collections of datasets, methods and experimental settings.

Those efforts improve comparability, but an operational benchmark needs one more layer: causes and responses. A detector that catches every unusual interval yet confuses a dead sensor with a real plant excursion can score well and operate badly. A detector that pages on every known weekly peak may have high recall and still be unusable.

The Numenta Anomaly Benchmark was an important streaming precedent because it rewarded early detection and penalised false alarms. Its timing-aware score is useful here, but labelled anomaly windows alone do not identify whether the process, the measurement path or both changed.

The proposed benchmark injects four mechanisms into a clean synthetic or replayable reference stream. It then scores the detector on the questions that determine action.

Ablate the evidence, not only the algorithm

A causal benchmark should show which observation makes each diagnosis possible. Start with the four synthetic generators, keep the detector thresholds fixed and remove one evidence channel at a time. This is more informative than replacing one detector with another while leaving the evidence contract implicit.

First remove hour-of-week conditioning. The seasonal negative control now enters the anomaly budget because its matched phase has disappeared; a more flexible point detector may alter the score, but it cannot recover the missing calendar comparison. Next retain phase but remove the peer channels. The injected sensor fault and the rare real event then present the same primary trace. Any confident distinction between them is supplied by an assumption, not by the observation.

Now restore peers and remove the persistence accumulator. The brief event remains visible, but the smaller level shift becomes a sequence of individually modest residuals. A point detector either misses it or lowers its threshold and pays with more false alerts. Finally, keep detection intact but omit recovery readback. The system can close an incident merely because the alert stops, even when the sensor has frozen, the stream has disappeared or an adaptive baseline has absorbed the fault.

The ablation target is the discriminating evidence. In this benchmark, phase explains recurrence, peers test measurement coherence, accumulation tests persistence and readback tests outcome. A replacement model earns credit only if it preserves those decisions under the same withheld-evidence tests.

Executable artefact: a four-cause drift laboratory

The laboratory below generates 35 days of synthetic hourly demand with daily and weekly seasonality across three noisy channels. The first 21 days form a frozen reference. Four scenarios are evaluated: expected seasonal variation, a four-hour real event, a persistent level shift and a fault injected into only the primary channel.

The simple detector fits an hour-of-week template, standardises residuals robustly, tests peer disagreement, checks a short high-amplitude run and checks sustained rolling residual evidence. It is deliberately modest. Its purpose is to reveal the classification logic, not to claim a generally optimal algorithm.

// Core decision logic used by the page.
// zPrimary, zPeerA and zPeerB are phase-adjusted residual z-scores.

const peer = medianSeries([zPeerA, zPeerB]);
const consensus = medianSeries([zPrimary, zPeerA, zPeerB]);
const disagreement = absSeries(subtract(zPrimary, peer));

const sensorPeak = max(disagreement);
const peerAtSensorPeak = Math.abs(peer[argmax(disagreement)]);
const eventPeak = max(absSeries(consensus));
const eventRun = longestRun(absSeries(consensus).map(v => v > 8));
const driftMean = rollingMean(consensus, 24);
const driftRun = longestRun(
  driftMean.map(v => Number.isFinite(v) && Math.abs(v) > 2.4)
);

let label = "expected variation";
if (sensorPeak > 12 && peerAtSensorPeak < 5) {
  label = "sensor fault";
} else if (eventPeak > 8 && eventRun <= 12 && driftRun < 36) {
  label = "rare event";
} else if (driftRun >= 36) {
  label = "regime change";
}

Production-shaped worked scenario: authorisation telemetry

Consider a synthetic card-authorisation service with four relevant series: incoming requests, gateway accepts, processor acknowledgements and ledger postings. Each has minute-level seasonality, weekday effects, known batch boundaries and a release calendar. The service also records exporter health and sequence gaps.

At 02:07 on Sunday, gateway accepts fall 28 per cent. The first-stage seasonal model expects a 16 per cent decline because maintenance traffic shifts at that phase, leaving a residual decline of 12 per cent. A raw detector pages at severity high. The typed monitor keeps four hypotheses alive.

Phase recurrence is only partial: previous maintenance windows were smaller. Persistence evidence grows for 18 minutes, then stabilises. Incoming requests and processor acknowledgements remain normal. Ledger postings match acknowledgements. Exporter health reports a shard restart and a 12 per cent partition gap. The measurement-fault hypothesis therefore dominates. The system quarantines the affected metric, substitutes acknowledgements for operational monitoring and opens an instrumentation incident. It does not roll back the payment service.

Now vary one fact. Processor acknowledgements and ledger postings fall with gateway accepts while exporter health remains sound. The same primary trace now supports a real service event. The correct response is traffic containment and service investigation. The detector’s value lies in changing the action, not in making the red line appear sooner.

The scenario is a worked composite, not a deployment record. Its operating claim is narrower: when independent evidence changes the safe response, the monitor must preserve that evidence in the alert.

The temporal diagnosis card

The card should be completed from a frozen evidence window before anyone retunes the detector. Record the event time in both source and processing clocks, the seasonal reference set, the first series to depart, dependent-series reactions, sensor health, deployment and policy changes, the proposed mechanism and the action that mechanism would justify. This order keeps a convenient alert label from becoming the explanation.

A useful counterfactual replay then changes one evidence channel at a time. Replace the suspect sensor with its redundant measurement while preserving the underlying event; remove a calendar feature while preserving the time stamps; replay the observations through the previous detector; and compare a stable pre-change baseline with an adaptively updated one. A true process change should survive sensor substitution. A measurement fault should follow the instrument. Expected seasonality should weaken when the relevant context is removed rather than when the process is repaired.

Report detection and diagnosis separately. Detection latency measures when the system first declared that the sequence was unusual. Diagnosis latency measures when evidence was sufficient to choose the correct runbook. False alerts consume attention, but a fast alarm with the wrong causal label can be worse than a slower, well-bounded diagnosis. The benchmark should therefore score cause classification, abstention, time-to-correct-action and consequence-weighted errors at a fixed alert budget.

Unknown outcomes need a stable state too. If telemetry is delayed or clocks disagree, the system should retain “unresolved observation” rather than impute normality or declare drift. It may request a second channel, narrow automatic action or escalate. Once delayed evidence arrives, the record should show whether the original classification would have changed. This creates feedback for the diagnostic architecture without rewriting the incident history.

Thresholds also expire. A residual limit calibrated during one traffic mix can lose meaning after a product launch, routing change or sensor migration. Recalibration is permitted only after separating a new expected baseline from an unresolved process fault. Otherwise the detector learns the incident as normal. Preserve the old model, replay both versions on a holdout spanning the change and require that improved alert economy does not erase known fault signatures.

The operating decision is consequently bounded. Automate a low-consequence reversible response when multiple channels agree and the proposed mechanism passes its counterfactual check. Request evidence when the cause remains ambiguous but delay is tolerable. Escalate or place the service in a safe mode when the consequence is high, the sensor contract is doubtful or dependent series contradict the leading explanation. A scalar anomaly score alone authorises none of these operational response branches.

Failure boundary: where the mechanism stops helping

The strongest boundary is identifiability. Given one observed series, a real process pulse and an additive sensor fault can generate exactly the same values. No detector can distinguish them from that series alone. The missing information must come from redundancy, a physical model, an external event, a second measurement path or an intervention.

Decomposition also becomes fragile with too little clean history, rapidly changing seasonal patterns, irregular sampling, long missing intervals, feedback loops or anomalies that contaminate the reference window. A model can absorb a fault into its seasonal component or call a newly recurring failure “normal”. Abrupt changes in variance can defeat mean-shift tests. Shared data pipelines can make several “independent” sensors fail together.

When the evidence cannot separate causes, the correct output is unresolved. The next action should gather discriminating evidence or move the system into a bounded safe mode. Confidence language cannot repair missing observability.

stateDiagram-v2
  [*] --> Claim
  state "Thought experiment: move the<br/>clock, change nothing else" as Claim
  state "Minimal worked example:<br/>twenty-one days of hourly<br/>orders" as Test
  state "Point anomaly or change?" as Revision
  Claim --> Test: expose mechanism
  Test --> Decision: evidence survives
  Test --> Revision: rival survives
  Revision --> Claim: narrow or rebuild
  Decision --> [*]

Compact glossary

Seasonality

Variation associated with a recurring phase such as hour, weekday, holiday or operating cycle.

Regime change

A sustained alteration in parameters or distribution that makes the prior process model persistently wrong.

Rare event

A transient real-process departure that later returns to the prior regime or another known state.

Measurement fault

A change in sensing, counting, timing or transport that corrupts the observation without requiring the underlying process to change.

Residual

The difference between an observation and its context-conditioned expected value.

Change point

A location at which the parameters governing a sequence are inferred to change.

Analytical redundancy

Independent estimates or constraints that allow one measurement to be checked against others or against a system model.

Identifiability

The ability to distinguish competing causes from the observations and assumptions available.

The changed architecture: alerts become small diagnoses

A time-series platform should stop treating every deviation as one object called anomaly. The platform needs a temporal baseline, a change detector, a measurement model and a typed alert contract. These components can be simple. Their separation is the important design decision.

The temporal baseline explains recurring phase and expected uncertainty. The change detector accumulates evidence for persistent departure. The measurement layer checks peers, constraints, timestamps and exporter health. The alert contract records which hypothesis is supported, which remains possible and what observation would discriminate them next. Recovery is verified against the same models rather than inferred from the disappearance of a red marker.

The benchmark must mirror that architecture. Inject seasonality, regime change, rare events and sensor faults. Freeze the reference period. Include a negative control where a simple threshold should win. Measure false pages, type confusion, detection delay, boundary error and whether the chosen runbook fits the cause. The goal is not to maximise anomaly scores; it is to minimise wrong operational decisions under changing time and imperfect measurement.

That is the durable intuition. A surprising point is only the beginning of an investigation. The useful unit is a causal hypothesis about what changed, where it changed and what evidence would make the system safe to act.