Two alarms, equally rare

A factory monitors 10,000 batches. Two proposed alarms each fire on exactly 100 batches, so either alarm event has probability 0.01 and surprisal −log₂(0.01) = 6.644 bits. One alarm is useless. The other identifies most of the batches that later fail.

For the first alarm, ten of its hundred firings precede failure. That is also the base failure rate: 1,000 of 10,000 batches fail. Knowing that the alarm fired leaves the failure probability unchanged at 10 per cent. The event is rare, visually dramatic and expensive to encode under the marginal model. It carries zero information about the target.

For the second alarm, ninety of its hundred firings precede failure. Its rarity is identical, but the joint arrangement is different. Knowing that it fired changes the target distribution sharply. Averaged across all batches, it saves 0.0259 bits when encoding failure or non-failure. That sounds small because the alarm occurs only one per cent of the time. On a fired-and-failed batch, however, the local evidence is 3.17 bits in favour of dependence.

The difference is not the amount of surprise in the alarm. It is the amount by which the alarm changes the code for something we care about. This distinction connects four ideas that are often blended together: surprisal, entropy, compression and mutual information. It also supplies a practical test for machine-learning features, anomaly signals, representation objectives and operational alerts.

Surprise belongs to a model

Imagine receiving the message “the pump stopped”. Its information content is not stored in the sentence like mass in a parcel. During a planned shutdown, the message was highly probable. During normal operation, it may have been unlikely. Under one model it costs almost no bits to predict; under another it costs many. The physical event can stay fixed while its formal surprise changes.

This formula earns its interpretation from coding. A prefix code cannot give every possible message a short word. Frequent messages should receive short words, and infrequent messages must pay with longer ones. Ideal lengths can be fractional even though a single binary codeword has an integer length. Across blocks of messages, the average can approach the ideal. Shannon made this bridge between probability and efficient communication precise.^[1]

The subscript on i matters. It records whose probability model priced the message. When the model is accurate, a long code identifies a genuinely uncommon outcome under the operating distribution. When the model is stale, a long code may identify a forecasting error. Surprise is always evidence about the pair consisting of an observation and a model.

Three different questions called entropy

Entropy appears across communication, thermodynamics, cryptography and machine learning. Shannon entropy describes average uncertainty under a probability distribution. Cryptographic work may emphasise min-entropy, a worst-case measure tied to the most likely outcome and guessing risk.^[10] Algorithmic information asks for the shortest effective description of an individual object.^[9]

These are not interchangeable badges for “interesting”. A random nonce can have excellent cryptographic unpredictability and no predictive value for a failure target. A repeated diagnostic pattern can have low marginal entropy and high decision value. An individually intricate string can be incompressible in an algorithmic sense while remaining independent of the label under investigation. Name the random variable, the probability model and the target before interpreting an entropy number.

Rare can also be evidence against the target

The phrase “rare and informative” often hides another mistake: assuming that a surprising observation must increase the probability of the event being investigated. Pointwise mutual information can be negative. Suppose a diagnostic marker appears on one per cent of all cases but almost never appears among genuine failures. Seeing the marker is still surprising under its marginal model, yet it shifts probability away from failure. Its target contribution is informative precisely because it contradicts the favoured hypothesis.

This matters in anomaly systems. A previously unseen client version may explain a burst of unusual logs and thereby reduce suspicion of an attack. A rare but authenticated maintenance token may make a dangerous-looking command more consistent with an approved intervention. The event’s rarity and the direction of its target evidence are independent questions. One is priced by −log q(x). The other is priced by a likelihood ratio comparing the joint event with what independence predicted.

Operational interfaces should therefore avoid a single “information score” that mixes novelty and target evidence. Show at least the model-relative surprise of the observation, the direction and magnitude of its evidence for the declared target, and the age of the model that supplied the baseline. A red novelty badge can coexist with evidence that lowers risk.

flowchart LR
  A["Two alarms, equally rare"] --> B["Surprise belongs to a model"]
  B --> C["Entropy becomes a compression<br/>limit"]
  C --> D["Dependence is target code saving"]
  D -. "re-test" .-> A
  classDef first fill:#0a1424,stroke:#087A98,stroke-width:2px,color:#e6f3ff
  classDef step fill:#0a1424,stroke:#3f7da0,stroke-width:1px,color:#e6f3ff
  class A first
  class B,C,D step

Entropy becomes a compression limit

Consider a source that emits four symbols with probabilities 1/2, 1/4, 1/8 and 1/8. A fixed-width code needs two bits for every symbol. A prefix code can assign lengths 1, 2, 3 and 3. The average is 1.75 bits, exactly the source entropy. Common symbols borrow short paths; rare symbols pay for the privilege by taking longer ones.

Huffman’s construction finds a minimum-redundancy prefix code for known symbol weights.^[2] For arbitrary probabilities, one-symbol code lengths must be integers and may sit above the ideal. Coding long blocks, or using arithmetic-style schemes, can approach the entropy rate. Shannon’s noiseless coding theorem gives entropy its operational force: for a stationary source under its assumptions, long typical sequences can be represented near H bits per symbol, while reliable lossless coding below that rate is unavailable in the limit.^[1]

Entropy is therefore an average, not the code length of every outcome. For the one-per-cent alarm, the fired event costs 6.644 ideal bits, yet the binary variable’s entropy is only 0.0808 bits per batch because ninety-nine per cent of batches produce the predictable “no alarm”. Confusing these two quantities makes rare events look as though they dominate every stream even when they barely affect average storage or prediction.

One-symbol entropy can miss sequence structure

Now compare two binary streams with the same histogram. The first is an independent fair coin: approximately half zeros and half ones in no predictable order. The second alternates 0, 1, 0, 1 without exception, but begins with an unknown first bit. Each individual position has marginal probability one half for either symbol, so the one-symbol entropy is one bit in both streams. Their long-run compressibility is radically different.

After the first symbol of the alternating stream, every remaining symbol is determined. A code need only transmit the starting bit and the rule “alternate”. The independent stream continues to require about one bit per symbol. The relevant quantity for a source with memory is its entropy rate, the limiting uncertainty per symbol after dependencies across the sequence are accounted for:

This distinction reaches directly into machine intelligence. Token frequencies do not capture grammar. Independent feature entropies do not capture temporal state. A context model earns compression by predicting the next event from prior events. When a larger context window lowers held-out code length, it has captured sequence dependence under that evaluation distribution. Whether the dependence is useful for a later decision remains a separate target question.

The codebook is part of the measurement

A deployed compressor or probabilistic predictor rarely knows the true distribution p. It uses a model q. The average length it pays is cross-entropy. The gap between this cost and the best cost available under p is Kullback-Leibler divergence, introduced as an information measure by Kullback and Leibler.^[3]

This decomposition makes log loss an especially useful evaluation instrument. A probabilistic model that places too little probability on realised outcomes pays longer codes. Under regularity conditions, the logarithmic score is strictly proper: in expectation, truthful probabilities optimise the score rather than merely producing correct labels.^[6] Calibration, discrimination and operational value still need separate checks, but the code interpretation keeps probabilistic accuracy concrete.

Compression is not semantic understanding

A compressor exploits regularity. It need not know why the regularity exists. Repeating “authorised” a thousand times is highly compressible whether the word appears in a valid policy, a malicious injection or a corrupt log. Conversely, an unpredictable cryptographic token can resist compression while carrying no broad semantic content.

Minimum-description-length methods extend the coding idea to model selection: pay for a model and the residual data under that model, then prefer a short total description.^[4] This disciplines complexity, but the selected regularities remain relative to the model class and coding scheme. Kolmogorov’s algorithmic approach pushes toward the shortest programme producing an individual object, which clarifies why one string may be simple and another incompressible without an ensemble.^[9] General Kolmogorov complexity is not computable, and individual-object brevity still does not specify relevance to a target.

Dependence is target code saving

Entropy prices uncertainty in one variable. To ask whether an observation helps, introduce a target. Let Y be the outcome we must encode and X a candidate signal. First build the shortest expected code for Y alone. Then allow the coder to see X. The expected reduction is mutual information.

This definition makes “useful” precise but local. The target must be named. An alarm can have high information about machine type and zero information about failure. A text embedding can retain document identity while discarding the contractual clause needed for a decision. Tishby, Pereira and Bialek formalised relevant information in exactly this target-relative way: compress X while preserving what it says about Y.^[5]

Local evidence and average information

The rare precursor’s fired-and-failed cell has pointwise mutual information log₂(0.009/(0.01 × 0.10)) = log₂(9) = 3.17 bits. The event is nine times more common than independence predicts. Yet it occurs in only ninety of 10,000 batches. Other cells contribute smaller positive and negative terms. The average settles at 0.0259 bits.

This resolves an apparent contradiction. A rare observation may be strongly diagnostic when it occurs while contributing little average mutual information across the population. Pointwise evidence is relevant to the current case. Average mutual information is relevant to expected coding and broad feature value. A consequential design may need both, plus tail-specific loss. Do not use an average dependence measure to erase a rare but costly stratum.

Incremental information needs context

A second sensor can look informative only because it copies the first. Let Z denote signals already available. The correct admission question is often conditional mutual information:

Feature-selection research has connected many information-theoretic criteria to conditional likelihood and the assumptions used to approximate this incremental quantity.^[7] The practical implication is straightforward: rank a proposed signal against the system it will join, not against an empty baseline. A duplicate feed can look excellent in isolation and add no new evidence to a mature model.

Recoding cannot manufacture true dependence

Suppose a representation T is produced deterministically from X. Under a fixed joint distribution, post-processing cannot increase the true mutual information with Y: I(T;Y) ≤ I(X;Y). An invertible relabelling preserves the quantity because no distinctions are lost. A many-to-one compression can preserve or discard target evidence, but it cannot create evidence that was absent from X. This data-processing boundary is one reason the information bottleneck frames representation learning as a trade between compression of X and retention of information about Y.^[5]

Estimated values can appear to violate the intuition because estimators, finite samples and model classes change. A neural critic may fit T more easily than raw X and therefore produce a larger lower bound, even though the underlying true information did not grow. That can make T a better practical representation for the estimator without proving that the transformation generated new dependence. Separate a better measuring instrument from a larger underlying quantity.

XOR gives a hard boundary for simplistic feature ranking. Redundancy means several variables repeat the same evidence; synergy means a group supplies evidence that none carries alone. Real systems can contain both. Evaluate candidate sets, interactions and model classes against a held-out target code rather than assuming that a list of marginal scores composes additively.

Estimation is itself a model choice

True probabilities are rarely available. Histogram bins, smoothing, nearest neighbours, density models and neural variational bounds each impose assumptions. Paninski’s analysis shows why plug-in entropy and mutual-information estimates become unreliable when samples are sparse relative to the alphabet.^[8] In high dimensions, Poole and colleagues found important bias-variance trade-offs among variational bounds.^[11] McAllester and Stratos proved a sharper distribution-free limitation: a high-confidence lower bound estimated from N samples cannot generally grow beyond order log N.^[12]

These results do not make mutual information unusable. They change the evidence standard. Use estimators matched to data type and dimensionality. Report uncertainty. Include exact synthetic distributions where truth is known. Compare against permutation controls. Test code savings out of sample. Inspect sensitivity to binning, representation and model capacity. When neural bounds are optimised as objectives, do not quietly reinterpret the bound value as an accurate measurement of true mutual information.

A measurement protocol for dependence

Begin with a synthetic calibration suite whose joint distributions are known. Include independence, perfect copying, weak rare dependence, nonlinear dependence, redundancy, XOR synergy and high-cardinality noise. An estimator that cannot recover these controlled cases within declared tolerances has not earned interpretation on the production data.

Then separate fitting from measurement. Fit density models, discretisers or neural critics on one partition. Estimate target code gain on another. Use a third temporal or domain holdout for the release claim when the data volume permits it. Repeat across seeds or resamples and report intervals, not only the best run. For feature discovery across many candidates, account for selection: the largest observed estimate is partly a winner of noise.

Finally, replay the measurement under plausible representation choices. Change bins, smoothing strength, embedding dimension and category pooling. A dependence claim that reverses under minor admissible choices is estimator-sensitive evidence, not a stable property of the system. Preserve the full manifest so that later data can reproduce the old estimate under the old boundary before replacing it with a new one.

Admit signals by what they save

A production team rarely wants information in the abstract. It wants to decide whether a new signal deserves collection, storage, model complexity, monitoring and influence over an action. Entropy alone answers the wrong question. Raw mutual information answers only part of the right one. The admission test should move through four boundaries: source, target, generalisation and consequence.

Time is part of the information contract

Production labels mature after actions occur. A specialist-review label may arrive hours later, be corrected after investigation or remain unknown when a case is abandoned. A signal can also be logged after the decision it supposedly predicted. The evaluation manifest must therefore record event time, evidence-available time, decision time and label-finalisation time. Otherwise a future fact can leak backwards and create impressive code savings that no live system could realise.

The same discipline governs drift. Store the baseline and conditional code lengths by decision cohort, not only as one cumulative average. A rising baseline loss means the target distribution or existing model has changed. A stable baseline with collapsing conditional gain means the proposed signal has lost its incremental relation. Both conditions can trigger withdrawal, but they diagnose different failures. The fallback should reconstruct the decision using Z alone rather than substituting a stale narrative about what X used to mean.

Unknown outcomes need explicit treatment. Silently dropping unresolved cases changes the evaluated population, often in a way correlated with difficulty. Report their rate and reason. Where later labels arrive, replay the original decision with the evidence and model version available at that time. Information is valid at a decision moment, not merely somewhere in the completed record.

This formula also clarifies authority. A feature can improve a prediction while remaining unauthorised for a protected decision. It can arrive too late to change the action. It can increase operational fragility beyond its expected benefit. It can be informative about a proxy that policy forbids. Information theory prices predictive dependence. Governance decides whether that dependence may be used.

The information admission record

The following instrument makes the hidden choices reviewable. It should accompany any claim that a feature, representation, alert or retrieval result is “informative”.

Read the admission result as a diagnosis

High signal entropy with zero held-out target gain means the source is varied but irrelevant under the tested target and context. Do not collect or retain it merely because it looks rich. Low signal entropy with stable positive gain can be valuable: a rare deterministic flag may remove exactly the uncertainty that matters. Complexity is not a prerequisite for usefulness.

Positive training information with zero or negative held-out gain points first to leakage, memorisation, sparse estimation or shift. Freeze deployment and inspect the split before changing the model. Stable code gain with zero decision value means the signal improves probability estimates without altering any permitted action enough to repay its costs. It may still support forecasting or monitoring, but it has not justified decision influence.

Positive decision value with an authority failure is not a technical pass. Route the signal to an authorised reviewer or redesign the action boundary. Positive average value with a harmed subgroup or catastrophic tail also remains blocked until the loss and constraint model reflects that consequence. The instrument is useful because each failure suggests a different remedy rather than compressing every outcome into “informative” or “not informative”.

The browser lab exposes exact discrete logic. The companion Python programme adds a seeded train/test experiment with a useful sensor, an equally entropic shuffled control and the unique-identifier trap. It uses no external packages and fails its assertions if controls begin to look reusable or predictive features lose held-out code gain.

gain_bits = baseline_test_log_loss_bits - conditional_test_log_loss_bits

assert rare_precursor_gain > 0.015       # positive case
assert abs(rare_random_gain) < 0.005     # independence control
assert session_id_train_mi > 0.40        # memorisation appears strong
assert session_id_test_gain == 0.0       # but cannot price unseen IDs

release = (
    gain_bits > declared_minimum
    and permutation_gain <= 0.0
    and stable_across_time
    and expected_decision_value > 0.0
)

Run the embedded programme in this page. Expected output includes 6.644 bits of surprise for either one-per-cent alarm, zero mutual information for the independent table, 0.025890 bits for the precursor table, positive held-out gains for the precursor and common sensor, and zero held-out gain for the unique session identifier.

Glossary

Surprisal

The ideal code length −log₂q(x) assigned to one observation by a stated model q.

Entropy

The expected self-information of a random variable under a specified distribution, commonly measured in bits.

Cross-entropy

The expected code length paid when data follow p but the code or predictor uses q.

KL divergence

The expected excess code length from using q rather than p, under suitable support conditions.

Mutual information

The expected reduction in the code length of one variable after observing another.

Pointwise mutual information

The local log likelihood ratio for one pair of outcomes relative to independence.

Conditional mutual information

The additional expected code saving from X about Y after existing context Z is known.

Min-entropy

A worst-case unpredictability measure determined by the most likely outcome, often used in cryptographic entropy assessment.