The number belongs to a question

Imagine a river leaving one source lake. Downstream it splits. One branch asks how unpredictable the source is. Another asks how much one observation tells us about another. A third chooses a distribution when knowledge is incomplete. A fourth asks how many bits preserve what a user cares about. A fifth asks whether several sources reveal a target separately, redundantly or only together. They share mathematical water, but they do not reach the same sea.

A coin before an equation

A fair coin is hard to predict before it lands. A coin that shows heads 99 times in 100 is easy to predict most of the time. Suppose heads has probability 0.75 and tails 0.25. Seeing tails is more surprising because the model assigned it a smaller probability. Shannon’s key move was to measure the surprise of an outcome by the negative logarithm of its probability, then average that surprise across outcomes.¹

For the 75:25 coin, the entropy is about 0.811 bits per toss. A fair coin reaches one bit. A deterministic coin reaches zero. The equation compresses a practical result: a long sequence from the biased coin can, in principle, be encoded with fewer bits per toss than a fair sequence because its regularity is predictable.

This is why statements such as “the information in the prompt” or “the information in the model” can be incomplete. A prompt may be decodable only with cultural knowledge. A model state may become useful only when paired with a tokenizer, runtime and observation interface. Information measures can represent those relations, but only after the variables and boundary are named.

A sample is not its own probability model

Write the sequence 0101010101 on a page. Count individual symbols and half are zero, half are one. An independent-symbol model therefore assigns one bit of entropy to each position. Yet after the first symbol, a reader who recognises the alternating rule predicts every later symbol perfectly. Under a model conditioned on the previous symbol, H(X_t|X_t−1) is zero. The page stayed fixed while the model’s state and unit of analysis changed.

This is not a trick. Entropy is calculated over random variables, not over uninterpreted ink. The variable might be the next bit, the next pair, a whole message, or a state transition. If we group the alternating sequence into pairs beginning at the first position, every pair is 01 and the pair entropy is zero. Shift the grouping by one position and every complete pair is 10. If the starting phase is itself uncertain, one bit may be needed once, after which the stream is determined.

Real datasets make this choice harder. A byte histogram ignores words, grammar and long-range structure. A token model sees some regularities and misses others. An empirical entropy estimate also carries finite-sample bias: unseen outcomes look impossible, rare outcomes are poorly estimated, and an over-flexible model can memorise the sample. Compression length can serve as a useful operational proxy, but it then measures performance of a particular compressor with its own vocabulary, context window and overhead.

flowchart TD
  A["The number belongs to a question"] --> B["Four measures, four contracts"]
  B --> C["When the whole knows what no<br/>part knows"]
  C --> D["The measure-comparison lab"]
  D -. "re-test" .-> A
  classDef first fill:#0a1424,stroke:#D95545,stroke-width:2px,color:#e6f3ff
  classDef step fill:#0a1424,stroke:#3f7da0,stroke-width:1px,color:#e6f3ff
  class A first
  class B,C,D step

Four measures, four contracts

Mutual information asks what one variable reveals about another

Take a rain sensor in a city where rain and dry weather are equally likely. The sensor reports the correct state 90 per cent of the time. Before reading it, uncertainty about the weather is one bit. After reading it, some uncertainty remains because the sensor can be wrong. The average reduction is about 0.531 bits.

Mutual information is zero when X and Y are independent under the distribution. It rises as knowledge of one reduces uncertainty about the other. It is symmetric, so I(X;Y) = I(Y;X). That symmetry is useful for dependence and dangerous for causal storytelling. Rain may cause a sensor reading; the sensor reading does not cause rain. A hidden controller could also cause two indicators to correlate.

Maximum entropy asks how not to invent what is unknown

A colleague says a six-sided die will be rolled, but supplies no other information. The distribution with maximum entropy assigns one sixth to each face. Now the colleague adds one reliable constraint: the expected face value is 4.5. Uniform probabilities no longer satisfy what is known. Maximising entropy subject to the new constraint produces a tilted distribution, with more mass on high faces.

Jaynes presented maximum entropy as a rule for constructing the least-committal distribution compatible with stated knowledge.² It does not say the physical die “seeks entropy”. It says the analyst should not insert distinctions that the constraints do not justify.

What maximum entropy cannot repair

The method inherits every mistake in the state space and constraints. If the die is chipped so that it can rest on an edge, a six-face support is wrong. If the reported mean came from a biased sample, enforcing 4.5 with certainty is too strong. If a hidden manufacturing process couples consecutive rolls, a distribution over isolated faces misses the dependence. Maximum entropy prevents unsupported detail within a model; it does not certify that the model contains the right possibilities.

There is also a difference between uncertainty about the world and randomness in the world. The maximum-entropy distribution may represent an analyst’s limited knowledge even when the underlying mechanism is deterministic. Conversely, a physically stochastic process can be described badly by a low-entropy model. The numerical entropy alone does not reveal whether uncertainty is epistemic, aleatory or a mixture.

Rate-distortion asks what may be lost at a given cost

Lossless compression tries to reconstruct the source exactly. Many systems do not need exact reconstruction. A music codec can discard differences people rarely hear. A maintenance system may discard most vibration detail while preserving short transients that predict bearing damage. Rate-distortion theory makes the trade explicit: what is the smallest information rate that keeps expected distortion below a chosen threshold?³

The distortion function is not an afterthought. It decides which differences count. Mean-squared error treats every numerical deviation according to its square. A lesion-weighted medical loss can penalise a tiny missed region more than a large harmless background shift. A text summariser might preserve named obligations while tolerating changes in style. The achievable rate changes because the task changes.

This reversal matters in machine learning because convenient losses often stand in for inaccessible consequences. Pixel error stands in for perceptual fidelity. Token likelihood stands in for usefulness. Reconstruction accuracy stands in for preserved state. These proxies can be good engineering choices, but their legitimacy is empirical. They must be checked against the downstream decision rather than treated as neutral definitions of quality.

Rate-distortion can also illuminate representation design without deciding semantics. If a compact state preserves all variables needed for a control task under the chosen loss, it is sufficient for that task. It may still discard autobiographical context, social meaning or any structure a theory of experience considers relevant. Task sufficiency is not ontological sufficiency.

The information bottleneck extends this intuition by defining relevance through another variable Y: compress X while preserving information useful for Y.⁵ This is powerful precisely because relevance is named. It is not an intrinsic halo around a representation. Change Y from “speaker identity” to “words spoken” and a different compression becomes useful.

When the whole knows what no part knows

Two fair coins are tossed. Let X₁ and X₂ record their outcomes. Define a target Y that is one when the coins differ and zero when they match. If you see only X₁, the target remains a fair guess. The same is true for X₂. See both and Y is certain.

Now change one causal property. Let X₁, X₂ and Y all be identical copies of the same fair bit. Each source individually provides one bit about Y. The pair still provides only one bit because the second copy adds nothing. This is redundancy. XOR and duplicate-copy systems have the same joint mutual information with Y, one bit, but distribute access differently.

A three-variable quantity called co-information can distinguish the signs of these clean cases. In one common sign convention, I(X₁;Y) + I(X₂;Y) − I((X₁,X₂);Y) is +1 for duplicate copies and −1 for XOR. Yet when redundancy and synergy coexist, the signed total can cancel them. It cannot tell how much of each is present.

Partial information decomposition, or PID, was introduced to split target information into redundant, unique and synergistic atoms.⁶ The ambition is clear. The measurement is not uniquely settled. Different redundancy definitions yield different decompositions; later work has supplied alternative operational criteria and logical constructions.⁷⁸⁹ A 2026 analysis proves important inconsistency results for lattice-based multivariate decompositions beyond the simplest cases, while proposing alternatives.¹¹

Synergy belongs to a target, not to a system in isolation

Keep the two input coins and their physical relation fixed. First choose Y = X₁ XOR X₂; the target is available only jointly. Then choose Y = X₁; the first source alone carries the target and the second adds nothing. Next choose Y as the ordered pair (X₁, X₂); each source contributes unique information about one component. The mechanism did not change. The informational decomposition changed because the question did.

Estimation adds another layer. Mutual information is difficult to estimate in high-dimensional or continuous spaces. Binning can create or erase dependencies. Flexible density estimators can overfit. Sparse samples can make rare joint events invisible. A PID then inherits the joint-distribution error and adds dependence on a decomposition rule. The correct response is not to abandon the question, but to use synthetic ground truths, estimator comparisons, uncertainty intervals and held-out perturbations.

A useful synergy claim therefore has two levels. The mathematical level says how information is decomposed under a named definition. The mechanistic level asks whether ablating, combining or rerouting sources changes target performance as predicted. Agreement strengthens the interpretation. A numerical atom without intervention remains a description of the fitted distribution.

The metric-laundering failure

Information language often travels through a chain of substitutions. High entropy becomes “rich information”. Rich information becomes “complexity”. Dependence becomes “integration”. Integration becomes “awareness”. Each step may be defensible inside a specific theory, but the numerical measure alone does not supply the bridge.

This underdetermination is not a defect in information theory. Shannon deliberately isolated an engineering problem from semantics.¹ Later proposals can add semantics by adding structure. Kolchinsky and Wolpert, for example, define semantic information through correlations that are causally necessary for a system to maintain its own viability.¹⁰ Their intervention and viability criterion do conceptual work that Shannon entropy alone does not.

stateDiagram-v2
  [*] --> Claim
  state "A coin before an equation" as Claim
  state "Why the rare outcome<br/>contributes more" as Test
  state "A sample is not its own<br/>probability model" as Revision
  Claim --> Test: expose mechanism
  Test --> Decision: evidence survives
  Test --> Revision: rival survives
  Revision --> Claim: narrow or rebuild
  Decision --> [*]

The measure-comparison lab

A realistic configured-system example

Consider a synthetic maintenance system monitoring a pump. Eight vibration sensors produce high-rate waveforms. A model estimates whether a bearing fault will occur within thirty seconds. A compressor stores only a compact trace. A human operator sees an alert and may stop the pump.

The raw stream can have high entropy because it contains noise. That does not make the noise useful. Mutual information between a sensor feature and the future-fault variable measures predictive dependence. It does not prove the feature causes the fault or that the model uses it. PID can ask whether sensors carry redundant or jointly available fault information, but the answer depends on the decomposition and estimator. Maximum entropy can fill an unknown joint distribution from declared moments, but only if those constraints are defensible. Rate-distortion can reduce storage, but the loss must penalise erased fault transients rather than only waveform error.

The failure mode is easy to imagine. Engineers optimise compression with mean-squared error, demonstrate a good rate-distortion curve, then discover that brief fault signatures were smoothed away. The mathematics was correct. The loss was wrong for the decision. A second failure occurs when a sensor’s mutual information with faults comes from a common operating regime rather than a causal precursor. The dependency disappears after the plant schedule changes.

The five-line information contract

The paper’s practical contribution is a compact contract that must accompany every consequential information claim. The first four lines determine what is measured. The fifth prevents the result from being promoted beyond its warrant.

1. Ensemble. Name the random variables, possible states, probability model, sampling window and units.
2. Target. State exactly which uncertainty or reconstruction is being evaluated.
3. Access. State what observations, keys, conditioning variables, boundaries and joint sources are available.
4. Loss or constraint. State which errors, moments, costs or fidelity conditions shape the quantity.
5. Licensed inference. Write the strongest conclusion allowed and the tempting conclusion that remains prohibited.

A contract-sensitivity protocol

The five lines become more useful when treated as variables rather than paperwork. Start with a baseline contract. Then define a small set of plausible alternatives before looking at the result. Change the temporal window, target granularity, access boundary or loss one at a time. Recompute the quantity and the resulting system ranking. Record which conclusions remain stable and which reverse.

For the pump system, compare two compressors. Under waveform error, compressor A wins. Under a fault-event loss, compressor B wins. Under a combined loss, the choice depends on the cost assigned to missed events. This does not mean “anything goes”. It identifies the decision surface that was hidden by a single score. Engineers can then negotiate costs, collect outcome data or design a representation that lies on a better frontier.

The protocol has three safeguards. Plausible alternatives must be specified from domain knowledge, not selected after seeing which one favours a preferred system. Estimation uncertainty must be separated from contract sensitivity. Finally, the allowed conclusion must remain scoped to the tested family. Stability across four losses does not prove stability across every future use.

The notebook tests whether a conclusion survives a change in specification. A positive stability result permits a claim within the tested contract family. It does not make the metric intrinsic. A ranking reversal means the original claim depended materially on target, boundary or loss. A flat result can mean genuine invariance, an insensitive example or insufficient measurement resolution.

const log2 = x => Math.log(x) / Math.log(2);
const h2 = p => (p <= 0 || p >= 1) ? 0 : -p*log2(p) - (1-p)*log2(1-p);

function entropy(values) {
  return -Object.values(values)
    .filter(p => p > 0)
    .reduce((s, p) => s + p * log2(p), 0);
}

function maxEntDie(targetMean) {
  const faces = [1,2,3,4,5,6];
  let lo = -40, hi = 40;
  const distribution = lambda => {
    const w = faces.map(x => Math.exp(lambda * x));
    const z = w.reduce((a,b) => a+b, 0);
    return w.map(v => v/z);
  };
  for (let k=0; k<100; k++) {
    const mid = (lo+hi)/2;
    const p = distribution(mid);
    const mean = p.reduce((s,v,i) => s + v*faces[i], 0);
    if (mean < targetMean) lo = mid; else hi = mid;
  }
  return distribution((lo+hi)/2);
}

sequenceDiagram
  participant Q as Question
  participant M as Mechanism
  participant E as Evidence
  participant D as Decision
  Q->>M: The number belongs to a question
  M->>E: A coin before an equation
  E-->>M: compare rival explanations
  E->>D: Write the question beside the information number
  D-->>Q: revise scope or act

Write the question beside the information number

Stop asking “How much information does this system contain?” as though the answer were a single property. Ask which uncertainty, for which observer or subsystem, under which distribution, with which access, and at what cost of error.

Use entropy when the decision concerns uncertainty or coding length. Use mutual information when the decision concerns statistical dependence between declared variables. Use maximum entropy when partial knowledge must become a probability distribution without adding unsupported structure. Use rate-distortion when loss is acceptable and the loss function is decision-valid. Use PID when the distribution of target information across sources matters, while declaring the decomposition and its limits.

For claims about minds, add a further obligation. State the consciousness theory or metaphysical bridge that makes the measured structure relevant. Then specify an intervention that could distinguish that account from a rival generator. A high number may become evidence inside such an argument. It is never the argument by itself.

The five-line contract changes design reviews, research papers and public claims. It turns “the model has more information” into a checkable statement. It exposes when a compression win depends on the wrong loss. It shows when a synergy claim depends on a contested decomposition. It prevents an information statistic from being laundered into meaning, agency or experience.

The practical decision is simple: never optimise, compare or philosophise from an information number until the question that gives the number its meaning has been written beside it.

Glossary

Entropy

Expected surprise of outcomes under a declared probability distribution.

Conditional entropy

Remaining uncertainty about one variable after another is observed.

Mutual information

Average reduction in uncertainty shared by two random variables.

Maximum entropy

A method for selecting the least-committal distribution satisfying stated constraints.

Distortion function

A declared penalty for reconstructing one source outcome as another.

Rate-distortion function

The minimum information rate needed to keep expected distortion below a tolerance.

Redundancy

Target information available from more than one source separately.

Synergy

Target information available only from sources considered jointly, relative to a decomposition.

PID

Partial information decomposition, a family of frameworks for splitting multivariate target information.

Measurement contract

The ensemble, target, access, loss or constraints, and licensed inference attached to a quantity.

References

1. Shannon, C. E. (1948). A mathematical theory of communication. Bell System Technical Journal, 27, 379–423 and 623–656.
2. Jaynes, E. T. (1957). Information theory and statistical mechanics. Physical Review, 106, 620–630.
3. Shannon, C. E. (1959). Coding theorems for a discrete source with a fidelity criterion. IRE National Convention Record, 4, 142–163.
4. Kullback, S., & Leibler, R. A. (1951). On information and sufficiency. Annals of Mathematical Statistics, 22(1), 79–86.
5. Tishby, N., Pereira, F. C., & Bialek, W. (2000). The information bottleneck method. Proceedings of the 37th Allerton Conference on Communication, Control and Computing.
6. Williams, P. L., & Beer, R. D. (2010). Nonnegative decomposition of multivariate information. arXiv:1004.2515.
7. Bertschinger, N., Rauh, J., Olbrich, E., Jost, J., & Ay, N. (2014). Quantifying unique information. Entropy, 16(4), 2161–2183.
8. Ince, R. A. A. (2017). Measuring multivariate redundant information with pointwise common change in surprisal. Entropy, 19(7), 318.
9. Gutknecht, A. J., Wibral, M., & Makkeh, A. (2021). Bits and pieces: understanding information decomposition from part-whole relationships and formal logic. Proceedings of the Royal Society A, 477, 20210110.
10. Kolchinsky, A., & Wolpert, D. H. (2018). Semantic information, autonomous agency and non-equilibrium statistical physics. Interface Focus, 8, 20180041.
11. Lyu, A., Clark, A., & Raviv, N. (2026). Multivariate partial information decomposition: constructions, inconsistencies, and alternative measures. Physical Review E, 113, 034102.