More channels can mean less evidence

Multimodal learning is often introduced as a way to exploit complementary signals. The foundational literature rightly shows that modalities can expose structure unavailable to any one stream, and that shared representations can help when one modality is absent.¹ The broader taxonomy of multimodal machine learning separates representation, translation, alignment, fusion and co-learning because combining modalities is only one part of the problem.² That separation matters: a perfectly implemented fusion layer cannot repair incorrect identity, time alignment or semantic scope.

A modality is not a fact. It is a measurement process with its own field of view, sampling rate, transformations, failure modes and relationship to the target. A photograph records reflected light from one viewpoint. Audio records pressure variations over an interval. Text records someone's description, intention or interpretation. Structured data records values already selected, named and transformed by an upstream system. Their differences are not nuisances to be normalised away. They determine what each observation can establish.

Four quantities are commonly collapsed into one attention weight. Availability asks whether an input exists. Quality asks whether its current form is usable. Informativeness asks whether it changes belief about a specific proposition. Dependence asks whether its information is already carried by another input. A fifth issue, authority, asks whether the observation may authorise an action. Statistical fusion can address the first four imperfectly. It cannot infer the fifth merely from predictive success.

Unique, redundant and synergistic information

A useful vocabulary comes from information decomposition. A modality can contribute unique information available nowhere else, redundant information repeated across streams, or synergistic information that becomes predictive only when streams are considered together.³ These are properties of modalities relative to a target and a distribution. They are not permanent labels attached to image, audio or text.

Redundancy is valuable when it provides graceful degradation. If audio disappears, a vibration sensor may preserve enough signal to continue. Yet redundancy does not license adding confidence twice. Synergy creates the opposite problem. Neither lip movement nor an ambiguous phoneme may identify a word alone, but their alignment can. A late ensemble of independent predictions may miss this interaction because the decisive evidence exists between representations rather than within either one.

This gives fusion two distinct jobs. The first is to accumulate unique support without double-counting redundant support. The second is to learn interaction terms without letting a flexible interaction model obscure the contribution of each modality. Architectures that mix every feature immediately can be expressive, but they make those jobs difficult to inspect. Architectures that preserve unimodal predictors and model interactions separately create a more testable decomposition, even if the decomposition remains approximate.⁴

flowchart LR
  A["The bearing that told four stories"] --> B["More channels can mean less<br/>evidence"]
  B --> C["What has earned influence"]
  C --> D["Reinforce, override or unresolved"]
  D -. "re-test" .-> A
  classDef first fill:#0a1424,stroke:#7B4CD8,stroke-width:2px,color:#e6f3ff
  classDef step fill:#0a1424,stroke:#3f7da0,stroke-width:1px,color:#e6f3ff
  class A first
  class B,C,D step

What has earned influence

The compressor question hides several propositions: there is oil on the housing; the sound originates near the bearing; vibration exceeds a threshold now; the bearing is damaged; failure is likely within forty-eight hours; a shutdown is warranted. A modality can be strong evidence for one proposition and weak evidence for the next. The camera may establish a visible sheen but not its chemical identity. Audio may localise a click poorly. A structured sensor can establish a numeric reading, provided identity, units and time are valid. None of them alone authorises a shutdown.

Fusion should therefore begin with a proposition ledger, not a fused vector. Each observation records the proposition it bears on, its source identity, observation time, processing lineage, quality indicators and known dependence groups. The model may still use high-dimensional representations, but its decision interface should expose bounded claims rather than one undifferentiated confidence score.

Per-instance reliability is also multidimensional. A useful quality vector includes presence, parseability, signal fidelity, temporal alignment, entity alignment, task relevance and calibration evidence. A sharp image of the wrong machine has high visual fidelity and zero entity relevance. A current sensor reading may be precise but outside calibration. A fluent note may be legible and still describe a hypothesis rather than an observation. Compressing these conditions into one learned gate is convenient, but it hides which assumption failed.

From conditional independence to a usable mechanism

There is a clean special case. Suppose the modalities are conditionally independent once the true class is known, and each unimodal probability is calibrated to the same prior. Their evidence can then be combined in log-odds form:

Here, y is the target class, xᵢ is modality i, and π is the shared prior probability. Each bracketed term is that modality's log-evidence beyond the prior. The sum becomes more extreme as independent evidence accumulates. Classical classifier-combination work derives product and sum rules under related assumptions and also shows why those assumptions matter.⁵

Real systems violate the clean case. Inputs share sources, encoders are miscalibrated, some modalities are missing, and interaction terms matter. A conservative engineering approximation is:

L₀ is prior log-odds. uᵢ is modality-specific evidence; qᵢ is a current quality gate between zero and one. r_g is evidence shared within dependence group g, discounted by c_g. The term s represents synergy that cannot be recovered from separate predictions. Raising a quality gate increases a modality's unique contribution. Raising assumed dependence lowers the gain from repeated agreement. Increasing the interaction term allows aligned combinations to matter even when individual streams are weak.

This equation is a design scaffold, not a universal estimator. The quantities can be learned, specified or bounded, but each choice creates an evaluation obligation. In particular, a gate trained from the model's own confidence can be confidently wrong. Quality monitors need targets that differ from the final label, such as blur, clipping, timestamp drift, entity mismatch or out-of-distribution distance. Their relationship to correctness must then be tested under failure injection.

Reliability is not confidence

A classifier's confidence describes its own predictive distribution. Reliability describes how often a claim remains correct under stated conditions. The two coincide only when the predictor is calibrated for the population and regime being used. Modern neural networks can be substantially miscalibrated even when their accuracy is high; temperature scaling is one simple post-hoc correction, not a cure for distribution shift.⁶ A model can assign 0.97 to an image prediction because the image resembles familiar training examples while missing that the asset identity is wrong.

For that reason, a modality gate should answer several smaller questions. Is the source the intended source? Is the observation current enough for this proposition? Is the signal inside the acquisition envelope? Is the encoder operating near data on which its calibration was measured? Does a second observation share the same likely failure? These answers may be deterministic checks, specialist models or human attestations. Their outputs should remain visible rather than being compressed immediately into one number.

Uncertainty-aware multi-view methods offer one route. Trusted Multi-View Classification, for example, uses evidential uncertainty to discount views at the sample level and to handle conflict among views.⁷ This is useful published evidence that dynamic influence can outperform fixed weighting under its evaluated conditions. It is not proof that evidential networks recover true source reliability. Their uncertainty estimates are themselves model outputs, so they need calibration, corruption tests and comparison with independent quality signals.

The most important distinction is between quality of observation and strength of support. A pristine photograph can be excellent evidence that a gauge needle points right and weak evidence that the process pressure is safe, because parallax, scale interpretation and gauge calibration remain outside the pixels. Conversely, a noisy signal can still be decisive if its discriminating pattern survives the noise. Quality gates should alter the warrant, not substitute for the claim model.

Sometimes the best fusion operation is to acquire another modality

Static fusion assumes all available modalities have already arrived. Operational systems can often choose what to observe next. After image and audio disagree, requesting a fresh structured reading may be cheaper and more informative than running a larger fusion model. After a sensor reports a threshold breach, asking for an image may establish whether the value corresponds to visible leakage. This turns fusion into a sequential decision problem.

A simple acquisition policy estimates three quantities for each possible next observation: expected reduction in decision loss, acquisition cost and delay, and the chance that the observation shares the current failure mode. A new camera angle may add little after two cameras behind the same window. A manual temperature reading may add much because it crosses both sensor lineage and acquisition method. The policy can stop when the expected value of another observation falls below its cost or when delay itself creates unacceptable risk.

This sequential view also clarifies missingness. “No audio” may mean the microphone failed, the environment was silent, the recording was withheld for privacy, or the process ended before capture. These causes carry different information. Imputing all four as a neutral audio embedding erases that information. A better state records why the modality is absent, whether it can be reacquired and whether absence changes the target probability.

Where fusion should happen

Early, intermediate and late fusion are often presented as competing architectural families. They are better understood as different places to expose interaction and preserve inspectability. Early fusion combines raw or lightly processed features. It can model fine cross-modal alignment, but it assumes compatible sampling and makes corruption easy to propagate. Intermediate fusion combines learned representations, often through cross-attention or bottleneck tokens. It supports rich interaction but can hide modality dominance. Late fusion combines predictions or evidence. It is easier to calibrate, ablate and degrade gracefully, but may miss synergy present below the decision level.

Attention bottlenecks show that multimodal interaction can be restricted through a small set of fusion tokens rather than allowing every token to attend to every other token.⁸ Multimodal transformers for unaligned language sequences use directional cross-modal attention to adapt one stream to another without requiring explicit alignment.⁹ These are mechanisms for interaction, not proofs of reliability. A production design can use intermediate fusion to capture synergy while retaining late, modality-specific heads for calibration, diagnostics and fallback.

Reinforce, override or unresolved

“Fusion” sounds as though every valid input must end in one blended answer. A safer decision vocabulary allows three outcomes. Evidence reinforces when observations bear on the same proposition, pass quality checks and add sufficiently distinct support. One source overrides another when it provides stronger, more direct or more current evidence on that same proposition. A case remains unresolved when credible observations conflict, required evidence is absent, or the system cannot justify a dependence assumption.

Override is proposition-specific, not a permanent hierarchy among modalities. A calibrated temperature sensor should outweigh a photograph when the proposition is “the surface temperature is 94°C now”. A current image may outweigh yesterday's inventory code when the proposition is “this valve is physically open”. A technician note can explain why a sensor is in test mode, but it does not alter the recorded voltage. The design question is not whether structured data outranks text. It is which observation has the strongest warrant for the claim being decided.

A minimal worked example

Use the compressor example with a prior failure probability of 0.35, whose log-odds are approximately −0.619. Let the image, audio, note and structured signal contribute signed log-evidence of 1.4, 1.2, 0.7 and 2.0. Their current quality gates are 0.90, 0.80, 0.70 and 0.95. Image and audio share a mounting environment, so place them in one dependence group with a conservative dependence factor of 0.65.

A static evidence sum produces log-odds −0.619 + 1.4 + 1.2 + 0.7 + 2.0 = 4.681, or probability 0.991. The reliability-gated sum reduces this to about 0.982. The laboratory's dependence-aware heuristic divides the image-audio group by 1 + 0.65(2 − 1), then adds text and structured evidence. It produces probability 0.958. All three methods support failure, but the last avoids treating two related observations as fully independent.

The lower probability is not automatically better. It is better only if the dependence assumption describes the data-generating process. This is why the grouping must be tested against cases where image and audio fail together, separately and not at all.

Conflict is information, not noise to average away

Now reverse the production-shaped case. Image and audio strongly support failure because one common interference source distorts both. The note and structured signal oppose failure. A weighted average can land near 0.5 and hide the fact that two credible groups disagree. The operational distinction is important: “little evidence either way” and “strong evidence on both sides” have similar means but demand different next actions.

A conflict detector should therefore operate on grouped contributions before they are collapsed. If strong, high-quality groups support opposite propositions, the action can remain unresolved even when the final scalar crosses a normal threshold. The next step might be a new sensor read, a fresh image, an identity check or human inspection. Abstention is useful only when it changes information seeking.

The unimodal negative control

Consider a dataset where the text field explicitly contains the target code and images and audio are unrelated background. A sufficiently flexible multimodal network may learn to ignore the extra streams, but it can also overfit their accidental correlations, slow optimisation or weaken the strongest branch. In this setting, text-only performance is not an ablation after the main experiment. It is the serious competing baseline.

Wang, Tran and Feiszli found across several audio-video classification tasks that multimodal networks could underperform their best unimodal component. They traced the difficulty to overfitting from added capacity and to modalities generalising at different rates.¹⁰ The result does not prove that fusion is generally harmful. It establishes the negative control a fusion claim must beat: the strongest unimodal model with comparable tuning and capacity.

A valid release test also includes a shuffled-modality control. Pair each text observation with an image from another case while keeping class frequencies unchanged. If performance barely moves, the image branch may be decorative. If performance rises, the dataset may contain an identity or leakage defect. If a multimodal model beats the unimodal baseline only on the clean test split but loses under realistic corruption, the improvement may be a benchmark-specific bargain rather than durable evidence use.

Abstention needs a risk-coverage contract

Selective classification evaluates a model at the subset of cases it chooses to answer. Coverage is the fraction decided; selective risk is the error rate among those decisions. Methods for selective classification make this trade explicit rather than treating rejection as an afterthought.¹¹ In multimodal fusion, the reject condition should include low evidence and high conflict. Those are different reasons, and they should trigger different evidence-gathering actions.

A threshold of 0.70 for positive and 0.30 for negative is merely an example. If a false shutdown is cheap and a missed catastrophic failure is expensive, the thresholds should be asymmetric. If the next observation costs little, the unresolved region can widen. If no further evidence can arrive and a decision is mandatory, the system must escalate authority rather than disguising the forced choice as high confidence.

A production-shaped fusion system

A production design for the compressor case can preserve rich modelling while keeping evidence legible. The following sequence is practitioner synthesis, not a claim that one architecture fits every domain.

1. Ingest observations with identity and time. Bind every image, audio segment, note and measurement to an asset, source, event time and retrieval time before encoding.
2. Align only what should be aligned. Build temporal windows and entity joins explicitly. Record unmatched or ambiguous observations rather than forcing nearest-neighbour alignment.
3. Produce modality-specific claims. Each encoder returns a bounded prediction, evidence features and a calibration estimate for named propositions.
4. Run independent quality monitors. Detect blur, clipping, silence, schema breaks, stale timestamps, entity mismatch and distribution shift using signals that do not simply repeat the final classifier.
5. Represent known dependence. Group observations that share sensors, preprocessing, physical mounting, upstream sources or generated descriptions. Treat uncertain dependence as a reason for conservative discounting.
6. Model interactions deliberately. Use cross-attention, bottleneck tokens or an explicit interaction predictor where the task contains tested synergy.
7. Fuse into a proposition ledger. Preserve unimodal contributions, group discounts, interaction terms and conflicts alongside the aggregate probability.
8. Apply a consequence-aware policy. Map evidence state to decide, gather, defer or escalate. Keep predictive evidence separate from the identity and authority required to execute an action.
9. Verify outcomes and learn selectively. Compare the predicted state with inspection findings, record which modalities failed, and update gates only through a governed release process.

Missing modalities are not one condition. A stream can be absent by design, unavailable at random, missing because the environment became difficult, or deliberately withheld. ModDrop introduced modality dropout to make multimodal networks less dependent on any one stream.¹² More recent evaluations show that multimodal transformers can remain sensitive to missing inputs and that the most robust fusion strategy varies by dataset.¹³ Training-time masking is useful, but it does not prove robustness to informative missingness or degraded content that still looks present.

MultiBench makes this evaluation burden concrete by comparing methods across fifteen datasets, ten modalities and twenty prediction tasks, including robustness to noise and missing data.¹⁴ The lesson is not to copy one benchmark score. It is to evaluate generalisation, complexity and robustness together, with failure regimes that match the intended environment.

stateDiagram-v2
  [*] --> Claim
  state "Thought experiment 1: two<br/>cameras and one dirty window" as Claim
  state "Unique, redundant and<br/>synergistic information" as Test
  state "Thought experiment 2: the<br/>number stays still while time<br/>moves" as Revision
  Claim --> Test: expose mechanism
  Test --> Decision: evidence survives
  Test --> Revision: rival survives
  Revision --> Claim: narrow or rebuild
  Decision --> [*]

The fusion laboratory

The executable artefact makes the central mechanism visible. It generates binary cases with four unequal modalities, injects local corruption, hidden shared corruption and missingness, then compares four rules: mean probability, static log-evidence addition, per-instance reliability gating, and dependence discounting with conflict-triggered abstention.

The laboratory is intentionally synthetic. Its purpose is to reveal a failure pattern, not to establish a universal performance ranking. The quality signal detects local image corruption reasonably well but does not fully reveal the shared image-audio fault. That distinction lets the reliability gate succeed in one regime and remain overconfident in another.

Accuracy uses a 0.5 threshold even for cases the policy would abstain on. Brier score measures squared probability error. Coverage is the fraction outside the example decision interval [0.30, 0.70] and not conflict-forced. Selective risk is the error rate among those decisions. Lower Brier and selective risk are better. The dependence-aware rule loses a small amount on clean data, then gives up coverage to reduce error when the common cause appears. That is a design bargain, not free robustness.

Inspect the implementation in this paper. Run python multimodal_fusion_lab.py. The file contains fixed example cases, seeded simulation, positive and negative regimes, metric definitions and input validation.

def fuse_dependency_aware(case, rho=0.65):
    image_audio = []
    other = []
    for name, value in case.evidence.items():
        if value is None:
            continue
        contribution = case.quality[name] * value
        (image_audio if name in {"image", "audio"} else other).append(contribution)

    group = sum(image_audio) / (1 + rho * (len(image_audio) - 1)) if image_audio else 0
    remaining = sum(other)
    strong_conflict = (
        min_present_quality(case, {"image", "audio"}) >= 0.65
        and abs(group) >= 1.25
        and abs(remaining) >= 1.25
        and group * remaining < 0
    )
    probability = logistic(logit(PRIOR_POSITIVE) + group + remaining)
    return probability, strong_conflict

A release decision instrument

A multimodal system should not pass review because its full-input score exceeds a single-modality score on one clean split. The release gate below asks whether fusion has earned the additional complexity and confidence it creates.

The failure boundary

Reliability-weighted fusion ceases to help when its reliability signals do not predict correctness, its dependence model misses the dominant common causes, or its proposition alignment is wrong. The system may then convert measurement defects into confident decisions more efficiently than a simpler model.

Three boundaries deserve particular emphasis. First, a learned quality gate can be manipulated by an attacker or shifted by a new device. Second, hidden dependence is not fixed: a software update can make previously independent sensors share preprocessing. Third, statistical reliability cannot resolve normative authority. A source can be highly predictive and still be prohibited, unentitled or inappropriate for the decision.

These limits change the architecture. Quality gates need independent tests and versioned thresholds. Dependence groups need lineage from physical and digital sources. Conflict needs to be retained before aggregation. High-consequence actions need a deterministic authority layer outside the fusion model. The operating evidence should support counterfactual replay: what would the decision have been without the image, with the current timestamp, or under a stronger dependence discount?

The adjacent article Multimodal Context for Lending and Financial Crime takes this last problem into document-heavy banking workflows, where provenance, purpose and authority determine whether multimodal context can support a regulated decision. The present article stays at the general fusion mechanism; the banking article carries the domain-specific evidence contract.

sequenceDiagram
  participant Q as Question
  participant M as Mechanism
  participant E as Evidence
  participant D as Decision
  Q->>M: The bearing that told four stories
  M->>E: Thought experiment 1: two cameras and one dirty window
  E-->>M: compare rival explanations
  E->>D: Fuse only what has earned influence
  D-->>Q: revise scope or act

Fuse only what has earned influence

The compressor did not provide four votes. It provided four observations produced by different processes, about partly different propositions, with different times and shared causes. Treating them as equivalent features created the appearance of corroboration. Treating them as evidence exposed identity error, staleness, redundancy and conflict.

The changed architecture is concrete. Preserve modality-specific claims. Validate quality per instance. Discount known dependence. Model synergy explicitly. Keep strong disagreement visible. Connect unresolved states to information seeking. Place action authority outside the predictive fusion mechanism. The result can still use deep encoders and cross-attention, but its operational contract is a reliability-weighted evidence braid rather than an opaque fused score.

The changed evaluation is equally concrete. No multimodal claim should survive without the strongest unimodal baseline, shuffled-modality negative controls, missing and corrupted inputs, shared-fault injection, calibration and risk-coverage analysis. Full-input accuracy is evidence about one regime. It is not evidence that the system knows when one channel lies, when two channels repeat the same error or when the correct answer is unresolved.

The practical decision is therefore restrained: use fusion when interactions or graceful degradation earn their cost, use simpler mechanisms when one modality already carries the task, and abstain when credible evidence groups cannot be reconciled. A multimodal system becomes trustworthy not when every channel speaks, but when each channel receives only the influence its warrant supports.