A gate, not a cell

Strip away the name and a familiar artificial neuron contains two operations. First it forms a weighted sum of its inputs. Then it transforms that score through an activation function.

Here x is the input vector, w is a vector of adjustable weights, b is a bias, z is the resulting score, φ is the activation and a is the output. Increasing one weight makes its corresponding input matter more. Changing the bias moves the point at which the unit switches or bends. Changing the activation alters how sharply the output responds and whether stacked units can create genuinely nonlinear functions.

With a hard threshold, the unit returns one when z ≥ 0 and zero otherwise. With a logistic sigmoid it returns a smooth value between zero and one. With a rectified linear unit it returns zero for negative scores and the score itself for positive ones. These choices matter for optimisation and function shape, but all begin with the same affine score.

The early neuron metaphor was not arbitrary. McCulloch and Pitts used idealised all-or-none units to relate nervous activity to logical propositions, while Rosenblatt's perceptron made the weights adaptive.[1][2] The abstraction was productive because it turned a vague ambition into a calculable system. Yet historical origin does not settle present meaning. A useful abstraction earns its name by the relation it preserves, not by the details it evokes.

A point model buys clarity by deleting causes

A biological neuron has a spatially extended dendritic tree, a membrane whose state evolves over time, voltage-dependent conductances, synapses with their own dynamics and a history that can change later responses. Hodgkin and Huxley showed that membrane excitation depends on interacting ionic currents with time-dependent dynamics.[9] Later work has established that dendrites can perform substantial linear and nonlinear processing rather than merely adding inputs at one point.[10]

The point unit deletes almost all of that. It accepts a vector now, calculates a score now and emits a value now. Any spatial arrangement, arrival sequence, refractory effect or biochemical state matters only if the modeller first encodes it into x or adds explicit dynamics elsewhere.

This deletion is often exactly right. A model built to classify a fixed feature vector does not become better merely because its unit simulates ion channels. Detailed mechanism would add cost without necessarily improving the decision map. The error begins when a claim silently crosses levels: from “this architecture approximates the observed input-output relation” to “this is how a biological neuron performs the computation”, or from “the model contains artificial neurons” to “the model therefore has brain-like cognition”.

Thought experiment: preserve the sum, change the time

Imagine two inputs, each delivering three pulses. In condition A, the six pulses arrive within a narrow interval. In condition B, the same pulses arrive widely separated. Keep the total count, source identities and nominal weights unchanged. A static point unit that receives only the two counts must produce the same result in both conditions. A dynamical cell may respond differently because membrane state, channel kinetics and dendritic location make timing causal.

The point is not that a particular biological cell must spike in condition A and remain silent in B. The figure is an intervention on the model boundary. If timing could change the target and timing is absent from the representation, no amount of fitting can recover the missing cause.

The geometry of one cut

Return to the equation z = wᵀx + b. In two dimensions, every location for which z = 0 lies on a line. In three dimensions it is a plane. In any number of dimensions it is a hyperplane. The weight vector points towards increasing score, while the bias shifts the boundary without changing its orientation.

A threshold unit therefore divides its representation space into two half-spaces. A sigmoid softens the transition, but because the sigmoid is monotonic, the probability contour at any fixed level is still a hyperplane. Smoothness helps gradient-based training. It does not turn one affine score into a curved or disconnected decision region.

Why XOR refuses one boundary

The parcel rule makes the limitation exact. Suppose a threshold unit must classify (0,0) and (1,1) as zero, while classifying (1,0) and (0,1) as one. Let a non-negative score mean class one.

The first negative point requires b < 0. The two positive points require w₁ + b ≥ 0 and w₂ + b ≥ 0, so each weight must be at least −b. Adding those requirements gives w₁ + w₂ + b ≥ −b, which is positive because b is negative. Yet the final negative point requires that same expression to be below zero. The requirements contradict one another.

This is more than a familiar classroom curiosity. It is a compact proof that an architectural failure can survive perfect labels, infinite repetitions of the four examples and an optimiser that has searched every possible setting. No parameter value exists inside the single-unit hypothesis class.

Minimal worked example: AND succeeds, XOR does not

An AND rule labels only (1,1) positive. The score x₁ + x₂ − 1.5 is positive only at that corner, so one unit solves the task exactly. OR is similarly easy with bias −0.5. These positive controls matter because they show that the implementation and learning procedure can work on a matched task.

On raw XOR, logistic training settles at equal probability for every case in the deterministic laboratory below. A different objective or an incomplete fit can instead choose a boundary that gets three of the four points right. Neither outcome repairs the structural defect. The model must compromise because its score orders all four cases along one axis.

The negative control: change the representation, keep the unit

Add one engineered feature, q = (x₁ − x₂)². It is zero when the two inspectors agree and one when they disagree. A single threshold on q now implements XOR. An equivalent feature set includes the interaction x₁x₂, because x₁ + x₂ − 2x₁x₂ equals XOR for binary inputs.

This negative control prevents a seductive but false conclusion: that hidden layers are inherently required for the concept of disagreement. They are required only when the input representation does not already expose that relation in a linearly separable form. Architecture and representation are alternative places to pay for the same expressive work.

flowchart TD
  A["The parcel that two inspectors<br/>disagree about"] --> B["A gate, not a cell"]
  B --> C["The geometry of one cut"]
  C --> D["Composition creates the<br/>missing representation"]
  D -. "re-test" .-> A
  classDef first fill:#0a1424,stroke:#B96D00,stroke-width:2px,color:#e6f3ff
  classDef step fill:#0a1424,stroke:#3f7da0,stroke-width:1px,color:#e6f3ff
  class A first
  class B,C,D step

Composition creates the missing representation

A hidden layer gives later units inputs that did not exist in the original data. Each hidden unit forms its own cut or bend. The output unit then combines those learned responses. The critical change is therefore not simply “more parameters”. It is the ability to construct intermediate coordinates.

XOR can be built with two threshold features. Let h₁ detect OR and h₂ detect AND:

The first hidden unit activates whenever at least one input is on. The second activates only when both are on. The output activates when OR is present but AND is absent. For (0,0), both hidden units are zero. For the mixed cases, only h₁ is one. For (1,1), both are one and the negative output weight on h₂ cancels the result.

A smooth multilayer perceptron generalises the same idea. With one hidden layer of m units, a scalar output can be written as follows:

Each wⱼ and bⱼ positions a hidden feature; φ makes that feature nonlinear; vⱼ tells the output how to combine it; c is the final bias; and m is the hidden width. Increasing m adds candidate pieces. Changing the hidden parameters changes the representation itself.

Backpropagation made these internal features trainable by propagating output error through local derivatives. Rumelhart, Hinton and Williams demonstrated how repeated weight adjustment could lead hidden units to capture task-relevant structure.[3] The important word is task-relevant. Hidden units are not assigned universal meanings. Their useful interpretation depends on the data, objective, architecture and optimisation path.

Production-shaped worked scenario: two routes to the same fault

Consider a synthetic cooling-system triage rule. A serious condition may arise when temperature is high while coolant flow is low, or when vibration is high while pressure drops. Either conjunction warrants escalation. One symptom from each route, such as high temperature plus high vibration with normal flow and pressure, should not trigger the same conclusion.

A single weighted score struggles because raising the weights enough to recognise each complete pair also raises mixed, incomplete pairs. Two hidden units can specialise: one detects the thermal-flow conjunction and the other detects the vibration-pressure conjunction. The output combines them as an OR. An engineer could instead supply the two interaction features explicitly. Which route is better depends on data volume, stability, audit needs and whether the interactions are already known.

The scenario also marks an important governance boundary. A learned interaction may improve discrimination without establishing that the model has identified the physical cause. Intervention, domain evidence and failure testing remain necessary. A hidden feature is a computational relation, not automatically a scientific explanation.

Universality without mythology

The move from one cut to many composed cuts leads to a powerful theorem. In one standard form, for every continuous target function on a compact domain and every positive error tolerance, a sufficiently wide one-hidden-layer network with a suitable activation has parameters whose output stays within that tolerance everywhere on the domain. Cybenko proved this for sigmoidal activations, Hornik and colleagues clarified the general approximation capacity of multilayer feed-forward networks, and later work characterised broad non-polynomial activation conditions.[4][5][6]

f is the continuous target, K is a compact set such as a closed bounded region, ε is the permitted approximation error, m is a width that may depend on the target and tolerance, and f̂ is the network. Making ε smaller demands a closer approximation and may require more units. Enlarging or changing K changes the claim. The theorem does not say that one fixed network works for every target.

Universality is a statement about representational existence. It says the function family is not blocked by the kind of geometric obstruction that defeated the single unit. It does not identify the needed width, provide the weights, certify that gradient descent will find them, show that finite data identify the target, or promise useful behaviour outside the domain on which approximation is claimed.

Thought experiment: every answer exists, but the index is missing

Imagine a library containing one book for every continuous function on a bounded region, each book encoding a network that approximates its function to a chosen tolerance. Your required book is certainly on a shelf. You possess only a finite set of examples, an optimisation rule and a limited search budget. The existence of the book does not reveal its shelf, distinguish it from books that match the examples but differ elsewhere, or tell you whether its encoding is economical.

Vary one feature at a time. First add a perfect catalogue that maps a complete target specification to the right shelf: the search problem disappears, but only because the target is already known. Next retain the missing catalogue and add more shelves: expressive capacity rises while search can become harder. Finally, keep capacity fixed but add informative data and an inductive bias aligned with the target: learnability may improve without any change to the approximation theorem.

Width proves possibility; depth can change the price

A shallow network may be universal yet require an impractically large number of units for a particular function. Depth can reuse intermediate computations and express some functions far more compactly. Formal separation results exhibit functions represented by modest deeper networks that demand exponentially greater width from shallower alternatives.[7][8]

These results do not imply that deeper is always better. Extra depth changes optimisation, latency, calibration and failure behaviour. It also creates more internal representations whose stability may be difficult to establish. Architecture, objective and learning rule must therefore be reviewed separately rather than attributed to the unit type alone.[15] The decision is comparative: which architecture reaches the required function, evidence and operating constraints at acceptable cost?

The biological comparison becomes stricter, not looser

Modern neuroscience makes the point-neuron analogy more qualified. Active dendrites support local nonlinear computations, and experiments in human cortical tissue have reported dendritic events with operations that a simple point model does not capture.[11] In a model-matching study, Beniaguev and colleagues needed a temporally convolutional deep network to reproduce the millisecond-scale input-output behaviour of a detailed cortical pyramidal-cell model; removing NMDA receptor dynamics greatly reduced the required depth.[12]

That finding should not be converted into a universal exchange rate such as “one biological neuron equals several artificial layers”. It concerns a particular detailed model, input regime, time resolution and approximation objective. It does show why the casual picture of a living neuron as one weighted sum can be materially wrong when cellular computation is the target.

The learning rule is another separation. Standard backpropagation uses downstream error information in ways that have long raised biological plausibility questions. Research explores mechanisms that could implement related credit-assignment principles, but no resemblance should be assumed merely because both systems adapt synaptic strengths.[13][14][16]

Composition changes the available questions

A useful way to understand a multilayer network is not as a crowd of individually intelligent units but as a sequence of coordinate changes. Each hidden unit asks whether its input lies on one side of a learned hyperplane and passes a graded answer through its activation. A following layer does not see the original case alone. It sees a new vector of these answers. A distinction that was not linearly accessible in the original coordinates can become accessible after this remapping.

For rectified linear units, the mechanism is especially concrete. Within any region where the active set is fixed, the network is an affine map. Crossing an activation boundary changes which affine map applies. The composed model therefore partitions input space into regions and assigns a locally affine rule to each. Depth can reuse earlier boundaries to build more intricate partitions, but the resulting function is still determined by weights, activations and architecture. Nothing in this construction grants goals, memory or understanding to a single unit.

Return to XOR. Two hidden units can create features corresponding roughly to “at least one input is on” and “both inputs are on”. The output unit can subtract the second feature from the first and separate the positive cases. The important event is representational: the hidden layer supplies a coordinate system in which the target is linearly accessible. Calling this merely a bigger classifier misses why it succeeds; calling each hidden unit a concept detector claims more than the evidence warrants.

This account yields a discriminating test. Freeze the output rule and intervene on hidden activations. If deleting one activation removes the distinction while matched perturbations with equal norm do not, that unit participates causally in the implemented computation. Yet participation is not semantic identity. A feature may be distributed across many units, one unit may serve several contexts, and an invertible change of hidden basis can preserve the overall function while changing which coordinate looks interpretable.

The same caution applies to feature visualisation. Maximising a unit can produce a compelling image without showing how typical data use that direction, whether downstream layers rely on it or whether the direction survives a harmless reparameterisation. Evidence should combine activation interventions, counterfactual pairs, downstream readout and stability across seeds. A picture of one maximiser is a hypothesis generator, not a certificate of meaning.

A causal test for the abstraction boundary

The abstraction should be expanded only when an omitted variable changes the required answer under a controlled intervention. Timing provides a clean example. Present two input streams with the same total weighted sum but different order. A static point unit must return the same value. A recurrent or stateful mechanism may distinguish them. If the task label changes with order, the static vector has erased a task-relevant cause before learning begins.

That diagnosis leads to different repairs. If a fixed window of lagged inputs restores the distinction, feature construction may be enough. If the relevant horizon varies, a recurrent state, attention over history or an explicit event model may be needed. If exact duration constraints matter, a timed automaton or symbolic guard can be cleaner than asking a dense network to infer the rule. Model choice follows the missing relation, not a generic preference for more layers.

A production evaluation should therefore include matched counterfactuals. Keep totals, marginal distributions and visible labels constant while changing chronology, grouping or intervention history. Measure whether the required output changes and whether the model follows. Then perform the reverse negative control: alter a nuisance detail while preserving the causal history. A useful representation should respond to the former and resist the latter within declared uncertainty.

Consider a fault-routing service that consumes aggregate telemetry. Two incidents have the same error count, mean latency and machine identifiers. In one, latency rises before errors; in the other, errors rise before latency. The first may indicate saturation, the second a retry storm. A point model receiving only aggregates cannot recover that ordering. Adding capacity or hidden units cannot restore information that the input contract deleted. The correct intervention is to represent sequence or causal landmarks and retest route selection.

Evaluation must also separate representability from learnability. Train several architectures under matched data, optimisation effort and parameter budgets. Include an engineered representation known to expose the target relation and a negative control with that relation deliberately destroyed. If the engineered baseline works while the end-to-end model does not, optimisation or sample efficiency remains a live explanation. If neither works, question the labels, support or claimed relation before celebrating a larger network.

The release record should name the claim level. For function approximation, record the covered input domain, tolerance and counterexample search. For learned prediction, add data exposure, calibration, shift tests and optimisation variability. For a mechanistic claim, add interventions on internal variables and rival explanations. For biological comparison, specify the measured cell type, time scale and causal process. Each higher level needs new evidence; performance at a lower level does not silently carry upward.

This hierarchy prevents two opposite mistakes. One is eliminative: rejecting artificial neurons because they omit cellular biology even when the engineering task only needs a stable nonlinear map. The other is inflationary: treating an accurate map as evidence that the model reproduces the natural mechanism or possesses the concepts described by its outputs. The point abstraction is valuable precisely when its deletions are declared and tested against the decision.

Resource accounting completes the comparison. A theorem that guarantees existence does not state how many examples, optimisation steps, parameters, joules or milliseconds are required to find and run the approximation. Record these budgets beside error and robustness. An architecture that represents the target but cannot be trained reliably within the available evidence is not an operational solution, while a simpler engineered feature may expose the same relation with clearer guarantees.

The evaluation set should include interpolation, controlled recombination and declared shift. Interpolation checks competence near observed cases. Recombination asks whether learned components work in combinations withheld during training. Shift changes a nuisance, measurement path or population factor while preserving the target mechanism. Results should remain separate because average success can hide a model that interpolates smoothly yet fails whenever the representation must transfer.

Calibration adds another boundary. A network can rank cases well while attaching probabilities that are too sharp, especially under shift. If a probability controls escalation or abstention, compare reliability curves and consequence-weighted calibration on the slices that determine action. Thresholds should be selected on separate evidence, then frozen for the release test. Retuning them after inspecting failures confounds model repair with policy repair.

Finally, retain rival explanations for apparent improvement. More layers may help because of representation, but also because parameter count, optimisation schedule, regularisation or data exposure changed. Use matched budgets and ablations where the conclusion depends on depth. If the advantage disappears when a shallow model receives the same engineered intermediate features, the evidence supports a representation claim, not a general claim that depth itself was necessary.

The monitoring plan must preserve this separation after deployment. Track input-support drift, calibration, counterfactual-pair performance and the frequency of unsupported cases independently. A stable average score cannot clear an architecture when the specific relation that justified it is deteriorating. If the critical pair test fails, narrow the action set or fall back before retraining; otherwise new data collected under the faulty decisions can make the original causal diagnosis harder.

Choose the abstraction by the claim

The artificial neuron remains one of engineering's most fruitful simplifications. The right response to its omissions is neither to discard it as biologically false nor to inflate it into a theory of mind. Use it where the preserved relation is the one under decision.

A practical review can proceed in three passes. First diagnose the geometry of the target in the current representation. Second ask whether feature construction or nonlinear composition is the cleaner source of expressive power. Third audit causal variables outside the static vector, especially time, state, intervention and constraints.

The abstraction contract

Before approving a neural model, record five short statements. Target relation: the input-output behaviour the system must reproduce. Retained variables: the features and state the model can actually use. Discarded variables: timing, morphology, context or mechanisms outside its boundary. Validation intervention: the controlled change that would reveal whether the simplification holds. Withdrawal trigger: the observation that requires a richer model or a narrower claim.

For the parcel example, the target relation is disagreement. Raw scanner and weight bits are retained; sensor histories are discarded. The geometry test shows one unit fails. The lifted-feature control shows that an explicit disagreement coordinate restores sufficiency. If decision quality later depends on sequences of sensor retries, the static abstraction loses validity and must be replaced by a stateful one.

This instrument changes both research and architecture. Research claims become conditional on the intervention that tested the abstraction. Architecture reviews stop treating more layers as a generic remedy. Teams can distinguish four diagnoses that are often conflated: missing representation, insufficient composition, failed optimisation and omitted causal state.