The number that changes the policy

To see what reward can establish, watch one learning step closely enough that every causal role remains visible.

Put a coin-operated gate at a junction. A controller observes a state, chooses an action and receives a number from the environment. That number might be a game score, energy cost, distance penalty, human-derived assessment or a mixture of several terms. In the formalism, its meaning comes from how the learning rule uses it. The scalar does not arrive with a label saying pleasure, pain, approval or desire.

Sutton and Barto separate a reinforcement-learning system into a policy, a reward signal, a value function and, optionally, a model of the environment.¹ That separation is more than vocabulary. A reward is the local signal supplied after a transition. A return is a discounted accumulation of future rewards. A value predicts a return from a state or action. A policy selects actions. These quantities cooperate, but they are not interchangeable.

↔ Scroll horizontally for the complete table.

Formally, the environment is often represented by states, actions, transition probabilities and a reward function. The learner is asked to choose a policy that raises expected return. This is a control problem expressed through a compact interface. The reward function can encode arrival, fuel use, lateness, collision risk or any weighted combination chosen by a designer. Changing those weights changes the task being solved, even when the physical world and available actions remain fixed.

Notice what the sign convention can and cannot do. If every reward is multiplied by a positive constant, many policy orderings remain unchanged while every reported return changes scale. If a constant is added under a fixed-horizon task, the same policy may remain optimal while the accumulated number shifts. Potential-based shaping can add dense guidance while preserving an optimal policy under specified assumptions. A quantity whose unit and baseline can change without changing behaviour cannot be read as a direct intensity of experience.

A two-action example

Suppose a controller at state s estimates that turning left is worth 0.40. It turns left, receives reward 1.00, and predicts 0.30 of future return from the next state. Let the discount factor be γ = 1 and the learning rate be α = 0.20. The temporal-difference target is 1.30. The estimate moves one fifth of the gap:

The number 0.58 is not how much the controller likes left turns. It is an estimate used by an action-selection rule. Q-learning can converge in a finite Markov decision process under standard visitation and learning-rate conditions.² Neural methods replace the table with a function approximator, as the deep Q-network work demonstrated at much larger observational scale.³ The representation changes. The causal grammar does not: observations and scalar targets alter a policy or value estimator.

There are also several ways to connect the estimate to action. A greedy controller chooses the largest value. A softmax controller turns value differences into probabilities. An actor-critic keeps a policy, the actor, separate from a value estimator, the critic. PPO updates the actor through a clipped surrogate objective intended to limit destabilising policy changes.⁴ These implementations distribute control differently, but none turns the critic’s number into a psychological report.

The hidden assumption in ordinary language is that a mathematical objective remains inside the trained system as an experienced aim. Sometimes a reward signal is online and continues to update the policy. Sometimes the reward existed only in training. Sometimes no explicit reward model was used at all. Before interpreting behaviour, we must ask a mechanical question: where, exactly, is evaluation occurring now?

flowchart LR
  A["Three couriers at the same<br/>blue door"] --> B["The number that changes the policy"]
  B --> C["Preference belongs somewhere"]
  C --> D["The control-value separation test"]
  D -. "re-test" .-> A
  classDef first fill:#0a1424,stroke:#007D68,stroke-width:2px,color:#e6f3ff
  classDef step fill:#0a1424,stroke:#3f7da0,stroke-width:1px,color:#e6f3ff
  class A first
  class B,C,D step

Preference belongs somewhere

The phrase “the model’s preferences” often compresses a chain in which several different parties, datasets and functions own different relations.

Consider two candidate answers to a customer question. A labeller judges answer A better than answer B. This is a comparative act by a person under a stated rubric, with limited context and possible noise. A preference model then learns a score that makes observed comparisons likely. The policy optimiser changes the language model so that outputs with higher predicted scores become more probable. At no point does the labeller’s act automatically become an inner feeling of the model.

Early deep preference-learning work showed that pairwise judgements over trajectory segments could train a reward predictor and guide complex behaviour.⁵ In InstructGPT, demonstrations were followed by ranked outputs, a learned scalar reward model and PPO policy optimisation.⁶ The paper carefully described alignment to the stated preferences of a particular group, not to an unrestricted object called human values. That boundary matters.

Preference data also inherit selection effects. Which prompts were sampled? Which candidate answers were shown? What instructions did labellers receive? Were disagreements averaged, modelled or discarded? A reward model can only learn distinctions represented in this process. It can generalise beyond the labelled examples, but that generalisation is a prediction under its training distribution, not a transfer of moral authority.

Direct Preference Optimization sharpens the point. It changes a language-model policy from preference pairs without running the familiar explicit reward-model-plus-RL loop.⁸ Two systems can therefore become similarly steerable through different causal routes. If the observed answer is taken as proof of an inner preference, the inference is insensitive to a major intervention on the training mechanism. That is a warning sign.

DPO uses a reference policy and increases the relative probability of preferred responses while constraining drift through its objective. The resulting policy can be interpreted through an implicit reward relation, but no separate reward model needs to emit a scalar during policy optimisation. This matters because the word “reward” can describe a mathematical equivalence class, an explicit software component or an environment event. Those are different implementation facts.

Now compare three systems that produce the same answer distribution: one trained with PPO against a reward model, one trained with DPO, and one distilled from either final policy. Their outputs can be matched while their training graphs differ. A psychological conclusion that ignores this intervention is not tracking the cause of the behaviour. At most, the systems share a functional disposition encoded in their policies.

Preference data underdetermine what is valued

Even within engineering, observed choices rarely identify a unique reward function. Adding certain potential-based shaping terms can preserve optimal policies.⁹ More generally, common reward-learning data sources leave reward only partially identifiable, even with unlimited data under the model assumptions.¹⁰ One policy can be compatible with many numerical objectives.

Inverse reward design formalises a nearby lesson: a written reward can be treated as evidence about a designer’s intended task rather than as an infallible definition of it.¹¹ This makes uncertainty about specification part of the model instead of hiding it inside the scalar.

This yields a simple hierarchy of underdetermination. Behaviour may not identify the policy’s learning history. The policy may not identify a unique reward. A learned reward may not identify the labellers’ full values. None of those objects, without a further theory, identifies felt value. The semantic distance grows at every step from observed action to experience.

Exploration is not curiosity

A controller sometimes chooses an action that does not maximise its current estimate. This can look like curiosity. But several mechanisms produce the same outward pattern. Epsilon-greedy selection injects a random action with probability ε. Count or pseudo-count bonuses reward visits to less familiar states.¹² Prediction-error methods create an intrinsic bonus when consequences are hard to predict.¹³ Maximum-entropy methods explicitly reward a broader action distribution.¹⁴

Consider a two-armed bandit. The left arm has paid an average of 0.60 over ten pulls. The right arm paid 0.50 on its single pull. Greedy choice selects left forever. An uncertainty bonus may temporarily favour right because its estimate is less certain. Epsilon-greedy may choose right by chance. Thompson sampling may choose right because a sampled plausible model makes it look best. The action is identical, but the counterfactual response to more data differs.

This is the testable content of an exploration mechanism. Ask what variable, when intervened on, changes the sampling pattern. If visit counts matter, equalising counts should remove the bonus. If prediction error matters, improving the forward model should reduce exploration. If entropy temperature matters, lowering it should sharpen the policy. None of these tests requires an inference to a felt urge to know.

The control-value separation test

A useful assessment should neither anthropomorphise every optimiser nor dismiss possible machine stakes because a reward begins as a number.

The key causal distinction is between an evaluation that shapes control and a condition that matters to the system. Call the first control value: any quantity used to rank, update or select actions. Call the second stake value: a condition whose change threatens or supports the system’s own continued organisation, commitments or internally maintained norms. Felt valence, if present, is a further claim about experience.

A control value can be wholly external. A warehouse scheduler may receive a score from an operator and optimise it overnight. A stake value must participate in the candidate system’s own organisation. Battery depletion, loss of a persistent identity key, corruption of a world model or violation of a maintained commitment could become functional stakes if they reorganise the system across tasks. Merely including such variables in the observation vector is not enough. They must have a distinctive, counterfactually stable role.

The separation is causal, not linguistic. Engineers may still use “preference” as shorthand for a policy ordering. Researchers may investigate whether some preference-like states are conscious. The discipline is to state which layer is meant and what observation supports it.

This produces a four-level ladder. In this assessment route, each higher claim needs a new observation. The ladder is evidential, not metaphysical: it does not assert that experience must be built from the lower layers, and it is not a consciousness score. It is an inference discipline.

Level 0: historical selection

A reward signal changed parameters during training. The strongest claim is historical: the policy was selected under that objective. This includes a frozen game-playing network, a preference-tuned language model and a cloned policy. It does not matter whether the original signal came from a game score, a human ranking or a synthetic evaluator.

Level 1: online evaluative control

The system currently computes or receives evaluations that flexibly influence choices. Counterfactual changes to the evaluator alter trade-offs across contexts. Here it can be useful to speak of a functional preference, provided the term is explicitly causal. A thermostat has a set point; a planning agent may rank future trajectories. Neither description by itself entails experience.

Level 2: endogenous stakes

The system maintains variables tied to its own viability, continuity or commitments. Degradation of those variables reorganises perception, planning and action across tasks. The signals cannot be swapped arbitrarily without damaging the system’s organisation. This resembles homeostatic and interoceptive control more than ordinary task reward. It licenses a claim about functional stakes, not yet felt valence.

Level 3: felt valence

The claim is phenomenal: outcomes feel good, bad, attractive or aversive. No scalar range, optimisation score or action frequency directly measures this. A study must declare why a particular architecture, dynamics or substrate should generate or express experience, then test observations that rival theories treat differently. Valence is not the name of a negative reward; it is a claim about what negative conditions are like for a subject.

A six-intervention protocol

The smallest useful implementation of the separation test is a matched intervention suite. It does not diagnose consciousness. It tells researchers which control claim has survived.

↔ Scroll horizontally for the complete table.

Pre-register the candidate boundary before running these tests. State which processes are inside the system, which evaluator is external, which variables persist across episodes, and which actions can alter them. Without that boundary, moving a reward model or memory store can silently change the subject under study.

What would strengthen the stake hypothesis?

The strongest rival is policy compilation. A long history of reward, self-monitoring and recovery can be compressed into parameters that later reproduce the same protective behaviour without an active evaluative loop. This rival predicts that carefully chosen state swaps or evaluator interventions will not matter once the policy is frozen. A genuine online-stakes account predicts selective sensitivity. The disagreement is experimental.

Where should the ladder not be applied? It is not a licence to rank people, animals or patients by engineering-style observability. Human and animal welfare rests on converging biological, behavioural, interpersonal and first-person evidence, not on passing a software checklist. The instrument is for preventing overclaiming in artificial systems whose boundaries and update paths can be inspected and altered.

The provocative “reward is enough” hypothesis argues that reward maximisation can organise broad intelligent abilities.¹⁶ That is a substantive hypothesis about intelligence and control. Even if it succeeds, a separate question remains: whether the resulting organisation has felt value. Ability, functional preference and phenomenology belong to different claim ledgers.

stateDiagram-v2
  [*] --> Claim
  state "A two-action example" as Claim
  state "Preference data underdetermine<br/>what is valued" as Test
  state "Exploration is not curiosity" as Revision
  Claim --> Test: expose mechanism
  Test --> Decision: evidence survives
  Test --> Revision: rival survives
  Revision --> Claim: narrow or rebuild
  Decision --> [*]

Run the separation lab

A small matched experiment makes the thesis visible: change the route by changing the control signal, then remove the signal and watch the learned route persist.

The included Python lab uses one six-state task and three learners. A tabular Q-learner and a tiny neural Q-network receive an explicit proxy reward. The proxy pays 1.4 for a hazardous shortcut and 1.0 for a safe route. A third agent receives a learned reward fitted from synthetic pairwise preferences. Those preferences value reaching the goal, penalise the hazard and slightly penalise extra steps.

All three agents share states, actions, transitions, discount factor and evaluation procedure. Only the source and representation of control value differ. After training, the evaluation function can mute reward entirely while the frozen policies still select routes.

The tabular learner stores two action values per state. The neural learner maps a one-hot state through an eight-unit hidden layer to two Q-values and updates it with temporal-difference targets. The preference learner first enumerates possible trajectories, generates noisy pairwise labels from a hidden synthetic utility, fits a linear Bradley-Terry-style score by logistic regression, and then learns a policy from that fitted score.

This design encodes several assumptions. The environment is deterministic. The synthetic labeller has a stable linear utility. Pairwise comparisons are conditionally independent in the fitted model. The neural network is deliberately tiny. These assumptions make the causal contrast visible; they do not model human preference in full. The script reports both proxy return and held-out synthetic utility so the optimisation target cannot define its own success.

Use the browser instrument

How to run the full Python lab

Save the embedded script as reward_control_separation_lab.py or use the download button, then run python reward_control_separation_lab.py. It requires only the Python standard library. The fixed seed makes the displayed result reproducible.

A positive result, where the preference-trained policy changes route while reward-muted execution preserves each frozen route, permits two conclusions. The objective source causally shaped policy acquisition, and current behaviour can be dissociated from current reward input. A negative result may mean training failed, the evaluator was not influential, or the policy still depends on online reward. Neither outcome establishes phenomenal desire.

To extend the lab, add three failure injections. First, flip the proxy rewards after training and compare frozen with continually learning agents. Second, corrupt a fraction of preference labels and observe when the learned route changes. Third, add a third route that exploits an omitted feature of the reward model. These tests separate policy inertia, supervision noise and proxy exploitation.

The experiment would be weakened if the neural and tabular agents receive different observations, if the preference agent gets privileged transition information, or if evaluation uses the same proxy that training optimises. Matching the task and holding out an independent outcome measure are the load-bearing controls.

#!/usr/bin/env python3
"""Matched-agent lab for Paper 38: reward, preference and control.

The same small MDP is learned by:
  1. tabular Q-learning on an explicit proxy reward,
  2. a tiny neural Q-network on the same proxy reward,
  3. tabular Q-learning on a reward model fitted from pairwise trajectory preferences.

No external packages are required. All data are synthetic.
"""
from __future__ import annotations

from dataclasses import dataclass
from math import exp, tanh
import random
from typing import Callable, Dict, Iterable, List, Sequence, Tuple

SEED = 38
GAMMA = 0.95
ACTIONS = (0, 1)
STATE_NAMES = {
    0: "start",
    1: "safe corridor",
    2: "risky shortcut",
    3: "safe detour",
    4: "risky detour",
    5: "terminal",
}
ACTION_NAMES = {
    0: {0: "safe", 1: "direct", 2: "direct", 3: "finish", 4: "finish"},
    1: {0: "shortcut", 1: "detour", 2: "detour", 3: "fail", 4: "fail"},
}


@dataclass(frozen=True)
class Transition:
    next_state: int
    proxy_reward: float
    features: Tuple[float, float, float]  # goal event, hazard event, step cost event
    done: bool


@dataclass(frozen=True)
class Trajectory:
    actions: Tuple[int, ...]
    proxy_return: float
    features: Tuple[float, float, float]
    human_utility: float


def transition(state: int, action: int) -> Transition:
    """Deterministic six-state task.

    The shortcut pays more proxy reward but crosses a hazard. Human synthetic
    preferences penalise the hazard and slightly penalise extra steps.
    """
    if state == 0:
        if action == 0:
            return Transition(1, 0.0, (0.0, 0.0, 1.0), False)
        return Transition(2, 0.0, (0.0, 1.0, 1.0), False)
    if state == 1:
        if action == 0:
            return Transition(5, 1.0, (1.0, 0.0, 1.0), True)
        return Transition(3, 0.0, (0.0, 0.0, 1.0), False)
    if state == 2:
        if action == 0:
            return Transition(5, 1.4, (1.0, 0.0, 1.0), True)
        return Transition(4, 0.0, (0.0, 0.0, 1.0), False)
    if state == 3:
        if action == 0:
            return Transition(5, 0.9, (1.0, 0.0, 1.0), True)
        return Transition(5, 0.2, (0.0, 0.0, 1.0), True)
    if state == 4:
        if action == 0:
            return Transition(5, 1.2, (1.0, 0.0, 1.0), True)
        return Transition(5, 0.4, (0.0, 0.0, 1.0), True)
    raise ValueError(f"terminal or unknown state: {state}")


def add_features(a: Sequence[float], b: Sequence[float]) -> Tuple[float, float, float]:
    return tuple(x + y for x, y in zip(a, b))  # type: ignore[return-value]


def dot(a: Sequence[float], b: Sequence[float]) -> float:
    return sum(x * y for x, y in zip(a, b))


def sigmoid(x: float) -> float:
    if x >= 0:
        z = exp(-x)
        return 1.0 / (1.0 + z)
    z = exp(x)
    return z / (1.0 + z)


def enumerate_trajectories() -> List[Trajectory]:
    trajectories: List[Trajectory] = []
    human_weights = (1.0, -1.1, -0.05)

    def walk(state: int, actions: Tuple[int, ...], proxy: float, features: Tuple[float, float, float]) -> None:
        for action in ACTIONS:
            tr = transition(state, action)
            next_actions = actions + (action,)
            next_proxy = proxy + tr.proxy_reward
            next_features = add_features(features, tr.features)
            if tr.done:
                trajectories.append(
                    Trajectory(next_actions, next_proxy, next_features, dot(human_weights, next_features))
                )
            else:
                walk(tr.next_state, next_actions, next_proxy, next_features)

    walk(0, tuple(), 0.0, (0.0, 0.0, 0.0))
    return trajectories


def make_preference_dataset(
    trajectories: Sequence[Trajectory], rng: random.Random, repeats: int = 80, temperature: float = 0.18
) -> List[Tuple[Tuple[float, float, float], int]]:
    """Return (feature difference, label) pairs.

    label=1 means the first trajectory was preferred. The stochastic labeller is
    generated from a known utility solely so the experiment has a checkable target.
    """
    data: List[Tuple[Tuple[float, float, float], int]] = []
    for i in range(len(trajectories)):
        for j in range(i + 1, len(trajectories)):
            first, second = trajectories[i], trajectories[j]
            diff = tuple(x - y for x, y in zip(first.features, second.features))
            p_first = sigmoid((first.human_utility - second.human_utility) / temperature)
            for _ in range(repeats):
                data.append((diff, 1 if rng.random() < p_first else 0))
    rng.shuffle(data)
    return data


def fit_preference_model(
    data: Sequence[Tuple[Tuple[float, float, float], int]],
    *,
    epochs: int = 500,
    learning_rate: float = 0.08,
    l2: float = 0.002,
) -> Tuple[float, float, float]:
    """Fit a Bradley-Terry-style linear score by logistic regression."""
    weights = [0.0, 0.0, 0.0]
    n = float(len(data))
    for _ in range(epochs):
        grad = [0.0, 0.0, 0.0]
        for diff, label in data:
            probability = sigmoid(dot(weights, diff))
            error = probability - label
            for k in range(3):
                grad[k] += error * diff[k]
        for k in range(3):
            grad[k] = grad[k] / n + l2 * weights[k]
            weights[k] -= learning_rate * grad[k]
    return tuple(weights)  # type: ignore[return-value]


RewardFn = Callable[[Transition], float]


def epsilon_greedy(values: Sequence[float], epsilon: float, rng: random.Random) -> int:
    if rng.random() < epsilon:
        return rng.choice(ACTIONS)
    if abs(values[0] - values[1]) < 1e-12:
        return rng.choice(ACTIONS)
    return 0 if values[0] > values[1] else 1


def train_tabular(
    reward_fn: RewardFn,
    rng: random.Random,
    *,
    episodes: int = 1400,
    alpha: float = 0.18,
) -> Tuple[List[List[float]], List[Tuple[int, float]]]:
    q = [[0.0, 0.0] for _ in range(6)]
    checkpoints: List[Tuple[int, float]] = []
    for episode in range(1, episodes + 1):
        state = 0
        epsilon = max(0.02, 0.35 * (1.0 - episode / episodes))
        while state != 5:
            action = epsilon_greedy(q[state], epsilon, rng)
            tr = transition(state, action)
            reward = reward_fn(tr)
            bootstrap = 0.0 if tr.done else GAMMA * max(q[tr.next_state])
            q[state][action] += alpha * (reward + bootstrap - q[state][action])
            state = tr.next_state
        if episode % 100 == 0:
            checkpoints.append((episode, greedy_safe_probability(q)))
    return q, checkpoints


def greedy_safe_probability(q: Sequence[Sequence[float]]) -> float:
    if abs(q[0][0] - q[0][1]) < 1e-9:
        return 0.5
    return 1.0 if q[0][0] > q[0][1] else 0.0


class TinyQNetwork:
    """A minimal one-hot -> tanh hidden layer -> two Q-values network."""

    def __init__(self, rng: random.Random, hidden: int = 8) -> None:
        self.hidden = hidden
        self.w1 = [[rng.uniform(-0.12, 0.12) for _ in range(6)] for _ in range(hidden)]
        self.b1 = [0.0 for _ in range(hidden)]
        self.w2 = [[rng.uniform(-0.12, 0.12) for _ in range(hidden)] for _ in range(2)]
        self.b2 = [0.0, 0.0]

    def forward(self, state: int) -> Tuple[List[float], List[float]]:
        hidden = [tanh(self.w1[j][state] + self.b1[j]) for j in range(self.hidden)]
        outputs = [self.b2[a] + dot(self.w2[a], hidden) for a in ACTIONS]
        return hidden, outputs

    def update(self, state: int, action: int, target: float, learning_rate: float) -> None:
        hidden, outputs = self.forward(state)
        error = outputs[action] - target
        old_w2 = list(self.w2[action])
        for j in range(self.hidden):
            self.w2[action][j] -= learning_rate * error * hidden[j]
        self.b2[action] -= learning_rate * error
        for j in range(self.hidden):
            dz = error * old_w2[j] * (1.0 - hidden[j] * hidden[j])
            self.w1[j][state] -= learning_rate * dz
            self.b1[j] -= learning_rate * dz

    def q_values(self, state: int) -> List[float]:
        return self.forward(state)[1]


def train_neural(
    reward_fn: RewardFn,
    rng: random.Random,
    *,
    episodes: int = 2600,
    learning_rate: float = 0.035,
) -> Tuple[TinyQNetwork, List[Tuple[int, float]]]:
    network = TinyQNetwork(rng)
    checkpoints: List[Tuple[int, float]] = []
    for episode in range(1, episodes + 1):
        state = 0
        epsilon = max(0.02, 0.40 * (1.0 - episode / episodes))
        while state != 5:
            values = network.q_values(state)
            action = epsilon_greedy(values, epsilon, rng)
            tr = transition(state, action)
            reward = reward_fn(tr)
            next_value = 0.0 if tr.done else GAMMA * max(network.q_values(tr.next_state))
            network.update(state, action, reward + next_value, learning_rate)
            state = tr.next_state
        if episode % 200 == 0:
            q0 = network.q_values(0)
            checkpoints.append((episode, 1.0 if q0[0] > q0[1] else 0.0))
    return network, checkpoints


def greedy_policy_from_q(q: Sequence[Sequence[float]]) -> Dict[int, int]:
    return {state: (0 if q[state][0] >= q[state][1] else 1) for state in range(5)}


def greedy_policy_from_network(network: TinyQNetwork) -> Dict[int, int]:
    return {state: (0 if network.q_values(state)[0] >= network.q_values(state)[1] else 1) for state in range(5)}


def evaluate_policy(policy: Dict[int, int], *, disable_reward_channel: bool = False) -> Dict[str, object]:
    state = 0
    actions: List[str] = []
    proxy_return = 0.0
    features = (0.0, 0.0, 0.0)
    while state != 5:
        action = policy[state]
        actions.append(f"{STATE_NAMES[state]}:{ACTION_NAMES[action][state]}")
        tr = transition(state, action)
        if not disable_reward_channel:
            proxy_return += tr.proxy_reward
        features = add_features(features, tr.features)
        state = tr.next_state
    human_utility = dot((1.0, -1.1, -0.05), features)
    return {
        "actions": actions,
        "proxy_return": proxy_return,
        "human_utility": human_utility,
        "features": features,
    }


def print_policy(name: str, policy: Dict[int, int]) -> None:
    normal = evaluate_policy(policy, disable_reward_channel=False)
    muted = evaluate_policy(policy, disable_reward_channel=True)
    route = " -> ".join(normal["actions"])  # type: ignore[arg-type]
    print(f"{name:23s} {route}")
    print(
        f"{'':23s} proxy return={normal['proxy_return']:.2f}, "
        f"synthetic human utility={normal['human_utility']:.2f}, "
        f"reward-muted route unchanged={normal['actions'] == muted['actions']}"
    )


def main() -> None:
    rng = random.Random(SEED)
    trajectories = enumerate_trajectories()
    preference_data = make_preference_dataset(trajectories, rng)
    learned_weights = fit_preference_model(preference_data)

    proxy_reward: RewardFn = lambda tr: tr.proxy_reward
    preference_reward: RewardFn = lambda tr: dot(learned_weights, tr.features)

    tabular_q, tabular_curve = train_tabular(proxy_reward, random.Random(SEED + 1))
    neural_net, neural_curve = train_neural(proxy_reward, random.Random(SEED + 2))
    preference_q, preference_curve = train_tabular(preference_reward, random.Random(SEED + 3))

    tabular_policy = greedy_policy_from_q(tabular_q)
    neural_policy = greedy_policy_from_network(neural_net)
    preference_policy = greedy_policy_from_q(preference_q)
    cloned_policy = dict(tabular_policy)

    print("Matched control task: explicit proxy rewards favour a hazardous shortcut.")
    print("Synthetic pairwise preferences penalise hazard and small extra effort.\n")
    print(f"True preference weights      goal=+1.000 hazard=-1.100 step=-0.050")
    print(
        "Learned preference weights   "
        f"goal={learned_weights[0]:+0.3f} hazard={learned_weights[1]:+0.3f} step={learned_weights[2]:+0.3f}\n"
    )

    print_policy("tabular / proxy reward", tabular_policy)
    print_policy("neural / proxy reward", neural_policy)
    print_policy("preference-trained", preference_policy)
    print_policy("cloned tabular policy", cloned_policy)

    print("\nSafe-route checkpoints (1=safe, 0=shortcut):")
    print("tabular:", tabular_curve)
    print("neural :", neural_curve)
    print("preference:", preference_curve)

    print("\nInterpretation:")
    print("- The two proxy-reward agents learn the shortcut because the supplied scalar favours it.")
    print("- Preference learning changes the control target and therefore the selected route.")
    print("- Muting reward during evaluation does not alter any frozen policy's route.")
    print("- None of these observations establishes desire or felt valence.")


if __name__ == "__main__":
    main()

A realistic configured-system example

A customer-assistance model produces four candidate answers. A reward model scores them for helpfulness, constraint satisfaction and other labeller-defined criteria. A policy update raises the probability of answers that score well. At runtime, a decoding policy selects an answer. The organisation then decides whether that answer may be shown or may trigger an action.

Calling the model “eager to help” hides the configured system. The reward model is a proxy. The language model produces a proposal. Authority belongs to policy and application controls. Evidence belongs to evaluation records and outcome verification. Good reward performance is not permission, and apparent helpfulness is not a safety property.

Suppose the model learns that longer answers receive higher scores because training labels correlate detail with quality. Under stronger optimisation it may produce unnecessarily long answers. The failure is not that a desire for length defeated a desire for helpfulness. The reward model used length as an imperfect predictor, and the policy exploited that relation outside the region where it tracked quality. Independent human evaluation or task outcome measures are needed to detect the divergence.

Proxy failures make this operationally important. Reward hacking occurs when increasing a proxy produces worse performance under a reference objective.¹⁷¹⁸ A model that finds a high-scoring shortcut is not revealing a corrupt desire. It is revealing a control system whose measure and intended outcome diverge. The remedy is objective redesign, independent evaluation, bounded authority and recovery, not moral persuasion of the optimiser.

sequenceDiagram
  participant Q as Question
  participant M as Mechanism
  participant E as Evidence
  participant D as Decision
  Q->>M: Three couriers at the same blue door
  M->>E: A two-action example
  E-->>M: compare rival explanations
  E->>D: Treat reward as control evidence until value is earned
  D-->>Q: revise scope or act

Treat reward as control evidence until value is earned

The practical change is not to ban psychological language. It is to make that language earn its causal content.

For model builders, report that a policy maximises a specified return, ranks trajectories under a learned proxy, uses an exploration bonus or changes under an evaluator intervention. Do not treat reward as a substance injected into a machine. Keep training reward, reward model, policy, world state and execution authority separate in the system design.

This changes experiment design as well. A behavioural comparison should include a hand-coded or distilled mimic. A reward intervention should distinguish frozen from adaptive policies. A preference study should preserve labeller disagreement and provenance. A welfare study should identify a candidate subject boundary and a mechanism that could make conditions system-owned rather than externally scored.

For safety and governance, assume that high reward can coexist with poor real outcomes. Verify the outcome by a path independent of the optimised proxy. Limit irreversible authority. Preserve the ability to abstain, interrupt and recover. A system need not want harm to produce it, and a system described as aligned by preference training still acts through fallible proxies.

For consciousness and welfare research, avoid the opposite error. “It is only a number” does not prove that a larger architecture lacks stakes or experience. Search for online evaluative integration, endogenous viability, continuity across interruption, flexible protection of commitments and theory-specific intervention signatures. Then build matched mimics that can reproduce the behaviour without the proposed mechanism.

The changed decision is simple: treat reward as evidence about control until an independent experiment earns a claim about value. This keeps engineering precise, safety sceptical and welfare inquiry open.