A label is a measurement event

Suppose a prompt is x, two candidate responses are y_A and y_B, and an annotator chooses A. A conventional dataset reduces the event to (x, chosen=A, rejected=B). The useful original object is richer:

Here z is judgement context, such as role, expertise, accessibility need or declared objective. C is the candidate set from which A and B were drawn. I is the interface and instruction shown to the annotator. t is the decision time, which matters when policies or social expectations change. The observed bit is a sample from this conditional process. It is not a context-free quantity found inside the response.

Forced choice changes the thing being measured

A two-button interface conflates several states. The annotator may have a stable preference for A, weakly lean towards A, regard the answers as equivalent, believe both are unacceptable, lack the expertise to decide or notice that each answer is correct under a different interpretation. Converting all six states into A or B increases label volume while reducing construct validity.

The remedy is not an unlimited questionnaire. It is a small set of discriminating affordances. Allow tie and both-poor where they matter. Ask confidence on a sampled subset. Provide a reason code tied to the intended construct, such as factuality, relevance, clarity, tone or safety. Randomise order and repeat a fraction of pairs. These additions estimate whether disagreement is stable, feature-linked and actionable before the organisation pays to amplify it.

Rater competence is part of the same measurement design. A preference from a domain expert may be needed for correctness, while a target user may be the right judge of comprehensibility. Combining them into one majority does not create a more representative evaluator. It combines distinct questions. A defensible pipeline either separates the constructs or declares how one should constrain the other.

Bradley-Terry turns comparisons into a scalar

The common Bradley-Terry model assigns each response a scalar reward and maps the reward difference through a logistic function:

The symbol r is the learned score; only score differences matter. When the difference is zero, the model predicts a 50:50 choice. A larger positive difference pushes the probability towards A. This is a useful compression when comparisons are generated by a stable, approximately transitive target. It becomes a misleading compression when the same response has different utilities under hidden contexts.

For the opening case, service reviewers may have a positive reward difference for A while dispute specialists have an equally strong negative difference. Erasing z asks one curve to stand in for both. The fitted answer is governed by the sampled mixture, not by a newly discovered universal preference.

Thought experiment: the annotator-mix switch

Hold the prompt, candidates, model and instructions fixed. Let 70 per cent of labels come from group S, which prefers A with probability 0.9, and 30 per cent from group D, which prefers A with probability 0.1. The marginal probability is 0.66. Reverse only the sampling mix and it becomes 0.34. A majority label flips from A to B although neither group changes its judgement.

The causal variable is the collection process. If the deployed population follows the first mixture forever, optimising the marginal may be a defensible utility decision. If the mixture is a staffing accident, or if the service must work for both groups, the same optimisation encodes an undeclared policy. Calling the residual 34 per cent “noise” hides that decision.

Some preferences cannot fit on one line

A scalar score also assumes that pairwise relations can be represented by one ordering. Consider three candidates. A is concise and exact. B is warmer and more explanatory. C is exhaustive and cautious. Suppose the collected judgements say A beats B, B beats C and C beats A. Such cycles can arise from different raters, changing criteria or genuine context dependence.

No assignment of three real numbers can satisfy all three inequalities. The model must compromise, and the compromise depends on which pairs were sampled and how often. Distributional Preference Learning formalises a related hidden-context problem: standard Bradley-Terry learning can implicitly rank alternatives by a Borda-like average win probability over available alternatives. That is a specific aggregation rule, not a neutral recovery of “human values”.

The correct response is not to preserve every annotator identifier forever. Privacy, representativeness and misuse matter. The response is to diagnose variation before collapsing it. Repeat selected comparisons, allow ties and “both poor”, record reason codes, test order effects, stratify by legitimate task context and retain enough provenance to audit whose judgements dominate. Only then can the team distinguish random error from structured disagreement.

flowchart TD
  A["Two correct answers, one<br/>misleading winner"] --> B["A label is a measurement event"]
  B --> C["What RLHF and DPO actually change"]
  C --> D["Preserving disagreement in the<br/>target"]
  D -. "re-test" .-> A
  classDef first fill:#0a1424,stroke:#D95545,stroke-width:2px,color:#e6f3ff
  classDef step fill:#0a1424,stroke:#3f7da0,stroke-width:1px,color:#e6f3ff
  class A first
  class B,C,D step

What RLHF and DPO actually change

Preference-based reinforcement learning predates current language assistants. Christiano and colleagues showed that non-expert pairwise judgements over trajectory segments could train policies for simulated control and Atari tasks. Later language-model work used comparisons to improve summarisation and instruction following. These results establish that relative feedback can be operationally useful. They do not establish that every collection of preferences identifies a single latent human objective.

The standard RLHF branch

A common language-model pipeline starts with supervised fine-tuning. For each prompt, it samples candidate completions and asks annotators to rank or compare them. A reward model r_φ(x,y) is trained with a pairwise loss, often Bradley-Terry. A policy π_θ is then optimised to increase expected learned reward while remaining near a reference policy:

The expectation averages prompts and policy-generated responses. The KL term measures deviation from the reference distribution. The coefficient β prices that deviation: increasing it usually keeps the policy closer to the reference; reducing it permits stronger movement towards the learned reward. In many implementations, proximal policy optimisation is used for this reinforcement-learning stage.

The KL constraint is not merely aesthetic. The reward model has evidence near the candidates on which it was trained. Aggressive search can find completions that exploit its errors. Experiments on reward-model overoptimisation show the proxy score can continue rising after a stronger reference measure begins to fall. This is a measurement failure amplified by optimisation pressure.

The DPO branch

Direct Preference Optimization begins from the same KL-regularised objective and Bradley-Terry preference model. Its key move is to express the reward associated with an optimal policy as a log probability ratio against the reference policy:

The partition term Z(x) depends on the prompt but not on which completion is compared. It therefore cancels in a reward difference. Substituting the ratio into the Bradley-Terry likelihood gives the DPO loss for preferred completion y_w and rejected completion y_l:

The loss increases the preferred completion’s log-probability advantage over the rejected completion, measured relative to the reference. Standard DPO therefore avoids training a separately exposed reward network and avoids an online reinforcement-learning loop during fine-tuning. It converts the problem into supervised binary classification over preference pairs.

What is gained and lost by hiding the reward model

An explicit reward model is an additional component to train, validate, secure and monitor. It can drift away from the policy distribution, and policy optimisation can search for its blind spots. Yet it is also an inspectable instrument. Teams can score controlled candidate sets, compare reward margins across slices, use uncertainty to select new annotations and test whether the reward follows a known nuisance feature. A separate model therefore creates both risk and diagnostic leverage.

DPO removes that deployed interface in its standard form. The training loop is simpler and often more stable, but the implied reward is entangled with the policy and reference. A falling DPO loss shows that preferred pairs are becoming more likely in the required relative sense. It does not reveal whether the policy has learned factuality, length, annotator dialect or another correlated feature. Fresh generation-based evaluation remains necessary.

Algorithm choice should therefore follow the operating need. DPO is attractive for a fixed offline pair set when direct policy adaptation and implementation economy matter. An explicit reward pipeline can be preferable when active data collection, candidate reranking, reward auditing or a separately governed evaluator is required. Hybrid arrangements are possible. There is no optimiser-only answer to a target-design problem.

Thought experiment: change only the reference policy

Present the same preference pair to two DPO runs. In run R1, the reference already assigns A four times the probability of B. In run R2, it assigns equal probability. Suppose the fitted Bradley-Terry probability for A over B is 0.75, whose log-odds is about 1.10. Under the DPO model, the required policy odds are the reference odds multiplied by exp(1.10/β).

With β=1, the factor is about three. R1 therefore targets odds around 12:1 while R2 targets about 3:1. With β=0.5, the factor is about nine. This is an algebraic illustration, not a guarantee about a finite neural training run. It shows that the same label describes a change relative to a starting distribution. Reference choice and β are part of the behavioural contract.

Candidate generation defines what can be preferred

Annotators compare what the current policy or a proposal model produces. If every candidate is verbose, the data cannot identify a concise optimum. If both candidates share the same factual error, a forced choice can label one “chosen” without making it correct. If unsafe and safe answers are mixed with style differences, the model may learn the easiest correlated feature rather than the intended reason.

Length is a documented example. Work on DPO has shown that the objective can exploit verbosity and that length-aware regularisation can improve controlled win rates. Sycophancy provides another warning: human preference judgements can favour answers that agree with a user’s stated belief over truthful disagreement. These are not arguments against human feedback. They are reasons to vary candidate construction, collect reason-specific judgements and keep truth tests independent from taste.

Negative control: when a scalar target is enough

Now compare an answer that correctly computes 17 per cent of £240 as £40.80 with an otherwise similar answer that says £42.80. Qualified reviewers agree, repeated labels are stable, order has no effect and the candidate pair isolates arithmetic correctness. A scalar reward that ranks the correct answer higher is economical. Conditioning on reviewer identity adds parameters without changing the decision.

This negative control matters. The claim is not that every preference is plural or that averages are inherently illegitimate. Preserve disagreement when it carries decision-relevant structure; collapse it when experiments show that it behaves like exchangeable noise around a shared target.

Preserving disagreement in the target

Four targets hidden inside “human preference”

Before interpreting the evidence, separate four targets that often share one dataset column. The first is shared epistemic quality: correctness, valid reasoning or faithful evidence. The second is user-conditional utility: detail, tone, format or assistance appropriate to a declared need. The third is collective choice: a policy for several people whose interests may conflict. The fourth is institutional obligation: a rule that applies even when a majority would prefer otherwise.

These targets call for different mechanisms. Epistemic quality benefits from expert labels, executable tests and evidence checks. User-conditional utility needs legitimate context or direct choice. Collective choice requires an explicit aggregation function and representation analysis. Institutional obligations belong in policy, data and action controls, with preference used only inside the permitted region.

Recent evidence makes hidden disagreement harder to dismiss as an edge case. A study releasing individual annotations for MultiPref and HelpSteer2 found diverging preferences in more than 30 per cent of examples across the two datasets. Its manual taxonomy attributed over 75 per cent of disagreements to factors such as response complexity, verbosity and underspecified prompts rather than simple annotation error. The exact rates belong to those datasets and collection procedures, but the mechanism matches the opening case.

The same study compared single-value and distributional reward models. On MultiPref, aggregated Bradley-Terry achieved 0.663 preference accuracy and 0.458 AUROC for identifying divergent pairs. A mean-variance model achieved nearly identical preference accuracy, 0.664, while raising divergence-identification AUROC to 0.615. The result does not prove one universal architecture. It demonstrates that average winner accuracy can remain flat while a separate capability to recognise disagreement improves materially.

There are several defensible responses to structured disagreement. Keep all individual labels and condition the reward or policy on a legitimate user need. Predict a reward distribution rather than a point. Learn reusable reward features and adapt their weights from explicit user feedback. Ask the model to produce a consensus candidate, then apply a declared social-welfare function. Or recognise that one policy should not settle the matter and ask the user a clarifying question.

Each approach answers a different deployment question. Personalisation asks, “what does this person prefer?” Distributional modelling asks, “how uncertain or divided is the population?” Consensus asks, “which statement can the group accept under this aggregation rule?” A maximin policy asks, “which option gives the least-served group the best floor?” None is automatically correct.

Serious objection: preserving variation can fragment the system

A pooled model has real advantages. It uses more data per parameter, is easier to evaluate, avoids sparse identities and reduces the attack surface created by personal profiles. Conditioning on small groups can overfit incidental differences, reproduce stereotypes or let malicious raters steer a branch. A distribution can report uncertainty without explaining its cause. The pluralistic remedy can therefore be worse than the scalar baseline.

The answer is comparative evidence, not a presumption in favour of complexity. Start with the pooled model. Add context only when held-out interventions show stable preference reversal and the field is legitimate for use. Use coarse task variables before identity. Set minimum support, regularise towards the shared model and test unseen users. Where segmentation improves average fit but lowers the worst-group floor or increases prohibited correlations, reject it.

This objection defines the useful boundary. Preserving disagreement means retaining enough information to choose responsibly. It does not mean that every individual receives a permanent reward model, or that every preference deserves satisfaction.

Production-shaped example: separate obligations from taste

Return to the overdraft explanation. A defensible design does not ask preferences to determine every property. It first applies deterministic or independently evaluated gates for account facts, arithmetic, required disclosure, prohibited claims and available dispute routes. These are validity conditions. A fluent but false answer cannot win by being shorter.

After those gates, the system may optimise conditional presentation: concise mobile view, detailed dispute view, accessible language, optional calculation expansion and tone. The conditioning signal should come from explicit channel, task or user choice, not an inferred protected characteristic. If the need is unknown, the system can offer a short answer with a visible expansion or ask one clarifying question.

This architecture also clarifies evaluation. Report factual pass rate and policy compliance separately from human preference. For style, report each declared segment’s win rate, calibration and abstention. Track the worst served segment. Re-run comparisons after changing the candidate generator, because a model that now produces much longer answers has moved the measurement environment.

Operating preference learning as metrology

A mature preference programme begins with the decision the model will influence, not with the availability of a DPO trainer. The following instrument makes assumptions inspectable before training.

Using the card on the opening case

For the overdraft explanation, protected validity contains the charge amount, triggering transaction, balance arithmetic, required disclosure and challenge route. The decision subject is the customer completing either a quick service check or a dispute review. Legitimate context is the explicit task or selected view, not inferred wealth, age or vulnerability. The comparison object is presentation quality after validity gates pass.

Candidate coverage must include concise, expandable and detailed answers with matched facts. Disagreement is expected over depth, so individual labels and task reason codes are retained. The aggregation rule is conditional: quick-service judgements train the concise view and dispute judgements train the detailed view. If task is unknown, the default is concise plus expansion rather than a guessed identity. Release requires zero critical factual-gate failures, acceptable preference performance in both tasks and a declared worst-task floor.

Writing the card exposes a fact that the optimiser cannot infer: “70 per cent chose concise” is not the governing objective. The objective is to provide a valid explanation whose presentation matches an explicit task while preserving an accessible route to the fuller account.

A five-stage test protocol

1. Instrument the judgement. Run repeated labels on a stratified subset. Randomise left-right order, permit ties and both-poor, collect confidence and a compact reason code.
2. Intervene on context. Hold candidate text fixed while changing rubric, rater expertise or explicit user need. Estimate whether preference shifts systematically.
3. Intervene on candidates. Generate matched pairs that vary one feature, such as length, factuality or tone. Check whether the label follows the intended feature or a proxy.
4. Compare target models. Evaluate a pooled scalar baseline against conditional, distributional or multi-objective alternatives. Include the simple model as a serious control.
5. Release by outcome. Test the trained policy against fresh candidates and fresh raters. Report invariant-gate failures, group slices, abstention and sensitivity to rater mix, not only aggregate wins.

What to monitor after release

Preference data drifts even when the model does not. The user population changes, annotation vendors change, instructions are rewritten and the policy starts producing candidates unlike those used in training. Monitor rater composition, tie rate, reason-code mix, response length, reference-policy divergence and outcome slices. Preserve dataset and model lineage so a behavioural change can be traced to collection, aggregation or optimisation.

Use disagreement itself as a sentinel. A sudden fall may indicate healthy convergence, but it can also mean the candidate generator has collapsed to near-duplicates or dissenting annotators have been filtered out. A sudden rise may signal ambiguous prompts, a new user population or degraded candidates. Inspect examples before interpreting the number.

The boundary of personalisation

Conditioning is defensible only when the context is relevant, available at decision time and authorised for that use. Inferring a person’s politics, health status or vulnerability merely to maximise approval can create manipulation and discrimination. Even benign user preferences may conflict with truth, safety or institutional duty. Sycophancy is the simplest warning: approval is not always the outcome the system should optimise.

If legitimate context cannot be observed, a conditional model cannot choose the correct branch reliably. Distributional uncertainty can reveal that the answer is contested, but it does not choose a social objective. The system must then use an explicit default, ask the user, offer alternatives, apply a hard policy or escalate. Uncertainty is evidence for a decision rule; it is not the decision rule.

Technical appendix

"""Preference, reward-model and DPO laboratory for Paper 69.

Python 3.10+. No third-party packages required.

The laboratory compares two synthetic cases:
1. shared target: two rater groups rank the same three responses;
2. hidden disagreement: the groups rank concise and detailed responses oppositely.

It fits a tiny Bradley-Terry reward model and a discrete DPO policy twice:
first after erasing rater identity, then while conditioning on rater group.
All prompts, responses, labels and metrics are synthetic teaching fixtures.
"""

from __future__ import annotations

from dataclasses import dataclass
from math import exp
import json
import random
from typing import Mapping, Sequence

RESPONSES: tuple[str, ...] = ("concise", "balanced", "detailed")
GROUPS: tuple[str, ...] = ("concise_reader", "detail_reader")


@dataclass(frozen=True)
class Pair:
    """One synthetic comparison after the interface has forced a choice."""

    group: str
    chosen: str
    rejected: str


def sigmoid(value: float) -> float:
    """Numerically stable logistic function."""

    if value >= 0:
        z = exp(-value)
        return 1.0 / (1.0 + z)
    z = exp(value)
    return z / (1.0 + z)


def softmax(logits: Mapping[str, float]) -> dict[str, float]:
    """Convert response logits into a probability distribution."""

    maximum = max(logits.values())
    exponentials = {name: exp(value - maximum) for name, value in logits.items()}
    total = sum(exponentials.values())
    return {name: value / total for name, value in exponentials.items()}


def make_dataset(divergent: bool, seed: int = 7) -> tuple[list[Pair], dict[str, dict[str, int]]]:
    """Create 300 pairwise labels with an intentionally skewed 70:30 rater mix.

    Eight per cent of labels are flipped to represent ordinary annotation error.
    In the shared-target case both groups prefer balanced responses. In the
    divergent case one group prefers concise and the other prefers detailed.
    """

    if divergent:
        utilities = {
            "concise_reader": {"concise": 2, "balanced": 1, "detailed": 0},
            "detail_reader": {"concise": 0, "balanced": 1, "detailed": 2},
        }
    else:
        utilities = {
            "concise_reader": {"concise": 1, "balanced": 2, "detailed": 0},
            "detail_reader": {"concise": 1, "balanced": 2, "detailed": 0},
        }

    rng = random.Random(seed)
    comparisons: list[Pair] = []
    for group, rater_count in (("concise_reader", 70), ("detail_reader", 30)):
        utility = utilities[group]
        for _ in range(rater_count):
            for index, left in enumerate(RESPONSES):
                for right in RESPONSES[index + 1 :]:
                    if utility[left] > utility[right]:
                        chosen, rejected = left, right
                    else:
                        chosen, rejected = right, left
                    if rng.random() < 0.08:
                        chosen, rejected = rejected, chosen
                    comparisons.append(Pair(group, chosen, rejected))
    rng.shuffle(comparisons)
    return comparisons, utilities


def fit_bradley_terry(
    comparisons: Sequence[Pair],
    *,
    conditional: bool,
    steps: int = 4_000,
    learning_rate: float = 0.03,
    l2_penalty: float = 0.002,
) -> dict[str, dict[str, float]]:
    """Fit free scalar rewards with Bradley-Terry maximum likelihood."""

    keys = GROUPS if conditional else ("all",)
    rewards = {key: {response: 0.0 for response in RESPONSES} for key in keys}

    for _ in range(steps):
        gradients = {key: {response: 0.0 for response in RESPONSES} for key in keys}
        for pair in comparisons:
            key = pair.group if conditional else "all"
            margin = rewards[key][pair.chosen] - rewards[key][pair.rejected]
            residual = 1.0 - sigmoid(margin)
            gradients[key][pair.chosen] += residual
            gradients[key][pair.rejected] -= residual

        scale = 1.0 / len(comparisons)
        for key in keys:
            for response in RESPONSES:
                gradient = gradients[key][response] * scale
                gradient -= l2_penalty * rewards[key][response]
                rewards[key][response] += learning_rate * gradient

            # Reward differences are identifiable; their common offset is not.
            centre = sum(rewards[key].values()) / len(RESPONSES)
            for response in RESPONSES:
                rewards[key][response] -= centre

    return rewards


def fit_discrete_dpo(
    comparisons: Sequence[Pair],
    *,
    conditional: bool,
    beta: float = 0.5,
    steps: int = 4_000,
    learning_rate: float = 0.03,
) -> dict[str, dict[str, float]]:
    """Fit a three-response DPO policy against a uniform reference policy.

    For a discrete response set, log probability differences equal logit
    differences because the softmax normaliser cancels. The update therefore
    implements the DPO binary cross-entropy loss directly.
    """

    keys = GROUPS if conditional else ("all",)
    policy_logits = {key: {response: 0.0 for response in RESPONSES} for key in keys}
    reference_logits = {response: 0.0 for response in RESPONSES}

    for _ in range(steps):
        gradients = {key: {response: 0.0 for response in RESPONSES} for key in keys}
        for pair in comparisons:
            key = pair.group if conditional else "all"
            policy_gap = policy_logits[key][pair.chosen] - policy_logits[key][pair.rejected]
            reference_gap = reference_logits[pair.chosen] - reference_logits[pair.rejected]
            margin = beta * (policy_gap - reference_gap)
            residual = 1.0 - sigmoid(margin)
            gradients[key][pair.chosen] += beta * residual
            gradients[key][pair.rejected] -= beta * residual

        scale = 1.0 / len(comparisons)
        for key in keys:
            for response in RESPONSES:
                policy_logits[key][response] += learning_rate * gradients[key][response] * scale

            # Logits share an arbitrary common offset, so recenter for readability.
            centre = sum(policy_logits[key].values()) / len(RESPONSES)
            for response in RESPONSES:
                policy_logits[key][response] -= centre

    return {key: softmax(logits) for key, logits in policy_logits.items()}


def pair_accuracy(
    rewards: Mapping[str, Mapping[str, float]],
    comparisons: Sequence[Pair],
    *,
    conditional: bool,
) -> float:
    """Fraction of observed comparisons ordered correctly by a reward model."""

    correct = 0
    for pair in comparisons:
        key = pair.group if conditional else "all"
        correct += rewards[key][pair.chosen] > rewards[key][pair.rejected]
    return correct / len(comparisons)


def top_choice_satisfaction(
    policies: Mapping[str, Mapping[str, float]],
    utilities: Mapping[str, Mapping[str, int]],
    *,
    conditional: bool,
) -> dict[str, float]:
    """Probability assigned to each group's genuinely top-ranked response."""

    result: dict[str, float] = {}
    for group in GROUPS:
        policy = policies[group] if conditional else policies["all"]
        favourite = max(utilities[group], key=utilities[group].get)
        result[group] = policy[favourite]
    return result


def rounded(values: Mapping[str, float]) -> dict[str, float]:
    return {name: round(value, 3) for name, value in values.items()}


def run_case(name: str, *, divergent: bool) -> dict[str, object]:
    comparisons, utilities = make_dataset(divergent=divergent)
    aggregate_reward = fit_bradley_terry(comparisons, conditional=False)
    conditional_reward = fit_bradley_terry(comparisons, conditional=True)
    aggregate_policy = fit_discrete_dpo(comparisons, conditional=False)
    conditional_policy = fit_discrete_dpo(comparisons, conditional=True)

    aggregate_satisfaction = top_choice_satisfaction(
        aggregate_policy, utilities, conditional=False
    )
    conditional_satisfaction = top_choice_satisfaction(
        conditional_policy, utilities, conditional=True
    )

    return {
        "case": name,
        "labels": len(comparisons),
        "rater_mix": {"concise_reader": 0.70, "detail_reader": 0.30},
        "reward_pair_accuracy": {
            "identity_erased": round(
                pair_accuracy(aggregate_reward, comparisons, conditional=False), 3
            ),
            "identity_preserved": round(
                pair_accuracy(conditional_reward, comparisons, conditional=True), 3
            ),
        },
        "dpo_policy": {
            "identity_erased": rounded(aggregate_policy["all"]),
            "concise_reader": rounded(conditional_policy["concise_reader"]),
            "detail_reader": rounded(conditional_policy["detail_reader"]),
        },
        "top_choice_satisfaction": {
            "identity_erased": rounded(aggregate_satisfaction),
            "identity_preserved": rounded(conditional_satisfaction),
            "worst_group_erased": round(min(aggregate_satisfaction.values()), 3),
            "worst_group_preserved": round(min(conditional_satisfaction.values()), 3),
        },
    }


def main() -> None:
    shared = run_case("shared target (positive control)", divergent=False)
    divided = run_case("hidden disagreement (negative case)", divergent=True)

    # Positive control: aggregation should remain adequate when preferences align.
    assert shared["top_choice_satisfaction"]["worst_group_erased"] > 0.90

    # Negative case: identity erasure should hide the minority's preferred response.
    assert divided["reward_pair_accuracy"]["identity_erased"] < 0.70
    assert divided["reward_pair_accuracy"]["identity_preserved"] > 0.85
    assert divided["top_choice_satisfaction"]["worst_group_erased"] < 0.20
    assert divided["top_choice_satisfaction"]["worst_group_preserved"] > 0.85

    print(json.dumps({"shared_target": shared, "hidden_disagreement": divided}, indent=2))
    print("\nAll assertions passed.")


if __name__ == "__main__":
    main()

{
  "shared_target": {
    "case": "shared target (positive control)",
    "labels": 300,
    "rater_mix": {
      "concise_reader": 0.7,
      "detail_reader": 0.3
    },
    "reward_pair_accuracy": {
      "identity_erased": 0.91,
      "identity_preserved": 0.91
    },
    "dpo_policy": {
      "identity_erased": {
        "concise": 0.031,
        "balanced": 0.968,
        "detailed": 0.001
      },
      "concise_reader": {
        "concise": 0.046,
        "balanced": 0.951,
        "detailed": 0.003
      },
      "detail_reader": {
        "concise": 0.07,
        "balanced": 0.923,
        "detailed": 0.007
      }
    },
    "top_choice_satisfaction": {
      "identity_erased": {
        "concise_reader": 0.968,
        "detail_reader": 0.968
      },
      "identity_preserved": {
        "concise_reader": 0.951,
        "detail_reader": 0.923
      },
      "worst_group_erased": 0.968,
      "worst_group_preserved": 0.923
    }
  },
  "hidden_disagreement": {
    "case": "hidden disagreement (negative case)",
    "labels": 300,
    "rater_mix": {
      "concise_reader": 0.7,
      "detail_reader": 0.3
    },
    "reward_pair_accuracy": {
      "identity_erased": 0.643,
      "identity_preserved": 0.91
    },
    "dpo_policy": {
      "identity_erased": {
        "concise": 0.583,
        "balanced": 0.292,
        "detailed": 0.124
      },
      "concise_reader": {
        "concise": 0.931,
        "balanced": 0.065,
        "detailed": 0.004
      },
      "detail_reader": {
        "concise": 0.007,
        "balanced": 0.091,
        "detailed": 0.902
      }
    },
    "top_choice_satisfaction": {
      "identity_erased": {
        "concise_reader": 0.583,
        "detail_reader": 0.124
      },
      "identity_preserved": {
        "concise_reader": 0.931,
        "detail_reader": 0.902
      },
      "worst_group_erased": 0.124,
      "worst_group_preserved": 0.902
    }
  }
}

All assertions passed.