Part IDistillation can transfer only what the queries expose

Begin with a classroom. A teacher can solve algebra, geometry, probability and formal logic. A student attends only algebra lessons. Even perfect imitation during those lessons does not imply that geometry crossed invisibly. Knowledge distillation has the same causal structure. The teacher is evaluated on selected inputs, and its resulting distribution, label or structured output becomes training evidence for the student.

The seminal formulation uses the teacher's softened output distribution rather than only its top class. At a higher temperature, probabilities that were nearly zero become visible, exposing relations among alternatives. A handwritten seven may be classified mostly as seven, with smaller probabilities assigned to one and nine. Those smaller values can tell the student more than the hard label alone because they encode how the teacher partitions nearby cases. Hinton, Vinyals and Dean showed that this signal could compress ensembles and larger networks into deployable students.[1]

The expectation over q is the quiet term that decides the transfer boundary. If an input region has no mass under q, its contribution to the loss is zero. If a rare override appears twice among a million ordinary examples, its gradient can be overwhelmed even though its business consequence is large. Reweighting can help only after the case is identified. The training objective cannot demand preservation of an unknown omission.

Several hidden teachers can produce the same lesson

There is a deeper identifiability problem. Imagine two teachers that return exactly the same labels and probabilities on every query in the transfer set. Teacher A uses a robust relation among approved fields. Teacher B uses a shortcut that happens to agree on those cases. From the observed outputs alone, the student cannot know which internal generator produced them. Any training method that consumes only those outputs receives identical evidence from both teachers.

Now move to an intervention where the shortcut breaks. The two teachers diverge, but the student has no basis for choosing the robust response unless the transfer design included that intervention, a validated intermediate artefact or an external rule. This is why an impressive agreement score can coexist with causal ambiguity. The problem is not that the student failed to read the teacher's mind. The transfer channel never contained the distinction.

Hidden-state matching can enlarge the channel, but it changes rather than removes the assumption. The student's layers may differ in width, organisation and function, so an aligned hidden vector is not automatically the same computation. Free-form rationales enlarge the observable trace, yet they can also be post-hoc, stylistic or impossible to verify. For a bounded skill, the strongest transfer target is usually the smallest intermediate artefact whose correctness can be checked independently: a cited evidence span, a normalised entity, a calculation, a rule identifier or a typed abstention.

Distillation is therefore an experimental design problem before it is a loss-function problem. Choose queries that discriminate desired rules from plausible shortcuts. Include counterfactual pairs that hold wording constant while changing the decisive field, and pairs that change wording while preserving the decision. Record which distinctions the teacher must expose, then test them again without relying on teacher agreement as the only oracle.

Soft targets reveal relations, not hidden procedures

Temperature is useful because a distribution contains more structure than an argmax. At temperature one, a teacher may assign 0.98, 0.015 and 0.005 across three classes. At temperature three, the ordering remains but the smaller alternatives become large enough to influence the student's gradient. This is sometimes called dark knowledge: information in the teacher's non-winning probabilities.

That phrase should not be inflated. A probability vector can reveal local class similarity under the teacher. It does not automatically reveal the causal procedure, external evidence, tool call, safety rule or search process that generated the score. A student may match the final distribution through a shortcut. Stanton and colleagues found that distillation could improve generalisation even while substantial teacher-student predictive discrepancies remained, and that closer matching did not always improve the student.[2] The result is a warning against treating imitation fidelity as a complete explanation of transferred capability.

Sequence models create a second support problem

Autoregressive students are trained on prefixes. During deployment, their own earlier token becomes part of the next prefix. A small error can therefore move the student into a state that never appeared in teacher-generated training sequences. Ordinary offline distillation then asks the student to imitate the teacher on teacher trajectories while deployment exposes student trajectories.

On-policy approaches address this mismatch by letting the student generate sequences and asking the teacher to provide feedback on those states. Generalized Knowledge Distillation was proposed for precisely this training-inference distribution gap.[3] More recent work, including DistiLLM-2 and TAID, changes losses or intermediate distributions to improve transfer across teacher-generated and student-generated data and to manage capacity gaps.[4][5] These methods expand and reshape the observed path. They do not remove the need to define which paths matter.

Structured rationales can also provide additional supervision. Distilling Step-by-Step reported strong task-specific data efficiency on four NLP benchmarks by using model-generated rationales as an auxiliary target.[6] For operational systems, however, the useful artefact is often a verifiable intermediate field: evidence span, rule identifier, extracted amount, calculation or abstention reason. A persuasive free-form rationale may be difficult to validate and can teach stylistic regularities instead of the required mechanism.

flowchart TD
  A["The teacher knew the override.<br/>The student never saw it."] --> B["Part I Distillation can<br/>transfer only what the…"]
  B --> C["Part II LoRA constrains the<br/>change, not the meaning…"]
  C --> D["Part III The release object is<br/>a transfer envelope,…"]
  D -. "re-test" .-> A
  classDef first fill:#0a1424,stroke:#B96D00,stroke-width:2px,color:#e6f3ff
  classDef step fill:#0a1424,stroke:#3f7da0,stroke-width:1px,color:#e6f3ff
  class A first
  class B,C,D step

Part IILoRA constrains the change, not the meaning of the skill

Distillation decides what evidence reaches the student. Adaptation decides how the student may respond to it. Full fine-tuning can update every selected parameter. Low-Rank Adaptation freezes a pretrained weight matrix and learns a factorised update. If a weight has d_out rows and d_in columns, a rank-r adapter replaces a dense change with two thin matrices.

The original LoRA study showed large reductions in trainable parameters and training memory on its evaluated models and tasks, while retaining strong downstream quality.[7] The mechanism is attractive for a fleet because adapters are small relative to independent full checkpoints and the base can remain shared. QLoRA further reduces fine-tuning memory by propagating gradients through a frozen quantised base into LoRA adapters.[8]

Neither result means that LoRA compresses inference by itself. An unmerged adapter still runs with the full base architecture. A merged adapter produces the same dense architecture with altered weights. Training memory and checkpoint storage can fall sharply while the forward pass remains broadly the base model's forward pass. Distillation into a genuinely smaller student, quantisation, pruning or a different architecture is what can reduce serving compute. The PyTorch distillation tutorial states this cleanly: the teacher changes the lightweight model's weights, while the student's forward-pass cost remains that of the lightweight model.[9]

Semantic narrowness and geometric rank can disagree

A task can sound narrow while demanding a broad update. “Format this output as one schema” may affect token selection, long-range ordering, refusal behaviour and several layers of representation. A task can also sound broad while requiring a compact direction: a pretrained base may already know the domain and need only a stable control signal that changes when to use existing knowledge.

This distinction explains why rank should be swept rather than inferred from the number of labels. A twelve-class classifier can have a low-rank correction if the base already separates the cases and only the readout orientation is wrong. A binary decision can require a broader change if the decisive feature is absent, distributed or entangled with behaviours that must remain intact. Rank also interacts with which matrices are targeted. Increasing rank in a module that does not carry the needed feature may add parameters without opening the required path.

A useful ablation holds data and training budget fixed, then varies rank, target modules and student size separately. If higher rank closes the gap while the base and data remain fixed, update capacity is implicated. If changing the base closes it at the same rank, reusable representation was the bottleneck. If neither helps until new contrasts enter the dataset, query support was the bottleneck. This factorial view turns hyperparameter tuning into diagnosis.

There is evidence that some language-model adaptation problems have low intrinsic dimension. Aghajanyan, Gupta and Zettlemoyer found low-dimensional reparameterisations that retained much of full fine-tuning performance on studied tasks.[10] This helps explain why small adaptation surfaces can work. It does not imply one universal rank, because intrinsic dimension depends on the base, task, data and required performance.

Negative evidence sharpens the boundary. Biderman and colleagues reported that standard low-rank LoRA underperformed full fine-tuning in their programming and mathematics settings, while preserving more out-of-domain behaviour. They also found full fine-tuning perturbations with ranks far above typical LoRA settings.[11] Shuttleworth and colleagues found that similar downstream scores could hide different spectral structure and continual-learning behaviour, including LoRA-specific “intruder dimensions”.[12] These results do not invalidate LoRA. They invalidate the inference that matching one task metric proves equivalence to full fine-tuning.

Rank is only one adaptation boundary

A LoRA configuration also chooses target modules, scaling, dropout, data weighting and whether adapters are merged, routed or composed. A rank-16 update on attention projections is a different hypothesis from rank 16 on attention and feed-forward projections. The same nominal trainable parameter count can therefore expose different behavioural capacity.

Variants such as DoRA change the parameterisation by separating magnitude and direction, and the authors reported gains over LoRA across their studied downstream tasks.[13] Such methods are valuable candidates when a plain adapter underfits, but the engineering question remains unchanged: which update family passes the required behaviours and regressions? A more expressive parameterisation enlarges the search space. It does not define the skill or prove that the teacher evidence covers it.

The base model matters just as much. A small student that lacks necessary vocabulary, context handling, numerical representation or modality cannot be repaired by rank alone. Conversely, a well-chosen pretrained base may already contain most needed structure, so a tiny update can reorient it. LoRA is economical when the base owns the reusable capability and the task requires a compact correction. It is a poor substitute for capacity the base never acquired.

Part IIIThe release object is a transfer envelope, not a benchmark score

A private skill model becomes defensible when four sets overlap. The query set covers the situations the model will face. The representation exposes the features needed to distinguish them. The student and update family can express the required mapping. The evaluation can tell acceptable behaviour from a fluent imitation. The deployment boundary is the intersection, not the union.

This formalism avoids two common confusions. First, “private” describes placement, access and control boundaries. It does not imply that the model is accurate, calibrated, secure or authorised. Second, “teacher-level” is incomplete without a denominator. Agreement on a random held-out split of the same logs can coexist with failure on new templates, rare cohorts, changed rules and student-generated trajectories.

A production-shaped case: payment exception handling

Consider a synthetic operations system that maps inbound payment exceptions to twelve handling codes plus abstain. The input contract contains parsed message fields, a normalised narrative, message standard, amount band, legal-hold flag and current rule version. The output contract contains the proposed code, evidence fields, confidence and reason for abstention. A deterministic authority kernel decides whether the proposal may enter an automated queue.

The teacher produces structured labels for historical cases and expert-designed counterfactuals. Ordinary examples are plentiful. Rare sanctions-adjacent, legal-hold and market-disruption cases receive deliberate coverage because their consequence is not proportional to frequency. The student is smaller than the teacher, so distillation can reduce serving compute. A LoRA sweep tests ranks 4, 8, 16 and 32 on the chosen base, with target modules and seeds held in the experiment manifest.

The first release gate is task quality by cohort, including an untouched temporal set and message-template groups that never cross between training and evaluation. The second is risk coverage: how error and abstention change as the automatic coverage threshold rises. The third is evidence integrity: the proposed code must cite fields that actually exist and the rule version must be current. The fourth is a negative-control baseline. A rules engine, sparse classifier or retrieval-plus-calculation path may solve more of the task with less behavioural uncertainty.

The test set should contain matched counterfactuals. In one pair, the narrative remains nearly identical while the legal-hold flag changes; the expected action must flip. In another, wording, sender order and harmless metadata change while the decisive fields remain fixed; the action must stay invariant. These pairs separate semantic robustness from template memorisation more directly than a random average.

Protected tails need denominators and uncertainty. Reporting “all legal-hold cases passed” is weak when the set contains four near-duplicates. Record unique source groups, scenario families and the smallest consequence-relevant subgroup. For automated coverage, plot error against coverage separately for ordinary and protected cohorts. A global threshold that looks safe can route an overconfident rare cohort through the automatic path.

Finally, compare student errors with teacher errors rather than treating all disagreement alike. A student-only error suggests transfer or capacity debt. A teacher-and-student error suggests oracle or task-specification debt. A student correction of a teacher error may be desirable when expert labels or deterministic checks support it. The target is verified business behaviour, not maximum obedience to the teacher.

Suppose rank 8 matches aggregate teacher accuracy but misses two rare cohorts, while rank 16 closes one gap and leaves another. Increasing to rank 32 improves neither. That pattern suggests the remaining problem is not rank alone. The query set may omit the decisive contrast, the base may represent the message standard poorly, or the teacher label may be inconsistent. Rank sweeps are diagnostic interventions, not an automatic staircase towards quality.

The system still needs fallback. A calibrated abstention routes ambiguous or out-of-envelope cases to a stronger model or a human queue. The fallback receives the original evidence, not merely the student's narrative. A new message type, rule version or error-rate change invalidates relevant parts of the release evidence and triggers evaluation. The private skill model is one bounded mechanism in a cascade, not a replacement for every upstream capability.

Four failure families that averages conceal

Support failure occurs when required cases or student-generated states are absent from the transfer evidence. Capacity failure occurs when the student or adapter cannot represent the mapping even with adequate cases. Oracle failure occurs when teacher outputs are wrong, inconsistent or unverifiable, causing the student to compress defects. Currency failure occurs when rules, language or world state change after training.

These failures require different remedies. More examples help support failure only when they contain the missing distinction. More rank helps capacity failure only when the target modules and base make that direction reachable. Better teacher sampling does not repair a stale rule embedded in weights. Calibration can route uncertainty but cannot turn an unauthorised action into an authorised one.

Teacher strength can itself create a capacity mismatch. Fan and colleagues describe evidence that a stronger teacher is not automatically a better distillation teacher for a weaker student and analyse the role of calibration in common KL-based objectives.[14] The practical response is to treat teacher, temperature, loss and student size as an experiment, rather than ordering teachers by one headline benchmark.

Part IVBuild the private skill model only after the skill separates

A private skill model is most compelling when repeated work has a stable decision boundary, a compact output contract and a measurable fallback. High volume can then amortise teacher queries, curation, evaluation and hosting. Data locality or third-party constraints may strengthen the case for controlled placement. None of these reasons should weaken the quality gate.

The sequence matters. Start with the outcome and boundary, not a preferred tuning technique. Define the smallest input required, the permitted outputs, the abstention behaviour, the source of current knowledge and the postcondition by which success is verified. Establish deterministic, retrieval and untuned-base baselines. Only then ask whether distillation supplies useful targets and whether a low-rank update captures the remaining behavioural change.

Separate inference compression from adaptation efficiency

There are two economic questions. The first is whether a smaller student can replace a larger teacher on a bounded workload. That is inference compression. The second is whether the chosen student can be adapted with fewer trainable parameters and smaller per-skill artefacts. That is adaptation efficiency. Distillation primarily addresses the first; LoRA primarily addresses the second. Combining them is useful precisely because they remove different costs.

A third question is the cost of keeping the claim true. Teacher calls, expert adjudication, drift tests, calibration, red-team cases, registry operations and fallback all belong in the unit economics. A model that is cheap per token but abstains on half the workload may be more expensive per accepted case than a stronger route. A model that needs weekly retraining because changing facts were embedded in weights may be less economical than retrieval, even when each training run is small.

The finance calculation should therefore begin after the quality gate. For each candidate, estimate the total monthly cost of student inference, expected fallback, amortised training and evaluation, infrastructure idle time and review. Divide by the number of cases whose outcomes meet the release definition. Apply the same boundary to the baseline. This prevents a cheap but low-coverage student from being compared with an expensive route that handles the full task.

Economics also changes the optimal envelope. A very cheap student may be valuable at conservative coverage, handling only the obvious centre while the frontier route retains the tail. There is no requirement that the private model absorb every case. The defensible objective is the largest verified region for which marginal savings exceed the marginal assurance and fallback cost.

A common mistake is to train LoRA on the same large base used in the frontier route and then claim model compression because the adapter file is small. Storage per skill may indeed fall, and training may become accessible. Serving still loads and executes the base. The correct cost comparison includes accelerator memory, token throughput, batching, adapter switching, quantisation, cold start, fallback rate, evaluation refresh and human review. Measure cost per verified accepted case, not price per generated token in isolation.

A workshop that makes both bottlenecks visible

The included NumPy script runs two controlled experiments. The first teacher uses an ordinary three-class surface plus a rare override. A student trained only on ordinary teacher queries reaches 100 per cent agreement on the ordinary slice and 25.8 per cent on the unqueried tail. Adding tail queries lets the same representational student reach 100 per cent on both synthetic sets. The second experiment constructs one rank-3 update and one broad-spectrum update, then computes the best rank-r approximation. Rank three exactly recovers the first update and retains 0.936 relative error on the second.

Those numbers prove nothing about a real model. Their purpose is intervention clarity. In the first experiment only support changes. In the second only update spectrum changes. Together they prevent “the student is small” from becoming a catch-all explanation for every failure.

#!/usr/bin/env python3
"""Synthetic workshop for Paper 71: Distillation, LoRA and the Private Skill Model.

The script isolates two claims:
1. Distillation approximates teacher behaviour on the query distribution. An
   unqueried override is not inherited merely because the teacher knows it.
2. A LoRA-style rank-r update can reproduce an intrinsically low-rank change,
   but the same rank can lose a broad update whose singular spectrum is flat.

Requirements: Python 3.10+ and NumPy.
All data and results are synthetic. Nothing here is a model benchmark.
"""
from __future__ import annotations

from dataclasses import dataclass
import numpy as np

SEED = 71071



def teacher_logits(x: np.ndarray) -> np.ndarray:
    """Three-class teacher with a rare, high-consequence override.

    x[:, 0:2] govern the ordinary decision surface. When x[:, 2] > 1.5,
    the teacher applies an override and strongly selects class 2.
    """
    logits = np.column_stack(
        [
            1.8 * x[:, 0] - 0.5 * x[:, 1],
            -1.0 * x[:, 0] + 1.5 * x[:, 1],
            -0.6 * x[:, 0] - 1.0 * x[:, 1],
        ]
    )
    override = x[:, 2] > 1.5
    logits[override, :] -= 4.0
    logits[override, 2] += 12.0
    return logits


def make_queries(
    rng: np.random.Generator,
    n: int,
    tail_fraction: float,
) -> np.ndarray:
    if n <= 0:
        raise ValueError("n must be positive")
    if not 0.0 <= tail_fraction <= 1.0:
        raise ValueError("tail_fraction must lie between 0 and 1")
    x = rng.normal(0.0, 1.0, size=(n, 3))
    x[:, 2] = rng.normal(0.0, 0.20, size=n)
    tail_count = int(round(n * tail_fraction))
    if tail_count:
        idx = rng.choice(n, size=tail_count, replace=False)
        x[idx, 2] = rng.uniform(1.6, 3.0, size=tail_count)
    return x


def student_features(x: np.ndarray) -> np.ndarray:
    """Features available to the student, including a representable tail gate."""
    return np.column_stack(
        [x, np.maximum(x[:, 2] - 1.5, 0.0), (x[:, 2] > 1.5).astype(float)]
    )


@dataclass
class LinearStudent:
    weights: np.ndarray
    bias: np.ndarray

    def logits(self, x: np.ndarray) -> np.ndarray:
        return student_features(x) @ self.weights + self.bias


def fit_student(
    x: np.ndarray,
    teacher_temperature_logits: np.ndarray,
    *,
    ridge: float = 1e-8,
) -> LinearStudent:
    """Fit a linear student by matching temperature-scaled teacher logits.

    This least-squares surrogate is deliberately simple. It makes the support
    boundary visible without hiding it behind optimiser behaviour.
    """
    features = student_features(x)
    design = np.column_stack([features, np.ones(features.shape[0])])
    gram = design.T @ design
    penalty = ridge * np.eye(gram.shape[0])
    penalty[-1, -1] = 0.0
    coefficients = np.linalg.solve(gram + penalty, design.T @ teacher_temperature_logits)
    return LinearStudent(weights=coefficients[:-1], bias=coefficients[-1])


def agreement(student: LinearStudent, x: np.ndarray) -> float:
    teacher_choice = teacher_logits(x).argmax(axis=1)
    student_choice = student.logits(x).argmax(axis=1)
    return float(np.mean(teacher_choice == student_choice))


def distillation_experiment() -> dict[str, float]:
    rng = np.random.default_rng(SEED)
    temperature = 2.0

    ordinary_train = make_queries(rng, n=5000, tail_fraction=0.0)
    ordinary_targets = teacher_logits(ordinary_train) / temperature
    ordinary_student = fit_student(ordinary_train, ordinary_targets)

    covered_train = make_queries(rng, n=5000, tail_fraction=0.10)
    covered_targets = teacher_logits(covered_train) / temperature
    covered_student = fit_student(covered_train, covered_targets)

    ordinary_test = make_queries(rng, n=4000, tail_fraction=0.0)
    tail_test = make_queries(rng, n=4000, tail_fraction=1.0)

    return {
        "ordinary_student_ordinary": agreement(ordinary_student, ordinary_test),
        "ordinary_student_tail": agreement(ordinary_student, tail_test),
        "covered_student_ordinary": agreement(covered_student, ordinary_test),
        "covered_student_tail": agreement(covered_student, tail_test),
    }


def best_rank_approximation(delta: np.ndarray, rank: int) -> np.ndarray:
    if delta.ndim != 2:
        raise ValueError("delta must be a matrix")
    if not 1 <= rank <= min(delta.shape):
        raise ValueError("rank must be between 1 and the matrix's smaller dimension")
    u, singular_values, vt = np.linalg.svd(delta, full_matrices=False)
    return (u[:, :rank] * singular_values[:rank]) @ vt[:rank, :]


def relative_error(target: np.ndarray, approximation: np.ndarray) -> float:
    numerator = np.linalg.norm(target - approximation, ord="fro")
    denominator = np.linalg.norm(target, ord="fro")
    if denominator == 0:
        raise ValueError("target must contain a non-zero update")
    return float(numerator / denominator)


def decision_agreement(
    rng: np.random.Generator,
    target: np.ndarray,
    approximation: np.ndarray,
    n: int = 5000,
) -> float:
    x = rng.normal(size=(n, target.shape[1]))
    target_choice = (x @ target.T).argmax(axis=1)
    approximation_choice = (x @ approximation.T).argmax(axis=1)
    return float(np.mean(target_choice == approximation_choice))


def rank_experiment() -> list[dict[str, float | int | str]]:
    rng = np.random.default_rng(SEED + 1)
    width = 48

    left = rng.normal(size=(width, 3))
    right = rng.normal(size=(3, width))
    low_rank_delta = left @ right
    low_rank_delta /= np.linalg.norm(low_rank_delta, ord="fro")

    q_left, _ = np.linalg.qr(rng.normal(size=(width, width)))
    q_right, _ = np.linalg.qr(rng.normal(size=(width, width)))
    broad_spectrum = np.linspace(1.0, 0.35, width)
    broad_delta = q_left @ np.diag(broad_spectrum) @ q_right.T
    broad_delta /= np.linalg.norm(broad_delta, ord="fro")

    rows: list[dict[str, float | int | str]] = []
    for name, delta in (("intrinsic-rank-3", low_rank_delta), ("broad-spectrum", broad_delta)):
        for rank in (1, 2, 3, 4, 8, 16):
            approximation = best_rank_approximation(delta, rank)
            rows.append(
                {
                    "case": name,
                    "rank": rank,
                    "relative_error": relative_error(delta, approximation),
                    "decision_agreement": decision_agreement(rng, delta, approximation),
                    "parameter_fraction": rank * (2 * width) / (width * width),
                }
            )
    return rows


def print_results() -> None:
    d = distillation_experiment()
    print("DISTILLATION: teacher-student top-class agreement")
    print(f"  ordinary-only student on ordinary slice : {d['ordinary_student_ordinary']:.1%}")
    print(f"  ordinary-only student on unqueried tail : {d['ordinary_student_tail']:.1%}")
    print(f"  tail-covered student on ordinary slice  : {d['covered_student_ordinary']:.1%}")
    print(f"  tail-covered student on covered tail    : {d['covered_student_tail']:.1%}")
    print()
    print("RANK WORKSHOP: best rank-r approximation of a target update")
    print("  case              rank  params     rel.error  decision agreement")
    for row in rank_experiment():
        print(
            f"  {row['case']:<17} {row['rank']:>4}  "
            f"{row['parameter_fraction']:>6.1%}     "
            f"{row['relative_error']:>7.3f}       "
            f"{row['decision_agreement']:>6.1%}"
        )


if __name__ == "__main__":
    print_results()

DISTILLATION: teacher-student top-class agreement
  ordinary-only student on ordinary slice : 100.0%
  ordinary-only student on unqueried tail : 25.8%
  tail-covered student on ordinary slice  : 100.0%
  tail-covered student on covered tail    : 100.0%

RANK WORKSHOP: selected rows
  case              rank  params     rel.error  decision agreement
  intrinsic-rank-3     3   12.5%       0.000       100.0%
  broad-spectrum       3   12.5%       0.936         7.8%
  broad-spectrum      16   66.7%       0.673        29.5%

The skill separability record

The following instrument is a gate rather than a weighted score. A high-volume workload does not compensate for an unverifiable high-consequence tail. Cheap training does not compensate for missing query support. Every failed gate names an architecture response.

Govern the whole released package

The model registry should bind the student base, adapter, tokenizer, prompt or instruction template, teacher version, query manifest, data rights, calibration, evaluator, runtime schema and fallback policy. A change to any of these can alter effective behaviour. NIST's final SSDF profile for generative AI and dual-use foundation models explicitly extends secure development practices across AI model development, which reinforces treating lineage and release controls as engineering artefacts rather than paperwork after training.[15]

The evidence should also record defeaters. “Teacher agreement fell below the predeclared margin on legal-hold cases” is a release blocker. “Rank 8 was cheaper” is not a rebuttal. “The teacher has a higher general benchmark” is not evidence that a weaker student can absorb the relevant policy. Claims should expire when the input population, teacher, base, rule set or serving configuration changes materially.

sequenceDiagram
  participant Q as Question
  participant M as Mechanism
  participant E as Evidence
  participant D as Decision
  Q->>M: The teacher knew the override. The student never saw it.
  M->>E: Figure 1. What actually crosses the teacher-student boundary
  E-->>M: compare rival explanations
  E->>D: Compress the skill slice, not the frontier model
  D-->>Q: revise scope or act

Compress the skill slice, not the frontier model

Distillation is powerful because a teacher's distribution can carry more task structure than a hard label. LoRA is powerful because a pretrained model often needs a compact behavioural correction rather than a complete rewrite. Their combination can turn an expensive, repeated workload into a smaller student with modular adaptation.

The same mechanisms define the limit. Distillation sees the teacher through a query distribution. LoRA sees weight change through a rank-constrained channel. Student architecture, input representation, optimisation and evaluation narrow the result further. A high score inside that intersection is valuable evidence. It is not evidence that the student contains a frontier model in compressed form.

The approval object must therefore be a transfer envelope with an escape route. State the task, query support, protected tails, representation, student, adapter, teacher, evaluator, calibration, currency trigger and fallback. Compare against deterministic and retrieval baselines. Measure inference savings only after quality, authority and recovery pass.

The changed decision is precise. Build a private skill model when the required capability is narrow enough to expose, low-dimensional enough to adapt, stable enough to version and observable enough to verify. Keep changing knowledge outside the weights. Keep exact rules in deterministic controls. Keep broad residual reasoning behind a stronger route. Compression then becomes an evidence-backed placement of one capability, rather than a claim to have recreated the teacher.