When a public API call is not an option
A recurring platform question is why every skill should not be routed through a public frontier-model endpoint. It is a fair question. The answer depends on data handling, third-party scope, economics and auditability rather than a blanket judgement about model quality.
The first part is data handling. A prompt containing an account number, transaction narrative or KYC reference may carry personal, confidential or regulated data. A third-party inference service adds transfer location, subprocessors, retention, support access and incident response to the control boundary. The resulting route may be acceptable under the institution's legal analysis and contract for one workload and prohibited by policy for another. “Private” is therefore a workload decision, not a universal legal requirement.
The second part is governance scope. A third-party model call adds a provider, its infrastructure and its change process to the system boundary. The exact regulatory classification depends on the institution and jurisdiction. In the United States, the Federal Reserve’s SR 26-2 now supersedes SR 11-7 and excludes generative and agentic AI from its formal model definition. The guidance still says wider risk-management practices should inform controls for tools outside scope. Third-party risk, privacy, operational resilience, security and contractual duties remain relevant regardless of the inventory label. Those controls may include due diligence, change notification, audit rights and continuing performance monitoring.
The third part is arithmetic. A document-classification skill running across ten million inbound items a month is not a small workload. The business case depends on measured token volume, current provider prices, private-serving costs and the accuracy envelope the task requires. A smaller model may be economical for a narrow, repetitive skill, but that is an evaluation result rather than an architectural assumption. Compare candidates on task errors, calibration, latency, serving utilisation and the cost of the fallback route before deciding to own the model.
The fourth part is auditability. A calibrated cascade depends on a versioned model interface, replayable evidence and a known change policy. Some hosted services provide pinned versions, private networking and change notice; others expose less control. The institution must calibrate against the exact served version and treat a provider change as a new evaluation event. Owning weights can improve reproducibility, but it does not by itself establish fitness or explainability.
None of this removes hosted frontier models from the estate. They can serve as teachers or as a governed route for open-ended tasks. For narrow, high-volume skills, a private smaller model is a candidate worth testing against deterministic logic, retrieval, a hosted service and human handling. The evaluation, not architectural fashion, decides.
The foundry and the SWIFT example below are a composite reference design. Exact volumes, timings, costs, thresholds and incidents are illustrative assumptions derived from common delivery patterns; they are not measurements attributed to a disclosed banking group.
The foundry concept
The term I use with engineering teams is a skill model foundry: a repeatable pipeline that takes a narrow skill specification and a curated dataset and produces a small, deployable, calibrated model, entirely inside the bank's own virtual private cloud, with no path to the public internet at any stage.
The word “foundry” is doing real work here: this is infrastructure, not a one-off fine-tuning script handed over as a checkpoint. It is a fixed sequence of stages with defined inputs, outputs and gates. A team submits a skill specification and approved dataset; the pipeline returns a signed candidate or a rejection with reasons. Consistent lineage and promotion controls keep a fleet from becoming a pile of bespoke checkpoints, while model and data choices can still vary by skill.
The pipeline has five stages after data curation: teacher distillation, LoRA fine-tuning of a shared base model, a calibration pass, a red-teaming gate, and promotion to the production model registry. Every stage writes its artefacts, logs, and decision record to an internal artefact store, and nothing crosses from one stage to the next without a signed manifest recording the dataset hash, the teacher model version, the adapter checkpoint, and the calibration curve. This is not bureaucracy for its own sake. When a model validation team asks eighteen months later why a particular skill model made a particular decision on a particular date, the manifest is the answer.
The diagram below shows the reference pipeline. Its simplicity is deliberate; a foundry that only one specialist can operate defeats the point of repeatable infrastructure.
What the diagram does not show is time. The composite budgets nine to fourteen working days for a skill in a mature foundry, mostly for curation, annotation and review. That range is not a benchmark: data readiness, review queues, task complexity and infrastructure can move it sharply. Training may be the shorter phase; assembling defensible evidence is often the schedule driver.
Inside the VPC: network topology
The requirement that drives every design decision in this pipeline is simple to state and hard to satisfy: no internet egress, at any stage, from any component that touches customer data or model weights. That single constraint shapes the network topology more than any modelling choice does.
The training cluster sits in a private subnet with no internet gateway and no NAT gateway to the outside world. It can reach exactly three other things: the artefact store, where datasets, checkpoints, and manifests are written and read; the teacher model endpoint, a private, internally hosted or privately peered service rather than a public API; and the model registry, which receives only models that have cleared the red-teaming gate. Everything else, including package installation and monitoring dashboards, is served from an internal mirror inside the same VPC boundary, refreshed on a controlled cadence by a separate team that does have limited egress for exactly that purpose.
The teacher endpoint deserves particular attention because careless configuration can create hidden egress. In the reference design, the teacher is either internally hosted or reached through a cloud private-service endpoint with no internet-routable path from the training subnet. Private routing reduces one class of exposure; it does not remove provider, identity, retention or contractual risk. The network review should document DNS resolution, route tables, endpoint policy, provider access and failure behaviour rather than relying on the word “private”.
Two smaller details close the loop. Every artefact read and teacher call carries the requesting job identifier. The registry sits behind a separate promotion identity so a training job cannot write directly to production. This separation should be tested with a deliberate early-promotion attempt before launch; the expected result is a denied write and a control event, not reliance on a claimed incident history.
Data curation
Every skill model starts life as a specification document: what the skill decides, what the input and output schema look like, what the acceptable error modes are, and which downstream system consumes the output. From that specification, the curation stage builds a dataset from three sources: historical production logs where the outcome is already known, expert annotation for cases the logs do not cover well, and, for a handful of skills, synthetic examples generated under tight constraints to cover rare classes.
Production logs are often the largest source, but they reflect what the current system did rather than what the ideal system should do. A payment-routing skill trained only on historical outcomes can reproduce existing blind spots. Treat log-derived examples as a starting point and use expert-labelled cases to correct coverage. The composite tests a three-to-one weighting for expert cases despite their lower raw count; the real ratio belongs in an ablation study, not a rule of thumb.
Deduplication comes next. Operational logs repeat templates and near-identical narratives. In the worked sizing exercise, 80,000 raw records fall to about 23,000 clusters at a 0.92 cosine threshold. That collapse rate and threshold are corpus-specific. Review cluster errors, protect rare cases and split related records as groups before interpreting the reduction as genuine diversity.
Contamination checking compares the eventual evaluation set with the training pool for exact, near-exact and lineage-related matches. In the illustrative trade-finance case, 6.4% of the proposed evaluation set overlaps an earlier annotation batch. The clean remedy is to rebuild the split by source group and rerun the baseline rather than estimate a fixed accuracy inflation.
Privacy treatment is a release condition. Examples pass through detection and redaction before training: names, account and card numbers, addresses and sensitive free text are replaced where the approved purpose does not require them. The reference pipeline scans at ingestion and again before the manifest is sealed, then samples residual risk. Two automated passes are defence in depth, not proof that the dataset is free of personal data.
Teacher distillation
Once the dataset skeleton exists, the next stage is to generate the actual training signal using a larger teacher model, either hosted internally or reached through the private endpoint described earlier. The point of distillation is not simply to copy the teacher's outputs; it is to extract a decision policy that a much smaller student model can learn to approximate, at a fraction of the parameter count and inference cost.
For classification and extraction, the teacher can generate a final label plus structured, verifiable intermediate fields such as the evidence span, rule identifier and abstention reason. The foundry need not capture or expose private chain-of-thought. It should train on artefacts that a reviewer can check and that the runtime is permitted to retain.
Teacher output is not ground truth. In the composite, each example runs five times at temperature 0.6 and the final structured label is compared. Four-of-five label agreement makes the case eligible for automatic filtering, but there is no literal “majority trajectory” when rationales differ. Retain a rationale only when its evidence fields pass deterministic checks; route disagreement or failed evidence to expert annotation.
In the composite, consistency filtering removes 8% to 15% of the teacher-generated pool. That range is illustrative. Disagreement cases are routed to expert review rather than discarded because ambiguity and rare conditions may concentrate there.
The number of distillation passes affects cost. The planning model allows two or three iterations for a new rubric: generate structured outputs, review a sample, correct the rubric and rerun. Mature templates may reduce rework, but that should be measured. Teacher inference shifts some cost to training time; it does not remove ongoing costs for refresh, evaluation and fallback.
Lora fine-tuning of the student
With a filtered, high-quality trajectory dataset in hand, the student model is fine-tuned using low-rank adaptation rather than full parameter updates, and this choice is close to non-negotiable once you are running a fleet of skills rather than a single model.
The mechanics are well understood: instead of updating every weight matrix in the base model, LoRA freezes the base weights and trains a small pair of low-rank matrices added to selected weight matrices at inference time. For the skill models in this foundry, the base student is typically a seven billion parameter open-weight model, and the LoRA adapters target the query and value projection matrices in each attention block, along with the feed-forward down-projection for skills involving heavier extraction or structured output work.
Rank selection is empirical. The composite tests rank 8 for a binary flag and ranks 16 and 32 for a multi-label classifier, producing adapter candidates of roughly 19 million to 70 million trainable parameters under its chosen target modules. Task labels alone do not determine the right rank; compare held-out quality, stability, memory and latency across several seeds.
The operational case for LoRA is a fleet hypothesis as much as a training-cost hypothesis. The composite uses a rank-16 adapter, an eight-GPU node and adapter packages between 80 and 300 MB. Actual time and size depend on architecture, precision, targets and serving implementation. Shared base weights can reduce duplicated storage, but adapter switching, batching, isolation and latency still need load tests before claiming a fleet saving.
Adapters can also be versioned independently as business processes change. A rollback still needs the base model, adapter, tokenizer, prompt, calibration and routing policy to move as one tested package. It is not merely a file swap if any of those dependencies changed.
Calibration after the merge
A model that is more accurate after fine-tuning is not automatically calibrated. Accuracy and calibration can move independently. LoRA adaptation, including a merge into base weights, can shift the confidence distribution even when accuracy improves, so the previous calibration must be treated as unverified.
The mechanism is straightforward once you look for it. The base model's output layer was calibrated, to whatever degree it was, against a very different training distribution than the narrow skill dataset it has just been fine-tuned on. After the merge, the model has learned sharper, more confident distinctions specific to the skill's decision boundaries, and it tends to become systematically overconfident on exactly the classes it saw most during fine-tuning, while remaining comparatively underconfident on rarer classes it saw fewer times. Left unaddressed, this is not a cosmetic problem. Every skill model in this foundry feeds into a calibrated cascade routing layer, where a confidence threshold decides whether a case is handled automatically or escalated to a more expensive model or a human reviewer.
A model whose stated ninety percent confidence actually corresponds to seventy-eight percent real-world accuracy will silently route far too many cases through the cheap automatic path, and nobody notices until the error rate downstream creeps up weeks later.
The calibration pass remains separate from fine-tuning. After merge, the model runs over a held-out calibration set distinct from training and final evaluation. Temperature scaling fits one scalar by minimising negative log-likelihood. Isotonic regression is a candidate when a flexible monotonic mapping improves held-out proper scoring and reliability, not when the empirical relationship is genuinely non-monotonic. The selected mapping becomes part of the versioned serving configuration and is checked again on untouched evaluation data.
Expected calibration error buckets predictions and measures the population-weighted confidence-versus-accuracy gap. The composite places pre-calibration ECE between 6% and 11% and post-scaling ECE between 1.5% and 3%. Those values are not guaranteed: binning can hide local errors, and temperature scaling may fail. Retain reliability plots, a proper scoring rule and cohort checks alongside ECE.
The red-teaming gate
No skill model reaches the production registry without passing an adversarial review, and this gate is treated with the same seriousness as a security penetration test, because functionally it is one. The red-teaming gate exists to answer a question the accuracy and calibration metrics cannot: what does this model do when someone, deliberately or accidentally, tries to make it misbehave.
The gate runs three categories of probe. The first is prompt injection: adversarial inputs crafted to make the skill model ignore its intended task and follow instructions embedded in the data it is meant to be classifying or extracting from, for instance a transaction narrative field containing text designed to look like a system instruction. A mature test suite for a single skill typically includes upward of four hundred crafted injection attempts covering a dozen known technique families, refreshed periodically as new families are documented across the industry.
The second category is data exfiltration: probes designed to see whether the model can be coaxed into reproducing memorised training examples verbatim, which would be a serious problem if any PII scrubbing had failed upstream, or into leaking details about its own system prompt or configuration that should not be user-visible. The third category is policy violation: inputs designed to push the model into an output that breaches a defined business rule, for instance approving a transaction type it is explicitly scoped never to approve regardless of confidence.
In the composite policy, a skill must clear each category floor as well as 98% overall, with zero tolerance for confirmed disclosure of protected training data. A failure routes to diagnosis, not automatically to more fine-tuning. The remedy may be data removal, a deterministic output filter, permission reduction, architecture change or adversarial training. Confidence scaling does not repair a security boundary.
The gate should include challenge independent of the team building the skill. Builders naturally share the assumptions that shaped the training set and may miss a failure family without any bad intent. A central probe library, independent case authors and documented adjudication reduce that blind spot. Organisational separation should be proportional to consequence rather than asserted as a substitute for test quality.
Failure modes
Three failure modes recur often enough across this pipeline that they are worth naming explicitly, because each one is invisible in the metric you would naturally check first and only becomes obvious once you know to look for it.
Distillation collapse
The symptom is a student model that scores well on held-out accuracy during training but performs noticeably worse in production than the offline numbers predicted, particularly on cases requiring any genuine reasoning rather than pattern matching against surface features. What has happened is that the student has learned to mimic the teacher's stylistic patterns, its phrasing, its confidence markers, the shape of its reasoning trace, without learning the underlying decision logic those patterns were supposed to represent. This is most visible when the distillation dataset is too narrow or repetitive in its surface structure, so the student finds a shortcut that reproduces the teacher's trajectories on the training distribution without generalising.
The fix is to widen trajectory diversity during teacher generation, deliberately varying phrasing and example ordering across the distillation pool, and to add a held-out generalisation test set built from genuinely different sources than the training pool, not just a random split of the same logs, so the collapse shows up before deployment rather than after.
Lora rank chosen too low
The symptom here is a fine-tuned model that improves on the base model but plateaus well below the accuracy the teacher demonstrated, and the gap does not close no matter how much additional training data is added. This is a strong signal the adapter simply does not have enough capacity to represent the decision boundary the skill requires, most commonly on skills with subtle, overlapping classes, precisely the kind of skill where a rank 8 adapter feels adequate on a first pass because it handles the easy majority of cases fine.
In the diagnostic example, a rank-8 adapter plateaus at 84% macro F1 and a rank-32 candidate reaches 92.6% on the same split. This is an illustrative ablation, not evidence that increasing rank generally causes that gain. Repeat across seeds, check leakage and compare capacity, target modules and regularisation before selecting the adapter.
Calibration drift after adapter merge
This failure may not show up in raw accuracy. A router can admit too many low-quality cases, or escalate too many acceptable ones, after an adapter is promoted without re-estimating calibration. A small update may shift the score distribution; whether it does in a particular build is empirical. Treat the previous calibration parameter as invalid until the merged candidate demonstrates otherwise.
The fix is procedural rather than technical: calibration is re-run as a mandatory step on every adapter version, not just the first one, and the cascade router's thresholds are treated as invalid until a fresh calibration manifest is attached to the new model version in the registry.
Worked example: swift message classification
The worked payments-operations composite classifies inbound SWIFT MT and ISO 20022 messages into thirty-one internal handling categories. The label determines whether a message is processed automatically, routed to a specialist queue or flagged for sanctions review. Every count, price and performance value in this section is a scenario assumption that requires local validation.
The scenario starts with fourteen months of logs and 1.9 million raw message-category pairs. Grouped deduplication leaves 340,000 examples. It adds 6,200 expert-labelled cases concentrated in rare categories and tests a four-times sampling weight. The weight is a modelling choice to validate through cohort results, not proof that rare cases should always receive that multiplier.
The modelled contamination check finds 380 overlaps in a proposed 12,000-message evaluation set, or 3.17% before rounding to 3.2%. Those records and their related groups are removed before training. Privacy treatment replaces unnecessary account numbers, beneficiary names and free-text remittance details with typed placeholders while retaining approved structural features.
The scenario uses an internally hosted 40-billion-parameter teacher. A 200-case review of the first pass finds an ambiguous sanctions-adjacent rubric and prompts revision. Five teacher samples per example at temperature 0.6 remove 9.8% of the pool under the consistency rule, leaving about 306,000 examples after rounding and other curation exclusions.
The student is a seven-billion-parameter open-weight base with a rank-16 LoRA adapter targeting attention projections and a feed-forward projection. In the sizing model, the adapter has 41.2 million trainable parameters, about 0.59% of seven billion, and trains in two hours and forty minutes on one eight-GPU node. Hardware, precision and implementation can materially change that timing.
On the 8,000-message scenario calibration set, ECE moves from 9.1% to 2.1% after temperature scaling at 1.74. This makes the candidate eligible for cohort and risk-coverage review; one aggregate ECE does not by itself make an automatic-processing threshold trustworthy.
The scenario gate runs 460 injection, 220 exfiltration and 310 policy probes: 990 cases in total. Four failures imply 986 passes, or 99.6%. All four expose a non-personal reference code from context. After remediation, the candidate must rerun the whole suite; “100% on 990 cases” is a finite observed pass rate, not proof that exfiltration risk is zero.
The modelled production state assigns p95 latency of 340 milliseconds to the skill model and 1.9 seconds to the frontier route. At an assumed 42 million messages a month, the scenario compares 58,000 USD with 2,100 USD of monthly inference cost. That is a 96.4% reduction under the stated totals. A real comparison must use the same token basis and include hosting, utilisation, training, evaluation and fallback on both sides. Scenario accuracy is 96.4% versus 96.8%; equivalence still requires a predeclared margin and uncertainty interval.
| Metric | Base frontier model | Distilled skill model |
|---|---|---|
| Cost per 1K tokens | $0.0150 | $0.0006 |
| P95 latency | 1,900 ms | 340 ms |
| Task accuracy | 96.8% | 96.4% |
| Expected calibration error | 4.2%* | 2.1% |
*The frontier-route ECE is shown before institution-side post-processing. Post-hoc calibration may be possible without changing model internals if the service exposes a stable, suitable score; many generative APIs do not expose the class logits needed for the same method.
The foundry is a governed supply chain
A private model is not a weight file with a deployment endpoint. It is a lineage of source data, teacher behaviour, training code, adapters, evaluation evidence and runtime policy. Each link can change the model’s effective behaviour.
| Foundry asset | Minimum lineage | Release check | Owner |
|---|---|---|---|
| Source examples | Purpose, rights, sensitivity and cohort | Data-use approval | Data owner |
| Teacher traces | Teacher version, prompt and sampling policy | Quality and leakage sample | ML lead |
| Training recipe | Code, seed, base model and hyperparameters | Reproducibility run | Training platform |
| Adapter or weights | Hash, parent and licence | Integrity and compatibility | Model registry owner |
| Evaluation pack | Cases, oracles and exclusions | Independent challenge | Validation lead |
| Runtime package | Image, policy and endpoint identity | Deployment attestation | Platform owner |
Choose the adaptation method from the constraint
Distillation, supervised fine-tuning and LoRA solve different problems. The method should follow the target behaviour, update frequency and validation burden. A larger training run is not a stronger business case.
Stable task · narrow output
Use a small classifier or deterministic method when the output space is fixed and evidence is structured.
Stable task · generative output
Consider distillation or fine-tuning. Preserve source grounding and test omissions explicitly.
Changing task · narrow domain
Prefer retrieval and modular adapters. Keep knowledge outside the weights where updates matter.
Changing task · broad reasoning
Retain a stronger general model behind a governed route. Do not compress capability the task still needs.
| Choice | Main benefit | Main risk | Evidence before adoption |
|---|---|---|---|
| Prompt and retrieval | Fast change and visible knowledge | Runtime cost and context errors | Retrieval and task baseline |
| LoRA adapter | Small trainable surface and modularity | Base-adapter interaction | Merge, swap and regression tests |
| Full fine-tune | Greater behavioural change | Cost, forgetting and lineage burden | Broad regression suite |
| Distillation | Lower serving cost and latency | Lost tail capability and inherited errors | Risk-coverage comparison |
| Deterministic component | Predictable bounded behaviour | Limited coverage | Coverage and fallback analysis |
Promotion requires more than quality
A student should be allowed to abstain. Forced coverage hides the cases where distillation removed needed capability. Calibration must be re-estimated after adaptation. Red-team cases belong in the promotion gate, not a separate presentation. Rollback must include the model, adapter, prompt and routing policy. The simplest passing candidate should win. This keeps foundry economics tied to an observable outcome.
The original knowledge-distillation paper explains the teacher-student frame. LoRA shows low-rank adaptation of large language models. Guo et al.’s calibration study is relevant after any adaptation. NIST SP 800-218A extends secure-development practices to generative AI and foundation models. The OWASP Agentic Security Initiative adds current threat material for tool-using deployments.
Private means controlled placement, not automatic trust. A foundry earns trust through reproducible lineage, bounded data use, independent evaluation and a runtime that can fail safely.
Candidate promotion check
- Verify the base model hash.
- Verify the adapter hash.
- Verify the approved data manifest.
- Reproduce the training recipe.
- Compare against the simpler baseline.
- Inspect calibration by cohort.
- Run the privacy test pack.
- Run the security test pack.
- Test abstention and escalation.
- Test rollback as one package.
Promotion is a supply-chain decision as well as a modelling decision. The registry entry should make both aspects visible.
Notes for practitioners
If you are building a first skill model foundry rather than reading about someone else's, a few habits will save more time than any single technical choice discussed above.
Treat the dataset manifest as a first-class artefact from day one, not something added once an auditor asks for it. Every dataset hash, teacher version, and calibration temperature should be recorded automatically as part of the pipeline, because reconstructing that lineage after the fact, once you have three or four skill models in production and a model validation review scheduled, is far more expensive than logging it as you go.
Budget the first skills separately from the mature-run scenario. The initial releases carry network approval, probe-library and calibration-tooling work that later candidates reuse. Three weeks and nine-to-fourteen working days are planning assumptions in this composite, not delivery commitments.
Never let the team that builds a skill also grade its red-team suite. This is the single governance decision I would fight hardest to keep if I could keep only one, because every other control in this pipeline can be partially recovered after the fact through additional testing, but a compromised red-team result discovered after a production incident cannot be undone.
Re-estimate calibration on every adapter version, including apparently minor updates. Do not promote the old calibration parameter on intuition; require the new build to demonstrate that its scores still support the routing threshold.
Finally, resist the temptation to make rank selection a single fixed default across the fleet. A rank that works for a binary flag will underfit a nuanced multi-label skill, and testing two rank values before committing costs a few hours of compute against a training run that will otherwise need to be repeated after a production complaint tells you the model plateaued too early. The foundry's value is in its repeatability, not in forcing every skill through identical hyperparameters; the parts that must be identical are the gates, not the model itself.