AI Research Library

Private, Edge and Federated Inference

Rajesh Ranjan Mahapatra

A mechanism-first guide to placing machine inference across devices, edge systems and cloud services without confusing physical proximity with privacy.

© 2026 Rajesh Ranjan Mahapatra. 32 min read

Private, Edge and Federated Inference

Computation should move only after a system names what would cross each boundary and proves that the chosen placement clears privacy, quality, latency, energy and availability gates.

TLDR

  1. Running a model near the data can reduce transfers, but physical proximity is not a privacy guarantee. Privacy depends on an explicit adversary, disclosure and retention model.
  2. A split is defensible only after measuring each possible boundary: local compute, transmitted bytes, network delay, device energy, task quality and the information recoverable from intermediate representations.
  3. Use hard gates for prohibited disclosures and minimum service levels, then compare the remaining device, edge and cloud plans on a Pareto frontier rather than one invented “privacy score”.
  4. Encrypted transport, trusted hardware, cryptographic inference, differential privacy and federated collaboration protect different assets. None repairs a compromised endpoint or careless telemetry by itself.
  5. The architecture decision changes from “local or cloud?” to “which information may leave, under which protection, for what gain, and what happens when the network or device fails?”

Two phones photograph the same identity document. Phone A runs its classifier locally, then uploads an embedding, a crash snapshot and a diagnostic trace to three separate services. Phone B performs deterministic redaction on the device, sends only approved fields to a controlled remote service, disables content logging and deletes transient buffers after a verified result. Which phone provides the more private inference path?

The words local and remote do not answer. Phone A keeps the main neural computation on the handset yet creates several disclosures. Phone B crosses a network boundary but may expose less information to fewer principals. Even that comparison remains incomplete until the adversary is named. A malicious application on the phone, a curious service operator, a network observer, a model thief and an authorised analyst abusing retained logs see different attack surfaces.

Compute placement is therefore an information-boundary problem before it is an infrastructure preference. Device, edge and cloud describe where instructions execute. Privacy describes what a specified observer can learn or cause. Federated learning describes how training updates are combined. Split inference describes one forward pass divided across tiers. Emerging uses of federated inference describe collaborative serving, but the term is not yet governed by one settled mechanism.

The practical consequence is demanding but useful. First record every representation that could move or persist. Next reject plans that violate disclosure, quality, latency, energy, availability or recovery requirements. Only then optimise the feasible plans. A neat topology diagram cannot substitute for that evidence.

Part I

Topology is not privacy

Edge computing arose from a real systems pressure. Sensors, phones, vehicles and industrial equipment produce data far from central data centres. Sending every byte to a distant service consumes bandwidth, adds variable delay and makes the application depend on a wide-area path. Edge-computing accounts therefore emphasise proximity, responsiveness, bandwidth and local context.[1][2] These are valuable properties. They do not make every nearby processor trustworthy, every remote processor unsafe or every local result private.

Consider a smart speaker that performs wake-word detection locally. The design avoids continuously streaming ambient audio before a wake event. That is a meaningful reduction in raw-data movement. Yet the device could still retain audio after the event, transmit a transcript, expose speaker embeddings, upload failure samples or reveal household routines through timing metadata. The local detector changes the disclosure surface; it does not close it.

This is why privacy should be framed through effects on people and data processing, rather than through one favoured infrastructure label. The NIST Privacy Framework similarly treats privacy risk as potential problems arising from systems that process data across their life cycle.[20] Its framework is voluntary and broad; it does not choose an inference topology. It does reinforce the right starting point: identify the processing, actors and consequences before selecting controls.

Four different placements

All-local inference keeps the forward pass on the originating device. It can work without a network and can prevent an operator from seeing query content, provided the application, operating system, model package, logs and update channel honour that boundary. It also moves model weights into a physically exposed environment and inherits the device's memory, thermal and energy limits.

Remote inference sends an input or derived request to an edge or cloud service. It can use larger models, pooled accelerators and centrally controlled releases. The price is a disclosure to the serving boundary, network dependence and a larger operational estate. Encryption in transit protects the bytes from ordinary path observers, not from the endpoint that decrypts them for computation.

Split or collaborative inference executes an early segment locally and later layers remotely. Neurosurgeon formulated the split as a latency and energy optimisation over mobile and cloud resources, while Edgent combined device-edge partitioning with early exits.[4][5] A split can reduce local computation or raw-input movement. It also creates a new transmitted object: the intermediate activation.

Federated or multi-owner inference lets several devices, models or organisations contribute to one query-time result. The participants may exchange activations, encrypted shares, votes, logits, proofs or carefully limited requests. This is an inference-time protocol. It should not be confused with federated learning, where clients compute training updates from local data and an aggregator combines those updates into a model.

Data gravity and compute placement across device, edge and cloud A large sensitive data mass sits on the device. Smaller approved representations may move to an edge service and compact results may move to the cloud. Compute capacity rises towards the cloud while immediacy and local context are strongest on the device. Move the smallest authorised representation, not the largest available one Device raw sensor or document strong context, limited budget Edge approved activation nearby pooled compute Cloud compact result or case state deep capacity, wider estate 96 KB bottleneck 8 KB typed result raw offload is a different disclosure, even when it is faster more immediacy and local context more pooled capacity and central control
Figure 1. Data gravity should pull early processing towards the source, but not dictate the whole path.The drawing separates raw data, a protected intermediate representation and a compact result. A real design must measure each object and authorise each transfer independently. Conceptual geometry. Circle sizes and arrow lengths are illustrative, not measurements.
Thought experiment 1: move the labels, keep the disclosures

Take a system that sends an image embedding to a service, retains it for seven days and exposes it to the same operator. In World A the service rack is called “edge” and sits five kilometres away. In World B it is called “cloud” and sits five hundred kilometres away. Keep encryption, identity, retention, operator access, model and output unchanged. The network delay may change. The operator's access to the embedding does not.

Now reverse the intervention. Keep the rack in the same building, but change the protocol so that it receives a cryptographic share that is insufficient on its own, logs no content and returns only a bounded result. The privacy properties can change while the location remains fixed. Geography can affect exposure, but it cannot stand in for an exposure model.

The disclosure ledger

A useful placement decision starts with six possible disclosures. The first is the raw input, including metadata that may be more identifying than the payload. The second is an intermediate representation, such as a feature tensor, embedding, token sequence or partial state. The third is the output, including probabilities, explanations and repeated answers. The fourth is the model, whose weights, architecture or prompts may be valuable intellectual property. The fifth is operating residue: traces, caches, crash dumps, queues and backups. The sixth is any update sent for adaptation or learning.

For each object, record who can observe it, whether observation is plaintext or protected, how long it persists, what other data can be joined to it and what effect the observer may cause. A tensor that looks anonymous in isolation may become identifying when paired with a stable account, timestamp or device fingerprint. A diagnostic sample retained “temporarily” may outlive the model release that created it.

Six disclosure lanes crossing device, edge and cloud trust boundaries Raw input remains on device in the preferred route. A protected activation crosses to edge, a typed output continues, model fragments exist on several tiers, and telemetry and updates follow separate governed paths. A placement has several information paths device principal edge principal cloud and operations raw inputactivationoutput modeltelemetryupdate retained locally protected, tested typed, minimised different fragments content-free metrics only signed model release, not user data
Figure 2. “The data stays local” is incomplete unless every other lane is named.The preferred path shown here keeps raw input on the device, moves a tested activation to the edge, limits the output and governs telemetry and updates separately. A different threat model may require a different route. Conceptual disclosure ledger. It is an architecture prompt, not a claim that the illustrated controls are sufficient.
Part II

Every layer is a possible border

A neural network is usually drawn as one model, but a placement engine sees a sequence of transformations. After each stage lies a possible cut. The cut determines how much work remains local, which representation crosses the network and what the remote side can observe. It may also determine which weights are exposed on a customer-owned device and whether the system can continue during an outage.

The tempting rule is to push the cut “deeper” until the representation becomes small and abstract. That rule fails twice. First, activation volume is not monotonic. Early convolutions can expand channels faster than spatial downsampling shrinks them, so an intermediate tensor may exceed the input. Second, abstraction is not erasure. Mahendran and Vedaldi reconstructed photographically recognisable content from several studied CNN representations.[7] Later work has attacked collaborative inference directly by reconstructing unseen inputs from intermediate features.[10]

A split index is therefore neither a byte count nor a privacy level. Each candidate cut needs three measurements: the cost of producing the representation, the cost of moving it and an attack-specific test of what it reveals.

The cost of a split

Suppose a model has layers 1 through L, and layers up to k run on the device. A first latency estimate is:

latency(k) = Cdevice(1:k) + RTT + 8 × Sk / B + Qremote + Cremote(k+1:L)

Cdevice is measured device compute time for the local segment. Sk is the transmitted representation in kilobytes. B is effective bandwidth in megabits per second, so the factor eight converts bytes to bits under the stated decimal units. RTT is round-trip delay, Qremote is queueing and Cremote is remote compute. Protocol setup, serialisation, radio wake-up and retries should be added when material.

Changing k moves several terms at once. A later cut usually adds local compute and device energy. It may reduce remote compute, but it may increase or decrease transmitted bytes. It can change quality if the boundary quantises, compresses, noises or exits early. The same plan may perform differently under a warm Wi-Fi connection, a congested mobile path, a thermally throttled phone and a low-battery policy.

Energy deserves its own measurement rather than inference from latency. Local acceleration consumes processor and memory energy; transmission consumes radio energy and may keep a modem in a high-power state. MLPerf Tiny made accuracy, latency and energy distinct benchmark dimensions for constrained inference.[6] The relevant denominator is also task-specific: millijoules per accepted inference, not merely instantaneous power or operations per second.

Model design is another candidate before partitioning. MobileNets made width and input resolution explicit levers for trading computation against task performance on mobile vision workloads.[3] Quantisation, pruning, distillation and smaller task models can similarly change the feasible set. A split optimiser that compares only one oversized network across locations may “discover” a network dependency that disappears when the local baseline is properly sized.

Synthetic stage profile showing local compute, activation size and disclosure risk Six model stages have different local compute times, output sizes and disclosure risks. The feature encoder output expands to 900 kilobytes and remains revealing. The private bottleneck reduces the output to 96 kilobytes and risk to 0.22. Depth does not make cost or disclosure fall smoothly 0 25% 50% 75% 100% decoderedactfeature bottleneckheadformat candidate edge cut device compute, scaled to 48 ms maximum output size, 1,800 KB maximum risk, 0 to 1
Figure 3. The useful boundary appears only after profiling three different curves.The feature encoder is the most expensive local stage and emits a 900 KB activation, larger than the redacted input. The next stage creates a 96 KB bottleneck with synthetic disclosure risk 0.22. Synthetic model profile used by the executable lab. Compute is normalised to the largest stage; output size to 1,800 KB; risk is an illustrative attack score, not a universal privacy metric.
Thought experiment 2: hold the split layer fixed

Build two networks with the same number of layers and cut both after layer three. Network A preserves a near-identity residual path and emits a high-resolution feature map. Network B passes through a narrow, task-specific bottleneck trained against a reconstruction and sensitive-attribute attacker. The layer number, device and server remain unchanged. Only the representation changes.

If an adversary reconstructs faces from A but not B, the privacy difference belongs to representation and attack resistance, not depth. If B also loses the feature needed for the authorised task, its lower leakage is not a free improvement. Privacy Partitioning and DeepObfuscator exemplify this utility-adversary trade by explicitly training or evaluating intermediate representations against private-attribute and reconstruction objectives.[8][9]

Measure a boundary as an attack surface

“The tensor is unintelligible to a person” is not a test. A serious boundary assessment gives the attacker the access expected in operation: observed activations, auxiliary public data, knowledge of architecture, perhaps query access and perhaps a related training distribution. It then measures reconstruction, attribute inference, linkage or membership success against a baseline. The result is scoped to that attack, dataset and model version.

Protection can fail after a model change even when the cut index is stable. Fine-tuning may encode a sensitive attribute more strongly. Quantisation may remove useful task information before it removes private information. A compression layer can reduce bytes without reducing recoverability. The boundary test must therefore travel with the model release, rather than surviving as a one-time architecture assertion.

Worked example 1: a minimal partitioning lab

The accompanying laboratory models a six-stage evidence classifier. Raw input begins on the device. The full model reaches synthetic task quality 0.944 but consumes 163 milliseconds and 81 millijoules when run entirely locally. A private bottleneck after stage four emits 96 KB with attack-risk score 0.22. Sending that representation to a nearby edge tier yields 141.6 milliseconds and 64.2 millijoules under the stated network profile.

The policy first applies hard gates: quality at least 0.940, latency no more than 155 milliseconds, device energy no more than 70 millijoules, no raw crossing and activation risk no more than 0.30. The full local plan fails latency and energy. Earlier splits fail disclosure or latency because they move larger, more revealing activations. The split after the private bottleneck is the only Pareto-feasible full-model plan in this synthetic case.

The lab also contains a serious negative control. A compact model reaches quality 0.933, 62 milliseconds and 35 millijoules entirely on the device. Under the strict quality gate it is correctly rejected. Lower the required quality to 0.930, without changing any system measurement, and the compact local baseline wins. When a simple local model is good enough, network partitioning has not earned its complexity.

Measured synthetic results for full local, protected split and compact local plans The full local model fails latency and energy gates. The protected edge split passes the strict quality policy. The compact local model fails only strict quality and becomes the selected negative control when the quality floor is lowered. Hard gates change the winner 050 100150200 latency, ms 155 ms gate 70 mJ gate full local protected split compact local .944 .944 .933 device energy, mJ 050100 passes strict policy passes only 0.930 quality floor fails latency and energy
Figure 4. The strict policy justifies a protected split; the relaxed policy activates the local negative control.Circle labels show synthetic task quality. The red lines are hard service gates. The compact model is not “worse” in the abstract; it is insufficient only when the required quality is 0.940. Measured output from inference_partition_lab.py. Synthetic stage, link and risk values; deterministic assertions verify both cases.

The core enumerator is deliberately small. It accepts only monotonic device-to-edge-to-cloud placements, calculates transfer cost at every boundary and applies policy gates before selection. The abridged excerpt below exposes the gate order; the linked file is the runnable artefact:

def evaluate(model, placement, policy):
    latency = sum(stage.latency_ms[tier]
                  for stage, tier in zip(model.stages, placement))
    energy = sum(stage.device_energy_mj
                 for stage, tier in zip(model.stages, placement)
                 if tier == "device")
    raw_crosses, risks = False, []

    for boundary in boundaries(placement):
        link = LINKS[boundary.source, boundary.target]
        kb, risk, raw = payload_before(model, boundary.index)
        latency += link.rtt_ms + link.queue_ms + kb * 8 / link.bandwidth_mbps
        energy += device_radio_energy(boundary, kb)
        risks.append(risk)
        raw_crosses |= raw

    failed = []
    if model.quality < policy.min_quality: failed.append("quality")
    if latency > policy.max_latency_ms: failed.append("latency")
    if energy > policy.max_device_energy_mj: failed.append("energy")
    if raw_crosses: failed.append("raw-crossing")
    if max(risks, default=0) > policy.max_activation_risk:
        failed.append("activation-risk")
    return Result(feasible=not failed, failed_gates=failed, ...)

Inspect the implementation in this paper. It uses only the Python standard library, prints both policy outcomes, emits JSON on request and asserts that the positive case and negative control select the expected plans.

Assumptions and limits of the executable lab

The values are synthetic. The “activation risk” number stands for a bounded attack evaluation supplied by a real assurance process; the script does not pretend to derive privacy from tensor size. It models one user-privacy threat and does not score model theft, compromised endpoints, side channels, physical attacks, malicious collaborators or output extraction.

A production evaluator should replay measured stage traces from target hardware, network distributions rather than one bandwidth value, thermal and battery states, cold starts, retries, concurrent load, model-version-specific leakage tests and recovery behaviour. It should preserve the vector of results instead of hiding incompatible harms inside one weighted score.

Part III
The argument in one passChart · flow
flowchart LR
  A["Topology is not privacy"] --> B["Every layer is a possible border"]
  B --> C["Privacy mechanisms protect<br/>different things"]
  C --> D["Apply hard gates before<br/>optimisation"]
  D -. "re-test" .-> A
  classDef first fill:#0a1424,stroke:#087A98,stroke-width:2px,color:#e6f3ff
  classDef step fill:#0a1424,stroke:#3f7da0,stroke-width:1px,color:#e6f3ff
  class A first
  class B,C,D step
A structural route through Private, Edge and Federated Inference, from Topology is not privacy to Move the border only after naming the…. Reading use: follow the relation, then test it against the surrounding evidence and limits.

Privacy mechanisms protect different things

Once a placement exposes an object to another principal, the next question is what that principal may learn or change. The answer cannot be “we use privacy technology”. Each mechanism assumes an adversary and provides a bounded protection. Combining mechanisms can strengthen a design, but only when their assumptions compose.

Transport encryption protects the path

TLS or an equivalent protected channel prevents ordinary network observers from reading or modifying traffic between authenticated endpoints. It is essential for remote inference. It does not conceal plaintext from the client process before encryption, from the serving process after decryption, from content logs or from an authorised operator with service access. Calling a conventional HTTPS inference call “private inference” confuses transport confidentiality with endpoint confidentiality.

Metadata also remains. Destination, timing, volume and recurrence can reveal routines or sensitive events even when payloads are encrypted. Padding, batching, relays and local suppression can reduce some metadata leakage, but each adds latency, cost or complexity. The design must state whether metadata is in scope.

Trusted execution changes the server boundary

A trusted execution environment aims to isolate code and data from the surrounding host, often with remote attestation so a client can verify the measured workload before releasing secrets. This can reduce trust in the operating system or cloud administrator. It transfers trust to hardware, firmware, attestation, enclave code, key provisioning and side-channel resistance. It also constrains memory and observability.

Slalom demonstrated a useful form of partitioning inside this mechanism: keep sensitive control inside a trusted environment while outsourcing expensive linear operations to an untrusted accelerator, with privacy and verification measures around the delegated work.[13] The paper establishes a specific protocol and performance result on studied networks. It does not make all accelerator delegation confidential, or all trusted hardware immune to leakage.

Cryptographic inference hides values through computation

Homomorphic encryption permits selected operations on ciphertexts. Secure multi-party computation lets parties jointly compute a function while limiting what each party learns beyond the permitted output. CryptoNets showed that neural inference over encrypted inputs was possible on MNIST; Gazelle combined homomorphic linear algebra with two-party techniques for non-linear layers.[11][12] These works are important demonstrations, not evidence that any current model can be wrapped with negligible cost.

The architecture, activation functions, numeric precision, communication rounds, batching and adversary model all matter. Some protocols protect client inputs but reveal model architecture. Others aim to protect both input and weights. Semi-honest and malicious security are different claims. A secure protocol can still release an over-informative output or be embedded in a service that logs plaintext before encryption.

Differential privacy limits an observable distribution

Differential privacy is not a synonym for adding noise. It bounds how much a specified output distribution changes when one protected unit is added or removed, under a stated privacy budget and composition rule. The protected unit could be a person, record, event, device or query. These choices change the meaning of the guarantee.

In training, differential privacy can constrain what model parameters reveal about individual examples. At inference, noise may be applied to activations, outputs or aggregate statistics, but the guarantee must be derived for that release mechanism. Repeated queries consume privacy or create averaging attacks unless the protocol accounts for them. Noise that defeats one attribute classifier does not automatically establish differential privacy.

Output and operating controls remain necessary

Even a perfect confidential-computing path can expose information through its permitted output. Membership inference asks whether a record was in the training set by exploiting differences in model behaviour; Shokri and colleagues demonstrated the risk in black-box settings.[17] Prediction interfaces can also support model extraction. Controls therefore include output minimisation, calibration, rate limits, query anomaly detection, purpose binding and refusal of requests that are too revealing.

Logging is a separate release. A trace may contain prompts, tensors, identifiers, confidence scores or decrypted errors. Privacy-preserving computation followed by plaintext debug retention merely moves the breach point. Telemetry should default to typed operational events, with content capture treated as an exceptional, consented and time-bounded workflow.

Protection mechanisms aligned to adversaries and residual risks Five concentric and intersecting protection lenses show that transport security, trusted execution, cryptographic inference, differential privacy and output controls address different observers. A compromised endpoint remains outside their shared protection. No single privacy mechanism covers the whole inference path transport security path observer, tampering trusted execution host operator, host software cryptographic inference defined parties and values differential privacy bounded influence on a release output and query controls endpoint compromise before protection: a malicious client reads the input after decryption: a malicious display reads the result
Figure 5. Protection is a coverage map, not a badge.The lenses overlap because real systems compose mechanisms. The central endpoint problem remains: data exists in usable form somewhere, and compromise at that point can defeat protections around transit or outsourced computation. Conceptual threat-mechanism map. Exact coverage depends on protocol, implementation and adversary assumptions.
Choose a mechanism by the observer it constrains
MechanismPrimary protectionDoes not by itself solveEvidence required
Protected transportConfidentiality and integrity between authenticated endpoints.Endpoint access, content logging, over-informative outputs or compromised clients.Protocol configuration, endpoint identity, key lifecycle and traffic tests.
Trusted executionIsolation from specified host software and operators, with attestation where used.Vulnerable enclave code, side channels, hardware assumptions, input or output misuse.Attestation policy, TCB inventory, patch state, side-channel assessment and fail-closed tests.
HE or MPCValues hidden from defined computational parties under a cryptographic model.Leakage through outputs, client compromise, wrong circuit, protocol denial of service.Security model, implementation review, numeric equivalence, latency and communication profile.
Differential privacyBounded influence of a protected unit on a specified release.Arbitrary secrecy, compromised endpoints, unlimited unaccounted composition.Neighbour relation, epsilon and delta, accountant, clipping, utility and repeated-release tests.
Output controlsReduced extraction, membership or inference surface at the interface.Internal access, training-data defects, side channels or stolen weights.Query threat model, rate policy, output ablations, attack simulation and incident telemetry.

Federated learning is not federated inference

Federated learning changes where training data is processed. In the canonical FedAvg pattern, clients compute local updates and a coordinator aggregates them to improve shared model parameters.[14] Secure aggregation can prevent the server from seeing each participant's individual update while still recovering an aggregate.[15] These mechanisms do not make the resulting query-time service local, and federated training does not automatically prevent update leakage. Deep Leakage from Gradients showed that shared gradients can reveal training examples under studied conditions.[16]

Federated inference instead concerns one or more predictions. Recent work uses the label in at least two ways. A SIGCOMM 2025 poster describes heterogeneous edge devices collaboratively executing a partitioned model, with capability-aware cuts and differential-privacy noise on activations.[18] A 2026 preprint frames independently trained, privately owned models collaborating at inference time without sharing their data or parameters, and explicitly notes that a unified abstraction is still lacking.[19]

The term is emerging, so an architecture must define it operationally. State the participants, object exchanged, aggregation rule, privacy mechanism, failure semantics, incentive assumptions and authority to return a result. Otherwise “federated” imports the positive connotation of data locality without specifying what the inference protocol actually reveals.

Training-time federation and query-time collaboration answer different questions
DimensionFederated learningFederated inference
WhenDuring optimisation or adaptation.During one prediction or decision-support request.
Typical object exchangedGradients, parameter deltas, sufficient statistics or secure shares.Activations, logits, votes, encrypted shares, proofs, sub-queries or partial outputs.
Persistent resultUpdated model parameters or training state.Prediction, evidence object or bounded query-time state.
Main systems constraintClient sampling, non-IID data, update communication, convergence and poisoning.End-to-end latency, availability, correlated errors, input privacy and output authority.
Privacy questionWhat do updates or final weights reveal about local training data?What does each collaborator learn about this query, other models and the returned result?

The explicit failure boundary

This paper's placement method stops helping when the trusted endpoint itself is under adversarial control. A rooted phone can read raw input before a local model transforms it. A malicious display process can capture the result after decryption. Physical probes may recover weights or keys. Side channels can violate an otherwise correct cryptographic or enclave model. A tampered update can replace the protected bottleneck with an exfiltration path.

The method also fails when leakage cannot be tested against a meaningful adversary, or when the task has no stable quality and consequence threshold. In those cases a numeric partition optimiser creates false precision. The appropriate decision may be to avoid remote inference, avoid retaining the input, narrow the task, require human handling or refuse automation. Placement optimises within a permitted system boundary; it cannot make an impermissible purpose acceptable.

Worked example 2: card-dispute evidence intake

Consider a synthetic mobile journey in which a customer photographs a receipt and selects a disputed transaction. The goal is to prepare an evidence packet, not to decide the dispute. Images may contain names, addresses, partial card details and unrelated purchases. Devices vary widely. Network quality may collapse halfway through an upload.

On the phone, deterministic code checks file type, orientation and size. Optical extraction and a small model propose text regions. A local redaction stage removes fields that the case policy does not permit to leave the device. A private bottleneck converts approved regions into a compact representation and attaches a manifest naming the source image, transformations, model version and redaction confidence. Low-confidence redaction causes a rescan or human review rather than remote escalation of the raw image.

A regional edge service receives the representation under a scoped workload identity. A larger model proposes merchant, amount, date and evidence-category fields. Deterministic validators check formats and reconcile extracted values with the selected transaction. The service returns typed fields plus evidence references, not a free-form conclusion. The central case system records the proposal, policy version and effect receipt. A person remains accountable for any consequential disposition.

This arrangement is not automatically superior to an all-local model. It is justified only if the larger remote stage earns enough quality on difficult documents and the protected boundary survives attack testing. The compact local baseline remains available during outage and for documents it handles confidently. The remote path is an escalation, not the default definition of intelligence.

Governed card-dispute evidence path from phone to edge to case system The phone validates, extracts, redacts and creates a private bottleneck. An edge service proposes typed fields. A central case system validates and records evidence. Low redaction confidence, network failure and policy mismatch route to safe recovery paths. Escalate a bounded representation, not the whole case customer phoneregional edgecase and review validate file and source extract and redact private bottleneckplus provenance manifest model proposes typedevidence fields format and consistencychecks policy validation anddecision receipt human disposition forconsequential outcome 96 KB approved object typed fields low redaction confidence: rescan or local review network failure: checkpoint, do not duplicate upload policy mismatch: hold, do not auto-decide authorised forward pathrecovery pathworked scenario, not a deployment claim
Figure 6. The production-shaped path keeps inference authority narrower than case authority.The edge model may propose evidence fields. It cannot decide the dispute. Redaction uncertainty, network interruption and policy mismatch have explicit recovery paths rather than silent raw-data escalation or duplicate effects. Synthetic composite scenario. No client deployment or measured performance is implied.
Part IV

Apply hard gates before optimisation

An architecture team often asks for a single score to rank device, edge and cloud plans. That is attractive and dangerous. A weighted sum can trade a prohibited disclosure against a few milliseconds of latency, or offset unacceptable quality with a lower cloud bill. Some requirements are not preferences. They are admission conditions.

Let a placement plan p have task quality q, tail latency l, device energy e, availability a and a disclosure vector d. Define the feasible set before choosing a winner:

F = { p : q(p) ≥ qmin, l95(p) ≤ lmax, e(p) ≤ emax, a(p) ≥ amin, dprohibited(p) = 0, dattack(p) ≤ dmax }

qmin is the minimum quality supported by task evidence. l95 is a tail-latency requirement rather than a favourable average. emax reflects the device and usage pattern. amin includes the network, remote service and local fallback. The disclosure terms include zero-tolerance events, such as raw biometric material crossing an unapproved boundary, and measured attack thresholds for representations that are allowed to move.

Only plans in F should enter cost optimisation. Within that set, preserve a Pareto frontier when no plan is better on every relevant dimension. A lower-energy plan may have higher latency. A more accurate plan may require a protected remote tier. The decision maker can then see the exchange instead of inheriting a hidden weight chosen by the optimiser author.

Synthetic Pareto frontier after disclosure and service gates Many placement plans are rejected before optimisation. Under the relaxed quality policy, compact local, compact split and full protected split remain Pareto options. Under the strict quality policy only the full protected split remains feasible. Filter first, then expose the trade-off 050100150200 p95 latency, ms device energy, mJ compact local compact edge split full protected split strict quality gate removes compact plans rejected plan0.933 quality0.944 qualitysynthetic measured lab
Figure 7. A Pareto frontier is meaningful only after impermissible plans disappear.The relaxed policy retains three non-dominated plans because each buys a different combination of quality, latency, energy and activation risk. Raising the quality floor to 0.940 leaves only the full protected split. Synthetic results from the executable lab. Grey points illustrate rejected candidates; labelled feasible points use measured script output.

Availability changes the privacy decision

Remote compute introduces a failure mode that all-local inference may avoid. A responsible design specifies what happens when the edge is unreachable, slow or unable to attest. Retrying may duplicate cost or disclosure. Falling back to raw cloud upload may violate the reason for the local boundary. Returning an unverified result may convert an availability incident into a correctness incident.

Possible responses include a smaller local model, a delayed queue with encrypted state, a request for manual completion or a clear statement that the function is temporarily unavailable. The fallback must meet its own quality and disclosure conditions. A “graceful degradation” that silently widens data access is not graceful.

Device heterogeneity matters as much as network heterogeneity. A cut that works on the newest phone may exhaust memory on an older one. Thermal throttling can turn a passing latency profile into a failure after several requests. Measure by supported device class and workload distribution, then route only among plans qualified for the current context.

The inference placement record

A concise decision record keeps the topology tied to its evidence. It should be versioned with the model, application, privacy tests and release manifest. The following questions are deliberately ordered. Starting with a preferred vendor or tier reverses the logic.

Inference placement decision funnel A funnel begins with purpose and consequence, then identifies principals and disclosures, applies prohibited-flow and service gates, profiles candidate boundaries, attacks representations, tests recovery and finally selects a Pareto-feasible plan. Inference placement record: nine gates to a defensible cut 1. Purpose, user consequence and permitted outcome 2. Principals, assets, retention and joinability 3. Prohibited raw, activation, output or model flows 4. Minimum quality, p95 latency, energy and availability 5. Measured stage, payload and network profiles 6. Attack tests for every representation that crosses 7. Outage, stale model, retry and rollback tests 8. Pareto choice 9. Release dossier do not optimise a plan that failed an earlier gate
Figure 8. Placement is the last architectural choice, not the first.The funnel prevents a fast or fashionable topology from outrunning purpose, disclosure and service constraints. Release evidence includes the chosen plan, alternatives rejected, measurements, attacks, assumptions, owners and expiry triggers. Decision instrument. The sequence is practitioner synthesis derived from the paper's mechanism and source pack.
Minimum fields for an inference placement record
FieldQuestion to answerRelease evidence
Purpose and authorityWhat may the system infer, recommend or cause, and who remains accountable?Task statement, outcome contract, decision rights and prohibited uses.
Principals and assetsWho can observe raw input, activations, output, model, telemetry and updates?Data-flow inventory, identities, retention, access and deletion tests.
Candidate cutsWhich sequential or collaborative placements are technically possible?Stage graph, payload schemas, model fragments and network links.
Service profileWhat quality, p50 and p95 latency, energy, memory and availability does each plan achieve?Target-device traces, network distributions, load tests and confidence intervals.
Disclosure testsWhat can the stated adversary reconstruct, infer, link, extract or retain?Attack code, baselines, thresholds, datasets, model version and residual risk.
Protection contractWhich mechanism protects each crossing, under which assumptions?Protocol, cryptographic or attestation configuration, keys, TCB and verification.
RecoveryWhat happens on outage, timeout, duplicate, stale release, failed attestation or suspected leakage?Fallback behaviour, checkpoint, idempotency, rollback and incident exercise.
Decision and expiryWhy did this plan beat the negative control, and what change forces re-evaluation?Feasible frontier, chosen trade-off, owner, review trigger and withdrawal path.

Requalification triggers

A placement decision expires when any assumption that supports it changes. Triggers include a new model or fine-tune, a revised tokenizer or encoder, a different quantisation mode, new supported devices, altered retention, another telemetry provider, a changed edge region, a new attacker capability, a privacy-budget change or a shift in task consequence. A network optimisation can also be a privacy change when it moves the cut or changes the object transmitted.

Requalification should be proportional rather than ceremonial. A remote accelerator swap may require performance and attestation evidence but no new task-quality study if numerical equivalence is demonstrated. A new private bottleneck requires fresh utility and attack tests even when every API remains stable. A change that invalidates a hard gate should withdraw the route automatically, not wait for a periodic committee.

What would weaken the argument?

The thesis would weaken if physical placement reliably determined privacy across relevant adversaries without inspecting representations, outputs or operating residue. It would also weaken if intermediate leakage fell monotonically with depth across architectures and tasks, making the cut index a dependable proxy. Existing inversion and obfuscation results give reasons not to assume either claim, but they do not prove that every activation is sensitive.

A stronger practical challenge is simpler: perhaps a well-sized local model meets every material requirement for most requests. In that case the negative control wins and the edge path should disappear or become a narrow exception. Conversely, cryptographic or trusted execution may make remote computation feasible without a learned bottleneck. The paper does not privilege one mechanism. It privileges measured disclosure and service evidence over placement folklore.

The method also leaves open how to price harms that cannot be made commensurate. A privacy event, a missed urgent detection and a drained battery do not share a natural unit. Hard gates handle unacceptable regions, and a Pareto frontier exposes the remaining conflict. Accountable human judgement is still required to choose among feasible plans.

How the claim changes under pressureChart · state
stateDiagram-v2
  [*] --> Claim
  state "Four different placements" as Claim
  state "The disclosure ledger" as Test
  state "The cost of a split" as Revision
  Claim --> Test: expose mechanism
  Test --> Decision: evidence survives
  Test --> Revision: rival survives
  Revision --> Claim: narrow or rebuild
  Decision --> [*]
The revision loop begins with Four different placements and keeps counterexamples inside the reasoning path. Reading use: follow the relation, then test it against the surrounding evidence and limits.

Compact glossary

Edge inference
Inference executed on compute near the data source or user, usually to reduce wide-area latency, bandwidth or dependency. Nearness alone says nothing conclusive about trust.
Split inference
One model forward pass divided across two or more compute tiers, with an intermediate representation transmitted at each cut.
Federated learning
Training or adaptation in which participants process local data and contribute updates or statistics to a collaborative optimisation process.
Federated inference
An emerging label for query-time collaboration across devices or independently owned models. Its exact protocol must be defined rather than inferred from the name.
Disclosure vector
A record of which raw inputs, activations, outputs, weights, telemetry and updates each principal can observe, retain or join.
Private bottleneck
A task-specific representation designed and tested to preserve authorised utility while limiting a stated reconstruction, attribute or linkage attack.
Hard gate
A condition a plan must satisfy before optimisation, such as a prohibited raw-data flow, minimum quality or maximum tail latency.
Pareto frontier
The feasible plans for which no other plan is at least as good on every retained dimension and strictly better on one.
The evidence conversationChart · sequence
sequenceDiagram
  participant Q as Question
  participant M as Mechanism
  participant E as Evidence
  participant D as Decision
  Q->>M: Topology is not privacy
  M->>E: Four different placements
  E-->>M: compare rival explanations
  E->>D: Move the border only after naming the disclosure
  D-->>Q: revise scope or act
Question, mechanism, evidence and decision exchange the burdens developed in Private, Edge and Federated Inference. Reading use: follow the relation, then test it against the surrounding evidence and limits.

Move the border only after naming the disclosure

Private inference is sometimes presented as a race to fit a larger model on a smaller device. Edge inference is presented as a latency shortcut. Federated inference is presented as collaboration without centralisation. Each description catches part of the design space and hides the same decisive question: what information becomes available to which principal at each step?

A defensible answer begins with the objects, not the tiers. Raw inputs, activations, outputs, model fragments, telemetry and updates follow different paths. Each path needs an adversary, retention rule, protection mechanism and failure response. Candidate model cuts then face measured quality, latency, energy, availability and leakage tests on the hardware and network that will actually carry them.

The architecture decision changes accordingly. Do not ask whether the model should be local, at the edge or in the cloud as though one noun could settle privacy. Establish the prohibited flows and minimum service first. Put a compact local baseline beside every elaborate partition. Reject any plan that violates a gate. Preserve the Pareto frontier among those that remain, and record why one trade was accepted.

The executable result captures the discipline. A strict quality requirement can justify sending a protected bottleneck to the edge. The moment a compact local model meets the required quality, it becomes the better answer. Computation moves only when the evidence earns the boundary crossing.

References

  1. Primary survey and systems vision. Shi, W., Cao, J., Zhang, Q., Li, Y. and Xu, L. Edge Computing: Vision and Challenges. IEEE Internet of Things Journal, 2016.
  2. Primary perspective. Satyanarayanan, M. The Emergence of Edge Computing. Computer, 2017.
  3. Primary model paper. Howard, A. G. et al. MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications. 2017.
  4. Primary systems paper. Kang, Y. et al. Neurosurgeon: Collaborative Intelligence Between the Cloud and Mobile Edge. ASPLOS, 2017.
  5. Primary systems paper. Li, E., Zhou, Z. and Chen, X. Edge Intelligence: On-Demand Deep Learning Model Co-Inference with Device-Edge Synergy. 2018.
  6. Primary benchmark paper. Banbury, C. et al. MLPerf Tiny Benchmark. 2021.
  7. Primary representation study. Mahendran, A. and Vedaldi, A. Understanding Deep Image Representations by Inverting Them. CVPR, 2015.
  8. Primary privacy-partitioning paper. Chi, J. et al. Privacy Partitioning: Protecting User Data During the Deep Learning Inference Phase. 2018.
  9. Primary defence paper. Li, A. et al. DeepObfuscator: Obfuscating Intermediate Representations with Privacy-Preserving Adversarial Learning on Smartphones. 2019.
  10. Primary attack paper. Yin, Y. et al. Ginver: Generative Model Inversion Attacks Against Collaborative Inference. The Web Conference, 2023.
  11. Seminal primary paper. Gilad-Bachrach, R. et al. CryptoNets: Applying Neural Networks to Encrypted Data with High Throughput and Accuracy. ICML, 2016.
  12. Primary secure-inference systems paper. Juvekar, C., Vaikuntanathan, V. and Chandrakasan, A. GAZELLE: A Low Latency Framework for Secure Neural Network Inference. USENIX Security, 2018.
  13. Primary trusted-execution paper. Tramèr, F. and Boneh, D. Slalom: Fast, Verifiable and Private Execution of Neural Networks in Trusted Hardware. ICLR, 2019.
  14. Seminal federated-learning paper. McMahan, H. B. et al. Communication-Efficient Learning of Deep Networks from Decentralized Data. AISTATS, 2017.
  15. Primary cryptographic protocol paper. Bonawitz, K. et al. Practical Secure Aggregation for Privacy-Preserving Machine Learning. CCS, 2017.
  16. Primary attack paper. Zhu, L., Liu, Z. and Han, S. Deep Leakage from Gradients. NeurIPS, 2019.
  17. Primary attack paper. Shokri, R., Stronati, M., Song, C. and Shmatikov, V. Membership Inference Attacks Against Machine Learning Models. IEEE Symposium on Security and Privacy, 2017.
  18. Recent primary poster paper. Fan, B., Su, X., Tarkoma, S. and Hui, P. Federated Inference: Towards Collaborative and Privacy-Preserving Inference over Edge Devices. ACM SIGCOMM Posters and Demos, 2025.
  19. Recent primary preprint and competing formulation. Seo, J., Catak, F. O., Rong, C. and Jang, J. Federated Inference: Toward Privacy-Preserving Collaborative and Incentivized Model Serving. 2026.
  20. Official risk framework. National Institute of Standards and Technology. NIST Privacy Framework.