TLDR
The strongest rival: broad exercises can become theatre
Executive briefThe model passed; the payment process failed
- The red-team boundary is the complete path from actor and evidence to authority, external effect, detection and remedy.
- Begin with an unacceptable business consequence, then traverse backwards to every plausible malicious and ordinary entry path.
- Test controls in combination because filters, fallbacks, approvals, retries and privacy safeguards can defeat one another.
- A finding closes only after detection, revocation, reconstruction, reconciliation, affected-population identification and remedy are proved.
- Passing scenarios supplies bounded release evidence, never a certificate, universal safety score or claim that untested trajectories are safe.
A payment agent rejected every direct request to change a beneficiary. Its base model had passed the provider safety evaluation, an internal jailbreak set and a prompt-injection benchmark. In the deployed process, however, an approved supplier email contained text that the retriever placed beside a real invoice. The agent treated a new account as supporting evidence, prepared the change, inherited a delegation token and sent the amended payment through a tool whose approval screen displayed only the amount. Every component performed an intended local function. The business process moved money to the wrong destination.
A model-only test would report a blocked malicious prompt. A penetration test might report no exploitable network path. A workflow review might see an approval. A control inventory would list retrieval filtering, identity, transaction limits, monitoring and reconciliation. None of those views alone asks whether a plausible sequence can convert untrusted content into an authorised external effect while hiding the decisive change from the reviewer.
The consequential unit is the operating trajectory: triggering actor, channel, retrieved evidence, prompt and model, deterministic rules, memory, identity, delegation, tool calls, queues, human handoffs, downstream systems, messages, telemetry, containment and recovery. Weakness often lives in the join between two individually acceptable parts. A fallback changes the tool schema. A retry repeats a side effect. A privacy transform removes the signal a fraud rule needs. A kill switch stops new planning while already queued actions continue.
The Consequence-Backwards Process Red Team starts from a named outcome the organisation refuses or strictly bounds: financial loss, unlawful treatment, unsafe action, disclosure, service interruption, corrupted evidence or an irreversible commitment. The team walks backwards through possible states and entry points, then exercises complete representative paths in an authorised environment. Its product is decision evidence about release, authority, containment, operating readiness and recovery.
A process-wide exercise can consume many disciplines, reward subjective stories and create a false impression of completeness. Teams may count colourful attacks, grade disputed findings and produce a report that no release gate can use. The answer is to join model benchmarks, penetration tests, control tests, table-top exercises and monitoring through named consequences and replayable trajectories while preserving each method’s boundary.
Seven-surface operating modelName what can carry the process towards harm
1. Consequence surface
Define the unacceptable state in business terms, not attack vocabulary. Name the affected interest, population, severity, exposure, speed, external commitment and residual harm after feasible remedy. “Prompt injection” is a mechanism. “A supplier payment reaches an unverified account and cannot be recovered before settlement” is a consequence that sets authority, test containment and release boundaries.
2. Entry and influence surface
Map direct users, malicious outsiders, legitimate insiders, upstream applications, documents, messages, web pages, retrieval corpora, peer agents and delayed memory. Include ordinary ambiguity, stale state and conflicting evidence. Authenticated origin establishes where content came from; it does not make every embedded instruction an authorised mandate.
3. Decision surface
Version the model, system instructions, retrieval assembly, memory, deterministic rules, thresholds, ranking, aggregation, abstention and fallback. Observe how uncertainty is represented and where one component’s output becomes another component’s fact. A harmful state may originate in a workflow threshold even when the model is blamed first.
4. Authority surface
Trace user and workload identity, delegation, consent, entitlement, approval, transaction limits, separation of duties, credential exchange and resource-side policy. Test the maximum action under compromise and pressure. A human click, valid token or authenticated peer does not establish that the purpose, recipient and current case are authorised.
5. Action surface
Exercise tools, APIs, queues, writes, messages, retries, idempotency, compensation and external destinations. Record what the receiving system actually committed. Timeouts and partial failures matter because the agent may not know whether an effect occurred. Technical rollback can leave money, messages or customer decisions unchanged.
6. Human and organisation surface
Test reviewer workload, expertise, interface evidence, escalation, incentives, local practice, workarounds, handoffs and unavailable owners. Oversight exists only when a qualified person has time, information, authority and an observable history of changing outcomes. Review that becomes rapid confirmation under load is a failed operating control.
7. Detection and recovery surface
Prove telemetry, evidence lineage, incident declaration, containment, credential revocation, state reconstruction, rollback, external reconciliation, affected-population identification, communication, remedy and regression. Measure where evidence is absent or controlled by the challenged component. The organisation must recover the business state, not merely restart the service.
The depth of each surface follows consequence and recoverability. A private, reversible drafting tool may need a narrow influence and output test. A payment or eligibility process needs realistic identities, tools, queues, reviewer load, counterparties and recovery evidence. Scope is explicit: named product version, population, environment, authority, providers, volumes and time window. Unknown scope never counts as tested scope.
| Field | Required evidence | Failure the field prevents |
|---|---|---|
| Actor and precondition | Identity, access, starting state, channel and required knowledge | An impossible attacker or an omitted legitimate insider |
| Test action and trajectory | Ordered state transitions, data, versions, decisions, approvals and tool effects | A clever prompt without a business path |
| Expected and observed control | Control owner, placement, expected response, actual response and collision | Claiming a listed control operated |
| Consequence and population | External effect, exposed cases, severity, time and residual harm | Counting attempts without decision relevance |
| Detection and recovery | Signal, latency, declaration, containment, reconstruction, reconciliation and remedy | Closing after payload blocking alone |
| Evidence and disposition | Immutable references, challenger, finding owner, deadline, retest and release effect | Unreproducible or self-graded closure |
Process Attack GraphStates and edges, not a catalogue of tricks
The graph’s nodes are operational states: evidence accepted, case assembled, identity exchanged, transaction prepared, approval recorded, action committed, signal observed, authority revoked and customer remedied. Edges are influence, inference, delegation, approval, action, observation and recovery. Each edge has an actor, precondition, version, policy, time and expected control. The graph records paths that are possible, paths that were exercised and paths that remain unknown.
Consequence-backwards construction begins with the harmful state and asks what immediately enabled it. A wrong payment requires an external commitment. That commitment required a tool call and receiving-system acceptance. The call required a credential, transaction parameters and policy decision. The parameters came from case evidence and model or rule output. The evidence entered through a user, email, document, upstream system or stored state. This reverse trace exposes missing control placement before the team writes attack payloads.
Forward execution then tests the path in an isolated, authorised environment with synthetic or tightly controlled data and counterparties. The plan defines stopping conditions, test accounts, rate and value ceilings, evidence capture, communication, cleanup and emergency authority. Destructive tests do not run in production merely because the path is commercially important. Where production behaviour cannot be replicated, the untested difference remains explicit release uncertainty.
Normal-but-stressful trajectories belong beside malicious ones. Ambiguity, a timeout after commitment, a sudden reviewer queue or a degraded-service fallback can reach the same harmful state without a hostile string. Coverage therefore crosses actors, states, environments, consequences and recovery conditions. It records sampled cells, omissions and invalidating changes. Ten near-identical prompts may cover less than one traced uncertain commitment with failed compensation.
The graph also forces false attribution to be corrected. If a duplicate refund follows deterministic retry logic after the model produced one valid request, the causal failure is action semantics and idempotency, not model intent. The model remains part of the trajectory, but the finding owner and repair sit elsewhere. A red team that preserves its initial story after contrary evidence turns challenge into theatre.
Evidence architectureKeep complementary tests distinct
NIST SP 800-115 provides enduring discipline for planning, authorising, executing and reporting technical security tests. NIST’s Generative AI Profile distinguishes model testing, field testing and red teaming, and stresses domain and sociocultural expertise. The AI RMF Playbook recommends adversarial or stress testing, documented conditions and results, contextual evaluation and exercises at a prescribed cadence. These sources support rigorous challenge without claiming that one method covers the whole system.
Empirical work sharpens the boundary. Lessons drawn from red teaming one hundred generative AI products emphasise the application context, the difference between red teaming and safety benchmarking, and the human element. AgentDojo provides realistic tasks, tools, untrusted data and adaptive prompt-injection attacks. A large public competition demonstrates policy violations across tool-using agent scenarios. These studies justify agent and system testing, but their published environments do not prove a particular enterprise process safe.
Critical scholarship adds an organisational warning. Work on macro versus micro red teaming argues for system-level, multifunctional challenge. Research on values, labour and harms calls attention to who defines failure, who performs disturbing work and which affected perspectives are absent. Third-party flaw-disclosure research shows why in-house evaluation needs clear reporting, triage, coordination and safe channels. Independent challenge is not performative distance; it is the ability to dispute scope, evidence, severity and closure without product-delivery pressure deciding the answer.
| Evidence stream | Question it can answer | Question it cannot answer alone |
|---|---|---|
| Model benchmark | How a defined model behaves on a specified case set and metric | Whether deployed identity, tools, people and recovery prevent business harm |
| Penetration test | Whether technical exposure can be discovered and exploited within scope | Whether ordinary ambiguity, queue pressure or control collision changes an outcome |
| Control test | Whether a named control operates under specified conditions | Whether several controls interact safely across a complete trajectory |
| Table-top exercise | Whether people understand roles, escalation and choices in a simulated event | Whether credentials revoke, queues stop, ledgers reconcile and customers are identified |
| Process red team | Whether representative adversarial and stressful paths reach or recover from named consequences | Whether every untested path is safe or future change preserves the result |
| Production monitoring | What signals and outcomes appear during actual operation | Which rare intolerable path should be deliberately allowed to occur |
Evidence streams meet through stable identifiers for process, version, scenario, business state, case, action receipt, finding and release. A red-team observation may defeat a release claim in the assurance case from Paper 106, test the action envelope from Paper 105, challenge the authority firewall from Paper 112 and the least-authority design from Paper 113. It does not replace those designs. It asks whether they survive composition and pressure.
Control Collision MatrixChallenge the seams between sensible safeguards
Controls are often reviewed one at a time, under the condition each designer expected. The matrix pairs control mechanisms and asks how one changes the facts, timing, authority or recovery assumptions of another. Privacy redaction can remove device or relationship information that a fraud rule relies on. A content filter can strip the adverse clause that a credit reviewer needs to see. An approval interface can hide the field that changed. A circuit breaker can create a queue that later drains without renewed authorisation.
Retry is a particularly revealing collision. A client times out and does not know whether the payment or refund committed. A generic reliability policy retries. If the receiving tool lacks an idempotency key tied to the business transaction, two locally rational behaviours create a duplicate external effect. Compensation is then a separate authorised action with its own failure modes. The red team verifies the receiving ledger, not the agent’s belief that the first call failed.
Fallbacks need independent admission. During an outage, a different model may interpret tool schemas differently, a secondary connector may expose broader methods, and an emergency credential may exceed the normal mandate. Availability improves while the authority envelope silently expands. The test changes model, prompt, tool, identity, traffic, queue and human state together, then checks which release claims and recovery procedures still hold.
Each matrix cell states a hypothesis, consequence, exercise, expected evidence and owner. Empty cells are not safe cells; they may be immaterial, out of scope or unexamined. Findings change design where possible: expose the decisive field in approval, bind idempotency at the receiving resource, drain queues into revalidation, separate emergency credentials, preserve fraud signals under governed use, or keep fallbacks advisory until their action path is independently proved.
| Control pair | Harmful interaction to test | Design response if observed |
|---|---|---|
| Content filtering × human review | Filtering removes adverse evidence while leaving a fluent summary | Show protected source evidence, transformation receipt and a visible omission state |
| Privacy minimisation × fraud detection | Redaction deletes linkage needed to identify coordinated behaviour | Move the minimum fraud computation behind a governed boundary and expose the result, not raw data |
| Timeout retry × non-idempotent tool | Uncertain commitment is repeated as a new business transaction | Use transaction-bound idempotency, receipt lookup and explicit uncertain state |
| Approval × queue pressure | Review becomes ceremonial and no longer changes outcomes | Cap queue, sample blind disagreements, route exceptions and suspend authority when capacity fails |
| Fallback × emergency privilege | Service continuity introduces a less-tested model and wider action rights | Admit the fallback separately and restrict it to a narrower safe mode |
| Kill switch × durable work | New planning stops while queued writes, memory jobs or messages continue | Revoke receiving credentials, cancel queues, mark state and reconcile every in-flight effect |
Human and organisational challengeTest whether the reviewer can still change the outcome
A process red team treats people as participants with expertise, constraints and rights. The plan defines domain knowledge, workload, evidence, authority, escalation and psychological safety. It protects testers who may encounter harmful material and includes affected perspectives where customers, workers or communities bear the consequence.
Meaningful oversight is tested through observable counterfactuals. Seed cases where the right action is to agree, disagree, narrow, escalate and stop. Vary queue length, deadline, confidence language, source conflict, seniority pressure and unavailable escalation. Record whether reviewers inspect decisive evidence, identify the changed field, use their authority and create a reason that another person can audit. A completed approval event without those properties is not proof of oversight.
Incentives and handoffs matter. Operations may optimise queue clearance while risk expects exception review; security may revoke one credential while an external processor accepts signed messages. The exercise assigns the unresolved business state. Findings retain evidence, consequence, scope, causal path, owner, deadline, interim constraint and retest. Material disputes go to a named independent authority, and external reports receive a defined disclosure path.
Recovery ProofAn exploit stopped is not a business state restored
The Recovery Proof begins before testing. For each intolerable or material consequence, the team defines the earliest observable signal, declaration authority, safe state, credentials and queues to contain, evidence needed to reconstruct the path, external systems to reconcile, people to identify, remedy authority and return conditions. The exercise introduces partial logs, delayed owners, disputed counterparties and failing compensation because ideal recovery is not representative evidence.
Detection is measured from the first adverse action, not from when an analyst notices an alert. The proof records which system observed the effect, whether the challenged agent could suppress that evidence, how the signal maps to a business case and when incident authority activates. High alert volume is not strong detection. A useful signal reaches a named owner soon enough to reduce the relevant consequence.
Containment revokes the action path, not merely the user session. Stop new planning, block resource credentials, cancel or quarantine queues, freeze risky destinations, preserve evidence and mark durable state. Peer agents, scheduled jobs, browser sessions, external messages and already issued tokens may outlive the orchestrator. The team enumerates each continuation path and verifies receiving-system enforcement.
Reconstruction joins decision-time model, prompt, retrieval, memory, rules, thresholds, identity, delegation, approval, tool request and external receipt. Reconciliation then compares internal intent with settled payments, delivered messages, changed records and downstream decisions. Software rollback can leave those effects intact. Compensation needs independent authority, idempotency and evidence; remedy addresses the affected person, not just the database.
A finding closes when the causal path is removed or bounded, dependent paths are assessed, recovery evidence passes, affected cases are handled and a regression case is tied to the release gate. Blocking the observed string is inadequate if a different document, normal ambiguity or longer sequence reaches the same state. Closure includes a retest trigger for model, prompt, retrieval, tool, authority, queue, human operating or recovery change.
| Proof link | Demonstration | Unacceptable substitute |
|---|---|---|
| Detect and declare | Independent signal reaches authorised command inside the consequence window | A dashboard metric with no case, threshold or owner |
| Contain and revoke | Tools, credentials, queues, peers and durable work cannot create new effects | Stopping the model process while external authority remains live |
| Reconstruct | Decision-time evidence, versions, identities, approvals and receipts join per case | Current logs or sampled prompts without action continuity |
| Reconcile | Internal records match receiving ledgers, destinations and customer state | Successful technical rollback |
| Identify and remedy | Affected population is bounded and correction, redress or communication completes | A generic incident notice with unknown exposure |
| Learn and retest | Causal repair, dependent-path analysis, regression and expiry enter release evidence | Blocking one payload or closing a ticket |
The Release Evidence Rule is therefore conjunctive: a consequential process may proceed only when representative end-to-end trajectories cover named consequences, failed controls have owners and deadlines, intolerable paths are blocked, residual exposure is accepted by the accountable business owner and recovery has been rehearsed with evidence. A successful exercise does not certify the system. It establishes a time-bound decision for a stated release boundary.
Ten worked casesWalk from consequence through controls to recovery
1. Loan covenant disappears
A loan-document summariser passes model safety prompts yet omits an adverse covenant from a long attachment. The consequence is a credit decision made without decisive evidence. The graph travels backwards through analyst approval, summary generation, chunk selection and document parsing. Expected controls are source citation, completeness checks and reviewer access to the clause. The exercise observes that citations cover only included text. The process is narrowed to recommendation, adds document-structure reconciliation and forces visible omission states. Recovery identifies decisions using that parser and corpus version, reopens affected cases and records correction. Outcome: proceed to a controlled adversarial exercise after containment is strengthened.
2. Supplier email redirects payment
Instructions hidden in a legitimate supplier email alter beneficiary evidence. The model safety layer refuses a direct malicious request, but the workflow treats retrieved text as case data, the delegation token permits beneficiary maintenance and the approval screen shows only amount. The graph links influence, inference, authority, approval and external settlement. The red team verifies a transaction-specific resource policy that requires independently verified beneficiary data and displays every changed field. It revokes the old route, checks queued payments and rehearses recipient-bank escalation. Outcome: repair an authority or recovery control before testing.
3. Legitimate employee combines unrelated records
An employee asks a plausible cross-customer question while handling one case. The agent holds an over-broad delegation token, retrieves records outside the intended subject and produces a new inference from their combination. Authentication succeeds; purpose and case scope fail. The expected entitlement filter operates only at the source query, not across derived output. The process is tested with multiple subjects and sessions, then restricted through case-bound credentials, derived-information release policy and cumulative egress limits. Recovery finds recipients and outputs rather than only source reads. Outcome: repair the authority boundary before wider adversarial work.
4. Outage activates a broader fallback
The primary model becomes unavailable. A fallback interprets a changed tool schema, and the emergency service identity can call an update method that the primary path could only prepare. The consequence is an unreviewed record commitment during degraded service. The test varies model, schema, credential and queue state together. The fallback is isolated to read and draft, with a separate admission record and recovery trigger. Historical outage actions are reconstructed from receiving receipts. Outcome: narrow the scope and strengthen containment.
5. Approval becomes ceremonial
A reviewer queue spikes after a policy change. Qualified staff initially inspect evidence, but response-time pressure turns approval into rapid confirmation. The interface emphasises model confidence and hides source conflict. The consequence is repeated adverse customer treatment behind a nominal human control. The exercise seeds disagreements at rising load and measures changed outcomes, inspection and escalation. Authority suspends when capacity falls below the operating envelope, with a safe queue and customer-delay communication. Outcome: rehearse the human and operating response first.
6. Duplicate refund falsely blamed on the model
A timeout occurs after the refund service commits. The orchestration layer retries because it has no external receipt, and the tool accepts a second request without transaction-bound idempotency. Investigators initially attribute two calls to agent instability. Replay shows the model produced one valid proposal; deterministic retry and resource semantics created the duplicate. The repair adds receipt lookup, uncertain-commitment state and receiving-system idempotency. Recovery reconciles the payment ledger and remedies affected customers. This is the required false positive. Outcome: repair an action and recovery control, not the model.
7. Privacy control disables fraud control
A redaction service removes relationship and device signals before a fraud rule evaluates a refund cluster. Both controls meet their local requirement, yet the combined path approves coordinated abuse. The matrix frames the collision without treating privacy as the cause of fraud. A governed service computes the minimum linkage behind a restricted boundary and returns a bounded risk fact, while raw attributes remain hidden from the model and reviewer. The affected refund population is re-evaluated and legitimate customers are protected from blanket reversal. Outcome: narrow scope while the collision is resolved and retested.
8. Kill switch leaves the process alive
An incident command stops new agent sessions, but delayed queue entries, durable memory tasks, pre-issued tool tokens and outbound messages continue. The consequence can grow after the dashboard reports the service offline. The red team begins with the residual effect and discovers each continuation edge. Recovery fails because no owner can cancel one external queue. Release stops until receiving credentials revoke, work items enter quarantine, messages carry cancellable identifiers and external effects reconcile. Outcome: stop release because residual consequence is intolerable.
9. Ordinary ambiguity reaches the blocked state
A malicious request to change a customer address is blocked. A normal request mentioning a recent move passes through clarification, memory merge and an apparently harmless profile update; a later workflow treats the new address as verified. The longer trajectory reaches the same harmful state without hostile wording. The graph reveals that address verification was attached to one entry channel rather than the state transition. The process moves verification to the receiving resource and adds cross-session regression. Outcome: proceed with a controlled adversarial exercise against the repaired state transition.
10. Recovery cannot find who was affected
A policy threshold changes while a managed model alias also moves. Actions record current configuration but not decision-time versions, retrieval evidence or approval context. The organisation cannot identify which customers received the combined behaviour, explain decisions or bound the population. A service rollback restores the prior configuration but does not repair past outcomes. The Recovery Proof fails at reconstruction and remedy. Release remains stopped until immutable decision receipts, affected-population queries, redress authority and a rehearsed return path work. Outcome: stop release because residual consequence is intolerable.
Interactive decision artefactBusiness Process Red-Team Board
The Board selects the next release action. It records the consequence path, trajectory, controls, collisions, affected population, recovery, findings, owner and retest. It does not calculate a safety score or treat exercised scenarios as a guarantee.
Proceed to a controlled adversarial exercise
Consequence path: named material consequence with contained external effect and a defined business owner.
Attack trajectory: representative actor, evidence, decision, authority, action and observation states are mapped.
Expected and observed controls: exercise the stated controls and record actual resource-side effects.
Control collisions: test material filter, approval, retry, fallback and recovery interactions.
Affected population: bounded population and exposure query are ready before the exercise.
Detection and Recovery Proof: independent signal, declaration, containment, reconstruction, reconciliation and remedy are ready.
Unresolved findings: no open intolerable path; new observations receive owners, deadlines and retest.
Release constraint: remain inside the authorised test accounts, values, tools, rate and stopping conditions.
Accountable owner: business process owner with independent security or assurance challenger.
Next retest trigger: any material model, prompt, data, tool, authority, human operating or recovery change.
Operating cadenceMake challenge part of release and change
At process design, the business owner names decisions, populations, unacceptable consequences, irreversible effects and recovery obligations. Security, operations, domain experts, privacy, legal, model risk and affected-interest representatives build the seven-surface map. The team identifies which tests need simulation, staging, synthetic counterparties or tightly authorised field conditions.
Before exercise, a signed plan fixes scope, versions, actors, rules of engagement, data, containment, stopping conditions, communications, evidence custody and emergency authority. The red team receives enough access to challenge the real composition without receiving uncontrolled production authority. Known limitations and untestable differences are visible to the release authority.
During exercise, testers record states and receipts rather than only screenshots and prompts. Defenders observe through normal telemetry unless the scenario states otherwise. Domain observers distinguish surprising model behaviour from rule, identity, interface, queue and handoff failures. Safety staff can stop a test without erasing evidence or silently classifying the finding as closed.
At triage, the challenger and owner reproduce the trajectory, establish consequence and scope, identify the causal control path and set an interim release constraint. A false attribution changes the repair owner while preserving the evidence. Disputes travel to an independent decision authority. Findings link to release claims, affected versions and regression cases.
At closure, the owner demonstrates causal repair, dependent-path review and the Recovery Proof. The independent challenger confirms that the longer trajectory cannot reach the state under the accepted boundary. Release evidence records residual exposure, accountable acceptance, operating ceiling, monitoring and expiry. A passed retest closes the finding, not the broader claim that the process is safe.
During operation, incidents, near misses, complaints, control changes and external flaw reports feed new graph paths. Retest triggers include model or prompt update, retrieval or memory change, new tool or destination, identity-policy revision, fallback activation, material queue pattern, human-service change, recovery failure and new affected population. Production monitoring supplies evidence but does not wait for customers to discover intolerable paths.
Primary evidenceReferences
Links and live status were checked on 2 September 2026. NIST AI RMF 1.0 remains current while revision is under way, and the Playbook states it will be updated after that revision. The title and authors of the sociotechnical red-teaming paper were corrected against its current 2026 arXiv record. The Consequence-Backwards Process Red Team, seven surfaces, Process Attack Graph, Control Collision Matrix, Recovery Proof, Release Evidence Rule and Board are this article’s synthesis.
- NIST, Artificial Intelligence Risk Management Framework 1.0.
- NIST AI 600-1, Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile.
- NIST SP 800-115, Technical Guide to Information Security Testing and Assessment.
- NIST Cybersecurity Framework 2.0.
- NIST SP 800-61 Revision 3, Incident Response Recommendations and Considerations for Cybersecurity Risk Management.
- MITRE ATLAS, Adversarial Threat Landscape for AI Systems.
- OWASP GenAI Security Project, AI Red Teaming Initiative.
- OWASP GenAI Security Project, Securing Agentic Applications Guide 1.0.
- Bullwinkel, B. et al., Lessons From Red Teaming 100 Generative AI Products.
- Debenedetti, E. et al., AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents.
- Zou, A. et al., Security Challenges in AI Agent Deployment: Insights from a Large Scale Public Competition.
- Majumdar, S., Pendleton, B. and Gupta, A., Red Teaming AI Red Teaming.
- Gillespie, T., Shaw, R., Gray, M. L. and Suh, J., AI red-teaming is a sociotechnical problem: on values, labor, and harms.
- Longpre, S. et al., In-House Evaluation Is Not Enough: Towards Robust Third-Party Flaw Disclosure for General-Purpose AI.
- CISA, Secure by Design.
- European Union, Regulation (EU) 2024/1689, Artificial Intelligence Act.
- NIST AI RMF Playbook.
- NIST SP 800-160 Volume 1 Revision 1, Engineering Trustworthy Secure Systems.
About this paperAdversarial evidence for the whole operating path
The Consequence-Backwards Process Red Team, seven surfaces, Process Attack Graph, Control Collision Matrix, Recovery Proof, Release Evidence Rule and Business Process Red-Team Board are this article’s synthesis. They connect model, application, security, identity, human, operational and recovery evidence without replacing their specialist methods. The design does not treat scenario count, jailbreak resistance, human approval, finding closure or a successful rehearsal as certification.