Adversarial Regression and Guardrail Drift

Illustrative Findings Boundary

The severity-labelled scenarios below are newly authored fictional examples informed by the control domains assessed in this work. They do not reproduce or summarize any engagement finding, do not describe the reviewed environment, and do not indicate that any weakness was observed, confirmed, remediated or remains outstanding. Severity labels apply only to these examples and illustrate prioritization methodology.

Each example combines a plain-English business summary, a technical condition, potential impact, the evidence an assessment would seek, the expected control and a measurable way to check the fix.

Model, prompt and data changes can alter security behavior without a conventional code defect. Replayable test cassettes (recorded model interactions) and adversarial regression cases provide a stable way to compare releases and detect changes in prompt-injection resilience, output validation and tool use.

Delivery pattern. Keep a reviewed test set of direct and indirect injection cases, disclosure attempts, malformed outputs and requests for unsafe actions. Run it when models, prompts, retrieval sources or control code change, then retain the result as release evidence.

Fictional Security Examples

Illustrative Regression and Guardrail Examples

Illustrative only - not an engagement finding

Model Output Can Select an Unapproved Downstream Action

Critical See where this finding applies on the diagrams: model output path · bounded output handling · CP5 AI-call boundary

Ease of Attack: Requires manipulated model output
Impact: Unauthorized external or irreversible action
Detection Difficulty: Visible in source review of how model output becomes actions
Pattern: Example bounded-agency pattern

Business Summary: A model may recommend an action, but it should not decide what the application is authorized to change, send or delete. High-impact authority belongs to deterministic code and accountable people.
Possible Weakness: Free-form model output is interpreted as a tool, destination or operation and executed without an allow-list, independent authorization or human approval.
Potential Impact: Manipulated or malformed output could trigger data disclosure, destructive changes, external communication or paid processing beyond the intended workflow.
What an Assessment Verifies: Trace every consumer of Azure OpenAI output, tool permission, destination rule, authorization check and approval step from response parsing in the async worker to the final side effect.
Expected Fix: Map model output only to closed, schema-validated actions in the async worker, reauthorize in code and require human approval for external, irreversible or high-impact operations.
How to Check the Fix: Actions, destinations and parameters outside the approved set must be rejected, and every high-impact test request must stop at a recorded human approval before any side effect occurs.

Cross-Reference Evidence (Illustrative). This is cross-reference evidence showing how the model-output consumer, tool policy and approval test are connected. In a real engagement each item below is a live pointer: a file-and-line link into the source repository, a reproducible read-only query against the deployed Azure environment, or replayable tool output. Here they lead to newly authored public demonstrations.

Example source reference for mapping output to actions
This is cross-reference evidence. A real assessment links the reviewed commit and exact lines across the route, service and data boundary so another reader can follow the complete control path.
```
if actor.tenant_id != record.tenant_id:
    raise Forbidden()
```
Example test rejecting an unsafe action
This is cross-reference evidence. A real report retains the tool version, safe test case, expected result and output needed to reproduce the conclusion.
```
security-regression --case "<illustrative-case>" --format json
Illustrative output: {"result":"blocked as expected"}
```
Example least-privilege workload record
This is cross-reference evidence. A real assessment records a dated read-only identity or configuration query and explains what that result does and does not prove.
```
az resource show --ids "<illustrative-resource-id>" \
  --query "identity.type"

Illustrative output: "SystemAssigned"
```

Illustrative only - not an engagement finding

Free-Form Model Output Reaches Persistent Records

High See where this finding applies on the diagrams: model output path · bounded output handling

Ease of Attack: Requires malformed or manipulated output
Impact: Persistent corruption or unsafe rendering
Detection Difficulty: Visible in source review of the code paths that write records
Pattern: Example schema-enforcement pattern

Business Summary: Model output becomes part of the trusted system when it is stored, rendered or assembled into a client deliverable. Free-form text can violate data contracts and carry unsafe content forward.
Possible Weakness: An application writes model-generated categories, identifiers or document content into records or reports without a closed schema, escaping and deterministic validation.
Potential Impact: Malformed values could corrupt workflow state, inject active content, break final documents or place unreviewed text into a customer deliverable.
What an Assessment Verifies: Inspect response contracts, parser failure behaviour, Cosmos DB constraints, rendering contexts and the human-review boundary before final PDF assembly.
Expected Fix: Constrain model responses to closed categories and typed schemas, reject additional fields before Cosmos DB writes, escape every rendering context and require review before the final PDF.
How to Check the Fix: Malformed, additional and out-of-category fields must fail closed, while valid outputs persist safely and only human-approved classifications appear in the final test working paper.

Cross-Reference Evidence (Illustrative). This is cross-reference evidence showing how the private reports connect this example’s verification to complementary sources. In a real engagement each item below is a live pointer: a file-and-line link into the source repository, a reproducible read-only query against the deployed Azure environment, or replayable tool output. Here they lead to newly authored public demonstrations.

Example source-code evidence reference
This is cross-reference evidence. A real assessment links the reviewed commit and exact lines across the route, service and data boundary so another reader can follow the complete control path.
```
if actor.tenant_id != record.tenant_id:
    raise Forbidden()
```
Example read-only Azure evidence record
This is cross-reference evidence. A real assessment records a dated read-only identity or configuration query and explains what that result does and does not prove.
```
az resource show --ids "<illustrative-resource-id>" \
  --query "identity.type"

Illustrative output: "SystemAssigned"
```
Example reproducible tool output
This is cross-reference evidence. A real report retains the tool version, safe test case, expected result and output needed to reproduce the conclusion.
```
security-regression --case "<illustrative-case>" --format json
Illustrative output: {"result":"blocked as expected"}
```

Illustrative only - not an engagement finding

Prompt Changes Ship Without Adversarial Regression Evidence

Medium See where this finding applies on the diagrams: delivery zone [planned] · AI analysis step · CP6 supply-chain gate

Ease of Attack: Not an attack path; permits guardrail drift
Impact: Known AI security failures can reappear
Detection Difficulty: Apparent from missing release test evidence
Pattern: Example adversarial-release pattern

Business Summary: A prompt, model or retrieval change can alter security behaviour without changing ordinary application tests. A release needs evidence that known injection and disclosure cases did not regress.
Possible Weakness: Model, prompt, retrieval or output-handling changes reach release without replaying a reviewed test set of direct and document-borne injection cases, disclosure attempts and malformed outputs.
Potential Impact: A previously contained attack pattern could reappear silently, or guardrails could drift while conventional code tests continue to pass.
What an Assessment Verifies: Review adversarial test coverage, where the replayable test cassettes come from, model and prompt versioning, approval of expected results, Azure DevOps release gates and retained comparison evidence.
Expected Fix: Run a versioned set of adversarial test cases whenever models, prompts, retrieval sources or control code change and block the Azure DevOps release on an unapproved security regression.
How to Check the Fix: Known cases covering direct and indirect injection, PII disclosure, malformed output and unsafe actions must run on every relevant change, fail the release when they regress and retain results as release evidence.

Example source-code evidence reference
This is cross-reference evidence. A real assessment links the reviewed commit and exact lines across the route, service and data boundary so another reader can follow the complete control path.
```
if actor.tenant_id != record.tenant_id:
    raise Forbidden()
```
Example read-only Azure evidence record
This is cross-reference evidence. A real assessment records a dated read-only identity or configuration query and explains what that result does and does not prove.
```
az resource show --ids "<illustrative-resource-id>" \
  --query "identity.type"

Illustrative output: "SystemAssigned"
```
Example reproducible tool output
This is cross-reference evidence. A real report retains the tool version, safe test case, expected result and output needed to reproduce the conclusion.
```
security-regression --case "<illustrative-case>" --format json
Illustrative output: {"result":"blocked as expected"}
```

Application Review · Control Domain 2 of 5 · Public Edition

Part of the two-report public security-review portfolio.

Report overview