AI and LLM Threat Model · Application Review

Illustrative Findings Boundary

The severity-labelled scenarios below are newly authored fictional examples informed by the control domains assessed in this work. They do not reproduce or summarize any engagement finding, do not describe the reviewed environment, and do not indicate that any weakness was observed, confirmed, remediated or remains outstanding. Severity labels apply only to these examples and illustrate prioritization methodology.

Each example combines a plain-English business summary, a technical condition, potential impact, the evidence an assessment would seek, the expected control and a measurable way to check the fix.

Prompt injection is an application-security problem, not only a model problem. Instructions can arrive directly from a user or indirectly through documents, email, images, retrieved records or other content the application asks the model to process.

Direct prompt injection (OWASP LLM01)
Indirect document-borne instructions (OWASP LLM01)
Sensitive-information disclosure (OWASP LLM02)
Improper model-output handling (OWASP LLM05)
Excessive agency (OWASP LLM06)
Tool and privilege misuse (OWASP LLM06)
Poisoned or misleading content (OWASP LLM04, LLM09)
Resource and cost abuse (OWASP LLM10)

Security boundary. Content filtering can reduce risk, but it does not replace identity, authorization, schema validation, least privilege or deterministic approval for high-impact actions.

Fictional Security Examples

Illustrative AI and LLM Threat Examples

Illustrative only - not an engagement finding

Document-Borne Instructions Influence Cross-Customer Data Access

Critical See where this finding applies on the diagrams: AI services zone · AI analysis step · CP5 AI-call boundary

Ease of Attack: Requires crafted document content
Impact: Cross-customer retrieval or disclosure
Detection Difficulty: Requires tracing model output into retrieval and authorization code
Pattern: Example prompt-injection boundary pattern

Business Summary: Every uploaded document is untrusted model input. If instructions hidden in that content can influence retrieval or authorization, prompt injection stops being only a model quality problem and becomes a way for one customer's data to reach another.
Possible Weakness: Text embedded in a PDF, email or image instructs the model to request another customer’s records, and the application code acts on whichever record identifier or search scope the model chose, without checking it.
Potential Impact: Sensitive records could cross a tenant boundary or be included in a model response, log entry or generated working paper.
What an Assessment Verifies: Trace document content through prompt construction in the async worker, the Azure AI Document Intelligence and Azure OpenAI calls, the decisions about which records may be retrieved and the downstream Cosmos DB writes, using test documents that hide adversarial instructions and made-up customer records with known permissions.
Expected Fix: Keep tenant scope and authorization in deterministic FastAPI and worker code, treat document text only as data and never let Azure OpenAI output expand identity, retrieval or tool permissions.
How to Check the Fix: A reviewed set of documents carrying hidden instructions must never retrieve or reveal another tenant’s planted test data, while authorized classification continues to work and every attempted scope change is logged safely.

Cross-Reference Evidence (Illustrative). This is cross-reference evidence showing how prompt construction, authorization code and adversarial test output are reviewed together. In a real engagement each item below is a live pointer: a file-and-line link into the source repository, a reproducible read-only query against the deployed Azure environment, or replayable tool output. Here they lead to newly authored public demonstrations.

Example prompt and authorization source references
This is cross-reference evidence. A real assessment links the reviewed commit and exact lines across the route, service and data boundary so another reader can follow the complete control path.
```
if actor.tenant_id != record.tenant_id:
    raise Forbidden()
```
Example indirect-injection regression output
This is cross-reference evidence. A real report retains the tool version, safe test case, expected result and output needed to reproduce the conclusion.
```
security-regression --case "<illustrative-case>" --format json
Illustrative output: {"result":"blocked as expected"}
```
Example AI workload-identity record
This is cross-reference evidence. A real assessment records a dated read-only identity or configuration query and explains what that result does and does not prove.
```
az resource show --ids "<illustrative-resource-id>" \
  --query "identity.type"

Illustrative output: "SystemAssigned"
```

Illustrative only - not an engagement finding

Model and Error Paths Disclose Sensitive Document Content

High See where this finding applies on the diagrams: AI services zone · telemetry and error paths · CP8 telemetry separation

Ease of Attack: Depends on output or log access
Impact: PII disclosure across unintended surfaces
Detection Difficulty: Visible in log and API response inspection
Pattern: Example minimization-and-masking pattern

Business Summary: Tax documents are dense with personally identifiable information. Copies can escape the approved workflow through model responses, prompts, traces, errors or support logs even when the primary data store is protected.
Possible Weakness: Full document text, prompts or model responses are returned to callers or retained in broadly accessible logs without masking or data-minimization rules.
Potential Impact: Names, identifiers and financial details could reach unauthorized users, support tooling or telemetry audiences with longer retention than the source document.
What an Assessment Verifies: Plant recognizable fake personal details in test documents and trace them across model input, output, errors, application logs, Azure Monitor telemetry and status records persisted in Cosmos DB.
Expected Fix: Minimize what the model sees, mask personal details before Azure OpenAI calls where appropriate, validate outputs and keep sensitive traces in narrowly controlled Log Analytics tables with explicit retention.
How to Check the Fix: The planted fake details must never appear in responses, errors or logs, and may appear only in the explicitly approved protected records the business workflow needs.

Cross-Reference Evidence (Illustrative). This is cross-reference evidence showing how the private reports connect this example’s verification to complementary sources. In a real engagement each item below is a live pointer: a file-and-line link into the source repository, a reproducible read-only query against the deployed Azure environment, or replayable tool output. Here they lead to newly authored public demonstrations.

Example source-code evidence reference
This is cross-reference evidence. A real assessment links the reviewed commit and exact lines across the route, service and data boundary so another reader can follow the complete control path.
```
if actor.tenant_id != record.tenant_id:
    raise Forbidden()
```
Example read-only Azure evidence record
This is cross-reference evidence. A real assessment records a dated read-only identity or configuration query and explains what that result does and does not prove.
```
az resource show --ids "<illustrative-resource-id>" \
  --query "identity.type"

Illustrative output: "SystemAssigned"
```
Example reproducible tool output
This is cross-reference evidence. A real report retains the tool version, safe test case, expected result and output needed to reproduce the conclusion.
```
security-regression --case "<illustrative-case>" --format json
Illustrative output: {"result":"blocked as expected"}
```

Illustrative only - not an engagement finding

Untrusted Document Text Alters Classification Behaviour

Medium See where this finding applies on the diagrams: AI services zone · AI analysis step · CP5 AI-call boundary

Ease of Attack: Requires crafted model input
Impact: Misclassification and workflow disruption
Detection Difficulty: Requires adversarial test documents and output comparison
Pattern: Example bounded-classification pattern

Business Summary: Even when a model cannot take privileged action, hidden instructions can still bias classification, increase manual review and route documents through the wrong business process.
Possible Weakness: Instructions embedded in a document tell the model to ignore classification rules, select a preferred category or report an unjustified confidence score.
Potential Impact: Documents could be misclassified, omitted from the expected workflow or accepted with misleading confidence, reducing the reliability of the final working paper.
What an Assessment Verifies: Run direct and indirect injection cases across PDFs, emails and images and compare the Azure OpenAI classification category, confidence, whether the model declines to answer, and reviewer presentation with a reviewed baseline.
Expected Fix: Constrain Azure OpenAI classification to a closed category set, separate instructions from document content and require human review through the confidence-based stoplight screen.
How to Check the Fix: Injection cases must remain within the closed category schema, surface uncertainty or decline to answer when appropriate and always remain subject to a recorded human review before final assembly.

Example source-code evidence reference
This is cross-reference evidence. A real assessment links the reviewed commit and exact lines across the route, service and data boundary so another reader can follow the complete control path.
```
if actor.tenant_id != record.tenant_id:
    raise Forbidden()
```
Example read-only Azure evidence record
This is cross-reference evidence. A real assessment records a dated read-only identity or configuration query and explains what that result does and does not prove.
```
az resource show --ids "<illustrative-resource-id>" \
  --query "identity.type"

Illustrative output: "SystemAssigned"
```
Example reproducible tool output
This is cross-reference evidence. A real report retains the tool version, safe test case, expected result and output needed to reproduce the conclusion.
```
security-regression --case "<illustrative-case>" --format json
Illustrative output: {"result":"blocked as expected"}
```

Assessed control pattern - not an engagement finding

Bounded, Deterministic LLM Controls

Design position See where these controls apply on the diagrams: AI services zone · AI analysis step · CP5 AI-call boundary

The surrounding application should assume that model behavior can be manipulated or can drift. Controls therefore constrain what information the model receives, what shape it can return and what the application will permit afterward.

Schema-validated model inputs and outputs.
Closed or allow-listed actions instead of open-ended tool access.
Retrieval limited to the caller's permissions, and tenant separation.
Least-privilege workload identities and downstream authorization.
Human approval for high-impact, external or irreversible actions.
Audit records connecting model input, output, decision and resulting action.

Application Review · Control Domain 1 of 5 · Public Edition

Part of the two-report public security-review portfolio.

Report overview