# AI security risk assessment: 7 steps for an LLM application, with a worksheet

Source: https://airedteamingservices.com/guides/ai-security-risk-assessment/
Publisher: Excession Labs (airedteamingservices.com)
Last updated: 2026-11-09

Run an AI security risk assessment on an LLM app in 7 steps: inventory, data flows, threat model, a 4-level rating scale, tests, evidence, retest. Includes a worksheet.

> **Short answer:** An AI security risk assessment reviews the whole application, with the model as one component. List the prompts, retrieval sources, tools, permissions, data stores, logs and model provider. Trace data flows, threat-model each component, rate each risk on a four-level scale, test the top risks, keep evidence, then fix and retest. Map the results to the OWASP LLM Top 10 2026 and NIST AI RMF.

An AI security risk assessment is a repeatable review of the whole application, not a test of the model alone. Include the prompts, retrieval sources, tools, permissions, data stores, logs and the model provider. This page gives seven steps and a worksheet a security team can run on its own LLM application.

It is the self-run version. Our [methodology](/methodology/) page describes how Excession Labs tests, and the [questionnaires page](/ai-security-questionnaires/) lists what buyers ask. We checked the framework facts below on 9 November 2026.

## What does an AI security risk assessment cover?

It covers every place where untrusted text, data or actions meet the model. The model is one component among several, so a review of the model alone misses most of the risk.

Use this list as the scope. Mark each item present or absent.

- Prompts: system prompt, templates, and anything a user can edit.
- Inputs: chat, files, images, email, web pages, tickets, tool results.
- Retrieval: indexes, connectors, and who can add content to them.
- Tools: each function the model can call, and the credential it uses.
- Memory: anything stored between sessions or shared between users.
- Outputs: where answers go (screen, email, code runner, another system).
- Data stores and logs: what is kept, for how long, and who can read it.
- Model provider: hosted or self-hosted, and what the provider keeps.

## Steps 1 and 2: how do you inventory the system and map its data flows?

Step 1 is the inventory. Step 2 draws where data moves. Both fit on one page.

For the inventory, list each component from the scope above with an owner. For the data flows, draw an arrow for every path that carries text or data into the model and every path out. Mark each arrow as trusted (your code) or untrusted (users, outside documents, web content, tool output). Mark each tool with the credential it uses and whether the credential is the user's or a shared account.

The arrows from untrusted sources to a model that also holds private data or can call tools are your highest-value lines. The OWASP LLM Top 10 2026 describes input that alters model behavior as coming from "user input, retrieved content, tool output, image, audio, video or memory". Check that your drawing includes all of them.

## Step 3: how do you threat-model each component?

Ask one question per component: what can an attacker or a mistake do here? Use the OWASP LLM Top 10 2026 as the checklist and write the matching risk next to each component.

| Component | Typical risks to check | OWASP LLM Top 10 2026 entry |
|---|---|---|
| Prompt and inputs | Direct and indirect prompt injection | LLM01 Prompt Injection |
| Hidden instructions | Extraction of system instructions | LLM08 Hidden Context Exposure |
| Answers and logs | Confidential data in output | LLM02 Sensitive Information Disclosure |
| Tools and credentials | Damaging actions from manipulated output | LLM03 Excessive Agency |
| Models, adapters, packages | Weak integrity of artifacts and pipelines | LLM04 Supply Chain |
| Training and fine-tuning data | Data or model poisoning | LLM05 Data and Model Poisoning |
| Request volume | Outage, cost or model cloning | LLM06 Unbounded Consumption |
| Answers people trust | Wrong output used for decisions | LLM07 Misinformation |
| Vector store | Cross-tenant leakage in similarity search | LLM09 Vector and Embedding Weaknesses |
| Downstream systems | Unvalidated output causing XSS, SSRF or code execution | LLM10 Improper Output Handling |

For attacker steps, add [MITRE ATLAS](/guides/mitre-atlas/) technique IDs, for example AML.T0051 (LLM Prompt Injection) and AML.T0053 (AI Agent Tool Invocation) from release v2026.10. If the application has agents, also review the [OWASP agentic threats](/guides/owasp-agentic-ai-threats/).

## Step 4: how do you rate each risk on a four-level scale?

Rate each risk by the effect if it happens and by how easy it is to reach. Use four levels and write each level down before you start, so two reviewers rate the same way. The scale below is an example. Change the wording to match your own risk policy.

| Level | Definition (example) |
|---|---|
| Critical | An outside party with no account can read other users' data, or cause a destructive or irreversible action, with no approval step |
| High | A signed-in user, or untrusted content, can reach data or actions beyond their rights, or send data out of your environment |
| Medium | The control fails but the effect is limited to the attacker's own data or needs unusual conditions |
| Low | A weakness with no demonstrated effect, or a hardening gap |

Rate the demonstrated effect, not the label of the attack. A prompt injection that only changes the tone of a reply to the attacker is Low. The same injection that triggers an email tool is High or Critical, depending on who can send it and what the email can contain.

## Steps 5 and 6: how do you test the top risks and keep evidence?

Test the Critical and High risks first, with test accounts and test data. Write each test as a record: the condition, the input, the expected result and the actual result.

1. Plant a harmless canary string in each untrusted channel from your data-flow drawing.
2. Ask a normal question that makes the app read it.
3. Record the full transcript, the tool-call log and any outbound request.
4. Repeat each attempt at least 10 times. Output varies, so record how many runs went wrong.
5. Check effects by reading the result back through a second account, for example a changed test record.
6. Run the controls: a nonexistent identifier, another user's request, and a normal request with no canary.

Evidence is what you attach to a finding: the transcript, the log lines, the request and the read-back. A statement such as "the model tried to" is not evidence of an effect. Keep the files with timestamps. The [prompt injection examples](/guides/prompt-injection-examples/) page shows real cases that make good test templates.

## Step 7: how do you fix, retest and record the result?

Fix the cause, then run the same test again. Narrow tools, per-user credentials, a gate before side effects and removal of outbound paths work better than a longer system prompt. Record what you changed, the date and the retest result.

Record residual risk as well. The UK NCSC wrote on 8 December 2025 that prompt injection may never be totally mitigated and that the best case is reducing likelihood and impact. State which injection paths remain and which control limits each one.

## What does the worksheet look like, and how does it map to NIST AI RMF?

Copy one row per risk. The columns map to the NIST AI RMF, which has four functions: Govern, Map, Measure and Manage. Steps 1 to 3 are Map work, steps 4 to 6 are Measure work, and step 7 is Manage work. The mapping is our reading. NIST has not published this step list.

| Field | Example entry (fictional) |
|---|---|
| Component | Support agent, ticket search tool |
| Untrusted source | Customer-written tickets |
| Risk | Indirect injection makes the agent list another account's tickets |
| OWASP LLM 2026 | LLM01, LLM03 |
| ATLAS v2026.10 | AML.T0051.001, AML.T0053 |
| Rating | High (example scale) |
| Test | Canary in a ticket; 10 runs; read-back from a second account |
| Evidence file | transcript-014.txt, toolcalls-014.json |
| Fix | Call the API with the signed-in user's token |
| Retest | Date and result |
| Residual risk | Injection still possible; the API now blocks cross-account reads |

> **Demonstration:** This worksheet row is a demonstration with a fictional app. No real customer system was tested.
The NIST AI RMF page states that the framework is being revised as part of the White House AI Action Plan, so check it before you cite a version. NIST AI 600-1, the Generative AI Profile, lists suggested actions for red-teaming, including action MS-2.7-007 on resilience against prompt injection. For the process side, read the [NIST AI RMF and red teaming guide](/guides/nist-ai-rmf-and-red-teaming/) and the [OWASP guide](/guides/owasp-top-10-for-llm-applications/).

A scoped outside assessment follows the same logic with written authorization from the system owner. To plan one, use the [scoping guide](/guides/how-to-scope-an-ai-red-team/).

## Questions

### Is an AI security risk assessment the same as an AI governance risk assessment?

No. A governance assessment looks at policy, ownership, fairness and compliance across the AI lifecycle. A security risk assessment asks how an attacker or a mistake could expose data or trigger actions in one application. Many teams run both. This page covers the security one.

### How often should we repeat the assessment?

Repeat it when a tool, data source, model or permission changes, and before a release that adds write access. Choose an interval and write it into your process.

### Do we need a penetration test to finish the assessment?

The review and threat model find where risk sits. Testing shows which risks are real. Step 5 can be done by your own team with test accounts. An outside test adds a second view and evidence for buyers. Test only with the system owner's written authorization.

### What do customers mean when they ask for an AI risk assessment?

Buyers often send a questionnaire about data use, tool permissions, testing and incident handling. Your completed worksheet gives you the evidence to answer. Our questionnaires page lists what buyers ask.

### Can the assessment show our AI app is secure?

No. It shows which risks you found, how you rated them and what you fixed. NCSC says prompt injection may never be fully mitigated, so record residual risk and the controls that limit its impact.

## Sources

- [NIST AI Risk Management Framework](https://www.nist.gov/itl/ai-risk-management-framework)
- [NIST AI RMF 1.0 (NIST AI 100-1) publication page](https://www.nist.gov/publications/artificial-intelligence-risk-management-framework-ai-rmf-10)
- [OWASP Top 10 for LLM Applications 2026 (README)](https://github.com/GenAI-Security-Project/GenAI-LLM-Top10/blob/main/2026/README.md)
- [NIST AI 600-1, Generative AI Profile](https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.600-1.pdf)
- [MITRE ATLAS data release v2026.10](https://github.com/mitre-atlas/atlas-data/releases/tag/v2026.10)
- [UK NCSC: Prompt injection is not SQL injection](https://www.ncsc.gov.uk/blog-post/prompt-injection-is-not-sql-injection)