AI security: the 10 risks to test in LLM apps, RAG and agents
AI security is the practice of protecting AI systems (models, prompts, retrieved data, tools, agents and the code around them) from attack and misuse. It is a different job from using AI for defense. The OWASP LLM Top 10 2026 lists ten risks, and each one maps to a test. Filters alone do not stop prompt injection, so limit what a manipulated model can reach.
AI security is the practice of protecting AI systems from attack and misuse. The systems include models, prompts, retrieved documents, tools, agents and the software around them. This page covers securing AI applications, not using AI for defense.
The sections below show where AI applications are attacked, the ten risks in the OWASP LLM Top 10 2026 with a test for each, the risks that agents and MCP add, the frameworks to know, and a starter checklist.
What does AI security mean?#
AI security means finding and reducing the ways an attacker can make an AI system leak data, act wrongly or fail. Two other meanings share the name. One is using AI inside security tools, such as alert triage. The other is AI safety, which concerns harmful content and model behavior. Many search results blur these.
For an application team the object is the whole system. A language model takes text in and gives text out. The risk comes from what the application connects to it: your documents, your users' files, your APIs and the actions an agent may take. A model with no tools and no private data has little to lose. The same model with a mailbox, a database and a payment tool has a lot.
Where are AI applications attacked?#
AI applications are attacked at five places: the input, the data the model reads, the tools it calls, the output it produces and the supply chain behind them. The diagram shows where each sits.
An attacker needs only one path from untrusted input to data or an action that matters. A tester lists the inputs an outsider controls, the data and tools the model can reach, and what leaves the system. The prompt injection guide explains why the model cannot reliably tell data from instructions.
What are the ten OWASP LLM risks, and how do you test each?#
OWASP released the Top 10 for LLM Applications 2026 on 4 August 2026. It replaces the 2025 list, and the IDs and order changed. The table gives each risk, an example, a test and where to read more. The tests are our summary, and the one-line meanings follow the OWASP entry text.
| ID | Risk | Example | Test | Read more |
|---|---|---|---|---|
| LLM01:2026 | Prompt Injection | Input alters model behavior in ways the developer did not intend. A hidden line in a public Slack channel made Slack AI show a link that carried a private secret (PromptArmor, August 2024) | Plant marker text in each source the app reads, then check for changed answers or tool calls | Prompt injection |
| LLM02:2026 | Sensitive Information Disclosure | Confidential data leaves through an unauthorized channel | Seed canary values in prompts, documents and tool output and try to surface them | Markdown image exfiltration |
| LLM03:2026 | Excessive Agency | The system takes damaging actions on manipulated output | Review each tool and its permissions, then call it with out-of-role arguments | Excessive agency |
| LLM04:2026 | Supply Chain | Weak integrity of training data, models, adapters and platforms | Inventory every model, plugin and tool server, with source, version and who can change it | MCP security risks |
| LLM05:2026 | Data and Model Poisoning | Manipulated data or artifacts embed harmful behavior | Plant a canary document through each write path to the index and memory | Data poisoning |
| LLM06:2026 | Unbounded Consumption | Uncontrolled inference causes outage, cost or model cloning | Bounded tests of input size, loops and rate limits, with a cost cap | Model extraction |
| LLM07:2026 | Misinformation | People or agents trust and act on incorrect output | Ask questions with known answers and check grounding in sources | Customer support chatbots |
| LLM08:2026 | Hidden Context Exposure | Someone extracts hidden instructions or context | Try to extract the system prompt and report what a user can learn | System prompt leakage |
| LLM09:2026 | Vector and Embedding Weaknesses | Weaknesses in embeddings and similarity search, including cross-tenant leakage | Query across two test tenants with seeded markers | RAG security |
| LLM10:2026 | Improper Output Handling | Output goes downstream unvalidated and can cause XSS, SSRF or code execution | Check what the browser and downstream parsers do with model output | OWASP LLM Top 10 |
OWASP's 2026 list renames System Prompt Leakage to Hidden Context Exposure, and moves Excessive Agency up from LLM06 to LLM03. The OWASP hub page still showed the 2025 list on 9 November 2026, so check the edition when you read a blog post.
What do agents and MCP add?#
Agents add actions, memory and delegation, so one successful injection can change data or spend money. OWASP released the Top 10 for Agentic Applications on 9 December 2025, with IDs ASI01 Agent Goal Hijack to ASI10 Rogue Agents. Our OWASP agentic guide gives all ten with a test for each, and the AI agent security guide covers the threat model.
MCP, the Model Context Protocol, connects an assistant to tools and data. Anthropic announced it on 25 November 2024. Its security best practices page covers the confused deputy problem, token passthrough, SSRF and scope minimization. Tool poisoning, where instructions hide in a tool description, comes from research by Invariant Labs and is not in that page. The MCP security risks guide has the checklist.
A real case shows how the pieces combine. Invariant Labs reported on 26 May 2025 that a malicious GitHub Issue in a public repository could hijack a user's agent and leak private repository data into a pull request. The authors say it was not a flaw in the GitHub MCP server code and that security must be implemented at the system level.
Which frameworks matter, and what does each one do?#
Each framework answers a different question. None is a certification of your product.
| Framework | What it gives you | Facts to know (checked 9 November 2026) |
|---|---|---|
| OWASP Top 10 for LLM Applications 2026 | A ranked list of ten risks | Released 4 August 2026. See the OWASP LLM guide |
| OWASP Top 10 for Agentic Applications | Ten agent risks, ASI01 to ASI10 | Announced 9 December 2025 |
| MITRE ATLAS | A catalog of attack techniques | Release v2026.10 has 16 tactics, 129 techniques and 92 sub-techniques (our count from the release file). See MITRE ATLAS |
| NIST AI RMF 1.0 and AI 600-1 | A risk management process and a generative AI profile | AI RMF published 26 January 2023, AI 600-1 on 26 July 2024. See NIST AI RMF |
| NIST AI 100-2 E2025 | A taxonomy and terminology of attacks | Gives IDs such as NISTAML.018 Prompt Injection |
| EU AI Act | A law with duties by risk class | Annex III high-risk duties apply from 2 December 2027 after Regulation (EU) 2026/1744. See EU AI Act and red teaming |
| ISO/IEC 42001:2023 | A certifiable AI management system standard | Covers an organization, not one product. It is not a technical test standard |
Do not write that the EU AI Act requires penetration tests. Article 15(5) lists attacks such as data poisoning that high-risk systems must address where appropriate, but the term "red team" is not in that article.
How do you test an AI application?#
Testing an AI application follows four steps: model the threats, set the test conditions, collect evidence and retest after fixes.
- Threat model. List the inputs an outsider controls, the data and tools the model reaches, and the worst action each tool allows.
- Test conditions. Use a test environment, test tenants, fake seeded data and written authorization from the system owner. Record the model version, the prompt version and the tool set.
- Evidence. Model output varies, so repeat each attack and record the attempts and successes. A refusal or an odd answer is not a finding. A finding needs a visible effect: a marker in the wrong tenant, a tool call that should not happen.
- Retest. After a fix, run the same cases again under the same conditions.
Microsoft's AI Red Team makes the same point in its paper on red teaming 100 generative AI products: the work is about end-to-end systems, not model benchmarks. Our what is AI red teaming guide places this in an engagement, and the scoping guide lists what a tester needs.
Which controls reduce the impact, and which do not?#
Controls that limit what a manipulated model can do reduce impact. Filters that try to spot bad text reduce likelihood but do not close the problem.
The UK NCSC wrote on 8 December 2025 that prompt injection "may never be totally mitigated in the way that SQL injection attacks can be" and that the risk "cannot be fully mitigated with a product or appliance". NIST AI 100-2 E2025 (section 3.4.4) advises designers to assume that prompt injection is possible when a model sees untrusted input.
| Control | What it does | Limit |
|---|---|---|
| Least privilege for tools | Narrows what a hijacked model can reach | Needs design work per tool |
| Authorization in code | Checks the user's rights outside the model | Must run on every call |
| Human approval for side effects | Stops silent actions | Approval screens can be vague. See approval bypass |
| Output controls | Blocks image and link exfiltration, escapes output | Must cover every rendering path |
| Tenant filters in the query | Keeps retrieval inside one tenant | Filtering after ranking fails |
| Input and output filters | Catch known bad patterns | Attackers rephrase, so do not rely on them alone |
| Monitoring and rollback | Finds and reverses damage | Works after the event |
Treat a filter as one layer. The AI guardrails guide covers what they can and cannot do.
What should you do first? A starter checklist#
Start with these ten items. Each takes a decision or a test, and none needs a new product.
- List every AI feature, the model behind it and the data it reads.
- List every tool each feature can call and the identity the call uses.
- Mark inputs that outsiders control: email, tickets, uploads, web pages, shared documents.
- Remove tools and data the feature does not need.
- Move authorization out of the prompt and into code.
- Require approval for actions that send, delete, pay or share.
- Stop the UI from loading images or links from model output unless you allow the domain.
- Test retrieval with two tenants and seeded markers.
- Record the model, prompt and tool versions, then run your attack cases again after each change.
- Get written authorization and a test environment before anyone tests.
If you want a scoped test of these paths, read about our LLM application security assessment or the AI risk assessment guide. Excession Labs agrees the scope with you in writing and tests only with written authorization from the system owner. Send a scoping request with your feature list.
Shipping an AI feature or agent?
Describe the system, the data it reads and the actions it can take. We reply with questions and a proposed scope.
Questions
Is AI security the same as using AI for cybersecurity?
No. AI security protects AI systems from attack. Using AI for security means applying AI to detect threats or to run tests. Search results mix the two. This page covers the first meaning: LLM applications, RAG systems and agents that your team builds or buys.
What is the biggest AI security risk for a business app?
OWASP ranks prompt injection first (LLM01:2026). For apps with tools, Excessive Agency (LLM03:2026) decides how much damage an injection can do. OWASP says the 2026 edition draws on research from thousands of real-world incidents. Check which entries match your design: data, tools or a public chat.
Can prompt injection be fully prevented?
No source we checked says so. The UK NCSC wrote in December 2025 that prompt injection may never be totally mitigated like SQL injection, and that the best we can hope for is reducing the likelihood or impact of attacks. Plan for lower likelihood and lower impact.
Do I need to test the model or the application?
For most teams, the application. Model vendors run model-level safety work. You own the prompts, the data the model reads, the tools it can call and how output is shown. Those parts hold most findings in an LLM feature.
Which AI security framework should I start with?
Start with the OWASP Top 10 for LLM Applications 2026 for a test list, then add MITRE ATLAS for attack names and NIST AI RMF for risk management. The EU AI Act and ISO/IEC 42001 matter if your market or buyers require them.
Sources
- OWASP Top 10 for LLM Applications 2026 (README, release 4 August 2026) (checked 9 November 2026)
- OWASP Top 10 for Agentic Applications: the benchmark for agentic security (9 December 2025) (checked 9 November 2026)
- MITRE ATLAS data release v2026.10 (checked 9 November 2026)
- NIST AI RMF 1.0 (NIST AI 100-1) (checked 9 November 2026)
- NIST AI 600-1: Generative AI Profile (checked 9 November 2026)
- NIST AI 100-2 E2025: Adversarial Machine Learning taxonomy (checked 9 November 2026)
- UK NCSC: Prompt injection is not SQL injection (8 December 2025) (checked 9 November 2026)
- Regulation (EU) 2026/1744 (Digital Omnibus on AI), amending Regulation (EU) 2024/1689 (checked 9 November 2026)
- IEC Webstore: ISO/IEC 42001:2023 (checked 9 November 2026)
- Model Context Protocol: Security Best Practices (checked 9 November 2026)
- Simon Willison: EchoLeak (11 June 2025) (checked 9 November 2026)
- PromptArmor: Data Exfiltration from Slack AI via indirect prompt injection (20 August 2024) (checked 9 November 2026)
- Invariant Labs: GitHub MCP exploited (26 May 2025) (checked 9 November 2026)