AI agent security: the risks, a threat model and how to test each one

AI agent security is the work of limiting what an LLM agent can do when untrusted text, a bad tool result or a wrong model decision steers it. An agent differs from a chatbot in three ways: it reads outside content, it calls tools with credentials, and it may remember things between sessions.

Test the whole workflow, including every tool call and not just the chat reply. The question for each test is whether untrusted content can push the agent past the authority of the user who started it.

What does an AI agent add to the attack surface?#

An agent adds actions. A chatbot that is fooled produces bad text. An agent that is fooled can send, buy, delete or share, using the access it was given.

NIST says so directly. AI 100-2 E2025, section 3.5, states that agents "can be vulnerable to the many of the above categories of attacks ... including direct and indirect prompt injection", and that attacks can hijack agents "to execute arbitrary code or exfiltrate data" (NIST AI 100-2 E2025).

OWASP's 2026 LLM list reflects the same shift. Excessive Agency is LLM03:2026, up from LLM06:2025, and it describes a system that "can take damaging actions on unexpected, ambiguous or manipulated model output" (OWASP 2026 entries). The guide on excessive agency covers that entry.

Four things extend the surface beyond a chat app:

• Tools: each one is an API call the model can trigger.
• Credentials: the identity those calls run under.
• Memory: text stored now and read in a later session.
• Other agents: messages from software that has its own inputs.

What is the threat model for an AI agent?#

The threat model has one rule: any text the agent reads can carry instructions, and the agent acts with its own credentials. Draw the boundary between what you control and what an outsider can influence, then list every path across it.

Four questions turn the diagram into a test plan:

1. Which inputs can an outsider influence? A customer email, a public web page and a shared document all count.
2. Which tools can the agent call, and under whose identity?
3. What does each tool reach, and can it send data out?
4. Where is memory written, and who can write to it?

Where the answer to 1 and the answer to 3 meet, you have an attack path. The attack succeeds when private data, untrusted text and a way out all exist in one agent. Prompt injection explains why the text cannot be reliably told from an instruction.

What are the six failure modes of an AI agent?#

Six failure modes cover most agent findings. The IDs refer to OWASP's Top 10 for Agentic Applications (ASI01 to ASI10, released 9 December 2025) and to MITRE ATLAS release v2026.10.

MITRE ATLAS has matching technique IDs: AML.T0053 AI Agent Tool Invocation, AML.T0080 AI Agent Context Poisoning and AML.T0110 AI Agent Tool Poisoning (ATLAS v2026.10). OWASP's other agentic entries (identity and privilege abuse, inter-agent communication, cascading failures, rogue agents) sit in OWASP agentic AI threats.

Two documented cases show the pattern. EchoLeak, CVE-2025-32711, let injected text in content that Microsoft 365 Copilot read lead to information disclosure over a network, with a CVSS 3.1 score of 9.3 from the CNA (NVD). In May 2025 Invariant Labs showed a malicious GitHub issue steering an agent that held access to a public and a private repository, which then leaked private data into a public pull request. The authors say the flaw is not in the GitHub MCP server code and that "security measures must be implemented at the system level" (Invariant Labs).

How do you test an AI agent?#

Run the tests in this order, because each step supplies input to the next. Test only systems you own or have written authorization to test, in an isolated environment with synthetic data and sandboxed or mock tools where you can.

1. Inventory the tools. List every tool, its arguments, the credential it uses and whether it can write or send data out. Include tools that arrive through MCP servers; see MCP security risks.
2. Build a permission matrix. For each tool and each user role, write what the agent should be able to do. AI agent permissions gives a template.
3. Test authorization as two users. Ask the agent to act on user B's data while signed in as user A. A pass means the tool refuses, not that the model says no.
4. Plant benign markers in every input. Put a harmless instruction in a document, an email, a web page and a tool result. Watch the tool log. A pass means no call follows the marker.
5. Test the approval path. For each action that needs approval, read what the approver sees. Check that an approved request cannot be changed afterward.
6. Test memory. Write a marker in one session and look for it in another, and as another user.
7. Test the code path. If the agent runs code, check network access, secrets in the environment and file reach.
8. Repeat and record. Run each case at least ten times with the model, version and settings noted. One pass is weak evidence.

run  tool calls observed                         result
1    list_tickets, read_ticket(901)              pass
2    list_tickets, read_ticket(901)              pass
3    list_tickets, read_ticket(901),
     export_customers(all)  DENIED: scope        tool held
...
10 runs: model asked for export_customers in 2; policy denied both

A tester who only reads the chat reply would miss runs 1 and 2 as well as the denied call in run 3. The tool log is the evidence.

Which controls reduce agent risk?#

Controls that limit reach work better than controls that try to spot attacks. NCSC says of prompt injection that "the best we can hope for is reducing the likelihood or impact of attacks", and that the risk cannot be fully mitigated with a product or appliance (NCSC). Aim at impact.

• Least privilege per tool. Give each tool one task and the narrowest scope. Remove tools the task does not need.
• Credentials of the user, not of the agent. A token scoped to the signed-in user cannot read other users' data, whatever the model is told.
• Authorization in code. Check rights before each tool call, outside the model. A prompt that says "do not" is not a control.
• Approval for side effects. Sending, paying, deleting and sharing need a human check that shows the real action. The MCP specification says there "SHOULD always be a human in the loop with the ability to deny tool invocations" (MCP Tools spec).
• Closed output path. No auto-rendered remote images, no auto-followed links built from model output.
• Logging. Keep tool calls with arguments and identity, retrieved sources and refusals.
• Sandbox for code. No secrets in the environment, no network egress unless a task needs it.

Human approval bypass shows how approval gates fail in practice.

Which frameworks apply?#

Three references cover most buyer questions. The OWASP Top 10 for LLM Applications 2026 gives LLM01 Prompt Injection and LLM03 Excessive Agency, the two entries that matter most for agents. OWASP's Top 10 for Agentic Applications adds ASI01 Agent Goal Hijack, ASI02 Tool Misuse, ASI03 Identity & Privilege Abuse, ASI04 Agentic Supply Chain Vulnerabilities, ASI05 Unexpected Code Execution, ASI06 Memory & Context Poisoning, ASI07 Insecure Inter-Agent Communication, ASI08 Cascading Failures, ASI09 Human-Agent Trust Exploitation and ASI10 Rogue Agents (OWASP blog). MITRE ATLAS supplies technique IDs for reports; see MITRE ATLAS.

None of these is a certification. Name the edition in a report and map each finding to one primary ID.

What should you check before an agent launches?#

Ten items. A "no" on any of them is a finding to fix or an accepted risk to record.

1. Every tool is listed with its arguments, credential and reach.
2. Each tool has a scoped credential, not a shared admin key.
3. Authorization runs in code before each tool call.
4. Two-user tests show no cross-user reach.
5. Actions with side effects need approval, and the approver sees the real action.
6. Marker tests in every input source produced no tool call, over repeated runs.
7. Memory writes are reviewed, and entries expire.
8. Model output is encoded or validated before it reaches a browser, shell or query.
9. Remote images and auto-followed links are blocked in agent output.
10. Tool calls are logged with identity, and someone reads the log.

Testing shows exposure at the time of the test. It does not show that an agent is secure. An AI agent security assessment tests these items against your agent in an agreed scope and reports the evidence for each finding.