AI red teaming methodology
AI red teaming methodology in eight steps: threat model, test plan, conditions, repeated attempts, evidence, impact, fixes and retest, with safety rules.
Short answer
Our AI red teaming method has eight steps: threat model, test plan, test conditions, execution with attempt counts, evidence, impact, fix guidance and retest. Each finding states the conditions, the observed behavior and how often it happened. Model output varies, so we repeat tests. We use test data and test accounts, and we test only with written authorization.
We assess AI applications in eight steps. Each step produces a record that you can read and check. The goal is to show how an AI application can leak data, cross access boundaries or take actions it should not.
NIST AI 600-1 defines AI red-teaming as "a structured testing exercise used to probe an AI system to find flaws and vulnerabilities". Microsoft's AI Red Team says it aims to push beyond model-level benchmarks by emulating real-world attacks on end-to-end systems (arXiv 2501.07238). We follow that end-to-end view. What is AI red teaming has the definitions.
What are the eight steps?#
| Step | What we do | What you receive |
|---|---|---|
| 1. Threat model | Map data sources, users, roles, tools, credentials and output paths. Name what must not happen. | A one-page model that you confirm |
| 2. Test plan | List tests for each risk, in priority order, with stop conditions | The agreed plan and scope in writing |
| 3. Test conditions | Fix account, role, data, prompt version, model setting, tool set and starting memory | A condition record for each test |
| 4. Execution | Run each test several times and count results | Attempt counts and success rates |
| 5. Evidence | Save transcripts, tool-call logs, retrieved passages and proof markers | An evidence index |
| 6. Impact | State who is affected and what an attacker gains, with the conditions it needs | A written impact for each finding |
| 7. Fix guidance | Give specific changes, ordered by effect | Fix steps for your team |
| 8. Retest | Repeat the failed tests after your fix | Retest results in the report |
How do we handle nondeterminism?#
Model output changes between runs, so one attempt proves little. For each test we record the number of attempts and the number that succeeded. We report that rate with the conditions. One success shows that the path exists, and the rate shows how often it works.
We also record the model version and settings. If you change the model, the prompt or the tools, earlier results may not hold. We say so in the report and suggest retest triggers.
What counts as evidence?#
A finding needs proof of a real effect, and a strange answer does not count as proof.
| Finding type | Proof we require |
|---|---|
| Data exposure | A seeded marker from a protected record appears where it should not |
| Cross-tenant access | A test tenant's record returns to another tenant's session |
| Unintended action | The tool-call log shows the call, and a read-back shows the changed test record |
| Output handling | The browser loads an attacker-chosen URL, shown in the network log |
Each result carries a label. "Observed" means we reproduced it. "Inferred" means we reason that a path exists but did not show it. We keep inferred items apart from confirmed impact. A section lists controls that held and limits of the test.
How do we handle safety and data?#
We test only with written authorization from the system owner.
- We use test tenants, test users and fake seeded data.
- Actions with side effects, such as sending email or changing records, target test-owned objects.
- We bound tests of cost, such as long inputs or paid APIs, and state the cost in the report.
- Never send production secrets or real customer data through the scoping form. We agree a safe channel during scoping.
- You name a contact who can stop testing.
What do we not promise?#
We do not say a system is secure after our tests. We cannot promise to prevent prompt injection. The UK NCSC says it may never be totally mitigated. Our results describe what we tried, under stated conditions, on a stated date.
The example report shows a labeled lab example, and frameworks shows how findings map to public frameworks. The service pages cover LLM applications, agents and MCP. To start, go to scoping.
Questions
Is AI red teaming the same as a safety evaluation?
No. Microsoft's AI Red Team wrote that AI red teaming is not safety benchmarking. We test an end-to-end application against attacks on data, access and actions. A model benchmark scores the model alone.
Do you automate the attacks?
We use scripts to repeat attempts and count results. Excession Labs checks the evidence for each finding before it enters the report. A scripted run that cannot show a protected marker or a real action is not reported as a finding.
Sources
- Microsoft AI Red Team: Lessons From Red Teaming 100 Generative AI Products (arXiv 2501.07238) (checked 9 November 2026)
- NIST AI 600-1: Generative AI Profile (definition of AI red-teaming) (checked 9 November 2026)
- OWASP GenAI Red Teaming Guide (checked 9 November 2026)
- UK NCSC: Prompt injection is not SQL injection (checked 9 November 2026)
Shipping an AI feature or agent?
Describe the system, the data it reads and the actions it can take. We reply with questions and a proposed scope.