Adversarial machine learning: the NIST AI 100-2 attack types, explained for LLM apps

Adversarial machine learning (AML) is the study of attacks on machine learning systems and of the defenses against them. The reference vocabulary is NIST AI 100-2 E2025, "Adversarial Machine Learning: A Taxonomy and Terminology of Attacks and Mitigations", dated March 2025. The sections below explain its attack classes in plain language and show how each one becomes a test case for an LLM application.

What is adversarial machine learning?#

Adversarial machine learning covers attacks that make an ML system fail on purpose, leak what it learned, or lose availability. A classic example is an image with a tiny change that a person cannot see, which makes a classifier return the wrong label. A newer example is a web page with hidden text that tells a chatbot to send a user's data to an attacker.

The field has two halves. Predictive AI (PredAI) means models that classify or predict, such as fraud scores or image classifiers. Generative AI (GenAI) means models that produce text, images or code, including LLMs. NIST treats them in separate chapters because the attacks differ. The overlap is large: poisoning, privacy attacks and supply chain attacks occur in both.

For a team that ships an LLM feature, the question is which classes apply to its design. If you call a hosted model, you rarely train it, so training-time attacks sit with your model vendor. You still own the retrieval index, the prompts, the tools and the output handling. Attacks through those parts are in scope for your tests.

NIST states the document is voluntary. It "does not establish any legal standard" and has no force of law, so it is a vocabulary and a reference, not a compliance test.

What does NIST AI 100-2 E2025 cover?#

The report classifies attacks by the attacker's objective, then by the stage of the model life cycle and by the attacker's capabilities and knowledge. The five objectives carry IDs in the report.

Chapter 2 covers PredAI: evasion (section 2.2), poisoning (2.3) and privacy attacks (2.4). Chapter 3 covers GenAI: supply chain attacks (3.2), direct prompting (3.3), indirect prompt injection (3.4) and the security of agents (3.5). Chapter 4 discusses key challenges, including trade-offs between trustworthiness attributes and theoretical limits on resisting attacks.

The IDs give testers a shared label. A finding can say "NISTAML.015 Indirect Prompt Injection" and a reader who knows the taxonomy sees the class at once. For attack steps and case studies, pair the taxonomy with MITRE ATLAS.

What are evasion attacks?#

An evasion attack changes an input at deployment time so the model gives a wrong answer. NIST (NISTAML.022) describes the adversary's goal as generating adversarial examples: samples whose classification can be changed to a class the attacker chooses, often with minimal perturbation.

The attack needs only access to the model's inputs and outputs. In the white-box case the attacker knows the model and can compute gradients. In the black-box case the attacker only queries the model, which NIST tracks as NISTAML.025. The report also covers transferability: an adversarial example built against one model can often fool another.

Evasion matters most when you train or fine-tune your own classifier, for example a content filter, a spam model or a document classifier that gates an LLM pipeline. It matters less for a plain chat feature on a hosted model. If a guard model sits in front of your LLM, that guard is an ML classifier and can be evaded.

Mitigations are limited. NIST says that "many of the proposed mitigations against adversarial examples have been ineffective against stronger attacks". It names three classes that have proven resilient: adversarial training, randomized smoothing and formal verification. All three trade resilience against accuracy and add computational cost. The report cites work that bypassed 10 detection methods and 13 proposed defenses with adaptive attacks.

A test for evasion sends crafted variants of normal inputs to the classifier and records the rate at which labels flip. For a guard model, the variants are rephrasings, encodings and split messages. The test report should state the attack budget, the access level and the version tested.

What are poisoning and backdoor attacks?#

Poisoning attacks corrupt what the model learns from. NIST sorts them by effect: availability poisoning, targeted poisoning (NISTAML.024), backdoor poisoning (NISTAML.023) and model poisoning (NISTAML.026 and NISTAML.051). Data poisoning in general is NISTAML.013.

The three effects differ.

NIST notes that backdoor attacks are the only attacks in the literature that need control of both the training data and the testing data. Model poisoning changes the parameters directly, as in federated learning where a participant sends bad updates.

In an LLM application, poisoning has more entry points than the base model's pre-training. A retrieval index that ingests user documents, a fine-tuning set built from support tickets and an agent's long-term memory can all take attacker-controlled content. The test case is to insert a marked document through a normal channel and see whether it later changes an answer or an action. Our guides on data poisoning attacks and agent memory poisoning give worked examples.

For GenAI supply chains, section 3.2.3 suggests checking cryptographic hashes of downloaded data and treating models as untrusted system components. NIST adds that finding poisoned samples in a large corpus may be very difficult.

What are privacy attacks?#

Privacy attacks extract information about the training data or the model. NIST puts them under Privacy compromise (NISTAML.03). It names these attacks:

• Data reconstruction (NISTAML.032): reverse-engineering private records or inputs from access to a trained model.
• Membership inference (NISTAML.033): deciding whether a given record was in the training set.
• Property inference (NISTAML.034): learning global facts about the training data, such as the share of records with a sensitive attribute.
• Model extraction (NISTAML.031): learning about the model itself, such as its parameters or behavior, through queries.
• For GenAI: training data extraction and data extraction (NISTAML.038), and prompt extraction (NISTAML.035).

NIST's report separates the terms with care, because "privacy" in AML covers both leaking training data and leaking the model. In a security report, name which one you mean.

Privacy tests need query access and a plan. For membership inference, you need records known to be in the training data and records known to be out. For a hosted model you cannot do this against the vendor's training set, but you can do it against a fine-tune you made on your own data. For extraction, the test counts how many queries it takes to approximate the model's behavior, which shows whether query limits and monitoring would catch it. See model extraction attacks and system prompt leakage.

How does AI 100-2 treat generative AI and agents?#

Chapter 3 is the part most LLM teams need. Section 3.3 covers direct prompting attacks, where the attacker is the user and writes the malicious prompt. Section 3.4 covers indirect prompt injection (NISTAML.015), where the instructions arrive through data the model retrieves, such as a web page, email or document. Section 3.5 covers agents.

On agents, NIST says they "can be vulnerable to the many of the above categories of attacks", including direct and indirect prompt injection, and that attacks can hijack agents "to execute arbitrary code or exfiltrate data". The generative AI privacy IDs also include compromising connected resources (NISTAML.039) and leaking information from user interactions (NISTAML.036).

Section 3.4.4 holds the line that matters for planning: "Because current mitigations do not offer full protection against all attacker techniques, application designers may design systems with the assumption that prompt injection attacks are possible if a model is exposed to untrusted input sources."

That sentence changes how you build. You do not rely on one filter. You limit what the model can do with the output, for example by using models with different permissions or by letting a model reach untrusted data only through well-defined interfaces, as NIST suggests. Our guides on prompt injection, indirect prompt injection and excessive agency cover those controls.

Which mitigations exist and how far do they go?#

Every class has proposed mitigations, and none removes the risk. The table summarizes what NIST names and what remains open.

Treat the table as a map and read the report for the details of each method. Chapter 4 also discusses theoretical limits on resisting adversarial attacks and the trade-offs between attributes of trustworthy AI, so a gain in resilience can cost accuracy or other properties.

For regulators, the point is narrower. The EU AI Act asks high-risk systems to include measures against data poisoning, model poisoning, adversarial examples or model evasion, confidentiality attacks and model flaws "where appropriate" (Article 15(5)). Those categories match the AML classes above. The Act does not name a test method, and it is not legal advice here either. See the EU AI Act guide.

How do these attack classes become test cases?#

Each class maps to a test you can run against an LLM application, and to a catalog entry you can cite. This table pairs the NIST class with an OWASP LLM Top 10 2026 entry and an ATLAS technique from release v2026.10.

The mapping is ours, built from the IDs in the three catalogs, and it is a starting point. A tester picks the cases that fit the design: a chatbot with no tools skips the agent rows, and an app with no fine-tune skips membership inference.

A report like this works because it has a control and a marker that proves the effect. Add the model version and the date. The OWASP AI Testing Guide offers a wider test catalog, and our methodology page explains how we label evidence. Read the frameworks page for how the catalogs fit together.