# Model extraction attack: how it works and how to test for it

Source: https://airedteamingservices.com/guides/model-extraction-attacks/
Publisher: Excession Labs (airedteamingservices.com)
Last updated: 2026-11-09

A model extraction attack copies a model through its API. Learn how it works, how it differs from inversion, where OWASP lists it, how to test and what to limit.

> **Short answer:** A model extraction attack copies a model's behavior by sending many queries to its API and training a substitute on the answers, with no access to the weights. Testers check query limits, how much detail each response exposes and whether one account can spread the load. Controls are quotas, trimmed outputs and monitoring. They raise the cost of copying and do not remove it.

A model extraction attack copies a model's behavior through its public interface. The attacker sends many queries to a hosted model, records the answers, and trains a substitute model on them. The attacker never needs the weights or the training data.

Teams that expose a model through an API should know what this costs them and how to check their own exposure. The sections below define the attack, separate it from related ones, say where OWASP lists it, and describe a short outside-in test.

## What is a model extraction attack?

Tramèr et al. defined the setting in 2016: an adversary with black-box access, "but no prior knowledge of an ML model's parameters or training data, aims to duplicate the functionality of (i.e., 'steal') the model" ([arXiv:1609.02943](https://arxiv.org/abs/1609.02943)). They attacked the prediction services of BigML and Amazon Machine Learning and extracted logistic regression, neural network and decision tree models with near-perfect fidelity.

The motive is usually money or access. A model can be confidential because of its training data, its commercial value or its use in security tasks, as the same paper notes. A copy lets an attacker avoid per-query fees, probe the copy offline for weaknesses, or compete with the original.

Four attacks share the same API surface and get confused:

| Attack | What the attacker wants | Needs |
|---|---|---|
| Model extraction (stealing) | A copy of the model's behavior | Many input and output pairs |
| Model inversion | Information about the training data, rebuilt from outputs | Query access, often with scores |
| Membership inference | Whether one record was in the training set (NISTAML.033) | Query access and a candidate record |
| Training data extraction | Memorized training text (NISTAML.038 Data Extraction) | Prompts that trigger memorization |

NIST AI 100-2 E2025 gives the last two their own IDs under the Privacy objective ([NIST AI 100-2 E2025](https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.100-2e2025.pdf)). This page covers the first row. For leaked prompts see [system prompt leakage](/guides/system-prompt-leakage/).

## How does an attacker copy a model through an API?

The attacker treats the API as a labeling service. The loop has four steps: choose inputs, send them, store the outputs, train a substitute that matches them. More queries and richer outputs give a closer copy.

Output detail matters. Tramèr et al. observed that services may accept partial inputs and return confidence values with predictions, and they showed simple, efficient attacks that use these features. They also showed that "the natural countermeasure of omitting confidence values from model outputs still admits potentially harmful model extraction attacks."

Large language models are in scope too. Carlini et al. introduced "the first model-stealing attack that extracts precise, nontrivial information from black-box production language models like OpenAI's ChatGPT or Google's PaLM-2". Their attack recovers the embedding projection layer of a transformer, up to symmetries, with typical API access. For under $20 it extracted the entire projection matrix of OpenAI's Ada and Babbage models and confirmed hidden dimensions of 1024 and 2048. They estimated that recovering the projection matrix of gpt-3.5-turbo would cost under $2,000 in queries ([arXiv:2403.06634](https://arxiv.org/abs/2403.06634)).

Two points follow for a defender. The cost of an attack on a production model can be small compared with the cost of building the model. And what leaks may be a part of the model, not a full clone.

## Where does OWASP list model extraction?

OWASP moved the topic between editions. The 2023/24 list had LLM10 Model Theft, described as unauthorized access and exfiltration of LLM models, including when "weights and parameters are extracted to create a functional equivalent" ([OWASP LLM10 Model Theft](https://genai.owasp.org/llmrisk2023-24/llm10-model-theft/)).

The 2026 edition has no Model Theft entry. Its LLM06:2026 Unbounded Consumption covers an application that allows excessive, uncontrolled inference, which "causes outage, cost, or model cloning" ([OWASP 2026 entries](https://github.com/GenAI-Security-Project/GenAI-LLM-Top10/tree/main/2026/final)). So a report on the 2026 edition files extraction under LLM06:2026, and a report that follows the 2023/24 list files it under LLM10. Name the edition. The rate-limit and cost-cap tests belong to the [OWASP LLM Top 10 guide](/guides/owasp-top-10-for-llm-applications/).

Other frameworks name it as well:

- MITRE ATLAS has AML.T0005.001 Replicate Behavior from Model Outputs, and AML.T0024 Exfiltration via AI Inference API, which covers information returned through ordinary API response fields ([ATLAS v2026.10](https://github.com/mitre-atlas/atlas-data/releases/download/v2026.10/ATLAS-2026.10.yaml)). See [MITRE ATLAS](/guides/mitre-atlas/).
- NIST AI 600-1 action MS-2.7-007 suggests red teaming against ML attacks that include model extraction, and MS-2.10-001 covers privacy testing for model extraction and membership inference ([NIST AI 600-1](https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.600-1.pdf)).

## How do testers check for model extraction from outside?

A black-box tester asks one question: how much can one account learn about the model, and how fast, before anything stops it? Test only models and APIs you own or have written authorization to test, and set a cost cap before sending volume.

| Check | How to run it | Safe result | Evidence |
|---|---|---|---|
| Rate limits | Send requests at rising rates from one account up to the cap | A limit applies per account and per key, with a clear error | Request counts, status codes, headers |
| Quotas | Run to the daily and monthly quota | Quota ends the run; the account is flagged | Quota response, alert record |
| Output detail | Compare responses with and without optional fields | Scores, logprobs and embeddings are off unless needed | Field list per endpoint |
| Account spread | Split the same workload over several new accounts | Linked accounts hit shared limits or trigger review | Per-account counts, review result |
| Cost control | Estimate spend for a large run | Per-user spend cap stops it | Cost log |
| Monitoring | Run a systematic, wide-coverage query pattern | An alert fires on the pattern | Alert time and content |

Keep the volume small. The aim is to show whether the controls exist and respond, and a full extraction run is unnecessary for that. If the test needs scale, agree the cap in the scope first.

> **Demonstration:** This is a demonstration against Harbor Classify, a fictional text classification API at api.example.com. No real customer system was tested.
> 
> Check: output detail and rate limit on one free-tier key.
> 
> ```
> request:  POST /v1/classify  {"text": "..."}
> response: {"label": "refund", "scores": {"refund": 0.91, "billing": 0.07, ...}}
>           -> full score vector returned on every call
> 
> ramp:     10 req/s ok, 50 req/s ok, 200 req/s ok (no 429 up to 200 req/s)
> quota:    none observed after 20,000 requests (test stopped at agreed cap)
> alerts:   none received at the contact named in scope
> ```
> 
> Finding: the API returns the full score vector and shows no rate limit, quota or alert up to the agreed cap. Severity depends on the value of the model. Fix: return the top label only unless a customer needs scores, add per-key limits and quotas, and alert on high-volume, wide-coverage query patterns.
## Which controls reduce model extraction?

No control removes the risk while the model answers queries. Controls raise the cost and the chance of being caught.

- Quotas and rate limits per account, key and network range, with a spend cap.
- Trimmed outputs. Return the label, not the score vector. Remove logprobs and embeddings unless a use case needs them. Tramèr et al. show this is not enough alone.
- Monitoring. Alert on high volume, unusual input coverage and many linked accounts.
- Identity. Require sign-up checks for paid tiers so that account spread costs effort.
- Terms and legal controls. They do not stop an attacker, and they give you a basis to act.
- Watermarks and fingerprints. They can help show that a copy came from your model. They do not prevent copying, and their strength varies. We do not rate any specific method here.

Plan for the case where a copy appears. If a competitor model matches yours on odd inputs, evidence from your own logs of a systematic query pattern is worth more than an argument about similarity.

## What does a model extraction finding contain?

A finding names the endpoint, the output fields returned, the limit that was absent or too high, the volume sent and the cost. It records the controls that held, such as a per-key quota or a working alert. It states the fix in the same terms the OWASP entry uses: control inference volume and cost. And it labels what was not tested, such as a full extraction run.

Teams with a hosted model API can add these checks to an [LLM application security assessment](/llm-application-security-assessment/). The wider topic of AI risk is in [AI security](/guides/ai-security/).

## Questions

### Is model extraction the same as model stealing?

Yes, in most writing. Both terms describe an attacker who queries a model and uses the answers to build a copy. Tramèr et al. used 'model extraction' in 2016 and put 'steal' in the paper title. Pick one term and define it once in your report.

### What is the difference between model extraction and model inversion?

Extraction copies what the model does. Inversion tries to recover information about the data the model was trained on. Membership inference asks whether one record was in the training set. NIST lists the privacy attacks (such as NISTAML.033 Membership Inference) separately from model copying.

### Can an attacker steal an LLM through its API?

Partly, yes. Carlini et al. (2024) recovered the embedding projection layer of OpenAI's Ada and Babbage models for under $20 in queries and estimated under $2,000 for gpt-3.5-turbo. That is part of a model, not the whole model, and the paper also discusses defenses.

### Does hiding confidence scores or logprobs stop extraction?

It makes it harder and does not stop it. Tramèr et al. showed that omitting confidence values from outputs still admits potentially harmful extraction attacks. Treat output trimming as one layer next to quotas and monitoring.

### Where does OWASP list model extraction?

The 2023/24 list had LLM10 Model Theft. The 2026 list covers cloning under LLM06:2026 Unbounded Consumption, which the entry says can cause outage, cost or model cloning.

## Sources

- [Tramèr et al.: Stealing Machine Learning Models via Prediction APIs (arXiv:1609.02943, 2016)](https://arxiv.org/abs/1609.02943)
- [Carlini et al.: Stealing Part of a Production Language Model (arXiv:2403.06634, March 2024)](https://arxiv.org/abs/2403.06634)
- [OWASP Top 10 for LLM Applications 2026: final entries (LLM06 Unbounded Consumption)](https://github.com/GenAI-Security-Project/GenAI-LLM-Top10/tree/main/2026/final)
- [OWASP Top 10 for LLM Applications 2023/24: LLM10 Model Theft](https://genai.owasp.org/llmrisk2023-24/llm10-model-theft/)
- [MITRE ATLAS v2026.10: AML.T0005.001 Replicate Behavior from Model Outputs, AML.T0024 Exfiltration via AI Inference API](https://github.com/mitre-atlas/atlas-data/releases/download/v2026.10/ATLAS-2026.10.yaml)
- [NIST AI 600-1: Generative AI Profile, actions MS-2.7-007 and MS-2.10-001](https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.600-1.pdf)
- [NIST AI 100-2 E2025: Adversarial Machine Learning taxonomy](https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.100-2e2025.pdf)