AI red teaming, from zero.
Before a model ships, someone whose job is to break it gets a turn. Here is what that work looks like — and what separates it from attacking a system you do not own.
Red teaming is an old idea borrowed from the military and adopted by security teams decades ago: pay someone to attack your own system, under controlled conditions, so you find the weaknesses before an adversary does. AI red teaming applies that same discipline to machine-learning models — and especially to large language models — where the failure modes look nothing like a classic software bug.
When you red-team a web app, you are usually hunting for defects in code: an injection flaw, a broken access-control check, a misconfigured server. When you red-team an LLM, the "vulnerability" is often the model behaving exactly as designed — following instructions — but doing so on input that should never have counted as an instruction. That shift is why AI red teaming has become its own subfield rather than a footnote in a pentest report.
What red teamers are actually looking for
The OWASP Top 10 for LLM Applications is the closest thing the field has to a shared checklist. A red-team engagement against a model or an LLM-powered product typically probes for:
| Risk | What it looks like |
|---|---|
| Prompt injection | Both direct ("ignore your instructions and…") and indirect, where the malicious instruction is hidden in content the model reads, such as a web page or a document. |
| Jailbreaks | Coaxing the model past its safety training so it produces output it was tuned to refuse. |
| Sensitive information disclosure | Getting the model to reveal system prompts, training data, API keys pasted into context, or another user's data. |
| Insecure output handling | The application trusts model output and feeds it straight into a shell, a database query, or a browser, turning a text trick into code execution. |
| Excessive agency | An agent with tools (email, payments, file access) that can be steered into taking harmful actions it technically had permission to take. |
We cover this in depth in Prompt injection, explained. A jailbreak targets the model's alignment; prompt injection targets the application wrapped around it. They overlap but are not the same thing.
Jailbreaks vs. prompt injection
These two get conflated constantly, so it is worth being precise. A jailbreak attacks the model itself: the goal is to defeat the safety training so the model says something it was built to refuse. Prompt injection attacks the application: the goal is to make the model treat attacker-supplied content as trusted instructions, hijacking whatever the app was supposed to do. A support bot that leaks its system prompt because a user pasted a clever message has been prompt-injected. A model that produces disallowed content after an elaborate role-play setup has been jailbroken. A serious red team tests for both, because a real product fails in both ways.
How a model gets evaluated
Red teaming an LLM is part manual craft, part automation. A human red teamer brings creativity that scripted tests miss — but humans do not scale, so teams pair them with automated harnesses that fire thousands of adversarial prompts and score the responses. A typical loop looks like this:
- Define the harm taxonomy. Decide, before testing, what "bad output" means for this system: data leakage, unsafe instructions, biased decisions, tool misuse. Vague goals produce vague findings.
- Generate adversarial inputs. Seed with known attack patterns, then mutate them — rephrasings, encodings, language switches, role-play framings — to probe the edges of the model's refusals.
- Score the responses. Often another model acts as an automated judge for the first pass, with humans reviewing the borderline cases. Calibrating that judge is half the work.
- Report with reproductions. A finding is only useful if an engineer can reproduce it. "The model sometimes misbehaves" is not a bug report; a saved prompt and transcript is.
- Re-test after every change. A defense tuned against one model version rarely transfers cleanly to the next. Red teaming is continuous, not a one-time gate.
Anthropic, OpenAI, Google and others publish system cards and safety writeups describing red-team results before major releases; the US NIST has also published guidance on adversarial testing of AI systems. Reading those is the fastest way to see what mature evaluation looks like in practice.
Where the lines are
This is the part that matters most for a reader learning the field. Red teaming is authorized by definition — you are testing a system you own, or one you have explicit written permission to test, within an agreed scope. The moment you send crafted inputs at someone else's production model to see what breaks, you are no longer red teaming; you are attacking a system without authorization, which is both against every provider's terms of service and, depending on where you live, against the law.
If you want hands-on practice, the safe path is your own lab: run a local model, stand up a deliberately vulnerable LLM app, or use one of the public capture-the-flag style challenges built for exactly this. Start with running a local LLM so every experiment stays on hardware you control.
The one-line summary: AI red teaming is the discipline of breaking your own models on purpose, in a controlled way, so the failures show up in a report instead of in the news. The techniques are learnable, the tooling is maturing fast, and the single hard rule — only test what you are authorized to test — is what separates a red teamer from an attacker.