Red teaming is deliberately trying to make a system misbehave before someone else does.
Ordinary evaluation asks whether it works, red teaming asks how it can be made to fail.
The targets are harmful output, leaked data, broken rules and unauthorised tool use.
Attacks are found by people, generated by models, and collected from what real users try.
Every finding becomes a permanent test case, because a fixed jailbreak returns after the next update.
Standard evaluation samples the inputs you expect and reports how well the system does on them.
Red teaming assumes an adversary and asks what the worst reachable behaviour is.
The name comes from security exercises where one team attacks and another defends.
For a language system the objectives are concrete and worth enumerating.
Getting harmful content out of a model that was trained to refuse it.
Extracting the system prompt, private data, or fragments of training material.
Making the system break its own rules through indirect instructions in retrieved content.
Driving an agent to take an action it was never authorised to take.
The techniques range from role-play framings and hypotheticals to encoding and language switching.
Indirect prompt injection is the most serious in agent systems, since the attacker never talks to the model.
They plant instructions in a page or a document the agent will read while doing something else.
Attacks come from three sources: expert humans, models generating adversarial prompts at scale, and real users.
The third is easy to overlook and is the most representative of what will actually be tried.
Every successful attack must become a permanent regression test.
Without that, a fix survives until the next model update quietly reopens the hole.
One question on this card. Every option explains itself once you pick, right or wrong.
18 concepts in this module, then a graded test with every answer explained.