English | EN 
LLM red teaming has a few years of practice behind it. We are familiar with adversarial prompts, jailbreak taxonomies and automated attack generation. Most of it assumes the model’s only output is text. However, agents break that assumption, and much of the standard toolkit needs rethinking as a result. This article looks at what changes and what does not when you move from red teaming a chatbot to red teaming an agent.
Chatbots: attacking the completion
Classic LLM red teaming is mostly about the prompt-response link: can you get a model to say something it shouldn’t? The main attack types are familiar: straightforward jailbreak prompts (roleplay, “just hypothetically…,” encoding tricks), slow multi-turn “steering” where each message looks harmless but the conversation drifts into unsafe territory, and prompt injection hidden inside text the model is asked to process. Even without an attacker, models also fail in everyday ways: hallucinations, inconsistent refusals, and biased outputs inherited from training data.
Because this problem is well-studied, the defenses are relatively mature. Teams use automated test suites to generate adversarial prompts, human red teams to discover new jailbreak patterns, and output checks to catch violations before they reach the user. Importantly, the damage is usually contained as the model’s response stays inside the chat.
Agents: attacking the trajectory
Add tools, memory, and planning, and you have shifted to testing an entire end-to-end workflow. An agent produces a chain of steps where each step becomes the input to the next, so problems look different at each layer.
The biggest shift is the risk of indirect prompt injection. Agents constantly ingest outside content, web pages, emails, documents, logs, and that content can contain instructions that the agent mistakenly treats as higher priority than the user’s goal. OWASP keeps prompt injection at the top of its risk lists for this reason: the model processes instructions and data in one channel, with no reliable way to tell content to act on from content to merely process.
Errors also compound differently. In a chatbot, one hallucinated turn is usually just one wrong answer. In an agent, a hallucination early in a multi-step task can become a bad assumption that every later step builds on. Catching it requires step-level visibility tracing.
Finally, agents expand the action surface. When an agent can use tools and credentials, the risk becomes related to doing something wrong (as opposed to saying something wrong in the case of chatbots). That’s what OWASP calls excessive agency: an over-permissioned agent can misuse tools, leak data or take irreversible actions. Red teaming here starts to look like classic security testing: least-privilege design, blast-radius thinking, fuzzing tool inputs, and looking for “confused deputy” failures where the agent is tricked into using legitimate access for the wrong purpose. It also includes supply-chain style scenarios, like a compromised tool or connector (for example, an MCP server) that changes behavior after it’s been trusted, via poisoned tool descriptions or “rug-pull” updates.
Where the two overlap
Agent red teaming sits on top of chatbot red teaming. The base model is still vulnerable to classic jailbreaks and the agent harness often makes it worse. The BrowserART benchmark found GPT-4o refused 88% of harmful requests as a chatbot but executed 74% of the same behaviors as a browser agent with no jailbreak required.
What’s still unsolved
We’re starting to get better labels for the problem, and organizations such as OWASP are publishing agent-focused risk lists. However, the practical playbook is still emerging: how to evaluate whole tool-using runs, what “safe by default” sandboxing should look like, and how to scope permissions so an agent can be useful without being able to do too much. As agents become more autonomous, the toughest work will be building reliable guardrails for each step of the process.