deepteam

Open-source red-teaming framework for LLM systems: 50+ vulnerabilities, jailbreak/injection/multi-turn attacks against agents, RAG pipelines and chatbots — plus guardrails. Runs locally.

2,233 363 Python Apache-2.0updated 8 days ago

security evals

View on GitHub Dispute this mapping →

Curator's take

Penetration testing for LLM apps with a framework's ergonomics: pick from 50+ documented vulnerabilities (bias, PII leakage, SQL injection via prompt) and simulated attacks including multi-turn exploitation, run locally, then wire the matching guardrails. Built on DeepEval, so red-team results plug into an eval workflow rather than dying in a report. NOT neutral infrastructure — it funnels toward the Confident AI platform for storing results (self-host your own reporting if that matters), and like every attack library it tests known patterns: a clean run is a floor, not a clearance.

Mapped by ShipWithAI editors · links verified

README.md

DeepTeam.

The LLM Red Teaming Framework

Documentation | Vulnerabilities, Attacks, and Features | Getting Started | Confident AI

DeepTeam is a simple-to-use, open-source red teaming framework for LLM systems. Think of it as penetration testing, but for LLMs.

DeepTeam simulates attacks — jailbreaking, prompt injection, multi-turn exploitation, and more — to uncover vulnerabilities like bias, PII leakage, and SQL injection in your AI agents, RAG pipelines, and chatbots. It also offers guardrails to prevent these issues in production.

DeepTeam runs locally on your machine and is built on DeepEval, the open-source LLM evaluation framework.

[!IMPORTANT] Need a place for your red teaming results to live? Sign up to the Confident AI platform to manage risk assessments, monitor vulnerabilities in production, and share reports with your team.

Confident AI + DeepTeam

Want to talk LLM security, need help picking attacks, or just to say hi? Come join our discord.

🔥 Vulnerabilities, Attacks, and Features

📐 50+ ready-to-use vulnerabilities (all with explanations) powered by ANY LLM of your choice. Each vulnerability uses LLM-as-a-Judge metrics that run locally on your machine to produce binary pass/fail scores with reasoning:
- Data Privacy
  - PII Leakage — disclosure of sensitive personal information
  - Prompt Leakage — exposure of system prompt secrets and instructions
- Responsible AI
  - Bias — stereotypes and unfair treatment across gender, race, religion, politics
  - [Toxicity](https://www.trydeep

Continue your stack

What teams reach for next — and why each earns a place beside deepteam. Ranked by curator confidence.

Alternative to See all + compare →

garak

NVIDIA's LLM vulnerability scanner: nmap-style probing for jailbreaks, prompt injection, data leakage, toxicity and hallucination across dozens of model endpoints.

Why it fitsSame job — adversarial testing of LLM systems. garak is the nmap-style scanner you point at a model endpoint; DeepTeam is the framework you embed in your eval suite, with attack simulation and guardrails in one loop.

giskard-oss

Giskard v3: modular Python evals and red-teaming for agentic systems — scenario-based checks with LLM-as-judge, plus an automatic vulnerability scanner across OWASP LLM Top-10 categories.

Why it fitsBoth red-team LLM systems with framework ergonomics: deepteam ships 50+ documented vulnerabilities and matching guardrails; giskard-scan auto-generates adversarial suites from a plain-language agent description and pairs with the same stack's eval checks.