giskard-oss

Giskard v3: modular Python evals and red-teaming for agentic systems — scenario-based checks with LLM-as-judge, plus an automatic vulnerability scanner across OWASP LLM Top-10 categories.

5,574 494 Python Apache-2.0updated today

evals security

View on GitHub Dispute this mapping →

Curator's take

One of the OG names in LLM testing, freshly rewritten: v3 drops the heavyweight monolith for focused packages — giskard-checks (scenario API for multi-turn agent evals: groundedness, conformity, LLM-judge) and giskard-scan (generates adversarial suites from a plain-language description of your agent, prompt-injection probes included). Async-first, wraps anything callable. Mind the transition, though: v3 is beta, RAG evaluation (RAGET) hasn't been ported yet, and v2 — where scan and RAG eval are mature — is no longer maintained. Python 3.12+ only, and libraries on giskard-core send aggregated telemetry (opt-out documented). Pick ragas for pure RAG metrics; Giskard earns its place on multi-turn agent scenarios plus red-teaming in one stack.

Mapped by ShipWithAI editors · links verified

README.md

giskardlogo

Evals, Red Teaming and Test Generation for Agentic Systems

Modular, Lightweight, Dynamic and Async-first

Docs • Website • Community

[!IMPORTANT] Giskard v3 is a fresh rewrite designed for dynamic, multi-turn testing of AI agents. This release drops heavy dependencies for better efficiency while introducing a more powerful AI vulnerability scanner and enhanced RAG evaluation capabilities. For now, the vulnerability scanner and RAG evaluation still rely on Giskard v2. Giskard v2 remains available but is no longer actively maintained. Follow progress → Read the v3 Announcement · Roadmap

Install

pip install giskard

Requires Python 3.12+.

Telemetry: Libraries built on giskard-core (including giskard-checks) may send optional, aggregated usage analytics to help improve the product. No prompts, model outputs, or scenario text are included. See what is collected and how to opt out.

Giskard is an open-source Python library for testing and evaluating agentic systems. The v3 architecture is a modular set of focused packages — each carrying only the dependencies it needs — built from scratch to wrap anything: an LLM, a black-box agent, or a multi-step pipeline.

Status	Package	Description
✅ Beta	`giskard-checks`	Testing & evaluation — scenario API, built-in checks, LLM-as-judge
✅ Beta	`giskard-scan`	Agent vulnerability scanner — red teaming, prompt injection, data leakage (successor of v2 Scan)
📋 Planned	`giskard-rag`	RAG evaluation & synthetic data generation (successor of v2 RAGET)

Giskard Checks — create and apply evals for testing agents

pip install giskard-checks

Giskard Checks is a lightweight library for creating evaluations (evals) that test LLM-based systems — from simple assertions to LLM-as-judge assessments. Unlike traditional unit

Continue your stack

What teams reach for next — and why each earns a place beside giskard-oss. Ranked by curator confidence.

Alternative to See all + compare →

deepteam

Open-source red-teaming framework for LLM systems: 50+ vulnerabilities, jailbreak/injection/multi-turn attacks against agents, RAG pipelines and chatbots — plus guardrails. Runs locally.

Why it fitsBoth red-team LLM systems with framework ergonomics: deepteam ships 50+ documented vulnerabilities and matching guardrails; giskard-scan auto-generates adversarial suites from a plain-language agent description and pairs with the same stack's eval checks.

Ragas

Evaluation toolkit for your RAG and agent pipelines — faithfulness, relevance, and more.

Why it fitsOverlapping LLM-evaluation job: ragas is the specialist for RAG pipeline metrics; Giskard v3 covers multi-turn agent scenarios with built-in checks and LLM-as-judge, with RAG evaluation still pending its v3 port.

garak

NVIDIA's LLM vulnerability scanner: nmap-style probing for jailbreaks, prompt injection, data leakage, toxicity and hallucination across dozens of model endpoints.

Why it fitsSame scanning job, different shapes: garak is NVIDIA's nmap-style standalone prober across dozens of model endpoints; giskard-scan is a library-first scanner designed to live inside your Python test suite next to your evals.