garak vs giskard-oss
NVIDIA's LLM vulnerability scanner: nmap-style probing for jailbreaks, prompt injection, data leakage, toxicity and hallucination across dozens of model endpoints. — versus — Giskard v3: modular Python evals and red-teaming for agentic systems — scenario-based checks with LLM-as-judge, plus an automatic vulnerability scanner across OWASP LLM Top-10 categories.
Same scanning job, different shapes: garak is NVIDIA's nmap-style standalone prober across dozens of model endpoints; giskard-scan is a library-first scanner designed to live inside your Python test suite next to your evals.
| garak | giskard-oss | |
|---|---|---|
| Stars | 8.5k | 5.6k |
| Forks | 1.1k | 494 |
| Language | Python | Python |
| License | Apache-2.0 | Apache-2.0 |
| Last activity | 2 days ago | today |
| Topics | security | evals, security |
| Curated connections | 4 | 3 |
garak — the curator's take
Run it before anything LLM-shaped ships: static, dynamic and adaptive probes for jailbreaks, injection, leakage and toxicity, with reports that name the failing probe and prompt — nmap for language models. Speaks to HF, OpenAI-compatible, Ollama and REST endpoints, so it tests what you actually deploy. NOT a guardrail — it finds holes, it doesn't plug them (pair with a runtime defense); a full probe run takes hours, and a clean scan tests the MODEL, not your product logic around it. Authorized targets only.
giskard-oss — the curator's take
One of the OG names in LLM testing, freshly rewritten: v3 drops the heavyweight monolith for focused packages — giskard-checks (scenario API for multi-turn agent evals: groundedness, conformity, LLM-judge) and giskard-scan (generates adversarial suites from a plain-language description of your agent, prompt-injection probes included). Async-first, wraps anything callable. Mind the transition, though: v3 is beta, RAG evaluation (RAGET) hasn't been ported yet, and v2 — where scan and RAG eval are mature — is no longer maintained. Python 3.12+ only, and libraries on giskard-core send aggregated telemetry (opt-out documented). Pick ragas for pure RAG metrics; Giskard earns its place on multi-turn agent scenarios plus red-teaming in one stack.