deepeval vs giskard-oss
Pytest for LLM apps: 40+ research-backed metrics — G-Eval, RAG suite, agent task completion, hallucination — as unit tests you run in CI, judged by any LLM including local ones. — versus — Giskard v3: modular Python evals and red-teaming for agentic systems — scenario-based checks with LLM-as-judge, plus an automatic vulnerability scanner across OWASP LLM Top-10 categories.
Both are modular Python eval frameworks with LLM-as-judge and safety scanning. Giskard leans scenario-based checks and OWASP-mapped vulnerability scanning; DeepEval leans metric breadth and pytest-in-CI ergonomics.
| deepeval | giskard-oss | |
|---|---|---|
| Stars | 17k | 5.7k |
| Forks | 1.7k | 498 |
| Language | Python | Python |
| License | Apache-2.0 | Apache-2.0 |
| Last activity | today | yesterday |
| Topics | evals | evals, security |
| Curated connections | 3 | 4 |
deepeval — the curator's take
The most complete general-purpose eval framework: pytest ergonomics ('assert_test' in CI), metrics with explanations grounded in the research they implement, component-level tracing for agents, plus benchmark harnesses (MMLU, HumanEval) when you need them. If you're evaluating LLM apps in Python and don't know where to start, start here. NOT vendor-neutral: the free framework feeds Confident AI's platform for reports and regression tracking — usable without it, but the gravity is real. And LLM-as-judge metrics inherit judge bias: treat scores as regression signals, not absolute truth. RAG-only shops may prefer Ragas' tighter focus.
giskard-oss — the curator's take
One of the OG names in LLM testing, freshly rewritten: v3 drops the heavyweight monolith for focused packages — giskard-checks (scenario API for multi-turn agent evals: groundedness, conformity, LLM-judge) and giskard-scan (generates adversarial suites from a plain-language description of your agent, prompt-injection probes included). Async-first, wraps anything callable. Mind the transition, though: v3 is beta, RAG evaluation (RAGET) hasn't been ported yet, and v2 — where scan and RAG eval are mature — is no longer maintained. Python 3.12+ only, and libraries on giskard-core send aggregated telemetry (opt-out documented). Pick ragas for pure RAG metrics; Giskard earns its place on multi-turn agent scenarios plus red-teaming in one stack.