Evals & Testing
Measure and monitor LLM/agent output quality.

Hooks Claude Code, Codex, OpenCode and Pi so every edit is checked against the rules in your AGENTS.md/CLAUDE.md by a decision model (Jev) — ~300 ms per check, violations fixed in-turn.
Uber's Agentic AI Detection and Response: inventory AI tools on endpoints, capture Claude Code/Cursor/Codex telemetry, and flag risky sessions with a two-tier detector; ships ADR-Bench (300+ tasks).

CLI + agent-skills layer that turns your coding assistant into a Google Cloud agent-lifecycle expert: scaffold ADK projects, run and evaluate them, then deploy and publish to Gemini Enterprise.
Mock everything an AI app talks to: 13 providers across 15 API surfaces plus MCP, A2A, AG-UI, vector DBs, search and rerank on one local port — with record-and-replay fixtures.
Multi-agent conversation framework for building LLM applications with cooperating agents.
Pytest for LLM apps: 40+ research-backed metrics — G-Eval, RAG suite, agent task completion, hallucination — as unit tests you run in CI, judged by any LLM including local ones.
Open-source red-teaming framework for LLM systems: 50+ vulnerabilities, jailbreak/injection/multi-turn attacks against agents, RAG pipelines and chatbots — plus guardrails. Runs locally.

Agentic e2e testing for web and mobile: describe a goal, an agent drives the app, locators assert the result. Verified agent steps replay with zero model calls until the UI changes.

Self-hostable platform for the whole agent-quality loop: tracing, evals, simulations, datasets, guardrails and an LLM gateway — one feedback loop from prototype to production. Apache 2.0.

Giskard v3: modular Python evals and red-teaming for agentic systems — scenario-based checks with LLM-as-judge, plus an automatic vulnerability scanner across OWASP LLM Top-10 categories.
CMU framework that turns real software — browsers, IDEs, EMRs, CAD — into standardized agent environments: start the app, hand the agent a task, score it with automatic verifiers.

Google Research's recursive harnessing framework: an agent refines its own harness (prompts, tools, memory, skills) per task via Propose–Probe–Compose, with model weights fixed.
Independent auditor for AI agents: drops the agent into a simulation of your business, sends realistic requests, and scores operational alignment, permissions and red-team risk in ~2 minutes.
Trace, test and monitor LLM apps in production.

Local pre-push code review for coding agents: deterministic scanners (gitleaks, semgrep) plus separate Claude Code/Codex reviewer sessions, with a push gate. No account or key of its own.
Open-source framework for AI SRE agents plus the RL training and evaluation environment they need — connect 60+ tools you already run and investigate incidents on your own infra.
Time travel for coding agents: a local proxy records any run (Claude Code, Codex, SDKs, even TLS-only harnesses), replays it byte-for-byte offline, and forks from any step onto a different model.

Evaluation toolkit for your RAG and agent pipelines — faithfulness, relevance, and more.
Benchmark plus memory pipeline for stale memories: STALE probes whether agents notice stored facts stopped being true; CUP-Mem adds conflict-aware writes, invalidation and premise verification.

Code quality and coverage for coding agents: a Rust CLI that scores files with Jev, runs your existing test command, and turns uncovered paths into small actionable targets.

Meta's benchmark for open-ended bug hunting: given a real repository and no hints, an agent must find and fix as many bugs as it can. Scored as a micro-average of bugs resolved.
Trace-native CI/CD for agents: OTLP traces are graded on arrival, failures cluster into issues, and one click freezes a failing run into a hermetic regression case that blocks the PR.
Microsoft's Go CLI for agent-skill quality: scaffold eval suites from SKILL.md, run them through Copilot, grade, compare models, replay snapshots and check spec and token budgets.