Evals & Testing
Measure and monitor LLM/agent output quality.
google 6k · 9 days agoCLI + agent-skills layer that turns your coding assistant into a Google Cloud agent-lifecycle expert: scaffold ADK projects, run and evaluate them, then deploy and publish to Gemini Enterprise.
CopilotKit 949 · 7 days agoMock everything an AI app talks to: 13 providers across 15 API surfaces plus MCP, A2A, AG-UI, vector DBs, search and rerank on one local port — with record-and-replay fixtures.
microsoft 61.2k · 5 months agoMulti-agent conversation framework for building LLM applications with cooperating agents.
confident-ai 18.5k · 3 days agoPytest for LLM apps: 40+ research-backed metrics — G-Eval, RAG suite, agent task completion, hallucination — as unit tests you run in CI, judged by any LLM including local ones.
confident-ai 3k · 10 days agoOpen-source red-teaming framework for LLM systems: 50+ vulnerabilities, jailbreak/injection/multi-turn attacks against agents, RAG pipelines and chatbots — plus guardrails. Runs locally.
future-agi 2.1k · 3 days agoSelf-hostable platform for the whole agent-quality loop: tracing, evals, simulations, datasets, guardrails and an LLM gateway — one feedback loop from prototype to production. Apache 2.0.
Giskard-AI 5.8k · 3 days agoGiskard v3: modular Python evals and red-teaming for agentic systems — scenario-based checks with LLM-as-judge, plus an automatic vulnerability scanner across OWASP LLM Top-10 categories.
cmu-l3 288 · 5 days agoCMU framework that turns real software — browsers, IDEs, EMRs, CAD — into standardized agent environments: start the app, hand the agent a task, score it with automatic verifiers.
Trace, test and monitor LLM apps in production.
Tracer-Cloud 11.2k · 3 days agoOpen-source framework for AI SRE agents plus the RL training and evaluation environment they need — connect 60+ tools you already run and investigate incidents on your own infra.
Continuum-AI-Corp 269 · 3 days agoTime travel for coding agents: a local proxy records any run (Claude Code, Codex, SDKs, even TLS-only harnesses), replays it byte-for-byte offline, and forks from any step onto a different model.
vibrantlabsai 15.9k · 7 months agoEvaluation toolkit for your RAG and agent pipelines — faithfulness, relevance, and more.
icedreamc 22 · 4 months agoBenchmark plus memory pipeline for stale memories: STALE probes whether agents notice stored facts stopped being true; CUP-Mem adds conflict-aware writes, invalidation and premise verification.
supercorp-ai 133 · 4 days agoCode quality and coverage for coding agents: a Rust CLI that scores files with Jev, runs your existing test command, and turns uncovered paths into small actionable targets.
Jwuthri 1.4k · 3 days agoTrace-native CI/CD for agents: OTLP traces are graded on arrival, failures cluster into issues, and one click freezes a failing run into a hermetic regression case that blocks the PR.