StackMap
Subscribe
Explore / topics

Evals & Testing

Measure and monitor LLM/agent output quality.

abide logo
3
coldteadotai avatarcoldteadotai 631 · 1 months ago
abide

Hooks Claude Code, Codex, OpenCode and Pi so every edit is checked against the rules in your AGENTS.md/CLAUDE.md by a decision model (Jev) — ~300 ms per check, violations fixed in-turn.

codingevals
→ pairs well with claude-reflect, failproofai
uber avatar
3
uber avataruber 2.1k · 6 days ago
ADR

Uber's Agentic AI Detection and Response: inventory AI tools on endpoints, capture Claude Code/Cursor/Codex telemetry, and flag risky sessions with a two-tier detector; ships ADR-Bench (300+ tasks).

securityevals
→ pairs well with Adrian, numbat
agents-cli logo
5
google avatargoogle 6.4k · 2 days ago
agents-cli

CLI + agent-skills layer that turns your coding assistant into a Google Cloud agent-lifecycle expert: scaffold ADK projects, run and evaluate them, then deploy and publish to Gemini Enterprise.

codingevalsskills
→ pairs well with claude-reflect, skillkit
CopilotKit avatar
5
CopilotKit avatarCopilotKit 1k · today
aimock

Mock everything an AI app talks to: 13 providers across 15 API surfaces plus MCP, A2A, AG-UI, vector DBs, search and rerank on one local port — with record-and-replay fixtures.

evals
→ pairs well with deepeval, litellm
AutoGen logo
9
microsoft avatarmicrosoft 61.9k · 7 days ago
AutoGen

Multi-agent conversation framework for building LLM applications with cooperating agents.

agentsevals
→ pairs well with Ragas, CubeSandbox
deepeval logo
8
confident-ai avatarconfident-ai 19.6k · today
deepeval

Pytest for LLM apps: 40+ research-backed metrics — G-Eval, RAG suite, agent task completion, hallucination — as unit tests you run in CI, judged by any LLM including local ones.

evals
→ pairs well with deepteam, aimock
deepteam logo
4
confident-ai avatarconfident-ai 3.1k · 27 days ago
deepteam

Open-source red-teaming framework for LLM systems: 50+ vulnerabilities, jailbreak/injection/multi-turn attacks against agents, RAG pipelines and chatbots — plus guardrails. Runs locally.

securityevals
→ pairs well with deepeval, garak
e2e logo
3
e2e-inc avatare2e-inc 14k · today
e2e

Agentic e2e testing for web and mobile: describe a goal, an agent drives the app, locators assert the result. Verified agent steps replay with zero model calls until the UI changes.

evalsweb
→ pairs well with browser, typesafe-computer-use
future-agi logo
3
future-agi avatarfuture-agi 2.3k · today
future-agi

Self-hostable platform for the whole agent-quality loop: tracing, evals, simulations, datasets, guardrails and an LLM gateway — one feedback loop from prototype to production. Apache 2.0.

evals
→ alternative to LangSmith, Tracely-ai
giskard-oss logo
5
Giskard-AI avatarGiskard-AI 6k · today
giskard-oss

Giskard v3: modular Python evals and red-teaming for agentic systems — scenario-based checks with LLM-as-judge, plus an automatic vulnerability scanner across OWASP LLM Top-10 categories.

evalssecurity
→ alternative to deepteam, Ragas
cmu-l3 avatar
5
cmu-l3 avatarcmu-l3 307 · 23 days ago
gym-anything

CMU framework that turns real software — browsers, IDEs, EMRs, CAD — into standardized agent environments: start the app, hand the agent a task, score it with automatic verifiers.

evalsagents
→ pairs well with browser-use, labs-molt
harness2 logo
3
google-research avatargoogle-research 24.5k · 17 days ago
harness2

Google Research's recursive harnessing framework: an agent refines its own harness (prompts, tools, memory, skills) per task via Propose–Probe–Compose, with model weights fixed.

agentsevals
→ alternative to JIT, agentic-context-engine
iFixAi logo
3
ifixai-ai avatarifixai-ai 46.5k · today
iFixAi

Independent auditor for AI agents: drops the agent into a simulation of your business, sends realistic requests, and scores operational alignment, permissions and red-team risk in ~2 minutes.

evalssecurity
→ pairs well with garak, giskard-oss
L
5
langchain-ai 0 · 1 week ago
LangSmith

Trace, test and monitor LLM apps in production.

evals
→ pairs well with LangGraph, deepagents
openqodex logo
3
openqodex avataropenqodex 1.2k · today
openqodex

Local pre-push code review for coding agents: deterministic scanners (gitleaks, semgrep) plus separate Claude Code/Codex reviewer sessions, with a push gate. No account or key of its own.

codingevals
→ pairs well with opencode, open-code-review
opensre logo
3
Tracer-Cloud avatarTracer-Cloud 13k · today
opensre

Open-source framework for AI SRE agents plus the RL training and evaluation environment they need — connect 60+ tools you already run and investigate incidents on your own infra.

agentsevals
→ pairs well with verl, superlog
OrcaReplay logo
5
Continuum-AI-Corp avatarContinuum-AI-Corp 1k · 1 months ago
OrcaReplay

Time travel for coding agents: a local proxy records any run (Claude Code, Codex, SDKs, even TLS-only harnesses), replays it byte-for-byte offline, and forks from any step onto a different model.

evalscoding
→ pairs well with aimock, agent-flow
Ragas logo
8
vibrantlabsai avatarvibrantlabsai 16.4k · 5 days ago
Ragas

Evaluation toolkit for your RAG and agent pipelines — faithfulness, relevance, and more.

evalsrag
→ pairs well with LlamaIndex, AutoGen
icedreamc avatar
6
icedreamc avataricedreamc 24 · 6 months ago
STALE

Benchmark plus memory pipeline for stale memories: STALE probes whether agents notice stored facts stopped being true; CUP-Mem adds conflict-aware writes, invalidation and premise verification.

evalsmemory
→ pairs well with metronix-memory, LightMem
supercov logo
4
supercorp-ai avatarsupercorp-ai 167 · 6 days ago
supercov

Code quality and coverage for coding agents: a Rust CLI that scores files with Jev, runs your existing test command, and turns uncovered paths into small actionable targets.

codingevals
→ pairs well with ripwire, open-code-review
swe-sweep logo
0
facebookresearch avatarfacebookresearch 246 · 7 days ago
swe-sweep

Meta's benchmark for open-ended bug hunting: given a real repository and no hints, an agent must find and fix as many bugs as it can. Scored as a micro-average of bugs resolved.

evalscoding
Tracely-ai logo
4
Jwuthri avatarJwuthri 1.1k · yesterday
Tracely-ai

Trace-native CI/CD for agents: OTLP traces are graded on arrival, failures cluster into issues, and one click freezes a failing run into a hermetic regression case that blocks the PR.

evalsagents
→ pairs well with superlog, deepeval
microsoft avatar
4
microsoft avatarmicrosoft 1.5k · today
waza

Microsoft's Go CLI for agent-skill quality: scaffold eval suites from SKILL.md, run them through Copilot, grade, compare models, replay snapshots and check spec and token budgets.

skillsevals
→ pairs well with SkillOpt, apm