Tracely-ai

Trace-native CI/CD for agents: OTLP traces are graded on arrival, failures cluster into issues, and one click freezes a failing run into a hermetic regression case that blocks the PR.

1,064 86 Python MITupdated today

evals agents

View on GitHubDispute this mapping →

Curator's take

The claim that separates it from every other eval tool: no hand-authored dataset. The recorded production trace is the test — promoted with its tool and LLM responses as fixtures, so CI replays it for $0 with zero live model calls, under a fail-to-pass contract that refuses the promotion unless the case fails on the old code and passes on the fix. Failure clustering is the other real feature: 31 broken runs collapse into one issue instead of 31 rows. Use it when agents are already in production emitting OTLP and your problem is regressions. NOT a lightweight library — self-hosting means Postgres, ClickHouse, Redis and MinIO — and with no traffic there is nothing to grade, so pre-production teams should start with a dataset-first tool.

Mapped by ShipWithAI editors · links verified

README.md

Tracely

Production failures become regression tests.

Trace-native CI/CD for AI agents. Tracely grades every agent trace as it lands, clusters the failures into issues, freezes the bad runs into hermetic replayable cases, blocks the pull request that would ship them again — and tells you the moment any of it happens.

production trace → failure detection → regression test → CI gate → alert

Website · Docs · Product guide · Agent skill · Guided tour · 2-min demo · Design dossier

Self-host the whole stack in one click — API, worker, UI, Postgres, ClickHouse, Redis and MinIO:

Tracely dashboard — trace and failure counts, the biggest failure clusters, recent traces and regression cases

_{One workspace: what ran, what broke, what is already pinned as a test.}

Why another agent-observability tool?

Because observability stops at the dashboard. You can see that your agent broke — then what?

Every eval tool asks you to hand-author a dataset: sit down, invent questions, write ideal answers, keep them current as the product changes. That dataset is a guess about what might break.

Production already handed you the real thing: a trace of the exact run that failed, with the exact input, the exact tool calls, the exact model responses.

The recorded run is the test. Tracely freezes that trace into a hermetic regression case and replays it on every PR. Everything else — quality scores, failure clusters, suggested fixes, CI verdicts, trends, alerts — is derived from the trace. There are no hand-authored datasets.

	Dataset-first tools	Tracely
Where tests come from	You write them	Promoted from real failing traces
Fidelity to production	A guess	The exact failing run, byte for byte
Cost to replay in CI	Live model calls	$0 — recorded tool/LLM fixtures
What happens on regression	A dashboard number moves	The PR is blocked
How you find out	You go and look	It comes to you — Slack, email, your own webhook

The loop

Five steps, five pages in the app.

1 · Observe — every run, hierarchically

Traces arrive over plain OTLP. Agent semantics (agent.id, conversation.id, turn, step) are promoted to first-class indexed columns, so runs group into conversation threads instead of a flat span soup. The waterfall shows agent → thinking → skill → generation → hand-off, with the failing span in red and its I/O beside it.

Trace timeline — agent, thinking, skill, generation and delegate spans, with a nested sub-agent and the span's input

Evaluators are columns on the trace table, not a separate tab — each one grades at conve

Continue your stack

What teams reach for next — and why each earns a place beside Tracely-ai. Ranked by curator confidence.

Pairs well with

superlog

Agentic observability for OpenTelemetry: ingest traces, logs and metrics, group noisy signals into incidents, then let pluggable agent runners investigate while you sleep. Self-hosted, open-core.

Why it fitsTwo consumers of the same OTLP stream. superlog is the ops side — group noisy signals into incidents and let agent runners investigate; Tracely is the dev side, turning the bad runs into frozen regression tests so the incident cannot come back.

Alternative to See all + compare →

deepeval

Pytest for LLM apps: 40+ research-backed metrics — G-Eval, RAG suite, agent task completion, hallucination — as unit tests you run in CI, judged by any LLM including local ones.

Why it fitsBoth gate LLM behaviour in CI, from opposite directions. DeepEval is pytest: you author the dataset and pick from 40+ metrics. Tracely inverts it — production traces become the cases automatically and replay from recorded fixtures, so CI costs nothing but you need real traffic first.

LangSmith

Trace, test and monitor LLM apps in production.

Why it fitsThe same trace-to-test loop, different ownership. LangSmith is the hosted incumbent for tracing, testing and monitoring; Tracely is MIT and self-hosted end to end, and pushes past the dashboard — a regression does not move a chart, it blocks the pull request.

future-agi

Self-hostable platform for the whole agent-quality loop: tracing, evals, simulations, datasets, guardrails and an LLM gateway — one feedback loop from prototype to production. Apache 2.0.

Why it fitsBoth are self-hostable platforms for the whole agent-quality loop. Future AGI is broader — simulations, datasets, guardrails and an LLM gateway; Tracely is narrower and sharper: everything derives from the trace and the output is a PR verdict, not a score.