StackMap
Subscribe
Explore / leviathan
elstongun

leviathan

Single Rust binary that turns JSONL, CSV, SQLite or any DB export into a ranked full-text index, giving agents short cited answer cards (~450 tokens) at any dataset size.

879 53 Rust Apache-2.0updated 4 days ago
memorydata
View on GitHubDispute this mapping →
Curator's take

Use it when an agent has to answer questions over large record dumps, like tickets, logs or CRM exports, and grep is flooding the context. The benchmark claims 245x fewer tokens than the best grep strategy. Skip it if you need semantic or vector recall: it is lexical full-text search. It is not a memory system that learns from conversations; you re-index when the data changes.

Mapped by ShipWithAI editors · links verified

Continue your stack

What teams reach for next — and why each earns a place beside leviathan. Ranked by curator confidence.

alternativealternativealternativexerjcontext-modeheadroomleviathan
pairs wellalternativebuilt withpick a node for the why · open it from the panel
Weekly digest
README.md2 min read

Leviathan

Deep memory for agents over large datasets.
Index any table, export or log once; your agent gets the few records that answer the question.

CI License: Apache-2.0 Rust 1.88+ MCP optional

Leviathan is a single binary that turns records (JSONL, JSON, CSV/TSV, SQLite, or any database CLI's output) into a ranked full-text index. Agents ask in plain words and get short, cited result cards: ~450 tokens per answer at any dataset size, instead of grepping and reading raw history.

Median tokens per question at 1M records: Leviathan 436, grep entity + question words 107K, grep entity history 209K, read all 203M

Median tokens per question vs dataset size Answer rate vs dataset size

Tokens per question vs entity history size Latency vs dataset size

At 1M records (678 MB) Leviathan best grep strategy
Median tokens per question 436 107,122 (245×)
Relevant record returned 99.0% top 5 · 98.5% rank 1 96.0% within a 30K-char output
Worst case (1,200 questions) 602 tokens 9.7M tokens
Median latency 33 ms 92 ms

Measured on a synthetic maintenance log (one example dataset; nothing in Leviathan is specific to it). Methodology, all six scales and caveats: docs/BENCHMARKS.md.

Quickstart

cargo install leviathan-index       # or: cargo install --git https://github.com/elstongun/leviathan
cd examples/tickets && leviathan index
leviathan search -g acme "sso login loop after password reset"

leviathan search · customer C-ACME "Acme Corp" (7 tickets) · query "sso login loop after password reset" · shown 3 of 3 · 24 tickets indexed
[1] T-1001 · 2024-01-08 09:12 · rel 16.9
  Login loops back to sign-in page after password reset
  status: closed · priority: high
  resolution: Cleared stale session cookies on password reset; shipped in 4.2.1. Workaround: clear site data.
  match: Users who reset their password get redirected to the sign-in page again in an endless loop.
...
  • -g accepts a key, name or partial name. Ambiguous or unknown groups list candidates and exit 3, never guess.
  • No match in the group falls back to other groups, labeled OTHER CUSTOMER.
  • Combine --where field=value, --since/--until, "phrases" and -exclusions.

Prebuilt binaries are on Releases. No runtime dependencies; the index is one SQLite file.

Map your data

Fields are paths (a.b, items[].name); only id is required.

Field Enables
id get, upsert, delete, citations
title / text Card headline (weighted 2×) / searched text (default: all strings)
group / group_name -g scoped search, name resolution, labeled fallback
date --since, --until, recent
filters / display --where facets / fields shown on cards
empty_values / rank.boost Placeholders treated as missing / favor complete records 
leviathan index ./export                      # everything in a fol