cocoindex vs duckle
Rust-core incremental indexing engine: declare Target = F(Source) in Python and it keeps vector/graph/relational targets fresh forever, reprocessing only the delta — with per-row lineage. — versus — Self-hosted ETL/ELT on DuckDB: author pipelines on a canvas, in SQL or Python, then ship the same file to your own server — 190+ sources, dbt, CDC, lineage, and an MCP server for agents.
Both keep derived data fresh from declared sources without a vendor cloud. CocoIndex is a Python library for incremental AI indexing with per-row lineage; Duckle is a full ETL product — canvas, connectors, scheduler, console — compiling to SQL on DuckDB.
| cocoindex | duckle | |
|---|---|---|
| Stars | 11k | 1.2k |
| Forks | 887 | 91 |
| Language | Rust | Rust |
| License | Apache-2.0 | Apache-2.0 |
| Last activity | 6 days ago | yesterday |
| Topics | rag | data |
| Curated connections | 6 | 2 |
cocoindex — the curator's take
The mental model sells it — 'React for data engineering': you declare what the index should contain, and the engine reconciles it against source changes forever, re-running only affected rows (cached by hash of input AND code, so editing your transform also invalidates precisely). That's the honest answer to stale agent context: sub-second freshness at a fraction of the re-embedding bill, with every vector traceable to its source byte. Sources span code, PDFs, Slack, audio; targets span pgvector, LanceDB, Neo4j, Kafka. The flagship application is cocoindex-code, an AST-aware incremental code-index MCP for coding agents. Use it when your corpus changes constantly and batch re-indexing is bleeding you; overkill for a static document pile — any one-shot RAG ingester handles that.
duckle — the curator's take
Reach for it when the objection to Fivetran or Airbyte is the bill and the vendor cloud: pipelines compile to SQL on DuckDB, use every core on the box, and every pipeline is one file in git that outlives whoever wrote it. `duckle-runner serve` runs it headless on a schedule with a web console, roles and an audit trail, and the MCP server means an agent can inspect and drive pipelines rather than you hand-writing glue. Where it's weak: single-engine by design, so it scales up a box rather than out a cluster; it's beta and Rust/Tauri-shaped, meaning a desktop authoring app is part of the workflow; and the 385-component count includes a long tail you should verify before betting a migration on it.