Loafer

Loafer is an open-source ETL/ELT engine for defining data pipelines in YAML and running them from the command line or through a self-hosted control plane.

It exists to make serious data movement understandable: a pipeline says where data comes from, how it changes, where it goes, and what correctness rules apply. The same definition can run locally during development or as a durable job handled by independent workers.

Loafer currently supports:

• PostgreSQL, MySQL, MongoDB, SQLite, CSV, Excel, REST, and PDF sources
• PostgreSQL, MongoDB, CSV, and JSON targets
• SQL, custom Python, multi-step, and optional AI-authored transforms
• incremental cursors, validation, schema-drift policies, quarantine, and upserts
• bounded row-local ETL and in-database ELT
• durable schedules, retries, cancellation, checkpoints, and role-isolated workers
• a self-hosted API and authentication boundary

The CLI engine and distributed runtime are implemented. The connected operations dashboard, distributed object storage, OCR/web-crawl workers, and native multi-pipeline DAGs are still in development. See Production readiness before choosing a workload or making scale guarantees.

Quick start

Python 3.11 or newer is required.

pip install loafer-etl
loafer --version

To run the included no-API-key example from a clone:

git clone https://github.com/lupppig/loafer.git
cd loafer
uv sync

uv run loafer validate examples/pipelines/04_bypass_ai.yaml
uv run loafer run examples/pipelines/04_bypass_ai.yaml --local --yes

--local is deliberate: local execution must be requested explicitly. Durable jobs are submitted with loafer enqueue and executed by a worker.

A pipeline at a glance

name: daily_orders
mode: etl
chunk_size: 5000

source:
  type: postgres
  url: ${SOURCE_DATABASE_URL}
  query: SELECT * FROM orders

transform:
  type: custom
  path: ./transforms/normalize_order.py

target:
  type: postgres
  url: ${WAREHOUSE_DATABASE_URL}
  table: analytics.orders
  write_mode: upsert
  key: order_id

incremental:
  column: updated_at
  initial: "1970-01-01"

execution:
  transform_class: row_local
  schema_drift: quarantine
  quarantine_path: ./rejected/orders.json

validation:
  required_columns: [order_id, updated_at]
  on_failure: quarantine

The important declaration is transform_class. Use row_local only when each output row depends on data in its current batch. Joins, aggregates, windows, sorts, and whole-dataset deduplication are global work and must use ELT pushdown or the materialized compatibility path.

Architecture

Loafer has one execution engine with two ways into it:

Local development
  CLI ───────────────────────────────────────────────┐
                                                     │
Durable deployment                                  ▼
  Browser → Next.js auth/BFF → loaferd          application service
  CLI/automation ────────────────┘                    │
                                  PostgreSQL          ▼
                              metadata + outbox → relay → NATS JetStream
                                                           │
                                               role-isolated workers
                                                           │
                                      extract → validate → transform → load
                                                           │
                                              sources, targets, artifacts

The metadata database is the authority for run state: PostgreSQL in a distributed deployment and SQLite in the embedded profile. NATS carries small job notifications, not pipeline definitions, credentials, or row data. A worker resolves the immutable pipeline version and its allowed secret references only after claiming a fenced lease.

Architectural decisions