[{"data":1,"prerenderedAt":4},["ShallowReactive",2],{"readme:loafer":3},"\u003Ch1>Loafer\u003C\u002Fh1>\n\u003Cp>Loafer is an open-source ETL\u002FELT engine for defining data pipelines in YAML and running them from\nthe command line or through a self-hosted control plane.\u003C\u002Fp>\n\u003Cp>It exists to make serious data movement understandable: a pipeline says where data comes from,\nhow it changes, where it goes, and what correctness rules apply. The same definition can run\nlocally during development or as a durable job handled by independent workers.\u003C\u002Fp>\n\u003Cp>Loafer currently supports:\u003C\u002Fp>\n\u003Cul>\n\u003Cli>PostgreSQL, MySQL, MongoDB, SQLite, CSV, Excel, REST, and PDF sources\u003C\u002Fli>\n\u003Cli>PostgreSQL, MongoDB, CSV, and JSON targets\u003C\u002Fli>\n\u003Cli>SQL, custom Python, multi-step, and optional AI-authored transforms\u003C\u002Fli>\n\u003Cli>incremental cursors, validation, schema-drift policies, quarantine, and upserts\u003C\u002Fli>\n\u003Cli>bounded row-local ETL and in-database ELT\u003C\u002Fli>\n\u003Cli>durable schedules, retries, cancellation, checkpoints, and role-isolated workers\u003C\u002Fli>\n\u003Cli>a self-hosted API and authentication boundary\u003C\u002Fli>\n\u003C\u002Ful>\n\u003Cp>The CLI engine and distributed runtime are implemented. The connected operations dashboard,\ndistributed object storage, OCR\u002Fweb-crawl workers, and native multi-pipeline DAGs are still in\ndevelopment. See \u003Ca href=\"https:\u002F\u002Fgithub.com\u002Flupppig\u002Floafer\u002Fblob\u002FHEAD\u002FPRODUCTION_READINESS.md\" rel=\"nofollow ugc noopener\">Production readiness\u003C\u002Fa> before choosing a workload or\nmaking scale guarantees.\u003C\u002Fp>\n\u003Ch2>Quick start\u003C\u002Fh2>\n\u003Cp>Python 3.11 or newer is required.\u003C\u002Fp>\n\u003Cpre>\u003Ccode class=\"language-bash\">pip install loafer-etl\nloafer --version\n\u003C\u002Fcode>\u003C\u002Fpre>\n\u003Cp>To run the included no-API-key example from a clone:\u003C\u002Fp>\n\u003Cpre>\u003Ccode class=\"language-bash\">git clone https:\u002F\u002Fgithub.com\u002Flupppig\u002Floafer.git\ncd loafer\nuv sync\n\nuv run loafer validate examples\u002Fpipelines\u002F04_bypass_ai.yaml\nuv run loafer run examples\u002Fpipelines\u002F04_bypass_ai.yaml --local --yes\n\u003C\u002Fcode>\u003C\u002Fpre>\n\u003Cp>\u003Ccode>--local\u003C\u002Fcode> is deliberate: local execution must be requested explicitly. Durable jobs are submitted\nwith \u003Ccode>loafer enqueue\u003C\u002Fcode> and executed by a worker.\u003C\u002Fp>\n\u003Ch2>A pipeline at a glance\u003C\u002Fh2>\n\u003Cpre>\u003Ccode class=\"language-yaml\">name: daily_orders\nmode: etl\nchunk_size: 5000\n\nsource:\n  type: postgres\n  url: ${SOURCE_DATABASE_URL}\n  query: SELECT * FROM orders\n\ntransform:\n  type: custom\n  path: .\u002Ftransforms\u002Fnormalize_order.py\n\ntarget:\n  type: postgres\n  url: ${WAREHOUSE_DATABASE_URL}\n  table: analytics.orders\n  write_mode: upsert\n  key: order_id\n\nincremental:\n  column: updated_at\n  initial: \"1970-01-01\"\n\nexecution:\n  transform_class: row_local\n  schema_drift: quarantine\n  quarantine_path: .\u002Frejected\u002Forders.json\n\nvalidation:\n  required_columns: [order_id, updated_at]\n  on_failure: quarantine\n\u003C\u002Fcode>\u003C\u002Fpre>\n\u003Cp>The important declaration is \u003Ccode>transform_class\u003C\u002Fcode>. Use \u003Ccode>row_local\u003C\u002Fcode> only when each output row depends\non data in its current batch. Joins, aggregates, windows, sorts, and whole-dataset deduplication are\nglobal work and must use ELT pushdown or the materialized compatibility path.\u003C\u002Fp>\n\u003Ch2>Architecture\u003C\u002Fh2>\n\u003Cp>Loafer has one execution engine with two ways into it:\u003C\u002Fp>\n\u003Cpre>\u003Ccode class=\"language-text\">Local development\n  CLI ───────────────────────────────────────────────┐\n                                                     │\nDurable deployment                                  ▼\n  Browser → Next.js auth\u002FBFF → loaferd          application service\n  CLI\u002Fautomation ────────────────┘                    │\n                                  PostgreSQL          ▼\n                              metadata + outbox → relay → NATS JetStream\n                                                           │\n                                               role-isolated workers\n                                                           │\n                                      extract → validate → transform → load\n                                                           │\n                                              sources, targets, artifacts\n\u003C\u002Fcode>\u003C\u002Fpre>\n\u003Cp>The metadata database is the authority for run state: PostgreSQL in a distributed deployment and\nSQLite in the embedded profile. NATS carries small job notifications, not pipeline definitions,\ncredentials, or row data. A worker resolves the immutable pipeline version and its allowed secret\nreferences only after claiming a fenced lease.\u003C\u002Fp>\n\u003Ch3>Architectural decisions\u003C\u002Fh3>\n\u003Ctable>\n\u003Cthead>\n\u003Ctr>\n\u003Cth>Decision\u003C\u002Fth>\n\u003Cth>Why it matters\u003C\u002Fth>\n\u003C\u002Ftr>\n\u003C\u002Fthead>\n\u003Ctbody>\u003Ctr>\n\u003Ctd>YAML is the pipeline contract\u003C\u002Ftd>\n\u003Ctd>Pipelines stay re\u003C\u002Ftd>\n\u003C\u002Ftr>\n\u003C\u002Ftbody>\u003C\u002Ftable>\n",1788652555743]