DataFlow vs GraphGen
Operator-based system for LLM data prep — 100+ operators composed into pipelines that generate, clean, evaluate and filter pretraining/SFT/RL data, with a WebUI and a pipeline-building agent. — versus — KG-guided synthetic SFT data: builds a knowledge graph from source text, finds the LLM's knowledge gaps via calibration error, and generates targeted long-tail QA pairs. ACL-published.
Both produce SFT training data, differently: DataFlow is a general operator-pipeline system for generating/cleaning/filtering at scale; GraphGen is one focused method — KG + calibration-error targeting of long-tail knowledge. DataFlow to industrialize a data pipeline, GraphGen when gap coverage is the point.
| DataFlow | GraphGen | |
|---|---|---|
| Stars | 7.1k | 1.2k |
| Forks | 929 | 85 |
| Language | Python | Python |
| License | Apache-2.0 | Apache-2.0 |
| Last activity | 19 days ago | 2 months ago |
| Topics | training | training |
| Curated connections | 4 | 3 |
DataFlow — the curator's take
Pick it when the model isn't the problem, the data is. Ready pipelines cover text/math/code synthesis, large-scale PDF→QA, Text2SQL and knowledge-base cleaning, all in a PyTorch-like Pipeline→Operator→Prompt hierarchy that makes data governance reproducible and shareable; the DataFlow-Agent assembles pipelines from a task description, and the WebUI gives non-coders drag-and-drop. Versus Data-Juicer/Nemo-Curator the differentiator is synthesis, with peer-reviewed pedigree (ICDE/KDD acceptances, arXiv report). NOT runtime data plumbing — this is offline training-data preparation, and budget for real LLM API burn since most interesting operators call models (vLLM/SGLang backends supported for local).
GraphGen — the curator's take
The interesting bit isn't "synthetic QA" — it's targeting: GraphGen measures the student model's expected calibration error to find what it *doesn't* know, then samples multi-hop KG neighborhoods to generate exactly those long-tail QA pairs. Use it when domain-adapting a model and naive synthetic data keeps rehearsing what the model already knows. Output feeds straight into LLaMA-Factory or xtuner. NOT a general data-prep system — it makes QA-style SFT/pretrain data from documents, nothing else; for cleaning, filtering and pipeline plumbing you still want an operator system like DataFlow.