Scrapegraph-ai

LLM-driven Python scraping: describe what you want and graph pipelines extract structured data from websites or local docs (HTML, XML, JSON, Markdown). 29k stars; cloud API upsell.

28,904 2,824 Python MITupdated 14 days ago

web

View on GitHubDispute this mapping →

Curator's take

The prompt-first scraper: say what you want, SmartScraperGraph builds the extraction pipeline — the fastest route to a structured-extraction prototype, and it eats local HTML/XML/JSON/Markdown too. The catch: an LLM call per page, and the repo increasingly funnels you toward the paid cloud API. For bulk RAG ingestion crawl4ai is cheaper per page; for stable high-volume production pipelines a deterministic scraper (scrapling, scrapy) beats re-asking an LLM the same question about every page.

Mapped by ShipWithAI editors · links verified

README.md

🚀 Looking for an even faster and simpler way to scrape at scale (only 5 lines of code)? Check out our enhanced version at ScrapeGraphAI.com! 🚀

🕷️ ScrapeGraphAI: You Only Scrape Once

ScrapeGraphAI is a web scraping python library that uses LLM and direct graph logic to create scraping pipelines for websites and local documents (XML, HTML, JSON, Markdown, etc.).

Just say which information you want to extract and the library will do it for you!

🚀 Integrations

ScrapeGraphAI offers seamless integration with popular frameworks and tools to enhance your scraping capabilities. Whether you're building with Python or Node.js, using LLM frameworks, or working with no-code platforms, we've got you covered with our comprehensive integration options..

You can find more informations at the following link

Integrations:

API: Documentation
SDKs: Python, Node
LLM Frameworks: Langchain, Llama Index, Crew.ai, Agno, CamelAI
Low-code Frameworks: Pipedream, Bubble, Zapier, n8n, Dify, Toolhouse
MCP server: Link

🚀 Quick install

The reference page for Scrapegraph-ai is available on the official page of PyPI: pypi.

pip install scrapegraphai

# IMPORTANT (for fetching websites content)
playwright install

Note: it is recommended to install the library in a virtual environment to avoid conflicts with other libraries 🐱

💻 Usage

There are multiple standard scraping pipelines that can be used to extract information from a website (or local file).

The most common one is the SmartScraperGraph, which extracts informatio

Continue your stack

What teams reach for next — and why each earns a place beside Scrapegraph-ai. Ranked by curator confidence.

Alternative to See all + compare →

crawl4ai

The 74k-star LLM-native crawler: turns any site into clean, RAG-ready Markdown — adaptive crawling, JS rendering, extraction strategies, Docker deploy. Python, Apache-2.0.

Why it fitsBoth are LLM-era scraping libraries. Crawl4ai turns sites into clean RAG-ready Markdown with the LLM optional; ScrapeGraphAI puts the LLM in the loop to extract a prompt-defined schema. Ingesting corpora → crawl4ai; asking pages questions → ScrapeGraphAI.

Scrapling

Adaptive Python scraping framework: selectors that relearn when sites redesign, stealth fetchers that pass Cloudflare, spiders with proxy rotation and an MCP server — request to full crawl.

Why it fitsOpposite bets on brittleness: Scrapling keeps deterministic selectors that self-relearn on redesigns; ScrapeGraphAI drops selectors entirely and pays an LLM call per page. Scrapling for cheap repeatable production runs, ScrapeGraphAI for schema-by-prompt flexibility.

autoscraper

Learn-by-example Python scraper: give it a URL and sample values you want, it infers the extraction rules and reapplies them to similar pages. Tiny, fast, zero selectors.

Why it fitsTwo ways to skip writing selectors: AutoScraper infers reusable rules from one example page (train once, scrape free); ScrapeGraphAI re-extracts with an LLM every time. AutoScraper wins on cost for uniform pages, loses on messy or varied layouts.