crawl4ai

The 74k-star LLM-native crawler: turns any site into clean, RAG-ready Markdown — adaptive crawling, JS rendering, extraction strategies, Docker deploy. Python, Apache-2.0.

74,558 7,675 Python Apache-2.0updated yesterday

web rag

View on GitHubDispute this mapping →

Curator's take

The default answer when the deliverable is Markdown for a model rather than structured data for a database: heuristic content filtering, LLM and CSS extraction strategies, deep-crawl dispatchers and a Dockerized API — battle-tested by the largest community in the category. NOT the stealth pick: for hostile anti-bot targets Scrapling's fetchers earn their keep, and for classic item-pipeline scraping at scale Crawlee's queue/proxy machinery is more mature. Watch the cloud beta — the open core is healthy, but the cost-effective-cloud pitch tells you where the roadmap's gravity is.

Mapped by ShipWithAI editors · links verified

README.md

🚀🤖 Crawl4AI: Open-source LLM Friendly Web Crawler & Scraper.

🚀 Crawl4AI Cloud API — Closed Beta (Launching Soon)

Reliable, large-scale web extraction, now built to be drastically more cost-effective than any of the existing solutions.

👉 Apply here for early access
We’ll be onboarding in phases and working closely with early users. Limited slots.

Crawl4AI turns the web into clean, LLM ready Markdown for RAG, agents, and data pipelines. Fast, controllable, battle tested by a 50k+ star community.

✨ Check out latest update v0.9.2

✨ New in v0.9.2: Maintenance patch release. Fixes a MemoryAdaptiveDispatcher task/page leak when a streaming crawl is closed, Docker Playground "Advanced Config" and Monitor WebSocket auth, Playwright headless-shell packaging, and GPU (ENABLE_GPU=true) Docker builds. Release notes →

✨ Recent v0.9.0: Major secure-by-default release of the Docker API server. Auth is on by default, the server binds loopback unless given a token, and the request body is now an untrusted trust boundary. Release notes →

✨ Recent v0.8.7: Security-hardening release. Fixes critical Docker API vulnerabilities (RCE, SSRF, auth bypass, file write, XSS, hardcoded JWT secret), adds DomainMapper, and ships scraping, deep-crawl, and LLM fixes. Release notes →

✨ Previous v0.8.0: Crash Recovery & Prefetch Mode! Deep crawl crash recovery with resume_state and on_state_change callbacks for long-running crawls. New prefetch=True mode for 5-10x faster URL discovery. Release notes →

✨ Previous v0.7.8: Stability & Bug Fix Release! 11 bug fixes addressing Docker API issues, LLM extraction improvements, URL handling fixes, and dependency updates. Release notes →

🤓 My Personal Story

I grew up on an Amstrad, thanks to my dad, and never stopped building. In grad school I specialized in NLP an

Continue your stack

What teams reach for next — and why each earns a place beside crawl4ai. Ranked by curator confidence.

Alternative to See all + compare →

crawlee

Apify's web scraping and browser automation library for Node.js/TypeScript — HTTP and headless-browser crawlers with human-like anti-blocking defaults, queues, storage and proxies.

Why it fitsThe two big crawl frameworks, split by output philosophy: Crawlee (Node/TS) is item-pipeline scraping with anti-blocking and storage; Crawl4AI (Python) optimizes everything toward clean Markdown for LLM consumption.

Scrapling

Adaptive Python scraping framework: selectors that relearn when sites redesign, stealth fetchers that pass Cloudflare, spiders with proxy rotation and an MCP server — request to full crawl.

Why it fitsSame Python scraping job, different bets: Scrapling bets on resilience (self-healing selectors, Cloudflare-passing stealth); Crawl4AI bets on LLM-ready output and crawl orchestration. Hostile targets → Scrapling; RAG pipelines → Crawl4AI.