autoscraper vs scrapy
Learn-by-example Python scraper: give it a URL and sample values you want, it infers the extraction rules and reapplies them to similar pages. Tiny, fast, zero selectors. — versus — The veteran Python web crawling framework: spiders, middlewares, pipelines and battle-tested scheduling — 60k+ stars and still the reference architecture for structured scraping.
Same job â get structured data off a site â from opposite ends: Scrapy is the full crawling framework you configure, autoscraper infers extraction rules from one example URL and a few sample values.
| autoscraper | scrapy | |
|---|---|---|
| Stars | 8.0k | 64k |
| Forks | 828 | 12k |
| Language | Python | Python |
| License | MIT | BSD-3-Clause |
| Last activity | 1 months ago | 2 days ago |
| Topics | web | web |
| Curated connections | 5 | 4 |
autoscraper — the curator's take
The cleverest 500 lines in scraping: show it one example of what you want off a page and it figures out the rules — no selectors, no XPath, and the learned model reapplies across similar pages. Perfect for quick structured grabs and prototyping. But check the commit log before adopting: it's been quiet for over a year, so treat it as a finished small tool, NOT a maintained framework — no JS rendering, no anti-bot, no crawling infrastructure. When sites fight back or scale arrives, move to a real framework.
scrapy — the curator's take
Fifteen-plus years of production hardening in one framework: spiders declare what to extract, middlewares/pipelines handle retries, throttling, dedup and export, and the ecosystem has an answer for everything. For large structured crawls in Python it's still the default. NOT a browser — JS-heavy or anti-bot-protected sites need Playwright bolted on or a different tool (Scrapling's stealth fetchers, Crawlee's browser mode), and the framework's inversion of control feels heavy when you just need one page: for that, requests + a parser beats a Scrapy project.