Three tools dominate Python web scraping: Scrapy, Playwright and Selenium. They overlap, but each has a clear sweet spot. Here's how we choose for client projects.
Scrapy: the high-volume crawler
Scrapy is a full crawling framework. It downloads pages over plain HTTP (no browser) and manages the queue, retries, throttling, pipelines and exports for you.
- Strengths: very fast and efficient, since thousands of pages per minute on modest hardware is realistic. Built-in politeness settings (AutoThrottle), middlewares, item pipelines, and mature deployment options.
- Weaknesses: doesn't run JavaScript on its own, so content rendered client-side needs extra help (for example the
scrapy-playwrightplugin). - Use it for: large catalogues, news and listing sites, and anything where the data is in the HTML or in an API the page calls.
Playwright: the modern browser
Playwright drives real Chromium, Firefox or WebKit browsers with a clean async API.
- Strengths: handles JavaScript-heavy single-page apps, logins, infinite scroll and clicks. Auto-waiting makes scripts much more stable, and it can intercept network requests, which is often the best way to grab the JSON a page loads.
- Weaknesses: each browser uses far more CPU and memory than an HTTP request, so it's slower and more expensive at scale.
- Use it for: dynamic sites, complex interactions, and sites where you need to see the page as a user does.
import asyncio
from playwright.async_api import async_playwright
async def main():
async with async_playwright() as p:
browser = await p.chromium.launch()
page = await browser.new_page()
await page.goto("https://example.com/listings")
titles = await page.locator(".listing-title").all_inner_texts()
print(titles)
await browser.close()
asyncio.run(main())
Selenium: the veteran
Selenium has been around the longest and supports the widest range of languages and browsers.
- Strengths: huge community, Selenium Grid for distributed runs, and lots of existing code and tutorials.
- Weaknesses: more boilerplate and more manual waiting than Playwright, and generally slower for scraping.
- Use it for: maintaining existing Selenium projects, or when a team already has deep Selenium expertise.
For new browser-based scrapers, we now default to Playwright.
Comparison
| Scrapy | Playwright | Selenium | |
|---|---|---|---|
| Runs JavaScript | With a plugin | Yes | Yes |
| Speed / cost at scale | Best | Moderate | Lowest |
| Built-in crawling features | Extensive | Minimal | Minimal |
| Script stability | High | High | Medium |
| Best for | Volume | Dynamic sites | Legacy / Grid |
The hybrid approach we use most
- Look for the API first. Open DevTools, watch the Network tab, and see whether the page loads its data as JSON. Calling that endpoint directly is often 10–100× faster than rendering the page.
- Use Scrapy or plain HTTP for the bulk of pages.
- Use Playwright only where needed: logins, tokens, or pages that genuinely require a browser.
Increasingly, a fourth tool sits on top of all three: an LLM that turns messy pages into structured data. More on that in a future post.
Not sure which approach suits your target sites? Our web scraping team can run a quick feasibility check.