Web Scraping

Web Scraping in 2026: Playwright vs Selenium vs Scrapy

Home Blog Web Scraping

Web Scraping

Three tools dominate Python web scraping: Scrapy, Playwright and Selenium. They overlap, but each has a clear sweet spot. Here's how we choose for client projects.

Scrapy: the high-volume crawler

Scrapy is a full crawling framework. It downloads pages over plain HTTP (no browser) and manages the queue, retries, throttling, pipelines and exports for you.

  • Strengths: very fast and efficient, since thousands of pages per minute on modest hardware is realistic. Built-in politeness settings (AutoThrottle), middlewares, item pipelines, and mature deployment options.
  • Weaknesses: doesn't run JavaScript on its own, so content rendered client-side needs extra help (for example the scrapy-playwright plugin).
  • Use it for: large catalogues, news and listing sites, and anything where the data is in the HTML or in an API the page calls.

Playwright: the modern browser

Playwright drives real Chromium, Firefox or WebKit browsers with a clean async API.

  • Strengths: handles JavaScript-heavy single-page apps, logins, infinite scroll and clicks. Auto-waiting makes scripts much more stable, and it can intercept network requests, which is often the best way to grab the JSON a page loads.
  • Weaknesses: each browser uses far more CPU and memory than an HTTP request, so it's slower and more expensive at scale.
  • Use it for: dynamic sites, complex interactions, and sites where you need to see the page as a user does.
import asyncio
from playwright.async_api import async_playwright

async def main():
    async with async_playwright() as p:
        browser = await p.chromium.launch()
        page = await browser.new_page()
        await page.goto("https://example.com/listings")
        titles = await page.locator(".listing-title").all_inner_texts()
        print(titles)
        await browser.close()

asyncio.run(main())

Selenium: the veteran

Selenium has been around the longest and supports the widest range of languages and browsers.

  • Strengths: huge community, Selenium Grid for distributed runs, and lots of existing code and tutorials.
  • Weaknesses: more boilerplate and more manual waiting than Playwright, and generally slower for scraping.
  • Use it for: maintaining existing Selenium projects, or when a team already has deep Selenium expertise.

For new browser-based scrapers, we now default to Playwright.

Comparison

ScrapyPlaywrightSelenium
Runs JavaScriptWith a pluginYesYes
Speed / cost at scaleBestModerateLowest
Built-in crawling featuresExtensiveMinimalMinimal
Script stabilityHighHighMedium
Best forVolumeDynamic sitesLegacy / Grid

The hybrid approach we use most

  1. Look for the API first. Open DevTools, watch the Network tab, and see whether the page loads its data as JSON. Calling that endpoint directly is often 10–100× faster than rendering the page.
  2. Use Scrapy or plain HTTP for the bulk of pages.
  3. Use Playwright only where needed: logins, tokens, or pages that genuinely require a browser.

Increasingly, a fourth tool sits on top of all three: an LLM that turns messy pages into structured data. More on that in a future post.

Not sure which approach suits your target sites? Our web scraping team can run a quick feasibility check.