Web Scraping

Scraping JavaScript-Heavy and Bot-Protected Sites, Responsibly

Home Blog Web Scraping

Web Scraping

The days of every page arriving as plain HTML are long gone. Many sites now load data with JavaScript after the page opens, and many use bot-management services to control automated traffic. Here's how we collect public data from these sites reliably, and where we draw the line.

Part 1: JavaScript-rendered content

Find the data source first

If a page shows data that isn't in the initial HTML, the browser fetched it from somewhere. Open DevTools, go to the Network tab, filter by Fetch/XHR and reload. Very often you'll find a clean JSON endpoint behind the page. Requesting it directly is:

  • much faster and cheaper than rendering the page,
  • more stable, because JSON structures change less often than layouts,
  • lighter on the target server.

Also check the page source for embedded data, such as __NEXT_DATA__, JSON-LD or initial state objects. Many modern frameworks ship the data inside the HTML.

When you really need a browser

Use Playwright when content depends on interactions (clicks, scrolling, filters) or when tokens are generated in the browser. Tips for stable browser scrapers:

  • Wait for specific elements or network responses, never fixed sleep() calls.
  • Block unnecessary resources such as images, fonts and analytics to save bandwidth and time.
  • Capture the JSON responses the page receives with page.on("response", ...) instead of parsing the rendered DOM.
  • Reuse browser contexts and limit concurrency to keep memory under control.
async def capture_listings(page):
    results = []
    async def on_response(resp):
        if "/api/listings" in resp.url and resp.ok:
            results.extend((await resp.json())["items"])
    page.on("response", on_response)
    await page.goto("https://example.com/search?city=thimphu")
    await page.wait_for_selector(".listing-card")
    return results

Part 2: Bot protection

Bot-management systems look at many signals, including request rate, IP reputation, browser fingerprints and behaviour, to decide whether traffic is automated. Their purpose is legitimate: protecting sites from abuse, credential stuffing and overload.

Our approach is to reduce friction by behaving well, not to defeat protections:

  • Crawl slowly. Low concurrency and randomised delays look like, and are, light traffic.
  • Request only what you need. Skip pages and assets you don't use.
  • Cache aggressively and re-fetch only pages that are likely to have changed.
  • Run at off-peak times for the target's audience.
  • Handle errors gracefully. Back off on 429 and 503 responses instead of retrying immediately.

Where we draw the line

If a site deliberately blocks automated access, through CAPTCHAs, login walls or explicit terms, that's a clear signal from the owner. In those cases we recommend:

  1. Look for an official API or data feed, even a paid one.
  2. Ask for permission or a data partnership. Many site owners agree when the use is reasonable.
  3. Use licensed data providers that already have agreements in place.
  4. Re-think the requirement. Sometimes a different public source has the same information.

We don't build tools to solve CAPTCHAs, bypass logins or impersonate users. Beyond the legal risk (covered in our guide on whether web scraping is legal), scrapers that fight protections break constantly and cost far more to maintain.

Monitoring keeps scrapers alive

Dynamic sites change often. Track success rates, record counts and field completeness per source, and alert when they drop. A broken scraper that is caught the same day is a small fix. One found a month later is a data gap.

Need reliable data from dynamic websites? Our web scraping team designs collection pipelines that are robust and responsible.