Web Scraping

AI-Powered Scraping: Using LLMs to Extract Structured Data from Any Website

Home Blog Web Scraping

Web Scraping

For years, web scraping meant writing CSS selectors or XPath for each site and fixing them every time the site changed. Large language models (LLMs) have changed that. They can read a messy page and return clean, structured data, even from sites they've never seen. Here's how we use them in production without runaway costs.

The basic idea

  1. Fetch the page (HTTP or a headless browser).
  2. Clean it: strip scripts, styles, navigation and footers, and convert to Markdown or simplified text.
  3. Send it to an LLM with a schema describing the fields you want.
  4. Validate the response and store it.
from pydantic import BaseModel, HttpUrl

class JobPosting(BaseModel):
    title: str
    company: str
    location: str | None
    salary_min: float | None
    salary_max: float | None
    currency: str | None
    employment_type: str | None   # full-time, contract, ...
    apply_url: HttpUrl | None

# Ask the model for output matching JobPosting's JSON schema,
# then validate: JobPosting.model_validate_json(response_text)

Most LLM APIs now support structured output that follows a JSON schema, which makes responses easy to validate.

Where LLM extraction shines

  • Many different sites with different layouts, such as hundreds of company career pages, supplier catalogues or event listings.
  • Unstructured text: pulling specifications, dates or terms out of paragraphs.
  • Normalisation: "₹12–15 LPA", "12 to 15 lakh per annum" and "1.2M–1.5M INR" all mapped to the same numeric fields.
  • Classification: tagging products, sentiment or categories at the same time as extraction.

Where traditional scraping still wins

  • High volume from a few sites. Selectors or a JSON API cost almost nothing per page. LLM calls don't.
  • Exact values such as prices and IDs, where any risk of a model mistake is unacceptable.
  • Structured data already on the page (JSON-LD, embedded JSON). Just parse it.

The hybrid pattern we use

  1. Try structured data first (JSON-LD, APIs).
  2. For high-volume sites, use the LLM to generate selectors once, then run cheap selector-based extraction, re-generating when the success rate drops.
  3. For the long tail of varied sites, use direct LLM extraction.

This keeps costs low where volume is high and stays flexible where variety is high.

Controlling cost

  • Shrink the input. Clean HTML to text or Markdown and send only the relevant section. This often cuts tokens by 80–90%.
  • Use smaller, faster models for straightforward extraction, and save the large models for hard cases.
  • Cache by content hash, so unchanged pages are never processed twice.
  • Batch requests where the provider offers discounted batch processing.

Keeping accuracy high

  • Allow nulls. Tell the model to return null when a field isn't present instead of guessing.
  • Validate with rules: prices are positive, dates are plausible, URLs resolve.
  • Spot-check samples against the source pages regularly, and track accuracy per field.
  • Keep the source text next to the extracted data, so any value can be traced and verified.

Responsible use still applies

AI doesn't change the rules: respect site terms and robots.txt, avoid personal data you don't need, and crawl politely. See our guide on the legal side of scraping.

Need structured data from hundreds of different websites? Our web scraping team builds AI-assisted extraction pipelines that stay accurate and affordable.