Web Scraping

Building a Price-Monitoring System with Python

Home Blog Web Scraping

Web Scraping

For e-commerce and retail businesses, knowing when a competitor changes a price, runs out of stock or launches a promotion is directly worth money. Here's how we design price-monitoring systems that keep working for months, not days.

The architecture

  1. Scheduler: decides which product pages to check and how often.
  2. Fetchers: download pages politely, with retries and rate limits.
  3. Parsers: extract price, currency, stock status and promotions.
  4. Matcher: links competitor products to your own catalogue.
  5. Storage: keeps every observation with a timestamp.
  6. Alerts and reports: notify people only when something meaningful changes.

Fetching: start simple

Many product pages include structured data. Check for JSON-LD Product markup or the JSON the page's own JavaScript loads before reaching for a headless browser:

import json, httpx
from selectolax.parser import HTMLParser

def get_price(url: str) -> dict | None:
    html = httpx.get(url, timeout=20, headers={"User-Agent": "PriceBot/1.0"}).text
    for node in HTMLParser(html).css('script[type="application/ld+json"]'):
        data = json.loads(node.text())
        if isinstance(data, dict) and data.get("@type") == "Product":
            offer = data.get("offers", {})
            return {"price": offer.get("price"),
                    "currency": offer.get("priceCurrency"),
                    "in_stock": "InStock" in str(offer.get("availability", ""))}
    return None

Structured data is far more stable than CSS selectors, which break whenever the site is redesigned. Fall back to selectors, then to a headless browser such as Playwright, only when needed.

Product matching: the hard part

Scraping a price is easy. Knowing that "Samsung 55in QLED Q60D" on one site is the same product as "Samsung QE55Q60DAUXXU" on another is hard. Match in layers:

  • Exact identifiers first: GTIN/EAN, UPC, MPN, model numbers.
  • Normalised titles next: lowercase, strip units and noise words, compare with fuzzy matching.
  • Human review for uncertain matches, with a simple admin screen to confirm or reject.

Store confirmed matches so the expensive work is done only once.

Storage: keep history, not just the latest price

Save every observation (product, competitor, price, stock, timestamp). History lets you spot patterns such as weekend discounts or stock-outs before holidays, and it protects you from a single bad parse triggering a false alert. PostgreSQL handles this volume comfortably for most businesses.

Alerts people actually read

  • Alert on meaningful changes (for example more than 3%, or out of stock), not every update.
  • Confirm an unusual change with a second fetch before alerting.
  • Send a daily digest by email or Slack, with real-time alerts only for key products.

Keeping it healthy

Sites change. Track the success rate of every parser and alert your team when a site's success rate drops. That's usually a sign of a redesign, and catching it the same day keeps the data trustworthy. And always scrape responsibly. See our guide on the legal side of web scraping.

Need competitor or market price data? Our web scraping team can build and run the whole pipeline for you.