Almost every client who asks us about a scraping project asks the same question first: "Is this legal?" The honest answer is that it depends on what you collect, how you collect it and what you do with it. This guide covers the main factors in plain English.
This article is general information, not legal advice. Laws differ by country, so for high-stakes projects talk to a lawyer in your jurisdiction.
1. Public vs. protected data
Collecting information that anyone can see without logging in, such as product prices, public listings or published statistics, is generally the lowest-risk kind of scraping. Risk rises sharply when you:
- scrape content behind a login or paywall,
- get around technical barriers designed to keep you out, or
- use someone else's account credentials.
In several jurisdictions, getting around access controls can fall under computer-misuse laws, not just contract law.
2. Terms of service
Many websites prohibit automated access in their terms. Whether those terms bind you often depends on whether you actively agreed to them, for example by creating an account. Even where enforceability is unclear, ignoring explicit terms raises your risk and can damage business relationships. Read them before you start.
3. Personal data
This is where most real-world problems happen. If you collect names, emails, phone numbers, photos or anything else that identifies a person, privacy laws such as the EU's GDPR, the UK GDPR, India's DPDP Act and California's CCPA may apply. That is true even if the data is publicly visible. These laws generally require:
- a lawful basis for collecting and processing the data,
- collecting only what you actually need,
- telling people how their data is used, and honouring deletion requests,
- keeping the data secure.
If your project doesn't need personal data, design the scraper so it never collects it.
4. Copyright and database rights
Facts such as a price or a date are generally not protected by copyright, but the creative expression around them (articles, photos, reviews) usually is. Republishing scraped content wholesale is risky. Using it to calculate your own insights, such as averages, trends or comparisons, is far safer. The EU also has a separate "database right" that protects substantial extraction from databases built with significant investment.
5. How you scrape matters
Aggressive scraping that slows down or crashes a site can cause harm and legal exposure on its own. Responsible scrapers:
- respect
robots.txtas a strong signal of the owner's wishes, - rate-limit requests and avoid peak hours,
- identify themselves with an honest User-Agent where appropriate,
- cache results instead of re-downloading unchanged pages,
- use official APIs whenever they exist.
A quick risk checklist
| Lower risk | Higher risk |
|---|---|
| Public pages, no login | Content behind login or paywall |
| Facts and numbers | Articles, images, reviews republished |
| No personal data | Names, emails, phone numbers |
| Polite rate limits | High-volume crawling that affects the site |
| Internal analysis | Reselling the raw data |
How we approach it
Every web scraping project we build starts with a short compliance review: what data is collected, whether an API exists, whether personal data is involved and how the output will be used. That review shapes the technical design, so the scraper collects only what is needed and runs politely.