URLtoText

Blog / Guides

How to Prevent Web Scraping on Your Website

By URLtoText Team · · 9 min read

You can't make a public website impossible to scrape, but you can make scraping slow, expensive, and easy to detect. The approach that works is layered: tell well-behaved bots what's off limits with robots.txt, rate-limit everyone, put a bot management service in front of the site, gate valuable data behind a login, and watch your logs. Back it up with clear terms of service and DMCA notices when someone republishes your content.

This guide walks through each layer, with working robots.txt and nginx examples.

Decide what you're protecting

Blocking every bot also blocks the search engines that send you traffic. Before you change anything, sort your content:

Content Typical approach
Marketing pages, blog posts Let search engines in, block AI training crawlers if you want, rate-limit everyone
Pricing, listings, inventory Rate limits plus bot management
User profiles, contact details, proprietary data Require login, strict rate limits, monitor for bulk access
Search and API endpoints Tight per-IP limits, since these are the cheapest way to pull lots of data

1. Use robots.txt (and know its limits)

robots.txt sits at the root of your domain and tells crawlers which paths they shouldn't fetch. Search engines and the major AI companies follow it. Scrapers that don't want to be stopped simply ignore it, and the robots.txt standard (RFC 9309) says plainly that its rules "are not a form of access authorization."

So use robots.txt for what it does well: steering legitimate crawlers and opting out of AI training. Here's an example that keeps search engines, blocks the main AI training crawlers, and keeps everyone out of private paths:

# AI training crawlers: block the whole site
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
User-agent: Google-Extended
Disallow: /

# Everyone else, including search engines
User-agent: *
Disallow: /admin/
Disallow: /account/
Disallow: /search
Disallow: /trap/

Sitemap: https://www.example.com/sitemap.xml

A crawler named in a group follows that group and ignores the * group, so GPTBot obeys only the first group. The /trap/ line is used for the honeypot in step 5.

Blocking AI crawlers

Use the user-agent tokens the AI companies publish, since guessed names do nothing:

Token Company What blocking it does
GPTBot OpenAI Signals your content shouldn't be used to train OpenAI's foundation models (OpenAI docs)
OAI-SearchBot OpenAI Removes your site from ChatGPT search answers
ClaudeBot Anthropic Excludes your future content from Anthropic's training data (Anthropic docs)
Claude-SearchBot, Claude-User Anthropic Reduces your visibility in Claude's search and user-requested fetches
Google-Extended Google Stops use of your content for Gemini training and grounding, with no effect on Google Search ranking (Google docs)
CCBot Common Crawl Stops the crawl that feeds the Common Crawl dataset, which is widely used for AI training (Common Crawl)

A few details from the official docs:

  • Google-Extended is a control token only. Google says it "doesn't have a separate HTTP request user agent string," so you can't block it in your server config. Use robots.txt.
  • OpenAI says that for ChatGPT-User, which fetches pages when a user asks, "robots.txt rules may not apply."
  • Anthropic advises against blocking its IP addresses as a way to opt out, because that can stop its bots from reading your robots.txt.
  • Common Crawl warns that some crawlers falsely identify as CCBot and recommends verifying with reverse DNS.

Deciding which to block is a trade-off. Blocking training crawlers keeps your content out of future models. Blocking search and retrieval bots can also remove you from AI search answers.

2. Rate-limit requests

Rate limiting is the single most effective technical control, because scraping at scale means making a lot of requests. nginx has it built in with the limit_req module:

http {
    # 10 requests per second per IP for normal pages, 1 per second for search.
    limit_req_zone $binary_remote_addr zone=pages:10m  rate=10r/s;
    limit_req_zone $binary_remote_addr zone=search:10m rate=1r/s;
    limit_req_status 429;

    # Return 403 to scrapers that announce themselves.
    map $http_user_agent $blocked_agent {
        default                                   0;
        ~*(GPTBot|ClaudeBot|CCBot|Bytespider)     1;
        ~*(python-requests|scrapy|curl|wget)      1;
    }

    server {
        listen 80;
        server_name example.com;

        if ($blocked_agent) {
            return 403;
        }

        location / {
            limit_req zone=pages burst=20 nodelay;
            proxy_pass http://127.0.0.1:8000;
        }

        location /search {
            limit_req zone=search burst=5;
            proxy_pass http://127.0.0.1:8000;
        }
    }
}

How it works:

  • limit_req_zone tracks each client IP in shared memory. A 10 MB zone holds state for many thousands of IPs.
  • burst=20 nodelay lets a real visitor load a page with a burst of requests, then rejects anything beyond the limit with a 429 Too Many Requests.
  • The search location has a lower rate and no nodelay, so excess requests queue up and slow down instead of failing immediately.
  • The map blocks clients by user-agent string. That only stops bots that identify themselves honestly, but plenty do.

Tune the numbers against your real traffic before you enforce them. If you're behind a CDN or load balancer, configure nginx's real IP module so you limit the visitor's IP and not your proxy's.

3. Add a WAF or bot management service

Per-IP limits struggle against scrapers spread over many addresses. Bot management services score each request using signals your server can't see on its own, such as browser fingerprints, behavior, and reputation across many sites.

  • Cloudflare offers Bot Fight Mode, Super Bot Fight Mode, and Bot Management, with the full Bot Management product for Enterprise customers. Its AI Crawl Control, available on all plans, shows which AI crawlers visit your site and lets you allow or block each one.
  • AWS WAF has Bot Control, a managed rule group with a common protection level for self-identifying bots and a targeted level for sophisticated bots that don't identify themselves. It costs extra on top of standard AWS WAF fees.

These services also verify good bots, so Googlebot keeps working while impostors using its user agent get blocked. You can do the same check yourself with the reverse DNS method in Google's verification guide.

4. Use CAPTCHAs and challenges selectively

A CAPTCHA on every page drives away real visitors. Use challenges where abuse happens: after a rate limit trips, on search forms, on signup, or on pages that expose bulk data. Cloudflare Turnstile and Google reCAPTCHA both have modes that don't make most users solve a puzzle.

5. Set honeypots

A honeypot is a link that humans never see and polite crawlers never follow. Anything that requests it is a bot ignoring your robots.txt.

  1. Add Disallow: /trap/ to robots.txt (as in the example above).

  2. Add a hidden link to your page template:

    <a href="/trap/" rel="nofollow" tabindex="-1" aria-hidden="true" style="display:none">Archive</a>

  3. Log every IP that requests /trap/, and block or challenge it.

Search engines that respect robots.txt will never hit it, so false positives are rare.

6. Require login for valuable data

Content behind a login is much harder to scrape in bulk, and it gives you more leverage. Accounts have to accept your terms, every request is tied to an identity you can rate-limit, and you can ban an account instead of chasing IP addresses. Watch for many new accounts from the same network or email pattern, which is a common way to spread scraping across accounts.

7. Monitor your logs

Scraping usually shows up in your access logs before anywhere else: one IP or subnet requesting thousands of pages, sequential URLs (/product/1, /product/2, ...), no CSS or image requests, or a single user agent at odd hours. This Python script lists the busiest IPs in a standard nginx access log:

import re
import sys
from collections import Counter

LINE = re.compile(r'^(\S+) \S+ \S+ \[[^\]]+\] "(?:\S+) (\S+)[^"]*" (\d{3}) \S+ "[^"]*" "([^"]*)"')

hits = Counter()
agents = {}

with open(sys.argv[1], encoding="utf-8", errors="replace") as log:
    for line in log:
        match = LINE.match(line)
        if not match:
            continue
        ip, path, status, agent = match.groups()
        hits[ip] += 1
        agents[ip] = agent

for ip, count in hits.most_common(20):
    print(f"{count:>8}  {ip:<40}  {agents[ip][:60]}")

Run it with python top_ips.py /var/log/nginx/access.log. Check the top addresses with reverse DNS before blocking anything, since some will be search engines you want to keep.

8. Make content less valuable to copy

A few changes reduce what a scraper gains:

  • Watermark images and serve full-resolution versions only to logged-in users.
  • Paginate or cap API results so no single request returns your whole catalog.
  • Avoid sequential IDs in URLs if the order lets someone enumerate every record.

Technical controls stop most scraping. Legal ones handle what gets through.

  • Terms of service. State clearly that automated access, scraping, and commercial reuse require permission. Terms that users accept by clicking when they sign up are much easier to enforce than a link in the footer. See Is web scraping legal? for how courts have treated these claims.
  • Cease-and-desist letters. If you can identify the scraper, a letter is often enough to make them stop.
  • DMCA takedown notices. When someone republishes your copyrighted content, you can send a notice under 17 U.S.C. § 512 to the site's host and to search engines to get the copies removed. The US Copyright Office has an overview of section 512.

Which measures to start with

For most sites, this order gives the best return:

  1. robots.txt with AI crawler rules
  2. nginx (or CDN) rate limits on all pages, tighter on search
  3. A bot management service if you're on a CDN that offers one
  4. Log monitoring and a honeypot
  5. Login gates and CAPTCHAs for your most valuable data
  6. Clear terms of service and DMCA notices for copies

Frequently asked questions

Can you completely prevent web scraping?

No. Anything a browser can display, a determined scraper can collect. The goal is to make scraping slow, costly, and detectable enough that most scrapers give up or never start.

Does robots.txt block scrapers?

Only the ones that choose to follow it, such as search engines and the major AI crawlers. The standard itself says robots.txt rules "are not a form of access authorization." Use it alongside rate limits and bot management, not instead of them.

How do I block AI crawlers like GPTBot and ClaudeBot?

Add User-agent: GPTBot, User-agent: ClaudeBot, User-agent: CCBot, and User-agent: Google-Extended with Disallow: / to your robots.txt, as in the example above. Cloudflare's AI Crawl Control can also block AI crawlers at the network level.

What are the best anti-scraping tools?

For most sites, a CDN with bot management (such as Cloudflare's bot products or AWS WAF Bot Control), server rate limiting with nginx's limit_req module, and a CAPTCHA like Turnstile or reCAPTCHA on high-risk forms. Pair them with log monitoring so you can see what gets through.

Will blocking scrapers hurt my SEO?

Not if you're careful. Let verified search engine crawlers through, and remember that blocking Google-Extended doesn't affect Google Search. Test new rate limits and bot rules against your real traffic before you enforce them.