Is Web Scraping Legal? US Law and Commercial Use

Web scraping is generally legal in the United States when you collect publicly available data without logging in, bypassing technical barriers, or harming the site. It becomes risky when you break a contract you agreed to, get around a login or block, copy copyrighted content, or collect personal data. Commercial use is not illegal in itself, but it raises the stakes on every one of those issues.

This guide explains the court cases that shaped US law, the other legal theories site owners use, and a checklist for scraping in a business. It's general information, not legal advice. For a specific project, talk to a lawyer.

The short answer by situation

What you're doing	General risk level in the US
Scraping public pages, logged out, at a gentle rate	Lower
Scraping after logging in to an account (you agreed to the terms)	Higher: contract claims are much stronger
Bypassing CAPTCHAs, IP blocks, or other technical barriers	Higher
Using fake accounts	High: this is what sank hiQ
Republishing copyrighted text, images, or articles	High: copyright applies regardless of how you got it
Collecting personal data about people	Depends heavily on privacy laws like GDPR and CCPA

The Computer Fraud and Abuse Act (CFAA)

The CFAA is the federal anti-hacking law. It makes it a crime, and a basis for civil lawsuits, to access a computer "without authorization" or to "exceed authorized access." For years, companies argued that scraping in violation of their terms, or after a cease-and-desist letter, met that definition. Two cases narrowed that argument.

Van Buren v. United States (2021)

Van Buren v. United States, decided June 3, 2021 by a 6-3 vote, was not a scraping case. A police officer used his valid access to a license plate database for an improper purpose. The Supreme Court held that someone "exceeds authorized access" only when they access areas of a computer that are off limits to them, not when they misuse information they were allowed to reach. The Court described this as a "gates-up-or-down inquiry."

For scrapers, the key point is that misuse of data you can already reach is not a CFAA violation. The Court left one question open in a footnote: whether the gates are only technical barriers, like passwords, or also "limits contained in contracts or policies."

hiQ Labs v. LinkedIn (2017 to 2022)

hiQ Labs scraped public LinkedIn profiles to sell workforce analytics. LinkedIn sent a cease-and-desist letter and blocked it, and hiQ sued. The case went like this:

2017: A federal district court ordered LinkedIn to stop blocking hiQ while the case went on.
2019: The Ninth Circuit affirmed, finding hiQ had raised serious questions about whether scraping public profiles was access "without authorization."
June 2021: The Supreme Court vacated that decision and sent it back for reconsideration in light of Van Buren.
April 2022: The Ninth Circuit ruled for hiQ again. Its reasoning was that the CFAA's "without authorization" concept likely does not apply to websites open to the general public.
November 2022: The district court found that hiQ had breached LinkedIn's User Agreement through its scraping and its creation of fake accounts.
December 2022: The parties settled. According to the proposed consent judgment, hiQ agreed to a $500,000 judgment and a permanent injunction, and to delete its scraping code and the LinkedIn data it had collected.

So hiQ is often cited as a win for scrapers, and on the CFAA question about public data it was. But hiQ lost on contract, largely because of fake accounts, and the company ended up shut out of LinkedIn entirely.

Terms of service and contract claims

After the CFAA cases, breach of contract became the main tool for site owners. Whether a site's terms bind you depends on how you encountered them:

You created an account and clicked "I agree." That's a clickwrap agreement, and courts usually enforce it. Scraping while logged in is where most contract risk lives.
The terms are only linked in the footer. That's browsewrap, and courts are more skeptical that a visitor ever agreed.

Meta v. Bright Data (2024)

Meta sued Bright Data, a data collection company, for scraping Facebook and Instagram. On January 23, 2024, Judge Edward Chen of the Northern District of California granted summary judgment to Bright Data on Meta's breach of contract claim. The court held that Meta's terms could not be read to prohibit scraping publicly available data while logged out of Facebook or Instagram. Weeks later, Meta dropped the lawsuit and waived its right to appeal.

The lesson: the exact wording of the terms matters, and logged-out collection of public data is in a much better position than logged-in collection.

X Corp. v. Bright Data (2024)

X (formerly Twitter) sued Bright Data on similar grounds. In May 2024, Judge William Alsup dismissed the suit, reasoning in part that federal copyright law preempted X's claims about the scraping and sale of users' posts. In November 2024 the court let X proceed with amended claims that the scraping harmed its servers, while keeping the copyright-preempted claims dismissed. Claims based on server burden, sometimes framed as trespass to chattels, are a reminder that aggressive request rates create their own risk.

Copyright and databases

Scraping and copying are separate questions. Facts are not protected by copyright in the US: in Feist v. Rural Telephone (1991), the Supreme Court held that a phone directory's names and numbers weren't copyrightable. Prices, dates, and similar facts are generally fair to collect.

The creative expression around facts is protected. Articles, reviews, photos, and product descriptions can't be republished just because you scraped them. Using copyrighted content internally, for analysis or search, can sometimes qualify as fair use, but that's a fact-specific defense, not a rule. How copyright applies to AI training data is still being argued in active lawsuits.

The US has no separate database right. The EU does: the Database Directive protects databases that required substantial investment, and extracting a substantial part of one can infringe even if the individual facts aren't copyrighted. If you scrape EU sites, factor that in.

Personal data is where public availability helps you least.

GDPR applies to data about people in the EU, wherever the scraper is based. Being public does not make personal data free to use. You still need a lawful basis, and Article 14 generally requires informing people when you collect their data from somewhere other than them. In September 2024 the Dutch data protection authority fined Clearview AI €30.5 million, in part over scraping facial images of people online.
CCPA applies to businesses that meet thresholds such as $25 million in annual revenue. The California Attorney General notes that personal information "does not include publicly available information," but that exception is defined narrowly, and other state privacy laws have their own rules.

If you don't need personal data, don't collect it. That one decision removes most privacy risk.

Is robots.txt legally binding?

Not by itself. The robots.txt standard (RFC 9309) states that its rules "are not a form of access authorization." It's a request to crawlers, not a lock.

It still matters legally. Many terms of service incorporate robots.txt, which turns it into part of the contract. Ignoring it can also undercut a good-faith defense, and it is evidence that you knew the site didn't want your bot there. Treat it as a rule you follow, not one you test.

Checklist for commercial web scraping

Before you build a product or business process on scraped data:

Read the target site's terms. Look for clauses on automated access, scraping, and commercial use. Note whether they apply to all visitors or only account holders.
Stay logged out if you can. Scraping behind a login makes contract claims much stronger.
Never use fake accounts or bypass technical barriers. Don't defeat CAPTCHAs, rotate around IP bans, or evade blocks the site put in place.
Respect robots.txt and any API the site offers. An official API with clear terms is usually the safest path. See web scraping vs. API.
Rate-limit your requests so you never burden the site's servers.
Collect facts, not expression. Don't republish copyrighted text or images without permission.
Minimize personal data. Skip it if you can. If you can't, document your lawful basis and follow GDPR and CCPA requirements.
Stop when asked. A cease-and-desist letter changes your risk. Get legal advice before continuing.
Keep records of what you collected, when, and under which terms.
Consult a lawyer if the data is core to your product or you plan to sell it.

For pages you're permitted to process, tools like URLtoText can convert them to clean text or Markdown. The tool doesn't change the legal analysis: the same terms, copyright, and privacy rules apply.

Frequently asked questions

Is web scraping legal in the US?

Generally yes, for publicly available data collected without logging in, bypassing barriers, or overloading the site. After Van Buren and hiQ v. LinkedIn, scraping public pages is unlikely to violate the CFAA. The main remaining risks are breach of contract, copyright, and privacy law.

Is web scraping legal for commercial use?

Commercial use is not illegal in itself. hiQ and Bright Data both scraped for commercial purposes. But commercial use makes terms of service, copyright, and privacy questions more important, and many sites' terms specifically ban commercial use of their data. Follow the checklist above and get legal advice for anything core to your business.

Can I get in trouble for scraping a website that bans it in its terms of service?

Yes, especially if you agreed to the terms by creating an account. hiQ was found to have breached LinkedIn's User Agreement. Meta v. Bright Data shows the outcome depends on the terms' wording and whether you were logged in. Account bans, IP blocks, and cease-and-desist letters are the more common consequences.

Is it legal to scrape data to train AI models?

It's unsettled. Copyright lawsuits over AI training are ongoing, and many sites now ban AI training in their terms and block AI crawlers in robots.txt. Indeed is one example: see our breakdown of Indeed's scraping rules.

Does ignoring robots.txt make scraping illegal?

Not on its own, since robots.txt is not access authorization under the standard. But many terms of service make it part of the contract, and ignoring it is evidence that you knew your bot was unwelcome.