Indeed Web Scraping: Terms of Service, robots.txt, and What's Allowed

Indeed does not allow scraping without its permission. Its terms of service prohibit using bots, scrapers, spiders, or AI agents to access or data-mine the site without Indeed's express written permission, and the only crawling it permits is what its robots.txt file allows. That robots.txt file blocks the pages most scrapers want, including individual job listings, for all general-purpose bots.

Below is what the terms and robots.txt actually say (we checked both on October 17, 2026), what happens if you ignore them, and the legitimate ways to get job data instead.

What Indeed's terms of service say about scraping

Indeed's terms of service live at indeed.com/legal. The version we reviewed is marked "Last Updated: September 29, 2026". The scraping rules sit in Section D (General Terms for All Users), part 21, Site Rules, which says breaking any rule is "a material breach of the Terms."

The key clause on automated access reads:

Use any automated system (bots, scrapers, spiders, AI or Agentic AI) to access, data-mine, or submit content to the Site, in bulk or otherwise, without Indeed's express written permission (we conditionally grant permission to crawl the Site solely as outlined in our robots.txt file). You may not crawl, scrape, extract data from, reproduce, duplicate, copy, sell, exploit, trade or resell any part of the Site or access the Site for the development, training, fine-tuning, or improvement of any third-party machine learning model, artificial intelligence (AI) system, or any related software program, model, algorithm, or generative AI tool;

The same list of Site Rules also tells users not to:

"Access the Site through any means other than the public interfaces provided by Indeed"
"Access any data, especially personal data, by automated means or otherwise, without permission"
"Bypass any security measures or Site limitations"
"Use or misappropriate the Site for your own commercial gain"
"Modify, adapt, translate, reformat, resell, frame, or reverse engineer any portion of the Site"

Other parts of the terms add more specific limits:

Automated applications: job seekers may not use "automation, scripting, or bots to automate the Indeed Apply process" outside Indeed's official vendors and tooling.
Accounts: you "may not create multiple accounts or use automated means to create accounts."
Resume data: for employers using Smart Sourcing, "scraping the database" is "strictly prohibited and will result in termination of your access."
AI connectors: where Indeed offers official connections to third-party AI platforms, the terms state that the prohibitions "against scraping, bots, and other automated activity, remain in full effect."

Indeed also reserves "the right to use technical measures to detect, investigate, or prevent unauthorized automated access or activity on the Site."

In plain terms: unless you have a written agreement with Indeed, or you are a crawler doing only what robots.txt allows, automated collection from Indeed breaks its terms.

What Indeed's robots.txt disallows

The terms point to indeed.com/robots.txt as the one place where crawling is "conditionally" permitted. The file sorts bots into groups with different rules. Here is how it looked on October 17, 2026.

Bot group	Examples named in the file	What's allowed
All other bots (`User-agent: *`)	Any crawler not named elsewhere, including Bingbot	Most of the site, but not job listing pages (`/viewjob`, `/job/`, `/m/viewjob`, `/rc/`), internal APIs (`/graphql`, `/rpc/`, `/api/getrecjobs`), company pages under `/company/`, account pages, or search results past the 10th page
Search engines and AI retrieval bots	Googlebot, Google-Extended, OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-SearchBot, Claude-User	Similar rules to the default group, including blocks on `/job/` and `/graphql`. Unlike the default group, `/viewjob` is not on their disallow list
AI training bots	GPTBot, CCBot, ClaudeBot, anthropic-ai, Applebot-Extended, Bytespider, Diffbot	Blocked from job search (`/jobs`, `/q-`), job listings, company pages (`/cmp/`), career advice, and many other sections
Fully blocked bots	Scrapy, GoogleOther, ia_archiver, JobdiggerSpider, Webzio	`Disallow: /` for the whole site

A few details stand out:

The default group starts with Allow: / but then disallows /viewjob and /job/. Those are the paths for individual job postings, so a generic crawler is not permitted to fetch the job detail pages.
Pagination is limited. The file allows &start=0 through &start=90 and disallows other &start= values, which caps crawlable search results at about ten pages.
Scrapy, the user agent of the popular Python scraping framework, is listed under "Full disallow for aggressive scrapers, legacy crawlers, and unwanted utility bots."

Two caveats. The robots.txt standard (RFC 9309) says its rules "are not a form of access authorization," but Indeed's terms make robots.txt the boundary of its permission, so ignoring it also breaks the terms. And following robots.txt doesn't cancel the rest of the terms, which still prohibit things like reselling data or training AI models on it.

Is scraping Indeed illegal?

Breaking a website's terms is mainly a contract issue, not automatically a crime. US courts have narrowed the Computer Fraud and Abuse Act in recent years, and some rulings have found that scraping public pages while logged out did not breach a site's terms. Other cases went the other way, especially where the scraper used fake accounts. We cover these cases (hiQ v. LinkedIn, Van Buren, Meta v. Bright Data) in Is web scraping legal?.

For Indeed specifically, the facts that matter are:

The terms explicitly ban scraping and automated access without written permission.
Logging in means you have clearly agreed to those terms.
Job postings often include names, emails, and phone numbers of recruiters, and resumes are full of personal data. Privacy laws such as GDPR and CCPA can apply to that data no matter how you collected it.
Job descriptions are written by employers and may be protected by copyright, so republishing them is a separate risk.

This is general information, not legal advice. If job data is core to a commercial product, talk to a lawyer before you build on scraped Indeed data.

The practical risks

Even before any legal question, scraping Indeed tends to fail:

Blocking. Indeed says it uses technical measures to detect unauthorized automated access. Expect CAPTCHAs, verification pages, and blocked IP addresses.
Account bans. The terms let Indeed "suspend or terminate your account... at any time, for any or no reason." If you scrape while logged in, the account you lose may be the one your company uses to hire.
Legal letters and lawsuits. A cease-and-desist letter is the usual first step from a site that wants scraping to stop. Continuing after one weakens any argument that you didn't know access was unwanted.
Unreliable data. Scrapers break when the layout changes, and you have no rights to what you collected.

Legitimate alternatives to scraping Indeed

Indeed's own APIs and programs

Indeed runs a developer platform, referenced in its terms as the "Indeed Developer Portal" and governed by separate Indeed API Terms. Its documented APIs are aimed at employers and applicant tracking system (ATS) partners. For example, the Job Sync API lets ATS partners create and manage job postings on Indeed, and other APIs cover Indeed Apply and Sponsored Jobs. Access generally requires onboarding with Indeed.

We could not find a current, public Indeed API for searching or downloading job listings. Indeed's older public job search API (part of its Publisher program) is no longer offered, according to several third-party developer resources. If you need Indeed data at scale, the realistic route is to contact Indeed about a partnership and get written permission, which is exactly what the terms ask for.

Public job feeds and APIs

Many jobs on Indeed are also posted on the employer's own careers site or ATS, and several sources publish job data openly:

ATS job board APIs. Greenhouse's Job Board API and Lever's Postings API return a company's published jobs as JSON. They are designed for public job boards, so you can read them without scraping HTML.
Government job data. The USAJOBS API provides US federal job announcements with documented terms and rate limits.
Job search APIs. Adzuna's API offers job ad search and labor market data under its own terms.
Structured data on careers pages. Many employer job pages include schema.org JobPosting markup so search engines can read them. Check each site's terms before you collect it.

Licensed job-data providers

Commercial job-data vendors sell aggregated listings. Before you buy, ask how the data is sourced and whether the vendor has the right to resell it. If a dataset was scraped from Indeed against its terms, that risk can travel with the data.

Working with job pages you are allowed to process

For job pages you do have rights to, such as your own careers site or a partner who has granted access, a converter like URLtoText can turn a job posting into clean text or Markdown for parsing or summarizing. Check each site's terms first. A tool doesn't change what a site's terms allow.

Frequently asked questions

Does Indeed allow scraping?

No, not without permission. Indeed's terms prohibit using bots, scrapers, spiders, or AI agents to access or data-mine the site without Indeed's express written permission. The only crawling it conditionally permits is what its robots.txt file allows, and that file blocks job listing pages for general-purpose bots.

What does Indeed's robots.txt disallow?

For bots not named in the file, it disallows individual job pages (/viewjob, /job/), internal API endpoints like /graphql, company pages under /company/, account pages, and search results past the tenth page. AI training bots such as GPTBot, CCBot, and ClaudeBot are also blocked from job search and company pages, and Scrapy is blocked from the entire site.

What do Indeed's terms of service say about automated access?

The Site Rules in Section D.21 forbid accessing the site "through any means other than the public interfaces provided by Indeed," accessing data "by automated means or otherwise, without permission," bypassing security measures, and using automated systems to access, data-mine, or submit content without written permission. They also ban using the site to train third-party AI models.

Does Indeed have a public API for job listings?

Not that we could find. Indeed's developer documentation covers employer and ATS partner APIs, such as the Job Sync API for posting jobs. Its old public job search API is no longer available. For job data, use public ATS feeds, government APIs like USAJOBS, licensed providers, or ask Indeed for a partnership.

How does Indeed get its jobs?

Indeed's terms say job ads on the site "are indexed and posted in an automated manner," and that ads can arrive through "XML feeds, APIs, ATS integrations, requests to index Ads from elsewhere," in addition to employers posting them on the site.