Python Web Scraping: Why Python Is the Best Choice (With a BeautifulSoup Tutorial)

Python is used for web scraping because it has the most complete set of tools for every step of the job: fetching pages, parsing HTML, rendering JavaScript, and cleaning the data afterward. The syntax is short and readable, so a working scraper often fits in 20 lines. And when the data is collected, the same language handles analysis with pandas.

This guide explains why Python is the default choice, how it compares to Node.js, and then walks through a complete BeautifulSoup scraper you can run today. It ends with how to handle JavaScript-rendered pages, including whether Scrapy can do it.

Why Python is used for web scraping

The libraries cover the whole pipeline

Library	What it does	Use it when
`requests`	Downloads pages over HTTP	Almost always, for static pages
`BeautifulSoup`	Parses HTML and finds elements	Small to medium scrapers
`lxml`	Fast HTML/XML parser with XPath	Large pages or speed matters
`Scrapy`	Full crawling framework	Crawling many pages or whole sites
`Playwright`	Controls a real browser	Pages that load content with JavaScript
`pandas`	Cleans, analyzes, and exports data	After scraping, for CSV, Excel, or analysis

BeautifulSoup doesn't download pages itself. It parses markup you give it, and it can use Python's built-in html.parser, or the faster third-party lxml and html5lib parsers, according to its documentation. That's why you'll almost always see it paired with requests.

It's easy to read and quick to write

Scraping code is mostly "fetch this, find that, save it." Python expresses that with very little boilerplate, which makes scrapers easy to fix when a site changes. That matters because scrapers break often.

Data work happens in the same language

After scraping, you usually need to deduplicate, clean, and analyze. Python's data stack (pandas, Jupyter, and machine learning libraries) means you don't switch languages between collecting data and using it.

It has a large scraping community

Python is a common language for scraping tutorials, Stack Overflow answers, and open-source examples. When you hit a problem, someone has usually written about it already.

Web scraping: Python vs Node.js

Node.js is the main alternative, and it's a strong one. Here's how they compare:

	Python	Node.js
HTML parsing	BeautifulSoup, lxml	Cheerio
Crawling framework	Scrapy	Crawlee
Browser automation	Playwright, Selenium	Puppeteer, Playwright, Selenium
Data analysis after scraping	Excellent (pandas)	Limited
Concurrency	asyncio, or built into Scrapy	Async by default
Learning curve	Gentle	Gentle if you know JavaScript

Choose Python if you want the widest choice of libraries, plan to analyze the data, or are new to programming.

Choose Node.js if your team already writes JavaScript, or the project is mostly browser automation. Puppeteer is a JavaScript library for controlling Chrome or Firefox, and Playwright started in the Node ecosystem. That said, Playwright is also available for Python, Java, and .NET, so browser automation alone isn't a reason to switch.

The same reasoning applies to the other comparisons people search for (Ruby, PHP, Java, R). They can all scrape, but none has Python's combination of parsing, crawling, browser, and data libraries in one place.

BeautifulSoup tutorial: scrape quotes into a CSV

This tutorial scrapes quotes.toscrape.com, a practice site built for learning web scraping. It collects every quote, author, and tag across all pages and saves them to a CSV.

Step 1: Install the libraries

pip install requests beautifulsoup4

Step 2: Inspect the page

Open the site, right-click a quote, and choose Inspect. You'll see that each quote looks like this:

<div class="quote">
  <span class="text">“The world as we have created it...”</span>
  <span>by <small class="author">Albert Einstein</small></span>
  <div class="tags">
    <a class="tag" href="/tag/change/page/1/">change</a>
  </div>
</div>

At the bottom of each page there's a Next link inside <li class="next">. Those class names are all you need.

Step 3: Write the scraper

import csv
import time
from urllib.parse import urljoin

import requests
from bs4 import BeautifulSoup

BASE_URL = "https://quotes.toscrape.com/"
HEADERS = {"User-Agent": "my-quotes-scraper/1.0 (you@example.com)"}


def scrape_page(url):
    response = requests.get(url, headers=HEADERS, timeout=30)
    response.raise_for_status()
    soup = BeautifulSoup(response.text, "html.parser")

    quotes = []
    for block in soup.select("div.quote"):
        quotes.append({
            "text": block.select_one("span.text").get_text(strip=True),
            "author": block.select_one("small.author").get_text(strip=True),
            "tags": ", ".join(tag.get_text(strip=True) for tag in block.select("a.tag")),
        })

    next_link = soup.select_one("li.next a")
    next_url = urljoin(url, next_link["href"]) if next_link else None
    return quotes, next_url


all_quotes = []
url = BASE_URL
while url:
    quotes, url = scrape_page(url)
    all_quotes.extend(quotes)
    print(f"Collected {len(all_quotes)} quotes so far")
    time.sleep(1)  # be polite: one request per second

with open("quotes.csv", "w", newline="", encoding="utf-8") as f:
    writer = csv.DictWriter(f, fieldnames=["text", "author", "tags"])
    writer.writeheader()
    writer.writerows(all_quotes)

print(f"Saved {len(all_quotes)} quotes to quotes.csv")

Step 4: Run it

python quotes.py

The script follows the Next link page by page and saves 100 quotes to quotes.csv.

How it works

soup.select("div.quote") uses CSS selectors to find every quote block.
select_one(...).get_text(strip=True) pulls the text out of one element and trims whitespace.
urljoin turns the relative link /page/2/ into a full URL.
time.sleep(1) spaces out requests so you don't hammer the server.

To load the results into pandas for analysis, run pd.read_csv("quotes.csv").

Scraping JavaScript-rendered pages

Try the same scraper on quotes.toscrape.com/js/ and it finds zero quotes. That version of the page builds its content with JavaScript in the browser, and requests only downloads the initial HTML. The quotes never appear in what you parse.

You have three options.

Option 1: Find the data source

Open your browser's developer tools, go to the Network tab, and reload. Sometimes the page loads its data from a JSON endpoint you can call directly, or the data is embedded in a <script> tag. The Scrapy documentation recommends this approach first because it gives you structured data with the least overhead.

Option 2: Use Playwright

Playwright controls a real browser (Chromium, Firefox, or WebKit), so JavaScript runs just like it does for a visitor:

pip install playwright beautifulsoup4
playwright install chromium

from bs4 import BeautifulSoup
from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch()
    page = browser.new_page()
    page.goto("https://quotes.toscrape.com/js/")
    page.wait_for_selector("div.quote")
    html = page.content()
    browser.close()

soup = BeautifulSoup(html, "html.parser")
for block in soup.select("div.quote"):
    print(block.select_one("small.author").get_text(strip=True))

wait_for_selector pauses until the quotes exist, then page.content() returns the rendered HTML, which BeautifulSoup parses as before.

Can Scrapy handle complex JavaScript rendering?

Not by itself. Scrapy downloads HTML over HTTP and doesn't run JavaScript. To render pages, you add a browser through a plugin. The Scrapy docs recommend scrapy-playwright, a download handler that fetches requests with Playwright while keeping Scrapy's scheduling, middlewares, and pipelines.

After pip install scrapy-playwright and playwright install chromium, the setup adds this to settings.py:

DOWNLOAD_HANDLERS = {
    "http": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
    "https": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
}
TWISTED_REACTOR = "twisted.internet.asyncioreactor.AsyncioSelectorReactor"

Then you mark the requests that need a browser with meta={"playwright": True}. Other requests still go through Scrapy's fast HTTP downloader.

Older tutorials use scrapy-splash, which sends pages to Splash, a separate rendering service you run in Docker. It still works, but scrapy-playwright is the option the Scrapy docs point to today. Selenium is another choice, though Playwright is usually simpler to set up from Python.

Option 3: Use a rendering API

If you only need the page's text and don't want to run browsers, a service can render the page for you. The URLtoText API returns any URL as text, Markdown, or HTML with JavaScript rendering turned on:

import requests

response = requests.post(
    "https://urltotext.com/api/v1/urltotext/",
    headers={"Authorization": "Token YOUR_API_TOKEN"},
    json={
        "url": "https://quotes.toscrape.com/js/",
        "output_format": "text",
        "render_javascript": True,
    },
    timeout=120,
)
response.raise_for_status()
print(response.json()["data"]["content"])

This suits text and content extraction. For specific structured fields, Playwright or Scrapy gives you more control.

Scrape politely

Check robots.txt at https://example.com/robots.txt and respect it. Python's built-in urllib.robotparser can read it for you.
Slow down. Add a delay between requests, as in the tutorial. In Scrapy, use DOWNLOAD_DELAY or the AutoThrottle extension.
Identify yourself with a descriptive User-Agent that includes a way to contact you.
Cache pages while developing so you aren't re-downloading the same page every time you tweak your parser.
Read the site's terms and avoid personal data. See is web scraping legal? for the details.

Frequently asked questions

Why is Python used for web scraping?

Python has mature libraries for every step: requests for fetching, BeautifulSoup and lxml for parsing, Scrapy for crawling, Playwright for JavaScript pages, and pandas for analysis. Its readable syntax makes scrapers quick to write and easy to fix.

Can Scrapy handle complex JavaScript rendering?

Not on its own, because Scrapy doesn't execute JavaScript. Add the scrapy-playwright plugin to render pages in a real browser while keeping Scrapy's crawling features. You can also look for the JSON endpoint the page loads its data from and call it directly.

Python Web Scraping: Why Python Is the Best Choice (With a BeautifulSoup Tutorial)

Why Python is used for web scraping

The libraries cover the whole pipeline

It's easy to read and quick to write

Data work happens in the same language

It has a large scraping community

Web scraping: Python vs Node.js

BeautifulSoup tutorial: scrape quotes into a CSV

Step 1: Install the libraries

Step 2: Inspect the page

Step 3: Write the scraper

Step 4: Run it

How it works

Scraping JavaScript-rendered pages

Option 1: Find the data source

Option 2: Use Playwright

Can Scrapy handle complex JavaScript rendering?

Option 3: Use a rendering API

Scrape politely

Frequently asked questions

Why is Python used for web scraping?

Can Scrapy handle complex JavaScript rendering?

Is Python or Node.js better for web scraping?

Is BeautifulSoup or Scrapy better?

What is the best Python library for web scraping?