How to Extract Just the Main Text from Any Webpage

by

Extracting the main text content from a webpage can be incredibly useful for various applications, such as data analysis, content curation, or creating clean reading experiences.

While there are several methods to accomplish this task, one of the simplest ways is to use a free online service like urltotext.com. 👇

In this article, we’ll explore this option along with other methods, ranging from simple browser tricks to more advanced programming techniques.

Table of Contents

  1. Using Browser Developer Tools
  2. The Easy Way: Using Free Online Services Like URLtoText.com
  3. Browser Extensions
  4. Python Libraries
  5. Using Natural Language Processing (NLP)

1. Using Browser Developer Tools

For a quick and easy method that doesn’t require any coding:

  1. Open the webpage in your browser
  2. Right-click and select “Inspect” or press F12
  3. In the developer tools, find the “Console” tab
  4. Paste and run this JavaScript code:
copy(document.body.innerText)
  1. The main text content is now copied to your clipboard

This method works well for simple pages but may include some unwanted text like navigation menus.

2. Using Free Online Services

Several websites offer text extraction services:

These services often provide a user-friendly interface and sometimes offer APIs for programmatic access.

3. Browser Extensions

Several browser extensions can extract main content:

  • Mercury Reader (Chrome)
  • Just Read (Chrome, Firefox)

These extensions often provide a clean reading view and the ability to copy the main content.

4. Python Libraries

For developers, Python offers powerful libraries:

BeautifulSoup

import requests
from bs4 import BeautifulSoup

url = 'https://example.com'
response = requests.get(url)
soup = BeautifulSoup(response.text, 'html.parser')

# Remove script and style elements
for script in soup(["script", "style"]):
    script.decompose()

# Get text
text = soup.get_text()

# Break into lines and remove leading and trailing space on each
lines = (line.strip() for line in text.splitlines())
# Break multi-headlines into a line each
chunks = (phrase.strip() for line in lines for phrase in line.split("  "))
# Drop blank lines
text = '\n'.join(chunk for chunk in chunks if chunk)

print(text)

Newspaper3k

from newspaper import Article

url = 'https://example.com'
article = Article(url)
article.download()
article.parse()

print(article.text)

5. Using Natural Language Processing (NLP)

For more advanced extraction, consider using NLP techniques:

  1. Tokenize the webpage content
  2. Remove stop words and punctuation
  3. Calculate term frequency-inverse document frequency (TF-IDF)
  4. Identify sentences with the highest TF-IDF scores as main content

Libraries like NLTK or spaCy can help with this approach.

Conclusion

Extracting main text from webpages can be achieved through various methods, from simple online tools like urltotext.com to sophisticated NLP techniques. Choose the approach that best fits your needs and technical expertise. Remember to respect website terms of service and copyright when extracting content.