If you need to extract basic structured data from a static, publicly accessible website, it can be tempting to immediately reach for frameworks like Selenium, Scrapy, or Playwright.
However, for simple static pages, Python's standard library can often handle the job without installing any external dependencies.
In this tutorial, we'll build a lightweight data extraction script using only urllib, re, html, csv, and logging.
Technical Scope
Before we start, it's important to understand what this approach can and cannot do.
What it does:
- Fetches static HTML
- Extracts specific data patterns
- Cleans HTML entities
- Exports structured data to CSV
- Includes basic logging and error handling
What it does not do:
- Render JavaScript
- Handle infinite scrolling
- Bypass CAPTCHAs
- Bypass anti-bot protections
- Access login-protected/private pages
- Crawl multiple pages automatically
Only automate access where you have permission to do so. A webpage being publicly visible does not necessarily mean every form of automated extraction is permitted.
The Code
For this example, we'll extract quotes and author names from the public scraping sandbox quotes.toscrape.com.
We'll separate fetching, parsing, and exporting into a small class so the code stays easy to understand.
import csv
import logging
import urllib.request
import urllib.error
import re
import html
from datetime import datetime
logging.basicConfig(
level=logging.INFO,
format="%(asctime)s - %(levelname)s - %(message)s"
)
class PublicDataScraper:
def __init__(self, target_url: str):
self.target_url = target_url
self.extracted_data = []
def fetch_page_html(self) -> str:
"""Fetch HTML content from the target URL."""
try:
request = urllib.request.Request(
self.target_url,
headers={"User-Agent": "Mozilla/5.0"}
)
with urllib.request.urlopen(request, timeout=10) as response:
return response.read().decode("utf-8")
except urllib.error.URLError as error:
logging.error(f"Failed to fetch URL: {error.reason}")
raise
def parse_html(self, page_html: str) -> list:
"""Extract quotes and authors from the static HTML."""
results = []
pattern = (
r'<span class="text".*?>(.*?)</span>'
r'.*?<small class="author".*?>(.*?)</small>'
)
matches = re.findall(pattern, page_html, re.DOTALL)
for quote, author in matches:
clean_quote = html.unescape(quote).strip()
results.append({
"quote_text": clean_quote,
"author_name": author.strip()
})
return results
def extract(self):
"""Run the extraction process."""
try:
logging.info("Fetching HTML...")
page_html = self.fetch_page_html()
logging.info("Parsing data...")
self.extracted_data = self.parse_html(page_html)
logging.info(
f"Successfully extracted "
f"{len(self.extracted_data)} records."
)
except Exception as error:
logging.error(f"Extraction failed: {error}")
def export_to_csv(self, filename: str):
"""Export extracted records to CSV."""
if not self.extracted_data:
logging.warning("No extracted data to export.")
return
fieldnames = self.extracted_data[0].keys()
try:
with open(
filename,
"w",
newline="",
encoding="utf-8"
) as output_file:
writer = csv.DictWriter(
output_file,
fieldnames=fieldnames
)
writer.writeheader()
writer.writerows(self.extracted_data)
logging.info(
f"Data successfully exported to {filename}"
)
except IOError as error:
logging.error(
f"Failed to write CSV: {error}"
)
if __name__ == "__main__":
scraper = PublicDataScraper(
target_url="https://quotes.toscrape.com/"
)
scraper.extract()
timestamp = datetime.now().strftime("%Y%m%d_%H%M")
scraper.export_to_csv(
f"extracted_data_{timestamp}.csv"
)
What the Script Does
When the script runs, it follows a simple pipeline:
1. Fetch the HTML
urllib.request sends an HTTP request and downloads the raw HTML returned by the server.
2. Extract the data
A regular expression searches the HTML for the quote and author elements used by the demo website.
3. Clean the text
html.unescape() converts HTML entities into normal readable characters.
4. Store structured records
Each extracted quote is stored together with its author.
5. Export everything to CSV
Python's built-in csv module writes the final data into a structured file that can be opened in spreadsheet software or processed by another program.
Why This Approach Can Be Useful
1. No external dependencies
Everything used here ships with Python.
There is no need to run:
pip install ...
That makes small scripts easier to move between environments.
2. Basic error visibility
The built-in logging module gives us useful information when a request fails or when the script completes successfully.
3. Clear separation of responsibilities
Fetching, parsing, and exporting are handled separately.
If the HTML structure changes, the parsing logic can be updated without rewriting the CSV export workflow.
Important Limitation: Parsing HTML With Regex
Regular expressions are intentionally used here because this is a small, predictable demonstration.
They are not a universal HTML parser.
Real websites may contain:
- deeply nested markup
- frequently changing HTML
- dynamically generated content
- malformed HTML
- multiple layouts
For larger or more complex projects, a dedicated HTML parser such as BeautifulSoup is generally easier to maintain.
If a page is rendered primarily through JavaScript, browser-automation tools such as Playwright or Selenium may be more appropriate.
Those capabilities are outside the scope of this zero-dependency example.
Final Result
With only Python's standard library, we now have a small extraction pipeline that can:
- request a static HTML page
- extract targeted information
- clean the data
- log execution status
- export structured records to CSV
For simple and authorized static-data tasks, this can be a useful starting point before introducing larger scraping frameworks.
If you need help with a small, authorized static-data extraction task, you can view my Upwork service
https://www.upwork.com/freelancers/~013b992149b9e111c4?p=2102919213606854656
Top comments (0)