DEV Community

Cover image for 5 Python Scripts That Taught Me Real-World Parsing and Automation
Fahad Shah
Fahad Shah

Posted on

5 Python Scripts That Taught Me Real-World Parsing and Automation

#python #ai #automation #programming

(From Course 2 & 3 of Python for Everybody – Applied Like a Pro)

Most beginners stop at print statements.
I used every course module to build scripts that scrape, parse, and automate real data pipelines.

Here are 5 scripts that went beyond the basics — each one feels like a tool, not a toy.

1️⃣ 📬 Spam Confidence Extractor

Parses through emails and calculates average spam confidence from X-DSPAM-Confidence: headers.

✅ Skills:

find(), float(), string parsing

File reading, data cleaning

count = 0
total = 0

with open("mbox.txt") as f:
    for line in f:
        if line.startswith("X-DSPAM-Confidence:"):
            num = float(line.split(":")[1].strip())
            count += 1
            total += num

print("Average spam confidence:", total / count)
Enter fullscreen mode Exit fullscreen mode

📎 Real-World Use: Email filtering, NLP pre-cleaning, header analysis.

2️⃣ 📧 Email Address Counter

Counts how many times each sender appears and prints the most frequent one.

✅ Skills:

dict counting, string parsing, file handling

emails = {}

with open("mbox.txt") as f:
    for line in f:
        if line.startswith("From "):
            parts = line.split()
            email = parts[1]
            emails[email] = emails.get(email, 0) + 1

max_email = max(emails, key=emails.get)
print(max_email, emails[max_email])
Enter fullscreen mode Exit fullscreen mode

📎 Real-World Use: Inbox analytics, sender clustering, contact insights.

3️⃣ ⏰ Hour Histogram

Parses timestamps from From lines and plots an hour-wise distribution.

✅ Skills:

split(), dict, sorting keys

hours = {}

with open("mbox.txt") as f:
    for line in f:
        if line.startswith("From "):
            time = line.split()[5]
            hour = time.split(":")[0]
            hours[hour] = hours.get(hour, 0) + 1

for hour in sorted(hours):
    print(hour, hours[hour])
Enter fullscreen mode Exit fullscreen mode

📎 Real-World Use: Time-based behavior analysis, email scheduling data, logs monitoring.

4️⃣ 🌐 BeautifulSoup Scraper

Pulls all anchor tag texts from a live webpage using BeautifulSoup.

✅ Skills:

HTTP requests, HTML parsing, bs4 tag navigation

import urllib.request
from bs4 import BeautifulSoup

url = input("Enter URL: ")
html = urllib.request.urlopen(url).read()
soup = BeautifulSoup(html, "html.parser")

for tag in soup("a"):
    print(tag.get("href", None))
Enter fullscreen mode Exit fullscreen mode

📎 Real-World Use: Link scraping, data crawling, sitemap audits.

5️⃣ 🔗 JSON API Extractor

Fetches data from a REST API, parses JSON, and processes nested fields.

✅ Skills:

urllib, json, nested dictionary access

import urllib.request, urllib.parse, json

url = "http://py4e-data.dr-chuck.net/comments_42.json"
data = urllib.request.urlopen(url).read().decode()
info = json.loads(data)

total = sum([int(item["count"]) for item in info["comments"]])
print("Sum:", total)
Enter fullscreen mode Exit fullscreen mode

📎 Real-World Use: API response processing, backend pipelines, data analytics inputs.

🧩 Why This Matters

These aren’t random exercises.
Each script taught me core data processing patterns that show up in real-world systems:

  • Parsing messy input → extracting value
  • Aggregating + filtering data
  • Understanding structure behind unstructured sources

Not toy problems — these are backend blueprints.

🔗 Follow My Build Journey

#1FahadShah #Python #DataParsing #BackendEngineering #BuildInPublic #WebScraping #JSON #APIs #LearningInPublic

Top comments (1)

Collapse
 
onlineproxy profile image
OnlineProxy •

Working with real-world data pipelines in Python? Шt's all about dealing with messy data - inconsistent formats, missing fields, weird encodings. You name it. In my world, messy data is just the norm. I leaned hard on try-except blocks, structured logging, and tools like pydantic to keep things in check and validate inputs. For automating stuff, libraries like pandas, requests, BeautifulSoup, and schedule were absolute lifesavers. I’ve even taken scripts like the Spam Confidence Extractor and added a bit more muscle, like parsing DKIM headers or checking subject lines for spam clues. And let’s not forget about scraping job boards, generating reports, and keeping tabs on stock news - everything built to scale, maintain, and flex as needed.