DEV Community: andy

Why Building Your Own Data Pipeline Usually Ends in Broken Selectors and 4 AM Alerts

andy — Wed, 05 Aug 2026 10:51:53 +0000

A few months ago, a friend who runs a sales team came to me with a problem. He wanted to time his outreach better. If a competitor just got hit with a big regulatory fine, or announced layoffs, or acquired a smaller company, that was his signal to reach out to their customers. But finding that signal manually meant checking dozens of websites every morning.

Since I'm the "tech guy" in my friend group, he asked if I could just scrape the data for him. Layoff announcements, regulatory fines, acquisition news — how hard could it be?

It turns out, it's a nightmare. And it's the exact reason we ended up building structured APIs for this stuff at ParheliaWeb.

The Scraper Reality Check

Here is what my first prototype looked like. It's the kind of script a lot of developers write when they just need data quickly:

import requests
from lxml import html

def scrape_layoffs(url):
    response = requests.get(url, headers={"User-Agent": "Mozilla/5.0"})
    tree = html.fromstring(response.content)

    # These selectors work... until the site redesigns
    companies = tree.xpath("//h3[@class='company-name']/text()")
    counts = tree.xpath("//span[@class='layoff-count']/text()")
    dates = tree.xpath("//time[@class='announcement-date']/@datetime")

    return [{"company": c, "count": n, "date": d} for c, n, d in zip(companies, counts, dates)]

I handed it over, and it worked great for about a week. Then the real world hit:

Cloudflare started blocking my IP. Sites that host high-value B2B data have aggressive anti-bot protections. Without rotating residential proxies and proper fingerprinting, you are done after 50 requests.
The DOM shuffle. Government regulatory sites and local business registries update their HTML structure without warning. One Tuesday morning, your CSS selectors break, your pipeline returns null, and your users get stale data. Maintaining these selectors is a full-time job.
The "noise" problem. News articles and press releases are unstructured. Extracting the exact number of layoffs, the specific regulatory body issuing a fine, or the financial terms of an acquisition is messy. You end up with "approximately 150 employees" and "a multi-million dollar settlement" — text that is useless for automated workflows.
Deduplication hell. The same layoff gets posted on the company's blog, TechCrunch, LinkedIn, and three industry newsletters. My script counted it four times.

I ended up spending more time fighting Cloudflare, normalizing text, and handling rate limits than actually helping my friend. I was building an infrastructure project, not a data tool.

The API Approach (What I Use Now)

After realizing that scraping was a massive time sink, I stopped fighting the websites and started structuring the data properly. We built ParheliaWeb's layoffs, fines, and acquisitions APIs to be the pipeline I wished I had when my friend first asked for help.

Here is how you get the same data in 5 lines of clean Python:

import requests

API_KEY = "your_api_key_here"  # Get one at parheliaweb.com

response = requests.get(
    "https://parheliaweb.com/v1/layoffs",
    headers={"x-api-key": API_KEY},
    params={
        "max_age_days": 30,
        "max_records": 10
    }
)

data = response.json()

for event in data["results"]:
    print(f"{event['company_name']} laid off {event['layoff_count']} people "
          f"on {event['announcement_date']}")

And instead of messy text strings, you get back clean, structured JSON that you can actually plug into an app or a dashboard:

{
  "user_tier": "pro",
  "count": 10,
  "max_age_days": 30,
  "last_crawled": "2026-07-31T10:30:00Z",
  "results": [
    {
      "company_name": "TechCorp Inc.",
      "layoff_count": 150,
      "percentage": "15%",
      "announcement_date": "2026-07-15",
      "source_url": "https://example.com/...",
      "source_status": "active",
      "sector": "SaaS",
      "country": "US"
    }
  ]
}

No parsing. No deduplication. No 4 AM alerts when a government website changes its layout. It just works.

The Real Cost Comparison

If you are trying to decide whether to build your own data pipeline or use an API, here is what I learned from the trenches:

Factor	Build Yourself	Use an API
Time to first result	4-8 weeks	5 minutes
Ongoing maintenance	10+ hours/week	0
Data coverage	Limited to the 2-3 sites you can scrape	Funding, Acquisitions, IPOs, Layoffs, Fines
Data structure	Messy, requires constant normalization	Clean, standardized JSON
Compliance (GDPR, etc.)	Your risk	We handle source attribution and data minimization
Cost	Your time + proxy infrastructure	€29/month per API

My rule of thumb: If your core business is data collection, build it. If your core business is using the data, use an API. Let someone else worry about the CSS selectors.

Getting Started

Try it free: 100 free API calls per day, no credit card required. See if it works for your use case.

Pro tier: Full dataset access for €29/month per API.

Get your free API key →

I'm Andy, the founder. I'm just an IT guy who likes solving problems for people. If you have questions about the data or the API, just drop me a line at info@parheliaweb.com.

I Built an Email Validation API. Here’s Why Microsoft Lies to You.

andy — Thu, 16 Jul 2026 10:39:24 +0000

Most email validation tools are black boxes. You send them an address, they spit out "valid" or "invalid", and you pray.

But when a "clean" list suddenly hits a 21% bounce rate overnight and your sender reputation tanks, you realize that a simple boolean isn't enough. You need to know why.

If you're building a SaaS onboarding flow, a newsletter platform, or a CRM, protecting your domain reputation is non-negotiable.

This post shows you why basic validation fails, the hidden traps of SMTP probing, and how to actually get the context you need to keep your emails out of the spam folder.

1. The Hard Way: Falling Down the Rabbit Hole

Most developers start with a simple Regex check. It catches typos, but it completely misses disposable domains.

So you add an MX record lookup. That works, until you realize the domain has a "catch-all" setup, meaning it accepts anything.

So you build an SMTP probe to have a real conversation with the mail server without actually sending an email. It works perfectly... until it doesn't:

Microsoft lies to you: Outlook/Hotmail servers will accept almost any address during the SMTP handshake to prevent directory harvesting, returning a fake "250 OK".
Gmail throttles you: If you probe too many Gmail addresses in a short window, they silently soft-bounce or temporarily block your IP.
Yahoo blocks you entirely: Aggressive probing triggers immediate firewall drops.
Your own IP gets blacklisted: If you don't rate-limit your probes, Spamhaus will flag your verification server as a dictionary attack source.

Three months in, you aren't building your core product. You are maintaining a fragile, rate-limited, IP-reputation-sensitive scraping operation.

2. The Smart Way: Context Over Certainty

After watching good domains get blacklisted by bad data, I completely redesigned our validation infrastructure at ParheliaWeb.

We moved to a dedicated Python infrastructure with clean, segregated IPs and strict rate limiting. But the biggest change wasn't the infrastructure; it was the output.

Instead of a black box "valid/invalid", we return context.

Here is how to validate an email with full transparency in Python:

import requests

API_KEY = "your_api_key_here"  # Get one at parheliaweb.com

response = requests.post(
    "https://parheliaweb.com/v1/email/validate",
    headers={
        "Content-Type": "application/json",
        "x-api-key": API_KEY
    },
    json={"email": "user@outlook.com"}
)

print(response.json())

What you get back (Example of a "Risky" Outlook address):

{
  "status": "ok",
  "result": {
    "email": "user@outlook.com",
    "status": "risky",
    "confidence": 45,
    "syntax_valid": true,
    "mx_valid": true,
    "smtp_check": {
      "performed": true,
      "result": true,
      "code": 250,
      "message": "Mailbox accepted (code 250)"
    },
    "catch_all": {
      "detected": true,
      "message": "Domain accepts all test addresses — likely catch-all"
    },
    "risk_factors": [
      {"factor": "catch_all_domain", "severity": "medium"}
    ]
  }
}

No guessing. No black boxes. Just the exact technical reason why an email is risky, so you can make an informed decision.

3. Build vs. Buy: The Real Math

Factor	Build Yourself	Use ParheliaWeb API
Time to first result	2-4 weeks	5 minutes
Ongoing maintenance	5-10 hours/week (managing IP reputation)	0
Disposable domain list	Manual, constantly outdated	260,000+ updated daily
SMTP Probe Risk	High (your IP gets blacklisted)	Zero (we use segregated clean IPs)
Output	"Valid" / "Invalid"	Detailed risk scoring & reasoning
Cost	Engineering time + proxy costs	Starts at €39/mo (€31.20 launch offer)

The rule: If your core business is email deliverability, build it. If your core business is using the data, buy it.

4. Getting Started

Start with 100 free API checks per month — no credit card required. Test the transparency, validate the data quality, and upgrade when you're ready.

👉 Get your free API key and view live pricing

5. Questions?

I'm Andy, the founder. I read every email.

📧 info@parheliaweb.com

How to Track Startup Funding Rounds Programmatically (Without Losing Your Mind)

andy — Tue, 07 Jul 2026 14:46:16 +0000

Every week, over €2 billion flows into European startups. But finding who got funded -- and how much -- means scrolling press releases, parsing LinkedIn posts, and paying for paywalled databases.

If you're building an investor dashboard, a CRM for venture capital, or a market intelligence tool, you need this data structured and accessible. Not in PDFs. Not in HTML. In JSON, via an API, ready to plug into your application.

This post shows you two ways to get there: the hard way (building your own scraper) and the smart way (using a structured API). By the end, you'll have working Python code and a clear decision on which path fits your project.

1. The Hard Way: Building Your Own Scraper

Most developers start here. I did too.

You find a site that lists funding rounds -- TechCrunch, local news, industry blogs. You open Chrome DevTools, find the CSS selectors, and write a scraper:

import requests
from lxml import html

def scrape_funding(url):
    response = requests.get(url, headers={"User-Agent": "Mozilla/5.0"})
    tree = html.fromstring(response.content)

    # These selectors work... until the site redesigns
    companies = tree.xpath("//h2[@class='company-name']/text()")
    amounts = tree.xpath("//span[@class='funding-amount']/text()")

    return [{"company": c, "amount": a} for c, a in zip(companies, amounts)]

It works for a week. Then:

The site adds a paywall
Cloudflare blocks your IP
A redesign changes every selector
You discover the amounts are inconsistent ("€5M", "5 million euros", "Series A of undisclosed size")
You need to deduplicate the same round from three sources

Three months in, you've spent more time maintaining the scraper than building your actual product.

This is the hidden cost of DIY data collection: it's not the first script that kills you. It's the 4 AM alerts when a site changes its HTML.

2. The Smart Way: Using a Funding API

After maintaining scrapers for too long, I built ParheliaWeb -- a structured API that handles the messy work for you.

Here's how to get funding data in 5 lines of Python:

import requests

API_KEY = "your_api_key_here"  # Get one at parheliaweb.com

response = requests.get(
    "https://parheliaweb.com/v1/funding",
    headers={"x-api-key": API_KEY},
    params={
        "max_age_days": 30,
        "max_records": 10
    }
)

data = response.json()

for round in data["results"]:
    print(f"{round['company_name']} raised {round['funding_amount']} "
          f"in {round['round_type']} ({round['announcement_date']})")

What you get back:

{
  "user_tier": "pro",
  "count": 10,
  "max_age_days": 30,
  "last_crawled": "2026-07-03T10:30:00Z",
  "results": [
    {
      "company_name": "Acme AI",
      "funding_amount": "$5 million",
      "round_type": "Series A",
      "announcement_date": "2026-06-15",
      "source_url": "https://techcrunch.com/...",
      "source_status": "active",
      "sector": "Artificial Intelligence",
      "currency": "USD",
      "country": "US",
      "lead_investor": "Sequoia Capital",
      "is_extension": false,
      "investor_count": "5"
    }
  ]
}

No parsing. No deduplication. No 4 AM alerts.

3. Build vs. Buy: The Real Math

Factor	Build Yourself	Use ParheliaWeb API
Time to first result	2-4 weeks	5 minutes
Ongoing maintenance	5-10 hours/week	0
Data quality	Variable	AI-verified & structured
Coverage gaps	Your problem	Handled
Compliance (GDPR, etc.)	Your risk	Our responsibility
Cost	Engineering salary	€29/month

The rule: If your core business is data collection, build it. If your core business is using the data, buy it.

4. Getting Started

Start with 100 free API calls -- no credit card required. Test your integration, validate the data quality, and upgrade when you're ready.

Pro tier: Full dataset access for €29/month. Launch promo: 20% off (€23/month).

👉 Get your free API key

5. Questions?

I'm Andy, the founder. I read every email.

📧 info@parheliaweb.com