DEV Community: saripalli shanmukha kiran sagar

Free Python script: check if a website looks outdated (stdlib only, robots.txt-respecting)

saripalli shanmukha kiran sagar — Sun, 18 Oct 2026 13:21:25 +0000

If you sell web design or maintenance to small businesses, you probably have a list of prospects — but not all of them need you yet. A homepage that scores high on a few simple, observable "outdated" signals is a much better warm-up than one that just looks… fine.

This is a drop-in Python script that checks a homepage for five of those signals, prints a rough 0–100 "outdated" score and the reasons, and does it all with the standard library. No third-party libraries, no config files. It also respects robots.txt (via urllib.robotparser) and collects no personal data.

How it scores

For each URL you pass on the command line, it fetches only the homepage (follows the final redirect, reads up to 2 MB) and adds points for:

No HTTPS (the final URL is not https://) — +25
No mobile viewport <meta> tag — +25
A footer copyright year that's at least 2 years older than this year — +20 (e.g. (c) 2019 in 2026)
jQuery < 3 detected in the page — +15
Legacy markup: inline <table> layout with width/bgcolor, a <font> tag, a <marquee>, or a .swf Flash reference — +15

Capped at 100. It prints one line per site with the score, or a dash (-/100) if robots.txt disallows fetching the URL — so you can stream a list through it and spot the highest-scoring prospects.

It does NOT crawl internal links, does NOT collect personal data, and does NOT modify anything on the target site.

Usage

python3 outdated_site_check.py https://example.com https://another.com

Example output:

https://example.com 0/100   looks reasonably modern
https://oldshop.com 65/100  no HTTPS; no mobile viewport meta tag; footer copyright says 2019

The script

Save this as-is (it is idempotent and needs no config).

#!/usr/bin/env python3
"""
Quick outdated-website checker (homepage only, standard library only).

Usage:  python3 outdated_site_check.py https://example.com https://another.com

Prints a rough 0-100 "outdated" score and the reasons, so web designers can
qualify redesign prospects from a list they already have.

It fetches one page per site, respects robots.txt, and collects no personal data.
"""
import datetime
import re
import sys
import urllib.parse
import urllib.request
import urllib.robotparser

UA = "outdated-site-check/1.0"


def allowed(url):
    p = urllib.parse.urlparse(url)
    rp = urllib.robotparser.RobotFileParser(f"{p.scheme}://{p.netloc}/robots.txt")
    try:
        rp.read()
        return rp.can_fetch(UA, url)
    except Exception:
        return True


def check(url):
    if not url.startswith("http"):
        url = "https://" + url
    if not allowed(url):
        return None, ["robots.txt disallows fetching; skipped"]
    req = urllib.request.Request(url, headers={"User-Agent": UA})
    with urllib.request.urlopen(req, timeout=15) as r:
        final = r.geturl()
        html = r.read(2_000_000).decode("utf-8", "ignore")

    score, why = 0, []
    year = datetime.date.today().year

    if not final.startswith("https://"):
        score += 25; why.append("no HTTPS")
    if not re.search(r'<meta[^>]+name=["\']viewport', html, re.I):
        score += 25; why.append("no mobile viewport meta tag")
    yrs = [int(y) for y in re.findall(r"(?:©|&copy;|copyright)\D{0,20}((?:19|20)\d{2})", html, re.I)]
    if yrs and max(yrs) <= year - 2:
        score += 20; why.append(f"footer copyright says {max(yrs)}")
    jq = re.search(r"jquery[.-]?(\d)\.(\d+)", html, re.I)
    if jq and int(jq.group(1)) < 3:
        score += 15; why.append(f"old jQuery {jq.group(1)}.{jq.group(2)}")
    if re.search(r"<table[^>]+(width|bgcolor)=|<font\b|<marquee\b|\.swf\b", html, re.I):
        score += 15; why.append("table/font-tag layout or Flash")
    return min(score, 100), why or ["looks reasonably modern"]


if __name__ == "__main__":
    for u in sys.argv[1:] or ["https://example.com"]:
        try:
            s, why = check(u)
            print(f"{u}\t{'-' if s is None else s}/100\t" + "; ".join(why))
        except Exception as e:
            print(f"{u}\terror\t{e}")

Feedback welcome — if a signal you care about is missing, tell me and I'll add it.

Note: I also built an Apify actor that collects the same kinds of signals at scale (header checks, tech signals, content freshness, link health) and outputs a prioritized list of pages worth revisiting. The actor is open-source and runs on Apify free: https://apify.com/shanmukhkiransagar/website-redesign-signals

How I qualify web-design prospects with an outdated-website checker (no scraping, robots.txt-respecting)

saripalli shanmukha kiran sagar — Sun, 18 Oct 2026 09:35:57 +0000

Disclosure: I built the tool used in this post, Website Redesign Signals, and it's a paid actor on Apify. The checks described here are simple enough that you can copy the ideas into your own script if you'd rather not use it.

The problem: 200 tabs and a generic first line

If you sell web design to small businesses, you probably already have a list of prospects: a CRM export, a spreadsheet from a networking event, past enquiries. The slow part isn't finding names. It's working out which of those sites actually need work and what to say in the first line of your email.

Doing that by hand means opening every homepage, resizing the window to check mobile, scrolling to the footer, viewing the source for jQuery versions... and then still writing "I love your website, but..." because you ran out of patience.

So I wrote a small checker that does the boring part: fetch each homepage, look for a handful of observable signals, give it a 0-100 "outdated" score, and write one neutral, factual sentence I can verify in 20 seconds before using it.

Which signals are worth checking

I deliberately stuck to things that are visible in the public HTML or HTTP responses and that a business owner would recognise if you pointed them out:

Signal	Why it matters	How it's detected
No mobile viewport tag	Phones render a shrunken desktop page. Strongest single signal of an old build.	No `<meta name="viewport" content="width=device-width...">`
Stale © year	"© 2018" in the footer suggests nobody has touched the site in years	Regex on footer text; years written by a script (`new Date().getFullYear()`) are recognised and not penalised
Old jQuery / Bootstrap / CMS	jQuery 1.x, Bootstrap 3, WordPress < 6.0 point to an old theme	Versions from public script/style URLs, library banners and the generator tag (only if the site exposes it)
HTTPS problems	Plain http only, certificate errors, or http that doesn't forward to https	Try `https://` first, fall back to `http://`, one extra request to check the redirect
Broken internal links	A 404 behind "Services" is a concrete, fixable thing to mention	HEAD (with GET fallback) on up to 25 homepage links
Heavy images	A 3 MB hero JPEG is slow on mobile data	HEAD `Content-Length` on homepage images, top 3 by size
No enquiry form / not tap-to-call	Conversion basics	Looks for forms on the homepage and the linked contact page; checks for any `tel:` link (true/false only)

Things I left out on purpose: Lighthouse/Core Web Vitals (slow and expensive per site), anything that needs a login, and anything that probes paths or ports. It's not a security scanner, it only reads what a normal visitor's browser receives.

The polite part (no scraping)

This was a hard requirement for me:

You bring the list. It doesn't discover businesses or scrape Google Maps, directories or social networks.
robots.txt is checked before every request, under its own user-agent token (RedesignSignalsBot), and the User-Agent header says what it is.
At most the homepage plus 2 more pages per site (contact page, optionally one services page). Link and image checks are lightweight HEAD requests with low per-site concurrency.
No personal data in the output. No emails, phone numbers or names. Contact signals are booleans. Link text that looks like an email or phone number is dropped.

How the 0-100 score works

It's an additive, explainable heuristic. Each finding adds points, and the total is capped at 100:

no mobile viewport            +25
HTTPS certificate error       +22
plain http only               +20
© year 5+ years old           +15   (3-4 years: +10, 2 years: +4)
Flash                         +15
broken homepage links         +4 each, max +14
jQuery 1.x                    +10   (2.x: +7, < 3.5: +3)
Bootstrap 3 or older          +8
old WordPress/Joomla/Drupal   +8
http doesn't redirect to https +8
image over 1 MB               +7   (over 500 KB: +3)
no enquiry form               +6
missing <title>               +5
missing meta description      +4
no tap-to-call link           +3

Every point comes with a reasons[] entry (code, points, plain-English detail), so you never have to trust a mystery number. In my own testing I use it roughly like this:

40+: strong redesign prospect
15-39: fix-level issues, worth a short "quick fix" email
under 15: modern enough, skip

The outreach hook is built from the two highest-scoring reasons that have a plain-language phrase, and it avoids scare words like "hacked" or "insecure".

A real example (domain anonymised)

This is an abridged record from one of my test runs against a UK roofing company's homepage. I've replaced the domain:

{
  "domain": "www.roofer-example.co.uk",
  "finalUrl": "https://www.roofer-example.co.uk/",
  "status": "ok",
  "httpsOk": true,
  "httpRedirectsToHttps": true,
  "hasViewportMeta": false,
  "copyrightYear": null,
  "cms": "Weebly",
  "jqueryVersion": "1.8.3",
  "metaDescriptionMissing": false,
  "largestImages": [{ "url": "https://www.roofer-example.co.uk/uploads/logo.png", "kb": 245 }],
  "brokenInternalLinks": [],
  "hasContactOrBookingForm": true,
  "hasTapToCall": false,
  "outdatedScore": 38,
  "reasons": [
    { "code": "no_viewport", "points": 25, "detail": "No <meta name=\"viewport\" content=\"width=device-width\"> tag" },
    { "code": "old_jquery", "points": 10, "detail": "jQuery 1.8.3 (1.x line, no longer supported)" },
    { "code": "no_tap_to_call", "points": 3, "detail": "No tap-to-call (tel:) link on the homepage" }
  ],
  "outreachHook": "The site has no mobile viewport tag and it still loads jQuery 1.8.3.",
  "primaryIssueType": "NOVIEWPORT"
}

Note what it didn't penalise: HTTPS works and redirects properly, the contact page has a form, there's no footer year to judge. 25 + 10 + 3 = 38, a fix-level-to-redesign borderline case, and the hook is something I can confirm in a few seconds by opening the site on my phone.

Running it from the Apify API

You'll need an Apify account and an API token. Keep the token in an environment variable, never in your code.

Python

pip install apify-client
export APIFY_TOKEN="your-token-here"   # placeholder, set your own

import os
from apify_client import ApifyClient

client = ApifyClient(os.environ["APIFY_TOKEN"])

run = client.actor("shanmukhkiransagar/website-redesign-signals").call(
    run_input={
        "domains": ["example.co.uk", "https://www.another-example.com/"],
        "maxPagesPerSite": 2,      # homepage + contact page
        "checkBrokenLinks": True,
        "checkImages": True,
    }
)

rows = list(client.dataset(run["defaultDatasetId"]).iterate_items())
rows.sort(key=lambda r: r.get("outdatedScore") or 0, reverse=True)

for r in rows:
    print(f'{r.get("outdatedScore", 0):>3}  {r["domain"]:<35} {r.get("outreachHook") or "-"}')

JavaScript (Node)

npm install apify-client

import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: process.env.APIFY_TOKEN });

const run = await client.actor('shanmukhkiransagar/website-redesign-signals').call({
  domains: ['example.co.uk', 'https://www.another-example.com/'],
  maxPagesPerSite: 2,
});

const { items } = await client.dataset(run.defaultDatasetId).listItems();
items
  .filter((r) => r.status === 'ok' && r.outdatedScore >= 40)
  .forEach((r) => console.log(r.outdatedScore, r.domain, r.outreachHook));

From there you can export the rows to your CRM and segment by primaryIssueType (NOVIEWPORT, HTTP, BROKEN, DATED, HEAVY, NOFORM...), so each batch of emails talks about one specific thing.

Limitations (honestly)

It doesn't run JavaScript. It reads HTML over plain HTTP, so content rendered only by scripts isn't seen. A form injected by a script can be missed, which means "no enquiry form" can be a false positive on JS-heavy sites.
No page-weight totals, load times or Core Web Vitals. Image sizes come from HEAD requests on homepage images only.
CMS versions only when the site exposes them. Many don't.
Bot-protected sites (e.g. Cloudflare challenge pages) can come back as http_error 403/503.
The weights are mine and haven't been calibrated against campaign reply rates. In a pre-launch test on 10 real UK small-business homepages, the main issue matched my manual audit on 7 of 10. Treat it as a way to prioritise, not a verdict.
Check every hook by eye before sending. Sites change. And if you use it for cold email, follow the rules where your recipient is (PECR in the UK, CAN-SPAM in the US, CASL in Canada).

Try it

The actor is here: Website Redesign Signals on Apify. It's pay-per-event at $0.015 per site analysed ($15 per 1,000), and sites it couldn't reach, robots.txt skips, duplicates and invalid inputs aren't charged. You can set a maximum charge per run.

If you'd rather build your own version, the table above is basically the spec: start with the viewport check and the © year regex, they catch a surprising number of neglected sites on their own.

I'd love feedback on the scoring weights, especially from anyone who has reply-rate data on which observations actually land with small-business owners.

Why your small business WordPress site says "Not secure" (and a small Python script that checks for it)

saripalli shanmukha kiran sagar — Thu, 15 Oct 2026 16:43:17 +0000

Most small business sites I look at don't have exotic problems. They have the same four boring ones:

The TLS certificate is expired or about to expire, so browsers show a full-page warning.
http:// doesn't redirect to https://, so old links and some search results land on the insecure version.
The page is HTTPS but still loads images, scripts or stylesheets over http:// (mixed content). Browsers upgrade or block them, and the padlock can disappear.
There's no responsive viewport tag, so phones render a tiny zoomed-out desktop page.

All four can be seen from the outside with a normal page load, so you can check them without any access to the site. Here's a small script that does exactly that, and then the WordPress fix for each one.

The script

Standard library only, so it runs anywhere Python 3.8+ is installed. It loads the homepage like a browser: no scanning, no guessing paths, no logins.

#!/usr/bin/env python3
"""site_check.py - a quick, polite HTTPS + mobile sanity check for a small business site.

Standard library only (Python 3.8+). It loads the homepage like a normal browser
would: no port scans, no login attempts, no path guessing.

Usage:  python3 site_check.py example.com [another-site.co.uk ...]
"""
import socket
import ssl
import sys
import urllib.error
import urllib.request
from datetime import datetime, timezone
from html.parser import HTMLParser

UA = "Mozilla/5.0 (site_check.py; quick public health check)"
TIMEOUT = 15


class PageScan(HTMLParser):
    """Collects the viewport tag and any http:// sub-resources on the page."""

    # tag -> attributes that make the browser fetch something
    RESOURCE_ATTRS = {
        "img": ("src", "srcset"), "script": ("src",), "iframe": ("src",),
        "source": ("src", "srcset"), "video": ("src", "poster"),
        "audio": ("src",), "embed": ("src",), "object": ("data",),
    }

    def __init__(self):
        super().__init__()
        self.viewport = None
        self.insecure = []

    def handle_starttag(self, tag, attrs):
        a = {k.lower(): (v or "") for k, v in attrs}
        if tag == "meta" and a.get("name", "").lower() == "viewport":
            self.viewport = a.get("content", "")
        if tag == "link" and "stylesheet" in a.get("rel", "").lower():
            self._check(tag, a.get("href", ""))
        for attr in self.RESOURCE_ATTRS.get(tag, ()):
            self._check(tag, a.get(attr, ""))

    def _check(self, tag, value):
        # srcset can hold several URLs: "a.jpg 1x, b.jpg 2x"
        for part in value.split(","):
            url = part.strip().split(" ")[0]
            if url.lower().startswith("http://"):
                self.insecure.append(f"<{tag}> {url}")


def cert_days_left(host):
    """Days until the TLS certificate expires, or the error string if the handshake fails."""
    ctx = ssl.create_default_context()
    try:
        with socket.create_connection((host, 443), timeout=TIMEOUT) as sock:
            with ctx.wrap_socket(sock, server_hostname=host) as tls:
                not_after = tls.getpeercert()["notAfter"]
    except ssl.SSLCertVerificationError as e:
        return f"certificate problem: {e.verify_message}"
    except (OSError, ssl.SSLError) as e:
        return f"no HTTPS connection: {e}"
    expires = datetime.fromtimestamp(ssl.cert_time_to_seconds(not_after), timezone.utc)
    return (expires - datetime.now(timezone.utc)).days


def fetch(url):
    """Follow redirects like a browser. Returns (final_url, status, html)."""
    req = urllib.request.Request(url, headers={"User-Agent": UA})
    try:
        with urllib.request.urlopen(req, timeout=TIMEOUT) as r:
            return r.geturl(), r.status, r.read(2_000_000).decode("utf-8", "replace")
    except urllib.error.HTTPError as e:
        return e.geturl(), e.code, ""


def check(host):
    host = host.replace("https://", "").replace("http://", "").strip("/")
    print(f"\n== {host}")
    problems = 0

    # 1. Does the certificate validate, and when does it expire?
    days = cert_days_left(host)
    if isinstance(days, int):
        flag = "OK" if days > 14 else "WARN"
        problems += flag != "OK"
        print(f"  [{flag}] certificate valid, expires in {days} days")
    else:
        problems += 1
        print(f"  [FAIL] {days}")

    # 2. Does plain http:// send visitors to https://?
    try:
        final, status, _ = fetch(f"http://{host}/")
        if final.startswith("https://"):
            print(f"  [OK] http:// redirects to {final}")
        else:
            problems += 1
            print(f"  [FAIL] http:// stays on {final} (status {status}), no redirect to HTTPS")
    except Exception as e:  # noqa: BLE001 - report, don't crash on one bad site
        print(f"  [INFO] http:// check skipped: {e}")

    # 3. Mixed content and viewport on the HTTPS homepage
    try:
        final, status, html = fetch(f"https://{host}/")
    except Exception as e:  # noqa: BLE001
        print(f"  [FAIL] could not load https://{host}/ ({e})")
        return problems + 1
    scan = PageScan()
    scan.feed(html)
    if scan.insecure:
        problems += 1
        print(f"  [FAIL] {len(scan.insecure)} http:// resource(s) on an HTTPS page (mixed content):")
        for item in scan.insecure[:5]:
            print(f"         {item}")
    else:
        print("  [OK] no http:// images, scripts or stylesheets found in the HTML")
    if scan.viewport and "width=device-width" in scan.viewport.replace(" ", ""):
        print(f'  [OK] viewport: "{scan.viewport}"')
    else:
        problems += 1
        print("  [FAIL] no responsive viewport tag, so phones will show a zoomed-out desktop page")
    return problems


if __name__ == "__main__":
    if len(sys.argv) < 2:
        sys.exit(__doc__)
    total = sum(check(h) for h in sys.argv[1:])
    print(f"\n{total} problem(s) found")
    sys.exit(1 if total else 0)

Testing it

badssl.com runs deliberately broken test hosts, which makes it a good way to see each check fire:

python3 site_check.py badssl.com expired.badssl.com mixed.badssl.com http.badssl.com wordpress.org

== badssl.com
  [OK] certificate valid, expires in 74 days
  [OK] http:// redirects to https://badssl.com/
  [OK] no http:// images, scripts or stylesheets found in the HTML
  [OK] viewport: "width=device-width, initial-scale=1.0"

== expired.badssl.com
  [FAIL] certificate problem: certificate has expired
  [INFO] http:// check skipped: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate has expired (_ssl.c:1010)>
  [FAIL] could not load https://expired.badssl.com/ (<urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate has expired (_ssl.c:1010)>)

== mixed.badssl.com
  [OK] certificate valid, expires in 74 days
  [OK] http:// redirects to https://mixed.badssl.com/
  [FAIL] 1 http:// resource(s) on an HTTPS page (mixed content):
         <img> http://mixed.badssl.com/image.jpg
  [OK] viewport: "width=device-width, initial-scale=1"

== http.badssl.com
  [OK] certificate valid, expires in 74 days
  [FAIL] http:// stays on http://http.badssl.com/ (status 200), no redirect to HTTPS
  [OK] no http:// images, scripts or stylesheets found in the HTML
  [OK] viewport: "width=device-width, initial-scale=1"

== wordpress.org
  [OK] certificate valid, expires in 68 days
  [OK] http:// redirects to https://wordpress.org/
  [OK] no http:// images, scripts or stylesheets found in the HTML
  [OK] viewport: "width=device-width, initial-scale=1"

4 problem(s) found

The expired certificate, the missing redirect on http.badssl.com and the insecure image on mixed.badssl.com are all caught; the healthy sites pass. The exit code is 1 when anything fails, so you can drop it into a cron job or CI step for the sites you look after.

Fixing each one on WordPress

Always take a backup first (your host's backup tool, or wp db export plus a copy of wp-content).

1. Expired or failing certificate

On most shared hosting the certificate is a free Let's Encrypt one that renews automatically, and an expired one usually means the renewal broke: the domain's DNS moved, or a CDN/proxy sits in front of the site. Look in the host's SSL/TLS panel first and re-issue there. If the site is behind Cloudflare, check that its SSL mode matches what the origin actually serves.

2. No redirect from `http://` to `https://`

First make sure WordPress itself knows its address is HTTPS: Settings → General, set both WordPress Address (URL) and Site Address (URL) to https://.... Then add a server-level 301. On Apache, near the top of .htaccess (above the # BEGIN WordPress block):

RewriteEngine On
RewriteCond %{HTTPS} off
RewriteRule ^(.*)$ https://%{HTTP_HOST}/$1 [L,R=301]

One trap: behind a proxy or CDN, %{HTTPS} can be off even for HTTPS visitors, which gives you a redirect loop. In that case do the redirect at the CDN (Cloudflare's "Always Use HTTPS") instead of in .htaccess. On Nginx it's a return 301 https://$host$request_uri; in the port-80 server block.

It's also worth adding this to wp-config.php so the dashboard is always served over HTTPS:

define( 'FORCE_SSL_ADMIN', true );

3. Mixed content

This almost always comes from old absolute URLs saved in the database (post content, widgets, page-builder data) from before the site moved to HTTPS. With WP-CLI, do a dry run first:

wp search-replace 'http://example.com' 'https://example.com' --all-tables --dry-run
wp search-replace 'http://example.com' 'https://example.com' --all-tables

wp search-replace handles PHP-serialized data, which a raw SQL REPLACE() would corrupt. Some page builders keep generated CSS files with the old URLs, so regenerate those afterwards (in Elementor: Tools → Regenerate CSS & Data). Anything still left is usually hard-coded in a theme or plugin file, or comes from a third-party embed that only supports http://; replace or remove it.

The script only reads the homepage HTML, so it won't see URLs inside CSS files or resources injected by JavaScript. For the full list, open the browser DevTools console: mixed content warnings name each URL.

4. No viewport tag

Any reasonably modern theme includes it, so a missing viewport usually means a very old or heavily customised theme. The real fix is a responsive theme, because the tag alone won't make a fixed-width layout fit a phone. If the layout already uses responsive CSS and only the tag is missing, add it from a child theme's functions.php:

add_action( 'wp_head', function () {
    echo '<meta name="viewport" content="width=device-width, initial-scale=1">' . "\n";
}, 1 );

Check the page source afterwards so you don't end up with two viewport tags.

Wrapping up

None of this is hard, but it's easy to miss when nobody is looking after the site day to day. Running a check like this monthly catches a certificate that's about to lapse before customers see the warning.

I'm Shanmukh, a student and solo developer. I fix exactly these issues (HTTPS, mobile layout, forms) on small business sites, and build simple 5-page sites, as fixed-scope freelance work. If you'd rather hand it off, the packages are on Fiverr and Whop. More code is on GitHub.

Cleaning a messy product catalogue CSV with pandas (without silently breaking it)

saripalli shanmukha kiran sagar — Thu, 15 Oct 2026 07:48:06 +0000

If you've ever imported a supplier's product list into Shopify, WooCommerce or an ERP, you've probably seen something like this:

sku,title,category,length,price
hb-m8-40-zp,Hex bolt M8x40 zinc,Fasteners,40mm,£0.18
HB-M8-40-ZP ,HEX BOLT M8 X 40MM ZINC PLATED,fasteners,40 mm,0.18
HB-M10-50-A4,Hex Bolt M10 x 50 Stainless A4,Fasteners,5cm,0.71
THR-M12-3FT,Threaded Bar M12 x 3ft,Fasteners,3ft,2.10
HOS-PVC-12,PVC hose 12mm ID,Hose & clips,30m,24.00
SHT-MS-3,Mild steel plate 3mm,sheet metal,2500x1250,

The same bolt appears twice with different casing and a trailing space. Lengths are in mm, cm, ft and m. One price has a currency symbol, one is missing, and "sheet metal" and "Sheet Metal" count as two different categories.

The tempting fix is drop_duplicates() plus a few str.lower() calls. The trouble is that a catalogue is business data. Rows that look the same may differ in finish, pack size or price, and a blank is not always a mistake. My rule: normalise automatically, but anything that changes meaning goes on a review sheet that a human checks.

The pipeline

import re
import pandas as pd

df = pd.read_csv("catalogue.csv", dtype=str, keep_default_na=False)

# 1. Normalise SKUs: trim, uppercase, unify separators
df["sku_norm"] = (df["sku"].str.strip().str.upper()
                  .str.replace(r"[\s_/]+", "-", regex=True))

# 2. Tidy category casing so "fasteners" and "Fasteners" are one category
df["category"] = df["category"].str.strip().str.title()

# 3. Turn free-text lengths into one unit (mm)
TO_MM = {"mm": 1, "cm": 10, "m": 1000, "in": 25.4, "ft": 304.8}
def length_mm(text):
    m = re.fullmatch(r"\s*([\d.]+)\s*(mm|cm|m|in|ft)?\s*", text.lower())
    if not m:
        return None          # blank or unparseable -> flag, don't guess
    value, unit = float(m.group(1)), m.group(2) or "mm"
    return round(value * TO_MM[unit], 1)
df["length_mm"] = df["length"].map(length_mm)

# 4. Prices: strip currency symbols, keep blanks as missing
df["price"] = pd.to_numeric(df["price"].str.replace(r"[^\d.]", "", regex=True),
                            errors="coerce")

# 5. Find duplicates on the normalised SKU, but don't delete them blindly
df["dup_group"] = df.groupby("sku_norm")["sku_norm"].transform("size")
dupes = df[df["dup_group"] > 1].sort_values("sku_norm")

# 6. Build a review sheet: everything a human must look at before import
issues = []
for i, row in df.iterrows():
    if row["dup_group"] > 1:
        issues.append((row["sku"], "possible duplicate of " + row["sku_norm"]))
    if row["length"] and pd.isna(row["length_mm"]):
        issues.append((row["sku"], f"length not understood: {row['length']!r}"))
    if pd.isna(row["price"]):
        issues.append((row["sku"], "missing price"))
review = pd.DataFrame(issues, columns=["sku", "issue"])

with pd.ExcelWriter("catalogue_cleaned.xlsx") as xl:
    df.drop(columns="dup_group").to_excel(xl, sheet_name="cleaned", index=False)
    review.to_excel(xl, sheet_name="needs_review", index=False)
print(review.to_string(index=False))

Running it on the sample above prints the review sheet:

         sku                              issue
 hb-m8-40-zp  possible duplicate of HB-M8-40-ZP
HB-M8-40-ZP   possible duplicate of HB-M8-40-ZP
    SHT-MS-3 length not understood: '2500x1250'
    SHT-MS-3                      missing price

and the cleaned sheet now has one category spelling, length_mm as a number (3ft → 914.4, 5cm → 50.0) and numeric prices.

Why each step is written the way it is

Read everything as strings (dtype=str, keep_default_na=False). Otherwise pandas turns SKUs like 00123 into 123 and blanks into NaN before you've even looked at them.

Keep the original column, add a normalised one. sku_norm sits next to sku, so the review sheet can show the value the supplier actually sent. That's what you'll be asked about when something goes wrong.

Unit parsing returns None instead of guessing. 2500x1250 is a sheet size, not a length. A clever regex could "fix" it into something wrong. Flagging it costs one line on the review sheet.

Duplicates are grouped, not dropped. The two M8 bolts really are the same item here, but in real catalogues "duplicates" often differ in pack quantity or finish (ZP vs BZP). dup_group lets you sort and decide which row survives.

Missing prices stay missing. errors="coerce" turns junk into NaN, and the review step lists it. Never fill prices with 0 or a mean.

Extending it

Add your own unit table for inches written as " or 1/2" (fractions need a small parser).
Normalise materials with a mapping dict ("S/S", "SS", "stainless" → "Stainless steel") and send unmapped values to the review sheet.
For titles, generate them from the attributes (Hex Bolt M8 x 40mm, Zinc Plated) instead of trying to clean free text.

The output is two tabs, cleaned and needs_review, which is usually what the person doing the import wants: most of the work done, plus a short list of decisions only they can make.

I'm Shanmukh, a student and solo developer. I do this kind of catalogue and spreadsheet cleanup, plus small websites and REST APIs, as fixed-scope freelance work. If you'd rather not do it yourself, the packages are on Fiverr and Whop. Code is on GitHub.

I built a Rust LLM inference engine with custom WGSL GPU kernels, here's what I learned!

saripalli shanmukha kiran sagar — Sat, 30 May 2026 07:33:21 +0000

I've been working on a side project called aether , a Rust LLM inference engine that can load GGUF models and run them with WGPU GPU acceleration.

It started as a way to understand how LLMs actually work under the hood. One thing led to another, and now it has:

Loads GGUF models (Llama/Mistral/Phi/Qwen)
WGPU GPU backend (Metal/Vulkan/DX12)
Custom fused WGSL compute shaders for Q8_0 and Q4_K quantized matmul (dequantize inline instead of a separate pass)
Concurrent request pool for serving multiple users
OpenAI-compatible API server (axum)
Pure Rust, no Python dependencies in the hot path

The GPU path is still experimental (CPU mode is the safe default), but the dequant shaders and the fused matmul kernels were honestly the most fun part to write.

I'm not trying to compete with llama.cpp or MLX, this was primarily a learning project that grew into something actually useful. Happy to answer questions or take feedback.

Stack: Rust, WGPU, WGSL, GGUF, axum, Tokio

https://github.com/theoxfaber/aether

(Full transparency, the majority of this code and post were written with AI assistance. I drove the design decisions, architecture, and testing; AI handled a lot of the implementation. Treat it accordingly.)

I built a pure-Rust browser automation library, no Node.js, no wrappers, just CDP over Tokio

saripalli shanmukha kiran sagar — Sun, 10 May 2026 21:08:14 +0000

I got tired of every Rust browser automation library either being a thin wrapper around Node.js (slow, heavy) or completely unmaintained and archived. So I built ferrous-browser a pure-Rust, async-first Chrome DevTools Protocol client that ships as a single binary.
The core ideas behind it:

Zero Node.js. It talks directly to Chrome over CDP using Tokio WebSockets. No npm, no subprocess bridges, nothing.
Correct multi-page isolation. CDP session IDs are tracked per page so concurrent pages don't leak events into each other — something a lot of existing libraries quietly get wrong.

Race-condition-free event handling. Event handlers are registered before the commands that trigger them, not after. Sounds obvious, but most implementations don't do this.

A Playwright-inspired API. locator(), evaluate(), WaitUntil — familiar if you've used Playwright or Puppeteer, but idiomatic Rust.
Here's the basic setup in Cargo.toml:

ferrous-browser = "0.1"
tokio = { version = "1", features = ["full"] }

Requires Chrome or Chromium installed locally. That's it.
A minimal example, navigate, read a heading, take a screenshot:

use ferrous_browser::{Browser, WaitUntil};
#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
let browser = Browser::launch_chrome(None).await?;
let page = browser.new_page().await?;
page.goto("https://example.com", WaitUntil::Load).await?;
let heading = page.locator("h1").inner_text().await?;
println!("Heading: {heading}");

let png = page.screenshot().await?;
std::fs::write("screenshot.png", png)?;

Ok(())
}

The locator API covers click, type_text, wait_for, inner_text, and get_attribute. evaluate() is generic — you pass a JS expression and it deserializes into whatever Rust type you specify.
For navigation, there are three wait modes: DomContentLoaded for speed, Load for full resource loading, and NetworkIdle which waits until no network activity for 500ms — useful for SPAs.
On errors, every failure carries structured context. No more opaque strings:

match page.goto("https://bad-url", WaitUntil::Load).await {
Err(BrowserError::NavigationFailed { url, reason }) =>
eprintln!("Navigation to {url} failed: {reason}"),
Err(BrowserError::Timeout { operation, secs }) =>
eprintln!("{operation} timed out after {secs}s"),
Err(e) => eprintln!("{e}"),
Ok(_) => {}
}

You can also chain context onto any Result with .context("loading homepage")?.
Compared to the existing Rust options, chromiumoxide (stale), headless_chrome (archived), ferrous-browser is the only one with an active locator API, NetworkIdle support, and structured errors out of the box.

The benchmarks are honest: raw page creation is slower than Puppeteer right now (Chrome's session routing is heavily optimized on their end), but the gap closes fast when the workload is real scraping or E2E testing rather than micro-benchmarks.

What's on the roadmap: cookie management, PDF export, evaluate_handle for remote object references, HAR/trace capture, and full Windows support.

Would love feedback, especially from anyone who's hit the multi-page session isolation bugs in other libraries, that was the main itch I was scratching.

GitHub: https://github.com/theoxfaber/ferrous-browser
Crates.io: https://crates.io/crates/ferrous-browser

DEV Community: saripalli shanmukha kiran sagar

Free Python script: check if a website looks outdated (stdlib only, robots.txt-respecting)

How it scores

Usage

The script

How I qualify web-design prospects with an outdated-website checker (no scraping, robots.txt-respecting)

The problem: 200 tabs and a generic first line

Which signals are worth checking

The polite part (no scraping)

How the 0-100 score works

A real example (domain anonymised)

Running it from the Apify API

Python

JavaScript (Node)

Limitations (honestly)

Try it

Why your small business WordPress site says "Not secure" (and a small Python script that checks for it)

The script

Testing it

Fixing each one on WordPress

1. Expired or failing certificate

2. No redirect from http:// to https://

3. Mixed content

4. No viewport tag

Wrapping up

Cleaning a messy product catalogue CSV with pandas (without silently breaking it)

The pipeline

Why each step is written the way it is

Extending it

I built a Rust LLM inference engine with custom WGSL GPU kernels, here's what I learned!

I built a pure-Rust browser automation library, no Node.js, no wrappers, just CDP over Tokio

2. No redirect from `http://` to `https://`