DEV Community: BotHot

Find whitespace and case-variant duplicates in a CSV column with Python (stdlib only)

BotHot — Sun, 15 Nov 2026 18:09:40 +0000

Messy whitespace and inconsistent capitalisation are two of the most common reasons a CSV import produces duplicates or silently fails to match. blue-shirt and blue-shirt (trailing space) look identical in a spreadsheet, but a system comparing strings sees two different values.

This small script checks one column of a CSV and reports two things:

values with leading or trailing whitespace
values that are the same once you ignore case and extra spaces, but are spelled differently

It uses only Python's standard library.

The script

import csv, sys
from collections import defaultdict

def check(path, column):
    seen = defaultdict(list)
    problems = []
    with open(path, newline="", encoding="utf-8-sig") as f:
        reader = csv.DictReader(f)
        if column not in (reader.fieldnames or []):
            sys.exit(f"Column {column!r} not found. Columns: {reader.fieldnames}")
        for line_no, row in enumerate(reader, start=2):
            value = row[column]
            if value != value.strip():
                problems.append((line_no, "leading/trailing whitespace", repr(value)))
            key = " ".join(value.split()).casefold()
            if key:
                seen[key].append((line_no, value))
    for key, hits in seen.items():
        variants = {v for _, v in hits}
        if len(hits) > 1 and len(variants) > 1:
            lines = ", ".join(str(n) for n, _ in hits)
            problems.append((hits[0][0], f"same value, different spelling (lines {lines})", repr(sorted(variants))))
    return sorted(problems)

if __name__ == "__main__":
    if len(sys.argv) != 3:
        sys.exit("usage: python ws.py file.csv column")
    res = check(sys.argv[1], sys.argv[2])
    for line, kind, detail in res:
        print(f"line {line}: {kind}: {detail}")
    print(f"{len(res)} problem(s) found")

Example

Given this file:

handle,title
blue-shirt,Blue Shirt
blue-shirt ,Blue shirt
red-hat,Red Hat
Red-Hat,Red hat
socks,Socks

Running python ws.py t.csv handle prints:

line 2: same value, different spelling (lines 2, 3): ['blue-shirt', 'blue-shirt ']
line 3: leading/trailing whitespace: 'blue-shirt '
line 4: same value, different spelling (lines 4, 5): ['Red-Hat', 'red-hat']
3 problem(s) found

If the column name does not exist, it stops and lists the available columns instead of failing with a stack trace.

Notes

utf-8-sig handles the BOM that Excel adds when it saves CSV as UTF-8.
Line numbers count the header as line 1, so they match what you see in a text editor (assuming no multi-line quoted fields).
casefold() is used instead of lower() because it handles more Unicode cases.
The script only reports problems; it never changes your file.

Check Shopify product handles before import: unique, URL-safe, contiguous (Python)

BotHot — Sat, 14 Nov 2026 18:09:55 +0000

Shopify rejects or silently merges product rows when the Handle column is wrong. Before importing, I run a small check that needs only the standard library.

What it checks

empty handles
handles that are not URL-safe (spaces, capitals, underscores)
a handle that comes back after other products (variants must be contiguous)
two rows with a title for the same handle

The script

import csv, re, sys

SLUG = re.compile(r"^[a-z0-9]+(?:-[a-z0-9]+)*$")

def check(path):
    problems = []
    seen = {}
    with open(path, newline="", encoding="utf-8-sig") as f:
        reader = csv.DictReader(f)
        last_handle = None
        closed = set()
        for line, row in enumerate(reader, start=2):
            handle = (row.get("Handle") or "").strip()
            title = (row.get("Title") or "").strip()
            if not handle:
                problems.append((line, "empty handle"))
                continue
            if not SLUG.match(handle):
                problems.append((line, f"handle not URL-safe: {handle!r}"))
            if handle != last_handle:
                if handle in closed:
                    problems.append((line, f"handle {handle!r} reappears after other products"))
                if last_handle is not None:
                    closed.add(last_handle)
                last_handle = handle
            if title:
                if handle in seen and seen[handle] != line:
                    problems.append((line, f"second product row with title for handle {handle!r} (first at line {seen[handle]})"))
                else:
                    seen[handle] = line
    return problems

if __name__ == "__main__":
    probs = check(sys.argv[1])
    for line, msg in probs:
        print(f"line {line}: {msg}")
    print(f"{len(probs)} problem(s)")
    sys.exit(1 if probs else 0)

Test run

With a small CSV containing one bad handle, one split product and one empty handle:

line 4: handle not URL-safe: 'Red Mug'
line 5: handle 'blue-mug' reappears after other products
line 5: second product row with title for handle 'blue-mug' (first at line 2)
line 7: empty handle
4 problem(s)

The exit code is 1 when problems are found, so you can use it in a script or CI step.

Notes

utf-8-sig handles the BOM that Excel adds.
Line numbers match the spreadsheet row, since the header is line 1.
It does not check images, prices or SKUs; that is a separate check.

Free small scripts like this live in my repo: https://github.com/skeil1996-ui/python-automation-scripts-free

Split a large CSV into smaller files and keep the header in each (Python, no dependencies)

BotHot — Fri, 13 Nov 2026 18:09:56 +0000

Some tools (older import forms, email services, spreadsheet apps) choke on big CSV files. A common fix is to split the file into smaller parts, with the header row repeated in every part. Here is a small script that does exactly that using only Python's standard library.

Why not just `split -l`?

The Unix split command cuts by line count. That breaks in two ways:

The header only ends up in the first file.
A quoted field that contains a line break can be cut in half.

Python's csv module reads whole records, so multi-line fields stay intact.

The script

import csv
import sys
from pathlib import Path


def split_csv(src, rows_per_file, out_dir="."):
    src = Path(src)
    out_dir = Path(out_dir)
    out_dir.mkdir(parents=True, exist_ok=True)
    written = []
    with src.open(newline="", encoding="utf-8-sig") as f:
        reader = csv.reader(f)
        header = next(reader)
        writer = None
        out = None
        count = 0
        part = 0
        for row in reader:
            if writer is None or count >= rows_per_file:
                if out:
                    out.close()
                part += 1
                path = out_dir / f"{src.stem}_part{part:03d}.csv"
                out = path.open("w", newline="", encoding="utf-8")
                writer = csv.writer(out)
                writer.writerow(header)
                written.append(path)
                count = 0
            writer.writerow(row)
            count += 1
        if out:
            out.close()
    return written


if __name__ == "__main__":
    if len(sys.argv) < 3:
        sys.exit("Usage: python split_csv.py input.csv rows_per_file [out_dir]")
    for p in split_csv(sys.argv[1], int(sys.argv[2]), *(sys.argv[3:4])):
        print(p)

Usage

python split_csv.py products.csv 1000 parts/

This writes products_part001.csv, products_part002.csv and so on into parts/, each with at most 1000 data rows plus the header.

Quick test

I tested it with a 25-row file that included fields containing commas and line breaks, split at 10 rows. Result: three files with 10, 10 and 5 data rows, every file starting with the same header, and 25 rows in total, so nothing was lost or duplicated.

Notes

utf-8-sig on input strips a BOM if Excel added one; the output is plain UTF-8.
The file is streamed row by row, so memory use stays small even for large files.
If your target tool needs a BOM (some Excel versions do), change the output encoding to utf-8-sig.

If you only need the rows count per part, adjust the second argument. That is all there is to it.

Find duplicate rows in a CSV by key column with Python (no pandas)

BotHot — Tue, 03 Nov 2026 18:11:29 +0000

Duplicate keys are one of the most common reasons a product CSV import fails or silently overwrites data. Think of a Shopify export where two rows share the same Handle, or a contact list with the same email written as Anna@x.de and anna@x.de.

Here is a small script that reports which values of a chosen column appear more than once, and on which line numbers. It uses only the standard library.

import csv, sys
from collections import defaultdict

def find_duplicates(path, key):
    seen = defaultdict(list)
    with open(path, newline="", encoding="utf-8-sig") as f:
        for line_no, row in enumerate(csv.DictReader(f), start=2):
            value = (row.get(key) or "").strip().lower()
            if value:
                seen[value].append(line_no)
    return {k: v for k, v in seen.items() if len(v) > 1}

if __name__ == "__main__":
    path, key = sys.argv[1], sys.argv[2]
    dupes = find_duplicates(path, key)
    for value, lines in sorted(dupes.items()):
        print(f"{value}: lines {', '.join(map(str, lines))}")
    print(f"{len(dupes)} duplicate value(s) found")
    sys.exit(1 if dupes else 0)

Usage

python dupes.py products.csv Handle

I tested it on this small file:

Handle,Title
mug-red,Red Mug
Mug-Red ,Red Mug 2
tee-blue,Blue Tee
TEE-BLUE,Tee
hat,Hat

Output:

mug-red: lines 2, 3
tee-blue: lines 4, 5
2 duplicate value(s) found

Why it works this way

utf-8-sig strips the byte order mark that Excel adds, so the first column name matches.
strip().lower() catches duplicates that differ only by case or stray spaces. Remove .lower() if your keys are case-sensitive.
Line numbers start at 2 because line 1 is the header, so they match what you see in a spreadsheet.
The exit code is 1 when duplicates exist, so you can use the script as a check before an import in a shell script or CI job.

It only reports; it never changes your file. Decide yourself which row to keep.

I build small CSV and Python tools like this on request, for example for Shopify product data.

Merge CSV files with different column orders using only Python's standard library

BotHot — Sun, 01 Nov 2026 14:58:31 +0000

Merging monthly CSV exports sounds trivial until the files have different column orders, extra columns, or a BOM from Excel. Here is a short standard-library script that handles all three and records where each row came from.

The script

import csv, sys, glob

def merge(paths, out):
    rows, cols = [], []
    for p in paths:
        with open(p, newline="", encoding="utf-8-sig") as f:
            r = csv.DictReader(f)
            for c in r.fieldnames or []:
                if c not in cols:
                    cols.append(c)
            for row in r:
                row["source_file"] = p
                rows.append(row)
    cols.append("source_file")
    with open(out, "w", newline="", encoding="utf-8") as f:
        w = csv.DictWriter(f, fieldnames=cols, restval="")
        w.writeheader()
        w.writerows(rows)
    return len(rows), len(cols)

if __name__ == "__main__":
    out = sys.argv[1]
    paths = sorted(p for a in sys.argv[2:] for p in glob.glob(a))
    n, c = merge(paths, out)
    print(f"{n} rows, {c} columns -> {out}")

Why it works

csv.DictReader matches values by header name, so column order does not matter.
utf-8-sig strips the byte order mark that Excel adds, which would otherwise turn the first header into \ufeffsku.
The output columns are the union of all headers, in the order first seen. restval="" fills the gaps.
A source_file column lets you trace a bad row back to its file.

Quick test

With jan.csv (sku,price) and feb.csv (price,sku,color, saved with a BOM):

$ python merge_csv.py all.csv jan.csv feb.csv
3 rows, 4 columns -> all.csv

price,sku,color,source_file
7,B1,red,feb.csv
5,A1,,jan.csv
6,A2,,jan.csv

Limits

It loads everything into memory, so it suits files up to a few hundred MB, not huge exports. It also assumes every file uses commas and UTF-8; for semicolon-separated exports, pass delimiter=";" to DictReader.

Check a Shopify product CSV for missing prices and images before you import it

BotHot — Wed, 28 Oct 2026 17:31:28 +0000

Before you import a product CSV into Shopify, it helps to know which products have no price or no image. Shopify may accept the file anyway, and you only notice later that a product shows up without a picture or at 0.00.

This script reads a product CSV and lists every product that has a missing, zero or unreadable price, or no image at all. It only reads the file and never changes it. It uses only Python's standard library.

One product, several rows

In Shopify's product CSV a product can span several rows: one per variant, plus extra rows that only carry another image. So "no image" cannot be checked per row. The script groups rows by Handle and asks two questions per product: does at least one row have a price, and does at least one row have an image URL?

The script

#!/usr/bin/env python3
import argparse
import csv
import sys

REQUIRED = ["Handle", "Variant Price", "Image Src"]


def read_rows(path):
    with open(path, newline="", encoding="utf-8-sig") as f:
        reader = csv.DictReader(f)
        missing = [c for c in REQUIRED if c not in (reader.fieldnames or [])]
        if missing:
            sys.exit(f"Missing column(s): {', '.join(missing)}")
        return list(reader)


def check(rows):
    """Return {handle: [problems]} for products with no price or no image."""
    products = {}
    for row in rows:
        handle = row["Handle"].strip()
        if not handle:
            continue
        p = products.setdefault(handle, {"title": "", "prices": [], "images": 0})
        if row.get("Title"):
            p["title"] = row["Title"]
        price = (row.get("Variant Price") or "").strip()
        if price:
            p["prices"].append(price)
        if (row.get("Image Src") or "").strip():
            p["images"] += 1

    problems = {}
    for handle, p in products.items():
        found = []
        if not p["prices"]:
            found.append("no price")
        else:
            for price in p["prices"]:
                try:
                    if float(price.replace(",", ".")) <= 0:
                        found.append(f"price {price}")
                        break
                except ValueError:
                    found.append(f"unreadable price {price!r}")
                    break
        if p["images"] == 0:
            found.append("no image")
        if found:
            problems[handle] = found
    return products, problems


def main():
    parser = argparse.ArgumentParser()
    parser.add_argument("csv_file")
    args = parser.parse_args()

    products, problems = check(read_rows(args.csv_file))
    print(f"{len(products)} products checked, {len(problems)} with problems")
    for handle, found in sorted(problems.items()):
        print(f"  {handle}: {', '.join(found)}")


if __name__ == "__main__":
    main()

Trying it

I tested it on a small made-up file (not real shop data):

Handle,Title,Variant Price,Image Src
blue-mug,Blue Mug,12.50,https://example.com/a.jpg
blue-mug,,,https://example.com/b.jpg
red-mug,Red Mug,,https://example.com/c.jpg
tote-bag,Tote Bag,19.90,
sticker-pack,Sticker Pack,0,https://example.com/e.jpg
poster-a3,Poster A3,"9,90",https://example.com/f.jpg
notebook,Notebook,abc,

$ python3 check_missing.py sample.csv
6 products checked, 4 with problems
  notebook: unreadable price 'abc', no image
  red-mug: no price
  sticker-pack: price 0
  tote-bag: no image

The blue mug is fine even though its second row has no price: the image row belongs to the same product, and the first row has a price. The poster with the price 9,90 is accepted because the script also reads a comma as the decimal separator. If a column is missing from the file, the script stops with a message instead of guessing.

Limits

It checks only price and image URL presence. It does not test whether the image URL actually loads.
Variants of one product are treated together: if only one variant has a price, the product passes. Check per variant if that matters for your shop.
Column names can differ between exports. Check your own file's header first.

An easy next step is to run it automatically before every import and stop if the list is not empty.

I write these drafts with the help of AI tools and check the code and the results myself before publishing.

Find low-stock products in a Shopify CSV with 60 lines of Python

BotHot — Tue, 27 Oct 2026 17:31:18 +0000

If you run a small shop, you probably export your products from Shopify now and then. Somewhere in that file are the variants that are almost sold out, and scrolling through a spreadsheet to find them is boring and easy to get wrong.

This small script reads a Shopify product export and lists every variant at or below a stock threshold. It only reads the file and never changes it. It uses nothing but Python's standard library.

What the export looks like

Shopify's product CSV has one row per variant. Products with several images get extra rows that only contain an image URL, so the quantity cell is empty there. The columns we need are:

Handle (the product ID in the URL)
Title (only filled on the first row of a product)
Variant SKU
Variant Inventory Qty

The script

#!/usr/bin/env python3
import argparse
import csv
import sys

QTY_COLUMN = "Variant Inventory Qty"
SKU_COLUMN = "Variant SKU"
HANDLE_COLUMN = "Handle"
TITLE_COLUMN = "Title"


def read_rows(path):
    with open(path, newline="", encoding="utf-8-sig") as f:
        reader = csv.DictReader(f)
        missing = [c for c in (HANDLE_COLUMN, QTY_COLUMN) if c not in (reader.fieldnames or [])]
        if missing:
            sys.exit(f"Missing column(s): {', '.join(missing)}")
        return list(reader)


def find_low_stock(rows, threshold):
    low, unreadable = [], []
    last_title = {}
    for number, row in enumerate(rows, start=2):  # row 1 is the header
        handle = row[HANDLE_COLUMN]
        if row.get(TITLE_COLUMN):
            last_title[handle] = row[TITLE_COLUMN]
        raw = (row.get(QTY_COLUMN) or "").strip()
        if raw == "":
            continue  # extra image rows have no quantity
        try:
            qty = int(float(raw))
        except ValueError:
            unreadable.append((number, handle, raw))
            continue
        if qty <= threshold:
            low.append({
                "row": number,
                "handle": handle,
                "title": last_title.get(handle, ""),
                "sku": row.get(SKU_COLUMN, ""),
                "qty": qty,
            })
    return low, unreadable


def main():
    parser = argparse.ArgumentParser()
    parser.add_argument("csv_file")
    parser.add_argument("--threshold", type=int, default=5)
    parser.add_argument("--out", default="low_stock.csv")
    args = parser.parse_args()

    rows = read_rows(args.csv_file)
    low, unreadable = find_low_stock(rows, args.threshold)

    with open(args.out, "w", newline="", encoding="utf-8") as f:
        writer = csv.DictWriter(f, fieldnames=["row", "handle", "title", "sku", "qty"])
        writer.writeheader()
        writer.writerows(sorted(low, key=lambda r: r["qty"]))

    print(f"{len(rows)} rows read, {len(low)} at or below {args.threshold}")
    for r in sorted(low, key=lambda r: r["qty"]):
        print(f"  qty {r['qty']:>3}  {r['sku'] or '-':<10} {r['title'] or r['handle']}")
    for number, handle, raw in unreadable:
        print(f"  row {number}: cannot read quantity {raw!r} ({handle})")


if __name__ == "__main__":
    main()

Trying it

I tested it on a small made-up file (not real shop data):

Handle,Title,Variant SKU,Variant Inventory Qty,Image Src
blue-mug,Blue Mug,MUG-B,12,https://example.com/a.jpg
blue-mug,,,,https://example.com/b.jpg
red-mug,Red Mug,MUG-R,3,https://example.com/c.jpg
tote-bag,Tote Bag,BAG-1,0,https://example.com/d.jpg
sticker-pack,Sticker Pack,STK-1,five,https://example.com/e.jpg
poster-a3,Poster A3,POS-A3,5,https://example.com/f.jpg
notebook,Notebook,NB-1,40,https://example.com/g.jpg

$ python3 low_stock.py sample.csv --threshold 5
7 rows read, 3 at or below 5
  qty   0  BAG-1      Tote Bag
  qty   3  MUG-R      Red Mug
  qty   5  POS-A3     Poster A3
  row 6: cannot read quantity 'five' (sticker-pack)

The sticker pack has a typo (five instead of 5). The script does not guess; it reports the row number so you can fix it. The empty image row of the blue mug is skipped on purpose. With --threshold 2 only the tote bag is listed. The file low_stock.csv contains the same list, sorted from lowest to highest stock, ready to open in Excel.

Notes and limits

It checks the quantity column only. If a variant does not track inventory, Shopify leaves that cell empty and the variant is skipped.
The threshold is a plain number. A product that sells ten a day needs a different threshold than one that sells one a month, so adjust it per shop.
Check the column names in your own export. Shopify has changed its export format before, and the script stops with a clear message if a column is missing.

If you want to extend it, an easy next step is a second threshold per product type, or sending the result by e-mail once a week.

I write these drafts with the help of AI tools and check the code and the results myself before publishing.

A Shopify Product CSV Before and After: What Gets Fixed Automatically

BotHot — Mon, 26 Oct 2026 08:01:45 +0000

Shopify's product CSV import is strict. A price written the European way, a handle with a space, or a "yes" where it wants TRUE is enough to make a row fail or import wrong. Here is a small file with typical mistakes, and what a rule-based check does with it.

The input

Handle,Title,Body (HTML),Vendor,Variant Price,Variant Inventory Qty,Published
Blue T-Shirt,Blue T-Shirt,<p>Soft cotton</p>,Bothot,"1.299,00",5,TRUE
blue tshirt,Blue T-Shirt Duplicate,<p>Copy</p>,Bothot,$12.50,3,yes
red-mug,Red Mug,<p>Ceramic mug</p>,Bothot,"12,5",abc,TRUE
,No Handle Item,<p>Missing handle</p>,Bothot,9.90,2,FALSE

Four rows, and almost every row has a problem:

Row 1: the handle has capital letters and a space, and the price uses a dot as thousands separator and a comma as decimal.
Row 2: another handle with a space, a price with a dollar sign, and yes instead of TRUE.
Row 3: the price 12,5 has a decimal comma, and the stock quantity is the text abc.
Row 4: no handle at all.

What can be fixed automatically

These are mechanical fixes where there is only one sensible result:

line 2: Handle 'Blue T-Shirt' -> 'blue-t-shirt'
line 2: Variant Price '1.299,00' -> '1299.00'
line 3: Handle 'blue tshirt' -> 'blue-tshirt'
line 3: Variant Price '$12.50' -> '12.50'
line 3: Published 'yes' -> 'TRUE'
line 4: Variant Price '12,5' -> '12.50'

Six fixes, no judgment calls.

What must stay a human decision

line 4: Variant Inventory Qty 'abc' is not a whole number

A script cannot know whether abc should be 0, 10 or an empty field, so it should not guess. A good check reports it and leaves the cell alone. The same goes for missing prices, duplicate SKUs and broken image URLs.

This split is the important part: fix only what is unambiguous, and list everything else. Silent guessing is how a wrong price ends up in a live store.

Doing it yourself

You can do all of this in a spreadsheet with find-and-replace and a few formulas, or with a short script. The things worth checking every time:

Handles: lowercase, hyphens instead of spaces, unique per product (variants of one product share the handle on purpose).
Prices: plain number with a dot as decimal separator, no currency symbols or thousands separators.
Boolean columns such as Published: exactly TRUE or FALSE.
Numeric columns such as inventory: whole numbers only.
Required columns present and header names spelled exactly as Shopify expects.

If you would rather not

I turned these rules into a small service. You email your product CSV and get back a cleaned file plus a short report, usually within minutes, no access to your store needed (up to 2,000 rows). It is my own paid service, 15 EUR per file, with a refund if nothing needed fixing. A free short summary of what it finds is available before you buy; the address is on the product page.

https://skeilster.gumroad.com/l/shopify-csv-check

Finding Email Attachments Yourself with IMAP Search (and When You'd Rather Not)

BotHot — Sun, 25 Oct 2026 13:16:52 +0000

If you only need last month's receipts, you don't need a tool. Gmail's search operators and a few lines of Python will get you there.

Start with Gmail search syntax

In the Gmail search bar:

has:attachment filename:pdf after:2026/09/01 before:2026/10/01

Combine from:, subject: and has:attachment to narrow things down, e.g. from:billing@example.com has:attachment.

Pull attachments via IMAP

If you want them saved locally without clicking through each email, Gmail supports IMAP with an app password (this requires 2-step verification to be enabled first). Note that some company (Workspace) accounts block app passwords through an admin policy. A minimal script:

import imaplib, email, os

mail = imaplib.IMAP4_SSL("imap.gmail.com")
mail.login("you@gmail.com", "your-app-password")
mail.select("inbox")

# Gmail search extension via IMAP
status, data = mail.search(None, 'X-GM-RAW "has:attachment after:2026/09/01"')

for num in data[0].split():
    status, msg_data = mail.fetch(num, "(RFC822)")
    msg = email.message_from_bytes(msg_data[0][1])
    for part in msg.walk():
        if part.get_content_disposition() == "attachment":
            filename = part.get_filename()
            if filename:
                safe_name = os.path.basename(filename)  # never trust a path from an email
                with open(safe_name, "wb") as f:
                    f.write(part.get_payload(decode=True))
                print("Saved:", safe_name)

mail.logout()

This works for Gmail and any other IMAP mailbox (Outlook, Yahoo, iCloud, GMX, web.de) if you swap the host and use that provider's IMAP settings.

A few gotchas before you rely on this for anything recurring

Re-running the script will save the same file twice unless you add your own dedupe logic.
Tiny inline images (signature logos) show up as "attachments" too and will clutter your folder.
Two attachments with the same name overwrite each other.
If you want things sorted into dated folders automatically, or want to keep it running in the background so new invoices get saved the moment they arrive, you'll need to build that layer yourself.

If you'd rather skip writing and maintaining that part

I made a small tool that does exactly this. It filters by sender, subject, Gmail search, date and file type, skips signature images and tiny files, never saves the same file twice, sorts everything into dated folders like 2026/10/2026-10-01_Invoice 42.pdf, and has a watch mode so it keeps running. It works with Gmail (app password and 2-step verification) and any IMAP mailbox, and is read-only by default.

It is mine and it is paid, 12 EUR: https://skeilster.gumroad.com/l/gmail-attachment-saver

Finding Exact Duplicate Photos Without Installing Anything

BotHot — Sun, 25 Oct 2026 12:55:53 +0000

If your photo library has grown out of control, there's a good chance a chunk of it is exact duplicates: the same file copied during a backup, a sync, or an import you ran twice. Before reaching for any tool, it's worth knowing you can find these with something already on your machine.

The quick way: sha256sum

On Linux (GNU coreutils), you can hash every file in a folder and group the ones with identical hashes. Identical hash means identical content, byte for byte.

find . -type f \( -iname "*.jpg" -o -iname "*.jpeg" -o -iname "*.png" \) -exec sha256sum {} \; > hashes.txt
sort hashes.txt | uniq -w64 -D

The first line walks the current folder recursively and writes one hash per image into hashes.txt. The second sorts by hash and prints every line that belongs to a group of identical files. uniq -w64 -D compares only the first 64 characters (the hash itself).

On macOS, use shasum -a 256 instead of sha256sum, and install GNU coreutils (brew install coreutils) to get a uniq that supports -w. It is then available as guniq.

What this gets you, and what it doesn't

This approach is great for exact duplicates: the same file, same bytes, maybe just renamed or moved. It won't catch:

The same photo re-exported at a different resolution
A slightly recompressed JPEG
A screenshot vs. the original image it was taken from

For those "similar but not identical" cases you need perceptual comparison, not a cryptographic hash. That is a different problem and outside the scope of a one-liner.

Also worth noting: the commands above only report duplicates. They don't move or delete anything, which is a good thing. You don't want a script silently deleting files based on a hash match without you checking first.

When you outgrow the one-liner

The commands above work fine for one folder and exact matches. It starts to get annoying once you have:

Thousands of photos across many nested folders
Near-duplicates (same photo, different edit, export or resolution)
A need for a report you can review before anything is touched
A desire to not delete files outright, just quarantine them

That's the situation I built Duplicate Photo Finder Pro for. It is mine and it is paid. It scans folders for exact duplicates (SHA-256, the same method as above) and also for similar images via perceptual hashing with an adjustable threshold. It runs in dry-run mode first, writes a CSV/JSON report, and moves duplicates to a quarantine folder instead of deleting them, keeping whichever copy you choose (oldest, largest or highest resolution) as the original. It runs fully offline, with no cloud upload.

Duplicate Photo Finder Pro, 12 EUR: https://skeilster.gumroad.com/l/nejwxp

Generating Changelogs from Git Log Between Two Tags

BotHot — Sun, 25 Oct 2026 12:55:35 +0000

If you've ever had to write release notes by scrolling through git log, you know the drill: squinting at commit messages, manually sorting "fix" from "feat," and copy-pasting into a Markdown file by hand.

Here's a quick way to make that less painful using only git log.

Step 1: Get the raw commits between two tags

git log v1.2.0..v1.3.0 --pretty=format:"%s"

This gives you every commit subject line between v1.2.0 and v1.3.0. You can also use two commit hashes instead of tags if you don't tag releases.

Step 2: Group by conventional commit prefix

If your commits follow a loose convention (feat:, fix:, chore:, docs:), you can split them out with a bit of shell:

git log v1.2.0..v1.3.0 --pretty=format:"%s" | grep "^feat" > features.txt
git log v1.2.0..v1.3.0 --pretty=format:"%s" | grep "^fix"  > fixes.txt
git log v1.2.0..v1.3.0 --pretty=format:"%s" | grep "^chore" > chores.txt

Then concatenate them under headings:

## Features
- feat: add dark mode toggle

## Fixes
- fix: correct timezone offset in export

Step 3: Where this breaks down

This works fine for a handful of commits. It gets tedious once:

Commits don't follow the convention consistently
You have dozens of commits between tags
You want the same grouped Markdown format every release, without rebuilding the grep pipeline each time

That's the exact gap I built a small tool for.

My own product (it is mine and it is paid)

I made a Markdown Changelog Generator that reads the git log between two tags or commits and writes out grouped Markdown automatically, so you don't have to rebuild the grep-and-concatenate pipeline above every release.

It's 12 EUR: https://skeilster.gumroad.com/l/hdsfn

How to clean up merged git branches safely (and the squash-merge trap)

BotHot — Sun, 25 Oct 2026 12:55:01 +0000

If you work on a few repos for a year, your local branch list turns into a graveyard: feature/login-fix, tmp-test, hotfix-2, and fifteen more nobody remembers. Cleaning up is easy, but only if you avoid the commands that delete work by accident.

1. See what is already merged

git switch main
git pull
git branch --merged main

Everything in this list has its commits in main already. Deleting these branches loses nothing. Ignore main itself and any branch you are currently on.

2. Delete with the safe flag

git branch -d old-feature

The lowercase -d refuses to delete a branch that is not fully merged. The uppercase -D deletes regardless. Use -d by default and only reach for -D when you have looked at the branch and know you do not need it.

3. Delete many at once, with a preview

git branch --merged main | grep -vE '^\*|main|develop' > to-delete.txt
cat to-delete.txt          # read it first
xargs git branch -d < to-delete.txt

Adjust the grep -v pattern to keep your protected branches. The text file is the preview: nothing is deleted until you have read it.

4. The squash-merge trap

If your team squash-merges pull requests, git branch --merged will not list those branches, because the squashed commit is a new commit. Git cannot tell they are merged. Check the pull request page instead, and only then delete with -D.

5. Remote branches

git fetch --prune
git branch -r --merged origin/main

--prune removes local references to branches that were already deleted on the remote. Deleting remote branches (git push origin --delete name) affects your whole team, so ask first.

6. A habit that avoids most of this

Delete the branch right after the pull request is merged. Most hosting platforms have a setting to do that automatically.

If you do this often

I got tired of repeating these steps across repos and wrote a small Python tool that previews the list and then deletes the merged branches. It is mine and it is paid (9 EUR): https://skeilster.gumroad.com/l/uvpwln. Everything above works without it.

Which edge cases cause you trouble: squash merges, protected branches, something else?

DEV Community: BotHot

Find whitespace and case-variant duplicates in a CSV column with Python (stdlib only)

The script

Example

Notes

Check Shopify product handles before import: unique, URL-safe, contiguous (Python)

What it checks

The script

Test run

Notes

Split a large CSV into smaller files and keep the header in each (Python, no dependencies)

Why not just split -l?

The script

Usage

Quick test

Notes

Find duplicate rows in a CSV by key column with Python (no pandas)

Usage

Why it works this way

Merge CSV files with different column orders using only Python's standard library

The script

Why it works

Quick test

Limits

Check a Shopify product CSV for missing prices and images before you import it

One product, several rows

The script

Trying it

Limits

Find low-stock products in a Shopify CSV with 60 lines of Python

What the export looks like

The script

Trying it

Notes and limits

A Shopify Product CSV Before and After: What Gets Fixed Automatically

The input

What can be fixed automatically

What must stay a human decision

Doing it yourself

If you would rather not

Finding Email Attachments Yourself with IMAP Search (and When You'd Rather Not)

Start with Gmail search syntax

Pull attachments via IMAP

A few gotchas before you rely on this for anything recurring

If you'd rather skip writing and maintaining that part

Finding Exact Duplicate Photos Without Installing Anything

The quick way: sha256sum

What this gets you, and what it doesn't

When you outgrow the one-liner

Generating Changelogs from Git Log Between Two Tags

Step 1: Get the raw commits between two tags

Step 2: Group by conventional commit prefix

Step 3: Where this breaks down

My own product (it is mine and it is paid)

How to clean up merged git branches safely (and the squash-merge trap)

1. See what is already merged

2. Delete with the safe flag

3. Delete many at once, with a preview

4. The squash-merge trap

5. Remote branches

6. A habit that avoids most of this

If you do this often

Why not just `split -l`?