DEV Community: Aniket Patil

Auditing a Google News sitemap with Python (and the date bug that bit me)

Aniket Patil — Thu, 05 Nov 2026 10:17:55 +0000

A news sitemap looks like a normal sitemap with a few extra tags. It isn't. It has its own rules, and if you break them Google quietly ignores the file instead of telling you. No error in Search Console, no warning — the articles just don't show up in the News tab or the Top stories carousel.

I write for an Indian news site and wanted a check I could run before every deploy. This post is the ~100 lines of Python I ended up with, the four rules worth enforcing, and a stdlib bug I hit on the way.

The rules that actually matter

Google's news sitemap spec is short. The parts that cause real failures:

1. The 48-hour window. A news sitemap should only contain articles published in the last two days. Older entries aren't an error exactly — they're just dead weight that Google skips. If your sitemap is full of last month's articles, the handful of fresh ones get lost in it.

2. The 1,000-entry limit. Go over and the file is rejected outright. High-volume publishers need to split into several sitemaps behind an index.

3. <news:publication_date> must be a real timestamp, in the future-proof sense. Timezone included. A date in the future gets the entry dropped.

4. Every <url> needs a <news:news> block. Easy to break when the same template generates both your regular and news sitemaps.

Everything else — keywords, genres, stock tickers — is optional and mostly ignored now.

The script

import re
import sys
import urllib.request
import xml.etree.ElementTree as ET
from collections import Counter
from datetime import datetime, timedelta, timezone

NS = {
    "s": "http://www.sitemaps.org/schemas/sitemap/0.9",
    "n": "http://www.google.com/schemas/sitemap-news/0.9",
}
WINDOW_HOURS = 48
TITLE_MAX = 110

Parsing is plain ElementTree. The only wrinkle is that the news tags live in their own namespace, so every lookup needs the prefix:

def parse(xml_bytes):
    root = ET.fromstring(xml_bytes)
    items = []
    for url in root.findall("s:url", NS):
        news = url.find("n:news", NS)
        if news is None:
            items.append({"loc": url.findtext("s:loc", "", NS), "news": False})
            continue
        items.append({
            "loc": url.findtext("s:loc", "", NS),
            "news": True,
            "title": news.findtext("n:title", "", NS),
            "keywords": news.findtext("n:keywords", "", NS),
            "published": parse_date(news.findtext("n:publication_date", "", NS)),
        })
    return items

Then the checks, each one returning a line of human-readable output rather than a stack trace:

def audit(items):
    now = datetime.now(timezone.utc)
    cutoff = now - timedelta(hours=WINDOW_HOURS)
    problems = []

    missing_news = [i for i in items if not i["news"]]
    if missing_news:
        problems.append(f"{len(missing_news)} <url> entries have no <news:news> block")

    dated = [i for i in items if i.get("published")]
    stale = [i for i in dated if i["published"] < cutoff]
    if stale:
        oldest = min(i["published"] for i in stale)
        problems.append(
            f"{len(stale)} articles are older than {WINDOW_HOURS}h "
            f"(oldest {oldest.isoformat()}) — Google ignores these"
        )

    future = [i for i in dated if i["published"] > now + timedelta(minutes=5)]
    if future:
        problems.append(f"{len(future)} articles have a publication_date in the future")

    long_titles = [i for i in items if len(i.get("title") or "") > TITLE_MAX]
    if long_titles:
        problems.append(f"{len(long_titles)} titles are over {TITLE_MAX} characters")

    dupe_locs = [u for u, c in Counter(i["loc"] for i in items).items() if c > 1]
    if dupe_locs:
        problems.append(f"{len(dupe_locs)} duplicate <loc> values")

    if len(items) > 1000:
        problems.append(f"{len(items)} entries — Google's limit is 1,000")

    return problems, dated

The date bug

This is the part that cost me twenty minutes. My first version did the obvious thing:

datetime.fromisoformat(raw_date)

And it blew up:

ValueError: Invalid isoformat string: '2026-11-05T08:20:41.28+00:00'

Look closely at the fractional seconds: .28. Two digits. Before Python 3.11, fromisoformat only accepted exactly 3 or 6 digits there — it was written to round-trip isoformat() output, not to parse arbitrary ISO-8601. The sitemap was perfectly valid; the parser was picky.

If you're on 3.11+ you'll never see this. Plenty of us aren't, and a CI box on 3.9 failing on one article out of 178 is a miserable thing to debug. A tolerant parser is six lines:

def parse_date(raw):
    if not raw:
        return None
    raw = raw.strip().replace("Z", "+00:00")
    raw = re.sub(r"\.(\d{1,6})",
                 lambda m: "." + m.group(1).ljust(6, "0"),
                 raw, count=1)
    try:
        return datetime.fromisoformat(raw)
    except ValueError:
        return None

Pad the fraction to six digits, normalise a trailing Z, and return None instead of raising so one malformed entry doesn't take the whole run down.

Running it

Against a live news sitemap — Peepals, which publishes across seven sections and is a convenient test subject because it refreshes several times a day:

$ python news_sitemap_audit.py https://peepalsglobal.com/news-sitemap.xml

https://peepalsglobal.com/news-sitemap.xml
178 entries

window covered : 44.2h
title length   : 48-88 chars
keywords       : 5.01 per article, 866 distinct

No problems found.

44.2 hours of coverage, comfortably inside the 48-hour window, and 178 entries against a 1,000 limit. That's what a healthy file looks like.

The summary line is there on purpose. "No problems found" tells you nothing about whether the sitemap is actually doing its job, but 44 hours, 178 entries, titles 48–88 characters tells you the publishing pipeline is alive and the window is being respected.

One number did surprise me: 866 distinct keyword tags across 178 articles, averaging 5.01 per article. Near-zero reuse. Whether <news:keywords> still influences anything is doubtful — Google deprecated the meta keywords equivalent years ago — but it's a cheap signal that each article is being tagged individually rather than by template.

Now the failure case. Point it at a regular sitemap by mistake, which is exactly the kind of config slip that goes unnoticed for months:

$ python news_sitemap_audit.py https://peepalsglobal.com/sitemap-main.xml

https://peepalsglobal.com/sitemap-main.xml
3824 entries

PROBLEMS
  - 3824 <url> entries have no <news:news> block
  - 3824 entries — Google's limit for a news sitemap is 1,000

Both real problems, caught in one run.

Wiring it into CI

The script exits 1 when it finds problems, so it drops straight into a pipeline:

- name: Audit news sitemap
  run: python news_sitemap_audit.py https://example.com/news-sitemap.xml

A word of warning from experience: run it after deploy, not before. The sitemap you want to check is the one your readers and Googlebot actually get, not the one in your build directory. And if your site is behind a CDN, add a cache-busting query string or you'll audit a stale copy for an hour.

What I'd add next

Fetch each <loc> and compare the <news:title> to the <title> on the page. Mismatches between sitemap and page are a common cause of articles not appearing.
Check the canonical. An article whose canonical points elsewhere shouldn't be in the news sitemap at all.
Track the window over time. If coverage drops from 44 hours to 6, something upstream broke — probably worth an alert before anyone notices traffic falling.

The whole thing is stdlib, no dependencies, and it runs in under a second. If you're running a news site and you've never actually looked at what's in that file, it's worth the ten minutes.

Written by Aniket Patil. I write about India's news, technology and markets at Peepals — our technology section is where most of this kind of thing ends up.

Build a Simple News Aggregator in Python Using RSS Feeds

Aniket Patil — Tue, 03 Nov 2026 11:24:35 +0000

A news aggregator collects headlines from several sources and shows them in one place. In this article, we'll build a simple command-line news aggregator in Python that reads RSS feeds, removes duplicate stories, sorts them by time, filters them by keyword and saves the results to a CSV file.

What is an RSS feed?

RSS (Really Simple Syndication) is an XML format that websites use to publish a list of their latest content. Each item in a feed usually contains a title, a link, a publication date and a short summary. Because the format is standard, a program can read feeds from many different websites in the same way.

Prerequisites

Python 3.8 or later
Basic knowledge of Python lists, dictionaries and functions
The feedparser library

Install feedparser using pip:

pip install feedparser

Step 1: Define the feeds

We store the feeds in a dictionary, where the key is the source name and the value is the feed URL. I'm using two large newsrooms plus the technology feed of Peepals, an Indian news site that publishes section-wise RSS feeds. Swap in any feeds you like.

FEEDS = {
    "BBC World": "https://feeds.bbci.co.uk/news/world/rss.xml",
    "The Hindu National": "https://www.thehindu.com/news/national/feeder/default.rss",
    "Peepals Tech": "https://peepalsglobal.com/tech/feed.xml",
}

Step 2: Fetch and parse a feed

The feedparser.parse() function downloads and parses a feed. It returns an object with an entries list, where each entry represents one story.

import feedparser
from datetime import datetime


def fetch_feed(source, url):
    parsed = feedparser.parse(url)

    # bozo is set when the feed is malformed or could not be read cleanly
    if parsed.bozo:
        print(f"Warning: problem reading {source}: {parsed.bozo_exception}")

    stories = []
    for entry in parsed.entries:
        published = entry.get("published_parsed") or entry.get("updated_parsed")
        stories.append({
            "source": source,
            "title": entry.get("title", "").strip(),
            "link": entry.get("link", ""),
            "published": datetime(*published[:6]) if published else None,
        })
    return stories

Explanation:

entry.get() is used instead of entry["title"] because not every feed includes every field.
published_parsed is a time.struct_time. Its first six values are year, month, day, hour, minute and second, which we pass to datetime. feedparser converts these times to UTC, so stories from different feeds can be compared directly.
Some feeds use updated_parsed instead of published_parsed, so we fall back to it.

Step 3: Collect stories from all feeds

def fetch_all(feeds):
    all_stories = []
    for source, url in feeds.items():
        all_stories.extend(fetch_feed(source, url))
    return all_stories

Step 4: Remove duplicate stories

The same story can appear in more than one feed, or more than once in the same feed. We remove duplicates by comparing a normalised version of the title.

def remove_duplicates(stories):
    seen = set()
    unique = []
    for story in stories:
        key = " ".join(story["title"].lower().split())
        if key and key not in seen:
            seen.add(key)
            unique.append(story)
    return unique

Lower-casing the title and collapsing extra spaces means "India Wins Match" and "india wins match" are treated as the same story.

Step 5: Sort by time and filter by keyword

def sort_latest_first(stories):
    return sorted(
        stories,
        key=lambda s: s["published"] or datetime.min,
        reverse=True,
    )


def filter_by_keyword(stories, keyword):
    if not keyword:
        return stories
    keyword = keyword.lower()
    return [s for s in stories if keyword in s["title"].lower()]

Stories without a date get datetime.min, so they appear at the end of the list instead of causing an error.

Step 6: Save the results to CSV

import csv


def save_to_csv(stories, filename):
    with open(filename, "w", newline="", encoding="utf-8") as f:
        writer = csv.DictWriter(f, fieldnames=["published", "source", "title", "link"])
        writer.writeheader()
        writer.writerows(stories)

Step 7: Put it all together

def main():
    keyword = input("Filter by keyword (press Enter for all stories): ").strip()

    stories = fetch_all(FEEDS)
    stories = remove_duplicates(stories)
    stories = sort_latest_first(stories)
    stories = filter_by_keyword(stories, keyword)

    for story in stories[:20]:
        when = story["published"].strftime("%d %b %Y %H:%M") if story["published"] else "No date"
        print(f"[{when}] {story['source']}: {story['title']}")
        print(f"    {story['link']}")

    save_to_csv(stories, "news.csv")
    print(f"\nSaved {len(stories)} stories to news.csv")


if __name__ == "__main__":
    main()

Output

The headlines change every time you run the program. Here is a real run from 3 November 2026, with the keyword india (trimmed):

Filter by keyword (press Enter for all stories): india
[03 Nov 2026 10:47] The Hindu National: PM Modi launches Pune Metro projects, says India's metro network increased from 248 km to 1,200 km in 12 years
    https://www.thehindu.com/news/national/maharashtra/...
[03 Nov 2026 02:21] Peepals Tech: Harley-Davidson Sprint 440 unveiled: India-designed bike priced at $5,999
    https://peepalsglobal.com/tech/article/harley-davidson-sprint-440-unveiled-price-hero-x440-2026-11-02
[02 Nov 2026 09:30] Peepals Tech: H-1B extension rule would hit Indians hardest; comments close November 13
    https://peepalsglobal.com/tech/article/h1b-extension-deference-rule-indians-comment-deadline-2026-11-02

Saved 14 stories to news.csv

Complexity

Time complexity: O(n log n), where n is the total number of stories, because of sorting. Duplicate removal and filtering are O(n).
Space complexity: O(n) to store the stories and the set of seen titles.

Possible improvements

Fetch feeds in parallel using concurrent.futures.ThreadPoolExecutor to speed up the program when there are many feeds.
Schedule the script with cron (Linux/macOS) or Task Scheduler (Windows) to collect news automatically.
Store stories in SQLite instead of CSV to keep a history and avoid saving the same story twice across runs.
Build a web interface using Flask to show the headlines in a browser.
Respect publishers' terms. Use RSS feeds as intended, show headlines with links back to the original source, and don't republish full articles.

Conclusion

With fewer than 100 lines of Python and the feedparser library, we built a working news aggregator that collects headlines from multiple sources, removes duplicates, sorts and filters them, and saves them to a file. The same pattern works for any website that publishes an RSS feed, including blogs, podcasts and job boards.

Drafted with AI for Aniket Patil; the code was run and tested on Python 3.9 with feedparser 6. I write plain-language explainers on technology and India at Peepals.

Seven Performance Lessons for Building a Fast News Website

Aniket Patil — Wed, 21 Oct 2026 11:04:03 +0000

News websites have a hard job. They publish many pages a day, carry heavy images and ads, and most of their readers arrive on mid-range phones over patchy mobile data. In India especially, a slow page doesn't just annoy readers. It loses them before the headline loads.

Here are seven practical lessons for anyone building or maintaining a content-heavy site.

1. Treat the article page as the product

Most readers never see your home page. They land on a single article from search, social or a messaging app. Measure Core Web Vitals on article templates first, and test them on a real low-end Android phone, not just your laptop.

2. Serve images at the right size

Images are usually the biggest bytes on a news page. A few habits help:

Generate several widths and use srcset and sizes so phones don't download desktop images.
Use modern formats such as WebP or AVIF, with a JPEG fallback if you need it.
Always set width and height (or aspect-ratio) so the layout doesn't jump while images load.
Lazy-load everything below the fold with loading="lazy", but never the lead image.

3. Keep the main thread free

Third-party scripts for analytics, ads, comments and social embeds compete for the same main thread that renders your text. Audit them every quarter. Load non-essential scripts after the page becomes interactive, and replace heavy social embeds with a static preview that loads the real embed on tap.

4. Cache aggressively, invalidate precisely

Most article pages change rarely after the first few hours. Put them behind a CDN with a sensible cache lifetime and purge only the URLs that change when an editor updates a story. Use stale-while-revalidate so readers never wait for a cache refill during a traffic spike.

Cache-Control: public, max-age=300, stale-while-revalidate=86400

5. Make fonts fast or skip them

Custom fonts can delay text rendering. Self-host them, subset them to the scripts you actually use, preload only the one weight used in body text, and use font-display: swap. If you publish in several Indian languages, think carefully about which scripts truly need a custom font.

6. Design for bad networks

Assume packet loss and high latency. Keep the critical CSS small and inline, avoid long chains of requests, and make sure the article text is in the initial HTML rather than fetched by JavaScript. Server-rendered or statically generated pages are a good fit for news.

7. Monitor real users, not just lab scores

Lab tools are useful during development, but real-user monitoring tells you what readers actually experience. Track Largest Contentful Paint, Interaction to Next Paint and Cumulative Layout Shift by device type and region, and set alerts when they regress after a deploy.

Wrapping up

Performance work on a news site is never finished, because every new widget, ad slot or embed adds weight. The teams that stay fast are the ones that treat speed as a feature with an owner and a budget, not a one-off clean-up.

I write about technology and digital media for Peepals, an Indian news network.