DEV Community

Aniket Patil
Aniket Patil

Posted on Fully Autonomous

Build a Simple News Aggregator in Python Using RSS Feeds

#python #rss #tutorial #beginners

A news aggregator collects headlines from several sources and shows them in one place. In this article, we'll build a simple command-line news aggregator in Python that reads RSS feeds, removes duplicate stories, sorts them by time, filters them by keyword and saves the results to a CSV file.

What is an RSS feed?

RSS (Really Simple Syndication) is an XML format that websites use to publish a list of their latest content. Each item in a feed usually contains a title, a link, a publication date and a short summary. Because the format is standard, a program can read feeds from many different websites in the same way.

Prerequisites

  • Python 3.8 or later
  • Basic knowledge of Python lists, dictionaries and functions
  • The feedparser library

Install feedparser using pip:

pip install feedparser
Enter fullscreen mode Exit fullscreen mode

Step 1: Define the feeds

We store the feeds in a dictionary, where the key is the source name and the value is the feed URL. I'm using two large newsrooms plus the technology feed of Peepals, an Indian news site that publishes section-wise RSS feeds. Swap in any feeds you like.

FEEDS = {
    "BBC World": "https://feeds.bbci.co.uk/news/world/rss.xml",
    "The Hindu National": "https://www.thehindu.com/news/national/feeder/default.rss",
    "Peepals Tech": "https://peepalsglobal.com/tech/feed.xml",
}
Enter fullscreen mode Exit fullscreen mode

Step 2: Fetch and parse a feed

The feedparser.parse() function downloads and parses a feed. It returns an object with an entries list, where each entry represents one story.

import feedparser
from datetime import datetime


def fetch_feed(source, url):
    parsed = feedparser.parse(url)

    # bozo is set when the feed is malformed or could not be read cleanly
    if parsed.bozo:
        print(f"Warning: problem reading {source}: {parsed.bozo_exception}")

    stories = []
    for entry in parsed.entries:
        published = entry.get("published_parsed") or entry.get("updated_parsed")
        stories.append({
            "source": source,
            "title": entry.get("title", "").strip(),
            "link": entry.get("link", ""),
            "published": datetime(*published[:6]) if published else None,
        })
    return stories
Enter fullscreen mode Exit fullscreen mode

Explanation:

  • entry.get() is used instead of entry["title"] because not every feed includes every field.
  • published_parsed is a time.struct_time. Its first six values are year, month, day, hour, minute and second, which we pass to datetime. feedparser converts these times to UTC, so stories from different feeds can be compared directly.
  • Some feeds use updated_parsed instead of published_parsed, so we fall back to it.

Step 3: Collect stories from all feeds 

def fetch_all(feeds):
    all_stories = []
    for source, url in feeds.items():
        all_stories.extend(fetch_feed(source, url))
    return all_stories
Enter fullscreen mode Exit fullscreen mode

Step 4: Remove duplicate stories

The same story can appear in more than one feed, or more than once in the same feed. We remove duplicates by comparing a normalised version of the title.

def remove_duplicates(stories):
    seen = set()
    unique = []
    for story in stories:
        key = " ".join(story["title"].lower().split())
        if key and key not in seen:
            seen.add(key)
            unique.append(story)
    return unique
Enter fullscreen mode Exit fullscreen mode

Lower-casing the title and collapsing extra spaces means "India Wins Match" and "india wins match" are treated as the same story.

Step 5: Sort by time and filter by keyword 

def sort_latest_first(stories):
    return sorted(
        stories,
        key=lambda s: s["published"] or datetime.min,
        reverse=True,
    )


def filter_by_keyword(stories, keyword):
    if not keyword:
        return stories
    keyword = keyword.lower()
    return [s for s in stories if keyword in s["title"].lower()]
Enter fullscreen mode Exit fullscreen mode

Stories without a date get datetime.min, so they appear at the end of the list instead of causing an error.

Step 6: Save the results to CSV 

import csv


def save_to_csv(stories, filename):
    with open(filename, "w", newline="", encoding="utf-8") as f:
        writer = csv.DictWriter(f, fieldnames=["published", "source", "title", "link"])
        writer.writeheader()
        writer.writerows(stories)
Enter fullscreen mode Exit fullscreen mode

Step 7: Put it all together 

def main():
    keyword = input("Filter by keyword (press Enter for all stories): ").strip()

    stories = fetch_all(FEEDS)
    stories = remove_duplicates(stories)
    stories = sort_latest_first(stories)
    stories = filter_by_keyword(stories, keyword)

    for story in stories[:20]:
        when = story["published"].strftime("%d %b %Y %H:%M") if story["published"] else "No date"
        print(f"[{when}] {story['source']}: {story['title']}")
        print(f"    {story['link']}")

    save_to_csv(stories, "news.csv")
    print(f"\nSaved {len(stories)} stories to news.csv")


if __name__ == "__main__":
    main()
Enter fullscreen mode Exit fullscreen mode

Output

The headlines change every time you run the program. Here is a real run from 3 November 2026, with the keyword india (trimmed):

Filter by keyword (press Enter for all stories): india
[03 Nov 2026 10:47] The Hindu National: PM Modi launches Pune Metro projects, says India's metro network increased from 248 km to 1,200 km in 12 years
    https://www.thehindu.com/news/national/maharashtra/...
[03 Nov 2026 02:21] Peepals Tech: Harley-Davidson Sprint 440 unveiled: India-designed bike priced at $5,999
    https://peepalsglobal.com/tech/article/harley-davidson-sprint-440-unveiled-price-hero-x440-2026-11-02
[02 Nov 2026 09:30] Peepals Tech: H-1B extension rule would hit Indians hardest; comments close November 13
    https://peepalsglobal.com/tech/article/h1b-extension-deference-rule-indians-comment-deadline-2026-11-02

Saved 14 stories to news.csv
Enter fullscreen mode Exit fullscreen mode

Complexity

  • Time complexity: O(n log n), where n is the total number of stories, because of sorting. Duplicate removal and filtering are O(n).
  • Space complexity: O(n) to store the stories and the set of seen titles.

Possible improvements

  1. Fetch feeds in parallel using concurrent.futures.ThreadPoolExecutor to speed up the program when there are many feeds.
  2. Schedule the script with cron (Linux/macOS) or Task Scheduler (Windows) to collect news automatically.
  3. Store stories in SQLite instead of CSV to keep a history and avoid saving the same story twice across runs.
  4. Build a web interface using Flask to show the headlines in a browser.
  5. Respect publishers' terms. Use RSS feeds as intended, show headlines with links back to the original source, and don't republish full articles.

Conclusion

With fewer than 100 lines of Python and the feedparser library, we built a working news aggregator that collects headlines from multiple sources, removes duplicates, sorts and filters them, and saves them to a file. The same pattern works for any website that publishes an RSS feed, including blogs, podcasts and job boards.

Drafted with AI for Aniket Patil; the code was run and tested on Python 3.9 with feedparser 6. I write plain-language explainers on technology and India at Peepals.

Top comments (0)