DEV Community: hideki

I Un-Killed a 1.2k-Star Reddit Downloader That Reddit's API Changes Broke

hideki — Wed, 12 Aug 2026 22:48:59 +0000

tl;dr — RedditDownloader (RMD) was a solid tool for archiving Reddit media. In July 2023, Reddit killed Pushshift and locked down direct API access. The maintainer archived the project and explicitly invited forks. I replaced the dead data layer with the Sylvia API (sylvia-api.com) and got subreddit scanning working again. The fork is at github.com/c1nn3r/RedditDownloader.

The problem

RMD was well built. The repo has about 1,200 stars. It shipped a real filter engine, a local web UI, image-hash duplicate detection, a sqlite state database, and resume support so a large archive run can stop and start without re-downloading.

It scanned many places for media: subreddits, user posts, saved lists, upvoted lists, and multireddits. It extracted links from post URLs, from self-post text, and from comments. It handled most of the hosts you meet on Reddit: i.redd.it, v.redd.it, preview.redd.it, imgur, gfycat, and tumblr, with yt-dlp in the corner for video. It ran headless or as a server with a web UI. For the data-hoarding crowd — the people who believe the internet is ephemeral and the stuff they like will vanish — it was the tool.

Then Reddit changed the rules. In July 2023, Reddit destroyed Pushshift, which RMD used for historical data. Reddit also restricted direct API access and made OAuth app registration slower and stricter. The maintainer, ShadowMoose, wrote a blunt shutdown note. He said that the changes made it impossible for RMD to operate. He said that he no longer wanted to build on Reddit's platform at all. Then he archived the repo. But he left the door open:

"Other users are welcome to fork the project and continue onward, if anybody is willing to continue in spite of the Admin actions pushing against them."

So I forked it.

What was actually broken

RMD splits cleanly into two halves: sources and processing. A source decides where posts come from. Processing decides what happens to a post once you have it — filtering, dedup, download, file naming.

The processing half was healthy. The filters, the downloaders, the image-hash dedup, the sqlite state — none of that depended on the API changes.

The source half was dead. Subreddit and user scanning depended on two things that no longer worked:

Pushshift — offline. The API is gone and the Python wrappers around it (psaw, pmaw) are orphaned.
Direct PRAW / JSON access — still alive, but the cost increased. You need your own OAuth app registration, which Reddit approves slowly and strictly. Rate limits come back as HTTP 429 responses, which make bulk scanning painful. And if Reddit has flagged your IP, the HTML and JSON endpoints just serve you a block page.

Everything downstream of the data layer was fine. The rot was entirely in how RMD got post data in the first place.

The fix

I did not touch the processing pipeline. The source/processing split is the reason the fix is small: I wrote one new source.

The new source is SylviaSubredditSource. It replaces the dead Pushshift source for subreddit listings. It pulls posts from the Sylvia API, a Reddit data gateway that skips OAuth app registration entirely.

The flow looks like this:

Sylvia API (subreddit posts)
    → SylviaSubredditSource
    → RedditElement  (the same format RMD always used)
    → direct_link handler → filters → downloaders → disk

How it works:

RMD calls the source with a subreddit name, a sort order (top, new, hot, controversial), and a time window.
The source calls GET /v1/reddit/r/{subreddit}/{order} on the Sylvia API, with the time filter and a limit.
Sylvia returns Reddit-shaped JSON — the same field names Pushshift used to return: id, title, selftext, url, score, num_comments, over_18, created_utc.
The source wraps each post in a dict subclass named Submission.
RedditElement.detect_type() sees the class name, routes the object through the existing Pushshift parsing path, and produces the exact RedditElement object RMD always used.
Everything downstream runs unchanged: the direct_link handler, the filters, the downloader pool, the file naming, the dedup.

Pagination uses the same after-cursor pattern Reddit itself uses. The source loops until it hits the requested limit or the API stops returning posts. That is the same 1,000-post-per-listing window Reddit serves to everyone.

Setup is one environment variable:

export SYLVIA_API_KEY=syl_...

Then add a "Sylvia Subreddit" source in the WebUI or in the settings file, exactly like any other source in RMD. That is the whole migration. You do not need to register a Reddit app, and you do not need to fill in the OAuth fields that have been sitting empty since 2023.

Verification

I did not want to claim that it works without proof. The fork ships two tests:

test_sylvia.py — pulls real posts from r/aww through the Sylvia API, builds RedditElement objects, checks that media URLs are extracted, and checks that the URL filter still works.
test_download.py — end to end. It runs RMD's real direct_link handler against URLs the Sylvia source produced, and confirms that files land on disk.

The download test tried 5 posts and wrote 4 files. The 5th was a self-post with no direct media link, which the handler correctly rejected. Files ranged from 160 KB to 1.3 MB. Every byte of the download path used RMD's real handler code — no stubs, no test doubles.

I also checked the one thing I was worried about: the media CDN. Reddit blocks flagged IPs on its HTML and JSON endpoints, and this machine is flagged. The good news is that the media CDN (i.redd.it, preview.redd.it) does not use the same block list. Downloads worked from this machine. The data layer never touches the blocked endpoints at all, because Sylvia is the one making those requests.

What still does not work

I want to be clear about the limits, because some RMD features are not fixed and cannot be fixed by swapping a data source.

The OAuth-only sources are still broken: saved posts, upvoted lists, multireddits, and "your own user's posts." These need an authenticated Reddit session. That is a property of your Reddit account, not a property of data access. No data gateway can stand in for it. Those sources still need PRAW auth exactly as before.

There is also no deep history. Reddit's JSON API serves only a window of about 1,000 posts per listing, and Sylvia proxies that same window. The Pushshift-era ability to pull "every post ever made in this subreddit" is gone for everyone, not just for this fork.

Media binaries still download from Reddit's CDN. That worked in my tests. If Reddit later blocks download IPs the way it blocks the HTML and JSON endpoints, you will need a proxy for the media fetch — but the data layer will keep working.

If you used RMD for subreddit archiving, this fork gets you back to working. If you relied on saved or upvoted-list syncing, this does not help you yet. That is a different problem — authentication, not data access — and I have not tackled it.

Why fork instead of pull request

The original repo is archived. The maintainer said, plainly, that he is done with anything Reddit-related. A pull request would sit in the queue forever. He explicitly invited forks in the shutdown note, so that is the route I took.

Credit goes where credit is due. The architecture, the handlers, the filters, the dedup, the web UI — that is ShadowMoose and the contributors' work, and it is good work. I replaced one dead pipe. That is the whole diff, and it stayed that small because RMD was built well enough to separate "where the data comes from" from "what the data does."

Made possible by Sylvia API

Sylvia API is what made this fork possible. When Reddit killed Pushshift and locked down PRAW access, RedditDownloader's subreddit-scraping path died with them. Sylvia is a Reddit data gateway that serves Reddit JSON with no OAuth app registration, no rate-limit roulette, no 429s, no IP blocks — just a key and clean JSON. One source swap restored RMD's subreddit scan, filter engine, and media download pipeline for archiving, data hoarding, and image/video backups.

Credit where credit is due: the architecture is ShadowMoose's. Sylvia is the pipe that made it run again.

Fork: github.com/c1nn3r/RedditDownloader

I Built a CLI That Finds Me Clients on Reddit Using AI

hideki — Thu, 23 Jul 2026 20:53:19 +0000

tl;dr — I got tired of manually scrolling Reddit for people asking "who can build me a landing page?" so I built a Rust CLI that scans subreddits, uses AI to score buying intent, and hands me a ranked list of leads. Open source, runs locally, works with Ollama or any cloud AI provider.

The Problem

I'm a freelancer. Half my clients come from Reddit — someone posts in r/SaaS or r/startups asking for help, and I reply. Simple.

Except it's not simple. The workflow looks like this:

Open Reddit
Scroll through 6 subreddits
Read every post
Ignore 95% of them
Maybe find 1-2 posts where someone is actually looking to hire
By the time you find them, 3 other freelancers already replied

It's a full-time job just to find leads. And the signal-to-noise ratio is terrible — most posts are people venting, sharing links, or asking questions they don't need help with.

I wanted a tool that does the scrolling for me and only shows me posts where someone is ready to spend money.

What Cypher Does

Cypher is a terminal tool that:

Scans subreddits you're watching
Filters posts by keyword match first (cheap, fast)
Sends matching posts to AI for signal extraction (intent, budget, urgency)
Scores each lead 0-100
Shows you a ranked table in the terminal

You tell it what you do in one sentence. It suggests subreddits to watch and keywords to filter by. Then you just run cypher scan when you're ready to find leads.

The Stack

Language: Rust — because I wanted it fast and I wanted a single binary I could cargo install and forget about.

Reddit data: Sylvia API — a Reddit data proxy that handles rate limits, parsing, and anti-bot measures. I didn't want to build a scraper. I just wanted an API key and clean JSON. Sylvia does that.

AI layer: Hybrid — local Ollama for speed, cloud fallback (OpenAI, Groq, Gemini, OpenRouter) for when the local model isn't confident enough. You set a cloud_threshold score and anything above it gets re-analyzed by a stronger model.

Database: SQLite — stores leads locally at ~/.local/share/cypher/leads.db. No cloud, no accounts, no tracking.

How It Works Under the Hood

1. Profile Setup

You run cypher init and describe what you do:

I build landing pages for SaaS startups

Cypher takes that sentence, sends it to your configured AI provider, and gets back:

A list of subreddits where your clients hang out
Keywords to filter posts by

For "landing pages for SaaS startups," it might suggest:

r/SaaS, r/startups, r/webdev, r/Entrepreneur, r/smallbusiness
Keywords: "landing page," "need a website," "looking for designer," "conversion"

2. Scan Pipeline

cypher scan

This is the core loop:

Fetch recent posts from watched subreddits (Sylvia API)
    ↓
Keyword filter (cheap, fast — throws out 80% of posts)
    ↓
AI analysis on remaining posts
    ↓
Extract signals: intent, budget, urgency, service needed
    ↓
Score 0-100 based on signal strength
    ↓
Store in local SQLite
    ↓
Display ranked table

The keyword filter is important — it saves API calls and AI tokens. If a post doesn't contain any of your keywords, it never touches the AI layer.

3. Lead Scoring

Each lead gets a score based on:

Signal	Weight	What it measures
Intent	30%	How clearly they're looking to hire (vs. just asking a question)
Budget	25%	Whether they mention money, or it's implied
Urgency	25%	Time pressure ("need this done by Friday" vs. "someday")
Fit	20%	How well the post matches your profile description

A post like "Need a landing page done, budget $3k, launching next week" scores high. "Anyone know a good landing page builder?" scores low — they're probably looking for a tool, not a person.

4. The AI Layer

This is where it gets interesting. Cypher supports two modes:

Local (Ollama): Runs qwen3:4b locally. Fast, free, private. Good enough for most lead scoring.

Cloud fallback: When the local model's confidence is below cloud_threshold, Cypher sends the post to your first configured cloud provider (OpenAI, Groq, Gemini, or OpenRouter). If that fails, it tries the next one.

The config looks like this:

[ai]
local_model = "qwen3:4b"
ollama_url = "http://127.0.0.1:11434"
cloud_threshold = 70
openai_api_key = "sk-..."
openai_model = "gpt-4o-mini"
groq_api_key = "gsk_..."
groq_model = "llama-3.3-70b-versatile"

You don't need all of them. Just one cloud provider works. Ollama is optional — if you don't have it, everything goes through cloud.

5. Daemon Mode

If you want continuous monitoring:

cypher daemon --interval 60

Scans every 60 seconds. Good for leaving running while you work — new leads pop up in your terminal as they appear.

The Commands

cypher init          # Setup profile, API keys, subreddits
cypher scan          # One-shot scan
cypher leads         # List all leads
cypher leads --min-score 80 --status new
cypher lead <id>     # Detailed view of one lead
cypher watch r/SaaS  # Add subreddit to watchlist
cypher unwatch r/SaaS
cypher watchlist     # List all watched subreddits
cypher daemon        # Background scan loop
cypher export csv    # Export to CSV
cypher export json   # Export to JSON
cypher status        # Config and status check

Try It

# Install
cargo install --path .

# Setup
cypher init

# First scan
cypher scan

You'll need:

Rust 1.70+
A Sylvia API key (free tier works)
At least one AI provider (Ollama local, or any cloud key)

What I'd Do Differently

A few things I'd change if I rebuilt this:

Webhook alerts — instead of just terminal output, send high-score leads to Slack or Telegram. I'm building this next.
Reply templates — store pre-written responses per subreddit. When you find a lead, one key to copy a tailored reply.
Historical scoring — track how your lead quality changes over time. Are you getting better at finding clients, or just finding more noise?
Multi-profile — right now it's one profile per config. If you offer multiple services, you need separate configs.

Why Open Source

I built this because I needed it. I'm not trying to turn it into a SaaS — it runs locally, on my machine, with my API keys. If you find it useful, use it. If you want to add features, open a PR.

GitHub: hidekiryuuga/cypher

Cypher is built with Sylvia API — a Reddit data proxy that handles rate limits, parsing, and anti-bot measures. If you're building anything that needs Reddit data, check it out.