tl;dr — RedditDownloader (RMD) was a solid tool for archiving Reddit media. In July 2023, Reddit killed Pushshift and locked down direct API access. The maintainer archived the project and explicitly invited forks. I replaced the dead data layer with the Sylvia API (sylvia-api.com) and got subreddit scanning working again. The fork is at github.com/c1nn3r/RedditDownloader.
The problem
RMD was well built. The repo has about 1,200 stars. It shipped a real filter engine, a local web UI, image-hash duplicate detection, a sqlite state database, and resume support so a large archive run can stop and start without re-downloading.
It scanned many places for media: subreddits, user posts, saved lists, upvoted lists, and multireddits. It extracted links from post URLs, from self-post text, and from comments. It handled most of the hosts you meet on Reddit: i.redd.it, v.redd.it, preview.redd.it, imgur, gfycat, and tumblr, with yt-dlp in the corner for video. It ran headless or as a server with a web UI. For the data-hoarding crowd — the people who believe the internet is ephemeral and the stuff they like will vanish — it was the tool.
Then Reddit changed the rules. In July 2023, Reddit destroyed Pushshift, which RMD used for historical data. Reddit also restricted direct API access and made OAuth app registration slower and stricter. The maintainer, ShadowMoose, wrote a blunt shutdown note. He said that the changes made it impossible for RMD to operate. He said that he no longer wanted to build on Reddit's platform at all. Then he archived the repo. But he left the door open:
"Other users are welcome to fork the project and continue onward, if anybody is willing to continue in spite of the Admin actions pushing against them."
So I forked it.
What was actually broken
RMD splits cleanly into two halves: sources and processing. A source decides where posts come from. Processing decides what happens to a post once you have it — filtering, dedup, download, file naming.
The processing half was healthy. The filters, the downloaders, the image-hash dedup, the sqlite state — none of that depended on the API changes.
The source half was dead. Subreddit and user scanning depended on two things that no longer worked:
-
Pushshift — offline. The API is gone and the Python wrappers around it (
psaw,pmaw) are orphaned. - Direct PRAW / JSON access — still alive, but the cost increased. You need your own OAuth app registration, which Reddit approves slowly and strictly. Rate limits come back as HTTP 429 responses, which make bulk scanning painful. And if Reddit has flagged your IP, the HTML and JSON endpoints just serve you a block page.
Everything downstream of the data layer was fine. The rot was entirely in how RMD got post data in the first place.
The fix
I did not touch the processing pipeline. The source/processing split is the reason the fix is small: I wrote one new source.
The new source is SylviaSubredditSource. It replaces the dead Pushshift source for subreddit listings. It pulls posts from the Sylvia API, a Reddit data gateway that skips OAuth app registration entirely.
The flow looks like this:
Sylvia API (subreddit posts)
→ SylviaSubredditSource
→ RedditElement (the same format RMD always used)
→ direct_link handler → filters → downloaders → disk
How it works:
- RMD calls the source with a subreddit name, a sort order (top, new, hot, controversial), and a time window.
- The source calls
GET /v1/reddit/r/{subreddit}/{order}on the Sylvia API, with the time filter and a limit. - Sylvia returns Reddit-shaped JSON — the same field names Pushshift used to return:
id,title,selftext,url,score,num_comments,over_18,created_utc. - The source wraps each post in a dict subclass named
Submission. -
RedditElement.detect_type()sees the class name, routes the object through the existing Pushshift parsing path, and produces the exactRedditElementobject RMD always used. - Everything downstream runs unchanged: the
direct_linkhandler, the filters, the downloader pool, the file naming, the dedup.
Pagination uses the same after-cursor pattern Reddit itself uses. The source loops until it hits the requested limit or the API stops returning posts. That is the same 1,000-post-per-listing window Reddit serves to everyone.
Setup is one environment variable:
export SYLVIA_API_KEY=syl_...
Then add a "Sylvia Subreddit" source in the WebUI or in the settings file, exactly like any other source in RMD. That is the whole migration. You do not need to register a Reddit app, and you do not need to fill in the OAuth fields that have been sitting empty since 2023.
Verification
I did not want to claim that it works without proof. The fork ships two tests:
-
test_sylvia.py— pulls real posts from r/aww through the Sylvia API, buildsRedditElementobjects, checks that media URLs are extracted, and checks that the URL filter still works. -
test_download.py— end to end. It runs RMD's realdirect_linkhandler against URLs the Sylvia source produced, and confirms that files land on disk.
The download test tried 5 posts and wrote 4 files. The 5th was a self-post with no direct media link, which the handler correctly rejected. Files ranged from 160 KB to 1.3 MB. Every byte of the download path used RMD's real handler code — no stubs, no test doubles.
I also checked the one thing I was worried about: the media CDN. Reddit blocks flagged IPs on its HTML and JSON endpoints, and this machine is flagged. The good news is that the media CDN (i.redd.it, preview.redd.it) does not use the same block list. Downloads worked from this machine. The data layer never touches the blocked endpoints at all, because Sylvia is the one making those requests.
What still does not work
I want to be clear about the limits, because some RMD features are not fixed and cannot be fixed by swapping a data source.
The OAuth-only sources are still broken: saved posts, upvoted lists, multireddits, and "your own user's posts." These need an authenticated Reddit session. That is a property of your Reddit account, not a property of data access. No data gateway can stand in for it. Those sources still need PRAW auth exactly as before.
There is also no deep history. Reddit's JSON API serves only a window of about 1,000 posts per listing, and Sylvia proxies that same window. The Pushshift-era ability to pull "every post ever made in this subreddit" is gone for everyone, not just for this fork.
Media binaries still download from Reddit's CDN. That worked in my tests. If Reddit later blocks download IPs the way it blocks the HTML and JSON endpoints, you will need a proxy for the media fetch — but the data layer will keep working.
If you used RMD for subreddit archiving, this fork gets you back to working. If you relied on saved or upvoted-list syncing, this does not help you yet. That is a different problem — authentication, not data access — and I have not tackled it.
Why fork instead of pull request
The original repo is archived. The maintainer said, plainly, that he is done with anything Reddit-related. A pull request would sit in the queue forever. He explicitly invited forks in the shutdown note, so that is the route I took.
Credit goes where credit is due. The architecture, the handlers, the filters, the dedup, the web UI — that is ShadowMoose and the contributors' work, and it is good work. I replaced one dead pipe. That is the whole diff, and it stayed that small because RMD was built well enough to separate "where the data comes from" from "what the data does."
Made possible by Sylvia API
Sylvia API is what made this fork possible. When Reddit killed Pushshift and locked down PRAW access, RedditDownloader's subreddit-scraping path died with them. Sylvia is a Reddit data gateway that serves Reddit JSON with no OAuth app registration, no rate-limit roulette, no 429s, no IP blocks — just a key and clean JSON. One source swap restored RMD's subreddit scan, filter engine, and media download pipeline for archiving, data hoarding, and image/video backups.
Credit where credit is due: the architecture is ShadowMoose's. Sylvia is the pipe that made it run again.
Top comments (0)