DEV Community

sensen0025
sensen0025

Posted on

Resources downloader

I Built a Downloader That Doesn't Ask for Permission

Here's a small frustration we all share.

You find a file. You click download. And then:

  • Login required (for a public file)
  • 14 pop-ups before the button appears
  • "Please verify you're human" — with 47 trackers loading in the background
  • A captcha that takes 3 attempts
  • Finally, an HTML page disguised as a PDF

It's not malicious. It's just how the web works now. Data collection, ad impressions, engagement metrics — everything comes before the user's actual goal.

I got tired of it. So I built a tool.


What I Built

resources_downloader — a DSH skill pack that downloads resources from virtually any site, using an LLM agent to plan and execute the strategy.

The code doesn't hardcode site-specific scrapers. Instead, it gives the agent a set of generic tools and lets it figure out the approach.

Tool Purpose
browser Playwright automation — login, modals, CAPTCHA solving, cookie export
download_file Range-resume downloads
download_hls M3U8 + AES-128 stream download
probe_file Magic bytes verification — ensures the file is real, not an error page
run_code Execute one-off Python scripts for undocumented sites
memory_remember/query Persist working strategies per domain

21 skill cards currently cover Bilibili (BBDown), YouTube (yt-dlp), Anna's Archive, Project Gutenberg, HuggingFace, Quark netdisk, LittleSkin, and more.

If your site isn't covered, the agent writes a crawler on the spot.


The Execution Flow

The agent follows a structured workflow:

  1. Check the catalog and query memory for past successes
  2. Prefer mature CLIs (BBDown, yt-dlp) before falling back to custom scripts
  3. Write and execute Python code if needed
  4. Verify the downloaded file — magic bytes, size, SHA-256
  5. Persist the winning strategy for next time
  6. If 3+ strategies fail, report failure with evidence and terminate gracefully
  7. Deliver full metadata — path, size, hash, source URL

The goal is deterministic delivery. No empty files. No error pages passed off as downloads.


Free API Access — No Strings Attached

Most projects require you to bring your own LLM API key. That's fine, but it adds friction.

So I'm providing free API keys for this service. No credit card required. No "start your free trial" nonsense. Just use it.

Top comments (1)

Some comments may only be visible to logged-in visitors. Sign in to view all comments.