Why I built DataHarvester
While working on a larger project involving the collection of public datasets, I encountered a recurring problem.
Downloading a file is easy. Downloading multiple files, verifying their integrity, tracking what succeeded or failed, and keeping a reliable record of the process is another story.
Sure, I could write Python scripts for everything. But I wanted a more accessible and reusable solution, especially for workflows involving numerous public data sources.
That's how DataHarvester started.
What is DataHarvester?
DataHarvester is a portable Windows application designed to collect files from public HTTPS sources, verify their integrity, and produce traceable collection reports.
The idea is simple: make file collection more reliable, repeatable, and easier to audit.
It's not intended to replace advanced web scraping frameworks or every download manager. Instead, it focuses on a specific problem: collecting files while maintaining clear records of what happened.
What can it actually do?
1. Collect files over HTTPS
DataHarvester retrieves files from supported public HTTPS sources, helping organize collection workflows without requiring a separate script for every task.
2. Verify file integrity with SHA-256
The application calculates SHA-256 checksums to help identify whether downloaded files have changed or match expected content.
A checksum isn't proof that a source is trustworthy, but it provides useful evidence when checking file integrity.
3. Reuse previously verified files
When appropriate, DataHarvester can reuse files that have already been collected and verified, avoiding unnecessary downloads.
This is particularly useful when working with recurring collection tasks or large datasets.
4. Use reusable JSON collection plans
Instead of manually rebuilding the same collection workflow, users can define and reuse structured JSON plans.
That makes repeated collections easier to manage and reproduce.
5. Generate traceable reports
DataHarvester records collection outcomes and provides reporting capabilities, including CSV exports.
The goal is to make it easier to review successful downloads, identify failures, and understand what happened during a collection.
6. Run locally on Windows
DataHarvester is a portable Windows x64 application.
It doesn't require a cloud account, a subscription, or a continuously running external service.
The interface supports both English and French.
A practical example
Imagine you're collecting public economic or demographic datasets from several institutions.
You might need to:
- Download files from multiple HTTPS sources.
- Verify that the collected files are intact.
- Keep track of successful and failed downloads.
- Avoid downloading previously verified files unnecessarily.
- Preserve records of the collection for later review.
You could implement this with scripts, and for experienced developers, that may be the preferred approach.
But not everyone wants to maintain a custom collection pipeline.
DataHarvester brings these tasks together in a dedicated desktop application.
Who might find it useful?
I originally designed DataHarvester around my own data collection needs, but I believe it could also be useful for:
- Researchers collecting publicly available datasets.
- Developers managing repeatable file acquisition workflows.
- Data analysts working with statistical archives.
- Independent projects that need verifiable collection records.
- Anyone who regularly downloads public files and wants better traceability.
It isn't a universal solution, and some workflows will still be better served by custom scripts.
The goal is to provide another option.
Development and current release
DataHarvester has gone through several development and reliability improvements.
The current 1.0.0 stable release builds on that work, with an emphasis on predictable behavior, integrity checking, reusable collection workflows, and clear reporting.
Earlier updates also focused on more informative error messages, handling storage failures, and improving download reliability.
Although version 1.0.0 is a stable release, I'm still interested in feedback about usability, limitations, and future improvements.
I would rather keep the application focused and dependable than add features simply to make the feature list longer.
Availability
DataHarvester is available for Windows x64 on itch.io.
Price: $4.99 CAD — one-time purchase.
Project page: DataHarvester on itch.io
The project page also provides additional information about the application and its capabilities.
I'd appreciate your feedback
I'm especially interested in hearing from developers, researchers, and people who regularly work with public datasets.
- Do you currently use Python scripts, download managers, or other tools for collecting files?
- How important are integrity verification and traceable reports in your workflow?
- Would reusable collection plans save you time?
- What would make a desktop collection tool useful enough for you to consider using it?
Even if DataHarvester isn't something you personally need, I'd be interested in understanding how others approach the same problems.
Thanks for reading!
Top comments (0)