PH3N1X

Posted on Nov 1 AI-assisted

I Built PaperHarbor: An Open-Source Python Tool to Automate Research Paper Discovery

#research #opensource #productivity #python

Turning scattered academic references into an organised collection of accessible research PDFs.

The Problem: Research Papers Are Everywhere, but Collecting Them Is Still Painful

If you're a researcher, PhD scholar, or graduate student, you've probably experienced this situation:

You have hundreds of references stored in Zotero. Your bibliography is organised, your citation metadata is available, and your research topics are clearly defined.

But there's one problem.

Many of the actual PDF files are missing.

You end up manually searching Google Scholar, publisher websites, arXiv, institutional repositories, and other scholarly platforms just to locate accessible copies.

Now imagine repeating this process for 200 or 500 research papers.

As a PhD researcher in Computer Science, I frequently encounter this challenge while preparing literature reviews and exploring related work.

I wanted a workflow where my reference library could act as the starting point for discovering and collecting research papers automatically.

That idea led me to build PaperHarbor.

Introducing PaperHarbor 🚀

PaperHarbor is an open-source, Python-based research utility designed to automate the discovery, retrieval, and organisation of publicly accessible scholarly PDFs.

🔗 GitHub: https://github.com/sayanmondal2098/PaperHarbor

The project follows a straightforward philosophy:

Your bibliography already contains information about what you need to read. Why not use that information to locate the corresponding papers?

Rather than manually opening each reference and searching for its PDF, PaperHarbor processes bibliographic records, identifies missing documents, queries scholarly discovery services, and retrieves available files into a local directory.

It supports researchers who maintain their references in Zotero as well as users working with standalone bibliographic files or individual article identifiers.

🔍 What Can PaperHarbor Do?

1. Import References from Multiple Sources

Research workflows are rarely limited to one reference format.

PaperHarbor supports:

Local Zotero databases
Zotero Web API, including personal and group libraries
BibTeX (.bib)
CSL-JSON
Individual DOIs
Research article URLs

This flexibility allows users to process an existing bibliography without manually rebuilding their reference collection.

2. Discover Missing Research PDFs

One of the central capabilities of PaperHarbor is identifying references that do not already have PDF attachments.

When working with Zotero, the tool can inspect attachment information and prioritize references with missing PDFs.

This reduces unnecessary retrieval attempts and helps researchers focus on incomplete portions of their collections.

3. Search Multiple Scholarly Sources

A major challenge in academic document retrieval is that a paper may be discoverable through one provider but unavailable through another.

PaperHarbor integrates multiple scholarly discovery services, including:

Unpaywall
OpenAlex
arXiv
Europe PMC
Semantic Scholar
Crossref

It can also explore supported repository, publisher, and author-hosted links.

The advantage of this approach is resilience.

If a source cannot provide an accessible PDF, another discovery route may identify a legitimate alternative.

4. Automated Downloading and Validation

Finding a URL is not the same as obtaining a valid research document.

For example, a URL may return an HTML landing page, an error response, or an unrelated document rather than the intended PDF.

PaperHarbor incorporates basic PDF verification and DOI-matching safeguards to reduce these problems.

These checks do not guarantee perfect document identification, but they help make the retrieval workflow more reliable.

5. Structured Local Archiving

Downloading papers is only half the problem.

The files must also be organized so that researchers can locate them later.

PaperHarbor supports configurable filenames based on bibliographic information such as publication year, author, and title.

Instead of accumulating files with arbitrary names, researchers can build a more consistent local archive.

6. Interactive Web Interface

Although PaperHarbor is developed in Python, it is not limited to terminal-based usage.

Its local web interface provides capabilities such as:

Browsing imported references
Searching and filtering records
Selecting individual papers
Processing multiple references
Monitoring retrieval progress
Inspecting download results and activity

This makes the tool more approachable for researchers who are not comfortable with command-line applications.

7. Command-Line Automation and Reports

For developers and technically inclined researchers, PaperHarbor also supports command-line workflows.

This is useful when processing large reference collections or repeating retrieval operations.

The reporting functionality provides structured CSV output for tracking successful downloads, unsuccessful attempts, source information, and follow-up actions.

⚙️ How PaperHarbor Works Internally

At a conceptual level, PaperHarbor follows a multi-stage processing pipeline.

Stage 1: Reference ingestion

The tool reads bibliographic records from supported sources and extracts metadata such as title, author, publication information, DOI, and available URLs.

Stage 2: Missing-PDF detection

The workflow determines which references require retrieval, using available attachment information where supported.

Stage 3: Scholarly discovery

Multiple scholarly services are queried to identify candidate locations for accessible PDFs.

Stage 4: Retrieval and validation

The tool attempts to download available documents and performs basic checks to reduce invalid files and incorrect matches.

Stage 5: Local organization

Retrieved PDFs are saved into a user-selected directory using structured filenames.

Stage 6: Reporting and recovery

The process produces a report, while unsuccessful cases can be revisited using alternative sources or manual follow-up links.

The overall workflow can be summarized as:

References → Missing PDFs → Scholarly Discovery → Retrieval → Validation → Local Archive → Reports

🧠 Why Use Multiple Discovery Providers?

This was an important consideration while developing PaperHarbor.

Academic publications are distributed across different ecosystems.

Some papers are accessible through preprint repositories. Others are available through institutional archives or publisher-hosted open-access links.

A single discovery provider cannot reliably cover every publication.

By combining multiple services, PaperHarbor can attempt alternative discovery paths when an initial lookup fails.

However, multi-source retrieval introduces additional engineering considerations:

Identifying the correct document from metadata
Handling incomplete DOI information
Managing unreliable links and server responses
Avoiding unnecessary duplicate downloads
Distinguishing PDF files from HTML responses
Handling unsuccessful requests without interrupting an entire batch
Producing useful diagnostics for manual follow-up

These challenges make scholarly document retrieval more interesting than a straightforward file-downloading script.

🔐 Responsible and Ethical Research Access

Automating academic downloads also requires respecting publisher access policies.

PaperHarbor is designed to retrieve publicly accessible scholarly PDFs.

It does not:

Bypass publisher paywalls
Circumvent CAPTCHAs
Evade authentication restrictions
Obtain restricted documents through unauthorized methods

If an accessible PDF cannot be located, the tool records the unsuccessful attempt and provides information for further investigation.

Another important design choice is keeping downloaded PDFs separate from Zotero's database.

PaperHarbor does not directly modify the original Zotero library or automatically attach its downloaded PDFs back to Zotero records.

This keeps the retrieval process separate from reference management.

💻 Getting Started

PaperHarbor is publicly available on GitHub.

You can clone the project using:

git clone https://github.com/sayanmondal2098/PaperHarbor.git

Then enter the project directory:

cd PaperHarbor

Follow the installation and usage instructions in the repository's README to configure the required dependencies and launch the supported interface or command-line workflow.

Depending on your preferred workflow, you can start with a Zotero library, an exported bibliography, a DOI, or an article URL.

Project repository: https://github.com/sayanmondal2098/PaperHarbor

🎓 Who Is PaperHarbor For?

PaperHarbor may be particularly useful for:

PhD and postgraduate researchers: Maintaining collections of papers related to their thesis or dissertation.

Academic researchers: Organizing documents across multiple research projects.

Students preparing literature reviews: Collecting accessible papers from existing bibliographies.

Research groups: Processing references and building local collections for shared research activities.

Python developers: Exploring scholarly metadata APIs, automated retrieval, validation, and research workflow automation.

Open-source contributors: Improving document discovery, interface design, reporting, and reliability.

🌍 Open Source and MIT Licensed

I have released PaperHarbor under the MIT License.

This allows developers, researchers, and academic institutions to use, modify, and distribute the software under the license terms.

I believe research tools benefit from community feedback, reproducibility, and collaborative development.

Whether you are interested in scholarly APIs, Python automation, reference management, or digital research infrastructure, your feedback is welcome.

🔭 What's Next?

PaperHarbor currently focuses on one part of a larger research workflow: discovering and collecting accessible scholarly PDFs.

There are several interesting directions for future development:

More robust metadata matching and document verification
Additional scholarly discovery integrations
Improved batch processing and recovery
Better document deduplication
Enhanced research-library organization
More detailed retrieval analytics
Optional integration with downstream document-processing workflows

These are potential development directions, not commitments to features already available.

Longer term, I see opportunities to connect scholarly discovery with semantic search, research knowledge graphs, and literature-analysis systems.

Final Thoughts

As researchers, we spend considerable effort designing experiments, reading publications, evaluating methods, and writing papers.

But many small administrative tasks still interrupt that work.

PaperHarbor grew out of my attempt to automate one of those tasks.

It is not intended to replace Zotero or other reference managers. Instead, it complements existing research workflows by helping locate and archive accessible PDFs.

I am sharing it publicly in the hope that it will be useful to other researchers and developers.

Less time searching for papers. More time doing research.

⭐ Explore and Contribute