Batch-Scan PDFs and Word Files for AI-Generated Text with Python

Teachers get a folder of 60 essays. HR gets a shared drive of cover letters. Editors get a pile of freelancer drafts. Checking each one by copy-pasting into a web tool doesn't scale.

In this tutorial we'll build a small Python script that:

1. Walks a folder of PDF, DOCX and TXT files
2. Extracts the text from each one
3. Sends it to an AI detection API, with rate limiting and retries
4. Writes a CSV report with the score and the passages that drove it

The examples use the ProofAI API (it's what we build), but the structure works with any detection provider that returns a probability.

Setup

pip install pypdf python-docx requests

Keep your API key out of the code:

export PROOFAI_API_KEY="sk_live_..."
export PROOFAI_ORIGIN="https://yourapp.com"

Step 1: Extract text from any file

# extract.py
from pathlib import Path
from pypdf import PdfReader
import docx

def extract_text(path: Path) -> str:
    suffix = path.suffix.lower()
    if suffix == ".pdf":
        reader = PdfReader(str(path))
        return "\n".join((page.extract_text() or "") for page in reader.pages)
    if suffix == ".docx":
        d = docx.Document(str(path))
        return "\n".join(p.text for p in d.paragraphs)
    if suffix in {".txt", ".md"}:
        return path.read_text(encoding="utf-8", errors="ignore")
    raise ValueError(f"Unsupported file type: {suffix}")

Two gotchas worth handling:

• Scanned PDFs return almost no text, because the pages are images. Flag them for manual review (or OCR them first) instead of scoring an empty string.
• Headers, footers and reference lists add noise. For essays, consider dropping everything after a "References" or "Bibliography" heading before scoring.

Step 2: Call the detector with retries

# detect.py
import os, time, requests

API_URL = "https://ai.proofai.in/api/v1/detect"

def detect(text: str, max_retries: int = 3) -> dict:
    for attempt in range(max_retries):
        resp = requests.post(
            API_URL,
            headers={
                "x-api-key": os.environ["PROOFAI_API_KEY"],
                "Origin": os.environ["PROOFAI_ORIGIN"],
            },
            data={"content": text},
            timeout=30,
        )
        if resp.status_code == 429:           # rate limited: back off and retry
            time.sleep(2 ** attempt * 2)
            continue
        if resp.status_code >= 500:           # server hiccup: retry once or twice
            time.sleep(2)
            continue
        resp.raise_for_status()
        return resp.json()
    raise RuntimeError("Detection failed after retries")

Exponential backoff on 429 is the difference between a script that finishes and one that dies halfway through a folder of 200 files.

Step 3: Score long documents in chunks

A 3,000-word report is better scored in chunks than as one blob: you get a per-section picture, and one AI-written section doesn't get averaged away.

# chunking.py
def chunks(text: str, size: int = 400):
    words = text.split()
    for i in range(0, len(words), size):
        piece = words[i:i + size]
        if len(piece) >= 50:                  # skip fragments too short to score well
            yield " ".join(piece)

Step 4: Put it together and write a CSV

# scan_folder.py
import csv, sys, time
from pathlib import Path
from extract import extract_text
from detect import detect
from chunking import chunks   # the chunks() function above

SUPPORTED = {".pdf", ".docx", ".txt", ".md"}

def scan(folder: str, out_csv: str = "ai_report.csv"):
    rows = []
    for path in sorted(Path(folder).rglob("*")):
        if path.suffix.lower() not in SUPPORTED:
            continue
        try:
            text = extract_text(path)
        except Exception as e:
            rows.append({"file": path.name, "status": f"error: {e}"})
            continue

        words = len(text.split())
        if words < 50:
            rows.append({"file": path.name, "words": words, "status": "too_short_or_scanned"})
            continue

        scores, top = [], []
        for piece in chunks(text):
            result = detect(piece)
            scores.append(result["ai_probability"])
            top += sorted(result.get("segments", []), key=lambda s: s["ai_prob"], reverse=True)[:1]
            time.sleep(0.5)                   # be gentle with rate limits

        rows.append({
            "file": path.name,
            "words": words,
            "chunks": len(scores),
            "max_ai_probability": max(scores),
            "avg_ai_probability": round(sum(scores) / len(scores), 1),
            "highest_passage": (max(top, key=lambda s: s["ai_prob"])["text"][:150] if top else ""),
            "status": "ok",
        })

    fields = ["file", "words", "chunks", "max_ai_probability", "avg_ai_probability", "highest_passage", "status"]
    with open(out_csv, "w", newline="", encoding="utf-8") as f:
        w = csv.DictWriter(f, fieldnames=fields)
        w.writeheader()
        w.writerows(rows)
    print(f"Scanned {len(rows)} files -> {out_csv}")

if __name__ == "__main__":
    scan(sys.argv[1] if len(sys.argv) > 1 else "./documents")

Run it:

python scan_folder.py ./submissions

You'll get a spreadsheet you can sort by max_ai_probability, with the single passage that scored highest for each file.

Why both max and average?

• Average tells you how the document reads overall.
• Max catches the case where one section (an introduction, a conclusion, a "research summary") was pasted in from an AI tool.

Show reviewers both, plus the passage, so they're looking at evidence rather than a single number.

Using the report responsibly

A batch report makes it tempting to sort and act. Please don't do that blindly:

• Treat scores as a reason to look closer, not a verdict. Detectors produce false positives, and studies (for example Liang et al., Stanford, 2023) have shown higher false-positive rates for non-native English writers.
• Talk to the person. Ask about their process, drafts or version history before any decision.
• Store reports securely. You're processing other people's writing. Delete what you don't need.

Wrapping up

In about 100 lines of Python you have a folder scanner for PDFs, Word files and text, with chunking, retries and a reviewer-friendly CSV. Next steps could be a simple web upload page, OCR for scanned files, or a nightly job on a shared drive.

Full API reference: ProofAI API docs. You can also try the free web detector first (no login needed).

What would you add: OCR, a Google Drive connector, or a Slack summary? Let us know in the comments. 👇

---

Disclosure: we're the team behind ProofAI at 4Born Solutions. This article was drafted with AI assistance and reviewed and edited by our team.