DEV Community

Michael Rivera
Michael Rivera

Posted on AI-assisted

Broken SFTP file Transfers Enraged me to Build a Proper Transfer Engine

#backend #devops #programming #softwaredevelopment

I Got Tired of Broken SFTP Scripts So I Built a Proper Transfer Engine

Every few months I'd find a half-transferred file, a corrupted download that looked complete, or a transfer that silently failed and left nothing in the log. The scripts people write for SFTP — including mine — are usually fine until they're not.

sftp-ultra is what I built to replace them. A production SFTP transfer engine with concurrent workers, resumable downloads, SHA-256 verification, a SQLite transfer journal, and deterministic exit codes.

Here's what it actually does and why each piece is there.

The Core Problem With Ad-Hoc SFTP Scripts

One-off SFTP scripts typically fail in the same ways:

  • Silent failures — the transfer errors out, nothing is logged, the script exits 0
  • Corrupt partial downloads — the connection drops mid-file, the partial file looks like a complete file
  • No audit trail — you can't tell what transferred, when, or whether the file changed since
  • No retry logic — one network hiccup and the whole run fails

sftp-ultra addresses all of these explicitly.

Resumable Downloads — The .part Pattern

Every download lands as filename.part first. The file is only renamed to its final name after the transfer completes and (optionally) passes checksum verification. If the connection drops mid-transfer, you get a .part file — not a corrupt complete file that looks valid.

On the next run with --resume, the engine detects existing .part files, checks how much was already downloaded, and picks up from there. This matters for large files over unreliable connections.

SHA-256 Verification 

sftp-ultra pull \
  --target 192.168.1.105 \
  --username side \
  --remote-root /home/side \
  --destination /media/drive/downloads \
  --pattern "*.mp4" \
  --checksum sha256
Enter fullscreen mode Exit fullscreen mode

With --checksum sha256, the engine computes the SHA-256 of the remote file and the local file after transfer and compares them. Mismatch = transfer failed, file is removed, logged as failed, does not silently sit in your destination directory.

Without --checksum, size and mtime are still checked before and after download — if the remote file changed mid-transfer, the download is aborted.

The SQLite Transfer Journal

Every run appends to a SQLite journal:

Column What It Contains
remote_path Full remote path
local_path Local destination path
status copied, skipped, failed, planned
checksum SHA-256 hex digest (if enabled)
message Error message on failure
updated_at Timestamp

Query it directly:

sqlite3 .sftp-ultra.sqlite3 \
  "SELECT remote_path, status, checksum FROM transfer_journal;"
Enter fullscreen mode Exit fullscreen mode

Dry-run mode (--dry-run) also writes to the journal with planned status — so you can inspect what would have transferred before committing.

Concurrent Workers 

sftp-ultra pull --workers 4  # default
sftp-ultra pull --workers 16 # faster on high-bandwidth links
Enter fullscreen mode Exit fullscreen mode

Configurable worker pool up to 32. Each worker maintains its own SSH connection. On a gigabit LAN with large files, 8–16 workers makes a real difference.

Deterministic Exit Codes 

0   — all transfers succeeded
1   — one or more transfers failed
2   — configuration error (bad flags, missing required args)
130 — interrupted (Ctrl-C)
Enter fullscreen mode Exit fullscreen mode

This is what makes sftp-ultra scriptable in a pipeline. You can && it, check $?, or wrap it in a retry loop and know exactly what happened.

Quick Start 

git clone https://github.com/BleedingCodes/sftp-ultra.git
cd sftp-ultra/sftp-ultra
pip install -e .

sftp-ultra pull \
  --target 192.168.1.105 \
  --username side \
  --remote-root /home/side \
  --destination ./downloads \
  --pattern "*.mp4" \
  --workers 4 \
  --resume \
  --checksum sha256
Enter fullscreen mode Exit fullscreen mode

Python 3.11+, Linux. Requires paramiko. MIT license.

What It Doesn't Do

Key-based auth is not currently supported — it always prompts for a password. That's a real limitation if you need fully unattended operation without a password. It's on the list.

When to Use It

If you're moving files between machines on a schedule, running lab data collection off a remote node, or just tired of checking whether last night's transfer actually finished — this is the tool.

Repo: github.com/BleedingCodes/sftp-ultra

Built by MainbyteLabs — Python tooling for electronics labs, hardware shops, and Linux-based tech teams.
github.com/MR-MainbyteLabs

Top comments (0)