DEV Community: Michael Rivera

Broken SFTP file Transfers Enraged me to Build a Proper Transfer Engine

Michael Rivera — Thu, 17 Sep 2026 13:36:44 +0000

I Got Tired of Broken SFTP Scripts So I Built a Proper Transfer Engine

Every few months I'd find a half-transferred file, a corrupted download that looked complete, or a transfer that silently failed and left nothing in the log. The scripts people write for SFTP — including mine — are usually fine until they're not.

sftp-ultra is what I built to replace them. A production SFTP transfer engine with concurrent workers, resumable downloads, SHA-256 verification, a SQLite transfer journal, and deterministic exit codes.

Here's what it actually does and why each piece is there.

The Core Problem With Ad-Hoc SFTP Scripts

One-off SFTP scripts typically fail in the same ways:

Silent failures — the transfer errors out, nothing is logged, the script exits 0
Corrupt partial downloads — the connection drops mid-file, the partial file looks like a complete file
No audit trail — you can't tell what transferred, when, or whether the file changed since
No retry logic — one network hiccup and the whole run fails

sftp-ultra addresses all of these explicitly.

Resumable Downloads — The `.part` Pattern

Every download lands as filename.part first. The file is only renamed to its final name after the transfer completes and (optionally) passes checksum verification. If the connection drops mid-transfer, you get a .part file — not a corrupt complete file that looks valid.

On the next run with --resume, the engine detects existing .part files, checks how much was already downloaded, and picks up from there. This matters for large files over unreliable connections.

SHA-256 Verification

sftp-ultra pull \
  --target 192.168.1.105 \
  --username side \
  --remote-root /home/side \
  --destination /media/drive/downloads \
  --pattern "*.mp4" \
  --checksum sha256

With --checksum sha256, the engine computes the SHA-256 of the remote file and the local file after transfer and compares them. Mismatch = transfer failed, file is removed, logged as failed, does not silently sit in your destination directory.

Without --checksum, size and mtime are still checked before and after download — if the remote file changed mid-transfer, the download is aborted.

The SQLite Transfer Journal

Every run appends to a SQLite journal:

Column	What It Contains
`remote_path`	Full remote path
`local_path`	Local destination path
`status`	`copied`, `skipped`, `failed`, `planned`
`checksum`	SHA-256 hex digest (if enabled)
`message`	Error message on failure
`updated_at`	Timestamp

Query it directly:

sqlite3 .sftp-ultra.sqlite3 \
  "SELECT remote_path, status, checksum FROM transfer_journal;"

Dry-run mode (--dry-run) also writes to the journal with planned status — so you can inspect what would have transferred before committing.

Concurrent Workers

sftp-ultra pull --workers 4  # default
sftp-ultra pull --workers 16 # faster on high-bandwidth links

Configurable worker pool up to 32. Each worker maintains its own SSH connection. On a gigabit LAN with large files, 8–16 workers makes a real difference.

Deterministic Exit Codes

0   — all transfers succeeded
1   — one or more transfers failed
2   — configuration error (bad flags, missing required args)
130 — interrupted (Ctrl-C)

This is what makes sftp-ultra scriptable in a pipeline. You can && it, check $?, or wrap it in a retry loop and know exactly what happened.

Quick Start

git clone https://github.com/BleedingCodes/sftp-ultra.git
cd sftp-ultra/sftp-ultra
pip install -e .

sftp-ultra pull \
  --target 192.168.1.105 \
  --username side \
  --remote-root /home/side \
  --destination ./downloads \
  --pattern "*.mp4" \
  --workers 4 \
  --resume \
  --checksum sha256

Python 3.11+, Linux. Requires paramiko. MIT license.

What It Doesn't Do

Key-based auth is not currently supported — it always prompts for a password. That's a real limitation if you need fully unattended operation without a password. It's on the list.

When to Use It

If you're moving files between machines on a schedule, running lab data collection off a remote node, or just tired of checking whether last night's transfer actually finished — this is the tool.

Repo: github.com/BleedingCodes/sftp-ultra

Built by MainbyteLabs — Python tooling for electronics labs, hardware shops, and Linux-based tech teams.
github.com/MR-MainbyteLabs

CPU path tracer in Pure Python — No GPU, No Dependencies, Physically Based

Michael Rivera — Thu, 17 Sep 2026 13:32:01 +0000

I Wrote a Path Tracer in Pure Python — No GPU, No Dependencies, Physically Based

PureTrace is a CPU path tracer written entirely in Python's standard library. No NumPy, no Pillow, no native extensions, no GPU. Everything — geometry, BVH construction, PBR materials, PNG encoding, OpenEXR encoding, multiprocessing, and checkpointing — is standard library.

This post is about the engineering decisions behind it and what it was like to build a production-quality rendering pipeline under that constraint.

Why the Constraint?

"No dependencies" sounds like a gimmick. It's not.

The real goal was to understand what rendering pipelines actually do — not what they look like through a library's API. When you can't import NumPy, you write your own vector math. When you can't use Pillow, you write your own PNG encoder. You end up understanding every piece you would otherwise have taken for granted.

The secondary benefit: zero install friction. Clone and render. No pip install chain, no native extension compilation, no version conflicts.

The Rendering Pipeline

Scene primitives → SAH BVH → Camera samples → Path tracing integrator → MIS → Tile workers → Merge → PNG / OpenEXR

Monte Carlo path tracing with multiple-importance sampling (MIS). Each pixel fires rays into the scene, bouncing off surfaces until they hit a light or exceed max depth. MIS combines direct light sampling and BSDF sampling using the power heuristic — this is what makes glass and metal converge without the firefly artifacts you get from naive path tracing.

Physically based materials (GGX). Diffuse, metallic, and dielectric glass — all using the GGX microfacet distribution. Roughness controls the shape of specular highlights and reflections in a physically correct way.

Volumetric rendering. Homogeneous participating medium with free-flight sampling. Fog, anisotropic scattering, visible light transport.

The BVH

Ray-scene intersection is the performance bottleneck in any path tracer. Without a spatial acceleration structure, every ray tests every primitive — O(n) per ray, which makes complex scenes completely intractable.

PureTrace uses a binned surface-area-heuristic BVH (bounding volume hierarchy). The SAH is a cost model that estimates how expensive a split will be based on the surface areas of the resulting child nodes. Binning approximates the optimal split point efficiently rather than testing every possible position.

The result: ray traversal is O(log n) in the average case. A scene with thousands of triangles becomes tractable.

Writing the BVH in pure Python — no SIMD, no Cython — is where the constraint hurts most. It's real. PureTrace is not fast. Use all your cores, use sensible image sizes, and use progressive sampling. But it's correct and it's readable.

Multiprocessing and Checkpointing

The render is split into tiles. Each tile runs in a worker process (Python's multiprocessing). The main process merges tiles as they complete and writes progressive output so you can see the image forming.

Every render writes an atomic .ptrchk checkpoint file. Interrupt mid-render with Ctrl-C and the completed tiles are saved. Resume with the same settings:

# Start
puretrace render glass-chess -W 900 -H 675 -s 1200 -j 8 --samples-per-pass 4 -o chess.png

# Interrupt, resume later
puretrace render glass-chess -W 900 -H 675 -s 1200 -j 8 --samples-per-pass 4 -o chess.png --resume

Glass caustics at 1200 samples per pixel takes a long time on a CPU. Checkpointing makes that practical.

Output Formats

PNG — 8-bit sRGB with ACES, Reinhard, or linear tone mapping. The encoder is hand-written: deflate compression via zlib (stdlib), PNG chunk structure by hand.
OpenEXR — uncompressed scanline RGB in linear scene space, half-float. Also hand-written. No openexr package.

Writing a PNG encoder is one of those things you do once and never forget. The format is simple enough to implement from the spec in an afternoon.

Built-in Scenes

puretrace render cornell -W 512 -H 512 -s 256 -j 8 -o cornell.png
puretrace render spheres -W 640 -H 400 -s 128 -o spheres.png
puretrace render glass-chess -W 900 -H 675 -s 1200 -j 8 -o chess.png
puretrace render fog -W 800 -H 600 -s 512 -j 8 -o fog.png

Scene	What It Tests
`cornell`	Diffuse walls, rough metal, glass, soft ceiling light
`spheres`	Chrome, copper, glass, depth of field, motion blur
`glass-chess`	Lathed glass pieces on reflective checkerboard
`caustics`	Glass and polished metal under compact area light
`fog`	Anisotropic participating media, visible light transport

A first render worth looking at is 256 samples per pixel. Glass caustics converge slowly — 1000+ is normal.

JSON Scene Description

{
  "camera": {
    "look_from": [4, 2, 6],
    "look_at": [0, 1, 0],
    "vertical_fov": 40,
    "aperture": 0.05
  },
  "materials": {
    "glass": {
      "base_color": [0.95, 0.99, 1.0],
      "roughness": 0.01,
      "transmission": 1.0,
      "ior": 1.52
    }
  },
  "objects": [
    { "type": "obj", "file": "room.obj", "scale": 1.0 }
  ]
}

OBJ/MTL import, HDR environment lighting, all material parameters exposed. Enough to build real scenes.

Running It

git clone https://github.com/BleedingCodes/PureTrace.git
cd PureTrace
pip install -e .
puretrace render cornell -W 512 -H 512 -s 256 -j 8 -o cornell.png

Python 3.11+. Zero pip dependencies. MIT license.

What This Project Is For

PureTrace is a deliberately readable renderer. Every algorithm is in plain Python, every decision is traceable to the source. If you want to understand how path tracers work — how MIS reduces noise, how BVH traversal works, how closures get closed upvalues — this is a codebase you can actually read.

It is not fast. It is correct, deterministic, and hackable.

Repo: github.com/BleedingCodes/PureTrace

Built by MainbyteLabs — Python tooling for electronics labs, hardware shops, and Linux-based tech teams.

github.com/MR-MainbyteLabs

Explored a new Programming Language in Pure Python — Lexer, Pratt Parser, Bytecode Compiler, Stack VM

Michael Rivera — Thu, 17 Sep 2026 13:29:17 +0000

I Built a Programming Language in Pure Python — Lexer, Pratt Parser, Bytecode Compiler, Stack VM

A few months ago I decided to tickle at making a programming language, have AI's help... implementation and English translation.

The result is lumen-lang — a small sorta complete programming language in Pure Python, no dependencies, no parser generators, no shortcuts. It has its own syntax, compiles to bytecode, and runs on a stack-based virtual machine I wrote from scratch.

Here's what's inside and why I made the decisions I did.

The Pipeline

Source text → Lexer → Token stream → Pratt Parser → AST → Compiler → Bytecode → VM → Output

Every stage is hand-written. That was deliberate — the goal was to understand what's actually happening at each step, not to produce something in the least amount of code.

The Lexer

The scanner is hand-written with line and column tracking. Nothing exotic — you walk the source character by character, emit tokens, handle string escapes and number literals, and report clean errors with position info.

The part people skip: good error messages at the lexer level save enormous debugging pain later. A bare unexpected character with no position is useless. [line 12, col 7] unexpected character: '@' is not.

The Parser — Why Pratt?

The parser uses recursive descent with explicit precedence-climbing methods. Each method handles one precedence level and calls the next:

_equality → _comparison → _term → _factor → _unary → _primary

This is sometimes called a Pratt parser (or top-down operator precedence parsing). The key advantage: adding a new operator or precedence level means adding one method and wiring it into the chain. It doesn't require touching a grammar table or regenerating anything.

It's also readable. You can look at _term and immediately see that it handles + and -. No indirection.

The AST

Typed node hierarchy covering all language constructs. Every node is a dataclass — no dict-based AST, no stringly-typed anything. The compiler walks the AST with a visitor pattern and emits instructions for each node type.

The Bytecode Compiler

The compiler tree-walks the AST and emits a compact instruction set into a Chunk — a flat array of opcodes and operands. Constants live in a constant pool indexed by operand.

You can inspect the compiled output for any program:

lumen --disassemble examples/demo.lm

Output for a counter closure:

== make_counter ==
0000 GET_LOCAL          1
0001 CLOSURE            0  <fn next>
0002 GET_LOCAL          3
0003 RETURN

== next ==
0000 GET_UPVALUE        0
0001 CONSTANT           0
0002 ADD
0003 SET_UPVALUE        0
0004 POP
0005 GET_UPVALUE        0
0006 RETURN

The disassembler is one of the most useful parts of this project. When something in the VM behaves wrong, you can look at exactly what the compiler emitted and trace the problem to the right stage.

The VM — Closures and Upvalues

The VM is stack-based. Each function call pushes a call frame with its own stack window and instruction pointer.

The interesting part is upvalue capture — how closures capture variables from enclosing scopes.

When the compiler sees a function reference a variable from an outer scope, it emits GET_UPVALUE / SET_UPVALUE instructions instead of GET_LOCAL. The VM maintains an upvalue list per closure object. While the enclosing function is still on the stack, upvalues point directly into the stack. When the enclosing function returns, open upvalues are "closed" — their value is copied out of the stack into the upvalue object itself.

This is the mechanism that makes this work correctly:

fn make_counter(start) {
    let value = start;
    fn next() {
        value = value + 1;
        return value;
    }
    return next;
}

let counter = make_counter(10);
print(counter());   // 11
print(counter());   // 12

next captures value from make_counter's scope. After make_counter returns, value no longer exists on the stack — but counter still has a valid reference to it through the closed upvalue.

Getting this right took the most time of any part of the project.

What the Language Looks Like

// Fibonacci
fn fib(n) {
    if (n < 2) { return n; }
    return fib(n - 1) + fib(n - 2);
}
print(fib(10));   // 55

// Lists and dictionaries
let primes = [2, 3, 5, 7, 11];
print(primes[2]);   // 5

let person = {"name": "Ada", "field": "computing"};
print(person["name"]);   // Ada

Dynamically typed, expression-oriented, C-ish syntax. Closures, recursion, lists, dicts, builtins (print, len, type, clock).

Running It

git clone https://github.com/BleedingCodes/lumen-lang.git
cd lumen-lang
python -m lumen                          # REPL
python -m lumen examples/demo.lm        # Run a file
python -m lumen --disassemble examples/demo.lm   # Inspect bytecode

No dependencies. Python 3.11+. MIT license.

Repo: github.com/BleedingCodes/lumen-lang

Built by MainbyteLabs — Python tooling for electronics labs, hardware shops, and Linux-based tech teams.
github.com/MR-MainbyteLabs

DEV Community: Michael Rivera

Broken SFTP file Transfers Enraged me to Build a Proper Transfer Engine

I Got Tired of Broken SFTP Scripts So I Built a Proper Transfer Engine

The Core Problem With Ad-Hoc SFTP Scripts

Resumable Downloads — The .part Pattern

SHA-256 Verification

The SQLite Transfer Journal

Concurrent Workers

Deterministic Exit Codes

Quick Start

What It Doesn't Do

When to Use It

CPU path tracer in Pure Python — No GPU, No Dependencies, Physically Based

I Wrote a Path Tracer in Pure Python — No GPU, No Dependencies, Physically Based

Why the Constraint?

The Rendering Pipeline

The BVH

Multiprocessing and Checkpointing

Output Formats

Built-in Scenes

JSON Scene Description

Running It

What This Project Is For

Explored a new Programming Language in Pure Python — Lexer, Pratt Parser, Bytecode Compiler, Stack VM

The Pipeline

The Lexer

The Parser — Why Pratt?

The AST

The Bytecode Compiler

The VM — Closures and Upvalues

What the Language Looks Like

Running It

Resumable Downloads — The `.part` Pattern