PhotoCraft is a modern, open-source desktop image editor with an internal plugin engine built around WebAssembly. Wasm delivers rock-solid security, sandbox isolation, and cross-platform portability. But when you want to run modern neural networks locally, it hits hard physical boundaries.
Three limits make local AI in Wasm painful:
- No direct GPU access. No DirectML, no CoreML, no CUDA or Tensor Cores. Everything runs on the CPU through Wasmtime. Running a 150MB neural model like RMBG-1.4 or LaMa on the CPU causes severe UI freezes and takes 8–12 seconds per operation.
- Memory sandboxing. Loading multi-hundred-megabyte model weights into a browser-grade or sandboxed Wasm linear memory pool quickly hits execution ceilings.
- No native floating UI. Sandboxed plugins cannot render floating toolstrips, custom HUDs, or overlays above the host canvas window. They are trapped within the editor's designated sub-panels.
For heavy creative workflows — studio-grade background removal, generative inpainting, and seam-blended super-resolution — users were trapped between cloud subscriptions with privacy concerns or awkward manual file-export roundtrips.
I wanted a local, offline, zero-telemetry alternative.
1. Solution Overview: The Companion Pattern
Instead of forking or patching the upstream editor codebase, I designed a standalone native companion in Rust. It runs as an independent OS process, interacts with PhotoCraft via its existing JSON-RPC control socket (TCP port 18492), and exchanges pixel buffers through the system clipboard.
The companion is structured as a Cargo workspace with 6 crates, ~25,000 lines of Rust, and governed by strict engineering constraints: a Definition of Done, resource guards, automated architectural linters, and bilingual localization enforced by CI.
┌─────────────────────────────────────────────────────────────┐
│ PhotoCraft (Host Editor) │
│ Canvas UI (Wasm Sandbox) ───► TCP Socket Server (18492) │
└───────────────────────────┬─────────────────────▲───────────┘
│ JSON-RPC Control │
│ (Layer copy/paste) │
▼ │
┌─────────────────────────────────────────────────┴───────────┐
│ PhotoCraft Companion (Rust Native) │
│ │
│ ┌──────────────────────┐ ┌────────────────────────┐ │
│ │ overlay_ui │ │ photocraft_bridge │ │
│ │ (egui floating panel)│ │ (transactional locks) │ │
│ └──────────┬───────────┘ └───────────┬────────────┘ │
│ │ │ │
│ ▼ ▼ │
│ ┌──────────────────────────────────────────────────────┐ │
│ │ core_contracts (Zero-Core API) │ │
│ │ plugin.json · .pcpl packaging · i18n │ │
│ └──────────────────────────┬───────────────────────────┘ │
│ │ │
│ ▼ │
│ ┌──────────────────────────────────────────────────────┐ │
│ │ ai_engine (Local Inference) │ │
│ │ DirectML (GPU) ──► Transparent CPU Fallback │ │
│ │ Session LRU Cache · Single-Session Tiling Engine │ │
│ └──────────────────────────────────────────────────────┘ │
└──────────────────────────────┬──────────────────────────────┘
│ In-Memory Pixel Transport
▼
[ System OS Clipboard ]
Key Architectural Principle — Zero-Core Hardcoding:
No model filenames (.onnx), neural input shapes, or plugin IDs are ever hardcoded in the Rust core. Everything is declared in dynamic JSON manifests. The runtime builds the UI, schedules the inference graph, and issues canvas commands purely based on the declared contracts.
2. Workspace Architecture (6 Crates)
The workspace is strictly partitioned by domain boundaries:
-
core_contracts— Pure domain types: plugin manifest specifications, marketplace registry,.pcplpackaging format, and i18n abstractions. Zero workspace dependencies. -
photocraft_bridge— Asynchronous JSON-RPC client (TCP port 18492) and clipboard interchange bridge with automatic reconnection. -
ai_engine— ONNX Runtime (ort) inference engine, DirectML hardware acceleration with transparent CPU fallback, and single-session tiling pipeline. -
overlay_ui— Always-on-topegui/eframefloating toolstrip, marketplace dialog, auto-updater, and dynamic widget generator. -
companion_app— Main executable binary, system tray integration (tray-icon), and runtime orchestration. -
plugin_validator— Standalone CLI utility for validating manifest schemas, path safety, and localization completeness.
The dependency flow is strictly unidirectional: core_contracts sits at the root with zero internal deps, every other crate depends on it, and companion_app binds them together into a single, cohesive binary.
3. The JSON-RPC Bridge: Taming Unreliable Peers
The bridge is the most delicate component because it crosses an inter-process boundary with an asynchronous peer. The editor might hitch during heavy operations, show a blocking modal dialog, or abruptly close. The companion must never hang.
Transactional Request Locking
The client uses Tokio's asynchronous TCP halves. Initially, writer and reader had independent mutexes. However, concurrent calls interleaved: Call A wrote, Call B wrote, Call A read — and received Call B's response!
The fix was a transactional lock that serializes the full request-response lifecycle:
pub async fn call(&self, method: &str, params: Value) -> Result<Value> {
// Acquire a single lock that spans both write and read
let _tx = self.call_lock.lock().await;
let id = self.next_id.fetch_add(1, Ordering::SeqCst);
let req = JsonRpcRequest {
id,
method: method.to_string(),
params,
};
let mut writer_guard = self.writer.lock().await;
let writer = writer_guard.as_mut().ok_or(CompanionError::Bridge("Disconnected".into()))?;
writer.write_all(req.serialize()?.as_bytes()).await?;
let mut reader_guard = self.reader.lock().await;
let reader = reader_guard.as_mut().ok_or(CompanionError::Bridge("Disconnected".into()))?;
// Read with a strict deadline; drop handles on timeout to reset state
let resp = tokio::time::timeout(Duration::from_secs(30), reader.read_response(id)).await??;
assert_eq!(resp.id, id, "Mismatched JSON-RPC sequence ID");
Ok(resp.result)
}
Contention Gotcha: The Color Poller
When a plugin panel is open, a background task polls the active brush color every 180ms to keep overlay palettes in sync. Because it shares the bridge client, it created constant contention for the socket lock.
The solution: background polling uses a non-blocking try_lock(). If the bridge is busy running an inference command or layer operation, the poller silently skips that tick rather than queuing behind long-running operations.
4. The egui Floating Overlay: Vector-First UI
The companion displays a two-tier floating toolstrip rendered with egui and eframe. It hovers at persistent screen coordinates above the canvas without creating clutter in the OS taskbar.
Zero-Tofu Policy (No Emoji in Text)
Default fonts in egui lack full Unicode Noto/Emoji coverage. Using emojis like ⏳, ⚠️, or ⚙ produces ugly "tofu" squares (□).
The overlay enforces a strict Vector-First policy:
- Status dots and indicators are drawn directly via
ui.painter().circle_filled(center, radius, color). - UI icons (gear, orientation flip, grab handle, close) are drawn with vector primitives (
line_segment,rect_stroke) or cached SVGs viaresvg. - Loading states use active pulsing rings rather than spinner emojis.
Dynamic Widget Generation
The overlay does not hardcode plugin forms. It reads the plugin.json manifest and instantiates sliders, dropdowns, and toggles on the fly:
// overlay_ui renders widgets dynamically from manifest schema
for widget in &manifest.ui.widgets {
match widget.kind {
WidgetKind::Slider { min, max, step } => {
ui.add(
egui::Slider::new(&mut state.get_mut(&widget.id), *min..=*max)
.step_by(*step as f64)
.text(widget.label.localize(&self.locale))
);
}
WidgetKind::Dropdown { options } => {
egui::ComboBox::from_label(widget.label.localize(&self.locale))
.selected_text(&state.get_string(&widget.id))
.show_ui(ui, |ui| {
for opt in options {
ui.selectable_value(&mut state.get_mut(&widget.id), opt.value.clone(), &opt.label);
}
});
}
}
}
A third-party creator can introduce a new image processing filter with four sliders and two dropdowns entirely in JSON. The companion renders it immediately without recompiling Rust.
5. ONNX Runtime & DirectML: Real-World Performance
The AI engine uses ort (Rust bindings for ONNX Runtime C++ backend) with DirectML for GPU acceleration on Windows, falling back seamlessly to CPU on non-compatible hardware.
Benchmark: Wasm vs Companion
To see why the companion pattern is worth the architectural complexity, consider RMBG-1.4 inference on a 1024×1024 image:
| Execution Target | Runtime / Hardware | Latency (1024×1024) | System Impact |
|---|---|---|---|
| Wasm Sandboxed (Wasmtime) | CPU (Multi-core) | ~8,200 – 12,000 ms | 100% CPU lockup, UI dropped frames |
| Companion (CPU Fallback) | Native C++ ONNX | ~1,850 ms | Background worker, 60fps overlay UI |
| Companion (DirectML Cold) | Discrete GPU (RTX / RX) | ~380 ms | 0% host CPU load, 25× speedup |
| Companion (DirectML Warm) | LRU Cached Session | ~210 ms | Zero initialization overhead |
Session Caching
Allocating an ONNX execution session and compiling model graphs to DirectML on every user action introduces 1–2 seconds of latency. The companion maintains an LRU session cache:
pub struct SessionCache {
entries: HashMap<PathBuf, (Arc<Session>, Instant)>,
max_capacity: usize,
}
Because Session::run takes &self in modern ort, an Arc<Session> can be safely shared across tasks and reused across sequential operations.
High-Resolution Tiling Pipeline
When users process 4K or 8K layers, running the full canvas through a model in one pass overflows VRAM. The engine slices the canvas into overlapping tiles, executes inference tile-by-tile within a single cached session, and blends edges using cosine feathering to avoid seam artifacts.
6. The Zero-Core Plugin System & Security
Plugins are distributed as .pcpl files — ZIP archives containing a manifest, optional scripts, icons, and model weights.
Manifest Contract (plugin.json)
{
"id": "remove_bg",
"name": { "en": "Background Removal", "ru": "Удаление фона" },
"version": "1.3.0",
"type": "native_ai",
"input": "active_layer",
"output": "new_layer",
"models": [
{
"role": "segmentation_mask",
"path": "models/rmbg-1.4.onnx",
"sha256": "5c1f0d..."
}
],
"commands": ["image.revealAll"],
"post_commands": ["layer.renameActive"]
}
Zip-Slip Mitigation
Unpacking community archives is fraught with danger. A malicious archive containing filenames like ../../appdata/startup.exe can overwrite arbitrary system files.
The archive extractor enforces three layers of validation:
for i in 0..archive.len() {
let mut file = archive.by_index(i)?;
// 1. Safe extraction path verification
let enclosed = file.enclosed_name()
.ok_or_else(|| CompanionError::Security("Illegal path in archive".into()))?;
// 2. Reject absolute paths and parent traversals
if enclosed.components().any(|c| matches!(c, Component::ParentDir | Component::RootDir)) {
return Err(CompanionError::Security("Path traversal attempt detected".into()));
}
// 3. Reject symlinks
if file.is_symlink() {
return Err(CompanionError::Security("Symlinks forbidden in plugin packages".into()));
}
let target = target_dir.join(enclosed);
// ... write file safely
}
7. Engineering Discipline & Automated Governance
When pairing with LLMs on an ambitious codebase, the primary danger is "architecture rot" — subtle hacks, keyword-sniffing shortcuts, and unprincipled workarounds.
To prevent this, the repository uses automated governance tests:
-
Resource Guard (
-j 1): All builds, tests, and CI jobs run with restricted thread counts (-j 1/codegen-units = 1) to ensure modest developer machines remain fully responsive. -
Architecture Integrity Linter (
architecture_integrity_test.rs): An automated test scans all crate source files. If anyone adds.contains("remove_bg"),.contains("depth"), or hardcoded model paths to the core runtime, CI fails immediately. -
Localization Linter (
i18n_lint_test.rs): Scans the UI crates for raw Cyrillic or hardcoded strings. Every user-facing message must use.localize(&locale)or.t("En", "Ru")with non-empty translation pairs. -
Standalone Package Validator (
plugin_validator): Validates the entire plugin directory and generated.pcplpackages before every commit.
8. Lessons Learned
- The Companion Pattern unlocks legacy and sandboxed runtimes. You don't need to rewrite a massive web or Wasm application to give it native GPU capabilities. A lightweight native companion over JSON-RPC + OS clipboard gives you the best of both worlds: a secure sandboxed editor and native hardware inference.
- Declarative architecture pays exponential dividends. Investing early in a dynamic UI schema and Zero-Core contract felt like over-engineering for the first plugin. But by plugin three, adding new neural tools required zero Rust coding and zero app rebuilds.
- Concurrency requires transactional discipline. Independent mutexes on socket read/write halves invite race conditions the moment background polling starts. Treat socket interactions as atomic transactions.
- LLMs need architectural guardrails. LLMs are phenomenal at writing boilerplate, refactoring algorithms, and generating test suites. But without automated linters enforcing architectural purity, codebases naturally drift toward expedient, messy shortcuts.
Repository & Open Source
PhotoCraft Companion is open source and dual-licensed under MIT / Apache-2.0:
👉 github.com/kobaltgit/photocraft-companion
Contributions, architectural discussions, and new plugin manifests are welcome!
Top comments (0)