Technical deep-dive into the architecture decisions behind CaptureOCR

CaptureOCR is a Windows desktop application that transforms screenshots into structured data (Markdown tables, CSV) using local OCR. Built with Python, PySide6, and ONNX Runtime, it processes everything on-device with zero cloud dependencies.

Product link: CaptureOCR on Microsoft Store — $2.99 one-time, no subscription.

---

High-Level Architecture

┌─────────────────────────────────────────────────────────────┐
│                        UI Layer (PySide6)                   │
│  ┌─────────────┐  ┌─────────────┐  ┌─────────────────────┐ │
│  │ Full-screen │  │ Region      │  │ Canvas/Annotation   │ │
│  │ Selection   │  │ Drawer      │  │ Toolbar             │ │
│  └─────────────┘  └─────────────┘  └─────────────────────┘ │
└────────────────────────────┬────────────────────────────────┘
                             │
                             ▼
┌─────────────────────────────────────────────────────────────┐
│                  Application Services                       │
│  ┌─────────────────┐  ┌─────────────────────────────────┐  │
│  │ Capture Manager │  │ OCR Workflow Orchestrator       │  │
│  │ - mss grab      │  │ - Queue management              │  │
│  │ - DPI mapping   │  │ - Model tier switching          │  │
│  │ - Multi-monitor │  │ - Progress reporting            │  │
│  └─────────────────┘  └─────────────────────────────────┘  │
│                             │                               │
│                    ┌────────▼────────┐                      │
│                    │ Export Service  │                      │
│                    │ - Markdown/CSV  │                      │
│                    │ - Session (.snipx)│                    │
│                    └─────────────────┘                      │
└────────────────────────────┬────────────────────────────────┘
                             │
                             ▼
┌─────────────────────────────────────────────────────────────┐
│                   Infrastructure Layer                      │
│  ┌──────────────┐  ┌──────────────┐  ┌──────────────────┐  │
│  │ OCR Backend  │  │ Tiling       │  │ Scroll Capture   │  │
│  │ (RapidOCR +  │  │ Engine       │  │ (Row signatures) │  │
│  │  ONNX Runtime)│  │              │  │                  │  │
│  └──────────────┘  └──────────────┘  └──────────────────┘  │
└─────────────────────────────────────────────────────────────┘

---

Key Design Decisions

1. Single OCR Worker Thread

Decision: All model inference runs on a dedicated background thread.

Rationale:

• ONNX Runtime's InferenceSession is not guaranteed thread-safe
• Avoids complex locking around model loads/unloads
• For a desktop tool processing ~10 regions, serial throughput is sufficient
• Predictable memory profile (no concurrent model instances)

# Simplified worker pattern
class OcrWorker(QThread):
    result_ready = pyqtSignal(OcrResult)

    def __init__(self):
        super().__init__()
        self.queue = Queue()
        self.session = None
        self.current_model = None

    def run(self):
        while True:
            task = self.queue.get()
            if task is None:
                break
            result = self._process(task)
            self.result_ready.emit(result)

2. Two-Tier Model Strategy

Tier	Model	Size	Behavior
Fast	PP-OCRv6 tiny	6.6 MB	Bundled in MSIX, preloaded at startup, stays resident
Accurate	PP-OCRv6 small	29.8 MB	Downloaded on demand (SHA-256 verified), unloads after 5s idle

Implementation:

class ModelManager:
    def __init__(self):
        self.tiny_session = self._load_tiny()  # Always loaded
        self.small_session = None
        self.last_small_use = 0

    def get_session(self, tier: str):
        if tier == "fast":
            return self.tiny_session
        if self.small_session is None:
            self.small_session = self._download_and_load_small()
        self.last_small_use = time.time()
        return self.small_session

    def maybe_unload_small(self):
        if self.small_session and time.time() - self.last_small_use > 5:
            self.small_session = None
            gc.collect()

3. DPI-Aware Multi-Monitor Capture

Windows captures in physical pixels; Qt works in logical coordinates. On 125-150% scaling (and mixed-DPI setups), this mismatch causes blurry captures or offset selections.

Solution: Per-monitor ScreenInfo with both logical and physical rects + device pixel ratio.

@dataclass
class ScreenInfo:
    index: int
    logical_rect: QRect      # Qt coordinates
    physical_rect: QRect     # Screen pixels
    dpr: float               # Device pixel ratio

def logical_to_physical(screen: ScreenInfo, logical_rect: QRect) -> QRect:
    """Convert Qt logical rect to physical pixels for mss capture."""
    x = int((logical_rect.x() - screen.logical_rect.x()) * screen.dpr)
    y = int((logical_rect.y() - screen.logical_rect.y()) * screen.dpr)
    w = int(logical_rect.width() * screen.dpr)
    h = int(logical_rect.height() * screen.dpr)
    return QRect(x, y, w, h)

4. Tiled OCR for Long Images

Feeding a 20,000px tall screenshot into a 1600px OCR cap scales text to ~2px — unreadable. Instead, tile vertically:

Tile parameters:
- Target height: 1600px
- Overlap: 64px (≈2 lines at 1080p)
- Core bands: [0, 1536), [1536, 3072), ...
- Each tile extends 32px above/below core for context
- Line assigned to tile where box center Y falls in core band

This ensures every line is processed at native resolution, claimed exactly once.

5. Row-Signature Scroll Stitching

Template matching fails for scrolling (neither frame contains the other). Instead:

1. Convert frame to grayscale, ignore right 8% (scrollbars)
2. Compress each row to 32 column-bucket means → (H, 32) signature
3. For each candidate shift s, compute MAE between prev[s:] and cur[:H-s]
4. Accept shift only if best score < threshold AND clearly better than median
5. Sticky headers: retry skipping 8%/16%/25% of top rows
6. Adaptive wheel step: measure first scroll, target ~45% of selection height

---

Performance Characteristics

Metric	Tiny (Fast)	Small (Accurate)
Model size	6.61 MB	29.79 MB
Load time	~1.35s	~1.06s
Avg latency (23-img set)	78.7 ms	257.5 ms
P95 latency	144 ms	471 ms
Char accuracy (CER)	99.24%	99.69%

Test set: 23 mixed Chinese/English screenshots, 1600px cap, local CPU. Not a public benchmark — treat as order of magnitude.

---

Distribution

• Format: MSIX (runFullTrust), distributed via Microsoft Store
• Size: ~126 MB (includes tiny model, ONNX Runtime, Python runtime)
• Dependencies: Zero — fully self-contained
• Auto-updates: Handled by Store

---

Lessons Learned

1. Detection config matters more than model choice: Switching from limit_type="min" (upscales short side) to limit_type="max" (only downscale long side) was a bigger win than model tuning.

2. Disable classification for screenshots: use_cls=False — the 0°/180° classifier adds latency with no signal for upright captures.

3. DPI bugs are subtle: Logical/physical coordinate mismatch only appears on scaled displays. Test on 125%, 150%, and mixed-DPI multi-monitor.

4. Serialization format for reproducibility: The .snipx session package (ZIP with session.json + original PNGs + Markdown/CSV) enables lossless round-tripping and collaboration.

---

Resources

• GitHub: github.com/lijiannan1241/CaptureOCR — product page & issue tracker
• Microsoft Store: CaptureOCR
• Source availability: Application is closed-source; repo hosts product page and issue tracker only

---

Disclosure: I'm the developer of CaptureOCR. This is a paid product ($6.99 one-time). The application source code is not public — the repository serves as the product page and issue tracker.