How CaptureOCR stitches scrolling screenshots without template matching
Why Template Matching Fails for Scrolling
cv2.matchTemplate expects the template to be a sub-image of the target. When a page scrolls, consecutive frames overlap — but neither frame contains the other. You can crop a strip and search, but it gets fragile fast:
- Different content at top/bottom each frame
- Sticky headers break the overlap assumption
- Animation frames have partial content
Product link: CaptureOCR on Microsoft Store — $2.99 one-time, includes best-effort scrolling capture.
Row-Signature Approach
Instead of 2D template matching, compress each frame to a 1D row signature and search for vertical shift.
Step 1: Frame → Row Signature
import cv2
import numpy as np
def frame_to_signature(frame: np.ndarray, buckets: int = 32) -> np.ndarray:
"""
Convert frame to (H, buckets) signature.
- Ignore right 8% (scrollbars)
- Compress each row to bucket means
"""
h, w = frame.shape[:2]
# Ignore scrollbar region
usable_w = int(w * 0.92)
roi = frame[:, :usable_w]
# Grayscale
if len(roi.shape) == 3:
gray = cv2.cvtColor(roi, cv2.COLOR_BGR2GRAY)
else:
gray = roi
# Compress columns into buckets
bucket_width = usable_w // buckets
signature = np.zeros((h, buckets), dtype=np.float32)
for b in range(buckets):
start = b * bucket_width
end = (b + 1) * bucket_width if b < buckets - 1 else usable_w
signature[:, b] = gray[:, start:end].mean(axis=1)
return signature # Shape: (H, 32)
Step 2: Shift Search with MAE
def find_vertical_shift(prev_sig: np.ndarray, curr_sig: np.ndarray,
max_shift: int = None) -> tuple[int, float]:
"""
Find vertical shift between two frame signatures using MAE.
Returns (shift_pixels, best_score)
"""
h = prev_sig.shape[0]
if max_shift is None:
max_shift = h // 2 # Don't search more than half frame
best_shift = 0
best_score = float('inf')
scores = []
for s in range(max_shift + 1):
# Overlap region: prev[s:] vs curr[:h-s]
if h - s < 10: # Too little overlap
break
overlap_prev = prev_sig[s:]
overlap_curr = curr_sig[:h-s]
# Mean Absolute Error
mae = np.mean(np.abs(overlap_prev - overlap_curr))
scores.append(mae)
if mae < best_score:
best_score = mae
best_shift = s
# Peak detection: best must be clearly better than median
median_score = np.median(scores)
if best_score > median_score * 0.8: # Not a clear peak
return 0, float('inf') # Ambiguous
return best_shift, best_score
Step 3: Sticky Header Handling
A fixed nav bar stays at top while content moves → overlap never matches.
def find_shift_with_sticky_retry(prev_sig: np.ndarray, curr_sig: np.ndarray) -> tuple[int, float]:
"""
Retry with increasing top-skip percentages for sticky headers.
Skip ratios: 0% → 8% → 16% → 25%
"""
skip_ratios = [0.0, 0.08, 0.16, 0.25]
for skip_ratio in skip_ratios:
skip_rows = int(prev_sig.shape[0] * skip_ratio)
if skip_rows >= prev_sig.shape[0] // 2:
break
shifted_prev = prev_sig[skip_rows:]
shifted_curr = curr_sig[skip_rows:]
shift, score = find_vertical_shift(shifted_prev, shifted_curr)
if score != float('inf'):
return shift + skip_rows, score # Adjust for skipped rows
return 0, float('inf') # All retries failed
Step 4: Stitch Frames
def stitch_frames(frames: list[np.ndarray]) -> np.ndarray:
"""
Stitch a sequence of frames into one tall image.
"""
if not frames:
return np.array([])
if len(frames) == 1:
return frames[0]
# Start with first frame
canvas = frames[0].copy()
prev_sig = frame_to_signature(frames[0])
for i, curr_frame in enumerate(frames[1:], 1):
curr_sig = frame_to_signature(curr_frame)
# Find shift
shift, score = find_shift_with_sticky_retry(prev_sig, curr_sig)
if score == float('inf'):
# Fallback: assume single-page scroll step
shift = estimate_wheel_step(canvas.shape[0])
# Append non-overlapping portion
if shift < curr_frame.shape[0]:
new_content = curr_frame[shift:]
canvas = np.vstack([canvas, new_content])
prev_sig = frame_to_signature(canvas[-prev_sig.shape[0]:]) # Update with tail
# Hard caps
if canvas.shape[0] > 20000: # Max height
break
if i > 999: # Max frames
break
return canvas
Why 32 Buckets?
| Buckets | 9 Real Frame Pairs Correct |
|---|---|
| 12 | 7/9 |
| 24 | 8/9 |
| 32 | 9/9 |
| 64 | 9/9 |
With periodic text (line numbers 007, 008...), coarse buckets average away distinguishing columns. The shift search then locks onto multiples of line height, repeating sections. 32 is the cheapest configuration that passes.
Adaptive Wheel Step
One scroll notch = wildly different pixels in different apps (Qt text vs browser vs chat).
class AdaptiveWheelStep:
def __init__(self, selection_height: int):
self.selection_height = selection_height
self.target_overlap_ratio = 0.45 # Target ~45% overlap
self.current_notches = 1
self.last_measured_shift = None
def get_notches(self) -> int:
return max(1, min(12, self.current_notches))
def update(self, measured_shift: int):
"""Called after successful stitch with actual shift."""
if self.last_measured_shift is None:
self.last_measured_shift = measured_shift
# Next step: target 45% of selection height
target = int(self.selection_height * self.target_overlap_ratio)
self.current_notches = max(1, int(target / max(1, measured_shift)))
else:
# Gradual adaptation, cap 2x growth per step
target = int(self.selection_height * self.target_overlap_ratio)
ideal = max(1, target / max(1, measured_shift))
self.current_notches = min(self.current_notches * 2, int(ideal))
self.last_measured_shift = measured_shift
Failure Handling (Code-Based, Not String-Based)
class StitchResult:
SUCCESS = "success"
NO_CHANGE = "no_change" # Wheel didn't move content
NO_OVERLAP = "no_overlap" # Scrolled too far / mid-animation
AMBIGUOUS = "ambiguous" # Multiple equally good shifts
MAX_FRAMES = "max_frames" # Hit 999 frame cap
MAX_HEIGHT = "max_height" # Hit 20,000px cap
def handle_failure(reason: str, wheel_notches: int) -> tuple[bool, int]:
"""Returns (should_retry, new_notches)"""
if reason == StitchResult.NO_CHANGE:
# Wheel probably didn't reach target — resend same notches
return True, wheel_notches
elif reason in (StitchResult.NO_OVERLAP, StitchResult.AMBIGUOUS):
# Likely mid-animation or scrolled too far — halve and retry
return True, max(1, wheel_notches // 2)
return False, 0
Hard Caps
| Cap | Value | Reason |
|---|---|---|
| Max frames | 999 | Prevent infinite loops |
| Max canvas height | 20,000 px | 999 frames × 4K would OOM |
| Min canvas height | 20,000 px | Actually a minimum for "long enough" detection |
| Retry limit | 4 per frame | After 4 failures, give up with reason |
Limitations (Honest)
Scroll capture is best-effort v1.0. It fails on:
- Virtualized lists (React Virtual, Flutter ListView) — items not in DOM until scrolled into view
- Lazy-loaded images — content appears after scroll stops
- Custom scroll animations — non-standard scroll behavior
- Sticky sidebars — horizontal sticky elements break vertical-only signature
Region capture is the reliable fallback. The app tells you this upfront.
Performance
| Metric | Value |
|---|---|
| Signature computation | ~0.5ms/frame |
| Shift search (32 buckets, 1080p) | ~1ms/frame |
| Stitching 50 frames | ~50-100ms total |
| Memory (20k px canvas) | ~50MB |
Resources
- CaptureOCR: Microsoft Store
- OpenCV MAE: docs.opencv.org
Top comments (0)