How to Control Any Legacy Desktop App from n8n with a Type-Safe Python Vision Endpoint

Legacy desktop applications—custom ERPs, Delphi clients, SAP GUI, Citrix windows, and air-gapped financial systems—remain the bane of modern workflow orchestration. They lack REST endpoints, expose zero webhook triggers, and fail to expose accessibility tree nodes for conventional UI automation.

Traditionally, dev teams face two flawed paths:

1. Enterprise RPA suites (UiPath, Automation Anywhere, Microsoft Power Automate Desktop): Costs scale into five figures annually, require dedicated Windows Server instances, and depend on proprietary orchestrators disconnected from modern CI/CD or n8n topologies.
2. Brittle native scripts: Unsynchronized pyautogui scripts running on infinite loops with static time.sleep() calls, collapsing at the first shifted pixel or dynamic modal dialogue.

This guide demonstrates an alternative: a local, type-safe Python FastAPI daemon that acts as an OS-level microservice, controllable directly via standard n8n workflows using screen capture, coordinate-aware OCR, and deterministic vision targeting.

---

1. The Bottleneck: The Fragility of Native Selectors

Commercial RPA platforms rely heavily on accessibility frameworks (Microsoft UI Automation, MSAA, IAccessible2). However, legacy Win32, Java Swing, Qt, and WPF apps often emit unpopulated element trees:

[Desktop]
  └─ [Window: LegacyBilling.exe]
       └─ [Pane: CustomCanvas] <-- Zero child nodes; totally opaque to DOM/UI inspection

When accessibility selectors fail, commercial tools fall back to basic coordinate clicks or image-template matching (cv2.matchTemplate), which fails under minor antialiasing shifts, dynamic theme updates, or resolution changes.

The Direct Vision-to-Action Protocol

By treating the desktop screen as an ephemeral API, we establish a stateless loop:

1. Inspect: Frame-buffer capture via native C-level screen grabbers (mss).
2. Extract & Locate: Type-safe OCR parsing with coordinate bounding boxes normalized to DPI.
3. Assert: Pydantic schema validation ensuring required text or anchors exist before execution.
4. Dispatch: Hardware-level cursor/keyboard dispatch.
5. Acknowledge: HTTP 200 payload returned to n8n with extracted data or execution state.

---

2. The Architecture

The automation stack decouples orchestration from execution:

┌─────────────────┐       HTTP / JSON        ┌──────────────────────────────┐
│   n8n Server    │  ────────────────────>   │   FastAPI Local Agent (OS)   │
│ (Cloud/Self-Host│  <────────────────────   │   (Listening on 127.0.0.1)   │
└─────────────────┘      Structured Data     └──────────────┬───────────────┘
                                                            │
                                    ┌───────────────────────┴───────────────────────┐
                                    ▼                                               ▼
                         ┌─────────────────────┐                         ┌─────────────────────┐
                         │  OCR / Vision Core  │                         │ PyAutoGUI / ctypes  │
                         │ (EasyOCR/Tesseract) │                         │ (Input Dispatch)    │
                         └─────────────────────┘                         └─────────────────────┘

• n8n Orchestrator: Handles scheduling, error retries, CRM integrations (PostgreSQL, HubSpot, Slack), and flow logic.
• FastAPI Daemon: Runs directly on the target host (Windows/macOS/Linux), exposing type-safe endpoints secured with an internal Bearer token.
• Vision Engine: Runs localized OCR with bounding boxes returned as normalized (x_min, y_min, x_max, y_max) values.

---

3. The Code & Logic

3.1 The FastAPI Agent

Below is the complete implementation of the core endpoint daemon handling coordinate targeting and OCR extraction.

# endpoint.py
import os
from typing import Optional, List
import mss
import numpy as np
import cv2
import pytesseract
import pyautogui
from fastapi import FastAPI, HTTPException, Security, status
from fastapi.security import HTTPBearer, HTTPAuthorizationCredentials
from pydantic import BaseModel, Field

pyautogui.FAILSAFE = True  # Slam cursor to corner to abort

app = FastAPI(title="Desktop Vision Endpoint", version="1.0.0")
security = HTTPBearer()
AUTH_TOKEN = os.getenv("AGENT_AUTH_TOKEN", "dev_secret_token_change_me")

def verify_token(credentials: HTTPAuthorizationCredentials = Security(security)):
    if credentials.credentials != AUTH_TOKEN:
        raise HTTPException(
            status_code=status.HTTP_401_UNAUTHORIZED,
            detail="Invalid automation bearer token"
        )
    return credentials.credentials

class Coordinate(BaseModel):
    x: int
    y: int

class OCRMatch(BaseModel):
    text: str
    confidence: float
    center: Coordinate
    bounding_box: List[int] = Field(..., description="[x, y, w, h]")

class ClickRequest(BaseModel):
    target_text: Optional[str] = None
    fallback_coords: Optional[Coordinate] = None
    double_click: bool = False
    offset_x: int = 0
    offset_y: int = 0

class TypeRequest(BaseModel):
    text: str
    clear_field_first: bool = True
    submit_enter: bool = False

def capture_screen_matrix() -> np.ndarray:
    with mss.mss() as sct:
        monitor = sct.monitors[1]  # Primary monitor
        screenshot = np.array(sct.grab(monitor))
        # Convert BGRA to RGB
        return cv2.cvtColor(screenshot, cv2.COLOR_BGRA2RGB)

@app.post("/ocr/scan", response_model=List[OCRMatch], dependencies=[Security(verify_token)])
def scan_screen(contains: Optional[str] = None):
    image = capture_screen_matrix()
    data = pytesseract.image_to_data(image, output_type=pytesseract.Output.DICT)

    matches: List[OCRMatch] = []
    n_boxes = len(data["text"])

    for i in range(n_boxes):
        text = data["text"][i].strip()
        conf = float(data["conf"][i])

        if conf > 40.0 and len(text) > 0:
            if contains and contains.lower() not in text.lower():
                continue

            x, y, w, h = data["left"][i], data["top"][i], data["width"][i], data["height"][i]
            matches.append(OCRMatch(
                text=text,
                confidence=conf,
                center=Coordinate(x=x + w // 2, y=y + h // 2),
                bounding_box=[x, y, w, h]
            ))

    return matches

@app.post("/action/click", dependencies=[Security(verify_token)])
def click_element(req: ClickRequest):
    target_x, target_y = None, None

    if req.target_text:
        matches = scan_screen(contains=req.target_text)
        if matches:
            # Pick the highest confidence match
            best = max(matches, key=lambda m: m.confidence)
            target_x = best.center.x + req.offset_x
            target_y = best.center.y + req.offset_y

    if target_x is None and req.fallback_coords:
        target_x = req.fallback_coords.x + req.offset_x
        target_y = req.fallback_coords.y + req.offset_y

    if target_x is None:
        raise HTTPException(status_code=404, detail="Target not located on display")

    pyautogui.moveTo(target_x, target_y, duration=0.2)
    if req.double_click:
        pyautogui.doubleClick()
    else:
        pyautogui.click()

    return {"status": "success", "clicked_at": {"x": target_x, "y": target_y}}

@app.post("/action/type", dependencies=[Security(verify_token)])
def write_text(req: TypeRequest):
    if req.clear_field_first:
        pyautogui.hotkey("ctrl", "a")
        pyautogui.press("backspace")

    pyautogui.write(req.text, interval=0.03)

    if req.submit_enter:
        pyautogui.press("enter")

    return {"status": "success", "typed_chars": len(req.text)}

if __name__ == "__main__":
    import uvicorn
    uvicorn.run(app, host="127.0.0.1", port=8765)

3.2 The n8n Integration Loop

In n8n, interact with the local agent using standard HTTP Request nodes over local loopback (or via a secure Tailscale mesh if n8n is hosted remotely).

Here is a raw n8n HTTP Request node configuration targeting the /action/click endpoint:

{
  "parameters": {
    "method": "POST",
    "url": "http://127.0.0.1:8765/action/click",
    "authentication": "genericCredentialType",
    "genericAuthType": "httpHeaderAuth",
    "sendHeaders": true,
    "headerParameters": {
      "parameters": [
        {
          "name": "Authorization",
          "value": "Bearer dev_secret_token_change_me"
        }
      ]
    },
    "sendBody": true,
    "contentType": "raw",
    "rawContentType": "application/json",
    "body": "={\n  \"target_text\": \"Submit Order\",\n  \"offset_x\": 0,\n  \"offset_y\": 0,\n  \"double_click\": false\n}",
    "options": {
      "timeout": 5000
    }
  },
  "name": "Click 'Submit Order'",
  "type": "n8n-nodes-base.httpRequest",
  "typeVersion": 4.2,
  "position": [460, 240]
}

---

4. Deployment, DPI Scaling & Performance

When orchestrating OS interactions from a server pipeline, three production factors determine stability:

1. High-DPI Display Scaling

On Windows systems with 125% or 150% UI scale (common on laptops and 4K workstations), OpenCV and PyAutoGUI coordinates may drift if your Python process is not marked DPI-aware. Fix this at initialization in your runner:

import sys
if sys.platform == "win32":
    import ctypes
    ctypes.windll.shcore.SetProcessDpiAwareness(2) # Per-monitor DPI aware

2. Tailscale & Remote n8n Instances

If your n8n orchestrator is cloud-hosted while your target client runs on a private Windows VM behind NAT:

• Do not expose the FastAPI port to the public internet.
• Install Tailscale on the Windows agent and n8n container.
• Direct n8n to communicate with the host's 100.x.y.z Tailscale mesh IP with token validation.

3. Deterministic Retries in n8n

Desktop GUIs exhibit variable latency. Avoid using raw delays (Wait nodes). Instead, implement a Polling Retry Loop in n8n using an If condition:

1. Execute /ocr/scan?contains=Invoice.
2. Check if the returned array length is > 0.
3. If 0, wait 500ms and increment a retry counter.
4. Route to /action/click only when the element is detected on screen.

---

5. Conclusion & Ready-to-Use Package

By converting raw OS displays into type-safe, HTTP-accessible endpoints, you eliminate proprietary RPA overhead and maintain all business logic within unified n8n workflows.

You can manually implement the boilerplate above, configure PyTesseract, tune the bounding-box logic, and map out the n8n HTTP schemas yourself.

Alternatively, you can grab the turnkey production-ready package—which includes pre-configured FastAPI binaries, packaged system-tray runners, auto-healing OCR wrappers, and importable n8n workflow templates:

• Instant Access on Whop
• Direct Download on Gumroad — Use coupon code EARLYBIRD at checkout for 20% off.

---

🛠️ D'autres outils MR Studio qui pourraient vous intéresser :

• Enterprise OpenTelemetry LLM Cost Guardrail & Semantic Cache Proxy
• Autonomous Desktop RPA Engine — Vision LLM Computer-Use Architecture (Python)
• Autonomous Visual QA & Broken UI Bug Detector — Python & Playwright CLI