Architecting a Vision-Language Desktop Agent: Neo v0.1 for Real-Time Screen Perception

What We Built Today

We engineered LuxurAI Cowork - an autonomous desktop agent powered by our in-house Neo v0.1 vision-language model. Unlike cloud vision APIs, Neo v0.1 runs on our dedicated dual-GPU cluster for low-latency screen perception.

The Problem We Solved

Desktop automation requires precise understanding of UI elements and application states. Existing solutions either use:

• Brittle coordinate-based scripting
• Expensive cloud vision APIs with 200-300ms latency

How The Architecture Works

1. Screen Capture Pipeline: Grabs downsampled frames at 15fps
2. Neo v0.1 Processing: Unified pixel-word encoding for:
   • OCR/text extraction
   • UI element detection
   • Application state classification
3. Action Execution: Uses Windows UIA/Win32 APIs with human confirmation gates

Transparent Economics

• ₹0.25 per credit (3 LC/minute for Cowork)
• Zero subscriptions - pure pay-as-you-go

How We Can Make This Better

We're particularly struggling with:

• High-DPI/multi-monitor coordinate mapping
• GPU memory optimization

Would love feedback from computer vision and Windows automation experts! Try the beta at luxurai.in