DEV Community: Mininglamp

Complex UIs, Cross-App Workflows, Long Tasks: What GUI Agents Actually Unlock

Mininglamp — Wed, 29 Apr 2026 09:44:16 +0000

AI agents have gotten remarkably good at text-based tasks. Platforms like OpenClaw and Claude Code can write code, manage files, search the web, analyze data, and orchestrate multi-step workflows. If the task lives in a terminal, an editor, or an API — agents handle it well.

But ask an agent to fill out a form in your CRM, adjust parameters in a design tool, or navigate a multi-step workflow in an enterprise system — and you'll hit a wall.

The problem isn't intelligence. It's that agents can't see your screen.

The GUI Gap in Agent Capabilities

Most agent platforms interact with computers through three channels: command-line interfaces (CLI), browser developer protocols (CDP), and APIs. These work well for code execution, web scraping, and cloud service calls. But they share a fundamental limitation: they only work with software that exposes a programmatic interface.

In practice, a large portion of the software people use daily has no API:

Enterprise systems (ERP, CRM, internal tools) often lack external interfaces
Desktop applications (office suites, design tools, specialized software) rely on mouse and keyboard interaction
Many web applications involve complex dynamic UIs that resist simple scripting

This is a structural gap in the agent technology stack. Agents have the "brain" to plan and reason, but they lack the "eyes" to see the screen and the "hands" to operate the interface.

Why GUI Vision Is the Missing Piece

Humans interact with computers through a visual feedback loop: observe the screen → understand the interface → locate the target element → perform an action → check the result → proceed. This process doesn't depend on any underlying API. It works through seeing and doing.

Traditional RPA (Robotic Process Automation) attempted to automate GUI interactions, but relied on hardcoded coordinates, element paths, and pixel matching. When the UI changes — which happens constantly in modern software — scripts break and need manual updates.

A more robust approach is GUI-VLA (Vision-Language-Action) models: architectures that unify visual perception (seeing the screen), language understanding (interpreting instructions), and action execution (clicking, typing, navigating) into a single framework. Instead of depending on fixed UI structures, the agent understands the interface through visual comprehension and acts accordingly.

The implication: if a piece of software has a graphical interface, an agent can potentially operate it.

From Theory to Working System

Mano-P is an open-source GUI-VLA agent model built for edge devices, released by Mininglamp Technology under the Apache 2.0 license. Its core approach: pure vision-driven GUI interaction — no DOM parsing, no system APIs, just screen understanding and action execution from screenshots.

The technical design involves three key mechanisms:

Three-stage progressive training. The model goes through supervised fine-tuning (SFT), offline reinforcement learning, and online reinforcement learning. Each stage builds on the previous one, progressively improving action accuracy and environmental robustness.

Think-act-verify reasoning loop. Before each action, the agent plans its intent. After execution, it verifies whether the result matches expectations. If the outcome deviates, the system automatically corrects course. This significantly reduces error accumulation in multi-step tasks.

Edge-optimized deployment. Through mixed-precision quantization and visual token pruning (GS-Pruning), the model runs locally on Apple M4 devices with 32GB RAM. All screenshots and task data stay on-device — no cloud calls required.

Benchmark Results

OSWorld benchmark: Mano-P 1.0-72B achieves a 58.2% success rate, ranking #1 among specialized GUI agent models — 13.2 percentage points ahead of the second-place opencua-72b (45.0%)
WebRetriever Protocol I: Mano-P 1.0 scores 41.7 NavEval, surpassing Gemini 2.5 Pro Computer Use (40.9) and Claude 4.5 Computer Use (31.3)
On-device inference: The 4B quantized model (w4a16) achieves 476 tokens/s prefill and 76 tokens/s decode on Apple M4 Pro, with only 4.3GB peak memory

What GUI Agents Actually Unlock

Once agents gain the ability to see and operate graphical interfaces, several previously impossible workflows become practical. Here are four scenarios demonstrated in the Mano-P project:

1. Fully Automated Application Building

The agent receives natural language requirements and autonomously completes the entire pipeline: requirement clarification → architecture design → code generation → local deployment → multi-level testing (API tests, LLM-based visual page inspection, and end-to-end GUI automation testing driven by VLA models). When tests fail, the system automatically diagnoses root causes, fixes code, redeploys, and retests — iterating until all test cases pass. No human intervention required. The final deliverable is a running application with complete documentation.

2. Commercial Video Production Pipeline

Starting from a user command, the system handles video generation, uploading, analysis, editing, and secondary evaluation. The agent independently operates web interfaces and editing software, performing file management, subtitle modifications, and other fine-grained GUI operations. It then generates analysis reports with both subjective assessments and objective metrics. This kind of cross-application, multi-step workflow is exactly what GUI agents enable.

3. Local On-Device Task Execution

The model runs inference directly on Mac devices (M4 chip + 32GB RAM required), breaking through the bottleneck where agent workflows previously had to pause and wait for human GUI interaction. The agent handles the entire flow autonomously, including steps that require screen-based operations.

4. Beyond Work: General-Purpose Visual Understanding

GUI vision capabilities extend beyond productivity scenarios. Through pure visual understanding of a game interface, the agent can perform tile recognition, analysis, and decision-making in Mahjong. This demonstrates the generality of the GUI-VLA approach — the same model framework applies across structured business processes and unstructured interactive environments.

What This Means for Developers

The agent ecosystem has been expanding steadily — from chat to code generation, from file management to data analysis. But the jump from "text-based assistant" to "desktop-native operator" requires a fundamentally new capability: visual understanding of graphical interfaces.

With GUI vision in place, agents are no longer limited to software that provides APIs or CLI access. Any application with a screen becomes a potential workspace.

For developers building agent-powered automation, this opens up scenarios that were previously out of reach: enterprise systems without APIs, cross-application data workflows, long-running business processes that span multiple desktop tools, and tasks that previously required a human sitting in front of a screen.

The desktop was the last frontier agents couldn't reach. That's changing.

1.6 Trillion Parameters Just Went Open Source. What About the Other Direction?

Mininglamp — Tue, 28 Apr 2026 10:54:16 +0000

On April 27, DeepSeek released its V4 model family and open-sourced the weights. The flagship V4-Pro Base has 1.6 trillion parameters (862B active), while V4-Flash comes in at 158B (Base 292B). Both use a Mixture of Experts (MoE) architecture. Within 48 hours of landing on HuggingFace, V4-Pro had already racked up 3,000+ likes and 174K downloads.

It's an impressive milestone for open-source AI. But it also crystallizes a question that's been brewing for a while: Is "bigger" the only direction AI models can go?

The case for Scaling Up

Let's be clear — Scaling Up works, and DeepSeek V4 is the latest proof.

The logic behind bigger models traces back to the Scaling Laws paper (Kaplan et al., 2020): model performance scales predictably with parameter count, dataset size, and compute. From GPT-3 (175B) to DeepSeek V3 to V4 (1.6T), each generation has pushed the ceiling higher on general reasoning, code generation, and mathematical problem-solving.

The engineering has matured too. MoE architecture is key — V4-Pro's 1.6T total parameters don't all activate at once. A routing mechanism selects which expert networks fire for each input, keeping per-inference compute manageable while retaining the knowledge capacity of a massive model. Combined with distributed inference, mixed precision, and optimized serving stacks (V4-Pro is already available on Together, Novita, Fireworks, and others), trillion-parameter models are becoming practically accessible.

None of this is hype. The results are real. For general-purpose tasks — open-ended reasoning, multilingual generation, complex code synthesis — larger models consistently outperform smaller ones.

But not every problem needs a trillion parameters

Here's where the story gets more interesting.

Running V4-Pro requires a multi-GPU cluster. Even using it through an inference API costs money per call. For high-frequency use cases — real-time interaction, continuous agent workflows, batch processing — that cost adds up fast. And for individual developers or small teams, the economics don't always work.

There are also structural constraints:

Data privacy. Cloud inference means your input data leaves your machine. For AI agent scenarios where the model needs to see your entire screen — emails, chat messages, bank statements — that's a non-trivial compliance issue.
Latency. Network round trips add delay. For agent workflows involving dozens of sequential steps (screenshot → understand → act → repeat), every millisecond of latency compounds.
Availability. No internet, no AI. But real-world use cases on airplanes, in secure facilities, or on unstable connections require AI that works offline.

These aren't criticisms of Scaling Up. They're boundary conditions that define where a different approach makes more sense.

The other direction: Scaling Out

If Scaling Up means making one model as large as possible, Scaling Out means distributing multiple smaller, specialized AI models closer to where they're actually needed — and having them collaborate.

This isn't a theoretical alternative. Several converging technical trends make it practical:

Model compression is real

Techniques like mixed-precision quantization (e.g., w4a16), visual token pruning, and knowledge distillation can shrink billion-parameter models to run on consumer hardware. On an Apple M4 chip, a 4B-parameter quantized model achieves 476 tokens/s prefill and 76 tokens/s decode, with a peak memory footprint of just 4.3GB.

Specialized models can beat general ones — in their domain

A general-purpose trillion-parameter model spreads its capacity across every conceivable task. A specialized model focuses all its parameters on one domain. In GUI automation specifically, a 4B-parameter model trained for this task has achieved #1 scores on domain benchmarks, outperforming models hundreds of times its size on the same tests.

Data sovereignty matters

When the model runs on the user's device, the data never leaves. No cloud upload, no network transmission, no third-party processing. For enterprise compliance, personal privacy, and regulated industries, this is a structural advantage that cloud-only models can't match.

Multi-agent collaboration

Instead of one giant model doing everything, multiple specialized agents can divide work — each running on different devices or nodes, communicating through standardized protocols. This architecture naturally fits the Scaling Out paradigm.

A concrete example: GUI agents on the edge

Let's make this concrete with a specific domain: GUI automation.

The task is straightforward in concept but demanding in practice: an AI agent looks at a screen, understands the interface elements, and performs operations — clicking buttons, filling forms, navigating menus — just like a human user would.

This is a natural fit for Scaling Out because:

Screen captures contain sensitive personal data — better processed locally
GUI tasks involve many sequential steps — latency accumulates
The task requires precise visual grounding and action planning, not broad general knowledge

Mano-P is an open-source project (Apache 2.0) by Mininglamp Technology that takes this approach. It's a GUI-VLA (Vision-Language-Action) agent designed for edge devices — specifically, it runs entirely on a Mac, with all data staying on the local machine.

The architecture integrates visual understanding, language reasoning, and action generation in a single end-to-end model, trained through a three-stage pipeline (SFT → offline RL → online RL) with a think-act-verify inference loop and GS-Pruning for visual token efficiency.

Published benchmark results (with evaluation framework and model specification noted):

OSWorld (72B model): 58.2% accuracy — ranked #1 (2nd place: 45.0%, a 13.2 percentage point gap)
WebRetriever Protocol I (72B model): 41.7 NavEval — ranked #1 (Gemini 2.5 Pro: 40.9, Claude 4.5: 31.3)
Edge deployment (4B quantized, w4a16): 476 tokens/s prefill, 76 tokens/s decode, 4.3GB peak memory on Apple M4

Hardware requirement: Mac with Apple M4 chip + 32GB RAM (or Mano-P Compute Stick via USB 4.0+).

The takeaway: a 4B-parameter model running locally on a Mac can achieve state-of-the-art results in its domain. Not because small models are universally better, but because the right model for the right task, deployed in the right place, can outperform a general-purpose giant.

Two tracks, one ecosystem

DeepSeek V4 pushing to 1.6 trillion parameters and a 4B model hitting #1 on GUI benchmarks are not contradictory developments. They're two sides of the same evolution in AI:

Scaling Up provides the general intelligence foundation — broad reasoning, complex generation, cross-domain capabilities
Scaling Out provides the execution layer — privacy-preserving, low-latency, offline-capable, specialized for specific tasks

The two can work together: edge models handle local tasks, and when something exceeds their scope, they call out to cloud models. This layered architecture may be closer to how AI actually gets deployed in the real world than any single-model paradigm.

For developers choosing a direction: it's not about picking the model with the most parameters. It's about picking the model that fits your constraints — compute budget, latency requirements, data sensitivity, deployment environment.

The trillion-parameter era is here. And so is the era of AI that runs on your machine.

Resources:

Mano-P (Apache 2.0): github.com/Mininglamp-AI/Mano-P

Happy 45th Birthday, GUI. Meet Your New Power User.

Mininglamp — Mon, 27 Apr 2026 11:54:15 +0000

On April 27, 1981, Xerox introduced the Star 8010 Information System — the first commercial computer with a graphical user interface.

Bitmapped display, desktop metaphor, icons, windows, mouse, WYSIWYG. Everything we take for granted about modern computing started with a $16,595 workstation that most people never used.

Today marks the 45th anniversary of that moment.

Five Milestones in 45 Years

The GUI's history can be traced through a handful of defining moments:

1981 · Xerox Star: GUI is born. The desktop metaphor becomes the foundational paradigm for human-computer interaction.
1984 · Macintosh: Apple brings GUI to the consumer market. Computing becomes visual for everyone.
1995 · Windows 95: The Start menu and taskbar. GUI becomes the global default.
2007 · iPhone: Multi-touch replaces the mouse. GUI extends from desktops to pockets.
2025–2026 · GUI Agents: AI learns to "see" screens and operate them autonomously.

The first four milestones share one constant: the user is always a human. Interface design revolves around human visual cognition — icons should be intuitive, layouts should follow natural eye movement, interactions should provide instant feedback.

The fifth milestone introduces a fundamental shift: the "user" can be an AI.

When AI Becomes the GUI Operator

Over the past two years, GUI Agents have emerged as a distinct technical direction. The core idea: train AI models to operate computers the way humans do — by looking at the screen and performing mouse/keyboard actions.

This is fundamentally different from traditional automation:

Approach	Dependency	Coverage
API/CLI	Target system must expose an API	Only apps with APIs
DOM/CDP parsing	Requires browser internals or accessible widget trees	Primarily web apps
Pure vision	None — works with any GUI	Any application with a visual interface

The vision-based approach inherits the exact principle that Xerox Star's designers articulated 45 years ago: a GUI should be self-explanatory — you should be able to understand how to use it just by looking at it. Back then, that capability belonged to humans. Now AI is developing it too.

Mano-P: A Vision-Only GUI Agent for Edge Devices

Mininglamp Technology open-sourced Mano-P under the Apache 2.0 license, taking a vision-only approach to GUI automation. Mano-P uses a GUI-VLA (Vision-Language-Action) architecture that integrates visual understanding, language reasoning, and action generation in a single end-to-end model.

Benchmark Results

OSWorld (verified, specialized model): Mano-P 72B achieves 58.2% accuracy, ranking #1 (runner-up: 45.0%)
WebRetriever Protocol I: 41.7 NavEval (ranked #1), surpassing Gemini 2.5 Pro (40.9) and Claude 4.5 (31.3)

On-Device Performance

The 4B quantized model (w4a16) runs locally on Apple M4 Macs:

Prefill: 476 tokens/s
Decode: 76 tokens/s
Peak memory: 4.3 GB
Fully local execution — screen captures and task data never leave the device

Hardware requirements: Mac with Apple M4 chip + 32GB RAM, or any Mac with a Mano-P Compute Stick (USB 4.0).

Technical Approach

Bidirectional self-reinforcement learning (Text ↔ Action cyclic consistency)
Three-stage training: SFT → Offline RL → Online RL
Think-act-verify reasoning loop
GS-Pruning for visual token reduction, optimizing edge inference

Full Circle

Forty-five years ago, the Xerox Star taught humans to interact with computers through visual interfaces. Today, AI agents are learning to do the same thing — looking at pixels, understanding layouts, clicking buttons.

The Xerox Star was a commercial failure but a technical triumph. Its design DNA — bitmapped displays, the desktop metaphor, WYSIWYG — lives on in every Mac, PC, phone, and tablet. GUI Agents are the next chapter: the interface designed for human eyes turns out to work for AI eyes too.

The GUI hasn't changed. What changed is who's looking at the screen.

GitHub: github.com/Mininglamp-AI/Mano-P
Technical Report: arXiv:2509.17336

Mano-P is developed by Mininglamp Technology and released under the Apache 2.0 license.

AI Got Hands: Breaking the Human Bottleneck in Agent Workflows

Mininglamp — Fri, 24 Apr 2026 07:35:34 +0000

Most AI agent frameworks can browse the web. Open a URL, read some HTML, click a button, fill a form. This works because browsers expose their internals through well-defined protocols — Chrome DevTools Protocol (CDP), DOM APIs, JavaScript injection.

But here's the problem: the majority of professional work doesn't happen in a browser.

CAD engineers work in SolidWorks. Video editors work in DaVinci Resolve. Data analysts switch between Excel, custom BI dashboards, and terminal sessions. System administrators navigate native configuration panels. Designers use Figma's desktop app, Photoshop, Blender.

None of these expose a DOM. None of them speak CDP. And most of the "AI automation" ecosystem simply cannot reach them.

This article examines the three main technical approaches to GUI automation, explains why the vision-only approach matters for breaking the browser boundary, and looks at measured results on cross-application benchmarks.

Three Approaches to GUI Automation

Approach 1: CDP and HTML Parsing

The Chrome DevTools Protocol gives programmatic access to Chromium-based browsers. You can:

Query the DOM tree
Execute JavaScript in page context
Intercept network requests
Simulate clicks and keyboard input at the DOM element level

Frameworks like Playwright, Puppeteer, and most browser-based AI agents use this approach. It's precise, fast, and reliable — within its domain.

Strengths:

Pixel-perfect element targeting via CSS selectors
Access to hidden elements, shadow DOM, iframe contents
Can read and modify page state programmatically
Low latency (no screen capture needed)

Limitations:

Browser-only. CDP doesn't exist outside Chromium. Firefox has a partial equivalent; Safari's is limited. Native desktop apps, mobile apps, and OS-level UI are completely out of scope.
Site-specific fragility. CSS selectors break when websites update their markup. A class name change, a restructured component tree, or a switch from server-rendered to client-rendered content can silently break automation scripts.
SPA complexity. Modern single-page applications with dynamic rendering, lazy loading, and virtual scrolling create timing dependencies that are hard to handle reliably.
Anti-automation measures. Many sites actively detect and block CDP-based automation through bot detection, CAPTCHAs, and behavioral analysis.

For browser-based tasks, CDP is the right tool. But framing "AI automation" as "browser automation" leaves most of the desktop untouched.

Approach 2: Accessibility APIs

Operating systems provide accessibility APIs (UI Automation on Windows, Accessibility API on macOS, AT-SPI on Linux) that expose a tree of UI elements with their roles, labels, and states. Screen readers use these APIs. So can automation frameworks.

Strengths:

Works across native applications, not just browsers
Semantic information (button labels, text field values, checkbox states)
Standardized per-OS (once you handle the platform API, it works across apps)
Doesn't require visual rendering — works even on headless systems

Limitations:

Inconsistent implementation. Application developers implement accessibility support to varying degrees. A well-built macOS app might expose a complete accessibility tree. A cross-platform Electron app might expose a flat, unlabeled hierarchy. A legacy Qt application might expose nothing useful.
Custom controls are invisible. Rendered canvases (games, CAD viewports, video timelines, terminal emulators with custom rendering) don't have accessibility tree entries for their internal elements. A 3D modeling tool's viewport is a single opaque rectangle to the accessibility API.
Platform fragmentation. Each OS has its own API, data model, and quirks. Code written for macOS accessibility doesn't transfer to Windows or Linux.
Performance overhead. Querying the full accessibility tree of a complex application can be slow — hundreds of milliseconds for apps with deep hierarchies.

Accessibility APIs are genuinely useful and underappreciated in the automation space. But they have a fundamental coverage gap: they can only see what developers explicitly expose, and many interfaces — especially professional tools with custom rendering — aren't fully accessible.

Approach 3: Vision-Only Understanding

The third approach skips the application's internal representation entirely. Instead of querying DOM trees or accessibility APIs, the agent looks at what's on screen — raw pixels — and reasons about what it sees.

This is how humans interact with computers. We don't parse HTML to find the "Submit" button. We see a rectangle that looks like a button, read its label, and click it.

Strengths:

Universal coverage. If a human can see it on screen, the agent can see it. Native apps, web apps, terminals, games, remote desktops, virtual machines — all the same to a screenshot.
No application cooperation required. The agent doesn't need hooks, APIs, or special access. Screen capture is a standard OS capability.
Resilient to UI changes. A button that moves from the left sidebar to the top toolbar still looks like a button. Visual understanding is inherently more robust to layout changes than coordinate-based or selector-based targeting.
Cross-platform by default. Screenshots are screenshots, regardless of OS. The same model that automates macOS can automate Windows or Linux without platform-specific code.

Limitations:

Requires capable vision models. The agent needs to accurately parse dense UIs, read small text, distinguish between similar-looking elements, and understand spatial relationships. This is a hard computer vision problem.
Higher computational cost. Processing a full screenshot through a vision model is more expensive than querying a DOM tree. This is where model optimization and edge deployment become critical.
Occlusion and overlaps. Dropdown menus, tooltips, and modal dialogs can cover important UI elements. The agent needs to handle these states.
No hidden state access. The agent can't see what's behind a collapsed menu or in an unscrolled region. It has to navigate to make information visible, just like a human would.

The trade-off is clear: vision-only gives you universal reach at the cost of requiring a strong vision model. The question is whether today's models are good enough to make that trade worthwhile.

Breaking the Browser Boundary

Let's make this concrete. Consider a workflow that's common in any organization:

"Pull Q1 sales data from the CRM, cross-reference it with the finance spreadsheet on the shared drive, and create a summary slide deck for the Monday meeting."

A browser-based agent can maybe handle the CRM part (if it's a web app). But the finance spreadsheet might be in a native Excel window. The slide deck is in PowerPoint or Keynote. The shared drive might be mounted as a local folder or accessed through a native file manager.

This is one task that touches three or four applications. A CDP-based agent taps out after step one. An accessibility-based agent might handle two of the three but struggle with Excel's complex grid rendering. A vision-based agent can navigate all of them — it sees what you see, clicks where you'd click, types what you'd type.

The same principle applies to more specialized work:

DevOps: Switching between a terminal, a monitoring dashboard (Grafana), a cloud console (AWS), and a ticket system (Jira) — mixing web and native UIs.
Design: Moving assets between Figma, Photoshop, and a file manager, with each tool having its own UI paradigms.
Data science: Interacting with Jupyter notebooks, database GUIs, Excel, and custom visualization tools.
System administration: Navigating OS settings panels, network configuration tools, and hardware management interfaces that have no web equivalent.

These aren't edge cases. They're the normal workday for millions of professionals. The browser boundary isn't a minor limitation — it's a wall that separates "AI demo" from "AI tool."

Measured Results on Cross-Application Benchmarks

Mano-P (GUI-Aware Agent Model for Edge Devices, open-source under Apache 2.0) uses the vision-only approach. The name stands for "Mano" (Spanish for "hand") and "P" (Person & Party).

Here's the architecture:

The model takes screenshots as input and outputs action sequences — click coordinates, keystrokes, scroll directions, and multi-step plans. No DOM parsing. No accessibility tree queries. Just pixels in, actions out.

On OSWorld — a benchmark specifically designed to test agents on real desktop environments across different operating systems and applications — the results look like this:

Mano-P achieves a 58.2% success rate on OSWorld, compared to 45.0% for the second-place model. This benchmark includes tasks spanning file management, office applications, web browsing, system configuration, and multi-application workflows — exactly the kind of cross-boundary work where vision-only approaches should theoretically shine.

On web-specific benchmarks, the vision-only approach remains competitive. On WebRetriever Protocol I, Mano-P scores 41.7 NavEval, ahead of Gemini 2.5 Pro (40.9) and Claude 4.5 (31.3). This is notable because web benchmarks should favor approaches that can access the DOM directly — yet the vision-only model still leads.

Why Vision-Only Can Win on the Web Too

This counterintuitive result — a vision model beating DOM-aware models on web tasks — has a plausible explanation.

Modern web pages are designed for human eyes, not for programmatic parsing. A typical SaaS dashboard might have:

Dynamically loaded content with JavaScript-rendered elements
Canvas-based charts and visualizations
Complex CSS layouts where the visual hierarchy doesn't match the DOM hierarchy
Shadow DOM components that hide internal structure
Iframes embedding third-party content

A DOM parser sees the structural complexity. A vision model sees the rendered result — the same clean layout the designer intended for human users. In many cases, the rendered output is actually easier to reason about than the underlying markup.

This doesn't mean vision-only is universally better for web tasks. DOM access provides exact text content (no OCR errors), hidden metadata, and element state information. But for navigation and interaction tasks — "find the settings button and change this option" — visual understanding can be more robust than structural parsing.

Running on the Edge

A vision-based agent is computationally demanding. Processing high-resolution screenshots through a vision-language model requires significant inference capacity. This is where model design and hardware optimization become critical.

Mano-P uses a 4B parameter model with w4a16 quantization (4-bit weights, 16-bit activations). On an Apple M4 Pro with 32GB RAM:

Prefill: 476 tokens/s (ingesting the screenshot and context)
Decode: 76 tokens/s (generating the action sequence)
Peak memory: 4.3 GB

These numbers mean the full perception-reasoning-action loop completes in under a second for typical interactions. The 4.3 GB memory footprint leaves plenty of room for the applications being automated to run alongside the agent.

Running locally also eliminates the latency of uploading screenshots to a cloud API. A screenshot from a 4K display can be several megabytes — sending that to a remote server for every action step adds meaningful delay, especially on typical upload speeds.

The local execution model also means screenshots and task data never leave the device. For workflows involving sensitive information — financial data, medical records, proprietary designs — this is often a hard requirement, not a nice-to-have.

The Training Challenge: Teaching a Model to See and Act

Building a vision-only agent that works across diverse applications requires solving several interconnected problems:

Visual grounding: The model must map regions of a screenshot to semantic UI elements. "The blue button in the top-right corner that says 'Save'" needs to become a precise coordinate.

Action planning: Given a goal ("rename this file to quarterly-report-v2.pdf"), the model must generate a sequence of actions: right-click the file → click "Rename" → select all text → type the new name → press Enter.

Error recovery: UI automation in real environments is noisy. Menus take time to open. Dialog boxes appear unexpectedly. Actions sometimes fail. The model needs to verify outcomes and adapt.

Mano-P addresses these through a three-stage training pipeline:

Supervised Fine-Tuning (SFT) on curated GUI interaction datasets builds foundational visual understanding and action generation.
Offline Reinforcement Learning on collected trajectories teaches multi-step planning from both successful and failed interactions.
Online Reinforcement Learning with a think-act-verify loop develops robustness — the model learns to check its work and recover from failures in live environments.

A technique called GS-Pruning (Gradient-based Structured Pruning) then compresses the model, removing redundant capacity to hit the 4B parameter target without proportional capability loss.

Implications for Agent Architecture

The vision-only approach has second-order effects on how agent systems are designed:

Simpler integration. Adding a new application to the agent's capabilities doesn't require building an adapter, writing selectors, or mapping accessibility trees. If the app has a GUI, the agent can use it.

Cross-system workflows. Tasks that span multiple applications — copying data from a web CRM into a native spreadsheet, then attaching it to an email — don't require different automation strategies for each app. The agent uses the same perception-action loop throughout.

Long-task planning. Because the agent perceives the full screen state at each step, it can maintain context across complex, multi-step workflows. The think-act-verify training means it checks whether each step succeeded before proceeding.

Reduced maintenance burden. Selector-based automation scripts break when UIs update. Vision-based automation is inherently more resilient because it relies on visual patterns rather than structural identifiers.

Current Limitations and Honest Assessment

Vision-only GUI automation is not a solved problem. Current limitations include:

Small text and dense UIs. Spreadsheets with tiny fonts, code editors with many similar-looking lines, and dashboards with packed metrics are still challenging.
Speed-sensitive interactions. Drag-and-drop, real-time canvas manipulation, and rapid sequential inputs are harder than discrete click-and-type actions.
Verification ambiguity. Sometimes it's hard to tell from a screenshot alone whether an action succeeded (e.g., a background save operation with no visual confirmation).
Training data coverage. The model performs best on application types well-represented in training data. Niche or custom enterprise software may require fine-tuning.

These are active research areas, not fundamental barriers. As vision models improve in resolution handling, temporal reasoning, and few-shot adaptation, the coverage gap will narrow.

Getting Started

Mano-P is open-source under Apache 2.0 with a three-phase release plan:

Phase 1 (released): Skills — task-specific capability modules
Phase 2: Local models and SDK — the inference runtime and integration tools
Phase 3: Training methods — the full pipeline for community extension

The code and documentation are at github.com/Mininglamp-AI/Mano-P.

If you're building agent workflows that stop at the browser boundary, it might be time to give your AI hands that can reach the rest of the desktop.

AI for Personal: How Edge-Native Agents Bring Data Sovereignty Back to Your Device

Mininglamp — Fri, 24 Apr 2026 07:34:50 +0000

When you ask a cloud-based AI agent to "summarize my last 20 emails" or "fill out this expense report from my receipts," you're making an implicit trade: convenience for control. Your screenshots, your documents, your workflow patterns — all uploaded to someone else's infrastructure, processed on someone else's GPUs, stored under someone else's data retention policy.

For many developers and enterprise users, that trade is becoming harder to justify.

This article explores the technical architecture behind running AI agents entirely on local hardware — no cloud round-trips, no data exfiltration, no API keys required — and how a 4B-parameter model running on Apple Silicon can match or exceed cloud-hosted alternatives on GUI automation benchmarks.

The Cloud Dependency Problem

Most AI agent frameworks today follow a predictable pattern:

Capture screen state (screenshot, DOM, accessibility tree)
Send it to a cloud API (OpenAI, Anthropic, Google)
Receive action instructions
Execute locally
Repeat

This works. But it has structural problems that no amount of prompt engineering can fix:

Latency compounds. Each action in a multi-step workflow requires a round-trip. A 10-step task that takes 500ms per API call adds 5 seconds of pure network overhead — before you account for token generation time on the server side.

Data leaves the device by design. Screenshots contain everything visible on screen: open tabs, notification previews, partial passwords in terminal windows, private messages, financial data. The agent doesn't selectively capture — it sees what you see.

Cost scales with usage. Vision API calls with screenshot inputs are expensive. A power user running an agent for 8 hours might generate hundreds of screenshots, each consuming thousands of tokens.

Availability depends on infrastructure you don't control. API rate limits, outages, region restrictions, and policy changes can break your workflow without warning.

None of these are hypothetical. They're the everyday reality of cloud-dependent agent architectures.

What "Edge-Native" Actually Means

Edge-native AI isn't just "smaller model on a laptop." It's a fundamentally different architecture where the entire inference loop — perception, reasoning, and action — runs on the device where the work happens.

Mano-P (GUI-Aware Agent Model for Edge Devices, open-source under Apache 2.0) is built around this principle. The name comes from "Mano" (Spanish for "hand") and "P" (Person & Party) — an agent that works with its hands, for its person.

Here's the architecture:

The key design decision: Mano-P uses vision-only understanding. It looks at screenshots — raw pixels — rather than parsing HTML, querying accessibility APIs, or injecting JavaScript into the DOM. This matters for edge deployment because:

No application-specific adapters. The same model works on browsers, native apps, terminal windows, and 3D tools.
No privilege escalation required. Screen capture is a standard OS capability. DOM injection and accessibility API access often require elevated permissions.
Reduced attack surface. The agent reads pixels. It doesn't hook into application internals.

In local mode, screenshots and task data never leave the device. There's no telemetry endpoint, no "anonymous usage data" upload, no cloud fallback. The inference happens on your hardware, and the data stays on your hardware.

Running a 4B Model on Apple Silicon

The practical question is: can edge hardware actually run a capable agent model at interactive speeds?

Here are measured numbers on an Apple M4 Pro with 32GB unified memory:

Metric	Value
Model size	4B parameters (w4a16 quantization)
Prefill throughput	476 tokens/s
Decode throughput	76 tokens/s
Peak memory	4.3 GB

Let's break down why these numbers matter.

476 tokens/s prefill means the model can ingest a screenshot (encoded as visual tokens) and the task context in well under a second. This is the "reading" phase — where the model processes what it sees on screen.

76 tokens/s decode means action generation (the "writing" phase — outputting what to click, type, or scroll) takes roughly 100-300ms for a typical action sequence. This is fast enough for real-time interaction.

4.3 GB peak memory means the model fits comfortably alongside your normal workload. On a 32GB machine, you have ~28GB left for browsers, IDEs, design tools — whatever the agent is supposed to be automating.

The w4a16 quantization scheme (4-bit weights, 16-bit activations) is the key enabler here. It reduces the model's memory footprint by roughly 4x compared to fp16, while preserving activation precision where it matters most — in the attention and reasoning layers.

Apple Silicon's unified memory architecture is particularly well-suited for this workload. There's no PCIe bottleneck between CPU and GPU memory; the model weights, the screenshot tensor, and the action output all live in the same memory space. The Neural Engine and GPU cores can be dispatched to different parts of the inference pipeline without data copies.

For machines without sufficient local compute, Mano-P also supports offloading to a compute stick connected via USB 4.0 — effectively adding a dedicated inference accelerator without changing the data sovereignty model (the stick is still physically local).

Benchmark Performance: Does Local Mean Worse?

The assumption that smaller, local models must sacrifice capability is worth testing empirically.

On OSWorld — a benchmark that tests agents on real desktop environments across operating systems — Mano-P achieves a 58.2% success rate, compared to 45.0% for the second-place model. This isn't a narrow domain-specific benchmark; OSWorld tests general GUI automation across diverse applications and multi-step workflows.

On WebRetriever Protocol I, Mano-P scores 41.7 NavEval, ahead of Gemini 2.5 Pro (40.9) and Claude 4.5 (31.3).

These results suggest that the "edge tax" — the performance cost of running locally instead of in the cloud — can be zero or negative when the model architecture is specifically designed for the task. A 4B model trained and optimized for GUI understanding can outperform much larger general-purpose models that treat GUI automation as one capability among many.

The Training Pipeline: How a Small Model Gets Good

Model size alone doesn't explain the benchmark results. The training methodology matters more at this scale because every parameter has to earn its keep.

Mano-P's training follows a three-stage progression:

Stage 1: Supervised Fine-Tuning (SFT). The base model is trained on curated GUI interaction datasets — screenshots paired with correct action sequences. This gives the model foundational competence in visual grounding (mapping screen regions to semantic elements) and action generation.

Stage 2: Offline Reinforcement Learning. Using collected interaction trajectories, the model learns from both successful and failed attempts. This stage improves multi-step planning — the ability to reason about sequences of actions rather than reacting to each screenshot independently.

Stage 3: Online Reinforcement Learning. The model interacts with live environments and learns from real outcomes. A think-act-verify loop ensures the model checks whether its actions achieved the intended result before proceeding. This is where the model develops robustness — learning to recover from unexpected states, handle loading delays, and adapt to UI variations.

An additional technique called GS-Pruning (Gradient-based Structured Pruning) removes redundant model capacity after training, further reducing the model size without proportional capability loss. This is how you get a 4B model that punches above its weight class.

What This Enables

When an AI agent runs entirely on your device with no cloud dependency, certain use cases become possible that were previously impractical or unacceptable:

Sensitive workflow automation. Automating tasks that involve medical records, legal documents, financial data, or classified information — where uploading screenshots to a third-party API would violate compliance requirements.

Air-gapped environments. Research labs, government facilities, and financial trading floors often operate without internet access. A local agent works regardless of network state.

Consistent performance. No API rate limits, no cold starts, no "the service is experiencing high demand" degradation. The model runs at the same speed whether it's Monday morning or Friday night.

Cost predictability. The hardware is a one-time cost. There's no per-token billing, no surprise invoices, no pricing changes.

Beyond single-device automation, the core capabilities extend to cross-system data integration (working across multiple apps to consolidate information), long-task planning (breaking complex goals into executable sequences), and intelligent report generation (synthesizing information from multiple sources into structured output).

Open Source Roadmap

Mano-P is released under Apache 2.0 with a three-phase open-source plan:

Phase 1 (released): Skills — the agent's capability modules for specific task domains
Phase 2: Local models and SDK — the inference runtime and developer integration tools
Phase 3: Training methods — the full pipeline so others can train specialized models

The phased approach is deliberate. Phase 1 lets developers use and evaluate the agent immediately. Phase 2 gives them the tools to integrate it into their own products. Phase 3 enables the community to extend the model to new domains and hardware platforms.

The Bigger Picture

The shift from cloud-dependent to edge-native AI agents isn't primarily a technical argument. It's an architectural one.

Cloud APIs are shared infrastructure. They're powerful, convenient, and constantly improving. But they come with structural constraints — latency, cost, data exposure, availability — that are inherent to the architecture, not bugs to be fixed.

Edge-native agents trade cloud-scale compute for data sovereignty, predictable performance, and zero marginal cost. For many workflows — especially those involving sensitive data or requiring low-latency interaction — that's a trade worth making.

The benchmark results suggest it doesn't have to be a trade at all. A well-designed, well-trained 4B model running on consumer hardware can match or exceed cloud-hosted alternatives on practical GUI automation tasks.

The code is on GitHub: github.com/Mininglamp-AI/Mano-P

If your data matters enough to keep it on your device, your AI agent should be able to stay there too.

Apple Took 50 Years for 3 CEOs — GUI Agents Went from Paper to Production in One

Mininglamp — Wed, 22 Apr 2026 04:56:34 +0000

Yesterday, Apple announced a landmark succession: Tim Cook steps down as CEO to become Executive Chairman, with John Ternus taking over on September 1. In its 50-year history, Apple has had just three CEOs: Jobs, Cook, Ternus.

Three people. Fifty years. Each transition spaced over a decade apart.

Now consider the AI Agent space: one year ago, most people were still debating whether AI could operate a computer at all. Today, there are open-source projects delivering usable on-device solutions.

This article breaks down the technical evolution of GUI Agents — using Mano-P, our open-source project, as a concrete example of what it takes to go from training to on-device deployment.

What Is a GUI Agent?

A GUI Agent's core mission: let AI operate a computer's graphical interface the way a human does — recognizing screen elements, understanding task intent, and executing clicks, typing, and drag-and-drop operations.

There are currently two main technical approaches:

Approach	Mechanism	Strength	Limitation
API/DOM-driven	Reads interface structure via accessibility APIs or DOM trees	Precise element targeting	Depends on app-specific interfaces
Pure vision	Understands UI from screenshots alone	Works across any application	Higher demand on visual comprehension

Mano-P takes the pure vision route. Designed for Mac, it's an on-device GUI Agent — "Mano" means "hand" in Spanish, "P" stands for Person. AI for Personal. It runs entirely locally; no data leaves the device.

Training: Bidirectional Self-Reinforcement Learning

The training pipeline follows a three-stage progressive framework:

Stage 1: SFT (Supervised Fine-Tuning)
    ↓  Build foundational capabilities
Stage 2: Offline Reinforcement Learning
    ↓  Learn strategy optimization from historical data
Stage 3: Online Reinforcement Learning
    ↓  Continuously improve through real-environment interaction

Stage 1 — SFT: Supervised fine-tuning on high-quality GUI operation datasets. The model learns basic interface understanding and action mapping — ground-truth capability building.

Stage 2 — Offline RL: Uses collected interaction trajectories to optimize policies via reinforcement learning. Extracts success/failure signals from historical operations without requiring live environment interaction, keeping training costs manageable.

Stage 3 — Online RL: Interacts with real GUI environments, adjusting strategy based on live feedback. The key challenge here is balancing exploration (trying new operation paths) with exploitation (reinforcing proven strategies).

Inference: Think-Act-Verify Loop

The inference mechanism uses a think-act-verify cycle:

while task_not_complete:
    # Think: analyze current screen, plan next action
    thought = model.think(screenshot, task_context)

    # Act: execute GUI operation (click, type, scroll)
    action = model.act(thought)
    execute(action)

    # Verify: capture new screenshot, check result
    new_screenshot = capture_screen()
    verified = model.verify(new_screenshot, expected_state)

    if not verified:
        task_context.update(error_info)  # back to Think

This gives the Agent self-correction capability. In real desktop environments, unexpected popups, loading delays, and dynamic element repositioning are common — the verify step catches these before errors cascade.

Core capabilities span four areas: complex GUI automation, cross-system data integration, long-task planning and execution, and intelligent report generation.

Benchmark Performance

OSWorld: Mano-P's 72B model achieves 58.2% success rate, ranking #1 among specialized GUI agent models. Second place scores 45.0%. OSWorld simulates real OS environments with cross-application tasks including file operations, browser interactions, and office software workflows.

WebRetriever Protocol I: Scores 41.7 NavEval, surpassing Gemini 2.5 Pro (40.9) and Claude 4.5 (31.3). This benchmark focuses on web information retrieval and interaction.

Edge Deployment: 4B Model Running On-Device

On-device deployment is a core feature of Mano-P. Here's the 4B quantized model (w4a16) performance on M4 Pro:

Metric	Value
Prefill Speed	476 tokens/s
Decode Speed	76 tokens/s
Peak Memory	4.3 GB

The w4a16 quantization scheme — 4-bit weights with 16-bit activations — strikes a practical balance: 4-bit weights dramatically reduce memory footprint while 16-bit activations preserve numerical precision during inference.

Hardware requirement: Apple M4 chip + 32 GB RAM. Fully local execution — your screen data never leaves your device.

Getting Started

Open-sourced under the Apache 2.0 license:

# Install
brew tap HanningWang/tap && brew install mano-cua

GitHub: https://github.com/Mininglamp-AI/Mano-P

Wrapping Up

From the three-stage progressive training framework, to think-act-verify inference, to w4a16 quantization enabling edge deployment — the path from "concept" to "locally usable" GUI Agents is becoming clear.

Apple took 50 years and three leaders. The GUI Agent space went from academic papers to open-source tools in roughly one year. These are two fundamentally different timescales.

For developers, Mano-P — Apache 2.0 licensed, runnable on a local Mac — is already a starting point for exploration and experimentation.

Tim Cook Steps Down — Is the Mac Becoming the Next AI Agent Platform?

Mininglamp — Wed, 22 Apr 2026 04:55:56 +0000

On April 20, Apple dropped a bombshell.

On April 20, Apple announced that Tim Cook will transition from CEO to Executive Chairman, with hardware engineering SVP John Ternus taking over on September 1. In its 50-year history, Apple has now had just three CEOs.

Cook's 14-year tenure defined two eras: making Apple the world's most valuable company, and driving the historic transition from Intel to Apple Silicon. Ternus's background is telling — he's not from the software or services side. He's Apple's hardware engineering chief, the person who shipped Apple Silicon. Choosing a hardware engineer as CEO is Apple signaling that hardware innovation remains the priority for the next decade.

This signal is especially interesting in the context of AI. For the past few years, AI development and deployment has been virtually synonymous with "NVIDIA GPUs + Windows/Linux." The Mac has been a non-factor in the AI ecosystem. But Apple Silicon is changing that — more and more developers are running AI workloads on Mac, and it's no longer just experimentation.

Why the Mac Couldn't Do AI Before

The answer is straightforward: the CUDA ecosystem. NVIDIA GPUs + CUDA have effectively monopolized AI training and inference infrastructure. Apple and NVIDIA parted ways after 2016 — Macs haven't shipped with NVIDIA GPUs since. Without CUDA, major deep learning frameworks (PyTorch, TensorFlow) treated Mac as a second-class citizen — technically supported, but performance-limited.

AI practitioners defaulted to Windows desktops or Linux servers. Mac was fine for writing code, but running models meant SSH-ing into a remote machine.

What Apple Silicon Changed

The M1 chip in 2020 was the inflection point. Apple Silicon's Unified Memory Architecture broke the traditional CPU-GPU separation — CPU and GPU share a single memory pool, eliminating the need to shuttle data between them. This design has natural advantages for AI inference:

No VRAM bottleneck: 32 GB or more of unified memory is directly available for model inference, unlike traditional GPUs constrained by dedicated VRAM
Superior power efficiency: Lower power consumption at equivalent compute, enabling MacBooks to run models on battery
Growing ecosystem: Apple launched MLX, a machine learning framework optimized for Apple Silicon; PyTorch now officially supports the MPS backend

From M1 through M4, each generation has delivered meaningful improvements in AI inference performance. With M4 and 32 GB RAM, Macs can now smoothly run models that previously required dedicated GPU servers.

A Real-World Example: GUI Agents on Mac

To make this concrete, consider GUI Agents — a fast-growing area in AI where models directly observe the screen, understand interface elements, and operate mouse and keyboard to complete complex computer tasks. These applications demand real-time local responsiveness, making them a natural fit for Mac deployment.

Mano-P is our open-source GUI Agent built specifically for Mac. "Mano" comes from the Spanish word for "hand," "P" stands for Person — AI for Personal. It uses pure vision — no accessibility APIs, no DOM parsing, just screenshot understanding. Everything runs locally on Mac; no data leaves the device.

How Does It Perform on Apple Silicon?

The question everyone cares about: is Apple Silicon actually fast enough for AI Agents?

OSWorld Benchmark (the standard end-to-end evaluation for GUI Agents): Mano-P's 72B model achieves 58.2% success rate, ranking #1. Second place scores 45.0% — a gap of over 13 percentage points.

WebRetriever Protocol I: Mano-P scores 41.7 NavEval, surpassing Gemini 2.5 Pro (40.9) and Claude 4.5 (31.3).

Local inference performance — Mano-P's 4B quantized model (w4a16) on M4 Pro:

Metric	Value
Prefill Speed	476 tokens/s
Decode Speed	76 tokens/s
Peak Memory	4.3 GB

At 4.3 GB peak memory on a 32 GB Mac, you can run the Agent alongside your IDE, browser, Slack, and everything else without breaking a sweat.

Hardware requirement: Apple M4 chip + 32 GB RAM.

Technical Overview

Training: Bidirectional self-reinforcement learning with three progressive stages — SFT → Offline RL → Online RL.

Inference: Think-act-verify loop. Analyze the screen state, execute an action, verify the result. If something unexpected happens (popup, loading delay), the system self-corrects.

Core capabilities: Complex GUI automation, cross-system data integration, long-task planning and execution, intelligent report generation.

brew tap HanningWang/tap && brew install mano-cua

Open-sourced under Apache 2.0: https://github.com/Mininglamp-AI/Mano-P

The Mac AI Ecosystem Is Taking Shape

Mano-P is our contribution, but it's one data point. The bigger picture:

MLX gives developers an efficient way to run models on Apple Silicon
Ollama and LM Studio make running open-source LLMs on Mac as easy as installing an app
Core ML continues to improve, with Apple investing in on-device AI infrastructure

The old consensus — "doing AI means Windows/Linux + NVIDIA" — is loosening. Not because the Mac is replacing GPU servers for large-scale training, but because for inference, personal development, and on-device applications, the Mac is becoming a genuinely viable platform.

Apple just chose a hardware engineer as CEO. The Mac's AI capabilities are only going up from here. We've experienced this trend firsthand building GUI Agents on Mac, and we're excited to see more developers explore this direction.

Google Released Android CLI Agent — Want to See an On-Device Agent on Mac?

Mininglamp — Mon, 20 Apr 2026 11:40:06 +0000

Google's Android team recently released a new CLI toolchain built for AI agents — packaging SDK management, project creation, and device debugging into streamlined commands with standardized Skills and a searchable Knowledge Base. It's a clear sign: on-device agents are moving from concept to production.

On mac, our open-source project Mano-P has been working on the same frontier — enabling AI agents to run locally and operate real GUI applications on your own machine.

What Is Mano-P?

Mano-P is an open-source, on-device GUI agent built by Mininglamp Technology, designed for macOS. It's based on a VLA (Vision-Language-Action) architecture.

"Mano" means "hand" in Spanish, and "P" stands for Person & Party — our vision is that every individual and organization can create their own personalized AI, running on their own hardware.

Mano-P is purely vision-driven: it understands screen content through visual models, plans action sequences, and executes operations via native OS input (mouse clicks, keyboard strokes). No system APIs or CLI access required — it can theoretically operate any GUI application on Mac.

Key Technical Features

think-act-verify Loop

Mano-P doesn't do single-pass inference. It uses a cyclical reasoning mechanism: observe → think → act → verify → repeat. After each action, the model re-examines the screen state to confirm success before deciding the next step. This enables complex workflows spanning dozens to hundreds of steps.

Edge Optimization

To run efficiently on consumer hardware, Mano-P uses mixed-precision quantization and the GS-Pruning algorithm for visual token compression. Performance of the 4B quantized model (w4a16) on Apple M4 Pro:

Prefill: 476 tokens/s
Decode: 76 tokens/s
Peak memory: 4.3 GB

A Mac mini or MacBook with an M4 chip and 32GB RAM can run Mano-P locally, with no cloud dependency.

Three-Stage Training

SFT → Offline RL → Online RL, with a bidirectional self-reinforcement framework (Text↔Action cycle consistency learning).

Benchmarks

Mano-P has achieved competitive results on public benchmarks:

Benchmark	Mano-P (72B)	Result
OSWorld (Specialized)	58.2%	#1
WebRetriever Protocol I	41.7 NavEval	#1

Data Privacy

In local mode, Mano-P keeps all screenshots and task descriptions on-device. Nothing leaves your machine. The full client code is open-source and auditable.

Three-Phase Open Source Plan

Phase 1 (released): Mano-CUA Skills for agent enthusiasts and Claude Code/OpenClaw users
Phase 2 (coming soon): Local model + SDK for developers with strict data security requirements
Phase 3 (planned): Training methods + pruning/quantization techniques

Get Started

Install via Homebrew:

brew tap HanningWang/tap && brew install mano-cua

GitHub: github.com/Mininglamp-AI/Mano-P (Apache 2.0)

The era of on-device agents is accelerating. Google took a step on Android; we've been building for Mac. If you're interested in local AI agents, give Mano-P a try and let us know what you think in the comments.

I Let an Open-Source GUI Agent Play Mahjong. Here's What Happened.

Mininglamp — Fri, 17 Apr 2026 10:57:52 +0000

The Challenge

Most GUI Agent demos show the same thing: open a browser, fill out a form, click "Submit." It works, it's useful, but it doesn't really stress-test what these agents can do.

We wanted to find out: what happens when you throw a GUI Agent into a completely unfamiliar, non-standard interface?

So we picked Mahjong — a Chinese tile game with complex rules, dense visual information, and a UI that has nothing in common with a typical web app.

Here's Mano-P playing:

Liquid error: internal

And the raw video:

https://github.com/user-attachments/assets/397a0552-9611-4d74-9f24-99544da272b6

What is Mano-P?

Mano-P (GUI-VLA Agent for Edge Devices) is an open-source project from Mininglamp Technology. The name comes from "Mano" (Spanish for "hand") and "P" for Person + Party.

The key differentiator: Mano-P is purely vision-driven. It doesn't parse DOM trees, doesn't use accessibility APIs, doesn't rely on OCR as a preprocessing step. It takes a screenshot, understands what's on screen, and outputs mouse/keyboard actions.

Think of it as an AI that operates a computer the same way you do — by looking at the screen.

🔗 Repo: github.com/Mininglamp-AI/Mano-P
📄 License: Apache 2.0

Why Mahjong is a Brutal Test Case

If you've played Mahjong, you know it's no joke. But even if you haven't, here's why it's an excellent stress test for a GUI Agent:

1. Dense, Visually Similar Elements

A Mahjong board has 136 tiles. Your hand has 13 tiles at a time. The tiles are small, visually similar (slight variations in dots, characters, bamboo patterns), and tightly packed. The agent needs pixel-level precision to distinguish a "3 of Dots" from a "5 of Dots."

2. Zero Structured Data

There's no HTML, no DOM, no accessibility tree. The game UI is rendered by a game engine — it's all pixels. This means any approach that relies on parsing page structure is out. Only pure vision works.

3. Strategic Reasoning Required

This isn't "see button, click button." The agent needs to:

Recognize all tiles in its hand
Evaluate possible winning combinations
Decide which tile to discard
React to other players' moves (pass, claim, or declare)

4. Asynchronous Multi-Player Flow

Mahjong is turn-based with 4 players. The agent has to wait for others, recognize when it's their turn, handle variable timing, and respond to unexpected events (another player declares a win, for instance).

How It Works: Think-Act-Verify

Mano-P doesn't just look once and act. It runs a continuous reasoning loop:

┌─────────┐     ┌─────────┐     ┌──────────┐
│  Think   │ ──▶ │   Act   │ ──▶ │  Verify  │
│ (analyze │     │(execute │     │(confirm  │
│  screen) │     │ action) │     │ result)  │
└─────────┘     └─────────┘     └──────────┘
      ▲                              │
      └──────── loop back ◀──────────┘

Think: Capture a screenshot, analyze the current game state. What tiles do I have? What's on the table? Is it my turn?
Act: Decide and execute an action — click a tile to discard, click "Pass," click "Claim."
Verify: Take another screenshot. Did my action register? Did the game state change as expected? If not, go back to Think.

This loop is critical for games, where animations, delays, and other players' actions create a constantly shifting interface.

Training Pipeline

Mano-P uses a three-stage training approach:

Stage	Method	Purpose
1	SFT (Supervised Fine-Tuning)	Learn basic GUI recognition and operation
2	Offline RL (Reinforcement Learning)	Optimize action policies from recorded trajectories
3	Online RL	Interactive learning in real environments

This progression moves from "imitate human actions" to "discover optimal strategies through exploration" — a pattern that's proven effective across many RL domains.

Benchmark Results

Numbers matter. Here's where Mano-P stands:

OSWorld (Desktop App Automation)

Model	Score
Mano-P 72B	58.2% (Rank #1 among specialized models)
opencua-72b	45.0%

WebRetriever Protocol I (Web Interaction)

Model	Score
Mano-P	41.7
Gemini 2.5 Pro	40.9
Claude 4.5	31.3

Edge Inference (4B Quantized, w4a16)

Running on Apple M4 + 32GB RAM:

Metric	Value
Prefill throughput	476 tok/s
Decode throughput	76 tok/s
Peak memory	4.3 GB

That's fast enough for real-time GUI interaction on a local device. No cloud API calls, no data leaving your machine.

Hardware note: The 4B model currently requires Apple M4 + 32GB RAM. Not all Macs can run it — be aware of this before trying.

What This Actually Means

The Mahjong demo is fun, but the real takeaway is about generalization.

Most GUI automation tools are brittle. Traditional RPA breaks when a button moves. DOM-based agents break when there's no DOM. Screen-scraping breaks when the UI updates.

A purely vision-driven agent doesn't have these dependencies. If a human can operate the application by looking at the screen, Mano-P can too — at least in principle. The Mahjong demo shows this isn't just theory:

✅ Non-standard UI? Handled.
✅ Visually dense interface? Handled.
✅ Strategic reasoning? Handled.
✅ Async multi-player flow? Handled.

The same architecture that plays Mahjong can automate legacy enterprise systems, operate desktop applications, or handle any GUI that doesn't expose a programmatic interface.

Open Source Roadmap

Mano-P is released under Apache 2.0 with a three-phase open-source plan:

Phase	Content	Status
Phase 1	Skills (core capabilities)	✅ Released
Phase 2	Local models + SDK	Coming soon
Phase 3	Training methodology	Planned

Try It Out

The project is live on GitHub. Whether you're interested in GUI automation, VLA research, or just want to see an AI play Mahjong, check it out:

👉 github.com/Mininglamp-AI/Mano-P

If you're working on GUI agents or have thoughts on vision-driven automation, I'd love to hear from you in the comments. What would you test a GUI Agent on?

Built by the open-source team at Mininglamp Technology. Mano-P is Apache 2.0 licensed — contributions welcome.

On-Device AI Agents vs Cloud AI Agents: Which Path Are You Betting On?

Mininglamp — Thu, 16 Apr 2026 11:22:29 +0000

Let me start with a question that's been bugging me lately:

Would you let an AI agent continuously stream your entire screen — emails, Slack DMs, browser tabs, documents — to a remote server?

If you hesitated, you've already identified the core tension in the AI Agent space right now.

Two Paths, One Goal

In 2026, the AI Agent world has split into two distinct camps.

Camp Cloud says: Throw the biggest models at the problem. 100B+ parameters, GPU clusters, infinite context windows. The raw intelligence approach.

Camp On-Device says: Run the model locally. Your data never leaves your machine. Trade some model size for privacy, speed, and zero marginal cost.

Both camps want the same thing — an AI that can actually use your computer for you. Open apps, fill forms, click buttons, extract data, automate workflows. The disagreement is about where the brain should live.

The Cloud Problem Nobody Talks About

Cloud-based GUI agents work like this: screenshot your screen → upload to cloud → model processes → send back instructions → repeat.

For a simple demo, this is fine. For daily use? Let's talk about the three elephants in the room.

1. Privacy Is Not a Feature — It's a Prerequisite

GUI agents need to see your screen. Everything on it. Your email drafts, your Slack conversations, your financial spreadsheets, your browser history. All of it gets uploaded to someone else's server for processing.

For individual developers? Maybe you're okay with that. For enterprise deployments? Compliance teams will shut this down before you finish the proposal deck.

2. Latency Compounds

A single cloud roundtrip might take 500ms. Sounds fast. But agents aren't single-shot — they're multi-step. A 10-step task means 10 roundtrips, and suddenly you're looking at 5+ seconds of cumulative network delay on top of inference time. That's the difference between "this feels instant" and "I could've done this faster myself."

3. Cost Scales Linearly (Your Patience Doesn't)

Vision model inference isn't cheap, especially with high-resolution screenshots. Every step costs tokens. Every retry costs tokens. Every mistake-and-recover costs tokens. Developers who prototyped with cloud APIs and then tried to run agents continuously were often surprised by the monthly bill.

The On-Device Bet

The on-device approach flips these trade-offs:

Privacy: Your screen data never leaves your machine. Period.
Latency: Local inference, no network roundtrip.
Cost: One-time setup, zero marginal cost per operation.

The catch? You need to fit a capable model into consumer hardware. And that's where things get technically interesting.

How Do You Fit an Agent Into a Laptop?

Three key techniques make on-device agents viable in 2026:

Quantization (W4A16)

Compress model weights from FP16 to 4-bit integers while keeping activations at FP16 precision. This cuts model size to roughly 1/4 while preserving most of the model's capability.

Real-world numbers on a 4B quantized model running on Apple M4 + 32GB RAM:

Metric	Value
Prefill speed	476 tok/s
Decode speed	76 tok/s
Peak memory	4.3 GB

Let that sink in. 4.3GB peak memory means your agent runs alongside your normal apps without breaking a sweat. 76 tok/s decode means action instructions are generated faster than you can read them.

Visual Token Pruning (GSPruning)

GUI screenshots are full of visual redundancy — blank areas, solid backgrounds, decorative elements. GSPruning identifies and removes low-information visual tokens before they hit the language model. The result: 30-50% fewer tokens to process, with minimal impact on task accuracy.

Pure Vision (No DOM, No API)

This is a deliberate architectural choice. Instead of parsing DOM trees or hooking into application APIs, on-device agents understand the screen purely through vision — the same way a human would.

Why? Because DOM parsing only works for web apps. Desktop applications, system dialogs, proprietary software — none of these expose DOM trees. A pure-vision agent can work with any interface a human can see and interact with.

"But Can Small Models Actually Do This?"

Fair question. Here's what the benchmarks say.

OSWorld

Approach	Score
72B model (pure vision)	58.2%
Runner-up	45.0%

That's a 13.2 percentage point gap on one of the most rigorous GUI agent benchmarks available.

WebRetriever

Model	Score
72B model	41.7
Gemini	40.9
Claude	31.3

Now, to be clear: the 72B model isn't what you'd run on your laptop. The deployment path is: 72B validates the architecture → knowledge distillation transfers capability to a 4B model → quantization makes the 4B model run on consumer hardware.

The 72B benchmarks prove the ceiling. The 4B quantized model is what you actually use.

The Trade-Off Matrix

Let me lay it out honestly:

	Cloud Agent	On-Device Agent
Raw capability	Higher (bigger models)	Lower (but closing the gap)
Privacy	Your data on their servers	Your data stays local
Latency	Network-dependent	Near-instant
Cost per use	Pay per token	Zero after setup
Offline support	No	Yes
Cross-app support	Varies (DOM/API dependent)	Any visible interface (pure vision)
Hardware requirement	Any device with internet	M4 + 32GB or equivalent
Setup complexity	API key	Local model deployment

Neither column is strictly better. It depends on what you're optimizing for.

An Open-Source Reference: Mano-P

We've been working on this problem at our team, and we open-sourced our on-device agent implementation as Mano-P under the Apache 2.0 license.

Key technical choices:

Pure vision approach (no DOM/API dependency)
W4A16 quantization for edge deployment
GSPruning for visual token efficiency
SFT + RL training pipeline
Native Apple Silicon support

We chose Apache 2.0 because we believe this space needs open collaboration. Restrictive licenses would only slow things down.

GitHub: https://github.com/Mininglamp-AI/Mano-P

What I Think (And Where I Might Be Wrong)

My current mental model: the future isn't purely cloud or purely on-device. It's a hybrid.

On-device handles the privacy-sensitive stuff — screen understanding, action execution, anything involving your personal data. Cloud provides optional capability boosts — complex multi-step reasoning, large-scale knowledge retrieval, tasks that genuinely need 100B+ parameters.

But I might be wrong. Maybe cloud providers will solve the privacy problem with confidential computing. Maybe on-device models will get good enough that cloud augmentation becomes unnecessary. Maybe a third path emerges that we haven't thought of yet.

The Discussion Part

I'm genuinely curious about how other developers and teams are thinking about this. A few questions I'd love to hear perspectives on:

1. Where's your privacy line? Would you use a cloud-based GUI agent for work? Personal use? Both? Neither?

2. Is the performance gap closing fast enough? 4B quantized models are usable today, but they're not as capable as cloud giants. Do you think the gap will close in 12 months? 24? Never?

3. What's your hardware reality? The benchmarks above use M4 + 32GB. That's not a budget machine. Is the hardware bar too high for on-device to go mainstream?

4. Pure vision vs DOM/API — which bet would you make? Pure vision is more general but harder. DOM/API is more reliable but limited in scope. Where do you land?

5. Does the license matter? Apache 2.0 vs GPL vs proprietary — does the licensing model affect whether you'd actually adopt an on-device agent?

Drop your thoughts in the comments. I'll be reading every one.

Disclosure: I work on the Mano-P project at Mininglamp Technology. All benchmark data cited is from public evaluations. This post represents our team's perspective, not an objective industry assessment.

Eyes and Hands for GUI Agents: How VLA Models Enable End-to-End Desktop Automation

Mininglamp — Wed, 15 Apr 2026 11:43:48 +0000

Eyes and Hands for GUI Agents: How VLA Models Enable End-to-End Desktop Automation

Most GUI automation today works by reading the app's internals — parsing HTML, querying the DOM, hooking into accessibility APIs. It works well... until you hit a native desktop app with no exposed interface.

At Mininglamp, we asked a different question: what if the model just looked at the screen, the way a human does?

That's the premise behind GUI-VLA (Vision-Language-Action), and we open-sourced our implementation as Mano-P.

What is GUI-VLA?

VLA comes from robotics — a robot sees the world through cameras, understands a spoken command, and moves its arms to act. GUI-VLA applies the same idea to screen automation:

Vision: the input is a raw screenshot
Language: the model understands your natural language instruction
Action: the output is a concrete GUI operation — click at (x, y), type text, scroll, drag

The pipeline is straightforward:

Screenshot → Visual Encoding → Language Understanding → Action Output

No CDP protocol. No HTML parsing. No accessibility tree. Just pixels in, actions out.

This means it can operate any application with a graphical interface — including native macOS apps, legacy desktop software, and cross-application workflows that no single API can bridge.

How We Train It: Three Stages

Getting a model to reliably operate GUIs from screenshots is hard. A single wrong click can cascade into a completely wrong state. Here's our three-stage training approach:

Stage 1: Supervised Fine-Tuning (SFT)

We start with large-scale supervised learning on (screenshot, instruction, correct action) triplets. This teaches the model the basics — what buttons look like, where text fields are, how menus work.

Stage 2: Offline Reinforcement Learning

SFT only shows the model correct actions. Offline RL introduces negative examples and reward signals, teaching the model to distinguish good actions from bad ones without live interaction.

Stage 3: Online Reinforcement Learning (Mano-Action)

The final stage. The model interacts with real environments, receives actual feedback, and refines its policy. We call this the Mano-Action method. This is where the model develops genuine error recovery skills.

Why three stages? Because GUI operations have cascading errors. Click the wrong button, and every subsequent step happens in the wrong context. SFT alone can't handle this. RL builds the judgment to recover from mistakes.

Think → Act → Verify

For complex, multi-step tasks, we use a Think-Act-Verify reasoning loop:

Think: Analyze the current screenshot, understand the state
Act: Execute the next operation
Verify: Check if the result matches expectations

If verification fails, the model loops back to Think and replans instead of blindly continuing. This is critical for long task chains like "gather data from three apps and compile a report."

Benchmark Results

Let's talk numbers.

OSWorld (Specialized Models)

OSWorld evaluates desktop OS-level GUI automation. Among specialized models:

Model	Score
Mano-P 72B	58.2%
opencua-72b	45.0%

That's a 13.2-point gap over the second-place model.

WebRetriever Protocol I

On web information retrieval:

Model	Score
Mano-P	41.7
Gemini 2.5 Pro	40.9
Claude 4.5	31.3

Note that Gemini 2.5 Pro and Claude 4.5 are flagship general-purpose models. A specialized GUI-VLA model outperforming them on this task suggests that purpose-built architectures still have an edge in vertical scenarios.

Running It on Your Mac

One of the things we're most excited about: the 4B quantized model runs locally on Apple Silicon.

Metric	Value
Prefill speed	476 tok/s
Decode speed	76 tok/s
Peak memory	4.3 GB
Hardware	M4 chip + 32GB RAM

4.3 GB peak memory means you can run it as a background service without impacting your daily workflow. Your data stays on your machine — no cloud uploads required.

The secret sauce here is GSPruning — visual token pruning that removes tokens corresponding to unimportant screen regions (blank backgrounds, decorative elements). This gives us a 2-3x speedup without meaningful accuracy loss.

What Can It Actually Do?

Based on the current capabilities:

Complex GUI Automation: Multi-step interface operations across applications
Cross-System Data Integration: Moving and combining data between different apps
Long-Task Planning: Workflows that require multi-step reasoning and planning
Report Generation: Extracting information from interfaces and producing structured outputs

The Honest Limitations

We believe in transparent communication about what works and what doesn't:

In highly standardized web scenarios, API-based approaches (CDP + DOM) can still be more reliable than pure vision
Screenshot resolution and interface complexity affect recognition accuracy
There's a capability gap between the 4B edge model and the 72B cloud model

GUI-VLA isn't here to replace API-based agents. It's here to handle everything those agents can't reach.

Try It Yourself

Mano-P is fully open source under Apache 2.0:

GitHub: github.com/Mininglamp-AI/Mano-P
Paper: arxiv.org/abs/2509.17336

We'd genuinely love feedback — issues, PRs, or just telling us where it falls short.

What do you think about vision-only GUI agents? Is the pure-vision approach the future, or will API-based methods always win in structured environments? Drop your thoughts in the comments — we're especially curious to hear from anyone who's tried building GUI agents in production.

How Mano-P Achieves #1 on OSWorld: Architecture, Benchmarks, and Edge Deployment

Mininglamp — Tue, 14 Apr 2026 12:15:51 +0000

Open-source GUI agents have been gaining traction, but most still rely on cloud inference, DOM parsing, or CLI hooks. Mano-P takes a different approach: pure vision-driven GUI automation that runs entirely on edge devices. And the benchmark results back it up — #1 on OSWorld among specialized models.

This article breaks down the architecture, benchmark data, and edge deployment performance, all from the project's public README and technical report.

The Numbers

OSWorld: #1 Among Specialized Models

OSWorld is the standard benchmark for GUI agent evaluation. Mano-P 1.0-72B achieves a 58.2% success rate, ranking first among all specialized GUI agent models. For context:

Model	OSWorld Success Rate
Mano-P 1.0-72B	58.2%
opencua-72b	45.0%
Gap	+13.2 percentage points

That's not a marginal improvement — it's a 29% relative gain over the second-place model.

WebRetriever: Beating Cloud Giants

On the WebRetriever Protocol I benchmark, Mano-P scores 41.7 NavEval, which puts it ahead of:

Model	NavEval Score
Mano-P 1.0	41.7
Gemini 2.5 Pro Computer Use	40.9
Claude 4.5 Computer Use	31.3

Worth noting: Gemini and Claude are cloud-based services with massive compute budgets. Mano-P achieves comparable or better results while running on local hardware.

13 Benchmarks, SOTA Across the Board

Beyond OSWorld and WebRetriever, Mano-P holds SOTA positions across 13 benchmarks spanning GUI grounding, perception & cognition, context learning, and pruning efficiency. The full benchmark data is available in the README.

The Architecture: Why Pure Vision?

Most GUI agents fall into one of these categories:

Approach	How It Works	Limitation
DOM/HTML Parsing	Read page structure directly	Web-only, breaks on native apps
CDP + CLI	Chrome DevTools Protocol + shell commands	Browser-dependent, fragile
Cloud Computer Use	Send screenshots to cloud API	Privacy concerns, latency, API costs
Pure Vision (Mano-P)	See the screen, understand it, act	Requires capable on-device model

Mano-P chose pure vision. No DOM access, no browser hooks, no platform-specific APIs. The model looks at the screen — the same pixels a human sees — and decides what to click, type, or scroll.

This is harder to build, but the payoff is generality: the same model works across any GUI application, any platform, without integration work per app.

Training Methodology: Mano-Action

The technical backbone is Mano-Action, a bidirectional self-reinforcement learning framework. The training follows three stages:

Stage 1: Supervised Fine-Tuning (SFT)
Starting from a base vision-language model, fine-tune on curated GUI interaction datasets.

Stage 2: Offline Reinforcement Learning
Learn from recorded interaction trajectories, optimizing action quality without live environment access.

Stage 3: Online Reinforcement Learning
The model interacts with real GUI environments, receiving feedback and iterating. This is where the "think-act-verify" loop reasoning mechanism comes in — the model plans an action, executes it, verifies the result, and adjusts.

The bidirectional aspect means Text→Action and Action→Text consistency are both optimized, creating a tighter loop between understanding and execution.

Edge Optimization: Running on Apple M4

The 72B model delivers SOTA benchmarks, but the edge story is equally important. Through mixed-precision quantization and a novel visual token pruning technique called GSPruning, Mano-P achieves practical performance on consumer hardware.

GSPruning: Preserving What Matters

GSPruning (Global Spatial Pruning) is designed specifically for vision-language models processing high-resolution interfaces. It:

Preserves global spatial structure through anchor points
Identifies semantic outliers for critical UI elements
Achieves 2-3× throughput speedup with minimal performance loss

On the Online-Mind2Web benchmark, GSPruning at 25% token retention achieves a success rate of 0.400 on Qwen3VL-4B, compared to 0.425 at full tokens — only a 6% drop while running significantly faster.

M4 Pro Performance

The 4B quantized model (w4a16) on Apple M4 Pro with 64GB RAM:

Metric	Value
Prefill Speed	476 tokens/s
Decode Speed	76 tokens/s
Peak Memory	4.3 GB
Prefill Time (4K context)	8.6s

4.3 GB peak memory means this runs comfortably alongside other applications. No dedicated GPU server required.

Hardware Requirements

Two deployment options:

Direct: Mac mini or MacBook with Apple M4 chip, 32GB+ RAM
Computing Stick: Any Mac + Mano-P computing stick via USB 4.0+

Data Privacy: The Edge Advantage

In Local Mode, all processing happens on-device:

✅ Screenshots never leave the device
✅ Task descriptions stay local
✅ No cloud API calls
✅ Full source code is open for audit

Cloud Mode is available as a fallback (screenshots sent to mano.mininglamp.com), but the local-first architecture means sensitive workflows can run with zero data exposure.

Getting Started

Three usage forms are available:

CLI (for terminal users):

brew tap HanningWang/tap
brew install mano-cua

mano-cua run "Open WeChat and tell FTY the meeting is postponed"

Python SDK (planned):

from mano_client import ManoClient
client = ManoClient()
client.run("Search for AI news on Xiaohongshu")

ClawHub Skill (for AI agents):

clawhub install mano-cua

The Skill form is designed for AI agents like Claude Code or OpenClaw — the agent automatically invokes Mano-P when GUI operations are needed.

What's Next

Mano-P is being released in three phases:

Phase 1 (now): Mano-CUA Skills — for agent enthusiasts to build CUA task workflows
Phase 2 (coming): Local models + SDK — for developers with high security requirements
Phase 3 (planned): Training methods + pruning techniques — for researchers who want to train their own GUI-VLA models

The project is Apache 2.0 licensed. Full source, benchmarks, and documentation: github.com/Mininglamp-AI/Mano-P