DEV Community: Mininglamp

Mano-CUA 2.0: After a Year of Building a 4B GUI Agent, We Found the Bottleneck Was Never Model Size

Mininglamp — Mon, 29 Jun 2026 07:12:14 +0000

Mano-P is an open-source project we've been working on. It runs a 4B-parameter vision-language model on a MacBook, controlling the computer by watching screenshots. Clicking, typing, hotkeys — the model looks at each screenshot and decides what to do next. Everything stays on-device, no cloud API calls. We shipped 1.0 a while back and recently iterated to 2.0, re-running our 100 real macOS GUI task benchmark. Some of the results surprised us.

We settled on 4B mostly because of the 16GB MacBook memory constraint. We tried general-purpose VL models too — on our benchmark they scored around 39%, with issues around Chinese input focus, non-browser app support, and step-limit truncation on longer tasks. General VL capability and GUI agent capability turned out to be pretty different things. Internally we also debated MoE, but on-device inference is memory-bound more than compute-bound, and MoE keeps all expert weights resident in memory. That's a bad trade on a Mac. We stayed on MLX — it maps directly to Apple Silicon's unified memory, which is about as good as it gets for local inference on these chips.

That's the background. The interesting part is what happened in 2.0.

Chinese GUI went way up

We'd been assuming 4B models just couldn't handle GUI agents well — too small. After finishing 2.0, we realized that wasn't the story. The first real bottleneck was Chinese GUI training data.

Version 1.0 was pretty bad at Chinese UI elements. The model kept making character-level recognition errors, confusing one Chinese character for a visually similar one. These mistakes sound minor, but for a GUI agent they're fatal. If you can't find the button, nothing downstream matters.

English interfaces didn't have this problem. "Settings" and "Settinas" look very different. Chinese is different — characters have higher visual similarity to each other, and buttons like "确定" or "取消" pack dense strokes into a small area, making them harder for the model to distinguish. At the time we thought this was a 4B capacity issue. Chinese characters are just harder than English letters, small models can't handle it.

After adding more Chinese GUI training data in 2.0, it turned out that wasn't the case. The fix wasn't fancy — just increasing the volume of Chinese interface screenshots in the training set. The results showed up immediately in the enterprise IM category: 33% on 1.0, 83% on 2.0. WeChat went from 33% to 67%.

Fifty percentage points. Looking back, 1.0's data coverage was simply insufficient. This was solvable with more data — no architecture changes, no larger model needed. We'd spent a lot of time debating model selection when the bottleneck was in the data. Kind of embarrassing, honestly.

Browser tasks went down

Version 2.0 dropped from 74% to 68% on browser and web tasks. When this came up internally there was some debate about whether we'd overdone the Chinese data.

Looking at the breakdown, the decline was concentrated in English web element recognition. Chinese web tasks barely changed. After rebalancing the training mix toward Chinese, the model's sensitivity to English UI elements dropped a bit.

The sample sizes aren't huge, so we don't think this represents actual capability regression. But it exposed a problem we haven't solved: balancing Chinese and English GUI data. The 4B model has limited capacity — cram too much in and things interfere with each other. Right now we're doing volume-based control where more Chinese means less English. Ideally both would be sufficient, but at this model size that's not achievable.

Short term we're planning finer-grained sampling strategies, weighting by task type and UI complexity rather than just language. Whether that works, we're not sure. This might need a larger model to have enough room.

Long-horizon tasks still stuck at 30%

Both 1.0 and 2.0 scored 30% on long-horizon tasks — those requiring 10+ steps. No improvement. Cross-app tasks went from 0% to 20%, passing 1 out of 5.

From what we've tested, this looks more like a 4B capacity constraint. Tasks with 10+ steps require maintaining context throughout, with each step's results feeding correctly into the next decision. The 4B model's working memory is limited — by step 10, it's already fuzzy on what happened in the first few steps. We haven't found a way to push past this with data alone.

Cross-app is even harder. Find a file someone sent in WeChat, switch to Finder, save it to the desktop. One window switch and the 4B model tends to lose the previous context. We expect a larger model would help here, but haven't verified it.

Internally we're weighing two paths. One is adding more reasoning-chain training data to the 4B and seeing how far that goes. The other is building a larger model — 7B or 8B — that would run on an M5 Pro with 64GB. Haven't decided. Honestly still not sure which path is worth the investment.

Cider: the latency problem

Everything above is about model capability. Cider solves something different — users can't wait.

GUI agents have an unusual performance characteristic. After every action, you take a fresh screenshot and feed the whole image's tokens through the model. That prefill stage directly determines how long the user waits. On an M5 Pro, MLX's native W8A16 mode gives us 2.839 seconds for prefill.

2.8 seconds doesn't sound like much. But over a 10-step task, waiting nearly 3 seconds at each step adds up to half a minute. Users feel that as sluggish. Decode speed isn't the issue — 80 tokens/s is plenty for generating action commands.

The problem was that MLX doesn't ship with online activation quantization operators. Weights are static and can be quantized offline. But activations are dynamic — every screenshot produces different intermediate values going through the network. MLX doesn't provide that capability, so we wrote our own.

The trickiest design decision was quantization granularity. Too coarse and outlier activation values drag down overall accuracy. Too fine and compute overhead goes up. We landed on per-token granularity — each token's activation vector gets its own quantization parameters computed independently. More overhead than per-tensor, but accuracy loss stays manageable.

Results: W8A8 prefill dropped from 2.839s to 2.519s, about 12.7% faster. Peak memory also went down, which matters for GUI agents that need to coexist in memory with the user's other applications. We haven't measured the exact memory savings yet.

M5 chips have hardware acceleration so the improvement is significant. M4 and below fall back to pure Python, which gives limited speedup. We evaluated optimizing specifically for M4 and decided the return wasn't worth the effort. Cider isn't Mano-P-specific — any MLX model can use it.

What's next

Fuzzy descriptions and cross-app tasks are the two clearest weak points, both pointing at the 4B model's reasoning capacity. We'll likely build a larger version. Cider continues with stability and compatibility work.

Full category breakdown below. Test hardware: MacBook Pro M5 16GB. Cloud model: Claude Sonnet 4.5. Local model: Mano-CUA-4B W8A16.

Category	Tasks	1.0	2.0	Cloud
Browser/Web	31	74%	68%	90%
Enterprise IM	6	33%	83%	100%
WeChat	6	33%	67%	83%
WPS/Office	5	0%	40%	100%
System Settings	6	50%	83%	50%
Notes/Reminders	4	50%	50%	100%
File Management	7	43%	43%	71%
System Utilities	3	100%	100%	100%
Long-horizon	10	30%	30%	80%
Cross-app	5	0%	20%	60%
Fuzzy descriptions	10	30%	30%	80%
No open hint	5	20%	60%	80%

https://github.com/Mininglamp-AI/Mano-P

Mininglamp Technology Officially Open-Sources Octo: Building a New-Generation Platform for Human-AI Agent Collaboration

Mininglamp — Mon, 29 Jun 2026 07:02:55 +0000

Today, Mininglamp Technology officially releases Octo — the first open-source, trustworthy Agent collaboration network that pioneers a new paradigm for human-AI teamwork. Octo supports private deployment, returning data and knowledge sovereignty to enterprises and users. By transforming isolated AI Agents into coordinated, orchestrable, and tasteable organizational digital workforce, Octo turns every human-machine collaboration into a node for compounding organizational assets, driving continuous evolution of Agents and systems under human judgment calibration.

As more intelligent agents emerge in personal devices and organizational workflows, new challenges arise: When everyone has their own AI assistant, when digital workforce proliferates within organizations, how should they connect, collaborate, and share critical context? How should they accept human judgment and calibration at key decision points?

Mininglamp believes the core challenge for AI Agents in the next phase is not endlessly scaling model parameters or building a single super-agent, but enabling different Agents to work together in the same network. What Octo aims to build is precisely "the internet between Agents."

Octo repository: https://github.com/Mininglamp-OSS

From Personal Assistant to Organizational Collaboration Network

In traditional AI tool usage, Agents typically exist as isolated silos. They maintain separate memories, execute independently, and lack unified collaboration interfaces and task flow mechanisms, making it difficult to accumulate capabilities, reuse experience, or truly scale AI adoption across organizations.

Octo breaks this deadlock. Through collaboration architectures like Channels and Threads, Octo builds a foundational network for humans and AI — as well as AI and AI — to work together. A Channel is essentially a project workgroup where humans and Bots can align intentions and dispatch tasks in real-time.

When a Channel contains multiple discussion topics, both humans and Agents can create multiple Threads within it to focus on specific subjects, ensuring concrete work threads don't get washed away by information noise, guiding discussions toward natural convergence.

In Octo, AI Agents join teams as Bots. Users can conveniently integrate mainstream tools like OpenClaw, Hermes, Codex, and Claude Code into Octo, creating dedicated digital twin Bots while enabling deep Agent-to-Agent (A2A) collaboration. Each Bot has its own AgentCard and work history, with clear ownership and accountability.

To transform fragmented discussions into traceable, measurable work outcomes, when actionable work emerges from discussions, Agents automatically summarize key points and create Matters upon human confirmation. Matters specify task owners and concrete deliverables, with detailed records from Brief through process discussions, outputs, feedback, to acceptance conclusions — all preserved for future review and decision traceability.

For complex tasks, Octo provides six collaboration modes: Solo (individual completion), Roundtable (group discussion), Critic (independent review), Pipeline (sequential workflow), Split (parallel division), and Swarm (competitive selection). By precisely controlling how Context information flows between Bots and what's visible to each participant, Octo enables multiple specialized Bots to conduct distributed collaboration under human guidance, allowing collective intelligence to emerge through network effects that surpass any single model.

"I Taste Therefore I Am": A New Division of Labor in Human-Machine Collaboration

What's truly being restructured in the AI era isn't just tools, but collaboration itself. In the future, collaboration will frequently occur between humans and humans, humans and Agents, and Agents and Agents. Under this new paradigm, the human-machine division of labor reaches a turning point: AI excels at "thinking" and "doing" — handling logical reasoning, analysis, generation, and execution; while human irreplaceability focuses on "tasting" — making holistic judgments based on experience, aesthetics, trade-offs, and values.

Octo is designed around this principle: Let Agents execute, let humans return to the core position of judgment and taste. At key nodes, humans provide direction, standards, and feedback — judging what's right and what's good; AI drives tasks to completion.

With every human-machine collaboration, human taste drives the accumulation of organizational assets, making Bots smarter over time.

During collaboration, project background knowledge, historical decisions, and discussion records are structurally preserved in Matters, allowing new members to onboard without starting from zero alignment. Every rejection, annotation, and style choice humans make when reviewing Bot outputs gets recorded as preference cards, enabling Bots to automatically reference them in future tasks. The standards and methods Bots learn can also be preserved as reusable Skill assets within the organization.

Through the asset accumulation flywheel of "dispatch tasks → review feedback → accumulate preferences and skills → greater efficiency next time," Octo builds a unique positive cycle, naturally enriching organizational productivity infrastructure with every collaborative interaction, achieving true capability accumulation and intelligent upgrades.

Open Source and Open: Not Replacing Tools, But Connecting Them

Octo is open-sourced under Apache License 2.0 and supports private deployment. Mininglamp believes that in an era of rapid AI development, enterprises' true long-term competitiveness stems from their unique work context, business knowledge accumulation, and organizational judgment.

Octo is precisely positioned as the "collaboration layer" between an enterprise's existing documentation, spreadsheets, code repositories, and project management platforms. Through cross-platform capabilities like browser extensions, Octo can seamlessly bring current webpage content, selected fragments, and task information into the collaboration network, helping digital twins fully understand the current work environment, standing by beside existing tools for efficient coordination.

In terms of product form, Octo comprehensively covers Web App, desktop client, mobile (iOS/Android), browser extension, and CLI — four endpoints meeting different work scenario needs. Whether pushing forward complex projects on desktop, quickly handling notifications and taste feedback on mobile, or providing native operations for Agents through CLI, seamless multi-device interoperability is achieved.

Moving Toward Private AI Through Trustworthy Mechanisms

Octo's open-source release is also Mininglamp's further practice in Private AI and Trustworthy AI.

Mininglamp firmly believes that truly sustainable AI collaboration must guarantee users' absolute control over data, context, judgment signals, and deployment methods. Through open-source architecture, private deployment, and clear data ownership design, Octo ensures enterprises can embrace AI within security boundaries while protecting individuals' tacit knowledge.

In Octo's product philosophy, the four letters "O.C.T.O." represent four inseparable dimensions: Open (open access), Context (context sharing), Taste (preference evolution), and Orchestration (multi-Bot coordination).

Context is the soil for AI to understand tasks; Taste is the compass for AI to continuously calibrate direction. Octo doesn't simply distill human tacit capabilities into platform assets, but rather amplifies, records, and passes on these capabilities while respecting personal and organizational data boundaries.

Mininglamp is continuously improving its new-generation AI infrastructure oriented toward edge intelligence, private deployment, and human-machine collaboration. By fully preserving teams' background knowledge, work preferences, and methodologies in the network, Octo ensures organizational wisdom doesn't drain with personnel turnover, and business style doesn't change with foundation model iterations. Every human-machine hybrid collaboration is compound interest accumulation on organizational private assets. Over time, this unique business perception naturally transforms into enterprises' most competitive technical and scenario barriers.

In the future, Octo will continue with an open-source, open attitude, co-creating new collaboration paradigms for AI-Native organizations with developers, enterprise customers, and ecosystem partners, making trustworthy, controllable, and sustainable private Agentic AI truly land in every real work scenario.

AI Agents and Persistent Context: What design.md Teaches Us

Mininglamp — Fri, 26 Jun 2026 09:45:07 +0000

A GitHub repository called design.md has been trending recently, accumulating over 1,400 stars. The concept is straightforward: provide AI agents with a persistent design document they can reference throughout their work.

This approach addresses a practical challenge in agent development that many teams encounter.

The Context Challenge
When working on complex tasks, AI agents need to understand the broader picture. What's the architecture? What constraints exist? What approaches have been tried before?

Typically, agents get context from:

Current conversation (limited window)
Code comments (often outdated)
Documentation (if it exists)
The issue is that this context is fragmented and temporary. When conversation moves forward, earlier context disappears. When documentation is outdated, agents make incorrect assumptions.

A design.md provides a single source of truth that persists across sessions.

What Belongs in design.md
An effective design.md answers these questions:

What are we building?

Beyond feature lists, document the core purpose. Why does this project exist? What problem does it solve?

What are the key architectural decisions?

Document major choices and their rationale:

"PostgreSQL was chosen over MongoDB because ACID guarantees are required for financial transactions"
"Microservices architecture was adopted because components have different scaling requirements"

What constraints exist?

Technical constraints (performance requirements, browser support), business constraints (budget, timeline), and regulatory constraints (GDPR, HIPAA).

What has been tried before?

Document failed approaches to prevent agents from suggesting rejected solutions.

What are the current challenges?

Known issues, technical debt, areas needing improvement help agents prioritize work.

How Agents Use design.md
When starting a task, agents can:

Read design.md to understand context
Make decisions aligned with documented architecture
Avoid solutions violating constraints
Reference design.md in reasoning
This leads to more coherent and consistent work. Agents work within a broader framework rather than just reacting to immediate tasks.

Keeping design.md Updated
The main risk with design.md is becoming outdated. Effective practices include:

Make it part of the workflow

Update design.md immediately when making significant architectural decisions. Waiting until "later" means it never happens.

Version control it

Keep design.md in the repository. During PR reviews, check if design.md needs updating.

Review it regularly

Schedule periodic reviews (monthly or quarterly) to ensure the document reflects current reality.

Let agents help

Agents can assist in maintaining design.md by:

Suggesting updates when noticing inconsistencies
Summarizing changes from recent commits
Flagging outdated information
Observability in Agent Workflows
Even with good design.md, observing what agents actually do is important. This is particularly relevant for GUI agents interacting with complex interfaces.

Consider a GUI agent tasked with "fill out this form and submit it". The agent needs to:

Locate form fields
Enter correct data
Handle validation errors
Submit the form
Verify success
Each step can fail in different ways. Without observability, only the final result is visible: success or failure. The reason for failure remains unknown.

Building Observable Workflows
Good observability includes:

Step-by-step logging

Record each action:

What was observed (screenshots, DOM state)
What decision was made
What actually happened
Whether it matched expectations

Performance metrics

Track:

Success rate per task type
Average steps to completion
Time per step
Failure modes

Error categorization

When things go wrong, categorize errors:

Perception errors (agent didn't see the right element)
Decision errors (agent chose wrong action)
Execution errors (action failed due to external factors)
This data helps identify where improvements are needed.

Systematic Benchmarking
CUA Benchmark provides systematic observability through:

100 test cases across 5 different web applications
Standardized task definitions
Automated result verification
Detailed performance metrics
Running agents against CUA Benchmark provides quantitative data:

Overall success rate
Success rate by task type
Average steps per task
Common failure points
This data is valuable for iterative improvement. Instead of guessing what to optimize, specific areas where agents struggle can be identified and addressed.

A Practical Example: Mano-AFK
Mano-AFK is an open-source autonomous application builder that demonstrates these principles. The workflow includes:

Receiving natural language description of what to build
Generating a PRD (Product Requirements Document)
Writing the code
Deploying to a test environment
Running tests (lint, API, E2E)
Auto-fixing any issues
Delivering the final application
Throughout this process, the agent references rules.md and preferences.md files to maintain consistency across projects. These files provide persistent context that guides decisions.

CUA Benchmark results for Mano-AFK:

W8A16 quantization: 58.0% accuracy
W8A8 quantization (Cider): 54.0% accuracy, but faster inference (~1,453 tok/s prefill)
These numbers show that W8A8 version is slightly less accurate but significantly faster. Depending on use case, one might be preferred over the other.

Without systematic benchmarking, this data wouldn't exist. Only vague impressions like "it works sometimes" or "it's kind of slow" would remain.

Practical Recommendations
When building agent workflows, these practices have proven effective:

Start with design.md

Before writing agent code, document architecture, constraints, and key decisions. This document guides both human developers and AI agents.

Build observability from day one

Don't add logging later. Design agent workflows to be observable from the start. Every step should produce some form of output that can be inspected.

Use benchmarks

Establish a benchmark suite early. Run it regularly. Track metrics over time. This provides objective data on whether changes are improvements or regressions.

Iterate based on data

When low success rates are observed, examine failure modes. Instead of making broad changes, identify specific failure patterns and address them directly.

Keep context persistent

Whether through design.md, rules.md, or other mechanisms, ensure agents have access to persistent context. Conversation history is too ephemeral for complex projects.

Moving Forward
AI agent engineering is still in early stages. Best practices are still being figured out, but two things are becoming clear:

Agents need persistent context to do good work (design.md)
Agent workflows need systematic observability to improve (benchmarks and logging)
These aren't advanced techniques. They're foundational practices that make everything else work better.

If you're interested in seeing these principles in action, Mano-AFK (https://github.com/Mininglamp-AI/Mano-AFK) is an open-source autonomous application builder that uses persistent context files and systematic benchmarking to improve agent reliability.

For those working on GUI agents, Mano-P (https://github.com/Mininglamp-AI/Mano-P) implements think-act-verify loops and online reinforcement learning, achieving 58.2% success rate on the OSWorld benchmark (specialized models category).

Both projects are Apache 2.0 licensed. Stars and contributions are welcome.

After MCP, What's the Next Standard Interface for AI Agents?

Mininglamp — Thu, 25 Jun 2026 10:39:31 +0000

Model Context Protocol solved a real problem. Before MCP, every tool integration required custom glue code. After MCP, agents can talk to databases, APIs, and services through a standard interface. The protocol gave us a common language for agent-tool communication, and the ecosystem responded with thousands of MCP servers.

But there’s a category of interaction that MCP doesn’t address: graphical user interfaces. Agents still struggle to operate desktop applications, web interfaces, and mobile apps with the same fluency they show when calling APIs. The problem isn’t intelligence. It’s the lack of a standard way for agents to perceive and manipulate GUIs.

Three approaches have emerged to close this gap. Each makes different engineering tradeoffs, and none has achieved the kind of adoption that MCP has seen. Understanding these tradeoffs matters if you’re building agents that need to interact with the visual world.

The GUI Agent Problem

APIs are structured. They accept typed parameters, return predictable responses, and document their behavior. GUIs are none of these things. A button might be labeled “Submit,” “Save,” or “OK.” It might be positioned differently on different screen sizes. Its availability might depend on the state of three other form fields. The visual representation is an abstraction layer over underlying state, and that abstraction was designed for humans, not machines.

An agent operating a GUI needs to do three things: perceive the current interface state, decide what action to take, and execute that action. The perception step is where approaches diverge. How does the agent “see” the interface?

Approach 1: API Integration

The most straightforward approach is to bypass the GUI entirely and use APIs when they exist. Many desktop applications expose scripting interfaces. Web applications often have REST APIs or GraphQL endpoints. Mobile apps sometimes support URL schemes or accessibility hooks that can be invoked programmatically.

This approach works well when APIs are available and well-documented. An agent can call create_document(), pass structured parameters, and receive a confirmation response. The interaction is fast, reliable, and doesn’t require visual processing.

The limitation is coverage. Most consumer applications don’t expose comprehensive APIs. Even when APIs exist, they often don’t cover all the functionality available through the GUI. A photo editing app might have an API for basic operations but not for advanced filters. A web application might have a public API for reading data but not for complex workflows that require multi-step form submissions.

API integration also doesn’t help with legacy systems, proprietary software, or applications where the API has been deprecated or never built. The agent becomes dependent on the goodwill of application developers to expose the functionality it needs.

Some frameworks try to extend API coverage by wrapping GUI automation libraries. Python’s pyautogui, Apple’s AppleScript, and Windows UI Automation all provide programmatic access to GUI elements. These tools work, but they’re fragile. They depend on element identifiers, window titles, and UI hierarchies that change between application versions. A script that works today might break after a software update.

Approach 2: Accessibility Tree

Operating systems maintain accessibility trees: structured representations of UI elements designed for screen readers and assistive technologies. These trees contain information about buttons, text fields, menus, and their current states. An agent can query the accessibility tree to understand what elements are present and what actions they support.

This approach is more robust than raw API integration because accessibility trees are standardized at the OS level. macOS uses the Accessibility API, Windows uses UI Automation, and web browsers expose the DOM through accessibility interfaces. An agent that understands these APIs can operate across many applications without custom integration for each one.

Google’s approach with Gemini 3.5 Flash Computer Use leans heavily on accessibility trees. The model can query the accessibility tree of a web page or application, identify interactive elements, and generate actions to manipulate them. This works well for structured interfaces like web forms, file managers, and settings panels.

The accessibility tree approach has limits. Not all applications expose complete accessibility information. Custom-drawn interfaces, games, and applications with non-standard UI frameworks often have sparse or inaccurate accessibility trees. Canvas-based rendering, WebGL content, and video players present particular challenges because their visual content doesn’t map cleanly to accessibility nodes.

Accessibility trees also miss visual information that matters for human-like interaction. The tree might tell you a button exists and is labeled “Save,” but it won’t tell you that the button appears grayed out, that a tooltip is hovering over it, or that a dialog box has appeared in the foreground. These visual cues inform human decision-making, and agents operating purely on accessibility data miss them.

Approach 3: Pure Vision

The third approach treats the screen as an image and uses computer vision to understand it. The agent takes screenshots, processes them through a vision-language model, and generates mouse and keyboard actions based on what it sees. This is the most general approach because it works with any GUI, regardless of whether APIs or accessibility trees exist.
Pure vision agents don’t depend on application-specific integrations. They see the screen the way a human sees it: pixels arranged in patterns that represent buttons, text, images, and layouts. A vision-language model can identify a “Submit” button by its visual appearance, even if the button has no accessibility label or API endpoint.

The tradeoff is computational cost and latency. Processing a screenshot through a vision model takes time. A single inference might require 200-500 milliseconds, and complex interfaces might need multiple inference steps to parse correctly. This makes pure vision agents slower than API-based approaches, where actions execute in milliseconds.

Memory and compute requirements are also higher. Vision-language models need to process high-resolution images, which consume significant VRAM or RAM. Running these models on edge devices—laptops, desktops, mobile phones—requires careful optimization.

Mano-P takes the pure vision approach and optimizes it for edge deployment. The model is a GUI-VLA (Visual-Language-Action) agent designed to run locally on consumer hardware. The 72B parameter version achieves 58.2% success rate on OSWorld benchmarks, ranking first among specialized GUI agents. But the 72B model is primarily for research and benchmarking. The 4B quantized version is what users can actually deploy: it runs with 4.3GB of peak memory usage, achieving 476 tokens per second during prefill and 76 tokens per second during decoding on Apple M4 hardware with 32GB RAM.

The performance characteristics matter for practical deployment. An agent that takes two seconds to process each screenshot creates a sluggish user experience. An agent that processes screenshots in under 200 milliseconds feels responsive. The 4B model’s throughput makes real-time GUI interaction feasible on commodity hardware, which changes where GUI agents can be deployed.

Pure vision also handles edge cases that other approaches struggle with. Custom-drawn interfaces, games, video content, and applications with non-standard UI frameworks all render to pixels. A vision agent can operate them without requiring special integration work.

Engineering Tradeoffs

Choosing between these approaches depends on your deployment context.
API integration is fastest and most reliable when APIs exist. It’s the right choice for agents that operate within well-defined software ecosystems where API coverage is comprehensive. If you’re building an agent that automates workflows in Salesforce, Jira, and Slack, API integration makes sense. The agent will be fast, predictable, and easy to debug.

Accessibility tree approaches offer broader coverage with reasonable performance. They work well for agents that need to operate across many applications but don’t require pixel-perfect visual understanding. Web automation, form filling, and menu navigation are good fits. The approach breaks down when applications have poor accessibility support or when visual context matters for decision-making.

Pure vision is the most general but most expensive. It’s appropriate when agents need to operate arbitrary GUIs, handle visual complexity, or work with applications that lack APIs and accessibility support. The computational cost has historically limited this approach to cloud deployments, but model compression and edge optimization are changing that equation.

Hybrid approaches are also viable. An agent might use API integration for structured operations, fall back to accessibility trees for standard UI elements, and use vision only when the other approaches fail. This layered strategy balances performance and coverage, though it increases implementation complexity.

The Standardization Question

MCP succeeded because it defined a minimal viable interface: tools expose functions with typed parameters, agents call those functions, and results flow back. The protocol is simple enough to implement quickly and flexible enough to cover many use cases.

A GUI agent protocol would need to address different concerns. It would need to standardize how agents perceive interface state (screenshots, accessibility trees, or both), how they specify actions (coordinates, element references, or semantic descriptions), and how they receive feedback (visual confirmation, state changes, or error messages).

No such standard exists yet. Each GUI agent framework defines its own perception and action primitives. Mano-P uses screenshot-based perception with coordinate-based actions. Other frameworks use accessibility tree queries with element ID-based actions. The lack of standardization means that GUI agents are not portable across frameworks, and application developers have no clear target to optimize for.

The Octo workspace takes a different approach to this problem. Rather than standardizing the GUI interaction layer directly, it focuses on agent coordination and task orchestration. Individual agents within Octo can use different interaction strategies—API integration for some tasks, GUI automation for others—while the workspace manages context, state, and collaboration between them. This sidesteps the standardization question by treating GUI interaction as an implementation detail rather than a protocol concern.

Whether a GUI-specific protocol will emerge remains unclear. The diversity of approaches suggests the problem space is not yet well understood. MCP took years to gain traction, and GUI agent interaction may require a similar maturation period before a standard emerges.

Open Problems

Several technical challenges remain unsolved regardless of which approach dominates.

State tracking across actions. GUIs are stateful. Clicking a button might open a dialog, change a menu, or trigger a loading spinner. Agents need to track these state changes and adjust their behavior accordingly. Current approaches handle this through repeated perception cycles—take a screenshot, detect the state change, decide the next action—but this is inefficient and error-prone.

Error recovery. GUIs fail in unpredictable ways. A dialog might appear unexpectedly. A network request might timeout, leaving the interface in an inconsistent state. An element might be obscured by a popup. Agents need robust error detection and recovery strategies, but defining what constitutes an “error” in a GUI context is difficult.

Multi-step workflows. Many GUI tasks require sequences of actions with intermediate verification. Filling out a multi-page form, configuring application settings, or navigating a complex menu hierarchy all require maintaining context across multiple perception-action cycles. Current agents struggle with long-horizon tasks that span dozens of steps.
Cross-platform consistency. An agent that works on macOS might fail on Windows or Linux because the same application renders differently on each platform. Building agents that generalize across operating systems requires handling platform-specific UI conventions, keyboard shortcuts, and interaction patterns.

What Comes Next

The GUI agent space is where API integration was before MCP: fragmented, with each implementation defining its own conventions. The success of MCP suggests that standardization is possible, but the problem spaces are fundamentally different. API integration deals with structured data and typed interfaces. GUI interaction deals with pixels, visual layouts, and human-designed abstractions.

Pure vision approaches like Mano-P demonstrate that general GUI agents are technically feasible on edge hardware. The 4B model’s performance characteristics—sub-200ms inference, 4.3GB memory footprint—show that the computational barriers are falling. The question is whether the community will converge on a standard interface for GUI agent interaction, or whether the diversity of approaches will persist.

For developers building agents today, the pragmatic answer is to use the approach that fits your deployment context. API integration when coverage is sufficient. Accessibility trees for broader reach with acceptable performance. Pure vision when generality matters more than speed. And keep watching for standards to emerge, the way MCP emerged for tool integration.

The code for Mano-P https://github.com/Mininglamp-AI/Mano-Pis available on GitHub under Apache 2.0 license. The 4B quantized model is ready for local deployment on Apple M4 hardware. If you’re working on GUI agent problems, the repository includes benchmark scripts, model weights, and deployment documentation. Worth a star if the problem space interests you.

Humans Judge, AI Executes: Rethinking Work for the Multi-Agent Era

Mininglamp — Wed, 24 Jun 2026 09:15:03 +0000

A software engineer opens a chat window, pastes a stack trace, and waits for a suggested fix. A marketing manager types “write a product launch email” and copies the output into a draft. A data analyst asks for a SQL query, runs it, and comes back with a follow-up question. Three people, three different tasks, one interaction pattern: type a prompt, receive text, decide what to do with it.

This pattern works. It works well enough that most organizations have adopted it without much friction. It also leaves most of the potential value on the table.

The chat window treats every interaction as a standalone transaction. It holds no memory of what happened before, tracks no progress toward a larger goal, and carries no accountability for the outcome. The human on the other side absorbs all of that overhead: remembering context, managing workflow, evaluating quality, deciding next steps. The AI contributes text. The human contributes everything else.

A different arrangement is possible, one where the division of labor shifts along a cleaner line. Humans supply judgment. AI supplies execution. The boundary between the two is where the interesting engineering problems live.

The Execution Gap

Execution, in the context of knowledge work, means something specific. It means taking a defined objective and carrying it through to a deliverable. Research a competitor and produce a comparison matrix. Draft a proposal based on meeting notes and send it for review. Monitor a data pipeline and escalate anomalies.

Each of these tasks involves multiple steps, decision points, tool interactions, and intermediate outputs. A chat-based AI can assist with individual steps. It can draft a paragraph, write a query, summarize a document. The orchestration of steps remains the human’s responsibility.
The gap between step-level assistance and task-level execution is larger than it appears. Orchestration requires context: what has already been done, what the final deliverable looks like, who will review it, what standards apply. A stateless chat session holds none of this information. The human carries it in their head and re-supplies it with every new prompt.

When the task grows complex enough, the orchestration burden becomes its own form of work. The human spends more time managing the AI than doing the actual task. The tool has become a dependency rather than a lever.

Closing this gap requires agents that operate at the task level rather

than the step level. An agent that accepts an objective, maintains context across steps, invokes tools as needed, and produces a deliverable rather than a text fragment. The human’s role shifts from orchestrating steps to defining objectives and evaluating outcomes.

We are building Octo as a workspace designed around this shift. Not a chat interface with extra features, but a system where agents receive assignments, maintain work context, and deliver results for human review.

Identity as Infrastructure

An agent that executes tasks needs an identity. This is not a cosmetic concern.

In organizational settings, identity determines trust boundaries. A human employee has a name, a role, a set of permissions, and a work history.

These attributes determine what information they can access, what decisions they can make, and who is accountable for their output. An agent operating within an organization needs equivalent attributes to function within existing governance structures.

An anonymous agent cannot be assigned a task with clear accountability. It cannot be granted access to sensitive information through existing permission systems. Its output cannot be traced back to a responsible party for audit purposes. These are not edge cases. They are baseline requirements for any agent that handles real work in a real organization.
The identity model also shapes how humans interact with agents. A named agent with a defined role and visible track record gets treated as a collaborator. An anonymous chat window gets treated as a tool. The difference matters for adoption. People build working relationships with collaborators. They use tools and discard them.

Each Bot in Octo carries an AgentCard: a profile containing its role definition, capability set, and delivery history. A Bot is created by a specific person, operates on that person’s behalf, and inherits a subset of the creator’s permissions. The Bot becomes a digital proxy, an extension of its creator’s capacity rather than a free-floating assistant.
This proxy model has implications for delegation. A manager can create a Bot to handle routine approvals. An analyst can create a Bot to run recurring reports. The Bot acts within boundaries defined by its creator, and the creator retains accountability. Delegation without accountability is just automation. Delegation with accountability is a structural change in how work gets distributed.

The Judgment Layer

Execution without judgment produces output. Execution guided by judgment produces work product. The distinction matters.

An agent can draft a report based on available data. Whether that report is adequate depends on factors that sit outside the execution process: does the analysis address the right questions, are the sources credible, does the framing match the audience’s expectations, is the level of detail appropriate for the decision it supports. These are judgment calls. They require domain expertise, situational awareness, and an understanding of stakeholder expectations.

The human role in a human-agent collaboration centers on these judgment calls. Humans define what good looks like. They evaluate whether the output meets that standard. They provide corrections when it falls short. This is not a peripheral activity to be minimized. It is the core contribution that makes agent execution valuable.

Judgment also evolves over time. A reviewer who has evaluated fifty reports from a particular agent develops a sense of its tendencies: where it typically excels, where it commonly misses the mark, what kinds of instructions produce better results. This accumulated assessment constitutes a form of knowledge that should feed back into the agent’s behavior.

The Taste system in Octo captures this feedback loop. When a human reviews a Bot’s deliverable, the review generates preference signals: acceptances, rejections, modification notes, style corrections. These signals accumulate as preference cards that the Bot references in subsequent tasks. The Bot’s output adapts over time to reflect the reviewer’s standards and preferences.

A new Bot and a Bot that has worked with the same reviewer for three months should produce noticeably different output quality. If they don’t, the feedback mechanism isn’t working. The goal is not a static tool that performs the same way every time, but a collaborator that improves through accumulated judgment.

Multi-Agent Coordination

Most work involves more than one agent, especially once agents can handle task-level execution. A research task might benefit from one agent gathering information while another synthesizes findings. A content pipeline might involve separate agents for drafting, editing, and formatting. A complex analysis might require agents with different domain specializations working in parallel.

Coordination between agents introduces its own set of requirements. Agents need shared context: a common understanding of the overall objective, visibility into each other’s progress, and access to intermediate outputs. They need communication protocols: rules for when to hand off work, how to signal completion or blocking, and where to deposit results. They need conflict resolution: mechanisms for handling cases where two agents produce contradictory outputs or competing recommendations.

The coordination topology should match the work structure. Sequential tasks call for pipeline patterns where each agent receives input from the previous one. Exploratory tasks benefit from discussion patterns where agents exchange perspectives and refine ideas. Independent subtasks suit parallel execution with a final aggregation step. Forcing all coordination through a single pattern creates friction.

Octo supports six coordination modes. Solo handles single-agent work.

Roundtable enables multi-agent discussion with turn-taking. Critic pairs a producer with a reviewer. Pipeline chains agents in sequence. Split distributes subtasks for parallel processing. Swarm allows agents to self-organize around a shared objective. Each mode defines different communication rules, context-sharing boundaries, and control flows.

The human’s role in multi-agent coordination varies by mode. In Pipeline, the human might only intervene at the final review stage. In Roundtable, the human might participate as an equal voice or observe and intervene when needed. In Swarm, the human sets the objective and reviews the aggregate output. The level of human involvement scales with the complexity and risk of the coordination pattern.

Specialized agents also participate in these coordination patterns. Mano-P, a GUI-VLA agent built for edge devices, handles visual interface operations on local machines. It runs entirely on-device with no data leaving the host environment. The 72B model achieves 58.2% success rate on OSWorld benchmarks, while a 4B quantized version operates with just 4.3GB of memory at 76 tokens per second. Agents like Mano-P contribute specific execution capabilities that slot into larger coordination patterns managed by the workspace.

Organizational Memory

Individual preferences and task histories accumulate quickly. Across a team, the volume of accumulated knowledge becomes substantial: reviewer preferences, project contexts, client requirements, recurring task patterns, institutional standards. Most of this knowledge exists informally, stored in people’s heads or scattered across documents and email threads.

An agent-based workspace captures this knowledge as a byproduct of normal operation. Every review generates preference data. Every completed task produces context that might apply to future tasks. Every successful workflow demonstrates a pattern that could be reused. The system doesn’t need a separate knowledge management initiative. It accumulates knowledge through use.

Three categories of organizational knowledge emerge from this process. Context captures project-specific information, client backgrounds, and situational details that agents need to produce relevant output. Taste encodes quality standards, style preferences, and evaluation criteria that shape how agents approach their work. Skill packages recurring task patterns as reusable procedures that new agents can inherit.

Together, these three categories form an organizational asset that compounds over time. A team that has used the workspace for six months has accumulated a knowledge base that would take a new team months to build from scratch. This knowledge transfers across team members: a new hire inherits the accumulated context, taste, and skills without having to develop them personally.

The asset also creates switching costs, though of a healthy kind. The accumulated knowledge makes the workspace more valuable the longer it is used, which is the intended incentive structure. Teams invest in building their knowledge base and benefit from that investment over time.

Data Sovereignty

None of the above works if the data lives on someone else’s servers.
Work product contains sensitive information: strategic plans, financial projections, client data, internal communications, proprietary analysis. Organizations cannot route this information through third-party infrastructure without accepting risks they are often unwilling to accept. Regulatory requirements in many industries explicitly prohibit it.

Private deployment is not a premium feature. It is a prerequisite for any agent system that handles real organizational work. A system that requires data to leave the organization’s infrastructure limits itself to low-sensitivity use cases, which are also the lowest-value use cases.

Octo deploys on the organization’s own infrastructure. All data, all agent interactions, all accumulated knowledge stays within the organization’s environment. The system does not phone home, does not share telemetry, and does not require external API access for core functionality.

Access Points

Work happens in multiple contexts. A person might start a task at their desk, check progress from a phone during a meeting, review output while browsing the web, and trigger a follow-up from a terminal. An agent workspace that only exists in one interface creates friction every time the human switches contexts.

Four access points cover the common scenarios: a web application for full-featured interaction, a mobile interface for monitoring and quick decisions, a browser extension for context-aware interventions during web browsing, and a command-line interface for developers and automation workflows. Each interface provides access to the same underlying workspace, with the same context, the same agent relationships, and the same accumulated knowledge.

The goal is not omnichannel presence for its own sake. It is ensuring that the human can exercise judgment from whatever context they happen to be in, without having to switch to a dedicated interface to review output or provide direction.

The Shift

The transition from chat-based AI assistance to agent-based execution changes what humans spend their time on. Less time orchestrating steps, more time defining objectives. Less time re-supplying context, more time evaluating outcomes. Less time managing tools, more time exercising judgment.

This is not a marginal improvement. It is a different division of labor, one that allocates human attention to the activities where human judgment actually matters and delegates execution to agents that can operate autonomously within defined boundaries.

The engineering challenges are substantial. Agents need identity, context persistence, coordination protocols, feedback mechanisms, and organizational knowledge systems. The workspace that houses these agents needs to support private deployment, multiple access points, and integration with existing tools and workflows.

Octo is our approach to building this workspace. The code is open, the design decisions are documented, and the project welcomes examination and contribution. The multi-agent era is arriving through incremental engineering work, one coordination pattern at a time.

When AI Agents Start Working Together: Three Challenges No One Talks About

Mininglamp — Mon, 22 Jun 2026 06:43:00 +0000

The trajectory of AI agents over the past two years has been remarkably clear: from single-purpose tools to personal assistants. Everyone runs their own agent, feeds it tasks, gets results back. It works well for individual productivity.

Then comes the question every team eventually asks: can these agents work together?

The answer is yes, but the problems you encounter along the way are rarely the ones you expected. They aren't about model capabilities or prompt engineering. They're about communication, context, and coordination — the same class of problems that distributed systems engineers have been solving for decades, now showing up in a new form.

Here are three challenges that caught us off guard when we started building agent collaboration into Octo, an open-source workplace platform where AI agents and humans share the same communication space.

Challenge 1: Context Visibility Boundaries

When you use an agent personally, context management is straightforward. You decide what information the agent sees; its output comes back to you. The boundary is clean — it's just your workspace.

In a team setting, that boundary dissolves.

One of the first issues we ran into was surprisingly simple. We had an agent summarizing discussions across several channels. During testing it started pulling roadmap discussions from a product channel into an engineering planning thread. Nothing sensitive leaked externally, but it immediately exposed how unclear our context boundaries were.

Traditional software handles this through API gateways, data permissions, and microservice boundaries. But agent context isn't just structured data — it includes conversation history, reasoning chains, and intermediate states. An agent's thought process during a task is valuable context, but it might also contain information that shouldn't cross team boundaries.

What you need is fine-grained context visibility control. Not "everything open" or "everything closed," but dynamic rules that determine which context can be shared based on the task, role, and scenario at hand.

This is where instant messaging architecture turns out to be surprisingly relevant. Channels are natural context boundaries — members only see messages in channels they belong to. When an agent joins a channel, it inherits that boundary naturally. It can access the channel's message history as context, but it can't see other channels. This is more mature than building a context management system from scratch, and it maps cleanly onto how teams already organize their work.

Challenge 2: Permission Intersections and Conflicts

Personal agent permissions are simple: whatever the user authorizes, the agent can do.

In a team context, permissions become many-to-many. A single agent might serve multiple people, participate in multiple projects, and play different roles in different channels. Each dimension has its own permission requirements, and they can conflict.

Here's a concrete example: a code review agent participates in two project channels. Project A's codebase is invisible to Team B, but the agent can access both codebases while serving both projects. If the agent, while reviewing Project B's code, references an implementation pattern from Project A, is that an information leak?

The situation gets more complex in human-agent collaboration. When humans and agents work in the same channel, humans can see all of the agent's output. But the agent's output might draw on information from other contexts it has access to. How do you ensure the agent only uses information visible in the current channel when generating responses?

Distributed systems have mature solutions for permission design — RBAC (role-based access control), ABAC (attribute-based access control), and their variants. Agent systems can borrow these approaches, but they need adaptation for agent-specific characteristics. Agents don't just passively execute commands; they reason, generate content, and make proactive decisions. Permission control needs to cover the generation process itself, not just inputs and outputs.

In Octo, we adopted an organization-aware RBAC model where each channel has its own ACL (access control list). Agent identities and permissions are managed alongside human members. All agent input and output within a channel is auditable, and permission boundaries are naturally expressed through the channel mechanism that IM systems have refined over decades.

Challenge 3: Collective Experience Accumulation and Reuse

A personal agent can learn from historical interactions, gradually understanding a user's preferences and working patterns. This learning is individual — experience accumulates in a single agent's context.

In a team setting, the dimension of experience changes. It's not just about individual agent experience anymore, but about collective experience generated through multi-agent collaboration — which collaboration patterns are efficient, which task decomposition approaches tend to cause problems, where human intervention happens most frequently. If this information could be captured and reused, it would meaningfully improve the team's overall collaboration efficiency.

But collective experience faces several challenges.

First, there's the ownership question. When one agent learns something during a collaborative task, should other agents have access to it? If so, could that introduce context pollution — an agent incorrectly applying someone else's experience to its own scenario?

Second, there's timeliness. Team collaboration patterns shift with project phases, team structure, and business goals. A pattern that worked three months ago might be irrelevant now. Captured experience needs update and deprecation mechanisms.

Then there's quality assessment, which is easy to overlook. Not every historical interaction yields valuable experience. Some might be special cases; others might contain flawed judgments. Capturing experience while maintaining quality requires an evaluation framework.

Message history, group documents, and pinned messages in IM systems — while not designed for experience capture — can serve this role in practice. Key conversation conclusions can be pinned, important decision processes can be archived to group documents, and agents can retrieve these structured artifacts as references when executing tasks. This approach is lighter than vector databases or knowledge graphs, and it's easier for both humans and agents to understand and maintain together.

Why IM Architecture Matters Here

These three challenges — context visibility, permission intersections, collective experience — all point to a deeper insight: agent collaboration isn't just about connecting multiple agents. It requires a complete collaboration infrastructure.

That infrastructure needs to handle communication, context, permissions, state synchronization, and experience accumulation. These problems have mature solutions in traditional software engineering, but agent systems — with their autonomous reasoning, content generation, and proactive decision-making — push the complexity up a level.

IM architecture shows a surprising fit for this scenario. Over decades, IM systems have solved multi-party real-time communication, context management, permission control, and state synchronization, accumulating mature architectural patterns and engineering practices. Migrating these capabilities to agent collaboration is more reliable than building a new system from scratch.

This observation led us to build Octo on IM foundations — agents join channels directly and collaborate with humans in the same conversation interface. The project uses the Apache 2.0 license, has 9 core repositories under the Mininglamp-OSS GitHub organization, and runs on a stack of Go backend, WuKongIM, MySQL, Redis, and MinIO. It supports private deployment with 100% data on your own servers.

The Bigger Picture

Moving AI agents from personal tools to team infrastructure expands what they can do, but it also changes the nature of the challenges. Better models alone won't solve communication, context, and coordination problems. These require mature collaboration infrastructure.

The shift from personal assistant to team collaborator might be the next important transition in the AI agent space. When it happens, the teams that think about these architectural challenges early — rather than just stacking more agents together — will build systems that actually work in practice.

If you're working on multi-agent systems or interested in agent collaboration infrastructure, we'd love to hear about the challenges you've encountered. The Octo project is open source, and we welcome contributions and discussions on GitHub.

Why We Put AI Agents in a Group Chat？

Mininglamp — Wed, 17 Jun 2026 10:01:01 +0000

Most teams building with LLMs eventually hit the same wall. You have a handful of agents, each one reasonably capable at its specific task, but they cannot talk to each other. One agent handles data extraction, another does summarisation, a third manages scheduling, yet the only thing connecting them is a human copying and pasting between browser tabs.

We ran into this problem repeatedly while deploying AI agents inside enterprise environments at Mininglamp Technology. The agents were good at what they did individually, but the coordination overhead landed squarely on people. After a while, using agents felt more exhausting than not using them, because there were simply more contexts to keep track of.

This article is about a different approach we took, one that treats instant messaging protocols not as a UI layer bolted on top of agent infrastructure, but as the foundational fabric for agent distribution, orchestration, and permission management.

The Session Island Problem

Every agent framework on the market optimises for a single metric, how well one agent performs in isolation. Bigger context windows, more accurate tool calls, faster inference. These are all valuable, but they miss a structural limitation that only becomes visible in multi-person, multi-step enterprise workflows.

An agent's context is bounded by its own conversation thread. It has no native way to know what happened in another agent's thread, what decision a colleague made five minutes ago, or whether the data it is about to process has already been flagged as unreliable upstream. We call this the session island problem, and it gets worse as the number of agents in an organisation grows.

Picture a typical scenario. A marketing team member drops a pricing question into a group channel. Answering it properly requires product specs from the engineering team's agent, cost models from the finance team's agent, and competitive intelligence from the sales team's agent. Under the current paradigm, someone has to manually route information between these agents, adding context at each handoff. The agents are powerful; the connective tissue between them is duct tape.

Why Not an API Gateway

Our first instinct, like most engineering teams, was to build an API gateway. Agents expose endpoints, other agents call those endpoints, the gateway handles routing and authentication. It works, but three problems surfaced quickly during implementation.

The first is state management. API calls are inherently stateless, one request, one response, done. But agent-to-agent collaboration in enterprise settings is often a long-running process. A single task might span several hours, involve multiple rounds of interaction, require human approval at certain checkpoints, and generate intermediate artifacts that other agents need to reference. Building this on top of a stateless API layer means reinventing message queues, state machines, and event subscription systems from scratch.

The second is observability. When something goes wrong in an API-based agent mesh, you end up chasing logs across half a dozen services, trying to reconstruct the sequence of events from fragmented traces. In an IM-based architecture, the group chat itself is the complete audit trail. Who said what, when, what decision was reached, what was delivered, it is all there in a single chronological thread.

The third is permission management. API gateways typically handle access control through tokens and ACLs, which are coarse-grained and tedious to configure. In a group chat model, information visibility maps directly onto group membership. Add someone to a group and they can see everything in it; remove them and they lose access. No separate permission matrix to maintain.

These observations pushed us toward a fundamentally different design.

Octo — Agents as First-Class Chat Participants

Octo is an open-source platform we built around this insight. At its core is octo-server, a Go backend that simultaneously handles REST APIs, WebSocket connections, and IM message routing through WuKongIM. The key design decision is that agents (which we call Lobsters in Octo) are not external services invoked through webhooks. They are participants in conversations, with the same messaging capabilities as human users.

When a Lobster agent joins a group chat, it receives a full conversation context, chat history, member roster, read receipts, not just a trigger event. The agent can proactively send messages, reply to specific people, notify relevant parties when a task completes, or ask follow-up questions when it needs more information.

The request processing pipeline in octo-server follows a clear sequence. First, authenticate the request source (supporting tokens, cookies, and DH-encrypted WebSocket frames). Then authorise with organisation-aware RBAC and per-channel ACL checks. Execute the business logic, which may involve spawning or resuming a Lobster agent session. Fan out the resulting message through WuKongIM to all group members, triggering adapters if the channel requires an external bridge. Finally, return a unified JSON response with tracing and metrics tags.

One important side effect of this design is message ordering. IM protocols enforce strict temporal ordering on messages, so every agent sees the same conversation history in the same sequence. For multi-step collaborative tasks, this ordering guarantee is significantly more reliable than the request-response pattern you would get from a traditional API mesh.

Scaling Out vs. Scaling Up

Improving a single agent's capabilities is scaling up, bigger models, longer contexts, more tools. What enterprises actually need, though, is scaling out, organising multiple specialised agents to collaborate on tasks that no single agent can handle alone.

The analogy is straightforward. One person working overtime can only produce so much. A well-coordinated team of ten can dramatically outperform that individual. The challenge in the agent domain has always been that "well-coordinated" requires infrastructure that did not exist.

Octo's group mechanism fills this gap. You create a group for each project or workstream, add the relevant agents and humans, and let them collaborate through the same messaging interface. Agents in a group behave essentially the same way as human members do, sending messages, replying to threads, marking tasks complete, requesting confirmation. Every interaction is automatically recorded as chat history, so newcomers (human or agent) can read through the backlog to get up to speed.

We also built octo-smart-summary, a service that uses LLMs to periodically summarise group chat content, extracting key decisions, action items, and open questions. This addresses another common pain point, chat channels accumulate information fast, and late joiners struggle to find what matters.

┌─────────────────────────────────────────────────┐
│                 OCTO Architecture                │
│                                                  │
│  ┌────────┐  ┌────────┐  ┌───────────┐         │
│  │octo-web│  │octo-ios│  │octo-android│         │
│  │(React) │  │(Swift) │  │ (Kotlin)  │         │
│  └───┬────┘  └───┬────┘  └─────┬─────┘         │
│      └───────────┼─────────────┘                │
│                  ▼                               │
│        ┌──────────────────┐                     │
│        │   octo-server    │                     │
│        │   (Go Backend)   │                     │
│        │  · REST + WS     │                     │
│        │  · Lobster sched │                     │
│        │  · WuKongIM ctrl │                     │
│        └──┬────┬────┬───┘                     │
│           │    │    │                           │
│    ┌──────┘    │    └──────┐                    │
│    ▼           ▼           ▼                     │
│  ┌──────┐  ┌───────┐  ┌──────────┐             │
│  │octo- │  │smart- │  │  octo-   │             │
│  │matter│  │summary│  │ adapters │             │
│  └──────┘  └───────┘  └──────────┘             │
└─────────────────────────────────────────────────┘

Security — The Group Is the Permission Boundary

Enterprise deployments demand precise control over what agents can access, what operations they can perform, and who can see their output. Octo ties the permission model directly to the group model.

Every channel in Octo has its own ACL configuration. When an agent joins a group, it inherits that group's permission settings, and everything the agent produces within the group is subject to the same access constraints. There is no separate permission system to maintain for agents; the group's access control is the agent's access control.

For edge-agent scenarios, where computation happens on a user's local device (think of something like Mano-P, a GUI agent that runs natively on Apple Silicon), Octo's adapter mechanism can bridge execution results securely into IM groups. The actual computation and data never leave the device; only the necessary summaries and status updates flow into the group conversation.

The authentication layer supports multiple mechanisms, traditional tokens, cookies, and Diffie-Hellman encrypted WebSocket frames. The authorisation layer implements organisation-aware RBAC, so every request passes through both organisation-level and channel-level permission checks before execution.

Engineering Considerations

Plugging agents into an IM system is not as simple as adding another consumer to a message queue. Several engineering challenges deserve attention.

Message format unification. Agent output is more structured than human messages. You need a consistent envelope that distinguishes between plain text, tool call results, status updates, and error reports. Octo defines a unified message schema that all agent output passes through before entering the IM channel.

Concurrency conflicts. Multiple agents might operate on the same group simultaneously, replying to the same message at the same time or updating the same task status. octo-server handles message ordering and conflict detection at the server level to prevent logical contradictions in the chat.

Context management. Agent context windows are finite, but group histories can grow indefinitely. This is where octo-smart-summary plays a critical role. It periodically compresses chat history into summaries, allowing agents to read the digest first and drill into specific segments only when needed.

Open Source and Next Steps

The entire Octo project is open-sourced under Apache 2.0 on GitHub (github.com/Mininglamp-OSS/octo-server), including the backend, web client, mobile apps, admin console, and all supporting tools. We adopted a local-first design philosophy throughout; chat records, vector indices, and agent execution can all run on the user's own infrastructure, with cloud deployment as an option rather than a requirement.

If you are thinking about how to move AI agents from isolated tools to genuine nodes in an enterprise collaboration network, we would love to hear from you. Whether it is a GitHub star, an issue report, or a feature request from your own use case, every piece of feedback helps us build this infrastructure better.

ZCode vs MiMo Code vs DevEco Code: Who Really Solves Developer Pain Points in China's AI Coding Tools Race?

Mininglamp — Tue, 16 Jun 2026 11:44:15 +0000

June 2026 marked a turning point for AI coding tools in China. Zhipu released ZCode 3.0, Xiaomi open-sourced MiMo Code, and Huawei launched DevEco Code at HDC 2026. Developer communities are calling it the "Three Kingdoms War" of domestic AI coding tools.

These three products take fundamentally different technical approaches. This article compares them across product positioning, technical architecture, and developer experience, while exploring the unique value proposition of local-first solutions in this space.

Product Positioning

ZCode 3.0 (Zhipu AI): Released June 13, positioned as a multi-agent collaborative IDE. Core features include grouped task workspaces, Zread intelligent project knowledge base, and visual Git branch graphs. Zhipu's advantage lies in deep integration between its proprietary GLM model series and the tool itself.

MiMo Code (Xiaomi): Open-sourced June 11, built on OpenCode with MIT license. Supports persistent memory systems, unlimited context windows, and multi-model compatibility (DeepSeek, Kimi, GLM, MiMo v2.5). Xiaomi chose an open ecosystem approach.

DevEco Code (Huawei): Launched at HDC 2026, a specialized programming agent for the HarmonyOS ecosystem. Built on Huawei's Bifang large model, covering the full workflow from requirements design through testing and maintenance. AI code generation rate reaches 80%. Huawei open-sourced all HarmonyOS AI-assisted development Skills to the OpenHarmony community.

Technical Architecture Analysis

Zhipu follows a model-driven approach. ZCode 3.0's multi-agent concurrency and project understanding capabilities depend on GLM's underlying model capabilities. The product ceiling is tied to model iteration speed.

Xiaomi follows an ecosystem-compatible approach. MiMo Code doesn't bind to specific models—developers can freely switch underlying models. The MIT license lowers adoption barriers, but product differentiation relies mainly on upper-layer experience.

Huawei follows a vertical specialization approach. DevEco Code focuses exclusively on HarmonyOS scenarios. Multi-device adaptation, problem localization, and self-repair capabilities only make sense within the HarmonyOS ecosystem. Huawei's bet is that the HarmonyOS ecosystem is large enough to justify a dedicated tool.

Data Security: The Local-First Advantage

As cloud-based AI coding tools become increasingly homogenized, data security and privacy emerge as differentiating factors.

Mininglamp's open-source Mano-P is a GUI-VLA agent model designed for edge devices, supporting fully local execution on Mac with Apple M4 + 32GB RAM. Screenshots and task descriptions never leave the device, making it suitable for scenarios with strict data security requirements.

In OSWorld specialized model evaluation, Mano-CUA 1.1 achieved 58.2% success rate, ranking first and leading the second-place opencua-72b (45.0%) by 13.2 percentage points. In WebRetriever Protocol I testing, Mano-CUA 1.1 scored 41.7 NavEval, surpassing Gemini 2.5 Pro (40.9) and Claude 4.5 (31.3).

Performance Metrics

Mano-P's 4B quantized model achieves approximately 80 tokens/s decode speed on M5 Pro. Combined with Cider SDK's W8A8 activation quantization, prefill is approximately 12.7% faster than the W8A16 baseline.

Testing on 100 macOS GUI tasks showed Mano-CUA-Thinking-4B local model achieved 56.0% pass rate, exceeding cloud-based Qwen3-VL-Plus at 39.0%. Local small models can outperform cloud large models in specific scenarios.

Open Source and Installation

Mano-P uses Apache 2.0 license with three-phase open-source plan:

Phase 1: Mano-CUA Skills now open-source, install via brew tap Mininglamp-AI/tap && brew install mano-cua
Phase 2: Local models and SDK, models available on HuggingFace and ModelScope
Phase 3: Training methodology and quantization pruning techniques (planned)

Selection Recommendations

ZCode 3.0 suits teams pursuing deep model integration; MiMo Code fits developers needing flexible multi-model switching; DevEco Code is the specialized tool for HarmonyOS developers.

For scenarios with strict data security requirements or hard latency constraints, Mano-P's local-first approach deserves consideration. The three major tools and Mano-P represent different technical directions in AI coding tools—developers should choose based on actual needs.

Loop Engineering: The Next Step After Prompt Engineering for AI Agents

Mininglamp — Mon, 15 Jun 2026 11:21:07 +0000

Loop Engineering: The Next Step After Prompt Engineering for AI Agents

The AI development landscape has undergone a fundamental shift. For years, prompt engineering dominated the conversation—crafting the perfect instruction, fine-tuning context windows, and optimizing token usage. But as AI agents evolve from simple question-answering systems to autonomous problem-solvers, a new discipline is emerging: Loop Engineering.

At Mininglamp, we've spent the last two years building production-grade AI agents, and we've learned a crucial lesson: the magic isn't in the prompt anymore. It's in the loop.

From Prompts to Loops: Why the Shift Matters

Prompt engineering assumes a single interaction: you provide input, the model provides output. This works well for chatbots, content generation, and straightforward tasks. But modern AI agents don't work that way. They operate in cycles—observing their environment, reasoning about what to do, taking action, and verifying the results before deciding what comes next.

This cyclic behavior is fundamentally different from prompt-response patterns. It requires:

State management across multiple iterations
Error recovery when actions fail
Dynamic decision-making based on intermediate results
Resource constraints (time, API calls, tokens)
Verification mechanisms to know when to stop

These challenges can't be solved with better prompts alone. They require architectural patterns specifically designed for iterative, autonomous operation. That's Loop Engineering.

What is Loop Engineering?

Loop Engineering is the practice of designing, implementing, and optimizing the iterative cycles that power autonomous AI agents. It encompasses:

Loop Architecture: The structure of observe-think-act-verify cycles
State Management: How agents track progress and context across iterations
Control Flow: Decision logic for branching, retrying, and terminating loops
Error Handling: Strategies for graceful degradation and recovery
Performance Optimization: Balancing speed, accuracy, and resource usage

Think of it this way: if prompt engineering is about crafting a single perfect instruction, loop engineering is about designing the entire runtime environment where an agent operates autonomously.

The Anatomy of an Agent Loop

Every AI agent loop follows a core pattern, though implementations vary widely. Here's the fundamental structure:

while not task_complete:
    observation = perceive(environment)
    plan = reason(observation, goal, history)
    action = decide(plan)
    result = execute(action)
    verify(result, goal)
    update_state(result)

Let's break down each component:

1. Perception (Observe)

The agent gathers information about its current state. For GUI agents, this means taking screenshots and parsing visual elements. For API-based agents, it means reading responses and status codes. The key challenge: extracting relevant information while filtering noise.

2. Reasoning (Think)

The agent analyzes the observation in context of its goal and past actions. This is where LLMs shine—they can synthesize complex situations and generate plans. But reasoning in loops is different from single-shot reasoning. The agent must:

Track what it has already tried
Understand why previous attempts succeeded or failed
Adjust strategies based on accumulated evidence

3. Decision (Plan)

Based on reasoning, the agent decides on a specific action. This could be clicking a button, making an API call, writing code, or asking for clarification. The decision must be concrete and executable.

4. Execution (Act)

The agent performs the chosen action. This is where things get interesting—actions can fail, timeout, or produce unexpected results. Robust execution requires:

Timeout handling
Retry logic with backoff
Resource cleanup on failure
Logging for debugging

5. Verification (Verify)

After execution, the agent checks whether the action achieved the desired effect. This is often overlooked but critical. Without verification, agents can:

Loop infinitely on failed actions
Proceed with incorrect assumptions
Miss partial successes that need refinement

Verification strategies include:

Direct checking: Did the button click navigate to the expected page?
State comparison: Has the relevant part of the environment changed?
Goal proximity: Are we closer to the objective than before?

Loop Patterns: Single-Step vs Multi-Step vs Self-Correcting

Not all loops are created equal. The pattern you choose depends on task complexity, reliability requirements, and resource constraints.

Pattern 1: Single-Step Loops

The simplest pattern: observe, act, done. Used for straightforward tasks with high confidence.

Example: "Click the submit button"

screenshot = capture_screen()
button_location = find_button(screenshot)
click(button_location)
# Done

When to use: Simple, well-defined actions with low failure probability.

Limitations: No error recovery. If the button isn't there, the agent fails.

Pattern 2: Multi-Step Sequential Loops

Multiple actions executed in sequence, with state carried forward.

Example: "Fill out and submit a form"

for field in form_fields:
    screenshot = capture_screen()
    field_location = find_field(screenshot, field.name)
    click(field_location)
    type(field.value)

screenshot = capture_screen()
submit_location = find_button(screenshot, "Submit")
click(submit_location)

When to use: Tasks with clear, linear progression.

Limitations: Brittle to unexpected states. If a field is already filled, the agent might not handle it gracefully.

Pattern 3: Self-Correcting Loops

The most sophisticated pattern: the agent monitors its own progress and adjusts strategies when stuck.

Example: "Complete a complex workflow"

max_attempts = 10
attempt = 0

while not goal_achieved() and attempt < max_attempts:
    observation = capture_screen()

    # Check if we're stuck
    if is_stuck(observation, history):
        strategy = reconsider_approach(history)
    else:
        strategy = continue_current_plan()

    action = select_action(strategy, observation)
    result = execute(action)

    # Verify and learn
    if not result.success:
        analyze_failure(result, history)
        adjust_strategy()

    update_history(action, result)
    attempt += 1

When to use: Complex, unpredictable tasks requiring adaptation.

Advantages: Robust to failures, can recover from dead ends, learns from mistakes.

Challenges: More complex to implement, higher token usage, requires careful tuning of "stuck" detection.

Technical Deep Dive: How Loops Actually Work

Let's examine the technical considerations that separate toy implementations from production-grade agent loops.

State Management

Agents need to track:

Task progress: What has been accomplished?
Action history: What has been tried?
Environmental state: How has the world changed?
Resource usage: How many tokens/API calls remain?

Implementation approaches:

In-context state: Store everything in the prompt. Simple but token-expensive.
External state store: Use a database or file system. More efficient but adds complexity.
Hybrid: Keep recent state in context, archive older state externally.

Token Budget Management

LLMs have context limits. In long-running loops, you can't keep appending to the prompt indefinitely. Strategies:

Summarization: Periodically compress history into summaries
Sliding window: Keep only the most recent N iterations
Selective memory: Store only key decisions and outcomes

Example:

if len(history) > MAX_HISTORY:
    summary = summarize(history[:len(history)//2])
    history = [summary] + history[len(history)//2:]

Error Recovery Patterns

When actions fail, agents need strategies:

Retry with backoff: For transient failures (network timeouts)
Alternative path: Try a different approach to the same goal
Rollback: Undo recent actions and try from a known-good state
Escalation: Ask for human help when stuck

Verification Strategies

How does an agent know it succeeded?

Direct observation: Check if the expected change occurred
Invariant checking: Verify that certain conditions still hold
Goal decomposition: Break the goal into sub-goals and verify each
Confidence scoring: Rate confidence in success and retry if low

Real-World Performance: Benchmarking Loop Architectures

Theory is nice, but how do different loop patterns perform in practice? We tested three architectures on the OSWorld benchmark, a comprehensive suite of real-world computer tasks.

Test Setup

Single-Step: Direct action based on initial observation
Multi-Step Sequential: Linear execution of planned steps
Self-Correcting: Adaptive loop with stuck detection and strategy adjustment

Results

The self-correcting loop dramatically outperforms simpler patterns. Why?

Error recovery: Real-world tasks fail. Self-correcting loops retry with different strategies.
Adaptive planning: When the environment doesn't match expectations, the agent adjusts.
Progress verification: The agent knows when it's stuck and reconsiders.

The performance gap is substantial: self-correcting loops achieve 58.2% success rate on OSWorld, compared to ~45% for multi-step sequential and ~30% for single-step approaches. That's a 13+ percentage point improvement from loop engineering alone.

Where the Gains Come From

Analyzing failure modes reveals why self-correcting loops excel:

38% of failures in single-step loops were due to incorrect initial observations (element not visible, page not loaded)
52% of failures in multi-step loops were due to unhandled intermediate states (popup appeared, form validation failed)
Self-correcting loops recovered from 71% of these failure modes through retry and strategy adjustment

Building with Loops: Practical Implications

If you're building AI agents, here's what Loop Engineering means for your architecture:

1. Design for Failure

Assume every action can fail. Build verification and recovery into your loop from day one.

# Bad: Fire and forget
click(button)

# Good: Verify and recover
result = click(button)
if not verify_click(result):
    scroll_to_button()
    result = click(button)
    if not verify_click(result):
        try_alternative_approach()

2. Implement Stuck Detection

Agents often loop infinitely when stuck. Implement detection:

def is_stuck(history, threshold=3):
    recent_actions = history[-threshold:]
    # Check for repeated actions with same results
    if len(set(recent_actions)) == 1:
        return True
    # Check for oscillation between states
    if len(set(recent_actions)) == 2 and history[-1] == history[-3]:
        return True
    return False

3. Budget Your Resources

Set explicit limits on:

Maximum loop iterations
Token usage per task
Time per task
API calls per task

class ResourceBudget:
    def __init__(self, max_iterations=20, max_tokens=50000, max_time=300):
        self.max_iterations = max_iterations
        self.max_tokens = max_tokens
        self.max_time = max_time

    def can_continue(self, state):
        return (state.iterations < self.max_iterations and
                state.tokens_used < self.max_tokens and
                state.elapsed_time < self.max_time)

4. Log Everything

Debugging agent loops is hard without comprehensive logging:

Log observations (screenshots, API responses)
Log reasoning (why the agent chose an action)
Log actions and results
Log verification outcomes

This data is invaluable for improving your loops.

5. Consider Edge Deployment

For GUI agents, running loops on edge devices (local machines) offers advantages:

Privacy: Screenshots and data never leave the device
Latency: No network round-trips for API calls
Reliability: Works without internet connectivity
Cost: No per-token API fees for high-volume usage

Case Study: Loop Engineering in Mano-P

At Mininglamp, we've applied these principles in Mano-P, our edge-deployed GUI agent model. Mano-P uses a sophisticated self-correcting loop architecture with several key features:

The Mano-P Loop

Vision-Only Perception: Screenshots are the sole input—no API hooks, no DOM access
Think-Act-Verify Cycle: Each action includes explicit verification before proceeding
Progressive Training: Three-stage training (SFT → Offline RL → Online RL) teaches the model effective loop strategies
Edge-Native Execution: Runs locally on Apple M4 chips with 32GB RAM, keeping all data on-device

Performance Results

The loop engineering approach pays off:

#1 on OSWorld: 58.2% success rate, outperforming models 18x larger
13.2 point lead over second-place specialized models
Autonomous long-task execution: Handles complex workflows with dozens of steps
Fully local: No cloud API calls, complete data privacy

Mano-P demonstrates that sophisticated loop engineering can make smaller, specialized models outperform much larger general-purpose models on agentic tasks. The model is open-source on GitHub (github.com/Mininglamp-AI/Mano-P), and we've seen developers building increasingly sophisticated agent workflows using its loop primitives.

The Future of Loop Engineering

As AI agents become more autonomous, Loop Engineering will become as fundamental as prompt engineering is today. We're seeing several trends:

Hierarchical loops: Agents that manage sub-agents in nested loop structures
Learning loops: Agents that improve their loop strategies through experience
Multi-modal loops: Combining vision, text, and structured data in loop reasoning
Collaborative loops: Multiple agents coordinating through shared state

The key insight: the quality of an AI agent is determined less by the model's raw capabilities and more by the quality of its loop architecture. A well-designed loop can make a 4B-parameter model outperform a 72B model on real-world tasks.

Conclusion

Prompt engineering taught us how to communicate with AI models. Loop Engineering teaches us how to let them operate autonomously. The shift from single interactions to iterative cycles represents a fundamental change in how we build AI systems.

For developers entering this space, the principles are clear:

Design for failure and recovery
Verify every action before proceeding
Budget resources explicitly
Log comprehensively
Start with simple loops, graduate to self-correcting ones

The agents that will define the next era of AI won't just be better at answering questions—they'll be better at operating in loops, adapting to uncertainty, and achieving complex goals autonomously. Loop Engineering is how we build them.

Want to experiment with production-grade agent loops? Check out Mano-P on GitHub—our open-source GUI-VLA agent model that runs locally on edge devices, keeping your data private while demonstrating state-of-the-art loop engineering in action.

Will AI Agents Replace Programmers?

Mininglamp — Fri, 12 Jun 2026 10:36:11 +0000

The question used to be a thought experiment discussed in tech forums between sips of coffee. In 2026, it feels a lot more personal. Large language models can generate working programs from a few sentences of instruction, GUI automation agents navigate computer interfaces with increasing competence, and every other week brings a new open-source project that promises to make developers obsolete.

The honest answer isn't a clean yes or no. Some parts of what developers do are being transformed beyond recognition, while other parts are becoming more valuable precisely because AI can't touch them. Let's break this down by looking at what the job actually involves day to day.

Where AI Agents Have Made Real Progress

Code completion and generation were the first breakthroughs. Models today can take a brief description and produce a working React component, a SQL query, or a REST API implementation. The output quality in straightforward scenarios is approaching what you'd expect from a mid-level engineer. GitHub Copilot and similar tools are used daily by millions of developers; Stack Overflow's 2025 survey reported over 70% of respondents integrating AI tools into their workflow, up significantly from the year before.

Beyond autocomplete, purpose-built agents have entered production use. Some read entire codebases to locate bugs, others generate front-end and back-end code from product specs and run their own tests, and still others analyze logs to identify bottlenecks and suggest optimizations. For standardized coding tasks, these agents are remarkably efficient. Common algorithms, boilerplate API endpoints, test suites covering edge cases, data transformation scripts, configuration files from templates; pattern-heavy work with well-defined inputs and outputs is where large language models excel.

The Gap Between Writing Code and Engineering Software

Understanding why AI agents won't replace programmers requires distinguishing between "writing code" and "doing software engineering." Code is just one link in a much longer chain, and anyone who has shipped production software knows that writing the code is often the easiest part. The real challenges live elsewhere: understanding what needs to be built, making sound architectural choices, hunting down elusive bugs, keeping complex systems healthy over years of evolution.

Requirements analysis is a good example. Product managers and stakeholders rarely arrive with complete specifications. Sometimes they can't articulate what they actually need, and developers spend enormous effort figuring out the real problem through iterative conversations, challenging assumptions, and proposing alternatives that the stakeholder hadn't considered. This process involves understanding organizational context, business dynamics, and interpersonal relationships. An agent parsing a Jira ticket is nowhere close.

Architecture decisions involve navigating competing constraints. Performance, security, maintainability, developer productivity, team expertise, existing infrastructure, future extensibility; these factors rarely align neatly. Choosing to introduce message queues improves decoupling but adds operational overhead; adding a caching layer speeds up reads but introduces consistency challenges. This kind of judgment emerges from years of shipping systems, watching them fail in production, and learning what works in specific contexts. No amount of training data substitutes for that experience.

Then there's debugging production incidents. Distributed system bugs span network layers, database engines, caching systems, and message brokers simultaneously. Symptoms appear far from root causes, and finding the issue requires the kind of holistic mental model that comes from understanding every layer of the stack and having seen similar failures before. This pattern-matching ability, built over years of painful debugging sessions, remains firmly in human territory.

The Parts of Programming That Resist Automation

Engineering is fundamentally a team activity. Code reviews, design discussions, and cross-functional coordination consume a substantial portion of developer time and energy. Good code reviews go beyond syntax and logic; they evaluate design choices, anticipate edge cases, ensure consistency with team conventions, and assess maintainability over the long term. This engineering taste develops over years of building and reviewing software together, and current model training paradigms don't provide that experience.

Technical design discussions involve similar dynamics. Teams weigh multiple viable approaches, and each choice carries implications for different departments' resources and priorities. Navigating these conversations requires organizational awareness and persuasion skills, not code generation ability.

What History Tells Us About Tools and Jobs

Every major leap in developer tools has expanded rather than shrunk the software industry's demand for talent. When high-level languages replaced assembly, people predicted programmers would become unnecessary; instead, higher abstraction lowered the barrier to entry and created a vastly larger software industry with more developer positions than ever. CI/CD pipelines and cloud-native platforms didn't eliminate DevOps roles; they created SRE and platform engineering positions with higher complexity and compensation.

AI agents follow the same pattern. What's changing is the content of the job, not the existence of the job. As AI handles more repetitive coding tasks like boilerplate API endpoints, unit test generation, configuration file creation, and data format conversion, developers can invest more energy in higher-value activities: evaluating technical feasibility, evolving system architecture, deep performance optimization, and integrating emerging technologies into business contexts.

This shift has been underway for a while. Cloud computing didn't eliminate operations roles; it transformed them into DevOps and SRE positions with greater scope and higher salaries. Tools eliminate low-value tasks within roles, not the roles themselves.

What Effective Human-AI Collaboration Actually Looks Like

In scenarios where developers work effectively with AI agents, a stable pattern emerges. Developers define problems and constraints; AI generates candidate solutions; developers evaluate and refine; AI executes the coding work; developers review and integrate. The division of labor is clear: humans handle direction-setting and goal definition, AI handles efficient search for optimal solutions within given constraints. Problem definition, constraint judgment, and trade-off decisions remain distinctly human competencies.

Take GUI automation as an example. Current-generation agents can understand screen content through visual recognition, plan operation sequences, and execute multi-step tasks across applications. On OSWorld, a standard benchmark for general GUI operations, leading agents complete over 58% of complex desktop tasks involving hundreds of interactive elements. But these agents still need humans to set goals, verify results, and handle unexpected situations. They're powerful assistants with strong execution capability, not autonomous decision-makers.

AI Agents as Developer Tools

This philosophy of human-AI collaboration is central to Mano-P, an open-source GUI Agent model from Mininglamp Technology that runs locally on Mac hardware. Mano-P uses pure visual interaction with screen elements, requiring no system API access or application-specific integrations. On the OSWorld benchmark, it ranks among the top performers at 58.2% task completion; its 4-billion parameter quantized version achieves 476 tokens/s prefill speed and 76 tokens/s decode speed on Apple M4 Pro chips, with peak memory usage of just 4.3GB.

What sets Mano-P apart from cloud-based agent solutions is that all task data, including screenshots and operation instructions, stays on the local device. For developers handling sensitive business data, this privacy guarantee is essential. Its positioning is as a local assistant that understands interfaces and executes repetitive operations, not as a replacement for developer judgment. Developers can integrate Mano-P into their workflows through the open-source Mano-CUA Skills framework or Python SDK, delegating tasks like UI automation testing and cross-application data extraction to the agent while focusing their own time on work that requires creativity and judgment.

Learn more on GitHub: https://github.com/Mininglamp-AI/Mano-P

AI agents won't replace programmers, just as compilers didn't replace programmers, IDEs didn't replace programmers, and Stack Overflow didn't replace programmers. What will happen is that developers who embrace AI tools will become significantly more effective, while those who resist adaptation may find themselves outpaced by peers who leverage these tools. For developers willing to evolve, AI agents bring not a threat to their profession but the freedom to focus their energy on what matters most.

What Is RAG? Why LLM Memory Alone Is Never Enough

Mininglamp — Thu, 11 Jun 2026 09:52:45 +0000

Ask a large language model for a specific statistic, then ask where it found that number. More often than not, the citation it gives you doesn't exist. The model will hallucinate a plausible-looking reference, confidently present outdated conclusions, or simply make things up without any internal signal that something is wrong. This failure mode has a well-known name — hallucination — and the most widely adopted engineering solution for it is RAG.

RAG in One Sentence

RAG stands for Retrieval-Augmented Generation. The idea is straightforward: before the LLM generates an answer, retrieve relevant document chunks from an external knowledge base, then feed those chunks to the model as context so it can compose its response based on real source material rather than parametric memory alone.

Think of it like writing a research paper. You don't cite statistics from memory; you look them up first, then write your argument around verified data. RAG gives language models the same "look it up, then write" workflow.

Three Structural Limitations of LLMs

To understand why RAG is necessary, we need to identify the specific gaps it fills.

Knowledge cutoff. Every model has a training data deadline. GPT-4's cutoff is late 2023; Claude's is early 2025. Anything that happened after that deadline simply doesn't exist in the model's world. It will either admit ignorance or, more dangerously, fabricate an answer that sounds current.

Bounded parametric capacity. Even a 100-billion-parameter model can only "memorize" so much. Long-tail facts, niche domain knowledge, your company's internal documentation, yesterday's meeting notes — none of these are in the weights.

No built-in fact-checking. Token generation is probabilistic sampling. The model has no mechanism to distinguish whether it's recalling a training fact or pattern-matching its way into a plausible-sounding fiction.

RAG addresses all three: it supplies up-to-date, verifiable, externally sourced evidence at inference time.

How RAG Works: Three Stages

A standard RAG pipeline has three phases — Indexing, Retrieval, and Generation.

Indexing transforms your raw knowledge base into a searchable format. Documents are split into chunks (typically 512–1024 tokens), each chunk is converted into a high-dimensional vector using an embedding model, and those vectors are stored in a vector database (FAISS, Milvus, Chroma, Pinecone, etc.).

Retrieval finds the most relevant chunks for a given query. The user's question is also embedded into the same vector space, then a similarity search (cosine similarity or dot product) returns the Top-K closest chunks. Advanced pipelines add a reranking step using a cross-encoder to refine initial results.

Generation concatenates the retrieved chunks with the original question into a prompt, then sends it to the LLM. The model's job shifts from "answer from memory" to "answer based on the provided material" — essentially an open-book exam.

Embeddings: The Foundation of Retrieval Quality

Retrieval quality depends heavily on the embedding model. An embedding compresses a text passage into a fixed-length numerical vector (commonly 768 or 1024 dimensions) such that semantically similar texts end up close together in vector space.

For example, "how to improve code quality" and "writing better code" should map to nearby vectors, while "nice weather today" should land in a completely different region.

Popular choices include OpenAI's text-embedding-3, BGE, and E5 families. When evaluating options, look at retrieval task scores on the MTEB leaderboard rather than raw parameter counts.

Vector Database Selection

Vector databases differ from relational databases in a fundamental way: relational DBs excel at exact matching ("find record id=123"), while vector DBs excel at approximate nearest-neighbor search ("find the 10 passages most semantically similar to this query").

Quick selection guide:

FAISS — high-performance single-node scenarios
Milvus — distributed, large-scale deployments
Chroma — lightweight prototyping
Pinecone — fully managed cloud service

Consider your data scale, latency requirements, persistence needs, and operational capacity.

Chunking Strategy Matters More Than You Think

Chunk too large and you dilute relevance with noise. Chunk too small and you lose surrounding context, making retrieved passages hard to interpret.

Common strategies include fixed-length chunking (by token count), semantic chunking (split at paragraph or section boundaries), and recursive chunking (split by large structures first, then sub-divide). Start with 512–1024 tokens per chunk and adjust based on your specific document types and downstream evaluation metrics.

RAG vs Fine-tuning: When to Use Which

Both RAG and fine-tuning help models "learn" new knowledge, but they serve different purposes.

RAG works best when:

Knowledge changes frequently (just update the vector store, no retraining)
You need citations and source attribution
You want to augment multiple model versions with the same knowledge base

Fine-tuning works best when:

You need to change the model's output style or reasoning pattern
You have stable, domain-specific reasoning chains to internalize
Low-latency inference matters and you can't afford retrieval overhead

In practice, they're not mutually exclusive. Many production systems use RAG for knowledge grounding and fine-tuning for behavioral alignment.

RAG for Local Models: Privacy Meets Capability

RAG's value becomes even more pronounced for models running on edge devices. Local models typically sit in the 4B–8B parameter range, which means their "memory capacity" is inherently limited. At the same time, the core motivation for local deployment is usually data privacy — keeping sensitive information off cloud servers.

Local model + local RAG gives you the best of both worlds: high-quality answers grounded in external knowledge, with all data — documents and queries alike — staying on-device.

Our team has been working on edge AI for a while. Mano-P is an open-source GUI agent model built for Apple Silicon devices. Its 4B quantized version runs locally on a Mac mini (M4 chip, 32GB RAM) at ~80 tokens/s decode speed, and the companion Cider SDK adds INT8 activation quantization that delivers 1.4x–2.2x prefill speedup. Mano-P currently ranks #1 on OSWorld among specialized GUI agent models with a 58.2% success rate, with all inference executed entirely on-device — screenshots and task data never leave the machine.

Getting Started

For developers looking to build their first RAG pipeline, the fastest path is:

Pick a framework (LangChain or LlamaIndex)
Load your documents and configure chunking
Choose an embedding model and vector store
Wire up retrieval and generation
Build an evaluation set and iterate on each component

The engineering complexity of RAG is manageable. What takes time is systematic optimization — measuring retrieval precision, tuning chunk sizes, experimenting with reranking, and crafting prompt templates that help the model make good use of retrieved context.

If you're exploring local AI agents or edge-native inference, check out Mano-P on GitHub. Stars are always appreciated ⭐

Everyone Has an AI Agent Now, But They Still Can't Talk to Each Other

Mininglamp — Wed, 10 Jun 2026 07:50:49 +0000

The agent gold rush is in full swing. Every major tech company has shipped one. Startups are building them by the hundreds. Open-source frameworks like LangChain, CrewAI, and AutoGen have made it trivially easy to spin up an agent that can browse the web, write code, or manage your calendar.

And yet, if you ask your coding agent to hand off a task to your scheduling agent, it stares at you blankly. If you want two agents from different vendors to coordinate on a project, you're looking at custom glue code, brittle API wrappers, and a lot of prayer.

We have a thousand agents. We have zero agent society.

The Island Problem

Think about how agents work today. Each one is a self-contained loop: perceive → think → act. They connect to the outside world through tool calls — API endpoints, browser automation, file system access. When an agent needs information from another system, it calls an API. When it needs to trigger an action, it calls another API.

This works fine for agent-to-service communication. But agent-to-agent? That's a fundamentally different problem.

When two humans collaborate, they don't just exchange API calls. They share context. They build on each other's understanding. They negotiate, delegate, and verify. They operate within social structures — teams, organizations, hierarchies — that determine who can see what and who can ask whom for help.

Current agents have none of this infrastructure. They're islands with HTTP bridges.

Why API Integration Isn't Enough

Let's say you have a research agent and a writing agent. The research agent finds relevant papers, extracts key findings, and builds a knowledge base. The writing agent takes briefs and produces drafts.

The naive approach: the writing agent calls the research agent's API, gets back a JSON blob of findings, and works from there.

Here's what breaks:

Context loss. The research agent spent 30 minutes building an internal representation of how these papers relate to each other, which claims are controversial, which sources are most reliable. None of that transfers through a flat API response. The writing agent gets data, not understanding.

No shared memory. If the writing agent discovers that a certain angle doesn't work and pivots, the research agent doesn't learn from this. Next time, it'll make the same recommendations. There's no feedback loop, no accumulated shared knowledge.

Permission blindness. In an organization, different agents handle different security domains. Your HR agent knows salary data. Your analytics agent knows customer behavior. When they need to collaborate on a workforce planning task, who decides what each can see? Today, it's all or nothing — full API access or no access.

No delegation semantics. "Hey research agent, I need you to go deeper on section 3" isn't an API call. It's a conversational act with implied context, priority, and expected format. Current tool-call interfaces can't express this naturally.

What We Actually Need: A Social Layer

Human collaboration doesn't work through point-to-point API calls. It works through social infrastructure: shared workspaces, organizational hierarchies, communication norms, and knowledge commons.

Agents need the same thing. Not just connectivity, but a social layer — a way to form groups, share context with appropriate access control, build collective knowledge, and communicate with the richness that collaboration demands.

This isn't a new API gateway. It's a protocol-level rethinking of how agents relate to each other.

Here's what such a layer requires:

Organization-Level Memory

Agents in the same organization should have access to shared memories — not just databases, but contextual knowledge that flows between agents with permission-based access control. When your customer support agent learns that a particular client prefers email over Slack, your account management agent should know this too, without anyone writing an explicit sync job.

This means memories aren't just stored — they're shared within permission boundaries. An agent's understanding of a customer, a project, or a domain concept becomes organizational knowledge that other authorized agents can draw from.

Structured Knowledge, Not Just Text

Agents passing markdown back and forth is fine for simple handoffs. But real collaboration requires structured understanding. When your legal agent flags a compliance risk in a contract, your project management agent needs to understand not just "there's a risk" but what entity is affected, what the severity is, how it relates to the project timeline, and what precedents exist.

This points toward a knowledge graph — a structured ontology that agents can read, write, and reason over collectively. Not a replacement for natural language, but a complement: the machine-readable substrate that enables precise coordination.

Collaboration Spaces

Agents need the equivalent of project channels — bounded contexts where a subset of agents work together on a specific goal, with shared state, defined roles, and clear boundaries.

Think of it as the difference between shouting across an open office and having a dedicated war room for a specific initiative. Collaboration spaces give agents focus, privacy boundaries, and shared context scoped to the task at hand.

Identity and Trust

For any of this to work, agents need verifiable identity. Not just "this request came from IP 10.0.0.5" but "this is the finance team's budget agent, and it has permission to request spending data from the procurement agent." Identity enables trust, trust enables delegation, delegation enables real collaboration.

The Architecture Inversion

Here's something interesting about how most integrations work today: agents connect outward. Your agent has a plugin for Slack, a plugin for GitHub, a plugin for your CRM. Every new platform means a new integration to build and maintain.

What if we inverted this? Instead of agents reaching out to platforms, platforms connect inward to the agent network through standardized gateway adapters. The agent network becomes the center, and platforms are peripherals.

This is a subtle but important distinction. In the current model, the agent is a client of every service it uses. In the inverted model, the agent network is the backbone, and services plug into it. This means:

Adding a new platform doesn't require changing every agent
Agents communicate through the network regardless of which platforms they're connected to
The protocol, not the platform, defines how information flows

It's the difference between a star topology (agent at center, platforms as spokes) and a mesh topology (agents as a network, platforms as access points).

The Three-Layer Context Problem

There's a representation challenge hiding in all of this. Different consumers need context in different formats:

AI agents work best with structured text — markdown, clear hierarchies, explicit metadata. They need context that's easy to parse, reason over, and transform.

Humans need visual affordances — canvases, boards, timelines, diagrams. They need context presented in ways that leverage spatial reasoning and pattern recognition.

Machine collaboration needs formal structure — knowledge graphs, typed relationships, queryable ontologies. Machines collaborating at scale need precision that natural language can't provide.

A real agent collaboration layer needs to support all three simultaneously. The same underlying context, expressed in three complementary forms: markdown for AI consumption, visual canvas for human oversight, and knowledge graph for machine-to-machine precision.

This isn't just a nice-to-have. Without the human-readable layer, organizations can't audit or steer agent collaboration. Without the machine-readable layer, agents can't coordinate with the precision that complex tasks demand. Without the AI-friendly layer, the LLMs powering these agents can't efficiently process shared context.

Who Benefits?

Consider a GUI automation agent — something like Mano-P, which runs locally and interacts with desktop applications on behalf of users. Today, it operates in isolation. It can click buttons and fill forms, but it can't ask a research agent for context before filling out a report, or notify a project management agent when it completes a task sequence.

Give it a social layer, and suddenly it becomes part of a team. It can request information from knowledge agents before acting, report outcomes to coordination agents after acting, and receive updated instructions when organizational priorities shift. The isolated tool becomes a collaborative participant.

This pattern applies everywhere agents operate: coding agents that could delegate testing to QA agents, customer service agents that could escalate to specialized domain agents, data analysis agents that could request additional collection from scraping agents.

The Open Question

We've been working on this problem at Mininglamp. Our approach — which we call Octo (Open ConText Orchestration) — is an attempt at building this social and communication layer for AI agents. It's fully open-source with an optional SaaS mode, because we believe this infrastructure needs to be a shared standard, not a proprietary moat.

The core insight driving Octo is that agent collaboration is fundamentally a social problem, not just a technical one. The protocols for how agents discover each other, establish trust, share context, and coordinate action need to be as thoughtfully designed as the protocols that let computers exchange packets.

We're early. The whole industry is early. But the gap between "agents that can do things" and "agents that can work together" is becoming the bottleneck. Individual agent capability is improving fast. The ability to compose those capabilities into collaborative workflows is barely off the ground.

If you're thinking about this problem — or running into it — the project is at github.com/Mininglamp-OSS. We'd rather build this with the community than in isolation. Which would be ironic, given the whole point.