I built a 17-agent AI swarm on my phone — here's how

I built a 17-agent AI swarm on my phone — here's how

Okay, buckle up. This is a bit of a wild ride. For the last few weeks, I've been obsessed with the idea of running a functional, multi-agent AI system entirely on my phone. Not just a single model, but a swarm of 17 distinct agents, all communicating and collaborating. It sounded crazy. Resource-intensive. Impractical. Turns out, with a bit of clever engineering and leveraging some fantastic open-source tools, it's surprisingly possible.

This isn't about running GPT-4 locally (yet!). This is about building something more… bespoke. Something that demonstrates the power of localized AI and the potential for truly personal, on-device intelligence. Here's the breakdown of how I did it.

The Goal: A Context-Aware Productivity Assistant

I didn't just want a bunch of bots chattering to each other. I wanted something useful. My vision was a productivity assistant that could handle tasks like:

• Note Taking & Summarization: Transcribing voice notes and generating concise summaries.
• Contextual Research: Researching topics based on the current content on my screen.
• Code Snippet Generation: Generating basic code snippets for quick tasks.
• To-Do List Management: Adding tasks, prioritizing, and setting reminders.
• Email Drafting: Composing basic email responses.

To accomplish this, I needed multiple agents, each specializing in a specific function.

The Tech Stack: Python, llama.cpp, and a lot of Optimization

The core of this project is built on the following:

• Python: For orchestration and logic. The glue that holds everything together.
• llama.cpp: This is the real hero. llama.cpp is a fantastic port of the Llama 2 model (and others!) to C++, optimized for running on CPUs, even low-power ones. Crucially, it has Python bindings. This allows me to run a quantized Llama 2 model directly on my Android phone. I specifically used the Q4_K_M quantization which provides a good balance between speed and accuracy.
• Android Termux: Termux is an Android terminal emulator and Linux environment. It provides a command-line interface and allows me to install Python and other dependencies.
• OpenAI-like API Wrapper: I built a simplified API wrapper around llama.cpp to mimic the OpenAI API. This allows the agents to interact with the model in a standardized way.
• LangChain (Lite): I needed a framework for building agent chains. LangChain is powerful, but heavy. I cherry-picked only the necessary components (primarily agents and tools) to keep the overhead minimal.
• Airdrop-inspired Communication: To facilitate agent communication, I created a simple message passing system where agents can "broadcast" messages and other agents can "subscribe" to specific topics.

The Agents: The 17 Members of the Swarm

Here's a simplified overview of the agents I created:

1. Voice Input Agent: Captures voice notes using the phone’s microphone.
2. Transcription Agent: Transcribes audio using a lightweight speech-to-text model (Wav2Vec2).
3. Summary Agent: Summarizes text using the Llama 2 model.
4. Research Agent: Uses a custom search tool (more on that later) to gather information.
5. Code Generation Agent: Generates code snippets based on prompts.
6. Email Draft Agent: Creates email drafts.
7. To-Do List Agent: Manages tasks and reminders.
8. Context Extraction Agent: Analyzes the current screen content (using accessibility services) to extract relevant information.
9. Prompt Refinement Agent: Improves prompts before sending them to the Llama 2 model. (This is surprisingly effective!)
10. Memory Agent: Stores and retrieves information across sessions (a simple in-memory store for now).
11. Routing Agent: Determines which agent should handle a given task. (The central coordinator)
12. Action Execution Agent: Executes actions like adding tasks to the to-do list.
13. Formatting Agent: Formats output for readability.
14. Sentiment Analysis Agent: Analyses the sentiment of text
15. Translation Agent: Translates text between languages
16. Filtering Agent: Filters out irrelevant information
17. Error Handling Agent: Handles errors and provides feedback

Code Snippet: Simplified Agent Interaction (Python)

This illustrates how the Routing Agent dispatches tasks:

class RoutingAgent:
    def __init__(self):
        self.agent_map = {
            "summarize": SummaryAgent(),
            "research": ResearchAgent(),
            "code": CodeGenerationAgent(),
            # ... other agents
        }

    def route_task(self, task_type, data):
        if task_type in self.agent_map:
            return self.agent_map[task_type].process(data)
        else:
            return "Error: Unknown task type."

Each agent has a process() method that handles its specific task. The MessageBroker (not shown) handles the broadcasting and subscription of messages between agents.

The Biggest Challenge: Performance and Optimization

Running Llama 2 (even a quantized version) on a phone is challenging. Here's what I did to optimize performance:

• Quantization: As mentioned, Q4_K_M quantization drastically reduces the model size and computational requirements.
• Offloading to GPU (Where Possible): Some phones have capable GPUs. llama.cpp can leverage them for acceleration.
• Asynchronous Processing: I used Python's asyncio library to run multiple agents concurrently, maximizing CPU utilization.
• Memory Management: I carefully managed memory usage to avoid crashes. Loading the model once and reusing it is crucial.
• Strategic Caching: Caching frequently used results reduced redundant computations.
• Minimal LangChain: As mentioned, I stripped down LangChain to only the essential components.

The Search Tool: A Lightweight Alternative

Using a full-fledged search API on a phone is impractical. I created a custom search tool that leverages DuckDuckGo's instant answer API for quick information retrieval. It’s not perfect, but it’s fast and doesn’t require excessive bandwidth.

Limitations and Future Improvements

This is a proof-of-concept, and it has limitations:

• Model Size: Even quantized, the Llama 2 model is still large. Larger models offer better performance, but also increase resource requirements.
• Processing Speed: Response times