I built a 17-agent AI swarm on my phone — here's how

I built a 17-agent AI swarm on my phone — here's how

Okay, buckle up. This is going to be a bit of a deep dive. For the past few months, I've been obsessed with the idea of running a meaningful AI swarm locally, on my phone. Not just a demo, not just a toy, but something that could genuinely process information and react in a somewhat intelligent way. And I did it. I built a system running 17 independent AI agents, all orchestrated on my Pixel 7.

This isn't about GPT-4 or large language models. This is about small, focused agents working together, powered by a surprising amount of ingenuity and a healthy dose of optimization. Here's the story, the tech, and the lessons learned.

The "Why" - Swarm Intelligence & Edge Computing

First, let’s talk motivation. I'm fascinated by swarm intelligence – the emergent behavior that arises from simple agents following simple rules. Think ants building a colony, or birds flocking. The idea of translating that to AI, where individual agents handle specific tasks and combine their efforts, feels powerful.

But why on my phone? Well, edge computing is a big deal. Processing data locally offers privacy, reduced latency, and resilience to network outages. Plus, it’s a fun challenge! The limitations forced me to be incredibly efficient. I wanted to prove that complex AI applications don't always need massive cloud infrastructure.

The Tech Stack: Python, TensorFlow Lite & Task Allocation

My core language is Python, so that was a given. The real game-changer was TensorFlow Lite (TFLite). TFLite is Google’s framework for running TensorFlow models on mobile and embedded devices. It’s optimized for low latency and small model size.

Here’s the breakdown of the key components:

• Python (backend): Orchestration, agent communication, data handling.
• TensorFlow Lite (agents): Each agent has a dedicated TFLite model.
• Protobuf (communication): Efficient, serialized communication between agents.
• SQLite (local data store): A lightweight database for shared knowledge and agent memory.
• Chainguard (for image processing): A lightweight object detection library.

The agents aren’t all LLMs. That's key. I focused on targeted, specialized tasks. The swarm’s function? Visual scene understanding and anomaly detection, specifically aimed at monitoring my workshop. I wanted it to "notice" when things are out of place, when tools are missing, or potentially, if something dangerous is happening.

Designing the Agents – A Diverse Collective

Here's a glimpse of the 17 agents I built, categorized by their function:

1. Perception (6 Agents):

• Object Detector (3 Agents): Each trained on a specific set of objects (tools, components, safety gear) using TFLite model conversion from a pre-trained MobileNet SSD model. These run Chainguard to identify and locate items in images.
• Color Analyzer (1 Agent): Determines dominant colors in a scene, used for identifying anomalies (e.g., a red wire where there shouldn't be one).
• Shape Recognizer (2 Agents): Identifies basic shapes (squares, circles, triangles) for assessing arrangement and changes in the environment.

2. Reasoning & Analysis (6 Agents):

• Anomaly Detector (2 Agents): Compares current scene data to a baseline (see Data Collection & Training). Uses a simple statistical model implemented in TFLite.
• Spatial Relationship Analyzer (2 Agents): Analyzes the positions of objects relative to each other. For example: "Tool A is usually on shelf B". Uses simple distance calculations and rule-based logic.
• Pattern Recognizer (2 Agents): Identifies repeating patterns (e.g., the arrangement of tools on a workbench) and flags deviations. Again, a simplified model in TFLite.

3. Communication & Coordination (5 Agents):

• Data Aggregator (1 Agent): Collects data from perception agents and forwards it to reasoning agents.
• Alert Manager (1 Agent): Receives alerts from reasoning agents and decides whether to notify the user.
• Memory Manager (1 Agent): Stores and retrieves information in the SQLite database.
• Task Distributor (2 Agents): Assigns tasks to agents based on availability and expertise.

Code Snippet: Basic Agent Communication (Protobuf)

This is a simplified example of how agents communicate using Protocol Buffers:

# Define the protobuf message
import my_swarm_pb2

# Agent 1 sending a detection
detection = my_swarm_pb2.DetectionResult()
detection.object_name = "Hammer"
detection.confidence = 0.85
detection.x = 100
detection.y = 200
serialized_data = detection.SerializeToString()

# Agent 2 receiving the detection
received_detection = my_swarm_pb2.DetectionResult()
received_detection.ParseFromString(serialized_data)
print(f"Received: Object: {received_detection.object_name}, Confidence: {received_detection.confidence}")

This shows how messages are defined, serialized, and deserialized, enabling efficient data exchange between agents.

Data Collection & Training – The Foundation

This was the most time-consuming part. I spent weeks taking hundreds of photos of my workshop in various states. I manually labeled objects in each image using a tool like LabelImg. This labeled data was used to train the TFLite object detection models.

The training process itself was done on my desktop using TensorFlow, then converted to TFLite using the TensorFlow Lite Converter. This conversion is crucial for reducing model size and optimizing for mobile performance.

Optimization – Squeezing Every Last Drop of Performance

Running 17 agents concurrently on a phone requires serious optimization. Here's what I did:

• Model Quantization: Using post-training quantization in TFLite, I reduced the precision of the models, significantly decreasing their size and inference time.
• Thread Management: Carefully managed threads to prevent resource contention and ensure smooth operation.
• Data Batching: Processed images in smaller batches instead of trying to analyze the entire frame at once.
• Selective Agent Activation: Not all agents run constantly. Agents are activated only when their expertise is needed. For example, the Color Analyzer only runs when significant color changes are detected.
• Code Profiling: Used Python’s cProfile module to identify performance bottlenecks and optimize critical code paths.

Challenges & Lessons Learned

This project wasn’t all smooth sailing. I ran into several challenges:

• Memory Constraints: Phones have limited RAM. Efficient memory management was critical.
• Battery Life: Running 17 agents drains the battery fast.