I built a 17-agent AI swarm on my phone — here's how

I built a 17-agent AI swarm on my phone — here's how

Okay, buckle up. This is a bit of a rabbit hole, and it involved a lot of late nights fueled by coffee and the sheer stubbornness to prove it could be done. I’ve built a 17-agent AI swarm that runs, surprisingly well, on my phone. Not in the cloud. On my phone.

Before you start picturing some Skynet-lite situation, it's not world-dominating AI. It’s a proof-of-concept for distributed intelligence and on-device machine learning, geared towards real-time environment analysis. Think sophisticated object recognition, anomaly detection, and potentially even predictive behavior modeling – all happening locally, without sending data to a server.

Why? Privacy, latency, and the challenge itself. I wanted to see if I could leverage the increasing processing power of modern smartphones to create a genuinely useful, complex system without relying on cloud connectivity.

The Core Concept: Agent-Based Swarming

The architecture is based on the idea of swarm intelligence. Each “agent” is a relatively simple AI module focused on a specific task. Individually, they're not terribly impressive. But when combined and allowed to communicate (albeit limitedly), they create emergent behavior that’s far more powerful.

My swarm comprises 17 agents, broken down into a few key categories:

• Perception Agents (6): These handle image processing. They include:
  • Object Detector (YOLOv5-Lite): Identifies common objects like people, cars, furniture, etc.
  • Face Recognizer (MobileFaceNet): Identifies known faces.
  • Emotion Detector (FERNet): Attempts to infer emotional state from facial expressions. (This is the least reliable, admittedly.)
  • Color Histogram Analyzer: Extracts color information for scene understanding.
  • Edge Detector: Identifies edges and shapes for rudimentary scene parsing.
  • Motion Detector: Detects movement in the camera feed.
• Analysis Agents (5): These take the output from Perception Agents and perform higher-level reasoning.
  • Anomaly Detector: Flags unusual combinations of detected objects or activity.
  • Contextual Awareness Agent: Uses time of day and location data (if permitted) to refine analysis.
  • Behavior Predictor (Simple LSTM): Attempts to predict potential actions based on observed patterns. (Very basic currently.)
  • Spatial Relationship Analyzer: Determines the relationships between detected objects (e.g., “person near car”).
  • Event Summarizer: Creates a concise text description of the scene.
• Communication & Control Agents (6): These manage the flow of information and overall system behavior.
  • Data Router: Directs data between agents.
  • Priority Manager: Determines which agents get access to processing resources.
  • Alert Trigger: Activates notifications based on detected anomalies.
  • State Manager: Maintains the overall system state.
  • Resource Monitor: Tracks CPU, memory, and battery usage.
  • Logging Agent: Records activity for debugging and analysis.

The Tech Stack: TinyML and Flutter

This wasn't going to be a straightforward Python/TensorFlow project. Running 17 AI models concurrently on a phone requires extreme optimization. That’s where TinyML comes in.

I primarily used TensorFlow Lite (TFLite) for deploying the pre-trained models. TFLite is designed for mobile and embedded devices and significantly reduces model size and computational requirements. I also experimented with CoreML on iOS, but TFLite offered better cross-platform compatibility.

For the user interface and the core logic of coordinating the agents, I chose Flutter. Flutter’s performance is surprisingly good, and its hot-reloading capabilities were invaluable during development.

Here’s a snippet of Flutter code showing how I load and run the YOLOv5-Lite model:

import 'package:tflite/tflite.dart';

Future<void> loadYolov5Model() async {
  String? res = await Tflite.loadModel(
    model: "assets/yolov5s_lite.tflite", // Path to your TFLite model
    labels: "assets/coco_labels.txt",   // Path to labels file
  );
  print('YOLOv5 Model loaded: $res');
}

Future<List<dynamic>?> runYolov5(Uint8List imageBytes) async {
  var recognitions = await Tflite.runModelOnFrame(
    bytesList: imageBytes,
    imageMean: 127.5,
    imageStd: 127.5,
    numResults: 5, // Number of top results to return
    threshold: 0.3,
  );
  return recognitions;
}

Important Considerations for Performance:

• Quantization: I heavily relied on model quantization. Converting models to 8-bit integer format dramatically reduced their size and inference time.
• Model Pruning: Removing unnecessary weights from the models further optimized performance.
• Asynchronous Processing: Each agent runs in its own isolate (a lightweight thread in Dart), preventing the UI from freezing.
• Limited Communication: Agents don't constantly broadcast data. Communication is event-driven and focused on essential information.

The Communication Protocol: A Simple Message Bus

The communication between agents is based on a rudimentary message bus. Each agent subscribes to specific "topics" (e.g., "object_detections", "motion_events"). When an agent detects something relevant, it publishes a message to the appropriate topic.

This is handled via a custom Dart class:

class Message {
  final String topic;
  final dynamic data;

  Message({required this.topic, required this.data});
}

class MessageBus {
  final Map<String, List<Function>> subscribers = {};

  void subscribe(String topic, Function callback) {
    if (!subscribers.containsKey(topic)) {
      subscribers[topic] = [];
    }
    subscribers[topic]!.add(callback);
  }

  void publish(Message message) {
    if (subscribers.containsKey(message.topic)) {
      subscribers[message.topic]!.forEach((callback) => callback(message.data));
    }
  }
}

This approach, while simple, keeps the overhead low. More sophisticated messaging systems (like MQTT) are possible, but would introduce significant complexity and potentially impact performance.

Challenges and Future Work

It wasn’t all smooth sailing.

• Battery Drain: Running this continuously will drain your battery. Optimizing power consumption is a