I built a 17-agent AI swarm on my phone — here's how

I built a 17-agent AI swarm on my phone — here's how

Okay, deep breath. This is a bit of a wild ride to explain, but I'm incredibly excited to finally share something I've been tinkering with for the past few months. I've built a 17-agent AI swarm that runs entirely on my Android phone. Not through cloud APIs (mostly), but actual on-device processing. It's a project born from curiosity, a love of emergent behaviour, and frankly, a desire to push the limits of what's possible with modern mobile hardware.

This isn’t about creating a single super-intelligent AI. It’s about a swarm – a collection of relatively simple agents, each with a specific task, interacting with each other to achieve a larger, un-programmed, emergent goal. Think ant colony, not Skynet.

Why on-device?

Good question. Why not just leverage the cloud? Several reasons. First, latency. Real-time interaction within the swarm is crucial. Cloud trips would introduce unacceptable delays. Second, privacy. I didn’t want data leaving my phone. Third, and honestly the most fun, the challenge. Could I actually run this on a phone? It turns out… mostly, yes.

The Tech Stack

This wasn't a straightforward Python/TensorFlow affair. While Python is fantastic, on-device execution requires a different approach. Here’s what I used:

• Language: JavaScript (specifically TypeScript for type safety). It runs natively in a web worker, and importantly, is well-supported by web assembly (more on that later).
• Framework: No heavy framework. I leaned heavily on vanilla JavaScript, supplemented with a lightweight vector math library, gl-matrix.
• On-device Execution: WebAssembly (WASM). This is the key. WASM allows me to compile optimized code (written in C++ for the computationally intensive parts) to a binary format that runs efficiently in the browser – or, in this case, a headless browser implemented with a webview in a native Android app.
• Android App: Built using Flutter. This handles the webview, sensor access, and UI.
• Communication: The agents communicate via a central message broker, implemented as a simple in-memory queue.
• Sensors: Accelerometer, gyroscope, and magnetometer. These provide the swarm with environmental input.

The Agents: A Diverse Crew

The beauty of a swarm is the diversity. Each agent has a dedicated purpose, contributing to the overall behaviour. Here's a breakdown of the 17 agents:

• Sensor Input (x3): Raw sensor data processing. These agents read data from the accelerometer, gyroscope, and magnetometer, smoothing it and converting it into meaningful signals (e.g., orientation changes, acceleration magnitude).
• Pattern Recognition (x4): These analyse the sensor data for specific patterns – sudden movements, sustained tilt, vibrations. They each look for different patterns using simple thresholding and moving average calculations.
• Spatial Mapping (x3): Based on the recognized patterns, these agents build a rudimentary map of the phone's movement and environment. Think of it as a highly abstracted understanding of 'up', 'down', 'left', 'right', and 'shake'.
• Navigation (x2): These use the spatial map to predict future movement. They’re not aiming for perfect accuracy, but rather probabilities.
• Action Selection (x3): These agents decide what "action" to take based on the navigation predictions. Actions are simple – sending commands to other agents (e.g., "increase sensor sensitivity," "flag potential event").
• Event Logger (x2): These log events triggered by the action selectors. This data is used for rudimentary learning – identifying which actions were correlated with significant sensor patterns.

Code Snippet: A Simple Agent (TypeScript)

Let's look at a simplified version of a Sensor Input agent:

class AccelerometerAgent {
  constructor(private messageBroker) { }

  processData(timestamp: number, data: Float32Array) {
    // Simple smoothing filter
    let smoothedX = 0;
    let smoothedY = 0;
    let smoothedZ = 0;

    for(let i = 0; i < data.length; i += 3) {
      smoothedX += data[i];
      smoothedY += data[i+1];
      smoothedZ += data[i+2];
    }

    smoothedX /= (data.length / 3);
    smoothedY /= (data.length / 3);
    smoothedZ /= (data.length / 3);


    this.messageBroker.sendMessage({
      type: "accelerometer_data",
      timestamp: timestamp,
      x: smoothedX,
      y: smoothedY,
      z: smoothedZ
    });
  }
}

This agent takes raw accelerometer data, smooths it, and broadcasts it to the message broker. Other agents subscribe to this message type to receive the data. The messageBroker is just a simple object with subscribe and sendMessage methods.

The WASM Component: Optimizing for Speed

The biggest bottleneck was processing the sensor data. JavaScript, while improved, isn't ideal for number-crunching. That's where WASM came in. I wrote the smoothing filters and some of the pattern recognition algorithms in C++, compiled them to WASM using Emscripten, and loaded them into the JavaScript environment.

This made a massive difference. The WASM-compiled code runs several times faster than the equivalent JavaScript, allowing the swarm to process sensor data in real-time without draining the battery.

Emergent Behaviour: What Does it Do?

So, what’s the point of all this? The swarm isn’t pre-programmed with a specific task. Instead, it’s designed to react to its environment. The emergent behaviour I've observed is fascinating.

For example:

• Activity Recognition: The swarm can reliably detect when I’m walking, running, or sitting still, based solely on the sensor data. It’s not perfect, but surprisingly accurate.
• Gesture Detection: It can detect certain gestures (e.g., a quick shake, a deliberate tilt) although these are very primitive.
• Environmental Awareness: It can build a very basic "awareness" of its surroundings by correlating sensor data with logged events.

The fascinating thing is that I didn't tell it to do these things. The behaviour emerged from the interaction of the agents and the feedback loops within the swarm. It’s like watching a simple ecosystem evolve.

Challenges & Future Work

This project wasn't without its hurdles.

• Debugging: Debugging 17 interacting agents