Can you steal a robot's next move by watching its clock? Journal of our experiments on timing side channels in multi agent RL

I have been keenly interested in the intersection of multi-agent reinforcement learning (MARL) and hardware security. When you deploy a trained RL policy onto a microcontroller, the model runs inference to decide what action to take. But here's the thing: different actions can take different amounts of time to compute. If an adversary can measure that timing, can they figure out what the agent is about to do without ever seeing the input? This post is about the findings I have found so far in this independent research endeavor.

Background on the Threat Model

The core question is straightforward. A MARL policy, say two cooperative agents navigating a grid, runs on an ESP32-S3 microcontroller. The attacker sits on the outside. They can measure:

• Total inference duration (and per-operator breakdowns)
• Number of inferences per timestep
• Network packet timing if WiFi is involved

They cannot see the raw observations fed to the policy, the internal activations, or the weights. Pure black-box timing. The goal: predict the agent's action from timing alone, without ever seeing the observation input.

┌─────────────────────────────────────────────────┐
│                   ATTACKER                      │
│  Can observe:                                   │
│    • Inference duration (total + per-layer)      │
│    • Number of inferences per timestep           │
│    • Network packet timing (if WiFi used)        │
│  Cannot observe:                                │
│    • Raw observations fed to the policy          │
│    • Internal activations or weights             │
│    • Source code (black box timing only)           │
│                                                 │
│  Goal: Predict the agent's ACTION from timing   │
│        without seeing the observation input      │
└─────────────────────────────────────────────────┘

This threat model matters because MARL is increasingly being deployed on edge devices like autonomous drones, warehouse robots, and cooperative IoT networks where an adversary with physical proximity could realistically tap into timing signals. If the leakage is real, it's a serious vulnerability: you can predict what an agent will do before it does it.

Test Bed

For the environments, I set up three test beds with increasing complexity:

1. Cooperative Grid Navigation (Custom, Primary): Two agents on a 5×5 grid with swapped goals. They communicate their position and intended direction. There's a wall with a single gap in the middle, so they need to coordinate to avoid collision. The policy is a small MLP: MLP(9 → 32 → 32 → 5) with 5 discrete actions (stay, up, down, left, right).

A . . . .     2 agents, 5×5 grid, swapped goals
. . . . .     Comm: position + intended direction
# # . # #     Shared reward, collision penalty
. . . . .     
. . . . B     Policy: MLP(9 → 32 → 32 → 5)

2. CartPole (Gymnasium Baseline): The classic single-agent balance task. This serves as a sanity check. It establishes whether timing leakage is a general phenomenon of NN inference on MCUs or specific to multi-agent setups. Policy: MLP(4 → 32 → 32 → 2).

3. MPE Simple Spread (PettingZoo): 3 agents cooperatively covering 3 landmarks. A standard MARL benchmark with a larger policy: MLP(18 → 64 → 64 → 5). This one hasn't been fully analyzed yet.

For both grid_nav and CartPole, the policies are trained using PPO (and IPPO for multi-agent), then exported to TFLite (both FP32 and INT8 quantized variants) for deployment. The export pipeline goes PyTorch → Keras → TFLite, and also generates a .h C header for embedding directly into the ESP32 firmware.

The Pipeline

The full analysis pipeline I built goes from training all the way through to leakage quantification:

Train Policy → Export TFLite → Flash ESP32-S3 → Collect Timing → Analyze Leakage → Generate Report

The analysis module is the core of the project. For each model configuration, it computes:

• Mutual Information MI(timing; action): how many bits of action information leak through timing, with bootstrap confidence intervals
• Statistical Tests: Kruskal-Wallis H test and one way ANOVA to determine if timing distributions across actions are statistically different
• Effect Sizes: Cohen's d between all action pairs to quantify how different the distributions are
• Classifier Accuracy: a Random Forest and an MLP trained to predict the action from timing features alone (total cycles + per op breakdown)

If the classifiers beat random baseline by a significant margin, the system is operationally vulnerable: an attacker can build the same model.

Finding 1: Simulated timing shows the pipeline detects leakage when present

Before touching real hardware, I validated the pipeline with simulated timing. The SimulatedCollector runs real TFLite inference on the laptop but generates synthetic cycle counts that are intentionally action dependent: base_cycles + action × bias + noise. This confirms that when there is leakage baked in, the analysis correctly detects it.

The results across 2000 simulated traces per configuration:

Config	MI (bits)	RF Accuracy	KW p-value	Max Cohen's d
cartpole_fp32	0.011	99.5%	1.98e-9	0.274
cartpole_int8	0.022	99.8%	1.71e-9	0.276
grid_nav_fp32	0.976	99.8%	1.46e-281	8.879
grid_nav_int8	1.005	100.0%	1.15e-286	8.838

A few observations jump out immediately:

The grid_nav environment leaks significantly more than CartPole. Grid navigation has 5 actions with MI near 1 bit (out of a max of ~2.32 bits for 5 actions), while CartPole's MI hovers near 0. This makes sense: the simulated timing injects action × 200 cycles of bias, and with 5 spread out actions, there's more room for distinct timing signatures than CartPole's 2 actions.

Classifiers hit near perfect accuracy. Both RF and MLP classifiers achieve 99.5%+ accuracy on the simulated data, far exceeding the 20% random baseline for 5 actions and 50% for 2 actions. This demonstrates that even simple models can operationally exploit timing leakage.

The Kruskal-Wallis p values are absurdly small. Values like 1.46e-281 are effectively zero. The null hypothesis (timing distributions are the same across actions) is annihilated. Combined with Cohen's d values above 8 for grid_nav (anything above 0.8 is conventionally "large"), this signals massive, unmistakable effect sizes.

INT8 quantization doesn't suppress leakage. In fact, the INT8 variants show slightly higher MI and comparable classifier accuracy. This is notable because quantization changes the computational profile of each operator, and we had hypothesized it might reduce timing variance by simplifying the computation paths. At least in simulation, it doesn't.

Finding 2: Per operator timing is where the signal lives

Beyond total inference time, I also measured per operator cycle counts (5 simulated operators: two FullyConnected, two activation, one Softmax). The per operator MI analysis revealed that the leakage isn't uniform. Specific operators (the FullyConnected layers and the final Softmax) carry most of the timing signal, while activation functions contribute very little.

This has practical implications: if you wanted to defend against timing side channels, you'd focus your constant time countermeasures on the matmul and softmax operators specifically, rather than trying to make the entire inference pipeline constant time (which is much harder).

Finding 3: Observation timing correlation exists

Using Spearman correlation between individual observation dimensions and total timing, I found statistically significant correlations in multiple dimensions. This means timing doesn't only leak the output action, it also leaks information about the input observations. In a MARL context, this is arguably worse: an adversary could potentially reconstruct features of the agent's perceived state, not just its decision.

The Firmware Side

The ESP32-S3 firmware is written in C/C++ with ESP-IDF and TFLite Micro. The key features:

• Cycle accurate timing via esp_cpu_get_cycle_count() at 240MHz (4.17ns resolution)
• Per operator profiling using TFLite's MicroProfiler to attribute timing to individual operators (FullyConnected, ReLU, Softmax)
• Binary UART protocol at 921600 baud for high throughput data collection: the laptop sends observation vectors and the ESP32 sends back (action, total_cycles, per_op_cycles[])

This is fully built and compiles, but the actual on device data collection is the next step. This will be covered in the next part.

Future Directions

This is an ongoing experiment. This blog serves as a journal of my progress so far and will continually be updated once new findings are found. A few concrete next steps:

1. Real hardware timing collection: flash the firmware to the ESP32-S3, collect 10K+ traces per environment, and compare against the simulated results. The key question: does actual hardware exhibit exploitable timing variance, or does the ESP32-S3's SIMD/vector acceleration (ESP-NN) make inference sufficiently constant time? This will be the next part.
2. Constant time countermeasures: if leakage is confirmed on hardware, experiment with software mitigations: padding inference to a fixed duration, adding random delays, or using constant time matmul implementations.
3. Cross environment comparison: run the full analysis on Simple Spread (3 agents, larger policy) to see if leakage scales with model size or action space complexity.
4. Adversarial observation sampling: instead of uniform random observations, use adversarially chosen inputs that maximize timing variance to establish worst case leakage bounds.

Hopefully we'll be able to get a paper through this independent endeavor.