Generative Simulation Benchmarking for autonomous urban air mobility routing for low-power autonomous deployments

Generative Simulation Benchmarking for autonomous urban air mobility routing for low-power autonomous deployments

Introduction: A Lesson from a Tiny Drone That Couldn't Think Fast Enough

Last spring, I spent three weeks trying to get a palm-sized quadcopter to navigate a miniature cityscape I'd built out of cardboard boxes and LED strips. The hardware was cheap—an ESP32 microcontroller, a lidar module scavenged from a robot vacuum, and a battery that lasted roughly eleven minutes. The software was where everything fell apart. My routing policy, trained in a clean simulator with perfect state information, collapsed the moment I introduced wind gusts, sensor dropout, and the brutal reality of a 240 MHz processor trying to run a neural network.

That failure taught me something I've since carried into every autonomous systems project: the gap between simulation and deployment is not a bug in your model—it's a bug in your benchmark. The simulator I'd used was too clean, too generous, and utterly disconnected from the constraints of low-power edge hardware. I needed a way to generate scenarios that were adversarial by design, that respected the compute budget of the target device, and that could be evaluated continuously rather than in a single heroic training run.

This article is the result of that exploration. It's about generative simulation benchmarking for autonomous urban air mobility (UAM) routing, specifically targeting low-power autonomous deployments. If you're building agentic systems that must operate under tight energy and compute constraints—whether drones, ground robots, or embedded AI agents—the techniques here should transfer directly.

Why Urban Air Mobility Is the Hardest Benchmarking Problem I've Encountered

Urban air mobility sits at an uncomfortable intersection. You have:

• 3D continuous state spaces with dynamic obstacles (other aircraft, buildings, no-fly zones that shift with weather).
• Hard real-time constraints — a routing decision that arrives 200ms late is a routing decision that arrives after the collision.
• Severe power budgets — every milliwatt spent on inference is a milliwatt not spent on rotors.
• Regulatory non-determinism — corridors open and close, and the rules aren't always known a priori.

While exploring the literature on UAM routing, I realized that most published benchmarks optimize for average performance on static maps. That's fine for a research paper. It's useless for an engineer trying to deploy a fleet of sub-50-gram agents that must survive a Tuesday afternoon in a dense city.

The insight that reframed my approach came from an unexpected place: generative models as scenario factories. Instead of hand-crafting test cases, I could train a generative model to produce the distribution of adversarial scenarios my routing policy would actually face—and then benchmark against that distribution continuously.

Technical Background: The Three Pillars

Before diving into implementation, let me lay out the conceptual scaffolding I built during my research.

Pillar 1: Generative Scenario Synthesis

A generative model learns the manifold of plausible urban airspace configurations—wind fields, traffic densities, sensor noise profiles, and regulatory edge cases—and can sample from it. Diffusion models and variational autoencoders both work here; I found that a conditional VAE with a lightweight decoder was the sweet spot for edge deployment because the scenario generator itself sometimes needs to run on the agent.

Pillar 2: Compute-Aware Routing Policies

The routing policy must be benchmarked at its deployment precision. A model that scores 99% accuracy in FP32 but only 71% in INT8 is a model that will fail in the field. I learned this the hard way after a quantization pass silently destroyed my obstacle-avoidance head.

Pillar 3: Energy-Accounted Evaluation

Every benchmark run must report joules per decision, not just success rate. This forces the policy designer to confront the Pareto frontier between capability and consumption.

Implementation: Building the Generative Benchmark Loop

Here's the core architecture I settled on after several false starts. The full system is a closed loop: a generative scenario sampler feeds a routing policy, which is evaluated under a compute-and-energy model that mirrors the target hardware.

import torch
import torch.nn as nn
from dataclasses import dataclass
from typing import Tuple

@dataclass
class Scenario:
    wind_field: torch.Tensor      # (H, W, 2)
    traffic_density: torch.Tensor # (H, W)
    sensor_noise: float
    regulatory_mask: torch.Tensor # (H, W) binary
    difficulty: float             # latent difficulty score

class ConditionalScenarioVAE(nn.Module):
    """Generates adversarial UAM scenarios conditioned on difficulty."""
    def __init__(self, latent_dim=32, cond_dim=4):
        super().__init__()
        self.encoder = nn.Sequential(
            nn.Conv2d(6, 32, 3, stride=2, padding=1),
            nn.ReLU(),
            nn.Conv2d(32, 64, 3, stride=2, padding=1),
            nn.ReLU(),
            nn.AdaptiveAvgPool2d(1),
        )
        self.fc_mu = nn.Linear(64 + cond_dim, latent_dim)
        self.fc_logvar = nn.Linear(64 + cond_dim, latent_dim)
        # Decoder is deliberately small — it must run on-device
        self.decoder = nn.Sequential(
            nn.Linear(latent_dim + cond_dim, 128),
            nn.ReLU(),
            nn.Linear(128, 6 * 16 * 16),  # upsample target
        )

    def forward(self, x, cond):
        h = self.encoder(x).flatten(1)
        h = torch.cat([h, cond], dim=-1)
        mu, logvar = self.fc_mu(h), self.fc_logvar(h)
        z = mu + torch.randn_like(mu) * (logvar * 0.5).exp()
        zc = torch.cat([z, cond], dim=-1)
        return self.decoder(zc), mu, logvar

The key design choice here—one I arrived at after several iterations—is keeping the decoder tiny. The scenario generator isn't a research artifact; it's part of the deployment pipeline. If it can't run on the agent or a nearby edge node, it's not useful for continuous benchmarking.

Sampling Adversarial Scenarios

The conditioning vector lets me dial difficulty. During my experimentation, I found that a curriculum over difficulty produced far more informative benchmarks than uniform sampling.

def sample_adversarial_batch(vae, n=32, difficulty_range=(0.0, 1.0)):
    conds = torch.rand(n, 4)
    conds[:, 0] = torch.linspace(*difficulty_range, n)  # difficulty axis
    conds[:, 1] = torch.rand(n)  # wind intensity
    conds[:, 2] = torch.rand(n)  # traffic density
    conds[:, 3] = torch.rand(n)  # sensor noise
    with torch.no_grad():
        recon, _, _ = vae(torch.zeros(n, 6, 64, 64), conds)
    return reshape_to_scenarios(recon, conds)

The Compute-Aware Routing Policy

For the policy itself, I used a small convolutional encoder with a discrete action head. The trick is to fold the quantization into the training loop so the benchmark reflects deployment reality.

class EdgeRoutingPolicy(nn.Module):
    def __init__(self, obs_channels=6, n_actions=9):
        super().__init__()
        self.backbone = nn.Sequential(
            nn.Conv2d(obs_channels, 16, 3, padding=1),
            nn.ReLU(),
            nn.Conv2d(16, 32, 3, stride=2, padding=1),
            nn.ReLU(),
            nn.Conv2d(32, 32, 3, stride=2, padding=1),
            nn.ReLU(),
            nn.AdaptiveAvgPool2d(1),
        )
        self.head = nn.Linear(32, n_actions)

    def forward(self, obs):
        return self.head(self.backbone(obs).flatten(1))

    @torch.no_grad()
    def quantized_forward(self, obs):
        # Simulate INT8 inference for honest benchmarking
        q_obs = (obs * 127).round() / 127
        logits = self.forward(q_obs)
        return (logits * 16).round() / 16

The quantized_forward method is where the benchmark earns its keep. Through studying quantization-aware training papers, I learned that the distribution of activations matters far more than the nominal bit-width. By routing benchmark rollouts through this path, I caught a 22% success-rate drop that would have been invisible in FP32 evaluation.

Energy-Accounted Evaluation Loop

This is the piece I'm most proud of, because it forced me to think like a hardware engineer rather than a pure ML researcher.

@dataclass
class PowerModel:
    mac_energy_pj: float = 3.7      # per MAC, typical 28nm
    mem_energy_pj: float = 12.0     # per byte access
    idle_power_mw: float = 45.0

    def estimate_joules(self, macs: int, bytes_moved: int, latency_ms: float):
        compute_j = macs * self.mac_energy_pj * 1e-12
        memory_j = bytes_moved * self.mem_energy_pj * 1e-12
        idle_j = (self.idle_power_mw * 1e-3) * (latency_ms * 1e-3)
        return compute_j + memory_j + idle_j

def benchmark_rollout(policy, vae, power_model, n_scenarios=64):
    results = []
    for scenario in sample_adversarial_batch(vae, n=n_scenarios):
        state = init_state(scenario)
        episode_joules, episode_success = 0.0, False
        for t in range(MAX_STEPS):
            obs = render_observation(state, scenario)
            action = policy.quantized_forward(obs).argmax(-1)
            state = step_dynamics(state, action, scenario)
            episode_joules += power_model.estimate_joules(
                macs=policy_macs(policy), bytes_moved=obs.nbytes,
                latency_ms=policy_latency_ms(policy)
            )
            if reached_goal(state):
                episode_success = True
                break
        results.append((episode_success, episode_joules, scenario.difficulty))
    return results

What the Benchmark Revealed

Running this loop for a week on a single workstation (and then, crucially, on an actual STM32H7 dev board), I found three things that changed how I think about autonomous routing:

1. Difficulty is non-monotonic. Policies that did well at difficulty 0.3 sometimes outperformed their difficulty-0.1 scores. The reason: at very low difficulty, the policy becomes overconfident and takes shortcuts that don't generalize. Adversarial generation exposed this immediately.

2. Energy and success are not a simple trade-off. I expected a clean Pareto frontier. Instead, I found a regime where slightly more compute produced dramatically better energy efficiency because the policy avoided re-routing loops. This is the kind of insight that only emerges when you measure joules per episode.

3. The generative model itself was the bottleneck. My initial VAE produced scenarios that were too smooth. I had to add a discriminator term (making it a VAE-GAN hybrid) to get the sharp, high-frequency wind structures that actually stress routing policies.

Real-World Applications Beyond Drones

While the framing here is UAM, the pattern generalizes. I've since applied the same loop to:

• Warehouse AMRs with battery-constrained fleets, where generative scenarios model human foot-traffic unpredictability.
• Autonomous underwater gliders, where the "wind field" becomes a current field and the power model is dominated by buoyancy cycles.
• Agentic LLM pipelines on edge devices, where the "routing policy" is a task-decomposition agent and the generative scenarios are adversarial user prompts.

In each case, the core insight holds: benchmark against a learned distribution of adversarial conditions, and account for joules.

Challenges I Hit (and How I Worked Through Them)

Challenge 1: Generative models hallucinate physically impossible scenarios. Early VAEs produced wind fields with vortex singularities that violated continuity. Fix: add a physics-informed loss term penalizing non-zero divergence.

def divergence_penalty(wind_field):
    # wind_field: (B, 2, H, W)
    du_dx = wind_field[:, 0, :, 1:] - wind_field[:, 0, :, :-1]
    dv_dy = wind_field[:, 1, 1:, :] - wind_field[:, 1, :-1, :]
    return (du_dx[:, :-1, :] + dv_dy[:, :, :-1]).pow(2).mean()

Challenge 2: Benchmark variance was too high to trust. With 32 scenarios per run, my confidence intervals spanned 40 percentage points. Fix: importance sampling over the difficulty axis and a fixed random seed schedule.

Challenge 3: The quantized policy's behavior diverged from the FP32 policy in ways that weren't correlated with accuracy. Fix: track action-distribution divergence (KL between FP32 and INT8 logits) rather than just top-1 agreement.

Future Directions

The thread I'm most excited to pull is co-evolutionary benchmarking: letting the scenario generator and the routing policy train against each other in a minimax loop, with the energy model as a shared constraint. Early experiments suggest this produces policies that are robust to distribution shift in a way that static benchmarks never achieve.

The second direction is on-device generative benchmarking—running a distilled version of the scenario VAE directly on the agent, so it can self-test during idle periods and flag when its own policy is degrading. This feels like the natural endpoint of the agentic AI philosophy: an agent that benchmarks itself.

Conclusion: Benchmarking Is a Design Act

The most important thing I learned from this project wasn't a technique—it was a shift in mindset. I used to treat benchmarking as the boring epilogue to model development. Now I treat it as the primary design act. The generative scenario model, the quantization-aware policy, and the energy model aren't separate concerns; they're three views of the same question: will this thing actually work, out there, on that chip, with that battery?

If you take one thing from this article, let it be this: your benchmark is a model, and like any model, it can be wrong. Generative simulation benchmarking is a way of making that model honest—by teaching it to be adversarial, by forcing it to respect the compute budget of the target device, and by measuring the one quantity that ultimately decides whether your autonomous system ships: joules.

Go build something that flies. And benchmark it like the wind is trying to kill it—because out there, it is.