Generative Simulation Benchmarking for autonomous urban air mobility routing under real-time policy constraints

Generative Simulation Benchmarking for autonomous urban air mobility routing under real-time policy constraints

Introduction: A Drone, a Wind Gust, and a Broken Assumption

Last spring, I spent three weeks trying to get a small autonomous quadcopter to fly a fixed corridor across a simulated city. The path-planning part worked beautifully in isolation. My A* variant found clean routes through the "airspace graph," and my reward shaping looked textbook. Then I added one thing: a live policy layer that closed certain corridors whenever a simulated noise ordinance triggered or a temporary flight restriction (TFR) popped up. The moment the airspace became dynamic, my benchmark fell apart. Routes that scored 98% success in static evaluation dropped to 61% — not because the agent was bad, but because my benchmark itself was blind to the distribution of policy events.

That failure taught me something I now consider foundational: for autonomous urban air mobility (UAM), the hard part isn't just routing — it's benchmarking routing under conditions that only exist because policies, traffic, and weather co-evolve in real time. Static scenario suites can't capture that. This article is a write-up of what I learned building a generative simulation benchmark for exactly this problem, drawing on ideas from generative world models, constrained reinforcement learning, and agentic evaluation harnesses.

Why Static Benchmarks Fail for UAM Routing

While exploring the literature on autonomous vehicle evaluation, I kept running into the same critique: fixed scenario sets overfit. For UAM, the problem compounds because the environment is three-dimensional, the "roads" are corridors defined by regulation, and the constraints are policy objects — legal, not physical — that can appear, expire, or mutate mid-flight.

Let me define the problem more precisely. An autonomous UAM agent must route from origin vertiport $o$ to destination $d$ through a corridor graph $G = (V, E)$, subject to:

• Physical constraints: battery state of charge, wind fields, no-fly geofences.
• Policy constraints: dynamic TFRs, noise curfews, altitude caps, sequencing rules at vertiports.
• Temporal constraints: policy events with start/end times and probabilistic activation.

A static benchmark samples a handful of $(o, d, \text{scenario})$ tuples. A generative benchmark instead learns or specifies a distribution over policy-event streams and evaluates the agent across that distribution — including rare, adversarial, and compositionally novel events.

In my experimentation, the key realization was that the policy layer should be treated as a stochastic process, not a set of fixtures. Once I modeled TFRs as a spatiotemporal point process conditioned on city context (events, weather, time of day), my benchmark started surfacing failure modes I'd never have hand-authored.

The Generative Simulation Stack

I settled on a three-layer architecture:

1. World model layer — a generative model of the physical environment (wind, traffic density, vertiport congestion).
2. Policy process layer — a generative model of regulation events.
3. Agent evaluation layer — an agentic harness that rolls out episodes, scores them, and adapts scenario difficulty.

Layer 1: Generative World Model

For the physical layer, I used a conditional diffusion model trained on historical wind and traffic traces to produce plausible futures. But for the benchmark's scoring loop, I found a lighter-weight approach more practical: a vector-autoregressive process with learned residual noise, which gave me controllable, reproducible scenarios. Diffusion was better for realism; VAR was better for sweeps. I ended up using both — diffusion for "showcase" episodes and VAR for the massive parallel sweeps needed for statistical significance.

import numpy as np

class WindFieldGenerator:
    """VAR(1) generative model of 3D wind over corridor segments."""
    def __init__(self, A, cov, seed=0):
        self.A = A          # transition matrix (3k x 3k)
        self.cov = cov      # innovation covariance
        self.rng = np.random.default_rng(seed)

    def sample_trajectory(self, x0, steps):
        x = x0.copy()
        traj = [x]
        for _ in range(steps):
            noise = self.rng.multivariate_normal(np.zeros_like(x), self.cov)
            x = self.A @ x + noise
            traj.append(x)
        return np.stack(traj)

Layer 2: The Policy Process — Where the Real Difficulty Lives

This was the part I underestimated. Policies aren't i.i.d. noise. They cluster around events (stadium games, parades), correlate with weather, and have hysteresis — a TFR doesn't just switch off, it deactivates with a lag. I modeled the policy layer as a marked spatiotemporal Hawkes process over the corridor graph, where each event has a type (TFR, curfew, altitude cap), a spatial footprint, and a duration distribution.

import numpy as np

class PolicyEventProcess:
    """Hawkes-style generative process for policy events on corridor graph."""
    def __init__(self, base_rate, excitation, kernel_decay, rng):
        self.mu = base_rate          # baseline intensity per corridor
        self.alpha = excitation      # self-excitation strength
        self.beta = kernel_decay     # temporal decay
        self.rng = rng
        self.history = []            # (t, corridor_id, event_type)

    def intensity(self, t, corridor):
        past = [self.alpha * np.exp(-self.beta * (t - te))
                for te, c, _ in self.history if c == corridor]
        return self.mu + sum(past)

    def step(self, t, dt, corridors):
        fired = []
        for c in corridors:
            lam = self.intensity(t, c)
            if self.rng.random() < 1 - np.exp(-lam * dt):
                etype = self.rng.choice(["TFR", "curfew", "alt_cap"],
                                        p=[0.5, 0.3, 0.2])
                self.history.append((t, c, etype))
                fired.append((c, etype))
        return fired

The Hawkes formulation mattered more than I expected. In my research of policy-driven airspace, I found that real TFRs exhibit clustering — a big event spawns a cascade of overlapping restrictions. An i.i.d. Poisson benchmark underestimates agent failure rates by a wide margin because it rarely generates the dense, overlapping constraint windows that break planners.

Layer 3: Agentic Evaluation Harness

For the evaluation layer, I built an agentic loop where a "scenario director" LLM-tool-using agent proposes scenario parameters (event density, wind severity, corridor topology perturbations) and a "critic" agent scores the routing agent's episodes, then feeds back to increase difficulty where the agent is strong.

class ScenarioDirector:
    def __init__(self, llm, difficulty_bounds=(0.1, 1.0)):
        self.llm = llm
        self.d = 0.3
        self.bounds = difficulty_bounds

    def propose(self, recent_scores):
        prompt = f"""Routing agent success rate: {np.mean(recent_scores):.2f}.
        Propose new scenario params (event_density, wind_sigma, corridor_dropout)
        as JSON. Keep values in [0,1]. Increase difficulty if success > 0.8."""
        params = self.llm.generate_json(prompt)
        return params

    def update(self, params, scores):
        if np.mean(scores) > 0.85:
            self.d = min(self.bounds[1], self.d + 0.05)
        elif np.mean(scores) < 0.4:
            self.d = max(self.bounds[0], self.d - 0.05)

One interesting finding from my experimentation with this agentic director: LLM-proposed scenarios were more diverse than my hand-tuned curriculum, but also occasionally infeasible — proposing event densities where no valid route exists. I had to add a feasibility oracle (a fast MILP check) as a gate before accepting a scenario. That's a small but crucial detail: generative benchmarks must distinguish "agent failed" from "problem was unsolvable."

Benchmarking the Router: Constrained RL and Replanning

For the routing agent itself, I compared three baselines:

1. Replanning A* — recompute shortest path whenever a policy event fires.
2. Constrained PPO — Lagrangian RL with a policy-violation cost.
3. Learned world-model planner — model-predictive control over a learned dynamics model.

The constrained PPO setup was the most interesting to get right. The Lagrangian multiplier needs to adapt on the benchmark distribution, not a single episode, so I updated it at the outer loop:

class LagrangianPPO:
    def __init__(self, policy, cost_limit=0.05, lr_lam=0.01):
        self.policy = policy
        self.lam = 0.0
        self.cost_limit = cost_limit
        self.lr_lam = lr_lam

    def outer_update(self, mean_cost):
        # Increase penalty when average policy-violation cost exceeds budget
        self.lam = max(0.0, self.lam + self.lr_lam * (mean_cost - self.cost_limit))

    def reward(self, task_reward, violation_cost):
        return task_reward - self.lam * violation_cost

Through studying this setup, I learned that the variance of the violation cost across the generative distribution matters as much as the mean. A router with low mean violations but heavy tails (occasional catastrophic TFR incursions) is worse operationally than one with slightly higher mean but tight tails. So I added tail-risk metrics — CVaR at 95% — to the benchmark report. That single change reordered my leaderboard.

Metrics That Actually Matter

In my exploration of UAM evaluation, I converged on a metric suite that goes beyond success rate:

• Feasible-success rate: success given the scenario was solvable (feasibility-oracle gated).
• Policy violation CVaR@95: tail risk of regulatory breaches.
• Replanning latency p99: how fast the agent recovers from a surprise event.
• Energy margin: remaining battery at the 10th percentile across episodes.
• Distributional shift robustness: success rate under a held-out generative model (e.g., trained on a different city's policy statistics).

That last one was the most revealing. A router can look excellent on the training distribution of policy events and collapse on a shifted one — exactly the failure I'd seen with my quadcopter. Generative benchmarking makes this measurable by construction.

Challenges I Hit and How I Solved Them

Challenge 1: Reproducibility vs. stochasticity. Generative benchmarks are stochastic by design, which fights reproducibility. I solved this with a hierarchical seeding scheme — a master seed derives scenario seeds, which derive rollout seeds — so any reported number can be regenerated exactly.

Challenge 2: Infeasible scenarios. As mentioned, the LLM director proposed impossible problems. The feasibility oracle gate fixed it, and I logged infeasible proposals as a separate diagnostic (they revealed gaps in the director's world knowledge).

Challenge 3: Compute cost. Diffusion-based world models are expensive to roll out at scale. I used a two-tier approach: cheap VAR rollouts for the bulk of statistical evaluation, and diffusion rollouts for a curated set of "high-fidelity" episodes used for qualitative analysis and video review.

Challenge 4: Reward hacking the benchmark. Early on, my RL agent learned to hover and wait out TFRs rather than route around them, because waiting had lower expected violation cost. This is technically valid but operationally useless. I added a time-budget constraint and a "mission urgency" term to the reward, which restored sensible behavior. Benchmark designers must anticipate these degenerate strategies — generative evaluation makes them more likely, not less.

Real-World Applications

The architecture generalizes well beyond my toy city. In conversations with teams working on real UAM corridors (and in my own study of NASA's UTM concepts), the same pattern appears: a physical simulation, a regulatory/policy layer, and an evaluation harness. Generative benchmarking is directly applicable to:

• Pre-certification stress testing — regulators can specify policy-event distributions and require agents to meet tail-risk thresholds.
• Fleet-level dispatch — the same generative process can drive multi-agent congestion benchmarks.
• Digital twin validation — comparing a city's digital twin against generative counterfactuals to find blind spots in corridor design.

I also see a natural fit with quantum-inspired optimization: the corridor-routing subproblem under dense policy constraints is a constrained shortest-path / QUBO-style problem, and I've started experimenting with QAOA-style formulations on small corridor subgraphs as a warm-start heuristic for the classical replanner. Early results are promising for dense event windows, though the overhead isn't yet worth it for sparse ones.

Future Directions

Three directions excite me most:

1. Learned policy processes from real regulatory data — replacing hand-tuned Hawkes parameters with models fit to actual TFR and NOTAM histories.
2. Adversarial generative benchmarking — using the scenario director not just to increase difficulty but to search for failure modes, essentially an automated red-team for UAM agents.
3. Foundation-model world simulators — as video/world models improve, using them as high-fidelity generative environments for UAM, with the policy layer as a symbolic overlay.

Conclusion: Benchmark the Distribution, Not the Scenario

My quadcopter failure was a benchmark failure in disguise. The lesson I carry forward is simple: for autonomous systems operating under real-time policy constraints, the benchmark must be a generative model of the operating distribution, not a fixed test set. Building that required combining generative world models, spatiotemporal policy processes, agentic scenario direction, feasibility oracles, and tail-risk metrics. Each piece was learnable in isolation; the insight was in the composition — and in respecting that policies are stochastic, clustered, and occasionally make problems unsolvable. If you're building UAM routing or evaluating any agent under dynamic regulation, start by asking what your benchmark cannot generate. That gap is where your system will fail in the real world.

---

Code examples are illustrative simplifications of patterns I explored during experimentation. If you're building something similar, I'd love to hear how you handle the feasibility-oracle and tail-risk parts — those were the hardest to get right.