Generative Simulation Benchmarking for circular manufacturing supply chains during mission-critical recovery windows

Generative Simulation Benchmarking for circular manufacturing supply chains during mission-critical recovery windows

Introduction: When a Broken Supply Chain Taught Me to Simulate the Impossible

Six months ago, I was deep into a personal research sprint on agentic AI systems when a friend who runs a mid-sized electronics remanufacturing operation called me in a panic. A typhoon had knocked out two of her key reverse-logistics hubs, and she had roughly 72 hours to reroute thousands of returned units, reallocate recovered components, and honor contractual recovery windows with three OEM partners. She asked a deceptively simple question: "Can your AI tell me what to do?"

It couldn't. Not in any useful way. My existing reinforcement learning agents were trained on steady-state conditions — smooth demand curves, predictable return rates, ample lead time. A mission-critical recovery window is the opposite of steady state. It's a chaotic, high-stakes, time-boxed regime where every decision compounds, data is sparse, and the cost of a wrong move is measured in broken contracts and scrapped material.

That phone call sent me down a rabbit hole that consumed the next several months of my learning journey: how do you benchmark decision-making systems for circular manufacturing supply chains when the supply chain itself is in crisis? The answer, I discovered through a lot of failed experiments, lies at the intersection of generative simulation, agentic AI, and a discipline I hadn't fully appreciated before — benchmarking under distributional shift.

This article is the write-up of what I learned. It's part tutorial, part research diary, and part cautionary tale about how easy it is to fool yourself with clean benchmarks.

Why Circular Supply Chains Break Differently Under Stress

Linear supply chains fail in fairly legible ways — a supplier misses a shipment, a port closes, a factory goes down. Circular supply chains, which depend on reverse logistics, refurbishment, component harvesting, and remanufacturing, fail in ways that are structurally harder to model.

While studying circular economy literature, I realized the core issue: a circular chain has coupled forward and reverse flows. The material you recover today determines what you can remanufacture tomorrow, which determines what you don't need to source, which changes your cost structure and your carbon footprint. When a recovery window opens — say, after a disruption — these couplings become the dominant dynamics rather than background noise.

The key characteristics of a mission-critical recovery window that make it brutal to benchmark:

1. Time-boxing: Decisions must be made within hours or days, not weeks.
2. Sparse and shifting data: Historical distributions no longer apply.
3. Cascading constraints: A single hub failure propagates through recovery, refurbishment, and resale.
4. Multi-objective pressure: Cost, service level, sustainability targets, and contractual penalties all pull in different directions simultaneously.
5. Irreversibility: Scrapping a recoverable unit is a permanent loss.

My exploration of this problem revealed that standard simulation benchmarks (fixed scenario libraries, static demand models) are essentially useless here. You need a generative approach — one that can synthesize plausible crisis scenarios on demand, adversarially, and at a scale that lets you actually trust your agent's performance.

The Core Idea: Generative Simulation as an Adversarial Benchmark

The conceptual leap I made — and it took a while — was to stop treating the simulator as a fixed environment and start treating it as a generative adversary. Instead of testing an agent against a handful of hand-crafted disruption scenarios, I'd train a generative model to produce recovery-window scenarios that are maximally informative about the agent's weaknesses.

This is essentially a minimax formulation:

$$
\min_\theta \max_\phi \mathbb{E}{s \sim G\phi} \left[ \mathcal{L}(A_\theta, s) \right]
$$

where $A_\theta$ is the decision agent, $G_\phi$ is the generative scenario model, and $\mathcal{L}$ is a loss capturing both operational performance and constraint violations. The generator's job is to find scenarios where the agent fails; the agent's job is to survive them.

Through studying this formulation, I learned that it maps remarkably well onto the structure of circular supply chains. The generator can perturb:

• Hub availability and capacity
• Return rates and quality distributions
• Lead times for recovered components
• Contractual penalty schedules
• Carbon accounting rules

And the agent must produce a recovery plan that holds up.

A Minimal Generative Scenario Model

Here's a compact example of how I structured the generative scenario sampler. I used a variational autoencoder over a structured scenario space, conditioned on a disruption "severity" latent:

import torch
import torch.nn as nn

class ScenarioVAE(nn.Module):
    """
    Generates plausible recovery-window scenarios for a circular
    manufacturing network. Input: flattened network state.
    Output: perturbed scenario parameters.
    """
    def __init__(self, n_nodes, latent_dim=16):
        super().__init__()
        self.n_nodes = n_nodes
        self.encoder = nn.Sequential(
            nn.Linear(n_nodes * 4, 128), nn.ReLU(),
            nn.Linear(128, 64), nn.ReLU(),
        )
        self.mu = nn.Linear(64, latent_dim)
        self.logvar = nn.Linear(64, latent_dim)
        self.decoder = nn.Sequential(
            nn.Linear(latent_dim + 1, 64), nn.ReLU(),  # +1 for severity
            nn.Linear(64, 128), nn.ReLU(),
            nn.Linear(128, n_nodes * 4),
            nn.Sigmoid(),  # normalized scenario params
        )

    def forward(self, x, severity):
        h = self.encoder(x)
        mu, logvar = self.mu(h), self.logvar(h)
        std = torch.exp(0.5 * logvar)
        z = mu + std * torch.randn_like(std)
        z_cond = torch.cat([z, severity.unsqueeze(-1)], dim=-1)
        return self.decoder(z_cond), mu, logvar

The severity conditioning is what makes this useful for recovery windows: severity=0 is business-as-usual, severity=1 is a full-blown crisis. By sweeping severity, I could generate a continuum of scenarios rather than a discrete set.

One interesting finding from my experimentation with this VAE was that the latent space organized itself along interpretable axes — one dimension corresponded roughly to "hub outage magnitude," another to "return quality degradation," and a third to "lead time inflation." That emergent structure turned out to be enormously useful for targeted stress testing.

Benchmarking Agents Under Distributional Shift

The next problem I hit — and it took me embarrassingly long to see it — was that benchmarking an agent on generated scenarios is only meaningful if the scenarios are (a) plausible and (b) not trivially overfit to the agent's failure modes.

In my research of adversarial benchmarking methods, I came across the concept of coverage-aware evaluation: you want to measure not just average performance but the fraction of the scenario space your agent can handle. This led me to a two-tier benchmark:

1. Plausibility gate: A discriminator network filters generated scenarios to ensure they resemble real recovery windows.
2. Coverage metric: We compute the volume of the scenario space (in latent terms) where the agent meets a performance threshold.

Here's the discriminator and coverage computation in sketch form:

class ScenarioDiscriminator(nn.Module):
    """Distinguishes real recovery-window logs from generated scenarios."""
    def __init__(self, n_params):
        super().__init__()
        self.net = nn.Sequential(
            nn.Linear(n_params, 64), nn.LeakyReLU(0.2),
            nn.Linear(64, 32), nn.LeakyReLU(0.2),
            nn.Linear(32, 1),
        )

    def forward(self, s):
        return torch.sigmoid(self.net(s))


def coverage_score(agent, generator, discriminator, n_samples=2048,
                   threshold=0.9, plausibility_min=0.5):
    """
    Estimate the fraction of plausible scenario space where the agent
    meets the performance threshold.
    """
    with torch.no_grad():
        z = torch.randn(n_samples, generator.latent_dim)
        severity = torch.rand(n_samples)
        scenarios, _, _ = generator.decode(z, severity)
        plausibility = discriminator(scenarios).squeeze(-1)
        plausible = scenarios[plausibility > plausibility_min]

        if len(plausible) == 0:
            return 0.0

        rewards = torch.stack([agent.evaluate(s) for s in plausible])
        return (rewards > threshold).float().mean().item()

While learning about this approach, I observed something that surprised me: agents that scored beautifully on a fixed benchmark suite often had coverage scores below 0.3. They were excellent at the scenarios I'd thought to test and brittle everywhere else. The generative benchmark exposed this immediately.

The Agent Side: Recovery Planning as Sequential Decision-Making

Of course, a benchmark is only as good as the agent you're benchmarking. For the recovery-planning agent itself, I experimented with several architectures and eventually settled on a hierarchical agentic design:

• A high-level planner (LLM-based) that reasons about the recovery window in natural language, proposes candidate strategies, and sets objectives.
• A low-level optimizer (a constrained RL policy or a mixed-integer solver) that executes the strategy against the live network state.
• A reflection module that post-hoc analyzes failures and feeds insights back into the planner's context.

The hierarchical split mattered because recovery windows are simultaneously strategic (which partners to prioritize, whether to invoke force majeure clauses) and operational (which truck goes where, which unit gets refurbished vs. harvested). Pure RL struggled with the strategic layer; pure LLM reasoning struggled with the combinatorial operational layer.

Here's a simplified sketch of the planner's decision loop:

class RecoveryPlanner:
    def __init__(self, llm, optimizer, reflection_memory):
        self.llm = llm
        self.optimizer = optimizer
        self.memory = reflection_memory

    def plan(self, network_state, window_hours):
        context = self._build_context(network_state, window_hours)
        strategy = self.llm.propose_strategy(context, self.memory.retrieve(context))

        # Translate strategy into concrete constraints for the optimizer
        constraints = self._strategy_to_constraints(strategy, network_state)

        # Solve the operational problem
        plan = self.optimizer.solve(network_state, constraints)
        return plan, strategy

    def reflect(self, plan, outcome):
        critique = self.llm.critique(plan, outcome)
        self.memory.store(plan.context, critique)

One thing I learned the hard way: the reflection memory is not optional. Without it, the planner kept proposing the same class of strategy and getting burned the same way. With it, performance on repeated disruption types improved measurably across episodes.

Quantum Computing: Where I Got Excited, Then Realistic

Given my interest in quantum computing, I spent a chunk of this project exploring whether quantum optimization could help with the routing and allocation subproblems inside recovery windows. These are combinatorial, and the problem sizes (hundreds of nodes, thousands of units) are exactly the regime where classical solvers start to strain.

I built a small QUBO formulation of a simplified allocation problem and ran it on a simulator using QAOA. The honest result: for the problem sizes I could realistically encode on current hardware, classical heuristics (simulated annealing, OR-Tools) still won comfortably. The quantum advantage window — if it exists for this problem class — is still ahead of us.

But the learning was valuable. Formulating the recovery allocation as a QUBO forced me to be extremely precise about the objective and constraints, and that precision improved my classical solver too. Here's the core of the QUBO construction:

import numpy as np

def build_allocation_qubo(cost_matrix, unit_supply, demand, penalty=10.0):
    """
    QUBO for allocating recovered units to demand nodes.
    x[i,j] = 1 if units from source i go to demand j.
    """
    n_src, n_dst = cost_matrix.shape
    n = n_src * n_dst
    Q = np.zeros((n, n))

    def idx(i, j): return i * n_dst + j

    # Linear cost terms
    for i in range(n_src):
        for j in range(n_dst):
            Q[idx(i, j), idx(i, j)] += cost_matrix[i, j]

    # Supply constraints: sum_j x[i,j] <= unit_supply[i]
    for i in range(n_src):
        for j in range(n_dst):
            for k in range(j, n_dst):
                Q[idx(i, j), idx(i, k)] += penalty

    # Demand constraints: sum_i x[i,j] == demand[j]
    for j in range(n_dst):
        for i in range(n_src):
            for k in range(i, n_src):
                Q[idx(i, j), idx(k, j)] += penalty
        for i in range(n_src):
            Q[idx(i, j), idx(i, j)] -= 2 * penalty * demand[j]

    return Q

My exploration of quantum approaches revealed something I now consider a general principle: even when the quantum method doesn't win, the discipline of quantum formulation often clarifies the classical problem. I'll take that trade.

Real-World Applications and What Actually Shipped

The full system — generative benchmark plus hierarchical agent — eventually got deployed in a limited pilot with my friend's operation. It wasn't magic. But the generative benchmark surfaced three failure modes her team hadn't anticipated:

1. Hub-outage cascades where rerouting to a secondary hub created a secondary bottleneck within 18 hours.
2. Quality-return shocks where a spike in low-quality returns silently invalidated the refurbishment plan.
3. Contract-penalty cliffs where a plan that looked fine on average was catastrophically bad in the tail.

Each of these became a targeted training scenario for the agent, and each was found by the generator, not by a human. That's the promise of generative simulation benchmarking: it finds the scenarios you wouldn't have thought to write.

Challenges and Solutions

Along the way I hit several walls worth documenting:

Challenge 1: Generator collapse. Early on, my VAE collapsed to producing near-identical scenarios. Solution: added a diversity regularizer based on pairwise latent distances and a KL annealing schedule.

Challenge 2: Unrealistic severity. The generator loved producing severity=1.0 scenarios that were technically valid but operationally absurd. Solution: the discriminator plausibility gate, plus a curriculum that ramped severity during training.

Challenge 3: Reward hacking. The agent learned to game the coverage metric by exploiting the discriminator's blind spots. Solution: held out a set of real recovery-window logs as a final, non-generative test set. If performance diverged between generated and real, I knew something was wrong.

Challenge 4: Compute cost. Running full co-training of generator and agent was expensive. Solution: alternating optimization with periodic freezing, plus a smaller "probe" generator for rapid iteration.

Future Directions

Through my investigation of this space, I see a few directions that feel genuinely promising:

• Foundation models for supply chain simulation: Pre-trained generative models that can be fine-tuned per network, dramatically reducing the data needed for a new deployment.
• Multi-agent recovery: Modeling each partner as an agent with its own objectives, turning recovery planning into a game-theoretic problem.
• Causal scenario generation: Moving beyond correlation-preserving generators to ones that respect the causal structure of disruptions, so counterfactuals are meaningful.
• Hybrid quantum-classical solvers: As quantum hardware matures, revisiting the allocation subproblems with a realistic eye on where advantage might actually appear.
• Human-in-the-loop benchmarking: Using the generative benchmark not just to score agents but to teach human planners, surfacing the scenarios they're least prepared for.

Conclusion: What I Actually Learned

If I had to compress this whole journey into a few takeaways:

1. Benchmarks are not neutral. A fixed benchmark suite encodes your assumptions about what matters. Generative benchmarks force you to confront the scenarios you didn't imagine — which are exactly the ones that break you in a real recovery window.

2. Circular supply chains deserve their own simulation frameworks. The coupled forward-reverse dynamics don't reduce cleanly to linear-chain tools. Building the right generative model is half the work.

3. Agentic hierarchies earn their keep under stress. Strategic reasoning and operational optimization are different problems; forcing one model to do both was a mistake I made repeatedly before splitting them.

4. Quantum is a lens, not (yet) a lever. For the problem sizes I could touch, classical methods still win — but thinking quantumly made my classical formulations sharper.

5. The most valuable output of a benchmark is surprise. The best moments in this project were the ones where the generator found a scenario that made me say "wait, that can happen?" Those are the scenarios worth training for.

My friend's operation survived its recovery window. Not because the AI was brilliant, but because the generative benchmark had already forced the system to rehearse the kind of crisis it was about to face. That, more than any single algorithm, is what I've come to believe matters: not building an agent that performs well on the tests you wrote, but building a benchmark that writes tests you never would have.