Human-Aligned Decision Transformers for satellite anomaly response operations with inverse simulation verification

Human-Aligned Decision Transformers for satellite anomaly response operations with inverse simulation verification

A Discovery Born from a Late-Night Simulation

It was 2:47 AM, and I was staring at a terminal window filled with telemetry data from a simulated satellite constellation. For weeks, I had been experimenting with Decision Transformers—a class of models that frame reinforcement learning as a sequence modeling problem—and I was stuck. The models could predict optimal actions for nominal operations, but when I injected anomalies—sudden thruster failures, power surges, or communication dropouts—the responses were brittle, often proposing actions that no human operator would ever approve.

That night, while re-reading the original Decision Transformer paper (Chen et al., 2021), a thought struck me: What if we could align these models with human operator preferences, not just through reward signals, but through an inverse simulation verification loop? The idea was simple yet profound—instead of training the model solely on historical data, we could simulate candidate responses, verify them against a set of human-defined constraints, and use that feedback to refine the model's latent representations.

This article documents my journey exploring Human-Aligned Decision Transformers (HADT) for satellite anomaly response, with a novel inverse simulation verification mechanism that ensures operational safety and human trust.

Technical Background: The Convergence of Sequence Modeling and Human Preference

Decision Transformers: A Primer

Traditional reinforcement learning (RL) for satellite anomaly response typically uses value-based or policy-gradient methods. However, these approaches struggle with long-horizon dependencies and require careful reward engineering. Decision Transformers (DT) reframe the problem: instead of learning a policy, they model the entire trajectory as a sequence of (state, action, return-to-go) tokens.

In my experiments, I found that DT's autoregressive nature naturally captures the temporal dependencies in satellite telemetry—thruster firings, power consumption spikes, and orbital perturbations all unfold as sequential patterns. The model predicts the next action by attending to the entire history of states and desired returns.

The Alignment Problem in Space Operations

While exploring human-AI alignment for space systems, I discovered a critical gap: satellite operators have implicit preferences that are rarely captured in reward functions. For example:

• Safety margins: Operators prefer actions that leave headroom for unexpected contingencies.
• Interpretability: A black-box action might be mathematically optimal but operationally unacceptable.
• Recovery trajectory: The path back to nominal operations matters as much as the immediate fix.

Standard RL alignment methods (like RLHF) require extensive human annotation, which is impractical for real-time anomaly response. My insight was to use inverse simulation—running candidate actions through a high-fidelity physics simulator and comparing the outcomes against human-defined verification rules.

Implementation Details: Building the HADT Framework

Core Architecture

The HADT consists of three components:

1. Decision Transformer backbone (GPT-like with causal masking)
2. Inverse simulator (differentiable physics model of the satellite)
3. Verification module (rule-based and learned preference models)

Let me walk you through the key implementation. First, the Decision Transformer encoder:

import torch
import torch.nn as nn
import numpy as np
from transformers import GPT2Model

class SatelliteDecisionTransformer(nn.Module):
    def __init__(self, state_dim=64, act_dim=6, max_ep_len=512, hidden_dim=512):
        super().__init__()
        self.state_dim = state_dim
        self.act_dim = act_dim
        self.max_ep_len = max_ep_len

        # Embedding layers for states, actions, and returns-to-go
        self.state_embed = nn.Linear(state_dim, hidden_dim)
        self.action_embed = nn.Linear(act_dim, hidden_dim)
        self.return_embed = nn.Linear(1, hidden_dim)

        # GPT-2 backbone for sequence modeling
        self.transformer = GPT2Model.from_pretrained('gpt2',
                                                      n_ctx=max_ep_len*3,
                                                      n_embd=hidden_dim,
                                                      n_layer=8,
                                                      n_head=8)

        # Action prediction head
        self.action_head = nn.Sequential(
            nn.Linear(hidden_dim, hidden_dim),
            nn.ReLU(),
            nn.Linear(hidden_dim, act_dim),
            nn.Tanh()  # Bounded actions for satellite control
        )

    def forward(self, states, actions, returns_to_go, timesteps):
        # Embed each modality
        state_emb = self.state_embed(states)
        action_emb = self.action_embed(actions)
        return_emb = self.return_embed(returns_to_go.unsqueeze(-1))

        # Interleave tokens: [R, S, A, R, S, A, ...]
        sequence = []
        for t in range(states.shape[1]):
            sequence.append(return_emb[:, t:t+1])
            sequence.append(state_emb[:, t:t+1])
            sequence.append(action_emb[:, t:t+1])

        x = torch.cat(sequence, dim=1)

        # Add positional embeddings
        pos_emb = self.transformer.wpe(timesteps)
        x = x + pos_emb

        # Transformer forward
        h = self.transformer(inputs_embeds=x).last_hidden_state

        # Extract action predictions (at positions 2, 5, 8, ...)
        action_preds = []
        for t in range(states.shape[1]):
            action_hidden = h[:, 3*t + 2]  # Position after state token
            action_preds.append(self.action_head(action_hidden))

        return torch.stack(action_preds, dim=1)

Inverse Simulation Verification

The key innovation is the inverse simulation loop. For each candidate action sequence predicted by the DT, we run it through a differentiable satellite simulator and compare the resulting trajectory against human-defined constraints:

class InverseSimulationVerifier:
    def __init__(self, satellite_model, constraints):
        self.sim = satellite_model  # Differentiable physics model
        self.constraints = constraints  # Dict of (name, lambda) pairs

    def verify_actions(self, states, candidate_actions, returns_to_go):
        """
        Run inverse simulation: given candidate actions,
        simulate forward and check constraints
        """
        # Simulate forward using differentiable physics
        simulated_states = self.sim.rollout(states[:, -1], candidate_actions)

        # Compute constraint violations
        violations = {}
        for name, constraint_fn in self.constraints.items():
            violation = constraint_fn(simulated_states, candidate_actions)
            violations[name] = violation

        # Compute alignment score (lower is better)
        alignment_score = sum(v.mean() for v in violations.values())

        # Compute trajectory preference score
        # (learned from human operator demonstrations)
        preference_score = self._preference_model(simulated_states,
                                                   candidate_actions)

        return {
            'alignment_score': alignment_score,
            'preference_score': preference_score,
            'violations': violations,
            'simulated_states': simulated_states
        }

    def _preference_model(self, states, actions):
        """
        Learned reward model from human operator preferences
        Trained via contrastive learning on operator demonstrations
        """
        # Simplified: compute cosine similarity with preferred trajectories
        preferred_encoding = self._encode_preferred_trajectory()
        current_encoding = self._encode_trajectory(states, actions)
        return torch.cosine_similarity(current_encoding, preferred_encoding)

Training with Human Feedback via Inverse Simulation

During training, I used a two-stage process. First, pre-train the DT on historical satellite telemetry. Then, fine-tune using the inverse simulation verifier:

def train_hadt_with_inverse_simulation(dt_model, verifier, dataset,
                                        num_epochs=100, lr=1e-4):
    optimizer = torch.optim.AdamW(dt_model.parameters(), lr=lr)

    for epoch in range(num_epochs):
        for batch in dataset:
            states, actions, returns_to_go, timesteps = batch

            # Forward pass through DT
            predicted_actions = dt_model(states, actions, returns_to_go, timesteps)

            # Inverse simulation verification
            verification = verifier.verify_actions(
                states, predicted_actions, returns_to_go
            )

            # Compute losses
            # 1. Behavioral cloning loss (match original actions)
            bc_loss = nn.MSELoss()(predicted_actions, actions)

            # 2. Alignment loss (minimize constraint violations)
            alignment_loss = verification['alignment_score']

            # 3. Preference loss (maximize operator preference)
            preference_loss = -verification['preference_score']

            # Combined loss with adaptive weighting
            loss = bc_loss + 0.3 * alignment_loss + 0.1 * preference_loss

            # Backprop through differentiable simulator
            loss.backward()
            optimizer.step()

Real-World Applications: From Simulation to Operations

Case Study: Thruster Anomaly Response

During my experimentation, I tested the HADT on a simulated GEO satellite with a stuck thruster. The standard DT proposed aggressive counter-thrusting to maintain orbit, which would deplete fuel reserves. The HADT, guided by inverse simulation verification, proposed a more conservative strategy:

1. Phase 1 (0-5 minutes): Reduce attitude control bandwidth to conserve reaction wheels
2. Phase 2 (5-30 minutes): Use magnetic torquers for coarse attitude hold
3. Phase 3 (30-60 minutes): Execute a fuel-optimal drift correction using remaining thrusters

The key insight was that the inverse simulation verifier had learned from human operators that "aggressive fuel usage" was a negative preference, even if it temporarily solved the anomaly.

Multi-Satellite Coordination

I extended the framework to handle constellations. The HADT was trained on sequences of inter-satellite link states and anomaly reports. When a single satellite experienced a power anomaly, the HADT coordinated actions across the constellation:

class ConstellationHADT:
    def __init__(self, num_satellites=12, state_dim=128, act_dim=8):
        self.num_satellites = num_satellites
        self.dt = SatelliteDecisionTransformer(
            state_dim=state_dim * num_satellites,  # Concatenated states
            act_dim=act_dim * num_satellites,       # Concatenated actions
            max_ep_len=256
        )
        self.verifier = InverseSimulationVerifier(
            satellite_model=MultiSatellitePhysics(num_satellites),
            constraints={
                'link_budget': lambda s,a: self._check_link_budget(s),
                'collision_avoidance': lambda s,a: self._check_collisions(s),
                'power_balance': lambda s,a: self._check_power(s),
                'human_preference': lambda s,a: self._operator_preference(s,a)
            }
        )

    def _operator_preference(self, states, actions):
        """
        Learned from inverse reinforcement learning on operator logs
        """
        # Simplified: prefer actions that maintain communication coverage
        coverage = self._compute_coverage(states)
        return -torch.sigmoid(1.0 - coverage)  # Higher coverage = lower violation

Challenges and Solutions

Challenge 1: Differentiable Physics Simulation

Problem: The inverse simulation verifier requires a differentiable satellite model for gradient backpropagation. Traditional physics engines (like GMAT or STK) are not differentiable.

Solution: I implemented a hybrid approach:

• Use a simplified differentiable model for training (learned neural ODE)
• Verify final actions with high-fidelity non-differentiable simulators at inference time

class DifferentiableSatelliteModel(nn.Module):
    """
    Neural ODE approximation of satellite dynamics
    """
    def __init__(self, state_dim=64):
        super().__init__()
        self.dynamics_net = nn.Sequential(
            nn.Linear(state_dim + 6, 256),  # State + action
            nn.ReLU(),
            nn.Linear(256, 256),
            nn.ReLU(),
            nn.Linear(256, state_dim)
        )

    def forward(self, state, action, dt=0.1):
        # Euler integration with learned dynamics
        delta_state = self.dynamics_net(torch.cat([state, action], dim=-1))
        return state + dt * delta_state

    def rollout(self, initial_state, actions):
        states = [initial_state]
        for action in actions:
            next_state = self.forward(states[-1], action)
            states.append(next_state)
        return torch.stack(states, dim=1)

Challenge 2: Sparse Human Feedback

Problem: Human operators cannot provide real-time feedback during anomaly response.

Solution: I used inverse simulation to generate synthetic feedback. The verifier checks candidate actions against human-defined safety envelopes, effectively creating a dense reward signal:

def generate_synthetic_preference(states, actions, safety_envelope):
    """
    Generate preference labels by comparing against human-defined
    safety envelopes (learned from historical operator actions)
    """
    # Check if actions stay within safe operating region
    safe_actions = torch.all(
        (actions >= safety_envelope['lower']) &
        (actions <= safety_envelope['upper']),
        dim=-1
    )

    # Check if resulting states are nominal
    nominal_states = torch.all(
        torch.abs(states) < safety_envelope['state_threshold'],
        dim=-1
    )

    # Preference is high when both actions and states are safe
    preference = safe_actions.float() * nominal_states.float()
    return preference.mean(dim=-1)  # Average over trajectory

Challenge 3: Real-Time Inference Latency

Problem: The inverse simulation loop adds computational overhead that may exceed real-time constraints.

Solution: I implemented a two-tier architecture:

• Fast path: Direct DT inference (sub-millisecond) for nominal operations
• Verification path: Only trigger inverse simulation when anomaly confidence exceeds threshold

class AdaptiveHADT:
    def __init__(self, dt_model, verifier, anomaly_detector):
        self.dt = dt_model
        self.verifier = verifier
        self.anomaly_detector = anomaly_detector

    def act(self, state, return_to_go):
        # Fast path: direct DT prediction
        fast_action = self.dt.infer(state, return_to_go)

        # Check if anomaly is detected
        anomaly_confidence = self.anomaly_detector(state)

        if anomaly_confidence > 0.7:
            # Verification path: run inverse simulation
            candidate_actions = self._generate_candidates(state, return_to_go)
            verification = self.verifier.verify_actions(
                state.unsqueeze(0),
                candidate_actions,
                return_to_go.unsqueeze(0)
            )
            # Select action with best alignment
            best_idx = verification['alignment_score'].argmin()
            return candidate_actions[best_idx]

        return fast_action

Future Directions

Quantum-Enhanced Inverse Simulation

While exploring quantum computing applications, I realized that the inverse simulation verification could be accelerated using quantum algorithms. The constraint satisfaction problem is essentially a combinatorial optimization—finding actions that minimize violations. Quantum annealing (via D-Wave) or variational quantum eigensolvers (VQE) could potentially explore the action space more efficiently:

# Conceptual quantum-enhanced verification
def quantum_verify_actions(hamiltonian, candidate_actions):
    """
    Use quantum computing to find optimal actions
    that minimize constraint violations
    """
    # Encode constraint violations as Ising Hamiltonian
    H = build_ising_hamiltonian(constraints)

    # Run quantum optimization (e.g., QAOA)
    optimal_actions = qaoa_optimize(H, candidate_actions)
    return optimal_actions

Federated Learning Across Satellite Constellations

Another direction is federated learning where each satellite learns local anomaly patterns and shares only model updates (not raw telemetry) to improve the global HADT. This is particularly relevant for military or commercial constellations where data privacy is paramount.

Conclusion: Lessons from the Trenches

Through this journey of building Human-Aligned Decision Transformers for satellite anomaly response, I learned several critical lessons:

1. Alignment is not just about reward: The inverse simulation verification loop taught me that human preferences are often implicit and multi-dimensional. A single reward signal is insufficient.

2. Differentiable simulators are game-changers: The ability to backpropagate through physics simulations opens up new possibilities for learning with constraints.

3. Trust through verification: Operators will never trust a black-box AI. The inverse simulation loop provides an auditable trail of why an action was chosen.

4. Simplicity wins: The most effective parts of the HADT were the simplest—the constraint functions defined by operators, not the complex neural networks.

As I finally shut down my terminal that morning, watching the simulated satellite gracefully recover from a power anomaly using the HADT's suggested actions, I felt a quiet satisfaction. The model had learned to prioritize fuel efficiency and safety margins—exactly what human operators would do. The inverse simulation verifier had effectively transferred human intuition into machine policy.

The code and experiments are available on my GitHub repository (link in bio). I encourage you to fork it, break it, and build something better. The future of autonomous space operations depends on systems that don't just optimize—they align.

---

This article is based on my personal research and experimentation with Decision Transformers and inverse simulation. All code examples are simplified for clarity but capture the essential implementation patterns.