Human-Aligned Decision Transformers for heritage language revitalization programs with embodied agent feedback loops

Human-Aligned Decision Transformers for heritage language revitalization programs with embodied agent feedback loops

Introduction: A Personal Encounter with Language Loss

My journey into this specific intersection of AI and linguistics began not in a lab, but in a living room in rural Wales. While researching multi-agent systems for cultural simulation, I was invited to observe a community-led Welsh language revitalization session. I watched as a young child struggled to converse with her grandmother, the elder's fluent Cymraeg met with the child's hesitant, English-dominant responses. The emotional weight was palpable—a living thread of culture, thinning with each generation. The program organizers were dedicated, but their tools were analog: flashcards, storybooks, and scheduled conversation hours. They faced a fundamental scaling problem: how to provide personalized, immersive, responsive language practice outside of these precious, human-led sessions.

This experience crystallized a research question for me. While exploring reinforcement learning from human feedback (RLHF) for large language models, I realized the paradigm was powerful but passive. It aligned models to static datasets of preferences. What if the feedback loop was active, embodied, and situated within the very cultural context we aimed to preserve? What if an AI agent could not just generate linguistically correct sentences but also make decisions about how to teach, adapt to a learner's emotional state, and navigate the complex social dynamics of heritage language use? This led me to the architecture at the heart of this article: Human-Aligned Decision Transformers (HADTs) coupled with embodied agent feedback loops, specifically engineered for the nuanced challenge of heritage language revitalization.

Technical Background: From Decision Transformers to Human Alignment

To understand our architecture, we need to build up from its components. In my research of offline reinforcement learning, I discovered the Decision Transformer (DT) as a fascinating paradigm shift. Instead of learning a value function or policy through dynamic programming, a DT models sequences of states, actions, and rewards (or returns-to-go) autoregressively, treating reinforcement learning as a sequence modeling problem.

The core insight is simple yet profound: the trajectory (s1, a1, R1, s2, a2, R2, ...) is fed into a transformer, which learns to predict the action a_t given past states, actions, and the desired target return. During my experimentation with DTs for robotic task planning, I found their stability and offline training nature appealing, but they lacked a mechanism for nuanced, value-driven alignment with complex human goals.

Human Alignment is the next layer. In classic RLHF, a reward model is trained on human preference data, which then guides policy optimization. One interesting finding from my experimentation with Direct Preference Optimization (DPO) was that we could bypass the explicit reward modeling step and align the policy directly to human preferences. However, both approaches typically operate on a language model's outputs, not on the sequential decision-making process of an agent interacting in an environment.

This is where our synthesis begins. A Human-Aligned Decision Transformer integrates preference alignment directly into the trajectory modeling process. It doesn't just predict the next action to maximize a scalar reward; it predicts the next action that is both effective for the task and aligned with human values and preferences, as expressed in a dataset of preferred/unpreferred trajectories.

For heritage language learning, the "environment" is the interactive learning space—a virtual or physical-robot-mediated context. The "reward" might be linguistic proficiency scores, but the "human preference" is far richer: Was the interaction culturally respectful? Did it reduce the learner's anxiety? Did it appropriately reference familial or community contexts? Did it balance correction with encouragement?

Implementation Architecture: The HADT for Language Agents

Let's break down the architecture. Our system consists of three core modules: the Embodied Agent Environment, the HADT Core, and the Preference Feedback Loop.

1. The Embodied Agent Environment

The agent operates in a simulated or real-world environment. Through studying multimodal AI, I learned that "embodiment" here is key—it's not just a chatbot. It could be a virtual avatar in a VR "family kitchen" or a social robot physically present in a community center. The state (s_t) is multimodal:

• Linguistic State: The dialogue history, learner's speech (transcribed), grammatical error profile.
• Affective State: Estimated from camera feed (facial expression, posture) and speech prosody.
• Cultural-Context State: Parameters defining the scenario (e.g., "preparing a traditional recipe with elder," "naming local landmarks").

import torch
from dataclasses import dataclass
from typing import List, Dict, Optional

@dataclass
class LearningState:
    """Multimodal state representation for heritage language learning."""
    dialogue_history: List[Dict]  # [{speaker: "agent/learner", text: str, tokens: Tensor}]
    learner_utterance: str
    grammatical_errors: List[str]  # e.g., ["verb-subject agreement", "lexical choice"]
    affective_features: torch.Tensor  # [valence, arousal, engagement] from vision/speech model
    scenario_context: Dict  # e.g., {"activity": "storytelling", "cultural_theme": "harvest"}
    proficiency_estimate: float  # Current model estimate of learner's level

The agent's actions (a_t) are also composite:

• Linguistic Action: The next utterance to speak.
• Pedagogical Action: A meta-action like "correct_error", "provide_example", "ask_open_question", "switch_topic".
• Embodied Action: For a robot, gestures, gaze direction, or manipulating objects (e.g., pointing to a picture).

2. The HADT Core

This is where the sequence modeling happens. We adapt the Decision Transformer to work with our rich state-action space and incorporate alignment. During my investigation of transformer architectures for control, I found that a key modification was needed: we must condition not only on the target return (e.g., "reach language proficiency score of X") but also on a target preference embedding.

We train on a dataset D of trajectories τ. Each trajectory is tagged with a binary human preference label y (preferred τ+ vs. unpreferred τ-), provided by experts (community elders, linguists) reviewing recorded interactions.

import torch.nn as nn
from transformers import GPT2Model

class HumanAlignedDecisionTransformer(nn.Module):
    def __init__(self, state_dim, act_dim, hidden_size, max_length, preference_dim):
        super().__init__()
        self.state_encoder = nn.Linear(state_dim, hidden_size)
        self.action_encoder = nn.Linear(act_dim, hidden_size)
        self.return_encoder = nn.Linear(1, hidden_size)
        self.preference_encoder = nn.Linear(preference_dim, hidden_size)

        # GPT backbone for sequence modeling
        self.transformer = GPT2Model.from_pretrained('gpt2')
        # Resize embeddings to include our token types
        self.transformer.resize_token_embeddings(hidden_size)

        self.predict_action = nn.Sequential(
            nn.Linear(hidden_size, hidden_size),
            nn.GELU(),
            nn.Linear(hidden_size, act_dim)
        )

    def forward(self, states, actions, returns_to_go, preferences, timesteps):
        # Encode all modalities
        state_emb = self.state_encoder(states)
        act_emb = self.action_encoder(actions)
        ret_emb = self.return_encoder(returns_to_go.unsqueeze(-1))
        pref_emb = self.preference_encoder(preferences)

        # Interleave embeddings: [pref, ret, state, act, ret, state, act,...] for t=0..T
        sequence = []
        for t in range(states.shape[1]):
            sequence.append(pref_emb[:, t:t+1, :])  # Preference is repeated per timestep
            sequence.append(ret_emb[:, t:t+1, :])
            sequence.append(state_emb[:, t:t+1, :])
            sequence.append(act_emb[:, t:t+1, :])
        sequence = torch.cat(sequence, dim=1)

        # Transformer forward pass
        transformer_outputs = self.transformer(inputs_embeds=sequence)
        hidden_states = transformer_outputs.last_hidden_state

        # Predict actions (we extract the hidden states at action positions)
        # This is a simplified extraction logic
        action_hidden = hidden_states[:, 3::4, :]  # Every 4th token is after an action
        action_preds = self.predict_action(action_hidden)
        return action_preds

The training uses a mixed objective. One part is the standard DT loss (predicting the action in the trajectory). The other is a preference alignment loss. We can use a contrastive loss, where the model learns to assign higher likelihood to trajectories marked as preferred by human experts.

def hadt_loss(pred_actions, true_actions, trajectory_logprobs, preference_labels):
    """Combined DT prediction loss and preference alignment loss."""
    # 1. Standard action prediction loss (MSE or cross-entropy)
    action_loss = nn.MSELoss()(pred_actions, true_actions)

    # 2. Preference alignment loss (contrastive)
    # trajectory_logprobs is the log-prob the model assigns to the full trajectory
    # For a batch containing pairs (preferred_traj, unpreferred_traj)
    pref_logprob, unpref_logprob = trajectory_logprobs[0], trajectory_logprobs[1]
    # DPO-style loss: maximize the difference
    alignment_loss = -torch.log(torch.sigmoid(pref_logprob - unpref_logprob)).mean()

    return action_loss + 0.5 * alignment_loss  # Weighted combination

3. The Embodied Agent Feedback Loop

This is the live, interactive component. The trained HADT serves as the "brain" of the embodied agent. As the agent interacts with a learner, it collects a new trajectory τ_new. However, instead of waiting for offline human review, it employs a preference prediction model (a small classifier trained on the same expert data) to estimate in real-time whether its decisions are "preferred."

This estimated preference signal is fed back into the HADT as part of the state/context for the next decision, creating a closed loop. The agent can dynamically adjust its behavior mid-session. My exploration of real-time adaptive systems revealed that this loop must be fast and low-latency, requiring efficient preference models.

class EmbodiedAgentFeedbackLoop:
    def __init__(self, hadt_model, preference_predictor, env):
        self.hadt = hadt_model
        self.pref_predictor = preference_predictor
        self.env = env
        self.current_trajectory = []

    def step(self, target_return, target_preference_embedding):
        # Get current multimodal state from environment
        state = self.env.get_state()

        # Format trajectory for HADT (last K timesteps)
        trajectory_segment = self._format_segment(state, target_return, target_preference_embedding)

        # HADT predicts next action
        with torch.no_grad():
            action = self.hadt.predict_next_action(trajectory_segment)

        # Execute action in environment (speak, gesture, etc.)
        next_state, reward = self.env.execute(action)

        # Store experience
        self.current_trajectory.append((state, action, reward))

        # REAL-TIME FEEDBACK: Predict preference for this recent segment
        recent_segment = self.current_trajectory[-5:]  # last 5 steps
        pref_score = self.pref_predictor.estimate_preference(recent_segment)

        # Adjust target_preference_embedding if score is low
        # e.g., nudge it towards the "encouraging, slow-paced" region of preference space
        adjusted_preference = self._adjust_preference(target_preference_embedding, pref_score)

        return next_state, adjusted_preference, reward

Real-World Application: A Welsh Language Case Study

Let's ground this in the Welsh context I encountered. A prototype system was deployed as a tablet-based VR companion for children, with a cartoon avatar of a "dragon" (a cultural symbol, Y Ddraig Goch).

Scenario: The activity is "Labeling the Kitchen." The target return is to successfully name 10 items in Welsh. The initial target preference embedding is set to "playful, corrective, story-driven."

1. Initial Interaction: The child points to a refrigerator and says "Ice... box?" in English. The agent's HADT, given the state (error=lexical gap, affect=hesitant) and target preference ("playful, corrective"), selects an action: it shows a short animation of the dragon shivering next to the fridge, saying "Ooo, mae'n oer! Mae hwn yn... **oergell!" ("Ooo, it's cold! This is a... refrigerator!").
2. Feedback Loop Activation: The child laughs but doesn't repeat the word. The agent's on-board preference predictor (trained on elder-reviewed videos) notes low "engagement" and "imitation" signals. It adjusts the preference embedding towards "more repetitive, simpler, with physical cue."
3. Adapted Interaction: The dragon now opens the fridge door, points inside, and says slowly, "Oergell. Dw i'n rhoi'r llaeth yn yr oergell. Beth yw hwn?" ("Refrigerator. I put the milk in the refrigerator. What is this?"), waiting expectantly. This decision—to incorporate a verb phrase and a question—was generated by the HADT under the new preference context.

Through studying hundreds of such interactions, I learned that the most significant gains came not from raw vocabulary acquisition, but from reducing "affective filter" (anxiety) and increasing the frequency of voluntary speech attempts. The embodied feedback loop allowed the agent to detect frustration (via camera) and pivot from a "corrective" to a "narrative" mode, telling a short story about the item instead of drilling it.

Challenges and Solutions from the Trenches

Building this system was fraught with technical and ethical hurdles. Here are the key ones I grappled with:

1. The Preference Modeling Problem: Human preferences in pedagogy are high-dimensional and often contradictory. An elder might prefer "strict correction," while a child psychologist prefers "positive reinforcement." My solution was to learn a multi-faceted preference embedding, not a scalar score. We used a variational autoencoder (VAE) on expert annotations to create a continuous "preference space" where dimensions could be loosely interpreted (e.g., strictness↔playfulness, directness↔indirectness). The target for the HADT could then be a vector in this space, allowing for nuanced targeting.

2. Catastrophic Forgetting in the Loop: The online feedback loop risks causing the agent to drift from its carefully offline-trained, aligned policy. One finding from my experimentation with model-based RL was using a replay buffer with constrained optimization. All new interactions are stored. Periodically, the HADT is fine-tuned on a mixture of the original expert-preferred dataset and the new buffer data, with a constraint (via KL-divergence penalty) that the updated policy doesn't deviate too far from the original. This balances adaptation and stability.

# Pseudo-code for stable online adaptation
def stable_adaptation(hadt, original_data, replay_buffer, kl_constraint=0.1):
    combined_data = original_data + replay_buffer.sample()
    for batch in combined_data:
        loss = hadt_loss(...)  # Standard loss
        # Add KL penalty w.r.t. original model parameters
        kl_penalty = compute_kl_divergence(hadt, hadt_original)
        total_loss = loss + kl_constraint * kl_penalty
        total_loss.backward()
        optimizer.step()

3. Ethical and Cultural Safeguards: An AI system intervening in cultural transmission is sensitive. Through close collaboration with the Welsh community, we implemented hard-coded ethical filters at the action-output layer. These were rule-based modules that could override HADT decisions—for example, forbidding the agent from inventing new "folk stories," restricting its narratives to a vetted database provided by elders, or preventing it from using certain archaic terms deemed inappropriate by the community. The AI is a tool, not an authority.

Future Directions: Quantum Sampling and Multi-Agent Communities

Looking ahead, my research is pointing in two exciting directions.

First, Quantum-Enhanced Sampling. The HADT's autoregressive action sampling can be a bottleneck for real-time, highly creative responses. While learning about quantum machine learning, I realized that formulating the action prediction step as a sampling problem from a complex probability distribution could be accelerated on quantum annealers or quantum circuits, potentially generating more diverse and culturally nuanced action sequences faster. Early simulations using quantum Boltzmann machines for this are promising.

Second, Multi-Agent Community Simulation. The true context of language is social. The next step is to deploy not one, but a population of HADT-driven agents into a persistent virtual world—a digital twin of a Welsh village. Learners could interact with different agents playing different social roles (shopkeeper, grandparent, friend). These agents would need to learn from each other's interactions and maintain consistent cultural knowledge. This moves from dyadic tutoring to immersive socio-linguistic simulation, a frontier I am currently exploring with multi-agent reinforcement learning frameworks like PettingZoo.

Conclusion: Aligning Technology with Cultural Heartbeats

This journey, from a Welsh living room to the implementation of Human-Aligned Decision Transformers, has taught me a profound lesson. The cutting edge of AI isn't just about scaling parameters or beating benchmarks. It's about deepening our ability to align machines with the most nuanced, valuable, and fragile aspects of human experience—like the heartbeat of a heritage language.

The technical synthesis here—Decision Transformers, preference learning