## Redistribution‑Based Cost Inference Improves Sparse Safe Offline RL

Redistribution‑Based Cost Inference Improves Sparse Safe Offline RL

Safe offline reinforcement learning normally needs a cost value for every step. In real applications, supervisors only give a binary stop‑feedback: the first unsafe step is marked, and nothing else is labeled. This creates a temporal credit‑assignment problem – the algorithm must figure out which step caused the unsafe signal.

The paper “Redistribution‑based Cost Inference Improves Sparse Safe Offline RL” proposes a solution. It treats the binary signal as a distribution of hidden costs over the preceding trajectory. The method learns a cost‑redistribution function (f_\theta) that spreads the observed stop‑signal back to earlier steps:

# Pseudo‑code for cost redistribution
def redistribute_cost(traj, stop_idx, theta):
    costs = np.zeros(len(traj))
    for t in range(stop_idx):
        costs[t] = f_theta(traj[t], theta)   # learned inference
    return costs

With the inferred per‑step costs, standard safe RL algorithms can be applied without needing dense annotations. Experiments show lower violation rates and higher returns compared to baselines that ignore the sparse feedback.

The approach bridges the gap between practical supervision (only trajectory‑level signals) and the need for step‑wise safety information. It opens a path for safer offline policies in domains where dense labeling is too expensive or impossible.

Read more: http://arxiv.org/abs/2608.12306v1