One channel decided whether my multi-agent RL agents learned at all

Rakib — Sun, 12 Jul 2026 05:37:07 +0000

I have a small cooperative multi-agent reinforcement-learning setup: eight agents on a 100×100 grid, learning to reach three goals while gathering resources and avoiding obstacles. Standard MAPPO — a shared actor, a centralized critic, per-agent advantages, the usual PPO machinery. Each agent sees the world through an 11×11 egocentric window.

That window is the whole story, and it took an ablation to make me take it seriously.

The setup, and the thing that's easy to miss

Look at the numbers for a second. The grid is 100×100. The agent sees an 11×11 patch centered on itself — reach ±5 in each direction. Goals are placed uniformly at random. So on any given step, what's the chance a goal is actually inside the patch an agent can see?

Almost never. A goal is a small target on a 10,000-cell map, and each agent perceives roughly 1% of that map at a time. For the overwhelming majority of steps, the agent's observation contains no signal about where it's supposed to go. It's a partially observed task in the most literal sense: the objective is usually outside the sensory window entirely.

There's a common fix for this, and my environment uses it: an objective beacon. Instead of hoping the goal wanders into view, you project the bearing to the nearest un-reached goal onto the border of the observation — a "go this way" arrow painted into the goal channel, always present, pointing toward the objective even when the objective itself is far outside the window. It's cheap, it's a handful of lines, and it turns an unobservable target into a directional gradient the policy can climb.

I knew, abstractly, that this kind of thing helps. What I didn't have was a sense of how much — whether the beacon was a nice-to-have that shaved some episodes off, or something more fundamental. So I ran the ablation: train with the beacon, train without it, change absolutely nothing else, and measure.

The ablation, kept honest

The only thing that differs between the two conditions is a beacon=True/False flag that wraps the beacon-drawing block. Same architecture, same hyperparameters (repo defaults — LR 1e-3, γ 0.99, GAE-λ 0.95, clip 0.2, entropy 0.005), same 250 training updates, same 64 parallel envs, same evaluation protocol, same four seeds for each condition. Greedy evaluation over 50 fixed-layout episodes, plus a random-policy baseline for reference.

A quick sanity check that the toggle does what it says: summing the goal-channel activation across a batch at reset gives 490.0 with the beacon on and 1.0 with it off. With the beacon disabled, the goal channel is essentially empty — which is exactly the point. Without the beacon, the goal almost never appears in any agent's window, so the channel carries nearly no information.

The result

Condition	Success rate	Reach rate	Return	Steps to solve
Beacon ON (mean ± std, 4 seeds)	0.995 ± 0.010	0.998 ± 0.003	+9.05 ± 0.22	43 / 200
Beacon OFF (mean ± std, 4 seeds)	0.000 ± 0.000	0.105 ± 0.053	−75.4 ± 3.9	200 (timeout)
Random policy	0.00	0.10	−72.8	200

"Success" means all three goals reached in an episode; the rate is the fraction of episodes fully solved.

With the beacon, MAPPO solves the task: 99.5% of episodes, in about 43 of the allotted 200 steps, consistent across all four seeds (per-seed success: 1.0, 0.98, 1.0, 1.0). The training return climbs from about −10 and crosses into positive territory around update 45.

Without the beacon — identical everything else — it does not learn at all. Zero percent success on every seed. Its reach rate (0.105) is statistically indistinguishable from the random policy (0.10). The training curve never rises; it drifts slightly more negative over time. This isn't "a bit worse." It's at chance.

The gap between conditions is roughly 100× the seed-to-seed noise on success rate, so it's a real effect, not variance. The one-line version is honest: 0% → 99.5%, from one channel's worth of directional signal.

Why this is the whole ballgame, not a tuning detail

It's tempting to file "add a goal beacon" under reward shaping or feature engineering — a knob you turn to speed things up. This ablation reframed it for me. In a task where the objective is almost always outside the observation window, the beacon isn't accelerating learning; it's supplying the only consistent signal about where to go. Take it away and there's nothing to climb — the policy is doing RL on observations that, most of the time, don't say anything about the goal. The agents don't learn slowly. They don't learn.

The general lesson, which I think travels beyond this toy: in a partially observed task, what you put in the observation can matter more than the algorithm you run on it. I could have spent that time tuning PPO clip ranges or critic architectures and gotten nowhere, because the bottleneck wasn't the optimizer — it was that the input didn't contain the answer. A single well-chosen feature — the bearing to the goal — was the difference between a policy that solves the map 99.5% of the time and one that never solves it at all.

Honest boundaries

A few things I want to be precise about, because the strong number invites over-reading:

This is one task configuration, one machine, four seeds, 250 updates. The effect is large and dead-consistent across seeds, but it's a single setup, not a law.
"Beacon-off can't learn" is shown within this fixed, fair budget. I did not go hunting for a no-beacon recipe — much longer training, denser reward, bigger networks, different exploration — that might eventually crack it. The honest claim is narrow and specific: under identical budget and hyperparameters, the beacon is required for learning here, not that the task is unlearnable without one in principle.
Both conditions keep the dense distance-to-goal reward term, which is why the beacon-off return sits around −75 (a persistent distance penalty with no way to reduce it). The success and reach-rate columns — which don't depend on that shaping — are the ones carrying the real 0% vs 99.5% story.

The code

This is from Kelvane, a small open-source (Apache-2.0) toolkit I've been building: a WebAssembly runtime for running untrusted, hot-swappable neural policies under a per-call compute and authority budget, paired with this compact multi-agent RL reference that trains policies and exports them to ONNX to run inside that sandbox. The gridworld, the MAPPO/QMIX trainers, and the beacon are all in kelvane-marl, and the ablation toggle is a few lines you can flip yourself.

Repo: https://github.com/rakib-nyc/kelvane

If you're doing partially observed multi-agent RL and your agents stubbornly refuse to learn, the first question I'd now ask isn't about the algorithm. It's: does the observation actually contain the objective? Mine didn't — until one channel said which way to go.