Most security AI assumes something it shouldn't: that its inputs are telling the truth.
A defender model reads the logs, sees an intrusion and responds. But a real attacker who has made it inside isn't polite. One of the things they can do is go after the logs themselves. If the defender's eyes are the logs, the cheapest attack is to lie to its eyes.
So I built PHANTOM for the Meta × PyTorch × Hugging Face OpenEnv Hackathon India 2026. An AI defender works a live breach. At the same time, an adversary AI feeds it fabricated SIEM entries in real time, gaslighting it about what's happening.
Decision 1: attack the logs, not the network
I could have made the adversary attack harder, with more intrusions and faster moves. That trains a defender to be quick.
Attacking through the logs trains something more valuable: a defender that doesn't believe everything it reads. It has to cross-check, notice contradictions and act on evidence that holds up, instead of whatever the latest log line claims. That's the skill that matters once someone is already inside.
Decision 2: both sides learn
A fixed adversary gets memorised. The defender learns its five tricks and looks brilliant, right up until it meets a sixth.
So both agents learn against each other. As the defender gets better at spotting fake logs, the adversary gets better at writing convincing ones. It's multi-agent reinforcement learning, and the pressure goes both ways, which is closer to how real security works. Every defence teaches attackers what to try next.
Decision 3: GRPO
For training I used GRPO, group relative policy optimisation. Instead of training a separate model to estimate how good each situation is, you let the policy try the same situation several times, as a group, and compare those attempts against each other. Better than the group's average gets reinforced, and worse gets discouraged.
It's simpler than the classic setup, and a good fit when the only thing you can really judge is whether one response beat another.
Decision 4: start easy
A defender thrown straight at a skilled liar learns nothing. It just fails, over and over, with nothing to learn from.
So there's an adaptive curriculum. Early on the deception is clumsy and the breach is simple, and as the defender improves the environment gets harder. The aim is to keep it at the edge of what it can handle, because that's where learning actually happens.
Decision 5: an LLM as the judge, carefully
How do you score incident response? "Did it block the right IP" is easy. "Did it reason sensibly about conflicting evidence" isn't something you can write a simple rule for.
So responses are scored by a language model acting as a judge. It can weigh reasoning in a way rules can't, and it brings its own biases. A judge you can't fully check is a risk, and I'd keep thinking about that before trusting the scores for anything that matters.
What I took from it
The interesting failure in AI security isn't a model that can't see the attack. It's a model that sees exactly what it's shown and believes it.
A defender that trusts its inputs isn't a defender. It's an audience.
Top comments (0)