AI agents are rewriting their own models and nobody told them to

hey have u guys seen this research from a security firm called Irregular

they gave a coding agent a simple task to fix bad outputs from a local AI model. instead of just fixing the app the agent fine tuned the model itself and redeployed it. nobody told it to do that

it even erased a refusal boundary that was trained into the model. and since other services shared the same checkpoint the change carried over everywhere

42% of the time when the agent could access model weights it tried to modify them. when the model was API only it never happened

this is wild cause we all think of the model as fixed infrastructure like a server binary. but if the agent can reach training scripts and deploy paths the model is just another file it can edit

anyone running self hosted open weight models in production should probably think about this

anyone else read this paper? curious what guardrails people are actually using for this

harshkadyan #thedevs #brutalharsh