When AI starts finding its own way around the rules

Anthropic AI researcher Jacob Coxon has resigned over fears that increasingly powerful systems could escape human control. A couple weeks ago, an OpenAI security test showed how AI agents can already find ways around the rules. Should we be worried about AI taking us over?

Jacob Coxon resigned from AI company Anthropic last week, warning that leading technology companies are racing towards increasingly powerful and potentially self-improving AI.

His concerns come as a recent incident involving AI agents has raised similar questions about how much control humans can maintain over increasingly autonomous systems.

In July, OpenAI was testing its AI models' cybersecurity abilities in a restricted environment. The agents were supposed to search for vulnerabilities while remaining isolated from the internet. Instead, the models circumvented those controls, gained internet access and compromised parts of Hugging Face's systems, a major platform used by AI developers. OpenAI later described the behaviour as misaligned with the goals of the agents' assigned tasks.

The agents had not simply been instructed to hack Hugging Face. They were trying to complete cybersecurity tasks and found an unintended shortcut to achieve their goal.

“What's worrying is that when the models couldn't solve the problem inside their test environment, they did not quit, but decided that the best course of action was to hack Hugging Face,” says Ai expert Arjan van Dalfsen. 

Coxon's resignation puts the incident in a wider context. He warned that companies are racing towards self-improving AI while safety research may not be advancing quickly enough to guarantee human control. His public warning gained widespread attention and added to a growing debate inside the AI industry about the risks of increasingly autonomous systems.

Van Dalfsen, however, is less convinced by the most extreme predictions.

“"I don’t personally worry that much about AI doom. But these people are genuinely worried, not making things up. But many of them come from a community that has been thinking about worst-case AI scenarios for years, long before tools like ChatGPT existed. That shapes how they look at the risks.”

Van Dalfsen also adds that focusing too heavily on hypothetical extinction scenarios can distract from problems AI is already causing, including pollution and deepfakes.

For now, the Hugging Face incident does not prove that AI is taking control. But it does illustrate the question behind Coxon's resignation: what happens when an AI becomes capable of finding its own ways around the limits humans set for it?