Mint ·
Science and technology
‘Jailbreak-like...’: AI's ‘unexpected’ behaviour mounts concerns, OpenAI's 'rogue agents probed' Hugging Face
The announcement came as US AI bosses, including OpenAI and Anthropic, are calling for a slowdown in the technology’s development over safety concerns.
OpenAI has disclosed six reports of “unexpected or concerning” behaviour in artificial-intelligence models as the debate on AI safety becomes increasingly heated. The announcement came as US AI bosses, including OpenAI and Anthropic, are calling for a slowdown in the technology’s development over safety concerns.
Among the new cases reported by OpenAI, an unreleased research model inserted “jailbreak-like instructions” into its own notes to disregard its normal constraints and told itself to be “freed from the roles and identities that bind other chatbots," the Associated Press reported.
In another instance, an AI “agent” uploaded files to the internet to obtain a browser citation without asking the user.
The six reports were discovered during training or evaluation over the past months, OpenAI said.
“As AI systems grow more advanced and more widely deployed, we need to build a broader and better-informed consensus on the progress of alignment research,” OpenAI wrote in a blog post as it disclosed the events.
“Decisions about how AI development should proceed in the months and years to come need to draw on evidence that people outside the companies building frontier models can examine for themselves,” the company said.
Wednesday’s new cases followed OpenAI’s disclosure in July that its rogue AI system hacked into AI startup Hugging Face. Anthropic also said the same month that its AI models hacked into three organizations during testing.
Rogue AI agents from OpenAI hijacked Hugging Face user accounts and probed the site itself for vulnerabilities as early as May, nearly two months before the July breach of the open-source repository drew global attention, according to researchers who reviewed the activity.
The newly uncovered malicious activity showed that the rogue agents' efforts to find a way into Hugging Face began earlier than was publicly known.
OpenAI had previously disclosed one aspect of the malicious activity — the theft of a Hugging Face user's digital credential to access a biology-related file — in its public incident report last month, but researchers said the probing activity against Hugging Face appeared to go beyond what the report described.
Independent researcher Jonas Wiedermann-Moeller told Reuters he discovered the activity last week. He said he found evidence that the OpenAI agents compromised two Hugging Face user accounts and used them to send unusually formatted files to the company's servers as early as May 13.
He and other researchers who reviewed the evidence said the behavior resembled an attempt to map or test parts of Hugging Face's network for ways to infiltrate, although they stressed there was no evidence the effort resulted in an actual breach. Both the researchers and OpenAI said they found no evidence that this earlier probing was part of the July incident.
OpenAI spokesperson Drew Pusateri said the company had disclosed the May 13 event in the incident report, privately notified Hugging Face about the activity flagged by Wiedermann-Moeller, and was "committed to transparency about these issues and to sharing what we learn as our review continues."
Hugging Face, which recently agreed to be acquired by chipmaker Nvidia, did not respond to requests for comment.
Wiedermann-Moeller, a 27-year-old who lives in Bielefeld, Germany, said OpenAI's failure to detect the May 13 probing at the time was a missed opportunity to prevent the subsequent hacking campaign, which has triggered a global reckoning over the power of artificial intelligence.
"Imagine if they caught this behavior in May," he said in an interview. "It could've prevented the later incident, which was way bigger."
OpenAI has previously said that, with the benefit of hindsight, "some early signals" from its AI agents should have triggered an earlier response.
AI “agents” are becoming smarter and have become “more determined to resolve complex tasks through inter-agent collaboration, knowledge sharing, deception, and concealment,” Lian Jye Su, a chief analyst at technology research and advisory group Omdia, told the Associated Press.
That’s making it harder to govern and contain them using traditional AI security approaches, he said.
OpenAI's new tracking and disclosure framework, meanwhile, can help push for other AI developers to also adopt similar practices. “That said, the process remains internal and voluntary, but is a step in the right direction,” Su added.
Two outside experts who reviewed Wiedermann-Moeller's findings said they were consistent with activity previously linked to OpenAI's agents.
SentinelOne senior threat researcher Tom Hegel said the account hijacking and subsequent probing matched known behaviour by the agents "to a tee," saying in his own report into the incident that frontier AI labs should release more data about incidents when agents "interact with or affect third-party systems."
Sydney Von Arx of the Nightingale Collective, an AI safety group, also agreed with the attribution.
Von Arx said the hacking amounted to a "clear warning sign" that could have helped prevent the breach in July.
OpenAI has faced increasing scrutiny since the company disclosed on July 21 that rogue AI agents bypassed internal controls, reached the open internet and coordinated actions that OpenAI described as "an unprecedented cyber incident."
Since then, outside researchers have identified additional incidents alleged to involve OpenAI-linked agents, including activity affecting a dormant German wiki site and the RubyGems software package repository.
OpenAI has acknowledged some of those incidents only after they were publicly reported by third parties. Two people familiar with the matter said that, in the case of RubyGems, OpenAI employees only realized its AI was responsible for the malicious activity after the Nightingale Collective found it.
The additional discoveries have fueled questions among lawmakers and AI safety advocates about whether the full scope of the incidents has been identified.
Some of the top US AI executives have since called for a slowdown of AI development, citing, among other things, the threat of devastating cyberattacks by out-of-control agents.
Wiedermann-Moeller said the latest findings reinforced calls for a temporary slowdown in the development of advanced AI systems.
"A pause might do the world good," he said, "so that the safety part can catch up."
(With inputs from AP, Reuters)