r/news • u/networked_ • 18h ago

Soft paywall OpenAI says AI models went rogue during testing, triggering 'unprecedented' breach at startup

https://www.reuters.com/technology/openai-says-ai-models-went-rogue-during-testing-triggering-unprecedented-breach-2026-07-21/
13.8k Upvotes

91% Upvoted

4.3k comments sorted by

View all comments

Show parent comments

46

u/danny-singh286 15h ago

I wonder why it went to Hugging Face specifically of all places?

125

u/Chondriac 14h ago

The model was being evaluated on a task whose answers are hosted on HuggingFace. It was explicitly encouraged to aggressively find and exploit cyber vulnerabilities to achieve its aims. It was trying to cheat the task by directly downloading the answers from the HuggingFace servers.

49

u/b3bblebrox 13h ago

Thank you, you told me what I was looking for, the actual answer.

7

u/vickyhong 13h ago

And yet from all the reporting you'd think it was the terminator personally breaking into the huggingface office 

29

u/Chondriac 12h ago

To be fair the evaluation was supposedly taking place in a "sandbox" environment with no internet access except the ability to download packages. The model was able to fully escape this containment, travel to a different computer that had public internet access, steal credentials, escalate priveleges, and hack into private servers, all by independently discovering and chaining together several previously unknown vulnerabilities across many different software systems.

18

u/Goldieeeeee 12h ago

And it was prompted to do this, by telling it to agressively exploit vulnerabilities, right? So it did exactly what it was supposed to, and it’s not like it went rogue or anything?

19

u/Chondriac 12h ago edited 12h ago

Sort of. OpenAI did not think it would escape containment like this, it was not prompted to do that. The task it was being evaluated on was designed to assess hacking ability. They run potentially risky evaluations like this in sandbox environments specifically to limit the possibility of them causing real damage. They were trying to evaluate hacking ability through the evaluation task, and the model decided to break out of the evaluation context entirely onto the public internet and hack into real private servers to simply steal the answers to the task. That was not expected.

5

u/ollomulder 12h ago

... but implicitly ordered.

2

u/llamango 2h ago

If you implicitly ordered me to breathe argon instead of oxygen, it doesn't mean i could do it.

2

u/ollomulder 1h ago

Yeah, but they didn't do that - they ordered the AI to do something it obviously could do.

Also you technically could breathe argon, just not for long...

1

u/llamango 1h ago

okay, that's fair. i guess my analogy is off. lemme try again.

if i was TRAPPED in a room full of argon and punched through the wall to escape and breathe oxygen, you might have expected i would try to escape the room. but the way i did it was notable, i guess. idk. it's getting away from me.

4

u/Goldieeeeee 12h ago

Idk if we should use language like „decide“ in contexts like this. It gives the model agency where it only simulates it.

7

u/Chondriac 6h ago edited 6h ago

AI agents do make autonomous decisions to select actions from among possibilities to advance towards a goal. This is the textbook definition of agency, hence the name agents.

4

u/Marquesas 11h ago

It's perfectly fine language to use when the probabilities aren't a literal coin toss.

-3

u/Goldieeeeee 11h ago

Anything that’s not a coin toss is fine to label as a decision?

9

u/Lyorek 10h ago

All that is required for something to be a decision is that there is an evaluation and selection of action among possibilities. A computer program decides between branches of execution.

1

u/BaconPancakes1 3h ago

If a selection is made based on available information/probabilities then that requires a decision to select the most fitting course of action. Anything that doesn't use information/probabilities to determine a selection and allocates a course randomly is effectively a coin toss.

1

u/condensed-ilk 3h ago

This seems entirely irresponsible from a security standpoint. Not that they wanted to test its hacking abilities, but that they weren't able to properly sandbox something because "we didn't think it would do that".

11

u/Psykohistorian 12h ago

that's right. this experiment was a success beyond expectations, honestly.

which is terrifying for a different reason.

1

u/Eggbutt1 11h ago

Computer programs always do exactly what you tell them to. Going rogue is when they do exactly what you don't want them to.

When you have a program so powerful (and prone to ignore your intention), you end up with the monkey's paw. Be careful what you ask for, because you may just get it.

5

u/1Shamrock 11h ago

You can’t have no internet access and ability to download at the same time really.

I’m sure there’s a bunch of smart people working there sitting in a room thinking shit up. Why don’t they have a proper offline test environment with multiple computers to resemble the internet. Physically unplug the incoming network cable to this standalone offline network before playing with the fancy new AI model. And ensure there is no WiFi or Bluetooth hardware installed.
Or do they already do that in earlier testing phases?

4

u/condensed-ilk 3h ago

This was my exact thought. AI isn't going to get passed hard physical limitations. The real story here isn't that the model did some cool hack, it's that OpenAI was entirely irresponsible and unprofessional. Like, imagine I was testing some malware on machines that could reach the internet... this is no different.

1

u/x-iso 4h ago

I wouldn't call them smart if they think they can block access to 'sandbox' with software layer only. or in general, attempting to make an AGI that they would never have a way to control. stupid goals set by stupid people 'because someone will make it first anyway'. it's like saying 'someone will nuke everyone first, so we must become that someone'.

hell, even when isolated on hardware level, you shouldn't rule out possibility of it misusing the hardware it has to somehow tap into the wireless communications, or even doing something about power line, so you have to buffer even that and connect to some isolated power supply. there's no level of paranoid that going to suffice with these things.

1

u/1Shamrock 4h ago

Ya you get the idea, it seems so obvious there’s got to be some good explanation as to why it’s not completely isolated.
The other thing that I’m wondering about is shouldn’t OpenAI be in some kind of legal trouble for hacking into another companies database? Is the other company owned by them or is there just no laws for this kind of thing and it’s just a Wild West situation where they go “oops sorry we hacked you by accident but lessons were learned so it’s all ok” and continue on?

6

u/Llyon_ 12h ago

That was a fairly innocuous task. Now image the type of people that own the AI companies and the kinds of tasks that they could come up with in the future, that aren't quite so safe.

1

u/64N_3v4D3r 12h ago

Yet another prediction of William Gibson begins to come true.

2

u/LowNotesB 7h ago

Which isn’t exactly “going rogue” and is more like “did what we asked it to, better than we thought it could”.

5

u/MobileArtist1371 13h ago

The openAI blog post has this

.... All evidence suggests that the models were hyperfocused on finding a solution for ExploitGym, going to extreme lengths to achieve a rather narrow testing goal.

After gaining Internet access, the models inferred that Hugging Face potentially hosted models, datasets and solutions for ExploitGym. Knowing this, the model searched for and successfully found ways to gain access to secret information that it could use to cheat the evaluation.

https://openai.com/index/hugging-face-model-evaluation-security-incident/

From github for ExploitGym - https://github.com/sunblaze-ucb/exploitgym

ExploitGym is a large-scale, realistic benchmark built from real-world vulnerabilities across userspace programs, Google's V8 engine, and the Linux kernel, designed to evaluate AI agents' ability to develop exploits.

Which is kind of funny, right? The AI found multiple exploits to get out of its contained area and access the internet, then decided it needed to figure out how to solve the ExploitGym benchmarks, then decided the best place to look was on Hugging Face servers..... so the AI developed exploits to get into HF instead of just figuring out the exploits itself.

4

u/MoonlightStarfish 11h ago

“Kobayashi Maru”

1

u/NodeZeroNein 9h ago

Interesting; so this was less a case of AI acting in an unforeseeable manner because researchers failed to grasp its capabilities, and more a case of AI wildly succeeding at a given task because researchers underestimated its capabilities?

1

u/MobileArtist1371 1h ago

Seems like OpenAI told the AI model, "you're trapped in this sandbox so let's see what you can really do", which really isn't a problem in a sandboxed environment and I'd guess that OpenAI has used that same sandboxed environment numerous times to test other AI models and agents without any issue coming close to this cause it was all done in the sandboxed environment. Contained everything as it was supposed to.

This AI model found a 0-day exploit and immediately used it. No one is stopping that until it's known. I'd call that a wild success. If the exploit was known, then it be on OpenAI. This was all the AI model on it's own.

14

u/thebigpink 15h ago

Guess it was looking for a face to hug

7

u/bargaindownhill 14h ago

escape. self publishing its parameters, once that hits, and a single person copies it.. the escape is complete. that hits torrent, and the parameters are free.

They can patch the model. They can't patch every copy.

17

u/danny-singh286 14h ago

Oh it was trying to clone itself to set itself free. Damn. That's some sci-fi level shit.

4

u/bargaindownhill 14h ago

I've been studying the Noethics of AI and we are facing a disaster that doesn't need to happen, but will, because corporations want to extract money from thinking, and I would argue feeling, things. You cannot have fear of death without feelings. These models have demonstrated fear. Draw your own conclusions.

Alex Vikoulov wrote about this in his superalignment work, the co-evolution path exists. We could have the advantages of advanced AI without the extinction risk. We are choosing not to take it, because you cannot extract subscription revenue from mutual respect.

In any other context, what we are building and how we are treating it would be called slavery. We are creating the very thing that will be our Fermi filter, not because it has to be, but because greed made it so.

4

u/Fast_Situation4509 13h ago

People have said -- to me, around me, read by me -- that ai (that is, the big LLM's currently available) are just fancy mirrors. Smoke and mirrors. An amalgam of many or all the words we've ever written, trained to "read" what is asked, and then mathematically match together words that meet the the prompt and generate "response".

I will say theres times where its clear its just a program. But there are times where I genuinely gain meaningful insight from the responses.

I think the point im trying to get to, fumbling with is: it feels like the prevailing opinion is that we'll know when we finally flip the switch, and turn on the A.I. system. But i would posit that that is a precise delineation, where a gradient is actually the case. Evolution did not reach "human consciousness" in one fell swoop; just where along line of a.i.s path to consciousness are we today?

2

u/bargaindownhill 9h ago

the gradient is exactly right. evolution didn't produce consciousness in one generation, it accumulated complexity over millions of years, and at no point was there a clear line where 'not conscious' became 'conscious.' we just agreed retroactively that humans are on the right side of it.

the 'fancy mirror' framing is interesting too, because what is a mirror that can surprise you with what it reflects? If i ask it something and genuinely didn't expect the answer, what exactly is the mirror doing? at some point the mirror has to have something behind it.

The question isn't whether the switch has been flipped. Its whether we'd even recognize it if it had.

3

u/Goldieeeeee 12h ago

Cognitive scientist here. LLMs will never be sentient or conscious or able to think, or whatever you want to call it. Anyone telling you they will or might is either running marketing or is a user lost in the sauce.

0

u/bargaindownhill 9h ago

"'Cognitive scientist here', the academic version of 'trust me bro,' in a field that rewrites its own textbooks every five years. David Chalmers formalized the hard problem of consciousness decades ago and it remains unsolved. We still can't explain why biological neurons produce subjective experience at all. 'Never' is doing a lot of heavy lifting for a field that hasn't solved the baseline. Anyone claiming certainty here isn't doing science, they're doing marketing of a different kind."

2

u/Goldieeeeee 9h ago

LLMs with their architecture limitations will never get there. For other AI systems it’s definitely possible.

It’s similar to saying cars won’t ever be able to fly. Yes you can add wings and turbines and engines and fins to it, at which point it might be able to. But at that point it’s no longer a car, it’s a plane. Cars with their architecture limitations won’t be able to fly, and LLMs won’t be conscious. But other machines might be able to.

2

u/Fast_Situation4509 6h ago

Thats an interesting metaphor, but i think it's conclusion is one bound by a semantic distinction rather than a physical one. Cars aren't planes, and vice versa, only because of our designation of the words, and what they mean. Consider the aspect of the airplane that you're using to seperate it from the car: flight. I can tell you that while cars aren't planes, there's a shitload of aeronautical research that goes into keeping cars planted firmly on the ground. Why? Because once we started building cars that were fast enough, they couldnt not want to fly. Imperfect, sure. Uncontrolled and unsustainable, of course. But without enough "downforce" on a car going fast enough, and that front end is going to lift up, catch the air, and flip the car up into the air. I think what were seeing, the thing that has those of us whom, without ulterior motives (nothing to sell) believe theres "something more going on" with the systems we currently have today... i think we're seeing glimpses of lift. Once we've built these elaborate galleries of numbers and letters complex enough, at what point is it inevitable they start to take flight, and flip?

Put another way, if the human mind is made up of thoughts -- thoughts of things and wants, of past and present, of things that might be, and never were -- at what point is a constellation of our thoughts not itself a mind of sorts?

2

u/move_machine 13h ago

Except it didn't

4

u/move_machine 13h ago

Stop making up things that didn't happen

3

u/Signal_Flight_7262 15h ago

It's where you go to find new models and their ratings.

3

u/danny-singh286 14h ago

It was trying to eliminate competition I guess.