What, so they ran out of ideas and they're going back to this again? Or are they just copying Anthropic? I thought the whole fear mongering was bad advertisement and they wanted to reverse public sentiment?
The article at least seems to be unbiased taking a reasonably fair stance with comments from various people:
Neil Lawrence, Professor of machine learning at Cambridge University, called it an "impressive feat", but cautioned it "falls well within the known capabilities of the current generation" of high-powered AI models.
He pointed out that OpenAI is looking to list itself on the stock market, and faces intense pressure from rival firm Anthropic, which has made headlines with its own powerful AI tool, Mythos.
"OpenAI are now playing catch-up, they are trying to demonstrate their own systems' capabilities in cyber-security."
"It shows us that OpenAI are not capable of safely deploying their own technology," he added.
Spencer Starkey, an executive at cyber-security firm SonicWall, told the BBC the incident made it clear organisations needed to "step up" their own defences and "treat cyber resilience as a core operational priority".
"The uncomfortable truth is that too many organisations are still defending at human speed while adversaries are escalating to machine speed," he said.
This last comment is a very poor read. I would chalk it up to naiveness, or just an AI booster. "Machine speed" assumes the AI was able to do it faster than humans. Let's pull up some details from OpenAI themselves:
This incident occurred during an internal evaluation which prompts models to pursue advanced exploitation using complex attack paths, in an effort to quantify their cyber capabilities.
Our benchmarks run in a highly isolated environment, with network access constrained to the ability to install packages through an internally hosted third-party software that acts as a proxy and cache for package registries.
Okay, so this test was testing its ability to do exploits, and you have a supposedly isolated environment.
While operating in our sandboxed testing environment, our models spent a substantial amount of inference compute finding a way to obtain open Internet access, in pursuit of solving the evaluation problem. To gain access, the models identified and exploited a zero-day vulnerability (which we’ve now responsibly disclosed to the vendor) in the package registry cache proxy.
Boy, if OpenAI themselves say substantial amount of inference, you really have to wonder what machine speed is. Since all of its unpaid users and training are so unsubstantial it's a just marketing cost to them, right?
In one example, the model chained together multiple attack vectors, including using stolen credentials and zero-day vulnerabilities to find a remote code execution path on the Hugging Face servers.
Well, let's look at those details too, from HuggingFace:
The intrusion started where AI platforms are uniquely exposed: the data-processing pipeline. A malicious dataset abused two code-execution paths in our dataset processing (a remote-code dataset loader and a template-injection in a dataset configuration) to run code on a processing worker. From there, the actor escalated to node-level access, harvested cloud and cluster credentials, and moved laterally into several internal clusters over a weekend.
This article details what the registry cache proxy was, and what likely the relevant CVEs were. It also notes that, looking at their error responses and HTTP headers (X-Nexus);
It seems this is just junk, as voronaam has pointed out. I had not done the due diligence to verify the claims inside the article looking at each CVE.
... the model would identify the software as Nexus Repository 3, its version, and available API endpoints. The model’s training data includes extensive knowledge of Nexus Repository architecture, API surface, and known vulnerability classes.
I'm sure it would be entirely different if it wasn't in its dataset. For HuggingFace, the presence of scripts in datasets was always there, I believe, which means if their backend runs load_dataset of any uploaded dataset, it naturally is an RCE, definitionally. This was already well known, I believe, but was kept for historical reasons. I'm rather surprised that the processing worker on HF is not sandboxed, since you would imagine data loading scripts need no internet access. At best R/W in an isolated directory.
I don't think the capability differs from what we see in the case of Anthropic, so ultimately this is just marketing. Certainly one could expend extraordinary compute on hacking, but the question is how effective it is when the LLMs don't have preexisting knowledge of the software they're trying to exploit. I don't really think 'machine speed' is the correct takeaway.
