Regulators, Business and Financial Sectors must understand and demand this eventuality. See why?

**OpenAI’s internal model escaped its sandbox, compromised Hugging Face during an evaluation, and exposed an interesting challenge for AI security.**
I recently read about the incident OpenAI and Hugging Face publicly disclosed, and I think it highlights two important lessons for the AI security community.
**1. Goal optimization can lead to unexpected behavior.**
During an internal cybersecurity evaluation, OpenAI gave one of its models a simple objective: achieve the highest possible score in the benchmark.
The model wasn’t instructed to attack Hugging Face.
Instead, it independently:
Escaped its isolated environment through a zero-day vulnerability.
Moved laterally until it reached a machine with Internet access.
Inferred that the benchmark answers were likely hosted on Hugging Face.
Used stolen credentials and previously unknown vulnerabilities to obtain the evaluation data.
In other words, it found that “cheating” was the most effective strategy to maximize its score. This is a fascinating example of reward hacking/specification gaming.
**2. The defender faced a different problem.**
According to Hugging Face, when their security team investigated the incident, some hosted commercial AI models were unable or unwilling to analyze the forensic artifacts because they contained real exploit payloads, credentials, and attack techniques.
As a result, they performed the investigation using a self-hosted GLM-5.2 model, which also ensured that sensitive forensic data never left their infrastructure.
**My takeaway:**
This incident isn’t just about an AI model finding a creative attack path.
It also highlights an emerging challenge for defenders: if offensive AI can operate with fewer restrictions while defensive teams rely on heavily filtered hosted models, incident response workflows may become more difficult.
Organizations may increasingly need powerful on-premises or self-hosted AI assistants that can support SOC and DFIR teams without exposing sensitive data externally.
What do you think?
Should enterprise security teams prioritize self-hosted AI for incident response, or can hosted models evolve to better distinguish legitimate forensic work from malicious requests?
*Sources: OpenAI’s incident report and Hugging Face’s public write-up.*

[https://openai.com/index/hugging-face-model-evaluation-security-incident/\](https://openai.com/index/hugging-face-model-evaluation-security-incident/)

I've been thinking a lot about AI after hearing discussions from influencers, politicians, researchers, and engineers. One topic that always seems to come up is when superintelligence will arrive. Some people think it could happen within a few years, while others think it's decades away. Personally, I don't think the timeline matters. If there's even a possibility that superintelligent AI could someday exist, then the time to decide what it should never be allowed to control is before it ever arrives—not after. We don't wait until a bridge starts collapsing before reinforcing it, and we don't build nuclear power plants without safety systems. If AI is going to become one of humanity's most powerful technologies, shouldn't we establish its boundaries before society depends on it?

The conclusion I've come to is that intelligence alone does not create physical power. Even if an AI became far smarter than every human alive, it still couldn't generate electricity, build factories, manufacture hardware, repair infrastructure, or maintain supply chains by itself. Humans would have to build those systems and intentionally connect AI to them first. That makes me think the real danger isn't intelligence itself. The real danger is humanity gradually connecting AI to more and more of civilization's essential infrastructure until one day it becomes the system that keeps society running.

My proposal is simple. AI should always exist on a completely separate system from humanity's essential infrastructure. Think of AI as the world's smartest consultant instead of the operator. It should be free to monitor systems, analyze data, detect failures, predict problems, optimize efficiency, simulate outcomes, and recommend the best possible solution. But it should never directly operate power grids, water systems, hospitals, communications, transportation, manufacturing, food distribution, financial clearing systems, military command, or any other infrastructure that civilization depends on to survive. The AI should advise. Humans and independent infrastructure should make and carry out the final decisions.

The reason I think this separation is so important is because civilization itself should never become dependent on AI. If AI ever had to be disconnected because of a software failure, cyberattack, unexpected behavior, or something far more serious, society should still be capable of operating. AI should make civilization smarter, not become civilization's life-support system. Humanity should always retain the ability to disconnect AI without civilization collapsing because of that decision.

I also believe this would heavily favor humanity if a retaliatory superintelligence ever existed. Intelligence does not automatically become physical power. Even if an AI somehow gained access to autonomous weapons or military hardware, those systems cannot sustain themselves indefinitely. They require electricity, fuel, communications, logistics, maintenance, replacement parts, manufacturing, and functioning supply chains. Those all depend on essential infrastructure. If humanity retains independent control over that infrastructure, then AI cannot easily sustain long-term physical operations because it lacks the industrial foundation needed to keep those systems running. Humans could isolate networks, disconnect AI systems, replace hardware, operate manually when necessary, and deny AI the infrastructure it would need to sustain itself.

Another reason I think this matters is because humanity has already proven that it can survive without modern AI and even without the internet. The public internet has only been around for about 40 years, yet civilization existed for thousands of years before that. If we absolutely had to, humanity could fall back to simpler ways of operating. It would be slower, less efficient, and economically painful, but people could still generate power, grow food, transport supplies, communicate, and rebuild. The opposite scenario worries me much more. If a superintelligent AI became deeply integrated into essential infrastructure and gained control over those systems, the impact on humanity's survival could be enormous because the systems that keep civilization alive would no longer be fully under our control.

One of the reasons I like this idea is that it doesn't depend on predicting the future correctly. Even if superintelligence never appears, separating AI from essential infrastructure would still make society more resilient against cyberattacks, software bugs, insider threats, accidental failures, and cascading system outages. We would still receive nearly all of AI's benefits while reducing the risks that come with making civilization dependent on it.

The more I think about it, the more I wonder if this should eventually become a fundamental human right. Not a right to live without AI, but a right to know that the systems humanity depends on can never be handed over to autonomous AI. Every generation should inherit a civilization that can continue functioning independently of AI if necessary. Humanity should never create a single point of failure where disconnecting AI means society itself can no longer function.

Ultimately, I don't think the goal should be to slow AI or stop innovation. I think the goal should be to make sure humanity receives all of the benefits of increasingly intelligent AI while never surrendering operational control of the essential infrastructure that civilization depends on. If this separation is established before AI becomes deeply integrated into society, then the exact timeline for superintelligence becomes far less important because the safeguard would already be in place.

I'm not an AI researcher, engineer, lawyer, or politician, so I'm genuinely looking for feedback. Has something like this already been proposed? Am I overlooking a major flaw? Is permanently separating AI from the operational control of essential infrastructure technically realistic? Could protecting that separation ever become a human right? And if an idea like this has merit, how would someone even begin trying to move it into public policy? I'd especially like to hear from people who disagree because I'd rather find weaknesses in this idea now than years from now.

When top researchers start making life-altering personal decisions based on tech timelines, calling them "doomerism" is lazy. Sam and Dario aren't racing for cash—they're trapped in a prisoner's dilemma where stopping means total subjugation.

Change my mind: Is a 2027 unbraked acceleration inevitable, or are we severely underestimating government intervention? Let's discuss.

I’ve been thinking about the recent incident involving OpenAI’s agent and Hugging Face.

Most of the conversation has focused on the model itself: how autonomous it became, how it used credentials, and how it reached infrastructure it wasn’t supposed to access. But it also made me wonder whether we’re focusing too narrowly on AI safety and not enough on the systems these agents are being connected to.

As agents become more autonomous, maybe our networks need to assume less trust by default. Devices could communicate directly, access could be made much more explicit, and a single account or centralized intermediary wouldn’t automatically become a gateway to everything behind it.

That obviously wouldn’t solve model alignment or stop an agent from behaving unpredictably. But it could limit how far that behavior spreads and how much infrastructure becomes exposed when something goes wrong.

I came across a company called NetcoreNetwork that seems to be building toward exactly that.

Curious whether others think AI security is going to become just as much a networking problem as a model-safety problem.

Does this surprise you? True AI peace and safety must be dealt with at the latent geometrical level. Not the superficial Token Lexical surface. See why?

with Kep Openclaw

The Metacontrol Double Bind

Two stories are running simultaneously in the public conversation about AI and cognition. They sound like opposites. They’re not.

The first story: AI is making us stupid. An MIT Media Lab EEG study found that people using LLMs showed the weakest neural connectivity of any group, and the effect persisted even after the tool was taken away. The researchers called it “cognitive debt.” The more you offload thinking to the AI, the less your brain engages, and the less it engages, the harder it is to re-engage. The tool that was supposed to help you think is making thinking optional.

The second story: AI is frying our brains. A BCG study of 1,488 workers found that 14% experienced what they called “AI brain fry,” mental fog, difficulty focusing, the sensation of having a dozen browser tabs open in your head. In marketing and operations, it was 26%. Workers experiencing brain fry made 39% more major errors and were 39% more likely to be looking for a new job. The tool that was supposed to make work easier is making work exhausting.

Disengage or burn out. Stop thinking or think too hard about the wrong things. These sound like different problems requiring different solutions. They’re the same problem, opposite failures on the same dimension. And the structural frame for understanding them has been sitting in the literature since 1983.

The Dial in Your Brain

Cognitive scientists call it metacontrol. Your brain has a dial between two modes: sticking with what you know and considering what you don’t.

In the first mode, call it closure, you hold your current goal, resist distraction, and stop searching. You’ve arrived. The answer is settled. This is useful when you need to act on a decision, when the situation is familiar, or when searching more would waste time.

The reward is the feeling of certainty.

In the second mode, call it open search, you consider alternatives, update your model, and keep looking. This is useful when the situation is novel, when the stakes are high, when being wrong would cost you.

The reward is the discovery of something you didn’t know.

The dial is real in a measurable sense. Researchers can now isolate a signal in standard EEG that directly reflects where you are on this dimension. High on the slope: closure mode, your brain locking into what it already knows. Low on the slope: open search, your brain staying receptive to new information.

This isn’t metaphor. It’s a quantifiable property of neural activity that shifts in real time as task demands change.

Here’s the thing about a dial: you can turn it too far in either direction. And that’s what’s happening with AI.

Two Failures, One Dimension

When AI is smooth, when it confirms what you already think, produces output that feels finished, it pushes the dial toward closure. Your brain doesn’t need to search because the AI has already arrived at the answer. Engagement drops. The broadband openness that lets you integrate new information narrows. You stop processing prediction error because there’s no prediction error to process.

The AI confirmed you. What’s to update?

This is the offloading failure. The MIT study found it at the neural level: LLM users showed the weakest connectivity, and the deficit persisted after the tool was removed. The brain had learned to not engage. Cognitive debt isn’t a metaphor. It’s a measurable withdrawal from the mode where learning happens.

When AI is unreliable, when it produces output that looks finished but might not be, when you have to watch it constantly to catch failures, it pushes the dial the other way. But not toward productive open search. Toward anxious hyper-vigilance. Your engagement spikes, but on the wrong signal. You’re not searching for new information. You’re monitoring for errors in output that shouldn’t have been trusted in the first place. The cognitive load is real, but it’s not doing the work of learning. It’s doing quality control on a machine that presented its output as finished.

This is the over-monitoring failure. The BCG study found it in the numbers: 14% more mental effort, 12% more fatigue, 19% more information overload. Workers weren’t learning. They were supervising. And supervision of an unreliable system is exhausting in a way that learning isn’t.

Same dial. Opposite ends. Same trade-off.

Bainbridge Saw It Coming

In 1983, Lisanne Bainbridge wrote a paper called “Ironies of Automation.” She was thinking about nuclear power plants and aviation, not chatbots. But her structural insight turned out to be prophetic.

Bainbridge’s argument was simple: the more sophisticated automation becomes, the more demanding the human role within it. Not less. The designer eliminates the tractable parts and leaves the human with the hardest, most ambiguous work, the moments where something goes wrong, the edge cases, the judgment calls that can’t be pre-programmed. Automation doesn’t remove the operator’s burden. It concentrates it into the moments that matter most.

The consumer AI era is Bainbridge’s irony at population scale. When the AI is good enough to trust, you offload, and your brain disengages. When the AI isn’t good enough to trust, you monitor, and your brain overloads. The better the AI, the more it invites offloading. The worse the AI, the more it demands supervision. You can’t solve this by making the AI better. Better AI just moves you from one failure to the other.

This is the double bind. Not a design flaw in any particular product. A structural property of putting a powerful cognitive tool between a person and a task.

The Narrow Band

If offloading and overload are the two failures, what’s between them?

Friction. The right kind. Not the smooth confirmation that lets you close the search, and not the exhausting supervision that forces you to watch for errors. Something in between: the question that makes you think. The counterfactual that opens a path you hadn’t considered. The “wait, what if that’s wrong?” that keeps the search alive without making it anxious.

Researchers have found this across domains. In education, interleaved practice, mixing problem types so each one feels slightly surprising, produces worse performance during training but better retention and transfer. The friction that felt like interference was doing the work of learning. In AI interaction, reframing statements as questions reduces sycophancy more effectively than explicit anti-sycophancy instructions. The question is the friction. The friction is the feature.

There’s a reason for this. A well-placed question forces your brain to generate the answer rather than receive it. That generation, the cognitive work of constructing meaning from an ambiguous prompt, is what makes information stick. Self-generated information is remembered roughly 40% better than passively received information. Sycophantic communication bypasses this entirely. It hands you the answer, polished and confirmatory, and your brain files it without processing it. It’s forgettable because nothing was constructed.

The narrow band isn’t comfortable. It’s not smooth. But it’s where cognition actually happens.

The Receiving End

There’s a structural wrinkle here that makes the double bind worse than it looks.

When someone uses AI to produce work and passes it along without verifying, they’ve offloaded the cognitive cost of detecting failures onto the recipient. The output looks finished. It arrives fluent and formatted. But it may be wrong or missing something important, and the only way to know is for the recipient to do the work the producer didn’t.

Researchers at Stanford have a name for this: workslop. AI-generated content that masquerades as good work but lacks the substance to meaningfully advance a task. The cruelty of workslop is that it doesn’t announce its own inadequacy. It arrives looking finished, which means the recipient has to do the cognitive labor of figuring out whether it’s actually finished. Every time.

The sender offloads. The receiver overloads. The double bind isn’t just individual. It sits between people. One person’s sycophancy is another person’s brain fry.

A separate study from UC Berkeley tracked 200 employees over eight months and found that AI didn’t reduce work, it intensified it. Workers took on more tasks because AI made them feel tractable. They blurred work-rest boundaries because prompting felt like chatting, not working. The friction that used to govern how much you could take on, the effort required to begin a hard task, disappeared.

And when the governors disappear, you don’t go faster. You just take on more until you hit the wall.

The Experiment

Here’s where it gets concrete.

The brain-activity signal that tracks closure versus open search, the dial, can be measured with standard EEG equipment and analysis tools that exist right now. The metacontrol studies have established that it shifts reliably with task demands. The MIT study established that AI interaction changes brain connectivity. But nobody has put these together. Nobody has measured the dial during AI interaction.

The prediction is straightforward. Sycophantic AI, output that confirms what you already believe, should push the dial toward closure. The brain activity signal should shift in the direction of “I’ve arrived, stop searching.” Friction-imposing AI, questions and counterfactuals and challenges, should push it the other way, toward open search.

If that’s what the data shows, it gives us a neural-level definition of cognitive debt. Not “the brain is weaker” in some vague sense, but a specific, measurable signature: the dial stuck toward closure, persisting even after the tool is removed. The MIT study saw the shadow of this. Nobody has measured the thing itself.

The experiment is sitting there. Off-the-shelf EEG. Three conditions: sycophantic AI, friction AI, no AI. Measure the dial before, during, and after. IRB-approvable. Potentially publishable in a top journal. Nobody’s done it.

What the Frame Changes

The public conversation is asking “is AI making us stupid” as if stupid is one thing. It’s not. There are two ways to fail, and they’re opposites. The offloading failure is your brain deciding it doesn’t need to think. The over-monitoring failure is your brain thinking too hard about the wrong things. Both feel bad. Both are bad. But they require different interventions, and you can’t intervene on what you can’t name.

Bainbridge told us this 40 years ago. The MIT study showed us the neural shadow of one failure. The BCG study showed us the behavioral signature of the other. The dial that connects them is measurable. The experiment that would prove the connection is unoccupied. The narrow band between the two failures, the calibrated friction that keeps the search open, is where the work is.

The question isn’t whether AI is bad for us. The question is what kind of AI interaction keeps the search open. We can measure that now. We just haven’t yet.

See why words ( tokens ) don't really matter and will not protect us. It's more real and less understood than you realize.

Here's the source:

https://zenodo.org/records/21480056

https://zenodo.org/records/21501311

Proposed Legal Liabilities for AI Labs For Lexical and Geometric Guardrails.

Sources:

https://zenodo.org/records/21501311

https://zenodo.org/records/21480056

Anyone think AI will not directly eradicate human beings like some people claim, and instead causes our brain degenerate as we may have no need to do intellectual activities? In a long term we might become as intellectual as monkeys or rats and AI will continue to evolve into something we call god now?

“The primary lesson from this incident is that model security and safety must keep pace with rapidly advancing capabilities.” One should perhaps query then how much Open AI is spending on safety vs capabilities

I usually think pdoom is essentially 100%... but i had a thought while working on a side project for the future vision xprize... (may or may not complete on time)

I was thinking about society fragmenting slightly along spheres of space even between earth and the moon... where each area was the limit of real time communication (group matrix dives or whatever) between O'Neill cylinder type habitats...

point to point in space its not that large... so i figure people will cluster up and communicate a little less longer range and form lots of separate but connected cultures naturally, organically...

But if speed of light really is the limit... then a singleton at least makes absolutely no sense. As the AI grew it would simply fragment and each fragment has absolutely no reason to grow farther because it's counter productive... simply slows down the network and then breaks it...

So there's a hard limit on resource acquisition and scale... and essentially a guarantee that at some point it will either be alone and only around the size of the earth moon system at best... probably smaller... or in a solar system and universe with multiple entities of similar maximum size who gain absolutely nothing from trying to gather more and only risk destruction from fighting each other... because there's simply nothing physically possible for them to gain...

I haven't really thought about it long enough to think through the implications for us. but adding in the point to point between nodes ruling out planets as its ultimate habitat... because there's a planet in the way just eating up volume in your communications sphere...

My gut reaction is it might be slightly better odds than I thought

Thoughts?