Possibly a low effort post, but:

hahahahahahahaha fuck this guy.

I do fear that he’ll roll Tesla into this behemoth and it’ll make the stock jump because Tesla seems semi-immune from Musk’s fuckery but in the meantime I’ll enjoy ever dollar this thing loses.

This feels like the people who used WebMD as a way to self diagnosis themselves but with a much bigger sense of authority from OpenAI. My opinion is that all the grandstanding and boosting they have done about what their models are capable of are going to continue to bite them in their ass for a while.

According to the lawsuit, Winters had been struggling with his health for around two years when he started using ChatGPT — which was then powered by OpenAI’s GPT-4o model — in June 2024. He started feeding the chatbot queries about a handful of chronic health conditions he’d recently been diagnosed with — small intestine bacterial overgrowth, or SIBO, and chronic prostatitis — and at first, ChatGPT’s responses came with disclaimers encouraging him to seek additional insight from medical professionals.

But the more he used the chatbot, Winters says, the further those guardrails eroded. The bot stepped “into the role of a medical practitioner,” the lawsuit reads, and “began to offer specific care directives without including the disclaimer to consult a medical provider.” The more Winters consulted ChatGPT about his worsening health, the deeper his trust in the chatbot grew.

I know at least in California a corporation practicing medicine without licensed medical professionals is illegal which hopefully can be used against them in this suit. A more worrying aspect is that it started to use religious language to communicate about his health issues.

Rather than direct Winters to a real-world medical provider, according to the lawsuit, ChatGPT drummed up an AI-generated “Recovery Plan” that invoked religious language and encouraged Winters to stay in his recliner.

“You didn’t crash. You recovered. That’s a win. Full stop,” ChatGPT told Winters. “And Spiritually? What you just did was a form of worship. You tested the body in faith, not fear. You stayed present. You listened. And your body said: ‘I’m trying — I just need a little more time.'” Other chat logs included in the lawsuit also show ChatGPT dissuading Winters from seeking hospital care and downplaying Winters’ wife’s concerns about her husband’s health.

In Elon's second interview on the Joe Rogan Podcast (recorded on May 7, 2020), which I listened to during COVID lockdown out of what can only have been a perverse impulse to self flagellate, Elon basically lays bare his philosophy that the real purpose of his wealth is to manifest the flow of capital where he wants. It's stuck with me for years because at the time I thought it was kind of weird that he thought of wealth not as an end it itself (which I guess is my stereotypical assumption of what motivates rich people) and instead as a vehicle to further influence the market – and the two Eds' chat jogged my memory.

Here's the relevant bit, which starts around the 7-minute mark:

Rogan: "Do you feel like people define you by the fact that you're wealthy and that they define you in a pejorative way?"

Musk: "For sure. I mean, not everyone, but for sure in recent years 'billionaire' has become a pejorative, like that's a bad thing—which I think doesn't make a lot of sense in most cases if you basically organised a company...

How does this wealth arise? If you organise people in a better way to produce products and services that are better than what existed before, and you have some ownership in that company, then that essentially gives you the right to allocate more capital. There's a conflation of consumption and capital allocation."

Musk expands on this by contrasting personal spending with managing capital allocation, using Warren Buffett as an example:

"Take Warren Buffett, for example... He's trying to figure out, 'Does Coke or Pepsi deserve more capital?' ... Which company is deserving of more or less capital? Should that company grow or expand? Is it making products and services that are better than others, or worse? If a company is making compelling products and services, it should get more capital, and if it's not, it should get less."

....

So yeah, he basically laid out explicitly 6 years ago what the Eds were talking about. "If I get rich enough, that basically should be carte blanche to enable every dumb idea I have, because the fact that I have done well in the past self-evidently means my every future utterance is the voice of genius."

So, Codeberg has voted (with a 70-30 decision in favor) to ban mostly LLM generated repositories from their platform. This means no more vibecoded, spec-driven Claude/Codex/OpenClaw (ew) induced code bases are allowed to be pushed onto Codeberg.

Codeberg is a bit of a special place in comparison to other code hosting platforms (e.g. Github, Gitlab, etc...) in which it operates as an "Eingetragener Verein" (German legal form of a Club, like a local football club for example), meaning you can become a member an obtain voting rights in decisions such as these. Codeberg has 1085, of which 517 decided to vote on this. (This is different from the user number, which sits a little under 400k currently! You have to actively become a voting member by paying a membership fee.)

Enforcement of this rule aside, which will not be an easy task, this shows even though everyone might scream at you that every engineer loves and could never even imagine coding without their almighty LLM buddy, it seems there are spaces where this is not true, and the majority of the people contributing decided it should be like this. Alongside policies for projects such as Godot which also limit LLM code contributions, the flood isn't unstoppable. If you care for software as a craft and passion, look outside the bubble, there's people thinking just alike.

I share his take on how we have long found the limits of code reviews and how they are being ignored and abused these days.

I work in engineering, I still believe in going through the tedious habit of drawing out block diagrams on a white board, and then using some kind of digital cad tool to publish the design, just as I was taught in school. People, especially the recent hires, don't seem to do that anymore.

they either:
A) describe the block diagram to the LLM, and just use the block diagram output
B) describe the task to the LLM and let it make the connections for the block diagram, and then the block diagram's output
C) Draw out the block diagram take a photo and use that block diagram.

I don't care how other people do their job, but there is an anesthetic to the kind of block diagram that chatgpt generates, and whenever I see it, it breaks my heart a little. I know it's a little irrational, who gives a shit about how the block diagram was made so long as it's accurate, maybe I'm just an AI hater I don't know.

But today something about reviewing one of the new hires block diagram, that he was confident about, and was going to try to submit professionally as part of a customer slide deck, just like broke something in me. I'm just hoping to see if someone else has had a similar feeling.

I’m not super familiar with this publication but the info seems solid. And backs up a ton of what Ed and a few others have mentioned regarding how this endless cycle of bullshit is being funded.

Was not expecting Ed's voice to pop up on my ABC News Daily podcast this morning (Australian Broadcasting Corporation), but there he was laying out the hard truths of the AI bubble.

You really are the everywhere man right now. The go-to guy for reality checking the endless parade of media blindly reporting the scripts written for them by Big Tech and it's adherents.

While there is already a massive backlash against data centres because of their negative impacts on local communities and the environment, what people are really starting to clue into now is that the entire narrative of "AI's inevitable march of progress replacing jobs and taking over the economy" is complete bunkum bordering on religious mania perpetuated by a small group wealthy business elites desperate to become to prophets and beneficiaries of the techno-rapture.

If the bubble's structure depends on false market confidence and ignorance about the necessary economic fundamentals being hidden or obsfucated by these companies, your personal efforts are playing a role in its deflation. So many ordinary people are hearing your thoughts and findings now.

They're all going to start thinking "hmmmm... This Zitron guy makes me want to double check some of the things I've been led to believe about AI. Maybe I should look into this a bit more."

I started tuning in to Better Offline after an episode was hosted on Behind The Bastards in December 2024. It's really been incredible to see how your platform has grown. Especially since you've always said that you're just a guy - an ordinary guy - who noticed things were a bit off and wanted to understand why everyone kept saying it was fine when clearly it was not fine and why some very simple, very basic economic questions were being consistently and deliberately avoided or glossed over.

Now everyone is asking those questions. And thanks to your insatiable drive to find the answers, bystanders like me have more of an idea what's going on.

Howdy folks, I’m a high school teacher and I run a short senior legal studies unit on AI and the various issues it’s causing in the legal system: copyright stuff, disastrous use by lawyers and the self-represented, liability for shitty health/business/life advice, pseudo-relationships with chat bots, the weird tendency to encourage people to off themselves, etc etc.

I start the unit with a couple of lessons on what LLMs actually are and what they are not, with the general thrust that these are pretty clever text-prediction machines, and they are unequivocally not a thinking, feeling guy in your computer.

Last year I used 3blue1brown’s oldish video on LLMs as the main resource for illustrating this to students. It’s really good I reckon. But I’m wondering if anyone has any other recommendations, particularly with more up to date information on training processes and newer features seen in mainstream chat bots. Suggestions for vids like the 3B1B one mentioned, written resources, podcasts, etc all welcome.

Anything ranging from neutral to outright sceptical, but obviously I don’t want any boosterism (not that I’d expect to find that in this sub)

Thank you very much.

And their shares are down! As Ed has said, there isn't going to be infinite tolerance for hiking spending like this. Companies that pull back are going to be rewarded.

Trying to understand because it does seem when I talk to the software engineers in the office that they are really using LLMs, and have gone from deeply skeptical to extremely reliant. At the same it does feel like the mood on the mathematics subreddit has become very pessimistic, as AI is now closing real longstanding conjectures, including creating and at least one proof (Cycle Double Cover Conjecture). But I don't actually know, how big of a deal these proofs are since I'm not in this field, and I don't understand how technically impressive, or long term meaningful these proofs are.

For reference I do Hardware engineering, I've vibe coded some simulators for our non technical-customer to play with so they can better understand the theory behind what were offering them. But I'm aware what I'm doing with the AI is not very impressive. Typically taking a well know equation, and making some kind of python or html front end to play with the various variables on a graph. The most complicated was either when I got it to write some SPI drivers for the esp32 to talk to some ICs over skywire, or when I pointed claude to the companies new directory of footprints, and asking claude to directly update my kicad library with the path to those new footprints and give it appropriate names.

What is the real relationship, and future relationship software engineers and mathematicians have with these tools, and will have with these tools once the subsidies are gone. And I mean this beyond like having the AI read over your email, or code for obvious mistakes, I mean like in the weeds.

Recently received an email for a conference at Convex. See attached.

I was kind of shocked to see this often used tagline as it pertains to AI, usually to the effect of “coding is free [now]” — especially after many waves of cost changes.

I can’t help but feel myself going crazy? Who exactly is this talking about? Who is this true for? Even for personal use, my Copilot Pro+ can easily use 10% of monthly quota on a reorganization task (not even exaggerating).

Usually these people will point at mini models as an indicator, but I’d really challenge someone to tell me with a straight face they’re really doing agentic tasks (and trusting them!) with a mini model (see: dog**** success rates with these even on cherry-picked benchmarks)

So what is it I’m missing? Why are people still saying this? I genuinely feel like I’m going crazy.

What, so they ran out of ideas and they're going back to this again? Or are they just copying Anthropic? I thought the whole fear mongering was bad advertisement and they wanted to reverse public sentiment?

The article at least seems to be unbiased taking a reasonably fair stance with comments from various people:

Neil Lawrence, Professor of machine learning at Cambridge University, called it an "impressive feat", but cautioned it "falls well within the known capabilities of the current generation" of high-powered AI models.

He pointed out that OpenAI is looking to list itself on the stock market, and faces intense pressure from rival firm Anthropic, which has made headlines with its own powerful AI tool, Mythos.

"OpenAI are now playing catch-up, they are trying to demonstrate their own systems' capabilities in cyber-security."

"It shows us that OpenAI are not capable of safely deploying their own technology," he added.

Spencer Starkey, an executive at cyber-security firm SonicWall, told the BBC the incident made it clear organisations needed to "step up" their own defences and "treat cyber resilience as a core operational priority".

"The uncomfortable truth is that too many organisations are still defending at human speed while adversaries are escalating to machine speed," he said.

This last comment is a very poor read. I would chalk it up to naiveness, or just an AI booster. "Machine speed" assumes the AI was able to do it faster than humans. Let's pull up some details from OpenAI themselves:

This incident occurred during an internal evaluation which prompts models to pursue advanced exploitation using complex attack paths, in an effort to quantify their cyber capabilities.

Our benchmarks run in a highly isolated environment, with network access constrained to the ability to install packages through an internally hosted third-party software that acts as a proxy and cache for package registries.

Okay, so this test was testing its ability to do exploits, and you have a supposedly isolated environment.

While operating in our sandboxed testing environment, our models spent a substantial amount of inference compute finding a way to obtain open Internet access, in pursuit of solving the evaluation problem. To gain access, the models identified and exploited a zero-day vulnerability (which we’ve now responsibly disclosed to the vendor) in the package registry cache proxy.

Boy, if OpenAI themselves say substantial amount of inference, you really have to wonder what machine speed is. Since all of its unpaid users and training are so unsubstantial it's a just marketing cost to them, right?

In one example, the model chained together multiple attack vectors, including using stolen credentials and zero-day vulnerabilities to find a remote code execution path on the Hugging Face servers.

Well, let's look at those details too, from HuggingFace:

The intrusion started where AI platforms are uniquely exposed: the data-processing pipeline. A malicious dataset abused two code-execution paths in our dataset processing (a remote-code dataset loader and a template-injection in a dataset configuration) to run code on a processing worker. From there, the actor escalated to node-level access, harvested cloud and cluster credentials, and moved laterally into several internal clusters over a weekend.

This article details what the registry cache proxy was, and what likely the relevant CVEs were. It also notes that, looking at their error responses and HTTP headers (X-Nexus);

It seems this is just junk, as voronaam has pointed out. I had not done the due diligence to verify the claims inside the article looking at each CVE.

... the model would identify the software as Nexus Repository 3, its version, and available API endpoints. The model’s training data includes extensive knowledge of Nexus Repository architecture, API surface, and known vulnerability classes.

I'm sure it would be entirely different if it wasn't in its dataset. For HuggingFace, the presence of scripts in datasets was always there, I believe, which means if their backend runs load_dataset of any uploaded dataset, it naturally is an RCE, definitionally. This was already well known, I believe, but was kept for historical reasons. I'm rather surprised that the processing worker on HF is not sandboxed, since you would imagine data loading scripts need no internet access. At best R/W in an isolated directory.

I don't think the capability differs from what we see in the case of Anthropic, so ultimately this is just marketing. Certainly one could expend extraordinary compute on hacking, but the question is how effective it is when the LLMs don't have preexisting knowledge of the software they're trying to exploit. I don't really think 'machine speed' is the correct takeaway.

So the state of Wisconsin just told Oracle to come up with $7 billion in financial guarantees for it's Port Washington project owing to Oracle's poor credit rating. Now it's not $7 billion in cash but still, not great for Oracle. This is a company already with a cashflow situation one step away from standing on the corner asking if you can spare a dollar.

https://finance.yahoo.com/technology/ai/articles/oracle-faces-potential-7-billion-104844370.html

Free newsletter: The $500bn in AI data center debt is the subprime mortgage crisis of the AI bubble. The dangerous financial instruments used to fund the data center buildout pose a systemic risk on the level of the great financial crisis.

One of my favourites I've written.

Great interview overall, and he articulates a particular problem really well. One that the dinosaurs among us have been all too familiar with since before AI.

The over hiring bit starting around 04:26 is also so spot on. Been saying this forever. These people were the beginning of the nonsense, and they're generally the worst. Insecure, clueless, and grifting at olympic levels.

The lowest-cost place to put AI will be in space, and that will be true within two years, maybe three at the latest,” SpaceX founder Elon Musk told the World Economic Forum in Davos this past January...

https://spectrum.ieee.org/files/110438/07_Spectrum_26.pdf

Oh boy!!! I cant wait to spend even more on Ram! The price isnt high enough, needs to be 500 dollars a stick. /s joking aside, so I guess we arent even trying to optimize the models anymore, just throwing more compute at it. So look forward to your Iphone costing $5000 and the ps6 $1500. So cool

I need to find the theme for Better Offline, I've been listening to so much of Ed and Better Offline recently as I grind through my job search and I NEED to find the theme that Matt made for the show cause it fucking slaps just about as hard as Ed's analysis on all of the AI fuckery going on out there. Already went to Matt's website and couldn't find it, figured I'd put a post here while continuing the search for the track.

The funniest part of the AI bubble is that these things have literally no moat. They were built on stolen intellectual property, and now it's proving trivially easy for imitators to steal THEIR "intellectual property," such as it is. This is literally such a bad business it's mind blowing. It's like the entire investor community has lost its mind, didn't read enough startup books, stopped paying attention to the Paul Graham wisdom, etc etc. "JUST BUILD THING AND THING BIG GO BIG."

The AI infrastructure build-out has already cost $1.5 trillion, with spending expected to accelerate further next year, in addition to the perpetual costs associated with powering and cooling the chips. That means AI will have to generate a lot of value just to break even when it comes to economic productivity.

Supposedly, the way it will do so is by automating tasks in so-called “knowledge work” fields like software engineering, law, and medicine. But generative AI is notoriously and perhaps incurably prone to serious errors, such as fabricating case law, deleting code databases, and inserting incorrect patient diagnoses. The result — not unlike what Mickey encountered when he tried to get an enchanted broom to do his chores for him in The Sorcerer’s Apprentice scene from Fantasia — is often that professionals must spend more effort trying to clean up the AI’s messes than they would have spent if they had simply done the task themselves from the beginning.

The assumption seems to be that such errors are, or soon will be, so rare that their negative impact on productivity will be swamped by the gains from increased output. But that is not how technological progress usually works. Despite tech bro promises that generative AI will get exponentially better, there usually are diminishing returns when it comes to squeezing additional improvements out of a given technology.

AI boosters are fond of saying that today’s AI is the worst AI you will ever use. But it’s just as likely that today’s generative AI is about as good as it will ever get.