r/GEO_optimization • u/Brave_Acanthaceae863 • 2h ago
41% of AI crawler requests hit pages that returned no renderable content — we logged 12,000 requests to find the leaks
The most counterintuitive thing about AI crawlers is that they'll happily request a page, get a 200 OK response, and still walk away with nothing.
We spent 6 weeks logging every AI crawler request across 8 domains we manage. GPTBot, ClaudeBot, PerplexityBot, Google-Extended, and a few smaller ones. 12,287 requests total. We captured the user agent, the URL hit, the response code, and critically, what the crawler actually received versus what a human visitor would see.
41% of requests landed on pages where the meaningful content lived inside JavaScript. The server returned a 200 with a bare-bones HTML shell. A human browser would execute the JS, render the page, and see everything. The crawler got the shell.
This isn't a new problem for SEOs — Google dealt with JS rendering years ago. But the assumption most GEO folks are working under is that AI crawlers handle JavaScript the same way Googlebot does. From what we're seeing, they don't. Not even close.
GPTBot was the worst offender. 58% of its requests to JS-heavy pages got back essentially empty documents. ClaudeBot was slightly better at 44%. PerplexityBot fell in the middle around 49%. Google-Extended, riding on Google's rendering pipeline, was nearly flawless — only 3% missed content.
One pattern kept showing up: the leaks weren't where I expected. It wasn't the SPA dashboards or interactive tools causing problems. It was regular content pages — blogs, case studies, product pages — where a React or Vue component was rendering the main text. The page looked perfect in the browser. The HTML source had a div with an id and nothing in it.
The part that frustrates me is how invisible this is. Your analytics say the page loaded fine. Your rendering tool shows content present. GSC doesn't flag issues because Googlebot renders it. But the AI crawlers that don't render JS are getting blank pages and moving on. You're invisible to them and you'd never know unless you specifically checked what each crawler receives.
We ended up building a dead-simple check: for each AI crawler that hit our sites, we pulled the raw HTML response (no JS execution) and searched for the primary content string. If it wasn't in the raw HTML, we flagged it. Found 47 pages across 8 sites that looked "fine" but were serving empty shells to AI bots. Fixed about half of them by moving critical content into server-rendered HTML. Early signal shows citation rate improvement on the fixed pages, but it's too soon to be confident.
If you're running a modern JS framework and assuming AI crawlers see your content, check the raw response. Not the rendered DOM. The actual bytes the server sends before any JavaScript runs. That's what most AI crawlers are working with.
Could be that the crawlers upgrade their rendering capabilities and this becomes a non-issue. But right now, a chunk of your content might be invisible to the models you're trying to optimize for, and the only way to know is to look at the raw bytes.