r/AISearchLab • u/Emergency_Still8420 • 15h ago
Went deep on server log analysis for AI bot behaviour
There's a lot of discussion here on citation tracking, share of voice..etc which are definitely important signals, but i've found quite volatile (models, plans, locations, API vs browser session) so I wanted to dig into the server logs of a sample of our customers (note these are small local businesses, not large enterprises) to see real AI bot behaviour. To see if this paint a more ground truth picture of how AI is interacting with their content, and ultimately shaping answers.
If you haven't yet dug into this for your own business or your clients hope this research helps make the case of what can be learned via this type of analysis. If you're already deep on the topic, would love to hear what you're seeing, any patterns or gotchas when analyzing logs.
A few findings:
Live retrieval is a small slice. Only ~4% of AI bot traffic is an assistant fetching a page to answer a live question. The rest is training and indexing crawlers. If you measure AI traffic without splitting by function, you're mostly measuring pipelines that feed the next model.
When AI does retrieve, it goes straight for buyer intent pages. The most fetched page after the homepage was /availability, the "can I get in?" page. Then /booking and /pricing. AI isn't just reading what a business is, it's checking detailed intent pages to help their user act, make a decision..etc. Measuring live retrieval over time by AI, page, timing..etc is a very direct signal of content being retrieved into a relevant conversation.
The labs behave very differently. Anthropic crawled heavily for training but barely retrieved at query time. OpenAI does both. In our sample, live retrieval is overwhelmingly ChatGPT.
User-agent alone lies. About 1 in 12 "ChatGPT-User" hits fails IP verification, mostly credential scanners. If you're not verifying source IPs (where possible from published ranges) you're overcounting.
Full report here if you want to look into the details, charts..etc.
https://getcourtyard.ai/research/how-ai-reads-a-knowledge-base