Look at your server, not your analytics. AI crawlers do not run JavaScript, so Google Analytics never records them. Your server logs, or a beacon that runs on the server, show each request with its user agent. Filter for GPTBot, OAI-SearchBot, ClaudeBot and PerplexityBot to see which bots came and which pages they read.
Where to look
Your server or CDN logs are the full record. Crawlers never fire analytics tags, so the logs are where every AI crawler request shows up. Search the user agent field for the bot names below.
A server-side beacon does the same job without reading raw logs. It runs on your server, sees each request as it arrives, and records the ones from AI crawlers. If you build with Claude Code or Codex, Shipfound installs one and shows whether GPTBot, PerplexityBot and ClaudeBot crawl your pages. If you do not use either agent, your logs answer the question just as well.
A robots.txt checker answers a different question. The Cloudflare AI Bot Visibility Checker reports each crawler as allowed, blocked or not specified, but it reports configuration only and cannot tell you whether a page was used by any AI product. Use it to rule out a block, then use your logs to see traffic.
Read the bot names before you act
AI bots do different jobs, and the name tells you which.
- Search bots such as OAI-SearchBot, PerplexityBot and Claude-SearchBot fetch pages so an assistant can answer a question and link to its sources.
- Training bots such as GPTBot, ClaudeBot and Google-Extended collect content to train future models.
- ChatGPT-User fetches a page when a person asks about it. It does not crawl the web automatically, and robots.txt rules may not apply to it.
Do not trust the label on its own. A user agent is self-reported and easy to fake, so confirm it with a reverse DNS lookup. On one measured site, 9,162 of 22,966 requests could not be tied to a named agent.
Frequently asked questions
Does blocking GPTBot stop my pages appearing in ChatGPT?
No. OAI-SearchBot builds ChatGPT's search index and ChatGPT-User fetches pages live, so blocking GPTBot alone still leaves you answerable. Opting out of OAI-SearchBot is different: those sites are not shown in ChatGPT search answers.
Why doesn't Google Analytics show AI crawlers?
Its tag fires from JavaScript, and crawlers do not execute JavaScript. The requests happen, but the tag never runs.
How do I stop AI crawlers?
Disallow them by name in robots.txt. Disallowing GPTBot tells OpenAI your content should not be used for training, and OpenAI says a robots.txt change can take about 24 hours to apply. Blocking a training bot generally does not remove you from AI search answers, with one exception: Google-Extended also affects Gemini grounding.
How do I check the AI visibility of my website?
Crawler hits show that a page was read, not that it was cited. To see whether you are named, ask AI engines the questions your buyers ask, more than once each, because answers change between runs. Shipfound runs those questions repeatedly and counts how often your product is named.