shipfound

Answer

How do I know if AI crawlers are hitting my site?

Updated 2026-11-02

Look at your server, not your analytics. AI crawlers do not run JavaScript, so Google Analytics never records them. Your server logs, or a beacon that runs on the server, show each request with its user agent. Filter for GPTBot, OAI-SearchBot, ClaudeBot and PerplexityBot to see which bots came and which pages they read.

How to check for GPTBot or ClaudeBot, step by step

  1. List every name the company uses. OpenAI runs GPTBot, OAI-SearchBot for ChatGPT Search and ChatGPT-User for pages a person asks about. Anthropic runs ClaudeBot, Claude-SearchBot and Claude-User. Search for all of them, not one.

  2. Search the logs that see the request first. If a CDN sits in front of your server, start with its edge logs, because an origin server often only sees requests that bypassed the CDN cache. On Nginx or Apache, this shows today's visits from OpenAI and Anthropic:

    grep "$(date +%d/%b/%Y)" access.log | grep -E "GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-SearchBot|Claude-User"
    
  3. Read the status code on each hit. A 2xx means the bot got the page, a 3xx means it hit a redirect, and a 403, 429 or 5xx means it was denied, rate-limited or failed. A request for /robots.txt alone means the bot read your rules, not that it fetched a page.

  4. Confirm the bot is real. A user agent is self-reported and easy to fake, so confirm it with a reverse DNS lookup.

  5. If a bot never shows up, rule out a block. A line such as User-agent: GPTBot followed by Disallow: / keeps it out.

Without reading raw logs

A server-side beacon does the same job. If you build with Claude Code or Codex, Shipfound installs one and shows whether GPTBot, PerplexityBot and ClaudeBot crawl your pages. If you do not use either agent, your logs answer the question just as well.

A robots.txt checker such as the Cloudflare AI Bot Visibility Checker reports each crawler as allowed, blocked or not specified. That is configuration only, so use it to rule out a block and your logs to see traffic. For tools that do this for you, see the best AI crawler SEO tools.

Read the bot names before you act

How soon each bot reaches a new page varies; one site's numbers are in how fast AI crawlers find a new page.

Frequently asked questions

Does blocking GPTBot stop my pages appearing in ChatGPT?

No. OAI-SearchBot builds ChatGPT's search index and ChatGPT-User fetches pages live, so blocking GPTBot alone still leaves you answerable. Opting out of OAI-SearchBot is different: those sites are not shown in ChatGPT search answers. To let the search bots in, see how to get indexed by PerplexityBot and GPTBot.

How can I check if my website is crawlable by AI bots?

Two checks. A robots.txt checker shows which AI bots your rules allow or block. Your logs show whether they got through: a named AI user agent with a 2xx response reached a layer that returned the page.

How do I stop AI crawlers?

Disallow them by name in robots.txt. Disallowing GPTBot tells OpenAI your content should not be used for training, and OpenAI says a robots.txt change can take about 24 hours to apply. Blocking a training bot generally does not remove you from AI search answers, with one exception: Google-Extended also affects Gemini grounding.

How do I check the AI visibility of my website?

Crawler hits show that a page was read, not that it was cited. To see whether you are named, ask AI engines your buyers' questions more than once, because answers change between runs. Shipfound does this and counts how often you are named. More in how to track if ChatGPT mentions your product.