On a new site with almost no links pointing at it, ClaudeBot and Googlebot reached nearly every new page on the day it went live. GPTBot and OAI-SearchBot reached 6 of 24 pages, and none launched after October 15. PerplexityBot reached one. Being fetched by one AI company's bot says little about the others.
I run shipfound.co, a young site. Since October 7, every request to it passes through a small server-side beacon that records the user agent and the path. Between October 7 and October 27 I shipped 24 new pages (answer pages, glossary pages and three free tools), so I could count how long each bot took to fetch each one for the first time.
What was measured
- The site: shipfound.co, a young Next.js site with few links pointing at it. It has a sitemap and an llms.txt file, and its robots.txt lets every AI bot in.
- The pages: 24 pages merged between October 7 and October 27. For each one, day 0 is the UTC day its pull request merged and the page went live.
- The visits: every request whose user agent names a known bot, read up to the end of October 27. Days are whole UTC days, so "day 0" means the same calendar day, not within 24 hours.
- What I did at launch: for most pages, I pressed Request indexing in Google Search Console and sent the URL through IndexNow on the day it went live or the day after. A few also went to Bing's URL submission.
Over the whole site in those three weeks, the beacon recorded 1,525 visits from 13 bots. ClaudeBot made 361 of them, Googlebot 257, OAI-SearchBot 207 and GPTBot 53.
How long each bot took to reach a new page
| Bot | New pages visited (of 24) | Median days to first visit | Fastest | Slowest |
|---|---|---|---|---|
| ClaudeBot | 23 | 0 | 0 | 3 |
| Googlebot | 22 | 0 | 0 | 1 |
| Google-InspectionTool | 19 | 0 | 0 | 1 |
| Amazonbot | 22 | 4 | 1 | 8 |
| Applebot | 16 | 4 | 1 | 9 |
| Meta-ExternalAgent | 19 | 8 | 0 | 12 |
| GPTBot | 6 | 1.5 | 1 | 6 |
| OAI-SearchBot | 6 | 3.5 | 0 | 5 |
| ChatGPT-User | 2 | 7.5 | 4 | 11 |
| Bingbot | 2 | 0.5 | 0 | 1 |
| PerplexityBot | 1 | 2 | 2 | 2 |
The only page ClaudeBot missed went live on October 27, the last day of data. Every page had at least one bot visit.
Same-day visits mostly follow a request
Most of the Google visits on day 0 are not discovery. For most of those pages I had asked Search Console to index the page that day, and Google-InspectionTool and Googlebot fetched it the same day.
ClaudeBot is the bot I asked nothing of. It still came on day 0 for most pages, usually once, and rarely came back to that page in the window. The site's sitemap.xml was fetched 208 times in three weeks, and ClaudeBot was among the bots reading it, which is the likeliest way it learned about each page so fast. Anthropic describes ClaudeBot as a bot for training data collection, so a fast ClaudeBot visit does not mean Claude can cite the page in search. Claude's search uses a curated web search, and the separate Claude-SearchBot is the one used for indexing for Claude search. Claude-SearchBot did not appear on these pages at all.
The OpenAI bots and PerplexityBot barely came
This is the part that surprised me. GPTBot reached 6 of the 24 pages and OAI-SearchBot reached 6. All of OAI-SearchBot's pages went live on or before October 12, and GPTBot's last new page went live on October 15. Pages shipped after that got nothing from either bot, even though both kept visiting the site: OAI-SearchBot spent its visits on the home page, robots.txt and a few older pages.
That matters because the two do different jobs. GPTBot is used by OpenAI for training data collection. OAI-SearchBot is used for indexing for ChatGPT search, so a page it has never fetched is unlikely to be a page ChatGPT search can cite. ChatGPT also leans on its own index plus Bing, and here Bingbot fetched only 2 of the 24 new pages, even though most of them had been sent through IndexNow.
PerplexityBot fetched one new page, two days after launch. Perplexity runs live web search on every query, so a missing PerplexityBot visit says less about Perplexity than a missing OAI-SearchBot visit says about ChatGPT. It may fetch the page when someone asks something it answers.
The likeliest explanation is crawl budget. Crawlers have a set number of pages they want to crawl per domain, usually determined by the popularity of the domain, and unknown websites will get very few crawls per day. A site this young is about as unknown as it gets. I can't tell from this data whether the budget will grow, only that it hadn't by October 27.
Visits from people asking
ChatGPT-User and Claude-User are not crawlers in the usual sense. They fetch a page when someone asks the assistant about it. Across the whole site they made 93 visits in three weeks, but ChatGPT-User reached only 2 of the new pages, on days 4 and 11, and Claude-User none. For a new site, that is the closest thing in the logs to someone finding a page through an AI answer, and it is rare.
What I would do differently on a new site
- Treat each bot as its own channel. A fast visit from ClaudeBot or Googlebot tells you nothing about OAI-SearchBot. Check each bot's own first visit to the page.
- Don't stop at notifying. Notification is not indexing, indexing is not ranking, ranking is not an AI citation. IndexNow can immediately notify participating search engines that a page exists, and here that notice did not bring Bingbot to most pages within the window.
- Rule out a block first. Many sites allow AI crawlers in robots.txt but block them at the CDN, bot manager or firewall, and a blocked bot will never receive the page's HTML. None of the requests here got an error, so a block isn't the reason OAI-SearchBot stayed away.
- Get the domain known. New domains begin with very little independent history, so links and mentions elsewhere are what move a site up the crawl queue.
- Expect a slow start. A site can be discovered in hours or days by some systems, while Google indexing and meaningful AI citations may take longer. That matches what I saw.
Limits of this data
This is one site over three weeks, so treat it as one data point, not a benchmark. Days are counted in whole UTC days. User agents are as reported: nothing here checks the requests against each company's published IP ranges, so a faked user agent would be counted as the bot it claims to be. Before October 23, Shipfound's own site checks also fetched pages with AI bot user agents and were counted; since then they carry a tag that keeps them out. Those checks send several AI user agents at once, and no new page shows GPTBot, OAI-SearchBot, PerplexityBot and ClaudeBot arriving on the same day, so they are unlikely to be in these numbers. The site is mine, and the beacon is part of Shipfound, the product I'm building.
Frequently asked questions
How does AI crawl websites?
Each AI company runs its own bots, and they do different jobs. Some collect training data, such as GPTBot and ClaudeBot. Others index pages for an assistant's search, such as OAI-SearchBot for ChatGPT search and PerplexityBot for Perplexity. Others fetch a page when a person asks about it.
Which AI crawlers should I look for in my logs?
At minimum: GPTBot, OAI-SearchBot and ChatGPT-User for OpenAI, ClaudeBot, Claude-SearchBot and Claude-User for Anthropic, and PerplexityBot for Perplexity. On this site Amazonbot, Applebot and Meta-ExternalAgent also visited most new pages within two weeks.
How do I get my website to show up in AI searches?
Make sure the search bots can read it, then make the site known enough to be crawled often. Pages shaped as a question-shaped heading, a one-sentence direct answer, then evidence are the format the engines tend to quote. Then check that you are named, because a crawl is not a citation.
Do AI crawlers respect robots.txt?
Reports differ. One site administrator saw PerplexityBot looking at robots.txt and ignoring it. Read your own logs: the beacon here saw robots.txt fetched 446 times in three weeks, more than any page.