Allow the search crawlers, not just the famous one. For ChatGPT, the bot that matters is OAI-SearchBot: it is used to surface websites in ChatGPT's search results, while GPTBot crawls content that may be used to train OpenAI's models. For Perplexity, allow PerplexityBot. Then make sure nothing between the bot and your page turns it away.
Which bots to allow
OpenAI runs separate crawlers for separate jobs. Sites that opt out of OAI-SearchBot will not be shown in ChatGPT search answers, and disallowing GPTBot says your content should not be used for training. You can allow one and block the other. Training bots send no users and produce no citations, so if you only want to be found, OAI-SearchBot is the one you cannot block.
On Perplexity's side, PerplexityBot respects robots.txt according to Perplexity's documentation. Its live fetcher is different: Perplexity-User may fetch a URL a user supplies even if robots.txt disallows it.
A robots.txt that allows both search bots looks like this:
User-agent: OAI-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
Add a Sitemap line with your sitemap's full URL, and check the Disallow lines too. A stray forward slash on a Disallow line will block that bot.
If they still do not crawl
Look past robots.txt. Bot protection on your CDN or host can reject a crawler that robots.txt allows. Serve the content in the HTML, because crawlers will bail if they wait too long for JavaScript to load it. Publish a sitemap so they have a list of your pages. After a robots.txt change, OpenAI says it can take about 24 hours for its systems to adjust.
Then confirm the visits. Your server logs show each bot by name, and how to tell if AI crawlers are hitting your site covers where to look. If you build with Claude Code or Codex, Shipfound records whether GPTBot, PerplexityBot and ClaudeBot crawl each page.
Frequently asked questions
Does being crawled mean I will be cited?
No. There is no organic way to guarantee that a site will show up in ChatGPT and Perplexity. Crawling makes you eligible. To see whether you are named, track how often ChatGPT mentions your product.
Can robots.txt block indexing?
It blocks crawling by the bots that respect it. Opting out of OAI-SearchBot keeps you out of ChatGPT search answers, though you can still appear as navigational links. Rules may not apply to ChatGPT-User, because its actions are started by a user.
Which AI bots visit most?
On one site over fourteen days, 3,392 fetches came from 13 distinct AI crawlers. Live agent bots such as ChatGPT-User and Perplexity-User were 44.3% of hits, and training crawls were 29.6%.