The syntax is the same for every AI bot: a User-agent line with the bot's name, then Allow and Disallow paths. What changes is whether the bot obeys. OpenAI's and Anthropic's crawlers follow robots.txt, and Anthropic also honors Crawl-delay. Some user-initiated fetchers and Bytespider do not. llms.txt allows and blocks nothing.
So the useful question is not "what syntax does GPTBot understand" but "which name do I write, and will that bot listen". The table below answers both, and the sections after it go company by company.
Every major AI crawler at a glance
| Name in robots.txt | Company | What it does | Follows robots.txt | How to verify it |
|---|---|---|---|---|
| GPTBot | OpenAI | Training | Yes | openai.com/gptbot.json |
| OAI-SearchBot | OpenAI | ChatGPT search | Yes | openai.com/searchbot.json |
| ChatGPT-User | OpenAI | Fetches a page for a user | Sources disagree (below) | openai.com/chatgpt-user.json |
| ClaudeBot | Anthropic | Training | Yes, including Crawl-delay | claude.com/crawling/bots.json |
| Claude-SearchBot | Anthropic | Search inside Claude | Yes | claude.com/crawling/bots.json |
| Claude-User | Anthropic | Fetches a page for a user | Yes | claude.com/crawling/bots.json |
| PerplexityBot | Perplexity | Perplexity's crawler | Yes, with a caveat | perplexity.com/perplexitybot.json |
| Perplexity-User | Perplexity | Fetches a page for a user | No | perplexity.com/perplexity-user.json |
| Google-Extended | AI training control | A token only, no crawler of its own | Nothing to verify | |
| Applebot-Extended | Apple | AI training control | Separate from Applebot | |
| CCBot | Common Crawl | Web archive | Yes | Reverse DNS to crawl.commoncrawl.org |
| Bytespider | Not documented | Crawler | Documented non-compliance | No published list |
Googlebot is not on this list on purpose. You do not need to touch it to control AI training, because Google handles that through Google-Extended (more on that below).
Why one company runs several bots
Each AI company splits its traffic by job. One bot collects pages to train future models. Another builds the index its assistant searches when it answers a question. A third fetches a single page right now, because a person pasted a link or asked about it.
That split is why a single "block AI" rule rarely does what people want. Disallow the training bot and the search bot can still read your pages and cite them. Disallow the search bot and you drop out of that assistant's answers while your pages may still end up in training data through someone else's crawler. You write one group per name, so you decide per job.
It also explains the exceptions. A user-initiated fetch is closer to a person clicking a link than to a crawler walking your site, and some companies treat it that way.
OpenAI: GPTBot, OAI-SearchBot and ChatGPT-User
GPTBot is the training crawler. Its full user agent is Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.3; +https://openai.com/gptbot), and it honors robots.txt. According to OpenAI, disallowing it signals that your content should not be used to train foundation models.
OAI-SearchBot is the one that decides whether you can show up in ChatGPT search. It is the crawler behind ChatGPT search answers, launched October 31, 2024, and it honors robots.txt. A rule change for it applies in about a day, so do not expect the effect within the hour.
ChatGPT-User is where the sources split. One guide lists it as honoring robots.txt. Another says it does not respect robots.txt since it is a direct user action. Plan for the second: if a page must stay private, put it behind a login, not a Disallow line.
The syntax for all three is plain robots.txt. Name the bot, then the paths:
User-agent: OAI-SearchBot
Allow: /
User-agent: GPTBot
Disallow: /
Anthropic: ClaudeBot, Claude-SearchBot and Claude-User
Anthropic runs the same split. ClaudeBot is the training crawler, with the user agent Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com). Claude-SearchBot is the in-product search crawler. Claude-User is the user-initiated fetcher. All of them honor robots.txt.
Anthropic goes further on the details. It honors Disallow and Crawl-delay and will not bypass CAPTCHAs. Crawl-delay matters if ClaudeBot is heavy on a small server: it is the one directive here that slows a bot down without shutting it out. The sources for this page document Crawl-delay only for Anthropic's bots, so do not count on it elsewhere.
If you want Claude to cite you but not train on you, allow Claude-SearchBot and Claude-User and disallow ClaudeBot. They are separate names, so each gets its own group.
Perplexity: PerplexityBot and Perplexity-User
PerplexityBot identifies itself as Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot). It respects robots.txt, but with a stealth crawler caveat.
The caveat comes from Cloudflare. On August 4, 2025, Cloudflare published a report showing Perplexity using undeclared crawlers that rotate user-agents, IPs and ASNs to evade no-crawl directives. In other words, a Disallow for PerplexityBot stops PerplexityBot, and the report says other, unnamed traffic from the same company may not stop with it.
Perplexity-User is the live fetcher. It claims to be an agent, not a bot, and does not respect robots.txt. If you want to block it, robots.txt is the wrong tool. Block its published IP ranges at your server or CDN instead.
Google-Extended and Applebot-Extended: tokens, not crawlers
This is the part people get wrong most often. Google-Extended is a robots.txt-only control token, not a crawler with its own HTTP requests. You will never see it in your logs. It is a name Google reads in your robots.txt, and nothing fetches pages under it.
So does it need special syntax? No. It is written like any other user agent: a User-agent line with the token, then Disallow or Allow. What makes it different is the effect, not the format. Google states that Google-Extended does not affect Search inclusion or ranking, so disallowing it leaves Googlebot and your rankings alone.
User-agent: Google-Extended
Disallow: /
Apple has a similar split. Applebot-Extended is distinct from standard Applebot, and blocking it does not affect Apple Search or Spotlight.
One newer Google name behaves differently. Google-Agent may ignore robots.txt rules generally, the same pattern as the other user-initiated fetchers.
CCBot and Bytespider
CCBot is Common Crawl's crawler. It respects robots.txt, so a Disallow group for it works like one for GPTBot. It is easy to forget because it does not belong to an AI company.
Bytespider is the hard case. It has documented non-compliance with robots.txt, and it does not publish a dedicated crawler documentation page or IP verification list, and site owners report it continuing to fetch disallowed paths. A Disallow line costs nothing, so write one, but the block that works is at the server or CDN. How the two layers fit together is in what are AI crawler rules.
How to check a bot is who it says it is
A user agent is a string anyone can send. Before you trust a "GPTBot" line in your logs, or before you block one, match the request's IP against the company's published list.
- OpenAI publishes one file per bot: openai.com/gptbot.json, openai.com/searchbot.json and openai.com/chatgpt-user.json.
- Anthropic publishes its crawler IPs at claude.com/crawling/bots.json. The sources here give one file for Anthropic, not one per bot, so match the IP first and then read the user agent to tell ClaudeBot, Claude-SearchBot and Claude-User apart.
- Perplexity publishes perplexity.com/perplexitybot.json and perplexity.com/perplexity-user.json.
- Common Crawl is checked by reverse DNS: legitimate CCBot requests resolve to [ip].crawl.commoncrawl.org.
Then look at what each bot actually fetched. Bots do not run analytics scripts, so this has to come from server logs or a server-side counter. How to tell if AI crawlers are hitting your site covers where to look. If you build with Claude Code or Codex, Shipfound records whether GPTBot, PerplexityBot and ClaudeBot crawl each page.
Where llms.txt fits
llms.txt is not an access rule. It is a Markdown file that describes your site and points to your best pages. It has no Allow, no Disallow and no user agents, so it cannot let a bot in or keep one out. None of the crawler rules above mention it.
That makes the answer to the llms.txt half of the question short: there is no llms.txt syntax that any crawler on this page obeys as a directive. If you want to control access, robots.txt and your CDN are the tools. Whether llms.txt is worth writing anyway is a separate question, answered in is writing an llms.txt file worth it for SEO.
A robots.txt that follows these rules
For a site that wants to be cited by AI search but kept out of training, this covers the names above. Several User-agent lines can share one group, which keeps the file short:
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
Allow: /
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
User-agent: Bytespider
Disallow: /
User-agent: *
Allow: /
Sitemap: https://yoursite.com/sitemap.xml
Before you ship it, read your live robots.txt at yoursite.com/robots.txt, not the file in your repo, since a host or plugin can rewrite it. And remember what this file cannot do: it will not stop Perplexity-User, it may not stop ChatGPT-User, and it will not stop Bytespider. Those need a server or CDN rule. Rules are also only one part of site crawlability; if a search bot is allowed and still never comes, see how to get indexed by PerplexityBot and GPTBot.
Frequently asked questions
Which robots.txt rules block crawlers?
A User-agent line naming the bot, followed by Disallow and a path. Disallow with a single slash blocks the whole site for that bot. It only works on bots that choose to follow robots.txt, which rules out Perplexity-User and Bytespider.
What is the correct syntax for robots.txt files?
Groups of lines. Each group starts with one or more User-agent lines, then the Allow and Disallow paths that apply to them. A User-agent of * covers every bot not named elsewhere. A Sitemap line with the full URL can go anywhere in the file. The file lives at the root of the domain.
Is robots.txt legally binding?
The file itself is a request, and nothing in it stops a bot that ignores it. Cloudflare's research on Perplexity's undeclared crawlers shows what that looks like in practice. Sites that need a hard block put it at the server or CDN. Whether ignoring robots.txt has legal consequences depends on where you are, and is a question for a lawyer.
What is robots.txt for AI?
The same robots.txt, with groups for AI bots by name. You can allow the search bots, such as OAI-SearchBot and Claude-SearchBot, and decide separately about training tokens such as Google-Extended.