AI crawler rules are the instructions a website gives AI bots about what they may fetch and how they may use it. They live in your robots.txt file and, more and more, in your CDN's bot settings. They matter because one rule can keep you out of AI search answers, or hand your content to model training you never agreed to.
How AI crawler rules work
The oldest layer is robots.txt. You name a bot by its user agent and allow or disallow paths for it. A well-behaved crawler reads the file before it fetches anything else. The file is a request, not a lock: it only works on bots that choose to follow it.
The second layer is your CDN or firewall, which can block a bot outright. Cloudflare now sorts AI traffic into three categories, Search, Agent and Training, with controls for each, and the feature is open to all Cloudflare customers, including the Free plan.
The categories map to what you probably care about. Search bots fetch pages to answer a question and cite you. Agent bots act for a user, such as a browsing assistant. Training bots collect pages to build future models. Most sites want the first and have opinions about the last.
Rules about use, not only access
Allow or block is a blunt choice. Cloudflare is extending its Content Signals format in robots.txt with a new use parameter, so a site can say what a bot may do with a page after reading it. The three levels are Immediate (no storage or reuse), Reference (indexing, excerpts and links back) and Full (summaries or reproduction).
Reference is the level that fits most businesses that want to be cited: the bot may index the page and quote it, with a link back.
There is some enforcement behind it. Verified Bots that ignore content use preferences or reproduce content in full may lose their Verified status. And being verified no longer means being let in. Previously all Verified Bots were allowed by default; under the new model, verification confirms only a bot's identity.
An example
A software company wants ChatGPT and Perplexity to cite its docs but does not want its pages used for training. In robots.txt it allows the search bots of each AI company and disallows their training bots, by name. On Cloudflare it sets Search to allowed, Training to blocked, and leaves Agent allowed so assistants browsing for a user can read the pricing page. The bot names differ by company; how to get indexed by PerplexityBot and GPTBot lists which bot does which job, and how AI crawlers handle text, video, images and structured data covers what they read once they are in.
How to check your rules
Read your live robots.txt at yoursite.com/robots.txt, not the file in your repo, since a host or plugin can rewrite it. Then open your CDN's bot settings and compare the two: a bot allowed in robots.txt but blocked at the firewall still never gets in.
Then check what actually happens. Bots do not run analytics scripts, so you need server logs or a server-side counter to see which AI crawlers fetch which pages. More on that in how to know if AI crawlers are hitting your site. If you build with Claude Code or Codex, Shipfound checks your AI crawler rules and counts GPTBot, PerplexityBot and ClaudeBot visits per page.
Common mistakes
Blocking training and losing search. Some crawlers do both jobs. On Cloudflare, multi-purpose crawlers are judged under both policies, so blocking Training also blocks Googlebot, Applebot and BingBot, even with Search allowed. Check this before you flip the switch.
Trusting the defaults. From September 15, 2026, new domains on Cloudflare block Training and Agent crawlers by default on pages that show ads, while Search crawlers stay allowed. If you set up a new domain, read what it does before assuming you are open.
Blocking everything with one wildcard rule. A blanket disallow meant to stop scrapers also stops the bots that would have cited you.
Frequently asked questions
How does an AI crawler work?
It fetches pages like a search engine bot, then hands the text to an AI system, either to answer a question now (search), act for a user (agent) or build a future model (training). Your rules can treat each job differently.
Why block AI crawlers?
Usually to keep content out of model training, or to cut server load. The cost is that a blocked search bot cannot cite you, so most sites block training and allow search.
Do AI crawlers follow robots.txt?
The well-known ones say they do, but robots.txt cannot force anything. That is why CDNs added blocking, and why Cloudflare ties Verified status to respecting content use preferences.