Start with Cloudflare AI Crawl Control if your site runs on Cloudflare. It is free and shows which AI bots visit. Add Known Agents for bot analytics anywhere else, and Semrush or Screaming Frog to audit what those bots can read. The right pick depends on whether you need to see, control or fix.
What an AI crawler tool does
General SEO tools were built around Googlebot. AI crawlers bring three new questions, and most tools answer only one or two of them.
The first is who is visiting. ChatGPT alone sends several bots that do different jobs. OAI-SearchBot decides whether you can appear in ChatGPT search, and sites that block it are not shown in ChatGPT search answers. GPTBot collects content that may train OpenAI's models, and you can allow OAI-SearchBot while disallowing GPTBot. Google works another way: Google-Extended is a robots.txt token with no user agent of its own, and it does not affect inclusion or ranking in Google Search.
The second is whether you let them in. That lives in robots.txt and your firewall, and a single line can hide you from an AI search engine.
The third is what they can read once inside. Most AI crawlers read only a page's basic HTML, not content loaded by JavaScript. A page that looks fine in a browser can be close to empty for them.
The tools at a glance
| Tool | Best for | What it does for AI bots | Starting price |
|---|---|---|---|
| Cloudflare AI Crawl Control | Sites already on Cloudflare | Shows AI crawler visits, allow or block each one | Free, on all plans |
| Known Agents | Bot analytics on any stack | Real-time bot visits, automatic robots.txt rules | Free plan, then $29/month |
| Semrush Site Audit | Auditing AI access and llms.txt | Flags pages blocked from AI search bots | $139/month |
| Screaming Frog SEO Spider | Seeing pages as a bot sees them | Crawls as an AI crawler, compares raw and rendered HTML | Free to 500 URLs, then £199/year |
| Ahrefs | Teams already paying for Ahrefs | Bot Analytics beta, Site Audit robots checks | Ahrefs Free; Bot Analytics on paid plans |
| Profound Agent Analytics | Large sites with server logs | Bot intent from logs, spoofed bots filtered out | Custom |
| Scrunch | Teams tracking AI traffic by bot type | Splits visits into training, indexing and retrieval | $250/month |
| Shipfound | Founders building with Claude Code or Codex | Counts AI bot visits and ships the fixes as pull requests | Free plan, then $49/month |
The list
1. Cloudflare AI Crawl Control
The easiest first step if Cloudflare already sits in front of your site. AI Crawl Control, formerly AI Audit, shows which AI services access your content and gives you tools to manage that access. It works on all Cloudflare plans with no configuration.
You get a breakdown of requests by individual crawler, such as GPTBot and ClaudeBot, and an Allow or Block choice for each AI crawler. Its Directives tab lists AI crawlers that requested paths your robots.txt disallows, which tells you who is ignoring your rules.
The limit: on the free plan it identifies AI crawlers by their user agent strings, and a user agent can be faked.
2. Known Agents (formerly Dark Visitors)
The pick when you are not on Cloudflare, or want bot analytics as its own product. Agent Analytics lets you see every crawler, scraper and AI agent visiting your website in real time, and it covers more than 2,000 crawlers, scrapers and AI agents across more than 15 categories.
Its Automatic Robots.txt writes a section of your robots.txt with rules for all of the bots in the categories you pick. You choose categories once, and you stop keeping a list of bot names yourself. The Free plan costs $0 a month for 1 project and 100K events, and Startup costs $29 a month for 3 projects and 1M events.
3. Semrush Site Audit
A good fit for checking AI access if you already pay for Semrush. Its Blocked from AI Search check shows how many of the pages in your audit are blocked from AI search engines, bot by bot, through robots.txt. Site Audit also flags missing or incorrect llms.txt files.
It reads your rules. It does not show you live bot visits. Site Audit comes with the SEO plan at $139 a month, or $117.33 a month billed annually.
4. Screaming Frog SEO Spider
The tool for seeing your site the way a bot sees it. You can crawl as a search bot, as an AI crawler, or with your own user agent, and the switch also changes which robots.txt directives the crawl follows.
It can also render pages in an integrated Chromium to crawl JavaScript-heavy sites built with frameworks like React and Vue. Crawl once without rendering and once with it, and the gap between the two is roughly what a non-rendering AI bot misses.
The free version crawls up to 500 URLs, and JavaScript rendering is a paid feature. A licence costs £199 per year.
5. Ahrefs
Two pieces, for teams already inside Ahrefs. Bot Analytics, in beta, has a toggle that isolates AI bot traffic across all reports. It is free during beta on all paid Ahrefs plans.
On the audit side, Ahrefs Webmaster Tools is now Ahrefs Free, and Ahrefs' own course says Site Audit will flag any robots.txt rules that might be blocking AI crawlers. Site Audit also executes JavaScript. The Lite plan costs $129 a month.
6. Profound Agent Analytics
Built for larger sites that can send server logs. Profound uses server logs instead of JavaScript trackers for AI bot analysis, and it classifies whether AI bots are indexing, training on or citing your content. It also tracks when, how often and which AI bots access your content.
There is no self-serve paid price on the page. The free trial runs for 7 days, and beyond that you talk to sales.
7. Scrunch
A middle option with a listed price. Agent Traffic separates bot traffic into training, indexing and retrieval and breaks down visits from ChatGPT, Claude, Gemini, Perplexity and more. The Core plan costs $250 a month.
8. Shipfound
For founders who build with Claude Code or Codex and would rather have the fixes than a report. It runs inside your coding agent, counts AI bot visits server side, and ships the changes as pull requests you review and merge. That covers robots rules, an llms.txt file and pages that answer buyers' questions. It is a narrower tool than the others here: if your team does not work in a repo with one of those agents, a tool higher on this list fits better.
Free checks worth running first
You can learn a lot before paying for anything.
Chrome's Lighthouse has an llms.txt audit in its Agentic Browsing category. That category reports a pass ratio instead of a 0 to 100 score.
Google Search Console's Crawl Stats report shows statistics about Google's crawling history on your website. It is about Google's own bots, so it will not tell you what GPTBot or ClaudeBot did.
And read OpenAI's crawler page before you touch robots.txt. A robots.txt change can take about 24 hours to reach ChatGPT search.
Which tools watch, and which change your site
Most of this list watches. Cloudflare and Known Agents count visits and write or enforce rules. Semrush, Screaming Frog and the Ahrefs audit read your site and report what is wrong. Profound and Scrunch analyse bot traffic for larger teams.
Knowing is half of it. If the audit says your pricing page is empty without JavaScript, someone still has to change the code. A practical setup pairs one tool that watches with one that audits, and a clear owner for the fixes.
How to pick
Pick by the question you need answered today.
If you do not know whether AI bots visit at all, start with visit analytics: Cloudflare if you are on it, Known Agents if not. Both have free plans.
If bots visit but you never show up in AI answers, audit access next. Run Semrush's Blocked from AI Search check, or crawl as an AI bot in Screaming Frog and compare the result with a rendered crawl.
If you have server logs and a budget, Profound and Scrunch go deeper on why each bot came.
Frequently asked questions
Which AI tool is best for SEO?
It depends on the job. For AI crawlers specifically, Cloudflare AI Crawl Control and Known Agents show who visits, and Semrush Site Audit and Screaming Frog show what those bots can read.
Do AI crawlers run JavaScript?
Mostly not. Most AI crawlers read only basic HTML, not content loaded by JavaScript, so key content should be in the HTML your server sends.
Should I block GPTBot?
That is a training choice, separate from search. Blocking GPTBot tells OpenAI not to use your content for training, while OAI-SearchBot decides whether you appear in ChatGPT search. More in how to get indexed by PerplexityBot and GPTBot.
Does blocking Google-Extended hurt my Google rankings?
No. Google says Google-Extended does not affect inclusion in Google Search and is not a ranking signal.
Can I see AI crawler visits in Google Analytics?
No. Bots do not run analytics scripts, so you need server logs or a server-side tool. See how to know if AI crawlers are hitting your site.