Site crawlability is how easily search engine bots can reach and read the pages on your website. Semrush puts it as how easily search engines like Google can discover the page. It matters because a page nobody can crawl goes nowhere: without first being crawled and indexed, a page will not be ranked by Google.
How crawling works
Search engines send out programs called web crawlers, also called bots or spiders, which follow links between pages to discover new or updated pages. Google's is sometimes referred to as Googlebot. They never stop: a crawler follows the links on the web 24/7, and crawlers can come to a site multiple times a day to discover new content or check for updates.
Before a bot fetches anything, search engines check the robots.txt file for restrictions. Then it follows links from page to page, and saves the HTML version in a gigantic database called the index.
AI search engines run bots of their own that crawl the same way, so a page that blocks or hides from bots can miss out on AI answers too.
Crawlability and indexability are different
Crawlability is whether a bot can reach the page. Indexability is whether the engine will keep it. Indexing usually follows crawling, but the two can split:
- A page can be crawled and still left out. If the robots meta tag on a specific page blocks the search engine from indexing that page, Google will crawl that page but will not add it to its index.
- A page can be blocked from crawling and still show up. Search engines can still discover and index pages that are linked from other websites, even if those pages are blocked in the robots.txt file.
What affects crawlability
- Internal links. Internal links act as a roadmap, guiding search engine bots from one page to another within a website.
- robots.txt. It tells search engine crawlers which pages or files the crawler can or cannot request from a site. If a robots.txt file blocks the crawler, Google will not come to the website or specific web page.
- Your sitemap. An XML sitemap shows search engine bots all the important pages on a website that you want crawled and indexed.
- Status codes. If the HTTP header status code says that a page does not exist, Google will not crawl the website.
- Technical problems. Slow page load times, broken links, and redirect loops can hinder bots' ability to navigate a website.
- Duplicates. Duplicate content issues or improper canonical tag usage can cause search engines to struggle to understand which version of a page to index and rank.
An example
A startup publishes a new pricing guide. The page works when you type its address, but nothing on the site links to it and it is not in the sitemap. A bot that follows links never reaches it. Two small changes fix that: a link from the blog index and a line in the sitemap. The next crawl finds it.
How to check your site's crawlability
- Open
yoursite.com/robots.txtand read every Disallow line. Make sure none of them covers a page you want found. What are AI crawler rules explains the AI bots' part of that file. - Open your sitemap and check that every page you care about is in it, then submit it to Bing and Google.
- Pick a few important pages and confirm each one answers with a normal page, not an error or a redirect chain.
- Check your server logs to see which bots fetch which pages. Analytics scripts do not see bots, so here is how to tell if AI crawlers are hitting your site.
If you build with Claude Code or Codex, Shipfound checks these from the outside and records whether GPTBot, PerplexityBot and ClaudeBot crawl each page.
Common mistakes
- Using robots.txt to keep a page out of search. Although robots.txt controls crawl accessibility, it does not directly impact the indexability of a website. To keep a page out of the index, use a robots meta tag and let bots crawl it.
- Leaving a staging rule in place. A Disallow line meant for a test site ships to production and blocks the whole site.
- Redirects that point to other redirects. Send each old URL straight to its final page.
Frequently asked questions
What does "crawlability" mean?
It means how easily a search engine bot can reach and read a page. Yoast says crawlability has to do with the possibilities Google has to crawl a website.
How do I check if my website is crawlable?
Follow the four steps above: robots.txt, sitemap, status codes, server logs.
Is ChatGPT a web crawler?
ChatGPT is an assistant, not a crawler. The bots that fetch pages, such as GPTBot, are separate, and you allow or block them in robots.txt like any other. How to get indexed by PerplexityBot and GPTBot covers which bot does what.
What is the difference between crawlability and indexability?
Crawlability is whether a bot can reach a page. Indexability is whether the engine adds it to its index once it has. A page needs both before it can rank.