Crawler efficiency is how much of a search engine bot's time on your site goes to the pages you actually want found. It matters because bots do not crawl everything equally. A site that sends them through errors, redirects and endless archive pages gets its important pages crawled less often, and a page that is not crawled cannot be indexed or ranked.
How a crawl works
Search engines get pages into results in four steps: Crawl, Render, Index, and Rank. Crawling comes first, and the rest depend on it.
Yoast describes Googlebot as working from a pile: it takes one page from a virtual pile, crawls it, indexes contents, and then adds all links on that page to the pile. So every link on a page decides what the bot will look at next. Links to useful pages keep the pile useful. Links to error pages, duplicate filters and empty tag pages fill it with work that brings nothing back.
Bots also choose what to crawl first. That choice is called crawl prioritization: the process search engines use to decide which pages to crawl first, and how frequently, based on their perceived value, relevance, and freshness. Crawlers allocate more crawl budget to pages that contain valuable updated content, earn strong internal or external signals, are essential for user experience, or are frequently updated.
Crawl depth and page depth
Two terms come up next to crawler efficiency. Crawl depth refers to how many pages search engine bots will access and index on a site during a single crawl. Page depth defines how many clicks a user needs to reach a specific page from the homepage using the shortest path.
They are linked. Pages on depth 3 and lower will generally perform poorer in organic search results because search engines may have issues reaching and crawling them within the site's allocated crawl budget. A page buried four clicks down is a page the bot may not reach this time.
When it matters
For most small sites, not much. Yoast's view is that if your site has 1,000 pages or less, crawl efficiency is not a topic you need to think about much. The bot gets through everything anyway.
It starts to matter on large sites, or on small sites that generate far more URLs than they have real pages. Calendars, tag pages and search filters can do that. Yoast gives an example from the Guardian, where Google indexed daily archives back to the year 1,600, approximately 150,000 clicks deep. That is crawling spent on pages nobody needed.
What wastes crawling
- Errors. When Googlebot encounters many errors during crawl, it will slow down because it is afraid that it is causing the errors by crawling too fast.
- Redirect chains. Most systems take up to a second to serve a redirect, which adds another second onto page load time. Point old URLs straight at their final page.
- Thin tag and archive pages. A rule of thumb from Yoast: a tag is only useful when it connects more than 3 pieces of content.
- Deep pages with few links. Pages with more links are crawled more frequently, so a page linked from one place gets less attention than it should.
How to improve it
- Keep your XML sitemap current. Most search engines crawl URLs in XML sitemaps more often than other URLs. Then submit it to Bing and Google.
- Link to important pages from pages that already get crawled, so they sit a click or two from the home page.
- Fix broken links and collapse redirect chains.
- Speed up the site. To raise crawl frequency on weaker pages, seoClarity suggests you improve site architecture, update the sitemap regularly, and speed up the site.
How to check yours
Your server logs show which bots fetched which URLs and what status they got back. Look for crawls landing on errors, redirects or URLs you never meant to publish. Analytics scripts do not see bots, so here is how to tell if AI crawlers are hitting your site. If you build with Claude Code or Codex, Shipfound records whether GPTBot, PerplexityBot and ClaudeBot crawl each page you ship.
For whether a bot can reach a page at all, see what is site crawlability. For how long a new page waits, see how fast AI crawlers find a new page.
Frequently asked questions
What does the term "crawler" mean?
A crawler is a program a search engine runs to fetch pages and follow their links. Google's is Googlebot. AI search engines run their own, such as GPTBot.
Do search engines still use crawlers?
Yes. Crawling is still the first of the four steps: Crawl, Render, Index, and Rank. Nothing gets indexed without it.
What is crawling vs scraping?
Crawling is a search engine fetching pages to index them. Scraping is pulling data out of pages for some other use. The fetching can look alike in your logs; the purpose differs.
How often will Google crawl my site?
There is no fixed schedule. It depends on how valuable, linked and fresh each page looks, which is what crawl prioritization decides. Pages in your sitemap, pages with many links and pages that change often get crawled more.