Technical SEO
What Is Crawl Budget, and When Does It Actually Matter?
Most sites never need to think about crawl budget. Here’s how to tell if yours is an exception.
Crawl budget is the number of URLs Google can and wants to crawl on a site. Google describes it as a combination of crawl capacity (how much crawling your server can handle) and crawl demand (how much Google wants to crawl your URLs).
Google’s crawl budget guide is written for large sites. It names roughly a million or more pages changing weekly, ten thousand or more pages changing daily, or a large share of URLs sitting in “Discovered, currently not indexed” as the situations where it matters.
Short answer: Crawl budget is how many URLs Googlebot can and wants to crawl on your site, set by crawl capacity (server limits) and crawl demand (how popular and fresh your URLs are). Google says it mainly matters for very large or fast-changing sites. To use it well, block unimportant URLs, remove duplicates, fix soft 404s and long redirect chains, return 404 or 410 for removed pages, and keep sitemaps updated.
Key Takeaways
- Crawl budget = crawl capacity + crawl demand.
- Google says it’s mainly a concern for large or fast-changing sites.
- Duplicates, filters, soft 404s and redirect chains waste crawling.
- 404 or 410 tells Google not to keep crawling removed URLs.
- The Crawl Stats report shows how Google crawls your site.
Do you need to worry?
| Situation | Crawl budget a concern? |
|---|---|
| Small business site with a few hundred pages | Rarely; Google says sites under about 1,000 pages usually don’t need the Crawl Stats report |
| Store or publisher with very large numbers of URLs | Possibly |
| Many pages changing daily | Possibly |
| Large share of URLs in “Discovered, currently not indexed” | Worth investigating |
What wastes crawling
- Duplicate URLs, such as parameter and filter variations (faceted navigation).
- Unimportant URLs that should be blocked in robots.txt.
- Pages with noindex that Google still requests.
- Soft 404s: error pages that return 200.
- Long redirect chains (redirects explained).
How to improve crawl efficiency
- Consolidate duplicates with canonicals or redirects.
- Block crawl traps such as infinite filters and calendar URLs with robots.txt.
- Return 404 or 410 for permanently removed pages.
- Fix soft 404s so missing pages return real error codes.
- Keep XML sitemaps accurate, with lastmod dates that reflect real changes.
- Make sure your server responds quickly and reliably.
Check the Crawl Stats report
In Search Console under Settings, Crawl stats shows total requests, download size, response times, host status and a breakdown by response type and purpose. Google notes it’s aimed at advanced users. For error patterns, see how to find crawl errors.
Frequently asked questions
Does crawl budget affect rankings?
Crawling is needed before indexing, but being crawled more doesn’t make a page rank higher.
Can I ask Google to crawl my site more?
You can’t buy or request more crawl budget. Google adjusts it based on your server and demand for your URLs.
Should I use noindex to save crawl budget?
noindex pages are still crawled. Use robots.txt for URLs you don’t want crawled at all.
Want a second pair of eyes on your technical SEO?
We review crawling, indexing, rendering and speed issues by hand and give you a prioritized list of fixes, starting with a free manual audit.
Sources
- Google Search Central: Crawl budget management for large sites
- Search Console Help: Crawl Stats report
Last reviewed October 14, 2026.
Table of Contents
Need Technical SEO Help?
We find what stops Google from crawling and indexing your important pages, and fix the cause.
Related Guides
Free SEO Audit
A manual on-page and technical SEO review of your website, with prioritized fixes.