How to Find Crawl Errors in Google Search Console (and What Each One Means)

Search Console used to have a single “Crawl errors” report. Today the same information is spread across three places: the Page indexing report (errors for specific URLs), the URL Inspection tool (what happened to one URL), and the Crawl stats report (how your server responds to Googlebot overall).

Not every error is a problem. A 404 for a page you deleted on purpose is working as intended. This guide shows how to find crawl errors, separate the harmless ones from the damaging ones, and fix the causes.

Where crawl errors appear in Search Console

Report	What it tells you	Use it for
Page indexing	Which URLs aren’t indexed and why (5xx, 404, redirect error, blocked, soft 404, 401/403)	Finding patterns across many URLs
URL Inspection	Last crawl, crawl allowed, page fetch result, indexing allowed, canonicals; live test of the current page	Diagnosing one URL and confirming fixes
Crawl stats (Settings)	Total requests, response codes, response time, host status for robots.txt, DNS and connectivity	Site-wide availability problems

Crawl stats is only available for root-level properties, and Google describes it as aimed at advanced users; sites with fewer than about a thousand pages usually don’t need it.

The crawl errors that matter, and what causes them

Server error (5xx)

Your server returned a 500-level error when Google requested the page. Common causes are overloaded hosting, PHP or plugin fatal errors, timeouts on slow database queries, and security rules that block Googlebot. Check your server error logs for the same timestamps and the Crawl stats host status.

Redirect error

Google hit a redirect chain that was too long, a redirect loop, a redirect to an overly long URL, or a bad or empty URL in the chain. Typical culprits are conflicting rules between a plugin, the CMS and the server (for example, one forcing a trailing slash and another removing it).

Blocked by robots.txt

A Disallow rule matches the URL. That’s fine for admin areas and internal search results, and a problem for pages you want in search. See common robots.txt mistakes.

Blocked due to unauthorized request (401) or access forbidden (403)

Googlebot never sends credentials, so a page behind a login or a firewall rule returns 401 or 403. If the page should be public, remove the login requirement or allow verified Googlebot through the firewall.

Soft 404

The page shows “not found” or almost no content but returns a 200 status. Return a real 404 for missing pages, or add genuine content if the page should exist. Empty category pages and out-of-stock templates are frequent sources.

Not found (404)

Usually harmless. If the page was removed on purpose and has no replacement, leave it. If it moved, add a permanent redirect to the new URL. Investigate 404s only when they hit important URLs or come from your own internal links.

Step by step: diagnosing one crawl error

1. In Page indexing, open the error reason and note a few example URLs (the report shows up to 1,000 examples).
2. Paste one into URL Inspection. Check “Last crawl”, “Crawl allowed?” and “Page fetch”.
3. Click Test live URL to see whether the problem still exists right now.
4. Open View tested page to see the HTTP response, headers and rendered HTML.
5. Reproduce it yourself: load the URL in a private window and check the status code and redirects.
6. Fix the cause (server, redirect rule, robots.txt, login), then re-test live.
7. When all examples are fixed, click Validate fix in the Page indexing report.

Site-wide problems: reading Crawl stats

If errors appear across many unrelated URLs at once, the cause is usually the host rather than individual pages. Crawl stats shows host status in three categories over the past 90 days:

• robots.txt fetching: Google slows or stops crawling if robots.txt doesn’t return a valid file or a 404. HTTP 429 and 5xx count as failures.
• DNS resolution: your DNS didn’t recognize the hostname or didn’t respond.
• Server connectivity: the server didn’t respond or returned an incomplete response.

Then look at the response breakdown. A rising share of 5xx or timeouts, or a jump in average response time, points to hosting capacity, a slow plugin, or a CDN or firewall rule. If Googlebot is overloading the server, Google suggests temporarily returning 503 or 429, but not for more than two or three days.

Errors you can safely ignore

• 404s for pages you deleted with no replacement.
• “Page with redirect” for old URLs that now redirect correctly.
• “Alternate page with proper canonical tag” for duplicates pointing to the main page.
• “Excluded by noindex” for pages you deliberately keep out of search (thank-you pages, internal search).
• Robots.txt blocks on admin, cart, checkout and internal search URLs.

Spend your time on errors that affect pages you want to rank. If you’re unsure whether a page should be indexed at all, see how to fix indexing problems.

Preventing crawl errors

• Keep one redirect layer: let either the server or one plugin handle redirects, not several.
• Update internal links when URLs change so they point straight to the final page.
• Keep the sitemap limited to live, canonical, indexable URLs.
• Monitor hosting resources and error logs after plugin or theme updates.
• Re-check Page indexing a week after any big change.

Our technical SEO checklist puts these checks in order, and our technical SEO service covers the fixes when the cause sits in hosting or code.

Frequently asked questions

Sources

• Search Console Help: Page indexing report
• Search Console Help: URL Inspection tool
• Search Console Help: Crawl Stats report
• Google Search Central: How Google interprets the robots.txt specification
• Google Search Central: Redirects and Google Search

Last reviewed October 12, 2026.