Robots.txt Mistakes That Hurt SEO (and How to Fix Each One)

robots.txt is a short text file with an outsized effect. One misplaced line can stop Google crawling your whole site, while a rule meant to hide a page can leave it in search results anyway.

This guide lists the mistakes we see most, explains the Google rules behind each one (precedence, wildcards, caching, status codes), and gives safe defaults. Google publishes how it interprets robots.txt, so everything here is checkable.

How Google reads robots.txt

• Location: the file must be at the root of each host, protocol and port (https://www.example.com/robots.txt). Subdomains need their own file; copies in subfolders are ignored.
• Size: Google reads up to 500 KiB; anything after that is ignored.
• Fields: Google supports user-agent, allow, disallow and sitemap. Other fields, including crawl-delay, aren’t supported.
• Case: field names are case-insensitive, but paths are case-sensitive. /Shop/ and /shop/ are different.
• Wildcards: * matches any sequence of characters and $ marks the end of a URL.
• Groups: a crawler follows only the most specific user-agent group that matches it. A Googlebot group and a * group are not combined.
• Caching: Google generally caches the file for up to 24 hours.

Mistake 1: blocking the whole site

User-agent: * followed by Disallow: / blocks everything. It often survives from a staging site or a development setting that was never switched off. Search Console’s Page indexing report will fill with “Blocked by robots.txt” and the site will lose visibility over the following weeks.

Fix: remove the rule, check the live file at /robots.txt, and confirm with URL Inspection’s live test that “Crawl allowed?” is Yes for key pages. On WordPress, also check Settings > Reading for “Discourage search engines”.

Mistake 2: using robots.txt to hide pages from search

Google’s documentation is explicit: robots.txt is not a mechanism for keeping a page out of Google. If other pages link to a blocked URL, Google can still index the URL and show it without a description. Search Console reports this as “Indexed, though blocked by robots.txt”.

Fix: allow crawling and add a noindex robots meta tag (or X-Robots-Tag header). Google must be able to crawl the page to see the noindex. For private content, use a login. See how to fix indexing problems.

Mistake 3: blocking CSS and JavaScript

Old advice suggested blocking theme and plugin folders. Google renders pages much like a browser, and it won’t use files you block. If those files build your layout, content or links, Google may see a broken or empty page. Check URL Inspection’s live test screenshot for missing content. Our JavaScript SEO guide covers rendering in more detail.

Mistake 4: rules that match more (or less) than intended

Rule	What it actually blocks
Disallow: /blog	/blog, /blog/, /blog/anything and also /blog-tips and /blogroll
Disallow: /blog/	Only URLs inside the /blog/ folder
Disallow: /*?	Every URL with a query string, including useful ones
Disallow: /*.pdf$	URLs ending exactly in .pdf
Disallow: /search	Internal search at /search, but not /?s= search on WordPress

Precedence matters too. If you have Disallow: /shop/ and Allow: /shop/sale/, the sale folder is crawlable because the longer rule wins. When two rules are equally specific, Google uses the least restrictive one.

Mistake 5: separate groups that cancel each other

If your file has a User-agent: * group with your main rules and a separate User-agent: Googlebot group with one extra line, Googlebot follows only its own group and ignores everything in the * group. Either repeat the shared rules in the Googlebot group or remove the separate group.

Mistake 6: robots.txt returning the wrong status code

Status when Google fetches robots.txt	What Google does
200	Reads and follows the file
3xx	Follows up to five redirect hops, then treats it as not found
4xx (except 429)	Treats it as no robots.txt: no crawl restrictions
5xx or timeout	Stops crawling for the first 12 hours while retrying, then uses the last cached copy for up to 30 days

So a host or firewall that returns 503 for robots.txt can pause crawling of your whole site, and using 401 or 403 to slow crawling doesn’t work. Google says not to use those codes for limiting crawl rate.

Mistake 7: relying on crawl-delay

Google ignores crawl-delay. If Googlebot is genuinely overloading your server, Google’s Crawl stats guidance suggests temporarily returning 503 or 429 to Googlebot, for no more than two or three days, while you add capacity.

Safe robots.txt patterns

WordPress

WordPress generates a virtual robots.txt that blocks /wp-admin/ but allows admin-ajax.php, plus a Sitemap line if your SEO plugin adds one. That’s a good default. Add rules only for clear problems, such as internal search (Disallow: /?s=) or plugin-generated duplicate parameters.

Shopify

Shopify ships a sensible default that blocks cart, checkout, account and internal search URLs. You can customize it with a robots.txt.liquid template, but edit only for a specific reason and keep Shopify’s defaults. Our Shopify SEO service covers these edits.

Every site

• Include a Sitemap: line with the absolute sitemap URL.
• Block internal search results and infinite filter combinations, not content pages.
• Never block CSS, JS or images that pages need to render.
• Keep it short and comment changes with dates.

How to test changes

1. Edit the file and load /robots.txt in a private window to confirm the live version.
2. Use Search Console’s robots.txt report (under Settings) to see the version Google last fetched and any parse issues.
3. Run URL Inspection’s live test on a few affected URLs and check “Crawl allowed?”.
4. Watch the Page indexing report over the following weeks for unexpected “Blocked by robots.txt” growth.

robots.txt problems often appear together with sitemap and canonical issues; the technical SEO checklist covers all three.

Frequently asked questions

Sources

• Google Search Central: How Google interprets the robots.txt specification
• Google Search Central: Introduction to robots.txt
• Google Search Central: Block Search indexing with noindex
• Search Console Help: Page indexing report
• Search Console Help: Crawl Stats report
• Search Console Help: URL Inspection tool

Last reviewed October 12, 2026.