Technical SEO
Noindex vs robots.txt: Which to Use (and the Mistake That Breaks Both)
robots.txt controls crawling. noindex controls indexing. Mixing them up is one of the most common technical SEO mistakes.
Google is clear that robots.txt is mainly for managing crawler traffic, and that it’s not a mechanism for keeping a page out of Google. A page blocked by robots.txt can still appear in results, without a description, if other pages link to it.
To keep a page out of search results, Google says to use noindex, and to make sure the page isn’t blocked by robots.txt, or Google will never see the noindex.
Short answer: Use noindex (a robots meta tag or X-Robots-Tag header) to keep a page out of Google’s index; the page must stay crawlable so Google can see the tag. Use robots.txt to stop crawling of URLs you don’t need crawled at all, such as filters or internal search. Don’t use both on the same URL: if robots.txt blocks a page, Google can’t read its noindex, and the URL may still appear in results.
Key Takeaways
- robots.txt = don’t crawl. noindex = don’t index.
- A robots.txt-blocked URL can still be indexed if linked.
- noindex only works if Google can crawl the page.
- Google doesn’t support noindex rules inside robots.txt.
- Themes, plugins and staging settings often add noindex by accident.
Which to use
| Goal | Use | Why |
|---|---|---|
| Keep a page out of search results | noindex (meta tag or X-Robots-Tag) | Google drops it after recrawling |
| Stop crawling of endless filter or search URLs | robots.txt Disallow | Saves crawling; these URLs don’t need to be fetched |
| Keep private content private | Password protection | Neither noindex nor robots.txt stops people opening the page |
| Remove a page quickly | Removals tool, then a permanent method (how to remove URLs) | Removals are temporary |
| Keep a PDF out of results | X-Robots-Tag: noindex header | Works for non-HTML files |
The mistake that breaks both
Adding noindex and then also blocking the URL in robots.txt means Google can’t crawl the page to see the noindex. If the URL is linked elsewhere, it can stay in the index. Remove the robots.txt block until Google has recrawled the page and dropped it.
Finding unwanted noindex
- Search Console’s Page indexing report lists “Excluded by ‘noindex’ tag” pages; check none are important (indexing reasons).
- Use URL Inspection to see whether indexing is allowed on a specific page.
- WordPress: check Settings > Reading and your SEO plugin’s per-page and per-type settings.
- Theme or plugin conflicts can add a second robots meta tag; view the page source and search for “robots”.
- After launch, check that staging noindex settings weren’t carried over.
How long noindex takes
Google removes the page after it next crawls it. For pages Google rarely crawls, that can take months; you can request a recrawl with URL Inspection.
Frequently asked questions
Can I put noindex in robots.txt?
No. Google doesn’t support noindex rules in robots.txt.
Why is a page blocked by robots.txt still in Google?
Google can index a blocked URL if other pages link to it, usually without a description. Use noindex and allow crawling to remove it.
Is “Excluded by noindex tag” in Search Console a problem?
Only if the page should be indexed. For thank-you pages, internal search or thin archives, it’s expected.
Want a second pair of eyes on your technical SEO?
We review crawling, indexing, rendering and speed issues by hand and give you a prioritized list of fixes, starting with a free manual audit.
Sources
- Google Search Central: Block search indexing with noindex
- Google Search Central: Introduction to robots.txt
Last reviewed October 14, 2026.
Table of Contents
Need Technical SEO Help?
We find what stops Google from crawling and indexing your important pages, and fix the cause.
Related Guides
Free SEO Audit
A manual on-page and technical SEO review of your website, with prioritized fixes.