Free tool
Free Robots.txt Checker
Learn more: Robots.txt vs Noindex: When to Use Each
Why a robots.txt block matters more than it looks
Blocking a path in robots.txt stops crawlers from fetching it — but it doesn't remove an already-indexed page, and it doesn't stop that URL from appearing in search results if other sites link to it. It just means search engines can't see the content behind it anymore, which often looks worse, not better.
Common robots.txt mistakes
- A broad rule blocks more than intended — a single Disallow: / line, even temporarily (e.g. during a staging-site copy), blocks the entire site.
- Blocking a path that's also meant to be noindexed — crawlers that can't fetch a page can't see its noindex tag either, so a blocked page can keep showing a bare URL in search results instead of disappearing.
- Rules meant for one subdirectory accidentally match a broader pattern due to how robots.txt wildcard matching works.
- A CDN, staging environment, or CMS default robots.txt is still live on the production domain after launch.
- Crawler-specific rules conflict — a rule aimed at one user-agent unintentionally also matches Googlebot due to overlapping User-agent: * blocks.
How to fix it
- If a page needs to disappear from search results, use noindex, not a robots.txt block — the two solve different problems, see robots.txt vs noindex.
- Check wildcard and path-prefix rules carefully; a rule written for one folder can silently match others.
- After any robots.txt change, re-check the specific URLs you care about — a single misplaced rule can block an entire section without an obvious error anywhere else.
- Remove staging or CMS-default robots.txt rules before or immediately after launch — don't assume the default file is safe for production.
Frequently asked questions
Does blocking a page in robots.txt remove it from Google?
No. Robots.txt only stops crawling, not indexing. A URL already indexed, or one Google learns about through external links, can still appear in search results — usually just as a bare link with no title or description, since Google can't read the blocked content.
Should I use robots.txt or noindex to keep a page out of search results?
Noindex, in almost every case. Robots.txt is for managing crawl budget on a large site (keeping crawlers away from pages you don't need indexed at all, like internal search results), not for removing specific pages from results.
How often does Google re-check my robots.txt file?
Frequently — Google generally caches a site's robots.txt for about 24 hours, so changes typically apply within a day, though individual URL re-crawls still happen on their own schedule after that.
Can robots.txt block images, CSS, or JavaScript by accident?
Yes, and it's a common issue — blocking a site's CSS or JS folders can stop Google from rendering a page properly, which can affect how it's indexed even though the HTML itself isn't blocked.