robots.txt & sitemap checker
See which crawlers a site lets in, which paths it blocks, and whether its sitemap loads and how many pages it lists.
- Free
- No sign-up
- Results in seconds
How the robots.txt & sitemap checker works
- 1Enter a website
- 2We fetch its robots.txt and the sitemaps it lists
- 3See what's blocked, for whom, and how many pages the sitemap has
What it checks
- Rules per crawler
- Blocked paths
- Search engines blocked?
- Sitemap found
- Pages in the sitemap
Frequently asked questions
- Does robots.txt hide pages?
- No. It asks well-behaved crawlers not to visit, but blocked pages can still appear in results if other sites link to them. Use a noindex tag to keep a page out of search.
- Where should the sitemap be?
- Anywhere, as long as robots.txt lists it with a Sitemap: line or you submit it in Google Search Console. /sitemap.xml is the usual place.
- A site disappeared from Google overnight
- Check for “Disallow: /” under “User-agent: *”. It's often left over from a staging site copied to production.
Get told the moment it breaks
Checking by hand only tells you about right now. Spot Downtime checks your sites around the clock and alerts you by email, Slack, SMS or a phone call.
Start monitoring free