Free tools

robots.txt & sitemap checker

See which crawlers a site lets in, which paths it blocks, and whether its sitemap loads and how many pages it lists.

  • Free
  • No sign-up
  • Results in seconds

How the robots.txt & sitemap checker works

  1. 1Enter a website
  2. 2We fetch its robots.txt and the sitemaps it lists
  3. 3See what's blocked, for whom, and how many pages the sitemap has

What it checks

  • Rules per crawler
  • Blocked paths
  • Search engines blocked?
  • Sitemap found
  • Pages in the sitemap
Example result, with sample data.

Frequently asked questions

Does robots.txt hide pages?
No. It asks well-behaved crawlers not to visit, but blocked pages can still appear in results if other sites link to them. Use a noindex tag to keep a page out of search.
Where should the sitemap be?
Anywhere, as long as robots.txt lists it with a Sitemap: line or you submit it in Google Search Console. /sitemap.xml is the usual place.
A site disappeared from Google overnight
Check for “Disallow: /” under “User-agent: *”. It's often left over from a staging site copied to production.

Get told the moment it breaks

Checking by hand only tells you about right now. Spot Downtime checks your sites around the clock and alerts you by email, Slack, SMS or a phone call.

Start monitoring free