robots.txt & sitemap checker

See which crawlers a site lets in, which paths it blocks, and whether its sitemap loads and how many pages it lists.

Good to know

Does robots.txt hide pages?: No. It asks well-behaved crawlers not to visit, but blocked pages can still appear in results if other sites link to them. Use a noindex tag to keep a page out of search.
Where should the sitemap be?: Anywhere, as long as robots.txt lists it with a Sitemap: line or you submit it in Google Search Console. /sitemap.xml is the usual place.
A site disappeared from Google overnight: Check for “Disallow: /” under “User-agent: *”. It's often left over from a staging site copied to production.

Checking by hand only tells you about right now. Spot Downtime checks your sites around the clock and alerts you by email, Slack or Discord.