Robots.txt tester
Check whether specific paths or URLs are allowed or blocked by a robots.txt file.
Robots.txt tester
Paste the contents of a robots.txt file, list the paths or full URLs you want to check underneath, and this tool tells you exactly which ones a crawler is allowed to fetch and which are blocked. It is built for the moment a page mysteriously stops ranking and someone needs to confirm, in seconds, whether a Disallow rule is the cause — no deploy, no waiting for a live crawl.
Pick the crawler from the user-agent list — Googlebot, Googlebot-Image, Bingbot, GPTBot, ClaudeBot — or type a custom token to test a bot of your own. The tool applies the same matching logic real crawlers do: the longest matching rule wins, an Allow beats a Disallow of equal length, and both "*" (any sequence of characters) and a trailing "$" (end of path) work exactly as they do in a live robots.txt. Turn on "Show deciding rule" to see precisely which Allow or Disallow line produced each verdict, so a confusing result stops being a guess.
Beyond individual paths, the report lists every Sitemap line the file declares and any Crawl-delay set for the selected user agent, and flags syntax problems — a directive with no colon, a rule sitting before any User-agent line (where it silently does nothing), or a field crawlers don't recognize. Choose a plain table for a quick read or JSON to feed the result into a script or a CI check.
Everything runs in your browser. The robots.txt content and the paths you test are never uploaded anywhere, so it is safe to check rules for a site that has not launched yet or that lives behind a login.