Robots.txt generator
Build a correct robots.txt with presets, custom rules, crawl-delay and a sitemap link.
Related tools
Robots.txt generator
Generate a syntactically correct robots.txt in seconds. Pick a preset — allow everything, disallow everything, or custom paths — set the user-agent, and the file appears ready to copy or download as robots.txt for the root of your site. The two presets produce the exact canonical forms: an empty Disallow line that permits all crawling, or Disallow: / that blocks the whole site, which is what you want on a staging domain.
The user-agent field accepts several bot names separated by commas, each emitted as its own User-agent line in the same group. That makes AI-crawler policies quick to write: enter GPTBot, ClaudeBot, Google-Extended, choose the disallow-everything preset, and the rules apply to all of them. Leave the field empty and the group targets every crawler with *.
With the custom preset you list Disallow and Allow paths separated by commas. A missing leading slash is added automatically, wildcards like /*.pdf pass through unchanged, and Allow lines come first so exceptions to broader blocks read naturally. An optional Crawl-delay adds the throttling hint some crawlers such as Bing respect, and the sitemap field appends a Sitemap line with the absolute URL search engines expect.
The file is generated entirely in your browser — nothing about your site structure is sent to a server. Note that robots.txt controls crawling, not access: obedient bots follow it, but it is not a security mechanism, so use authentication for anything genuinely private.