How to control AI crawlers with robots.txt

A few years ago the crawlers hitting your site were search engines and a long tail of scrapers. Now a large share of them belong to language models: some collect pages to train on, some fetch a page live because a user asked a question about it, and some do both under different names. The control surface for all of them is the same file it has always been — /robots.txt, a plain text file at the root of your domain — but the decisions behind it are new.

Training, retrieval and search are three different asks

The most common mistake is treating "AI crawler" as one thing. It is at least three, and most vendors run a separate user agent for each:

1. Training collection. A bot that crawls broadly and stores pages for a future model. Blocking it costs you nothing today and gains you nothing today — it is a licensing decision, not a traffic decision.
2. Live retrieval. A fetch triggered by a person in a chat window who pasted your URL or asked a question your page answers. Blocking this means the assistant cannot read you, cannot quote you, and cannot link you.
3. Answer-engine indexing. A crawler that builds the index an AI search product answers from. Blocking it removes you from that product's results the same way blocking a search engine would.

Publishers routinely block all three by pasting one rule, then wonder why their brand stopped appearing in assistant answers. Decide per category before you write a line.

Who is who

The names you will actually see in your logs, and what each one is for:

• GPTBot — OpenAI's broad crawler, associated with training data collection.
• OAI-SearchBot — OpenAI's crawler for search-style results; blocking it is a visibility decision, not a training one.
• ChatGPT-User — a live fetch made because a user asked. There is a person waiting on the other end of this request.
• ClaudeBot — Anthropic's broad crawler. Claude-User and Claude-SearchBot cover user-initiated fetches and search indexing.
• Google-Extended — not a crawler at all. It is a token that lets you opt out of Gemini training while Googlebot keeps crawling for Search. Disallowing it does not affect your search ranking; disallowing Googlebot very much does.
• PerplexityBot — indexing for an answer engine that cites sources inline.
• CCBot — Common Crawl. Not an AI company, but its archive is a training input for many of them.
• Applebot-Extended, meta-externalagent, Amazonbot, Bytespider — the same pattern from Apple, Meta, Amazon and ByteDance.

Vendors add and rename agents regularly, so treat any list — including this one — as a snapshot. The durable habit is reading your own access logs: pull the distinct user agent strings from a day of traffic and run them through the user agent parser to see which are bots, which are browsers, and which are pretending to be browsers.

Writing the rules

The syntax is unchanged and unforgiving in exactly one place: each User-agent group applies to the bot that matches it most specifically, and a bot that matches its own name ignores the * group entirely. So this does not do what it looks like it does:

A group for * that disallows nothing, followed by a group for GPTBot that disallows /, works fine. But if you put a crawl-delay or a sitemap rule only in the * group and assume named bots inherit it, they do not. Repeat anything that matters in each group, or keep your rules broad.

The robots.txt generator builds the file from checkboxes — pick the agents you want to allow or block, add your disallowed paths, and it emits correctly grouped output with your sitemap line at the end. Once the file exists, paste it into the robots.txt tester with a URL and a user agent to confirm the answer is the one you meant. Testing matters more than usual here because the failure is silent: a rule that accidentally blocks everything looks exactly like a rule that works, until traffic disappears a month later.

What robots.txt cannot do

Three limits are worth internalising before you rely on the file:

It is a request, not a fence. Well-behaved crawlers honour it. Scrapers that want your content ignore it, and nothing in the protocol stops them. If you need enforcement, that lives in your server config, a rate limiter or a WAF rule — and even then you are matching on user agent strings and IP ranges that can be faked.

Disallow is not noindex. A disallowed URL can still appear in results if other sites link to it, because the crawler is told not to fetch the page but is not told to forget it exists. If you want a page out of an index, let the crawler fetch it and serve a noindex meta tag or header. Blocking in robots.txt actively prevents the crawler from ever seeing that tag.

It is per host, scheme and port. https://example.com/robots.txt governs that origin only — not www., not the staging subdomain, not the CDN hostname serving your images. Every host that answers requests needs its own file.

Point crawlers at what you do want indexed

The other half of the file is invitation, not exclusion. A Sitemap: line gives every crawler an explicit list of your canonical URLs and their last-modified dates, which is the cheapest way to get new pages discovered — the sitemap generator turns a list of URLs into valid XML you can upload alongside the robots file. Keep the two consistent: a URL that appears in your sitemap while being disallowed in robots.txt is a contradiction that shows up as an error in every reporting tool that reads both.

You may also see llms.txt proposed as a companion file — a markdown summary of a site aimed at models rather than crawlers. It is a convention, not a standard, and no major vendor treats it as a directive. Adding one is harmless; relying on it for control is not.

A reasonable default

If you publish content and want reach, the setup most sites land on is: allow live retrieval and answer-engine crawlers so assistants can find and cite you, block or allow bulk training crawlers according to how you feel about that trade, never touch Googlebot or Bingbot unless you know exactly why, and keep a sitemap line pointing at a current file. If you publish behind a paywall or sell licences to your archive, the calculation inverts and blocking the broad crawlers is the point.

Either way, write it deliberately, test it against real user agent strings, and re-check it whenever a vendor announces a new bot. Both the generator and the tester run entirely in your browser — your robots file, your URLs and your log lines never leave the page.