robots.txt AI training opt-out with __usage_ai-training: Google's mapping will also cover Gemini grounding

The robots.txt on this site is a handful of lines, and one of them reads Content-Signal: ai-train=yes, search=yes, ai-input=yes. I added it in April, partly to pass an agent-readiness scanner. On 30 October three things landed at once: an IETF draft that defines a shared robots.txt token for AI training, a Google post saying Google will honor it, and an OpenAI post saying the same. None of the three mentions my line.

That is fine for me, because I want everything allowed and allowed is roughly what happens when you say nothing. It is less fine for anyone who wants to say no to training and yes to everything else, because at Google the new token will reach further than its own definition.

What is the __usage_ai-training robots.txt token?

It is an ordinary robots.txt group whose User-Agent line names a purpose instead of a crawler. Disallow under it means a preference against AI training for those paths, Allow means a preference for it, and no such group means no preference at all. The draft adds no new directives and no new grammar.

The draft is draft-kelley-aipref-training-preference-signal-00, written by Kevin Kelley of Anthropic and posted on 30 October. It is an individual submission: the document itself says it is not endorsed by the IETF and has no formal standing. It borrows its definition of AI training from the AIPREF working group’s vocabulary draft, where training means using an asset “to modify the learned parameters of a generative AI model”. Its example:

User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/

Two rules in it matter more than the example. The token is not a crawl directive: a crawler that honors it may still fetch the page, and crawl limits stay with each crawler’s own product token. And the wildcard group does not count:

Wildcard groups (User-Agent: *) MUST NOT be interpreted as
matching the policy token.

So a file whose only group is User-Agent: * with Disallow: / says “don’t crawl me” to crawlers and says nothing at all about training.

What did Google and OpenAI announce for the shared AI training token?

Google announced a mapping and a target year; OpenAI announced it will honor “a common signal” without naming the token or a date. Neither has implemented anything yet.

Google’s post, by John Mueller, says what the token will mean for Google specifically:

When a website uses this policy token to opt out of training
in its robots.txt file (and Google-Extended is not explicitly
specified), Google plans to interpret that as if the publisher
had disallowed Google-Extended.

The post says Google aims to support it “in 2027”; the crawler documentation, updated the same day, says “early 2027” and names the token. Existing controls like Google-Extended keep working alongside it.

OpenAI’s post says it will honor a common signal in robots.txt “for foundational generative AI training”, aims to implement it “in the coming months”, and promises a second post with implementation details. Its existing crawler controls stay in force. How the signal will interact with GPTBot is for that second post to say.

Will opting out of AI training with the shared token remove a site from Gemini grounding?

Read together, Google’s two documents say yes, once the mapping ships. The token opts you out of training, Google plans to turn that into a Google-Extended disallow, and Google-Extended was never a training-only control. Here is what the crawler documentation says it governs:

Google-Extended is a standalone product token that web
publishers can use to manage whether content Google crawls
from their sites may be used for training future generations
of Gemini models that power Gemini Apps and Vertex AI API for
Gemini and for grounding (providing content from the Google
Search index to the model at prompt time to improve factuality
and relevancy) in Gemini Apps and Grounding with Google Search
on Vertex AI.

And the blog post, a paragraph after the mapping:

Websites that specifically allow Google-Extended are eligible
to appear in responses and receive clicks via grounding in
Gemini Apps.

The draft defines a training preference and calls usage preferences orthogonal to everything else. Google, the one vendor so far that has said how it will read the token, reads a training opt-out as a Google-Extended opt-out, and Google-Extended bundles training with grounding in Gemini Apps and on Vertex AI. A publisher who only meant “don’t train on me” would also drop out of the grounding that cites and links sources in Gemini Apps. Google does not frame it as a cost: the post says what allowing Google-Extended earns, one paragraph after the mapping, and leaves the reader to subtract. That subtraction is my reading of the two documents side by side. Google Search itself, including its AI features, is a separate control and is not affected.

The way out is the precedence rule in the same sentence: the mapping applies only when Google-Extended “is not explicitly specified”. Write an explicit Allow for Google-Extended and Google reads that instead. But that Allow covers training too. Through the robots.txt controls Google documents today, Gemini training and Gemini grounding are one switch.

OpenAI does split them. Its crawler documentation calls each setting “independent of the others”: GPTBot is the training control, OAI-SearchBot decides whether a site appears in ChatGPT search answers. A third agent, ChatGPT-User, fetches pages when a user asks, and OpenAI says of it that “robots.txt rules may not apply”.

Where does Cloudflare’s Content-Signal fit next to Google-Extended and the token?

Content-Signal is a separate vocabulary from Cloudflare, published in September 2025, written into the User-Agent: * group. Under the new draft’s own wildcard rule, nothing in that group speaks for the token, and neither Google’s post nor OpenAI’s mentions Content-Signal.

It still earns its line: it is the only one of the three with a word for grounding. It defines three signals:

search : building a search index and providing search results
  (e.g., returning hyperlinks and short excerpts from your
  website's contents). Search does not include providing
  AI-generated search summaries.
ai-input : inputting content into one or more AI models (e.g.,
  retrieval augmented generation, grounding, or other real-time
  taking of content for generative AI search answers)
ai-train : training or fine-tuning AI models

The AIPREF vocabulary has similar categories (AI Training, AI Use, Search), with definitions that do not line up exactly, and the 30 October draft turns only the first into a token. As of today:

Content-Signal also comes with a legal paragraph: Cloudflare’s suggested policy text calls any restriction an express reservation of rights under Article 4 of the EU copyright directive. The training token makes no such claim.

Why does a standard robots.txt parser misread the __usage_ai-training token?

Because a stock parser answers a crawl question, and the token is a preference question. Under RFC 9309, * covers any crawler without its own group, so a parser treats the token as one more crawler name and returns yes or no where the draft wants a third answer: no preference. Python’s built-in parser, asked about the token against four files:

from urllib.robotparser import RobotFileParser
def check(name, txt, path="/post"):
    rp = RobotFileParser(); rp.parse(txt.splitlines())
    print(name, rp.can_fetch("__usage_ai-training", "https://example.com" + path))
check("star-disallow", "User-Agent: *\nDisallow: /\n")
check("star-allow", "User-Agent: *\nAllow: /\n")
check("token-disallow", "User-Agent: __usage_ai-training\nDisallow: /\n\nUser-Agent: *\nAllow: /\n")
check("draft-example", "User-Agent: __usage_ai-training\nDisallow: /\nAllow: /press-kit/\n", "/press-kit/logo.png")

Output on Python 3.12.3:

star-disallow False
star-allow True
token-disallow False
draft-example False

By the draft’s rules the first two should be “no preference”, not an opt-out and an opt-in; a staging site with Disallow: / for everyone has not thereby opted out of training. Only the third is right. The fourth is the draft’s own example, and the parser gets it wrong for a different reason: it takes the first matching rule, where RFC 9309 says the longest match wins, so /press-kit/ reads as opted out. A checker for this token needs three things a crawl library does not give you: the wildcard exception, longest-match precedence and a no-preference state. If you audit robots.txt files with an off-the-shelf parser, check the token group by hand.

Which robots.txt to write for AI training in 2027

Decide which of three things you mean and write it out. Every implicit case above gets read differently by different consumers. In each file the * group carries the default crawl rules, and vendor groups are the exceptions. The shared token only reaches participating providers, so each file also sets today’s vendor controls, which work now and keep working after the token ships.

No training by participating providers, and you accept losing Gemini grounding:

User-Agent: __usage_ai-training
Disallow: /
 
User-Agent: Google-Extended
Disallow: /
 
User-Agent: GPTBot
Disallow: /
 
User-Agent: *
Allow: /

No training except by Google, to stay in Gemini grounding. You cannot opt out of Gemini training and keep Gemini grounding. The closest file opts out with the token and GPTBot, and allows Google-Extended explicitly, which means Google may train:

User-Agent: __usage_ai-training
Disallow: /
 
User-Agent: Google-Extended
Allow: /
 
User-Agent: GPTBot
Disallow: /
 
User-Agent: *
Allow: /

Yes to everything:

User-Agent: __usage_ai-training
Allow: /
 
User-Agent: Google-Extended
Allow: /
 
User-Agent: *
Allow: /
 
Content-Signal: ai-train=yes, search=yes, ai-input=yes

That last one is my intent. My file today has the * group, the Content-Signal line and a sitemap, and no explicit groups, because it predates the token. Under normal robots.txt rules, the * group already allows Google-Extended. The explicit group is there for one word in Google’s post: sites that “specifically allow” Google-Extended are eligible for grounding, and the post does not say whether a default allow counts.

Two mechanics to know when you merge these into a file you already have: two groups for the same User-Agent are combined, not overridden by position, and within a group the longest matching path wins, with Allow winning a tie.

What can’t robots.txt tell you about AI crawler access?

Whether anyone is actually getting in. robots.txt is a preference you publish; the access you serve is decided at your edge. Last week, when I checked my own sites for leaked paths, the platform’s bot protection put the requests behind a challenge and the uptime monitor reported the site down until it lifted. A file that says yes to every AI crawler can sit in front of an edge that sometimes says no. If being read by AI systems matters to you, check your edge logs for the crawlers you allowed and measure whether answers cite you, rather than trusting the file.

Is the robots.txt training opt-out the part that matters for publishers?

My read is that for most sites it is the smaller half. Training changes what the next model generation knows, slowly and without a link back. Grounding is retrieval at answer time, the step where an answer can cite a page and send a click. That is the purpose the shared token does not cover, except at Google, where it will switch it off along with training.

My site sells nothing and runs no ads, so every agent that reads it is distribution, and yes-to-everything is the cheap choice. Nothing reads the token yet. Whatever you write today is a bet on how two vendors implement it next year.