Analysis
One robots.txt signal to opt out of AI training: what OpenAI, Google, Microsoft and Anthropic back
A draft proposes one robots.txt token, __usage_ai-training, to refuse AI training. What it covers, what it does not, and when Microsoft's side starts.
As of October 31, 2026, the proposal is a robots.txt group named __usage_ai-training, described in an IETF draft dated October 30. According to SiliconANGLE, Google, Microsoft, Anthropic and OpenAI back an upcoming standard on this basis. It is not yet a standard: the draft is individual and not endorsed by the IETF, and the signal is a preference that crawler operators follow voluntarily.
The essentials
- What it is: an ordinary robots.txt group whose name is
__usage_ai-training. No new directive is added. - What it covers: the use of content for AI training. It does not control crawling, search or grounding.
- Where things stand: an individual draft. Microsoft confirms its own dates (February 1, 2027). Google is said to follow in 2027 and OpenAI in the coming months, according to the press. Meta has not committed.
- Limits: it states a preference. Compliance is voluntary and nothing guarantees exclusion from training.
What exactly do these companies back?
The text is "Machine-Readable Training Preference Signal" (draft-kelley-aipref-training-preference-signal-00), by Kevin Kelley of Anthropic. It is dated October 30, 2026 and expires on May 3, 2027. It is an "Individual Internet-Draft" aimed at the Standards Track, and the draft states that it is not endorsed by the IETF (IETF Datatracker).
According to SiliconANGLE (October 30, 2026), Google, Microsoft, Anthropic and OpenAI support an upcoming standard. A single robots.txt entry would block all crawlers collecting training data for generative models. It would not apply to models doing "exclusively non-generative tasks". Meta expressed support for the idea but has not committed (SiliconANGLE).
Only Microsoft's calendar is confirmed by a primary source we read.
How do you write it in robots.txt?
The draft's own example:
User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/Disallow expresses a preference against using the content for training, and Allow a preference for it. The rules are those of RFC 9309, and the longest match wins. Section 4.2 of the draft says a generic User-Agent: * group does not apply to this token, so the group has to be written explicitly.
The meaning follows the "AI Training" definition of the IETF's AI Preferences working group (IETF AI Preferences).
Source: IETF draft (October 30, 2026), Bing blog, verified 2026-10-31.
What does it cover, and what does it not?
Keep three things apart: training (modifying a generative model's learned parameters), search (indexing and ranking) and grounding (using a page to answer a question).
The draft states that the token is "not a directive for crawling behavior". To limit crawling, it points to per-crawler groups such as GPTBot. Private agreements may prevail (section 4.5).
Microsoft draws the same line on the Bing blog. It distinguishes "AI Training" from "AI Grounding", and classic search indexing and ranking are not grounding. Training uses the __usage_ai-training token, with Allow as the default and Disallow as the opt-out. Grounding has no robots.txt setting: it is set page by page with a meta tag (name="Bingbot") or an X-Robots-Tag header, with values such as ai-grounding:disallow or ai-grounding:disallowextended. The two settings are independent, and a page-level setting prevails over robots.txt (Bing blog).
Microsoft adds that these preferences do not change eligibility for classic Bing results, and that a robots.txt blocking Bingbot still blocks crawling: the setting governs use, not access. Refusing grounding may reduce brand visibility in AI experiences. It guarantees no inclusion, traffic or payment and creates no licence.
For OpenAI, the press says the token does not replace existing controls. OpenAI's existing documentation says refusing GPTBot signals that content should not train its generative foundation models, that OAI-SearchBot handles search separately, and that ChatGPT-User, which acts on a user's request, may not follow robots.txt (OpenAI).
The signal has no legal force on its own, which depends on each country's law. It does not erase content already used.
When does it take effect?
On Microsoft's side, according to its blog: temporary grounding settings arrive in Bing Webmaster Tools on December 15, 2026, only for sites using NoArchive or NoCache on that date. The new training and grounding preferences start applying as crawls happen on February 1, 2027. On March 1, 2027, NoArchive and NoCache are no longer read as training or grounding preferences. The temporary grounding settings end on September 30, 2027.
For the others, only the press speaks. According to SiliconANGLE, Google will follow "next year" (2027) and OpenAI plans similar controls "in the coming months". We found no implementation statement from Anthropic in the sources we read, so we state none.
Source: IETF draft, Bing blog, SiliconANGLE, verified 2026-10-31. Past facts are shown filled, expected dates outlined.
Comparison table
| Company | Position on Oct. 31, 2026 | Timeline | Source |
|---|---|---|---|
| Microsoft | Documents the __usage_ai-training token and separate grounding settings | Feb. 1, 2027: preferences start to apply | Bing blog |
| Backs the upcoming standard, according to the press | "Next year" (2027), according to the press | SiliconANGLE | |
| OpenAI | Backs it; plans similar controls, which do not replace existing ones, according to the press | "In the coming months", according to the press | SiliconANGLE |
| Anthropic | Backs it, according to the press; its employee is the draft's author | No implementation statement found | SiliconANGLE, IETF draft |
| Meta | Expressed support for the idea, no commitment | None | SiliconANGLE |
Frequently asked questions
Do I need to change my robots.txt now?
Nothing in the sources we read promises immediate compliance. The token does not replace per-crawler rules such as GPTBot, so existing groups stay relevant. This is not legal advice.
Does the signal block crawling?
No. Section 4.4 of the draft says it is not a directive for crawling behavior. To limit crawling, use per-crawler groups.
Is it a standard?
Not yet. It is an individual Internet-Draft, not endorsed by the IETF, with no formal status, which expires on May 3, 2027.
Method and limits
This article relies on the IETF draft, the Bing blog, SiliconANGLE and OpenAI's bot documentation, read on October 31, 2026. The OpenAI announcement page cited by the press returned a 403 error and was not read: what we report about OpenAI's plans comes from SiliconANGLE and from the existing documentation. Google's "next year" and OpenAI's "in the coming months" also come from the press, not from a company page we read. This is not legal advice.
Sources
- IETF: Machine-Readable Training Preference Signal (draft) (verified 2026-10-31)
- Microsoft, Bing blog: clearer preferences for training and grounding (verified 2026-10-31)
- SiliconANGLE: Google, Anthropic and others to support a new AI crawling opt-out standard, October 30, 2026 (verified 2026-10-31)
- OpenAI: bot documentation (verified 2026-10-31)
- IETF: AI Preferences working group, vocabulary draft (verified 2026-10-31)
Spotted an error in this article? Report it · Our corrections policy
