OpenAI, Google, Microsoft and Anthropic Back One Opt-Out From AI Training. What It Covers, What It Doesn’t, How to Set It

By Thorsten Meyer | October 31, 2026 | ThorstenMeyerAI.com

Buying for a business?Offer from Amazon

Get business pricing on tech for your team

• Business-only prices and quantity discounts
• Tax-exempt purchasing
• Multiple users, one account, clear invoices

Create a free Business account →Free to register

As an affiliate, we earn on qualifying purchases.Hide

If you run a website and do not want AI companies to train on it, you have needed a list. One rule for OpenAI’s crawler, GPTBot. One for Google-Extended. One for Anthropic’s ClaudeBot. For Microsoft, special page tags called NoArchive and NoCache. Miss one company, and your “no” never reached it.

On Thursday, 30 October, OpenAI, Google, Microsoft and Anthropic each said they will honor one shared line in robots.txt instead. Write it once, and every participating company should read it as “do not train on this”. Meta’s communications chief said Meta supports the idea, but did not say whether Meta will honor it.

That sounds like a big win for publishers. It is a real step. It is also smaller than the headlines suggest: the signal is a request, not a lock, it covers training and not AI answers, it starts at different dates for each company, and it is not yet an official internet standard. This piece explains what changes, what does not, and how to set it, working from the companies’ own announcements, the Internet-Draft itself and the working group’s public records.

Two disclosures. The draft was written by a standards lead at Anthropic, and this article was researched and drafted with AI assistance, including Anthropic’s Claude. I treat Anthropic’s statements exactly like everyone else’s. And thorstenmeyerai.com belongs to a network of publisher websites, so how AI companies use web content affects us too. I mark company claims, documented facts and my own reading as such.

Key takeaways

• What it is: a shared robots.txt rule group, User-Agent: __usage_ai-training plus Disallow: /. It expresses a preference against using your content to train generative AI models. It comes from an individual Internet-Draft by Kevin Kelley of Anthropic, published 30 October.
• Who said what: Anthropic says it “has already started implementing support”. OpenAI aims to implement it “in the coming months”. Microsoft will honor it from 1 February 2027. Google aims for 2027; its crawler documentation says “early 2027”. Meta voiced support without an implementation commitment.
• What it does not do: it does not block crawling, it does not remove content from models already trained, it does not cover AI answers built from live web content, and it does not create payment. Microsoft: “Neither choice guarantees inclusion, traffic, or payment.”
• What stays the same: existing company rules (GPTBot, Google-Extended, ClaudeBot) keep working. Microsoft says these preferences do not change eligibility for Bing search results, and Google-Extended does not affect Google Search.
• Status: the draft “has no formal standing in the IETF standards process”. At the working group’s meeting on 21 October, one of its chairs said: “We haven’t declared consensus on anything yet.”
• For companies: decide three things separately: training, AI answers, search. Add the new line if you want out of training, keep the per-company lines until each company has switched, and plan for Microsoft’s deadline of 1 March 2027 if you use NoArchive or NoCache.
• For users: if you publish anything, these are two lines you can add. It is not the setting for your own chats with an AI assistant. Those are account settings.

Products Worth Considering

LG UltraGear 32GX850A-B 32-Inch 4K UHD Glossy OLED Gaming Monitor

AmazonView Latest Price

Amazon Kindle Scribe Colorsoft 64GB (1st Generation)

AmazonView Latest Price

Bose QuietComfort True Wireless Earbuds with Active Noise Cancellation, Black

AmazonView Latest Price

As an affiliate, we earn on qualifying purchases.

What was announced, company by company

All four announcements point to the same document: draft-kelley-aipref-training-preference-signal-00, titled “Machine-Readable Training Preference Signal”. It builds on the AI Preferences working group (AIPREF) at the Internet Engineering Task Force (IETF), the body that writes many of the internet’s technical standards. The story was first reported by Search Engine Roundtable and reported by The Verge and AFP, as published by The Straits Times.

• Google: In a Search Central blog post, Google says it plans to support the shared control. If a site opts out with the new token and has no rule for Google-Extended, Google “plans to interpret that as if the publisher had disallowed Google-Extended”. Google “aim[s] to support this new control along with other AI providers in 2027”; its crawler documentation now says “early 2027”.
• OpenAI: OpenAI says it “will honor a common signal” about “foundational generative AI training”, calls it a “voluntary effort” and aims to implement it “in the coming months”. Its existing crawler controls “will continue to apply”.
• Microsoft: The Bing blog goes furthest. Microsoft adopts the shared training preference and adds a separate page-level preference for “AI Grounding”, meaning web content used to inform AI answers in Bing, Copilot and Microsoft 365. Both take effect from 1 February 2027 as Microsoft crawls. They replace NoArchive and NoCache, which stop counting as a training or grounding preference after 1 March 2027.
• Anthropic: Kelley announced the draft on the IETF mailing list and wrote that Anthropic “has already started implementing support for this draft”, adding: “It’ll take us some time to add support across our services.” On 31 October, Anthropic’s crawler help page still described only its per-crawler controls.
• Meta: Spokesperson Andy Stone wrote on X that Meta “strongly support[s] publishers’ ability to express preferences” and called a common signal “an important step”. As The Verge notes, he did not say whether Meta will honor it.

Which other AI companies plan to honor the signal is open. I found no statement from Apple, Amazon, xAI or Perplexity in the sources I reviewed. The draft’s acknowledgments name people from Amazon Web Services, Meta, Cloudflare, Common Crawl, publishers such as News Corp, the Financial Times and the BBC, and groups such as the Electronic Frontier Foundation and Creative Commons. The draft stresses that affiliations are listed “for identification only”. It is a list of contributors, not of supporters.

How the signal works, in plain words

Robots.txt is a small text file at the root of a website, like a note on the front door. It tells automated visitors, called crawlers, which rooms they may enter. Until now it mostly answered one question: may this crawler fetch this page?

The new signal answers a different question: what may you do with the page once you have it? The trick is a made-up visitor name, __usage_ai-training. No crawler actually calls itself that. The draft calls it a “policy token”. Rules written under that name do not open or close doors. They express a preference about use. Here is the draft’s own example:

User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/

This reads as: please do not train on anything on this site, except the press kit, where training is welcome. Three design choices matter for site owners.

• The catch-all rule does not count. Many sites already have a group for User-Agent: *, which means “all crawlers”. The draft says such wildcard groups “MUST NOT be interpreted as matching the policy token”. If you want to say no to training, you must name the token.
• Silence is not “no”. “If no group contains the __usage_ai-training policy token”, the draft says, “then no preference has been expressed.” Microsoft states it plainly: training is allowed by default.
• It is about use, not access. The token “is not a directive for crawling behavior”. A company that crawls only for training “MAY” skip pages marked “no”, but nothing requires it. If you want to keep a crawler out, you still need a rule for that crawler’s own name.

Two more details. “AI training” has a narrow meaning here, taken from the working group’s vocabulary draft: “using an asset to modify the learned parameters of a generative AI model”. Training models used only for ranking or classification is not included. And company-specific rules can win: the draft says “specific expressions SHOULD take precedence over general ones”, and “private agreements MAY override” the signal. So a site that licenses its content to one AI company can still say no to everyone else.

What the signal does not do

Read the announcements closely and the limits are written into them.

• It is not a lock. The IETF vocabulary draft says plainly that it does not “ensure that preferences are followed”. OpenAI calls its support a “voluntary effort”. A scraper that ignores robots.txt today will ignore this line too. Cloudflare made the same point about its own similar convention: such signals “are not technical countermeasures against scraping”.
• It is not retroactive. Microsoft describes the preferences as covering “future AI generative model training”. Content already in a training set stays there.
• It does not cover AI answers. Training changes a model. Grounding feeds live web pages into an answer, the AI Overview or the chatbot reply you read today. Each company handles that separately. Microsoft adds its own grounding setting. Google offers a separate Search generative AI control in Search Console for AI Overviews and AI Mode, and says that control “doesn’t affect AI training”. OpenAI uses OAI-SearchBot for ChatGPT search, Anthropic uses Claude-SearchBot and Claude-User.
• It may not reach user-requested visits. OpenAI’s crawler documentation says that for ChatGPT-User, which fetches a page when a person asks, “robots.txt rules may not apply”. How to treat such uses is still debated in the working group.
• It does not pay you. Microsoft: the preferences “do not establish licensing or payment arrangements”. For payment, publishers look elsewhere, for example the Really Simple Licensing standard mentioned by The Verge and AFP.

One trade-off deserves attention. Google-Extended, the token Google will map the new signal to, covers both training of Gemini models and grounding in Gemini Apps. Google writes that sites “that specifically allow Google-Extended are eligible to appear in responses and receive clicks via grounding in Gemini Apps”. My reading: a blanket “no training” line, without a separate Google-Extended rule, may also cost you visibility in Gemini Apps. AFP reports that Microsoft and Google both warn that opting out can reduce how often content surfaces in AI apps.

Not a standard yet

The draft is labelled an “Internet-Draft (I-D)” that “is not endorsed by the IETF and has no formal standing in the IETF standards process”. It is version 00 and expires on 3 May 2027 unless it is updated. Kelley presents it as input to the working group’s work on attaching preferences to content.

The vocabulary underneath is also still moving. At the working group’s interim meeting on 21 October, one of the chairs said: “We haven’t declared consensus on anything yet.” The vocabulary draft carries notes that key sections do “not yet have consensus”, and the minutes show open debates over where AI answers end and user requests begin. The next sessions are scheduled for 16 and 19 November at IETF 127 in San Francisco.

Why ship before the standard? Microsoft says the work “represents significant progress that merits adoption”. Google calls it a way to give site owners “early and expanding access to the preference signal”. My reading: running code creates facts. Once four large companies honor the line, it becomes hard for any final standard to ignore it.

Is it legally binding?

On its own, no. It is a voluntary signal. What it means in law depends on where you are, and none of this is legal advice.

In the United States, there is no general opt-out rule for AI training in copyright law. The fights run through the courts. The New York Times sued OpenAI and Microsoft in December 2023, alleging unauthorised use of millions of articles. The companies invoke fair use, AFP notes. Those are claims, not rulings.

In the European Union, the situation is different. The copyright directive of 2019 allows text and data mining unless rightholders have reserved their rights “in an appropriate manner, such as machine-readable means in the case of content made publicly available online” (Directive (EU) 2019/790, Article 4(3)). The EU AI Act requires providers of general-purpose AI models to have a policy that respects such reservations. The voluntary Code of Practice, copyright chapter, commits signatories to crawlers that follow robots.txt “as specified in” the IETF’s RFC 9309, “and any subsequent version of this Protocol for which the IETF demonstrates that it is technically feasible”, plus other machine-readable protocols that standards bodies have adopted or that are widely adopted. It also encourages signatories that run search engines to make sure such reservations do not hurt indexing. Google says its plan “is consistent with” this code.

My reading: the new line sits in robots.txt, the file the code names, so it has a good chance of counting as a machine-readable reservation in the EU. But it is an individual draft, not an RFC, and I know of no court that has ruled on it. Cloudflare’s separate Content Signals convention from September 2025 already declares itself an express reservation under Article 4. Rightsholders who care should ask counsel whether to use both.

How to set it: a short guide for site operators

These steps follow the official documentation from Google, Microsoft, OpenAI, Anthropic and the draft. Check them against your own setup before you deploy.

1. Decide three things separately. Training of future models. Use in AI answers. Classic search. One setting no longer covers all three, and saying no to one does not have to mean no to the others.
2. Say no to training once, for every participating company. Add this group to the robots.txt at the root of each host, including every subdomain you want covered:

User-Agent: __usage_ai-training
Disallow: /

3. Keep your company-specific lines for now. Only Anthropic says it has started. OpenAI follows “in the coming months”, Microsoft from 1 February 2027 and Google in 2027. Until then, rules for GPTBot, Google-Extended and ClaudeBot are what those companies already read. They also stay valid afterwards, and the draft lets companies treat them as more specific.
4. Use exceptions where you want training. A press kit, documentation you want models to know, or a product catalogue can be carved out with Allow: lines, as in the draft’s example.
5. Set AI answers separately. For Microsoft, page by page, with a meta tag or an HTTP header. Microsoft also offers ai-grounding:disallowextended, which keeps grounding in Microsoft’s own products but not in grounding services it sells to others. For Google Search, use the Search generative AI control in Search Console. For ChatGPT search and Claude, use the rules for OAI-SearchBot and Claude-SearchBot.

<meta name="Bingbot" content="ai-training:disallow">
<meta name="Bingbot" content="ai-grounding:disallow">

X-Robots-Tag: Bingbot: ai-training:disallow
X-Robots-Tag: Bingbot: ai-grounding:disallow

6. If you use NoArchive or NoCache today, publish the new preferences before 1 March 2027. After that date Microsoft no longer reads the old tags as a training or grounding preference. Microsoft’s temporary grounding setting in Bing Webmaster Tools expires on 30 September 2027.
7. If you want to keep crawlers out entirely, a preference is not enough. Use rules for each crawler’s own name and, if needed, bot management at your host or CDN.
8. Keep a dated copy of your robots.txt. The draft ties preferences to the file “in effect when the asset was retrieved”. My suggestion, not a legal requirement: a record of when you said no can matter later.

User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/

What this means for companies

• Publishers and media: the cost of saying no to training drops to two lines. The harder question is the one this signal does not answer: AI answers that summarise your work and send fewer clicks. Reddit’s move to a login wall shows where that pressure leads: Reddit Is Putting Up a Login Wall. Decide training and grounding separately, and measure what each choice does to your traffic.
• Shops and brands: product pages usually want to be found, including by AI assistants and shopping agents. A blanket “no” may cost more visibility than it protects. Consider allowing product and help pages while saying no to paid content. How AI agents change shopping is covered in Amazon Blocks AI Shopping Agents, Walmart Lets Them In.
• SEO teams: Microsoft says the preferences do not change eligibility for Bing search results, and Google-Extended does not affect Google Search. That separation is the point. The trade-off sits in AI features, where Google is now putting ads into answers: When the Answer Carries the Ad.
• Companies that build or buy AI models: if you train models or license training data, ask your data suppliers whether they record and honor this signal. The working group is discussing how preferences should travel with scraped data. In its October minutes, one chair summed it up: “If you’re training models, wilful blindness isn’t compliance.”
• Hosting and platform providers: many site owners never touch robots.txt themselves. If you manage it for customers, add the option, and say clearly what it does and does not cover.

What this means for users

• If you have a blog or website: you can now say “no training” once instead of tracking every AI crawler. Check whether your site builder lets you edit robots.txt. Remember it is a request, and it only works for companies that honor it.
• If you post on a platform: your posts are governed by the platform’s robots.txt and its licensing deals, not by you. The signal changes nothing about that.
• If you read AI answers: sources that opt out of grounding may appear less often. Answers can only cite what the systems are allowed to use.
• If you use an AI assistant: this signal is about websites. Whether your own chats are used for training is a separate setting in your account.

Honest limits

• Only Anthropic says it has started. OpenAI, Microsoft and Google describe plans and dates, and none has published how outsiders can verify compliance.
• I could not test whether any crawler already honors the new token.
• Meta’s statement is support for the idea, not a commitment to implement it. I found no statements from other AI companies.
• The draft and the vocabulary can still change. The working group has not declared consensus.
• The legal effect in the EU is my reading, untested in court. In the U.S., training disputes remain in litigation. Nothing here is legal advice.
• Microsoft’s page-level settings and Google’s Search Console control are product features that can change. Check the current documentation before you rely on them.

Sources

• IETF: draft-kelley-aipref-training-preference-signal-00, “Machine-Readable Training Preference Signal”, 30 Oct 2026, and the announcement on the ai-control list, 30 Oct 2026
• IETF: draft-ietf-aipref-vocab-08, “A Vocabulary For Expressing AI Usage Preferences”, 14 Sep 2026; AIPREF interim meeting minutes, 21 Oct 2026
• Google: Supporting a shared AI training control for websites, 30 Oct 2026; Google’s common crawlers (Google-Extended), updated 30 Oct 2026; Search generative AI control
• OpenAI: Making it easier for websites to express AI training preferences, 30 Oct 2026; Overview of OpenAI crawlers
• Microsoft Bing: Your content, your choice: clearer preferences for training and grounding, Oct 2026
• Anthropic: Does Anthropic crawl data from the web, and how can site owners block the crawler?, checked 31 Oct 2026
• Meta: Andy Stone on X, 30 Oct 2026
• European Union: Directive (EU) 2019/790, Article 4; General-Purpose AI Code of Practice, copyright chapter, July 2025
• Cloudflare: Giving users choice with Cloudflare’s new Content Signals Policy, 24 Sep 2025
• The Verge: The biggest names in AI are backing a new way to opt out of scraping, 30 Oct 2026
• AFP via The Straits Times: AI giants promise publishers new way to block some scraping, 31 Oct 2026
• Search Engine Roundtable: Google Adopts New AI Training Control, 30 Oct 2026

Researched and drafted with AI assistance, including Anthropic’s Claude.