Web publishing · Reporting
OpenAI and Google back a shared training signal. Keep search controls separate.
The October 30 announcements promise simpler AI training preferences. Publishers should distinguish planned support, search discovery and user-triggered access.
A shared preference is coming; existing controls still matter
A publisher may want an article discoverable in AI search while declining its use for model training. OpenAI and Google both announced support plans on October 30 for a shared training-preference signal in robots.txt. That could reduce the work of maintaining a separate training rule for every participating provider. It does not remove the need to decide how the same page should behave in search or when a person asks an assistant to visit it.
The timetable is the first detail to check. OpenAI says it aims to implement the signal in the coming months and will publish implementation details later. Google's announcement targets 2027; its crawler reference specifies early 2027. Both describe continued company-specific controls. The useful action today is to inventory those controls and record the intended outcome for each, rather than assume the announcement has already changed how a live site is fetched.
From a provider announcement to a website decision
- 1Check the timetable
OpenAI says coming months; Google's crawler documentation says early 2027.
- 2Name the intended use
Training preference, search visibility and a user-requested fetch need separate decisions.
- 3Verify the current rule
Read each provider's existing documentation before changing robots or edge access.
For OpenAI, training and search have different bot controls
OpenAI's crawler documentation distinguishes GPTBot, which concerns content that may be used in foundation-model training, from OAI-SearchBot, which supports ChatGPT search discovery. It explicitly describes these settings as independent. A site can allow the search crawler while disallowing the training crawler. That is a more precise configuration choice than a blanket rule based only on the provider's name.
User-requested access adds another distinction. ChatGPT-User handles certain visits initiated by users and is not the automatic crawler that determines search inclusion. OpenAI says robots.txt rules may not apply to these user-initiated actions. A training preference therefore should not be treated as a complete access-control system for every request made by an assistant.
Google-Extended has its own product scope
Google's crawler reference says Google-Extended does not affect inclusion in Google Search and is not a Search ranking signal. Its described scope includes future Gemini training and specified grounding uses. That last detail matters: reducing the question to training alone can obscure other uses covered by a provider's control. Read the affected-products description before choosing a rule.
Google's proposed shared-signal handling also has a specific condition. Its announcement says a training opt-out would be interpreted like disallowing Google-Extended when Google-Extended is not explicitly specified. Existing company-specific choices would continue alongside the shared signal. This is a plan described by Google, not evidence that every other AI provider will use identical precedence or timing.
What each existing control actually addresses
Scroll sideways to read all columns.
| Purpose | OpenAI example | Google example / distinction |
|---|---|---|
| Training-related use | GPTBot | Google-Extended also covers specified grounding uses |
| Search discovery | OAI-SearchBot | Googlebot; Google-Extended is not a Search ranking signal |
| User-requested visit | ChatGPT-User; robots.txt rules may not apply | Do not infer user-fetch behavior from training controls |
Write a three-part request policy before editing the file
Start with a small inventory for one representative public article: whether training-related use is acceptable, whether search discovery is wanted, and how user-requested visits should be handled. Beside each choice, record the provider, documented control, source date and expected effect. This makes a disagreement about editorial policy visible before it becomes an accidental broad crawler block.
Then inspect the actual robots.txt response and any firewall rule affecting those requests. A crawler allowance in one layer cannot establish that another layer permits the request. Preserve the previous configuration and inspect request evidence after an authorized change. Keep the operational claim narrow: a rule being served is evidence of configuration, while discovery, citation and visits need their own observation.
Sources & verification
Provider documentation supports the facts referenced below. Our explanations and recommendations are editorial interpretation.
- Making it easier for websites to express AI training preferences ↗Checked Nov 1, 2026
- Google: Supporting a shared AI training control for websites ↗Checked Nov 1, 2026
- OpenAI: Overview of crawlers ↗Checked Nov 1, 2026
- Google: List of common crawlers ↗Checked Nov 1, 2026
Corrections: hello@aiapicost.com
