Google and OpenAI's shared AI training control: a website-owner action plan.
Google and OpenAI plan to support a shared robots.txt signal for AI training preferences. See current controls, documented timing and what website owners should do now.
Web governance · Current analysisBy Orbital Content Team · Published October 30, 2026 · Updated November 1, 2026
AI-assisted research and drafting with editorial review. Google, OpenAI and RFC 9309 sources were checked November 1, 2026. Reported plans and crawler behavior remain statements from the named provider unless explicitly labeled Orbital analysis. Orbital did not participate in the IETF work, test the proposed signal, inspect Google or OpenAI crawler systems, verify adoption beyond Google and OpenAI, determine legal rights or obligations, or measure effects on search visibility, traffic, citations, model training or business results.
Google and OpenAI each announced on October 30 that they plan to support a shared signal in robots.txt through which website operators can express preferences about using public content for foundational generative AI training. Both providers say the effort is intended to reduce the need to maintain a separate instruction for every participating provider.
The most important word is plan. Google aims to support the control in early 2027. OpenAI says implementation will occur in the coming months. Neither provider says the shared signal should replace its current controls today: Google-Extended and OpenAI's GPTBot controls remain relevant until dated implementation guidance says otherwise.
For a business, this is a governance change to prepare for—not a reason to paste an unverified directive into production. Training preference, AI-search visibility, user-requested access and protection of private content are four different decisions.
What Google and OpenAI announced—and what they did not
Google and OpenAI's October 30 announcements support a shared robots.txt signal for foundational generative AI training preferences and connect it to work in the IETF AI Preferences working group. OpenAI calls the effort voluntary. Google says it plans to interpret a shared opt-out as a Google-Extended disallow when Google-Extended is not explicitly specified, while preserving provider-specific choices.
| Documented statement | What it means now | What it does not establish |
|---|---|---|
| Google and OpenAI say they plan to support a shared training-preference signal. | Website owners have evidence of support from two providers and a reason to prepare a provider-neutral policy. | That either provider has implemented the signal today or that other providers will adopt it. |
| The control will be expressed in robots.txt. | The public crawler-policy file will be part of the mechanism. | That every provider will use identical precedence, scope, refresh timing or conflict handling. |
| Google aims for early 2027; OpenAI says the coming months. | Both announcements are roadmap commitments rather than current production behavior. | An exact effective date, synchronized rollout, enforcement test or service-level promise. |
| Existing provider-specific controls continue. | Current Google-Extended, GPTBot and OAI-SearchBot decisions should be preserved until implementation guidance is effective. | That the shared signal replaces provider-specific rules. |
| The work draws on ongoing IETF discussions. | The proposal is being developed through an open standards process. | That the linked drafts are final standards or that their current text will remain unchanged. |
Google's current common-crawlers documentation identifies the proposed token as __usage_ai-training and says Google intends to support it in early 2027. The October 30 Google announcement says an opt-out would be treated like a Google-Extended disallow when Google-Extended is not explicitly specified; an explicit Google-Extended choice can still tailor Google's behavior. This is Google's announced interpretation, not proof of current enforcement or a universal precedence rule.
The IETF Datatracker pages linked by OpenAI presented automated-browser verification during this review, so Orbital did not reproduce or interpret the draft syntax. That is a meaningful evidence limit. Website owners should wait for accessible, dated implementation documentation from the provider they intend to control rather than copying a token from commentary, screenshots or an earlier draft.
Separate the four policy layers
A single phrase such as “block AI” hides several independent choices. Create a decision record for each layer.
| Layer | Business question | Current evidence |
|---|---|---|
| Foundation-model training | May a provider use automatically crawled public content to train generative AI foundation models? | OpenAI currently describes GPTBot as the crawler for content that may be used in foundation-model training; the common signal is planned. |
| AI search and discovery | May the site be crawled for inclusion in ChatGPT search answers? | OpenAI says OAI-SearchBot manages search inclusion and is independent from GPTBot. |
| User-initiated retrieval | May an assistant retrieve a page because a user requested it? | OpenAI says ChatGPT-User is used for certain user actions and that robots.txt rules may not apply because those actions are user initiated. |
| Access control | Who is actually authorized to view customer, employee, account or confidential content? | RFC 9309 says robots rules are not access authorization and are not a substitute for security controls. |
This separation matters for marketing. A company may prefer not to contribute content to foundation-model training while still wanting public pages eligible for AI search. Another company may allow broad public crawling but require login, authorization and no public URL for client documents. Those are different configurations and risk decisions.
Audit current provider controls before changing them
Google-Extended and OpenAI's documented controls do different jobs today. Preserve both providers' current behavior until their dated shared-signal implementation guidance is active.
OpenAI's crawler documentation currently identifies three relevant user agents:
- GPTBot is described as crawling content that may be used to make OpenAI's generative AI foundation models more useful and safe. OpenAI says disallowing GPTBot indicates that site content should not be used for foundation-model training.
- OAI-SearchBot is described as the crawler used to surface websites in ChatGPT search features. OpenAI says sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers, although they may still appear as navigational links. The documentation says systems may take about 24 hours to adjust after a robots.txt update.
- ChatGPT-User is described as supporting certain user-initiated actions rather than automatic web crawling. OpenAI says robots.txt rules may not apply to those requests and directs website owners to OAI-SearchBot for search opt-outs.
OpenAI also says these settings are independent. Allowing OAI-SearchBot does not require allowing GPTBot. The company may use one crawl for both permitted purposes to avoid duplicative crawling, but the permission choices remain separate.
Google says Google-Extended is a standalone robots.txt product token rather than a separate HTTP user agent. It controls whether content Google crawls may be used to train future Gemini models and for grounding in Gemini Apps and Grounding with Google Search on Vertex AI. Google says Google-Extended does not affect inclusion or ranking in Google Search. Its October 30 announcement additionally says sites that specifically allow Google-Extended are eligible to appear in and receive clicks through grounding in Gemini Apps. Eligibility is not a visibility or traffic guarantee.
Before any edit, save a dated copy of the current robots.txt file and record:
- every user-agent group and applicable allow or disallow rule;
- whether a wildcard group also applies;
- the sitemap locations and any other records;
- the business owner for search, training and security decisions;
- which platform, plugin, CDN or deployment process generates the file;
- the production hostnames covered by each file;
- the last change, reason, reviewer and rollback method; and
- any conflict between the file, CDN bot controls, application middleware and vendor settings.
Do not assume the copy visible in a CMS is the production response. Fetch the public /robots.txt from each hostname and verify the HTTP status, content type and actual text after the delivery network and application rules have run.
Classify content before writing crawler policy
A whole-site rule may be simple, but it can erase useful distinctions. Inventory content by business purpose and sensitivity.
| Content class | Questions to answer | Likely control owner |
|---|---|---|
| Public marketing pages | Should the pages be discoverable in AI search? Is the training preference different? | Marketing, SEO and brand. |
| Editorial articles and guides | What reuse preference applies? Are licenses, syndication or contributor terms relevant? | Editorial, legal and marketing. |
| Product documentation | Does broad discovery support customers? Are versions, code samples or licensed components involved? | Product, support and legal. |
| Paid or licensed content | Is the page public, preview-only or access controlled? What rights were granted by authors and licensors? | Product, legal and finance. |
| Client, employee or account content | Why is it reachable publicly at all? Which authenticated users and services need access? | Security, privacy, IT and the data owner. |
| Staging, test and administrative routes | Should the environment be public? Does it contain real data, secrets or unfinished claims? | Engineering and security. |
Training preference should follow the company's rights and policy, not the technical convenience of one rule. If content includes licensed images, contractor writing, customer material or data from another provider, confirm that the business has authority to express the intended preference and that the public page itself is appropriate.
Use Orbital's website maintenance checklist to assign ownership and change records. The website redesign SEO checklist can help prevent crawler-policy changes from being lost or broadened during a migration.
Protect search and measurement decisions
The shared signal described by Google and OpenAI concerns foundational generative AI training. It should not be treated as a universal search control, ranking switch or traffic guarantee. OpenAI separates GPTBot from OAI-SearchBot, while Google says Google-Extended does not affect inclusion or ranking in Google Search. That distinction belongs in the acceptance test.
Before implementation, define the desired state for:
- traditional search crawling and indexing;
- ChatGPT search eligibility through OAI-SearchBot;
- foundation-model training preference;
- user-initiated retrieval;
- licensed feeds, APIs or syndication; and
- private authenticated content.
Do not promise that allowing a crawler will create citations, referrals, rankings or revenue. Do not claim that a disallow rule removes content already collected, controls every provider or prevents a person from copying a public page. Those outcomes require provider-specific evidence and, where relevant, legal review.
Measure changes with dated evidence: raw robots.txt responses, server or CDN logs where lawfully available, search-console data, analytics referral data, crawl error reports and periodic spot checks. Correlation after a rule change does not prove causation. AI answer visibility is also not the same as a conventional search ranking.
The SEO content strategy checklist can connect crawl policy to content ownership, sources, internal linking and measurement without inventing search demand or outcomes.
Robots.txt is public policy, not protection
RFC 9309 standardizes the Robots Exclusion Protocol for crawler access and says crawlers are requested to honor its rules. It explicitly says those rules are not access authorization. Its security section warns that listing paths in robots.txt exposes them publicly and makes them discoverable, and recommends actual application-layer security for restricted content.
Therefore:
- do not list a secret route in robots.txt as a way to hide it;
- do not publish passwords, keys, private tokens or personal data in the file;
- use authentication and authorization for protected content;
- remove sensitive pages from public hosting and caches where appropriate;
- separate production, staging and administrative access;
- verify storage, backups, logs and third-party integrations; and
- treat crawler compliance as voluntary policy behavior, not a security boundary.
RFC 9309 also allows crawlers to interpret records outside the standardized user-agent, allow and disallow fields, provided other records do not interfere with the defined protocol. That general mechanism does not tell a website owner how OpenAI's future common signal will be parsed. Provider implementation guidance is still required.
Prepare a safe implementation without guessing the syntax
The right preparation is procedural:
- Name owners. Assign one accountable person for the content policy, one for SEO and discovery, and one for technical deployment. Add legal or rights review where the content or decision requires it.
- Write the desired state. For every content class, state the training, search, user-retrieval and access-control decision separately.
- Preserve current rules. Keep provider-specific controls until each provider documents how the common signal interacts with them.
- Wait for dated implementation documentation. Google has named a proposed token and early-2027 target; OpenAI has said the coming months. Record each provider's effective date, final syntax, scope, precedence and known limitations before deployment.
- Stage the change. Validate formatting in a non-production copy and confirm that existing user-agent groups, sitemaps and path rules remain parseable.
- Review the diff. Require marketing or editorial approval for policy, technical review for syntax and security review for any newly exposed path.
- Deploy with rollback. Keep the previous file, deployment identifier and owner who can reverse the change.
- Verify the public response. Check status, encoding, content type, text and the result from every applicable hostname.
- Monitor for one full crawl cycle and beyond. OpenAI currently says search changes may take about 24 hours; neither provider has documented a production refresh interval for the future shared signal.
- Review quarterly and after provider changes. Crawler names, documentation, policy and business preferences can change.
A CMS plugin or CDN switch can reduce manual editing but does not own the policy decision. Confirm what it writes, which hosts it affects, how it handles conflicts and whether it can restore the previous version.
Build an evidence register for the unknowns
OpenAI's announcement leaves reasonable questions unanswered. Record them instead of filling the gaps with assumptions.
| Question | Why it matters | Acceptable evidence |
|---|---|---|
| Will each provider use the announced token and interpretation in production? | A draft token, example or precedence rule may change before rollout. | Dated Google and OpenAI implementation documentation tied to production behavior. |
| When does each provider begin honoring it? | Publishing a preference before enforcement may create false confidence. | Exact effective dates or rollout notices; Google currently targets early 2027 and OpenAI says the coming months. |
| How does it interact with Google-Extended and GPTBot? | Explicit and shared directives need predictable precedence. | Provider documentation and reproducible tests for each provider. |
| Does it apply by origin, path or another scope? | Businesses may want different policies for different content classes. | Normative syntax plus provider-specific examples. |
| How quickly are changes observed? | Rollout and rollback need a realistic verification window. | Documented refresh behavior and server-log observations. |
| Which other providers honor the same version? | “Common” does not mean universal. | Each provider's own dated implementation statement. |
| What happens to previously collected content? | A prospective crawler preference may not address historical data. | Provider policy or a written response applicable to the site owner. |
Absence of public evidence is not proof of unfavorable behavior. It is a reason to avoid a production claim until the provider publishes the missing information.
A bounded rollout after implementation details exist
- Days 1–3 — inventory. Capture current robots.txt responses, platform generators, CDN controls, sitemaps, crawler rules and responsible owners.
- Days 4–7 — decide. Classify content and approve separate positions for training, AI search, user retrieval and protected access.
- Wait for provider implementation evidence. Do not force the calendar. Record each provider's effective syntax, date and interaction with Google-Extended, GPTBot and other existing controls when published.
- Days 8–10 — stage and review. Produce one proposed file, validate the diff and confirm no existing search or security control is weakened.
- Days 11–14 — deploy and verify. Publish through normal change control, fetch the public file, inspect logs and keep a rollback path.
- Days 15–30 — observe. Record crawler behavior and search or referral changes without claiming causation. Resolve discrepancies with the provider or platform before broader changes.
Stop and roll back if the file becomes unavailable, intended search crawling is blocked, a protected path is exposed, the delivered text differs from the approved version, or ownership of the policy cannot be established.
Orbital's business take
A provider-neutral training-preference signal could reduce operational clutter for website owners. Separate October 30 announcements from Google and OpenAI make the direction more credible, but two announced implementations are not a universal standard or current enforcement.
The immediate opportunity is governance, not deployment. Decide what the business wants for public marketing content, editorial work, licensed material and private data. Keep AI training separate from Google Search, ChatGPT search, Gemini grounding and user-initiated retrieval. Treat robots.txt as a public request, never as a lock. Then wait for dated production documentation from each provider before editing the live file.
Explore Orbital's marketing strategy, website design, AI automation, tech stack consulting and connected-business planning. A 60-minute consultation can turn one website-governance question into a decision brief; implementation, monitoring, legal review and provider-specific work are separately scoped.
Sources and editorial limits
Primary sources were checked November 1, 2026. Google and OpenAI each published announcements dated October 30 supporting a shared robots.txt signal for foundational generative AI training preferences. Google aims for early 2027 and documents a proposed __usage_ai-training token plus an announced interaction with Google-Extended; OpenAI says implementation will occur in the coming months. Both preserve current provider-specific controls. Google's crawler documentation says Google-Extended does not affect Google Search inclusion or ranking. OpenAI's crawler documentation distinguishes GPTBot, OAI-SearchBot and ChatGPT-User and says the controls are independent. RFC 9309 is the September 2022 standards-track specification for the Robots Exclusion Protocol; it says robots rules are not access authorization and are not a substitute for security. The linked IETF draft pages presented automated-browser verification during this review, so Orbital did not reproduce or interpret their draft syntax. The policy layers, audit, content classification, evidence register and rollout process are Orbital editorial recommendations. Public drafts, documentation, crawler behavior, provider participation and business requirements can change. Orbital did not test the proposed signal, inspect Google or OpenAI systems, verify adoption beyond those providers, determine legal obligations or guarantee crawling, training, grounding, search visibility, traffic, citations, security or business results. This is business analysis, not legal, privacy, security or SEO-performance advice.
Primary source links
- Google Search Central: Supporting a shared AI training control for websites
- Google Crawling Infrastructure: Google-Extended and future shared-signal support
- Google Crawling Documentation: October 30, 2026 changelog
- OpenAI: Making it easier for websites to express AI training preferences
- OpenAI Developers: Overview of OpenAI crawlers
- RFC Editor: RFC 9309, Robots Exclusion Protocol
- IETF Datatracker: proposed AI training preference signal
- IETF Datatracker: AI preferences vocabulary work