Four AI companies back one robots.txt opt-out from AI training

Regulators, publishers and civil society groups had pressed for a simpler way to opt out of training, Google told Search Engine Roundtable.

Microsoft went further. Alongside the training signal, Bing’s post adds a page-level grounding setting, grounding being the use of a page to write an AI answer, with links back to it. Publishers can refuse training yet still be cited in Copilot, or limit grounding to Microsoft’s own products. Bing reads the new settings from 1 February 2027 and stops treating its old NoArchive and NoCache tags as AI preferences after 1 March. Microsoft is blunt about the cost: a brand that blocks grounding shows up less in those AI answers.

The token covers what crawlers collect from your site. What a team types into an assistant follows the plan instead: our 31 October check records Claude’s Team seat, $25 a month or $20 billed annually, with no model training on your content by default.

So once all four support the token, from 2027 for Google and Microsoft, one robots.txt line keeps a site’s pages out of their training, while on Bing a brand can leave grounding open so AI answers still cite those pages.