OpenAI, Google, Microsoft and Anthropic have pledged to honour a single, shared AI scraping opt-out for publishers, in the clearest joint move yet by the biggest artificial intelligence companies to give websites one straightforward way to say no to AI training. The announcement, made on Friday, October 30, 2026, promises to replace today's patchwork of separate blocking instructions for every company's web crawlers with one standard signal that all four firms recognise. For newsrooms and independent publishers that have watched their reporting swept up to train chatbots, it is a practical concession — but one that will arrive in stages, relies on voluntary compliance, and covers only future training, not the vast archives already collected.
The AI scraping opt-out matters because of how the web currently works. A publisher that wants to keep its articles out of AI training data cannot simply flip one switch. Instead, it must write and maintain a different blocking command for each bot, from each company, and hope that every crawler reads and respects it. The new plan collapses that chore into a single instruction, published once in a site's robots.txt file, that tells participating companies the site does not want its content used to train foundation models. That simplicity is the whole point: a small local news site with no legal team should get the same protection, in principle, as a global newspaper group.
The four companies announced their support for a common training-preference signal on Friday, as publishers face a surge in automated bots harvesting articles to feed AI models. Under the plan, a website operator adds one standardised instruction to robots.txt — the long-standing text file that tells automated visitors which parts of a site they may access — and every participating AI company is expected to honour it for training purposes. The companies described the move as a shared standard rather than four separate favours, and they framed it as a response to years of complaints that opting out of AI training was needlessly complicated.
The rollout will not be instant, and the companies were candid that the solution will not fully take effect for months. Anthropic, whose employee Kevin Kelley authored the underlying draft, said it has already started honouring the token. Microsoft said it will implement the signal beginning February 1, 2027. Google said it will follow sometime in 2027. OpenAI said it will roll the support out in the coming months, without giving a firm date. That staggered timetable means a publisher that adds the token now will see uneven protection for a while: some crawlers will respect it at once, while others will only catch up during 2027.
Meta, which was not listed among the four companies adopting the token, nevertheless welcomed the plan. Meta spokesperson Andy Stone wrote on X: "We believe creating a common signal that conveys such preferences to multiple providers at once is an important step. And we look forward to working with other AI companies on the standards and their implementation across the industry." The endorsement matters because Meta operates its own large AI systems and its own crawlers, and its backing signals that the idea could spread beyond the original four.
The dispute over AI scraping has been building since chatbots became mainstream. To train large language models, AI companies need enormous quantities of text, and much of the most valuable text on the open web is journalism: reported, edited and fact-checked work that cost real money to produce. Publishers argue that vacuuming up that work without permission or payment undercuts the business that creates it, especially when AI chatbots then answer users' questions directly, reducing the traffic that keeps newsrooms alive. AI companies have generally replied that training on publicly available material is permitted under copyright doctrines such as fair use, and that their products expand, rather than replace, access to information.
That argument is now being tested in court. In December 2023, The New York Times sued OpenAI and Microsoft, alleging that millions of its articles were used without authorisation or payment to train ChatGPT. The two companies have invoked fair use as their defence. Other publishers have chosen technical defences instead: since July 2025, internet infrastructure provider Cloudflare has blocked AI bots by default on new customer sites, and a standard called Really Simple Licensing already allows publishers to charge bots that visit their pages. The new shared token sits between those approaches — simpler than a lawsuit, cheaper than a paywall, but also weaker than either, because it depends on the goodwill of the very companies doing the scraping.
The technical roots of the plan lie in robots.txt itself, a convention that has governed polite crawling for decades. The file was never a law; it was a voluntary agreement that respectable crawlers would read a site's instructions and obey them. AI training has strained that convention, because a single company may run several bots with different purposes — one for search indexing, one for live answers, one for training — and a publisher may want to allow one while blocking another. OpenAI, for example, already distinguishes between GPTBot, which governs crawling for training its foundation models, and OAI-SearchBot, which governs whether a site can surface in ChatGPT's search features. The new token is designed to cut through that complexity for the training question alone.
The shared token is called __usage_ai-training. It is defined in an individual draft at the Internet Engineering Task Force (IETF) titled "Machine-Readable Training Preference Signal", authored by Kevin Kelley of Anthropic, dated October 30, 2026, and set to expire on May 3, 2027. An individual draft is an early stage of standardisation: the proposal is documented and open for review, but it has not been adopted as a formal standard, and its future depends on wider industry adoption and on the IETF process.
In practice, a publisher adds the token to its robots.txt file as a single machine-readable preference, stating that its content should not be used for foundation-model training. Once each company completes its rollout, that one line is meant to speak to all four companies' crawlers at once, instead of requiring a separate opt-out rule for every crawler. OpenAI has stressed that the new signal will not replace the controls it already provides, so existing robots.txt rules stay in place, and the token does not by itself guarantee exclusion from training. It expresses a preference and relies on voluntary compliance by the crawler operator; it creates no enforcement mechanism of its own, and the legal consequences of ignoring it would depend on the law in each jurisdiction, not on the token.
The limits are equally important. The setting applies only to future training of models. It does not prevent bots from crawling a site for other purposes, such as search indexing or real-time retrieval, unless a site restricts those separately. And content scraping is allowed by default until a publisher changes the setting, Microsoft said in its blog post — meaning the burden still falls on publishers to act, rather than on AI companies to ask first. Microsoft and Google also warned, in separate blog posts, that opting out of AI training could reduce the chances that a publisher's content will surface in AI apps and experiences, a trade-off that will weigh heavily on smaller outlets that depend on discovery.
The companies presented the announcement as a genuine simplification. Microsoft's blog post framed the change as giving publishers a clearer choice about training use, while noting the default-allowed position and the possible visibility trade-off. Google's blog post made a similar point about surfacing in AI experiences. OpenAI said it aims to implement the shared signal in the coming months and will publish a further update with implementation details when it does, while keeping its existing crawler controls in force.
Meta's public support, through spokesperson Andy Stone on X, was the most quotable industry reaction: "We believe creating a common signal that conveys such preferences to multiple providers at once is an important step. And we look forward to working with other AI companies on the standards and their implementation across the industry." The statement stops short of a commitment to honour the token on Meta's own crawlers, but it signals that the standard could widen beyond its four initial backers.
Publishers and analysts, meanwhile, are reading the fine print. The voluntary nature of robots.txt has always been its weakness: a well-behaved crawler obeys it, but nothing in the file itself can compel a bad actor to do so, and the token does nothing about content already scraped and baked into existing models. For news organisations, the practical question is whether a single opt-out will become a genuine industry norm — like the early web's respect for robots.txt — or whether it becomes another setting that polite companies honour while the fiercest competition for training data continues elsewhere. The answer will depend less on the token's wording than on whether courts, regulators and the largest platforms treat it as the expected standard.
This pledge matters for three reasons. First, it lowers the cost of saying no. Today, only well-resourced publishers can realistically track every AI crawler and maintain a wall of blocking rules; a single token gives every site, from a national newspaper to a one-person blog, the same one-line tool. Second, it creates a visible norm. If the four biggest AI developers all honour the same signal, it becomes harder for any one of them to argue that respecting publisher preferences is technically impractical — and easier for regulators to point to an existing industry practice when they write rules. Third, it clarifies the bargain: publishers that opt out may lose some visibility in AI answers, while publishers that stay in may keep that visibility but keep feeding the systems that compete with them. That is a real choice, and for the first time it is being offered in a single, legible form.
The limits matter just as much. A voluntary, future-only, default-allowed signal does not pay journalists, does not remove already-scraped articles from trained models, and does not stop a crawler that simply ignores it. It also arrives at a moment when AI companies are under intense financial pressure — OpenAI itself has just cut its fundraising target amid a tech sell-off — which makes publishers sceptical that goodwill alone will hold when training data is the industry's most valuable raw material. The token is therefore best understood as infrastructure, not a settlement: useful plumbing that makes a fairer deal possible, but not the deal itself.
The immediate next step belongs to publishers: those that care about training use can add the token now, then verify crawler behaviour against each company's stated rollout date as it passes — Anthropic first, Microsoft on February 1, 2027, Google during 2027, and OpenAI in the coming months. Site owners should also check that their existing robots.txt controls remain intact, since the new signal supplements rather than replaces them, and they should separate their training preference from their search and retrieval preferences, which follow different rules.
Beyond that, watch three things. The first is whether Meta and other AI developers formally adopt the token, turning a four-company pledge into a genuine web standard. The second is whether the IETF draft advances from an individual proposal toward a recognised standard before its May 3, 2027 expiry. The third is whether courts and lawmakers treat the token as evidence of an industry norm — which would give it teeth that robots.txt alone has never had. If all three move in the same direction, Friday's announcement could be remembered as the day the web regained a simple way to say no. If they do not, publishers will have gained a tidier opt-out form and little else, and the fight over payment for AI training will return to the courtroom.
It is a single shared signal, the token __usage_ai-training, that a website adds to its robots.txt file to tell participating AI companies not to use its content for training foundation models.
OpenAI, Google, Microsoft and Anthropic pledged support on October 30, 2026. Meta publicly backed the effort on X but was not listed among the four adopting companies.
Anthropic has already started, Microsoft begins February 1, 2027, Google follows sometime in 2027, and OpenAI says it will roll out in the coming months, so enforcement will be uneven through most of 2027.
No. It applies only to future training use. Bots may still crawl for search or live retrieval unless a site restricts those separately, and scraping remains allowed by default until a publisher adds the token.
No. The signal is voluntary, creates no enforcement mechanism of its own, and does not cover content already scraped. OpenAI says its existing controls remain in place and the token does not guarantee exclusion.
Related coverage on US News Pulse: OpenAI cuts its funding target by $10 billion as the AI sell-off bites; Anthropic files for its Nasdaq listing as ANT in a $2 trillion test; and Claude and the Georgia vote-data controversy. Sources for this report: AFP via Malay Mail, October 31, 2026; SiliconANGLE, October 30, 2026; OpenAI, Microsoft and Bing blog posts; and the IETF draft "Machine-Readable Training Preference Signal", October 30, 2026.
Leave a Comment
| Today's Date:
