Google, OpenAI, Anthropic, and Microsoft have agreed to honor a new robots.txt standard that lets publishers block AI training scrapers with a single directive—and the consolidation of control over who gets to build the next generation of AI is being sold as a win for the little guy.

The proposal, drafted by Anthropic technical lead Kevin Kelley and submitted to the Internet Engineering Task Force, would add a new user-agent flag—__usage_ai-training—to the familiar robots.txt file. Website operators could then disallow all AI training crawlers in one entry instead of tracking and blocking each company's bots individually. Google, OpenAI, and Microsoft say they'll honor it. Meta expressed support but hasn't committed to implementation, according to SiliconANGLE.

Here's the catch the establishment tech press won't name: this framework doesn't empower anyone outside the club. The biggest AI companies in the world are colluding on a standard they wrote, submitted through a process they dominate, and are now voluntarily enforcing—before IETF even finalizes it. If you're an independent AI developer, an open-source project, or a free-speech platform like Gab, you don't get a seat at this table. You just get a new set of rules designed by your competitors.

Engadget framed the announcement as tech giants "promising new scraping controls" for publishers. SiliconANGLE played it straight as a "crawling opt-out standard." The Verge offered little beyond calling it "a new way to opt out of scraping." None of the three outlets raised the obvious antitrust question: when the four largest AI companies coordinate on data-access infrastructure, who exactly benefits—and who gets frozen out?

The technical details matter. As Engadget noted, the new flag only blocks scrapers "specifically meant to gather data for AI training." Google confirmed it will apply the block only to its "Google-Extended" agent, which trains Gemini models. Regular search indexing continues untouched. Microsoft is also introducing a separate "AI Grounding" setting, SiliconANGLE reported, letting publishers block AI models from accessing webpages to answer user questions—even when no training is involved. That's not just training-data control. That's a censorship lever: the ability to decide whether your content can be retrieved and cited by AI systems at all.

OpenAI and Microsoft plan to implement the standard in the coming months; Google says sometime next year. The timeline is leisurely for companies that move at breakneck speed when it suits them.

The founders didn't draft the First Amendment to protect comfortable arrangements among entrenched powers. They wrote it for the pamphleteer, the dissenter, the voice the establishment wanted silenced. When the biggest AI companies on earth write the rules for who gets to access information, the question isn't whether publishers gain a tool—it's whether anyone outside the oligopoly can still compete.

The open internet was built on the principle that nobody gets to gatekeep who can read and learn from public information. The new scraping standard doesn't just let publishers opt out. It lets Big Tech opt everyone else out of the future.