If it’s a day ending in Y, you can count on a story that further erodes the public’s trust in Big Tech. This week it was Sony Music and Warner suing Anthropic over copyright, accusing the AI company of illicitly pirating music and song lyrics from its catalog to train its AI models. With the lawsuit, Anthropic is now officially the target of all three major music publishers, since Universal brought a similar action in January.
As copyright lawsuits go, the allegations are pretty juicy. They paint a picture of Anthropic brazenly pirating music catalogs through torrenting and copying huge troves of lyrics from third-party websites wholesale. The filing claims Anthropic co-founder Benjamin Mann personally conducted or directed the torrenting and discussed it openly in Slack channels. Anthropic, which agreed a year ago to pay $1.5 billion in a settlement over pirated books, responded curtly, telling Axios “we intend to defend ourselves robustly in court.”
To any media executive, content creator, or news publisher, the lawsuit is more evidence that tech companies can’t be trusted with content. And that conclusion is correct. Ever since OpenAI then-CTO Mira Murati was caught like a deer in the headlights when asked about what training data had been used to train Sora, the company’s now-discontinued video model, it’s been clear AI companies will always take the most liberal view of “fair use” when it comes to harvesting content for their models.
The all-or-nothing trap
The problem with “don’t trust them” is that it often fuels a binary perspective: that the only reasonable reaction is to lock down your corpus, blocking AI bots from ever ingesting a single character. Opening it up, even a little, to a bot—even a supposedly “legit” one—means trusting the company to play by certain rules, a key one being: content used for AI search won’t be thrown into training data. So you can open up and hope, or block and stay safe.
I see this more and more in my consulting work: the instinct to protect IP makes publishers reluctant to even do GEO testing. This perspective is understandable, but ultimately self-defeating. Blocking bots means sacrificing visibility in AI answers. And while translating that visibility into good business outcomes is far from guaranteed, AI experiences are rapidly becoming the future. What publishers need is an approach that preserves AI as a path for audiences to discover them and build their authority while not taking on faith that the AI companies will play by the rules.
Bot blocking is generally centered around the Robots Exclusion Protocol (a.k.a. robots.txt), which governs which bots can scrape content on a site. Importantly, there are different kinds of bots. For this discussion, you really only need to know that there are training bots and retrieval (i.e. search/answer) bots. The former harvests information into vast archives to build new models, while the latter grabs specific info to answer individual queries in real time. Training copies data and keeps it; retrieval uses it once, then poof. (The search bots that power discovery do keep an index, the way Google always has, but that’s a card catalog, not a model.)
It’s becoming more or less standard for publishers to block training bots absent some kind of licensing deal. Retrieval, however, is how articles appear in AI answer engines. If your article is blocked, the engine only has metadata to go on, so if a competing site is open and yours is blocked, there’s a high likelihood the engine will favor your competition in the answer.
This is where many publishers trip up. They want to compete in the answer, but they don’t trust the AI company to simply scrape the article for just the one query. Many will assume the AI company will keep that article and use it for either training or for allowing their users to access the full text—which might hurt even more if there’s a paywall. So they block everything since it’s better than risking giving away the store.
Verification > trust
There is a happy medium here. You can open up content to retrieval bots without blindly trusting the AI companies’ claims that they’ll never train on it. The approach starts with blocking training bots (of course) and then selectively allowing retrieval bots where AI visibility is important. Then, you build your own verification: your CDN (content distribution network) checks every bot and verifies it against the vendor. It also logs the visit—what the bot scraped and when. That’s evidence you can use later if the vendor does something they shouldn’t.
To monitor whether an AI vendor may be training on your retrieved content, you can seed your site with “tracer” phrases, checking whether they show up in the raw model with a set of queries run on a schedule. If they do, it’s a strong indicator that your content is being thrown into training data. And any testing that opens up content to new kinds of crawlers should be done in pieces—a slice of the content—before deploying site-wide. Problems? Reverse course with a single file change. And measure against the only outcome that matters: whether your visibility in AI answers is improving or not.
While the media industry has good reason to be paranoid, it’s worth pointing out that the AI companies are incentivized to ensure their bots play by the rules. The companies that determine AI visibility (OpenAI, Anthropic, Perplexity, et al.) are the same ones spending fortunes on licensing deals and courtroom settlements. They’ve also learned, expensively, what courts do with sloppy acquisition. If they were to cheat and use retrieval content for training, that would transform a murky fair-use fight into clear evidence of misrepresentation. You don’t have to believe in their virtue to understand it’s in their interest to avoid that kind of exposure.
A crawl is really a skim
AI retrieval bots don’t actually read all your content anyway. When a crawler scans a web page, they expend the absolute minimum number of tokens to figure out what’s on the page to make a judgment about whether it’s worth citing. So even if you do make the entirety of a page available to crawlers, they often don’t read it—at least when it comes to determining presence in AI answers.
But making the full text available to crawlers means there’s a greater chance of that extra context mattering in deeper, research-oriented queries—the exact kind where people tend to check their sources. Limiting what the crawlers can see means transferring your authority to more visible publishers for no real benefit.
No one is asking publishers to trust companies that torrent online libraries. But the web has never run on trust—it runs on logs, verification, and who has the leverage. “Don’t trust” and “be discovered” aren’t opposites. Handled right, the first is how you afford the second.
