There’s a contradiction emerging in how publishers get paid when AI uses their content. The licensing deals that OpenAI, Meta, Amazon, and others have been signing these past few years appear to be slowing down, at least for the publishers still waiting on one.
Publishers are speaking about this openly. Digiday hosted a London town hall recently, and one publishing executive said discussions with AI companies have slowed down. Also, Katya Gorchinskaya, a Ukrainian journalist and media consultant who built J·Index, a database of thousands of AI implementations in newsrooms, separately told the Reuters Institute for the Study of Journalism that the market for deals with model makers is effectively closed.
At the same time, AI demand for content continues to go up. I recently spoke with Jonathan Roberts, chief innovation officer at People Inc., for The Media Copilot podcast, and he revealed that the number of different bots that try to scrape content from the company’s media properties (including People, InStyle, and Investopedia) now numbers in the tens of thousands every day. The company blocks all but 38 of them. At that Digiday London event, Fiona Spooner, managing director of consumer revenue at the Financial Times, said one trial account tried to access about a million articles in eight hours from 700,000 IP addresses.
I also see this in my GEO client work: the media properties often start from the position that having bots hammer your site all day is cost-prohibitive, so they block them outright. It makes sense, but if the end goal is to monetize your content in AI systems, the slowdown in licensing agreements suggests an all-or-nothing stance won’t give you much leverage.
All of this points to an evolving approach to payment: metered pricing. The industry has already been moving in this direction, but the question that doesn’t yet have a clear answer is what those meters should measure.
Paying for what gets used
Metered pricing has been around for a while. TollBit was one of the first to offer a “pay per crawl” model, directing bots to a gate where they could elect to pay a price set by the site, or go away (the company says its technology is now installed on more than 7,000 publisher sites). Cloudflare followed suit last year with a fanfare announcement that it would block AI bots by default for new sites on its network and provide the tools for publishers to charge by the crawl.
Things have evolved since then. There’s now more energy behind a “pay per use” model, meaning a content provider would only get paid when their content contributes to an answer, not at the moment it’s scraped. Cloudflare itself made that switch in July, with Ceramic.ai and You.com as its first partners, and Google and Microsoft are testing similar models. Cloudflare’s program is still in beta, but Lara Cohen, Cloudflare’s VP of strategic partnerships for media, creators, and AI, told me the company is working toward a system that alerts publishers when a buyer wants their content and lets them accept, decline, or negotiate.
The logic is informed by data: According to research from AirOps covering more than 548,000 pages, only about 15% of what ChatGPT retrieves ends up in its final answers. This is intuitive and matches human experience: When you do research to inform a report, you don’t typically use all of it.
However, we should be careful of stretching the stat too far: Just because chatbots cite only 15% of what they look at, that doesn’t mean the 85% left out is worthless. In fact, uncited sources still shape answers. An Ahrefs study of 1.4 million prompts found ChatGPT leans heavily on Reddit to understand topics while rarely giving it credit. That material might also be exactly what’s needed for a different query, so there is value in having it available.
This is why, when determining value (and thus payment), who’s paying is as important as how much. If it’s an AI app answering a user’s question, it only wants to pay for what’s used. If it’s whoever builds the index that app draws on, there’s value in having the widest corpus of content available.
There’s actually a third party here: the middleman. And the question is even more relevant for them. As I’ve explored before, a whole industry of AI data brokers has emerged over the past couple of years, with hugely valuable companies like Exa, Parallel, and Tavily leading the pack. These companies crawl the internet, then sell access to the corpus to buyers—which are often AI developers. Parallel, valued at $2 billion after a $100 million round in April, says more than 100,000 developers use its tools, at companies like Harvey, Notion, and Opendoor.
Roberts doesn’t think much of the business model. “These businesses take content from the internet for free and sell it for money,” he told me. “They do not create things, they resell things.”
Pricing varies, but they bill customers by the request, which amounts to charging per delivery. Exa charges $1 for every 1,000 pages of full text, and Parallel’s Extract is $1 per 1,000 URLs. In publisher terms, that’s about a tenth of a cent per article.
Some of the services don’t pay publishers at all, but that’s starting to change. Since May, Parallel has partnered with several publishers, including The Atlantic, Fortune, and PR Newswire, on payments through a program called Index. The payouts depend on the measured value of the publication’s contribution to any work Parallel’s agent does on behalf of customers. That incentivizes unique, hard-to-replace content, since it’ll earn more.
The case for an index fee
Although the market for middlemen clearly needs more structure, Parallel’s approach is a real step toward the sophisticated payment system the industry needs. That said, it’s one part of an economy with several layers. Assuming the system can scale, paying publishers only when their content is used is a tough sell since it gives them nothing up front, and it doesn’t address what I mentioned earlier: the value of having the content in a larger corpus in the first place.
For example, say you had a query about a company’s layoffs, and the agent looks at three sources: Source A, an original report with the layoff numbers, which nobody else has; Source B, a wire story that repeats A’s numbers; and Source C, background on the company’s finances. The agent would likely use A, discard B, and maybe lightly incorporate bits of C. Money would flow accordingly.
But B is still valuable to validate Source A, and it’s important to have in cases where there is no Source A. And the next time, Source B might be the one with the original report. To incentivize the widest number of sources, a nominal, flat index fee (not the same as pay-per-crawl) could go a long way toward convincing publishers to participate in the broader system.
That’s less hypothetical than it sounds. Cohen told me that on Cloudflare’s platform, each buyer defines “use” for itself: “Some define this as when content is indexed, while others define it as when content is used to generate answers.” In other words, some buyers are already paying for the library.
As we’ve seen with the bigger licensing deals, this kind of system might be difficult to scale, but that can be addressed by scaling the fee to how much demand there is for a publisher’s content, not how big the publisher is. That’s important because smaller publishers can punch above their weight for the right query. Cohen said the biggest lesson from pay per crawl’s first year was “the tremendous value of long-tail, smaller publishers’ content,” especially local news. “If I’m asking ChatGPT what’s happening today in Duluth, Minnesota, there’s probably only one or two local news sources to pull that information from.” A small paper with no competition is Source A for its whole town.
TIME’s Mark Howard has seen evidence of the library’s value, too. He told me the models appear to store what they scrape, since some TIME articles get more AI referrals than citations, something that shouldn’t happen. And most of the agent crawls TIME sees are of archive stories more than a year old. The library is getting used whether or not anyone pays for it.
Roberts pointed to Spotify as a good analogy for what the industry currently lacks: “We’ve basically got 18 Napsters of content, but there’s no Spotify.” Spotify is actually a great model for this system: It pays rights holders based on their share of streams, but its deals with the major labels have included large advances and guaranteed minimums. A leaked 2011 contract with Sony, for example, showed up to $42.5 million in advances over three years.
In the case of the media, the middlemen would pay publishers a nominal amount for inclusion, then the real money comes in as their content provides actual value for customers.
Google holds the meter
It’s a solid model, but it doesn’t necessarily translate from the “middlemen” market to the mass market. OpenAI already has many licensing deals in place and doesn’t seem interested in other approaches. Perplexity experimented with rev-share models (an ad revenue share in 2024, then Comet Plus, which sends 80% of subscription revenue to participating publishers), but they haven’t had much effect. And the biggest fish, Google, is unique: It’s recently introduced a pilot called News AI for publishers to collect payments for use in Gemini, AI Overviews, and AI Mode, but the pricing is a total black box, and since it has all the leverage, there’s no reason to think that will change.
Which brings it back to the meters. The industry is finally building them, and that’s real progress. But a meter that only counts answers will always undervalue the library, and the library is what makes the answers any good. If publishers are going to keep feeding these systems, someone has to pay for the pantry as well as the ingredients that end up on the plate. Roberts has a simple test for whether the price is right: “If we would create new content for this market, then the pricing is right. If we would not create new content for this market, then the pricing is wrong.” By that measure, we’re not there yet.
