*

Growth of AI bot crawlers squeezing content creators'

search

revenue

*

Cloudflare ( NET ) tool marks shift from traditional search engine

traffic model

*

Publishers and social media firms, including Reddit ( RDDT ) and

Condé

Nast support initiative

By Krystal Hu

NEW YORK, July 1 (Reuters) -

Cloudflare ( NET ) has launched a tool that blocks bot crawlers from

accessing content without permission or compensation to help

websites make money from AI firms trying to access and train on

their content, the software company said on Tuesday.

The tool allows website owners to choose whether artificial

intelligence crawlers can access their material and set a price

for access through a "pay per crawl" model, which will help them

control how their work is used and compensated, Cloudflare ( NET )

said.

With AI crawlers increasingly collecting content without

sending visitors to the original source, website owners are

looking to develop additional revenue sources as search traffic

referrals that once generated advertising revenue decline.

The initiative is supported by major publishers including

Condé Nast and Associated Press, as well as social media

companies such as Reddit ( RDDT ) and Pinterest ( PINS ).

Cloudflare's ( NET ) Chief Strategy Officer Stephanie Cohen said the

goal of such tools was to give publishers control over their

content, and ensure a sustainable ecosystem for online content

creators and AI companies.

"The change in traffic patterns has been rapid, and

something needed to change," Cohen said in an interview. "This

is just the beginning of a new model for the internet."

Google, for example, has seen its ratio of crawls to

visitors referred back to sites drop to 18:1 from 6:1 just six

months ago, according to Cloudflare ( NET ) data, suggesting the search

giant is maintaining its crawling but decreasing referrals.

The decline could be a result of users finding answers

directly within Google's search results, such as AI Overviews.

Still, Google's ratio is much higher than other AI companies,

such as OpenAI's 1,500:1.

For decades, search engines have indexed content on the

internet directing users back to websites, an approach that

rewards creators for producing quality content. However, AI

companies' crawlers have disrupted this model because they

harvest material without sending visitors to the original source

and aggregate information through chatbots such as ChatGPT,

depriving creators of revenue and recognition.

Many AI companies are circumventing a common web standard

used by publishers to block the scraping of their content for

use in AI systems, and argue they have broken no laws in

accessing content for free.

In response, some publishers, including the New York Times,

have sued AI companies for copyright infringement, while others

have struck deals to license their content.

Reddit ( RDDT ), for example, has sued AI startup Anthropic for

allegedly scraping Reddit ( RDDT ) user comments to train its AI chatbot,

while inking a content licensing deal with Google.