Cloudflare has announced a new tool that allows website owners to block AI bots from collecting their content without permission. The feature, introduced on July 1, aims to help protect the economic interests of online creators as artificial intelligence increasingly relies on scraping published material to train models and generate responses.
According to Cloudflare, the internet has long depended on a simple exchange: search engines index content, send users back to the original sites, and drive traffic and ad revenue. But AI crawlers are disrupting this balance by harvesting words and images to answer user questions without linking back to the source, depriving creators of both recognition and income.
With this new tool, website owners can decide whether to allow AI bots to access their content and even set fees for usage through a “pay per crawl” system. Cloudflare’s CEO, Matthew Prince, said that for the internet to remain sustainable in the age of AI, publishers need more control over how their work is used. He emphasized that original content fuels the internet’s value and must be protected to keep creators motivated to produce it.
The company also pointed out that beyond lost revenue, unregulated scraping can hurt site performance by overloading servers, slowing load times, and distorting analytics data, which in turn raises operational costs.
While some experts have questioned whether anti-scraping tools can truly stop determined bots, others are hopeful this approach will create fairer terms between AI developers and content creators.
Last year, Cloudflare rolled out a related tool designed to block AI data harvesters selectively, allowing people and search engines to access sites without restrictions. This approach aims to maintain search engine optimization benefits while preventing unauthorized scraping.
With AI development accelerating, Cloudflare’s new bot blocker highlights the growing debate over who controls the content that powers modern technologies — and how creators can be compensated fairly for their work.
