Cloudflare, a major player on the internet, is deploying an unprecedented offensive strategy. Artificial intelligence robots, once omnipresent, are now experiencing a halt. The blocking system implemented by Cloudflare aims to curb abuses observed in the use of online content.
The consequences of this initiative are considerable for businesses and publishers. The era of digital plundering is over. Creators, faced with this threat, aspire to a fair economic model that protects their works from unsolicited intrusions. Cloudflare is redefining the rules of the game.
A new policy for blocking AI bots
Cloudflare, a leader in content delivery, has launched an offensive against AI bots. Since July 1st, the company has been blocking access to its services by default for AI bots like GPTBot from OpenAI and ClaudeBot from Anthropic. This change addresses growing concerns from website owners regarding the slowdown caused by these crawlers.
Website owners have reported a dramatic increase in the number of unwanted requests generated by these bots. Cloudflare’s infrastructure thus protects the over two million users who represent around 20% of the web. Owners must now explicitly consent for their content to be accessed by these bots, providing greater security to existing states.
Issues raised by uncontrolled access
Many publishers and content companies complain about a phenomenon that some describe as a true digital plunder. Companies like The Associated Press, Condé Nast, and ZDNET have expressed their dissatisfaction with the unauthorized use of their content by AI companies. Most of these operations occur without consent and bypass standard protocols such as robots.txt, which prevent access.
Measures to include publishers
Cloudflare’s new policy aims to restore balance between publishers and AI companies. Matthew Prince, CEO of Cloudflare, emphasized that this initiative seeks to “give publishers the control they deserve.” Additionally, Cloudflare has launched a program called “Pay Per Crawl”. This program allows publishers to set fees for AI companies to access their content.
Notable consequences on web traffic
Data reveals that since the rise of AI technologies, traffic to news sites has decreased. For example, the Business Insider site experienced a 55% drop in traffic between April 2022 and April 2025. If this trend continues, experts like Nicholas Thompson from The Atlantic predict a drastic reduction in traffic from Google, threatening the economic sustainability of online publishers.
A reaction to a growing phenomenon
This change has also intensified following legal debates surrounding copyright. Recent decisions have ruled that certain companies, including Meta and Anthropic, have the right to use copyrighted works under fair use. However, creators and writers remain concerned about the increasing impact of AI on their works.
The future and evolution of the market
As Cloudflare takes the lead, other content delivery networks, like Akamai, may follow this trend. The trend to restrict AI bot access on the web is a significant evolution, marking the end of an era of unlimited access. AI companies will face significant obstacles, now having to negotiate for licenses and pay for access to content.
For more information on the subject, it is relevant to consider recent developments in the field of AI, such as the impact of these new technologies on web traffic here, as well as educational initiatives here. The debate over copyright and the environmental implications of these technologies continue to evolve, as do efforts to reduce AI’s carbon footprint, detailed here.
FAQ on Cloudflare’s offensive against AI bots
Why does Cloudflare block AI bots?
Cloudflare blocks AI bots to protect its clients’ websites, as these bots generate a massive number of automated requests that significantly slow down site performance.
What types of AI bots are affected by this measure?
AI bots such as GPTBot from OpenAI and ClaudeBot from Anthropic are explicitly blocked, along with other bots attempting to access content without authorization.
How does Cloudflare determine which bots to block?
Cloudflare uses behavioral analysis and machine learning techniques to detect bots known as “shadow scrapers” that try to evade detection.
What are the consequences of this policy for companies using competing CDNs?
This new policy could prompt other CDNs, such as Akamai, to adopt similar measures, thus increasing pressure on AI bots and changing the dynamics of online content access.
What is the “Pay Per Crawl” program offered by Cloudflare?
The “Pay Per Crawl” program allows publishers to set fees for AI companies to access their content, making scraping potentially costly for bots.
How does Cloudflare protect the content of online publishers?
Cloudflare blocks AI bots by default, only allowing access with the explicit approval of site owners, thus giving them more control over their content.
What are the repercussions of this measure for AI companies?
AI companies will now need to negotiate access to content and potentially pay licensing fees, thereby changing their current approach to scraping content without consent.
Does this measure also affect regular Internet users?
However, this measure should not affect regular users; however, it could influence access to certain online information if content agents cannot negotiate access.
What are the concerns of publishers regarding the use of AI bots?
Publishers fear that AI bots exploit their content without compensation, harming their revenues and ability to create quality content.
What changes have been observed in the traffic of news sites due to AI bots?
Since the rise of AI bots, traffic to many news sites has significantly dropped, with some companies reporting a 55% decrease in their traffic.
