Cloudflare has changed how AI crawlers scrape the internet

Cloudflare has announced it is now blocking AI crawlers accessing content without permission or compensation, by default. Starting on 2 July 2025, website owners can choose if they want AI crawlers to access their content and decide how AI companies can use it. AI companies can also now clearly state their purpose — if their crawlers are used for training, inference or search — to help website owners decide which crawlers to allow. Cloudflare says its new default setting is the first step towards a more sustainable future for both content creators and AI innovators.

For decades, the internet has operated on a simple exchange: search engines index content and direct users back to original websites, generating traffic and ad revenue for websites of all sizes. This cycle rewards creators who produce quality content with money and a following, while helping users discover new and relevant information.

That model is now broken. AI crawlers collect content like text, articles and images to generate answers, without sending visitors to the original source — depriving content creators of revenue, and the satisfaction of knowing someone is viewing their content. If the incentive to create original, quality content disappears, society ends up losing, and the future of the internet is at risk.

“If the internet is going to survive the age of AI, we need to give publishers the control they deserve and build a new economic model that works for everyone — creators, consumers, tomorrow’s AI founders and the future of the web itself,” said Matthew Prince, co-founder and CEO of Cloudflare. “Original content is what makes the internet one of the greatest inventions in the last century, and it’s essential that creators continue making it.

“AI crawlers have been scraping content without limits. Our goal is to put the power back in the hands of creators, while still helping AI companies innovate. This is about safeguarding the future of a free and vibrant internet with a new model that works for everyone.”

Cloudflare powers one of the world’s largest networks, helping to manage and protect traffic for 20% of the web. The company handles trillions of requests daily, and has advanced bot management solutions, distinguishing between human users and AI crawlers. In September 2024, Cloudflare introduced the option to block AI crawlers in a single click. More than one million customers have since chosen this option, meant to be an aggressive but easy solution that halts scraping while they determine their AI strategy.

Now, Cloudflare is taking the next step to enforce a permission-based model for AI crawlers. AI companies will now be required to obtain explicit permission from a website before scraping. Upon sign-up with Cloudflare, every new domain will now be asked if they want to allow AI crawlers, giving customers the choice upfront to explicitly allow or deny AI crawlers access. This significant shift means that every new domain starts with the default of control, and eliminates the need for webpage owners to manually configure their settings to opt out. Customers can easily check their settings and enable crawling at any time if they want their content to be freely accessed.

Leading content, media and technology companies are in support of creating a more sustainable future that values original content.

“Cloudflare’s innovative approach to block AI crawlers is a game changer for publishers and sets a new standard for how content is respected online. When AI companies can no longer take anything they want for free, it opens the door to sustainable innovation built on permission and partnership,” said Roger Lynch, CEO of Condé Nast. “This is a critical step toward creating a fair value exchange on the internet that protects creators, supports quality journalism and holds AI companies accountable.”

“The information landscape continues to change rapidly, but the value of accurate, factual, nonpartisan journalism has never been more essential. We’re pleased to participate in this important framework that will help ensure intellectual property is protected and all content creators are fairly compensated for their work,” said Kristin Heitmann, Chief Revenue Officer, The Associated Press.

“As the largest publisher in the country (USA), comprised of USA TODAY and over 200 local publications throughout the USA TODAY Network, blocking unauthorised scraping and the use of our original content without fair compensation is critically important,” said Renn Turiano, Chief Consumer and Product Officer of Gannett Media. “As our industry faces these challenges, we are optimistic the Cloudflare technology will help combat the theft of valuable IP.”

“We applaud Cloudflare for advocating for a sustainable digital ecosystem that benefits all stakeholders — the consumers who rely on credible information, the publishers who invest in its creation, and the advertisers who support its dissemination,” said Vivek Shah, CEO of Ziff Davis.

The company recently proposed new ways for AI bots to authenticate themselves as well as for websites to identify those bots — giving creators and website owners new identification mechanisms and control over what crawlers they want to allow. Cloudflare is participating in the development of a new protocol to provide bot owners and AI agent developers with a public, standard way to identify themselves.

Image credit: iStock.com/Sundry Photography