Cloudflare cofounder and CEO Matthew Prince (Noam Galai/Getty Images)

You down with PPC

Could Cloudflare’s “pay per crawl” save news from AI?

The novel plan would let publishers control AI bot access and collect micropayments to access content.

7/1/25 12:02PM

AI is eating the news.

Publishers large and small are bracing for a grim reality that is starting to reveal itself: as readers increasingly turn to AI chatbots like ChatGPT for their queries, or skim over Google AI overviews, news publishers are seeing visits from search engines drop off a cliff.

AI companies have slurped up billions of web pages to train their models and fetch query results. Big publishers like The New York Times have filed suit against OpenAI and Microsoft, accusing the companies of stealing its content without permission or compensation. Many publishers have opted to cut deals with AI companies to license their work and appear in results.

While many of today’s chatbots surface citations with links in query responses, it generates a fraction of the traffic that traditional search engine results saw (and that was already in decline).

An unlikely company is stepping in with a novel solution to this problem that could provide a way for AI companies to crawl a publisher’s website with permission and pay for the access.

Cloudflare is a content delivery network — it ensures that customers’ websites, images, and videos will be accessible quickly around the world, sitting between the publisher and the web traffic hitting its site. That gives Cloudflare the unique ability to control who gets to see the content that it’s distributing. And while individual website owners can try to block AI bots from scraping their sites, Cloudflare can do it for billions of web pages at a time across 125 countries. The company says it serves about 20% of the web.

Cloudflare is introducing an experiment that it’s calling “pay per crawl,” which acts as a gatekeeper (and a toll booth) for AI bots crawling the web. Here’s how it works:

Cloudflare detects traffic that is coming not from a human user, but from an AI crawler.
Depending on a publisher’s choice, the “pay per crawl” system controls access to the site.
The AI bot can be allowed to access the site for free, it can pay to access the site, or it can be blocked altogether. Publishers can also tailor this to specific companies.
Cloudflare collects a micropayment from the AI bot, which it passes along to the publisher.

The pay per crawl plan is currently in private beta, and the company has also announced that all new Cloudflare customers will be set to block AI bots by default.

Cloudflare cofounder and CEO Matthew Prince wrote in a blog post declaring “Content Independence Day”:

“Instead of being a fair trade, the web is being stripmined by AI crawlers with content creators seeing almost no traffic and therefore almost no value. That changes today, July 1, what we’re calling Content Independence Day. Cloudflare, along with a majority of the world’s leading publishers and AI companies, is changing the default to block AI crawlers unless they pay creators for their content. That content is the fuel that powers AI engines, and so it’s only fair that content creators are compensated directly for it.”

The rub is that both AI companies and publishers need to opt in to the plan for payments to be processed, but several big publishers have signed up, including Condé Nast, Time, Associated Press, and The Atlantic, according to TechCrunch.

More Tech

See all Tech

Jon Keegan5h

xAI’s trade secrets lawsuit against OpenAI dismissed

In September, xAI sued rival OpenAI, accusing the company of stealing trade secrets by hiring away key employees.

After a key employee was poached by OpenAI, xAI filed its lawsuit in Northern California federal court, alleging that the company was after its “secret sauce” — the methods by which xAI was able to build data centers so quickly.

Today Judge Rita Lin dismissed the case, citing a lack of any direct accusations against OpenAI itself. In the order dismissing the case, Lin wrote:

“The sole defendant in this lawsuit is OpenAI, whom xAI accuses of misappropriating its trade secrets. But xAI does not point to any misconduct by OpenAI. Instead, it points to eight former xAI employees who left for OpenAI at around the same time.”

The judge said that xAI is allowed to file an amended complaint if it wants to pursue the case, but must do so by March 17.

US judge dismisses xAI trade-secrets lawsuit against rival OpenAI for now

Today Judge Rita Lin dismissed the case, citing a lack of any direct accusations against OpenAI itself. In the order dismissing the case, Lin wrote:

“The sole defendant in this lawsuit is OpenAI, whom xAI accuses of misappropriating its trade secrets. But xAI does not point to any misconduct by OpenAI. Instead, it points to eight former xAI employees who left for OpenAI at around the same time.”

The judge said that xAI is allowed to file an amended complaint if it wants to pursue the case, but must do so by March 17.

Jon Keegan8h

Chinese drone maker DJI sues to overturn FCC foreign drone ban

Chinese drone maker DJI has filed a lawsuit against the FCC challenging the December 2025 decision that effectively bans all foreign drones and components from the US market. DJI and all other foreign drone makers were added to the FCC’s “covered list” of equipment and services that the agency says “pose an unacceptable risk to the national security of the United States.”

In a lawsuit filed with the US Court of Appeals for the Ninth Circuit, DJI argues that the company is “severely harmed” by the ruling, and seeks review of the decision: “The FCC exceeded its statutory authority, failed to observe statutorily required procedures, and violated the Fifth Amendment when it purported to add DJI’s products to the Covered List.”

In a statement to Sherwood News, a DJI spokesperson said:

“The FCC can add products to the Covered List only when they present a national security threat, yet it has never identified any threat associated with DJI or its products. Despite repeated efforts to engage with the government, DJI has never been given the chance to provide information to address or refute any concerns. These procedural and substantive deficiencies violate the Constitution and federal law.”

The FCC decision has cleared the way for the nascent US drone industry just as the US military urgently races to catch up in the race to acquire drones.

Jon Keegan10h

Anthropic follows OpenAI in rolling out agentic tools for enterprise

Just a day after OpenAI rolled out its agentic platform for enterprise, Anthropic has announced its own. Built from existing pieces of Anthropic tech that have already been previewed, the new platform essentially ties together plug-ins that can be tailored by enterprise customers into Claude Cowork.

Companies can customize their version of the tool to use their branding, communication style, and private data to speed up a long list of common tasks like performing financial analyses, human resources tasks, design, and engineering workflows. New connectors tie Claude Cowork into third-party platforms like Salesforce’s Slack, Google’s apps, LegalZoom, and DocuSign, among others.

The announcement gave a lift to some beaten-down software companies.

While OpenAI was releasing consumer apps like Sora, Anthropic was busy improving Claude’s ability to make spreadsheets and PowerPoint presentations — the boring but essential tools of the workplace.

The two competing solutions will be battling it out in the enterprise marketplace as both Anthropic and OpenAI seek to grow revenue streams to power their ambitious AI infrastructure projects.

Cowork and plugins for teams across the enterprise | Claude

The announcement gave a lift to some beaten-down software companies.

The two competing solutions will be battling it out in the enterprise marketplace as both Anthropic and OpenAI seek to grow revenue streams to power their ambitious AI infrastructure projects.

Rani Molla11h

Alphabet’s Waymo is now available in 10 cities

Today, Alphabet subsidiary Waymo announced it’s now welcoming public riders to its driverless car service in four additional US cities: Houston, Dallas, San Antonio, and Orlando.

The company said the service will be available first to “select riders” who’ve downloaded the app in those cities, and it will invite new riders on a rolling basis before opening the service to everyone “later this year.”

The latest announcement brings Waymo’s total service area to 10 cities, mostly located in California and across the Sun Belt, and doubles its footprint from a few months ago.

Ready to Ride: Dallas, Houston, San Antonio, and Orlando

The latest announcement brings Waymo’s total service area to 10 cities, mostly located in California and across the Sun Belt, and doubles its footprint from a few months ago.

Rani Molla12h

TSMC trades near record high as Apple plans to purchase “well over 100 million” chips from its Arizona facility this year

Last year Apple said it would invest $600 billion to expand in the US over four years, in an effort to reduce its reliance on overseas suppliers and avoid tariffs.

We’re now getting more detail on what that involves. According to a new announcement from Apple, it includes expanding AI server production and moving future production of suddenly trendy Mac Minis to Houston.

It also involves the purchase of “well over 100 million” chips from TSMC’s Arizona facility this year, “a significant increase from 2025.”

“We’re buying as much of the output of this fab as we can,” Apple’s global head of procurement, David Tom, told The Wall Street Journal.

Apple has long been one of TSMC’s largest customers, even as the iPhone maker shifted to designing its own processors in-house — chips that TSMC overwhelmingly manufactures. Apple’s demand is helping fund TSMC’s massive multibillion-dollar chip plant expansion in Arizona.

Shares of TSMC were up 3% and near record highs in early trading Tuesday, as the Apple announcement underlines huge demand for TSMC’s AI chips

Apple accelerates U.S. manufacturing with Mac mini production

It also involves the purchase of “well over 100 million” chips from TSMC’s Arizona facility this year, “a significant increase from 2025.”

“We’re buying as much of the output of this fab as we can,” Apple’s global head of procurement, David Tom, told The Wall Street Journal.

Shares of TSMC were up 3% and near record highs in early trading Tuesday, as the Apple announcement underlines huge demand for TSMC’s AI chips

Latest Stories

Sherwood Media, LLC produces fresh and unique perspectives on topical financial news and is a fully owned subsidiary of Robinhood Markets, Inc., and any views expressed here do not necessarily reflect the views of any other Robinhood affiliate, including Robinhood Markets, Inc., Robinhood Financial LLC, Robinhood Securities, LLC, Robinhood Crypto, LLC, or Robinhood Money, LLC.