Tech
tech
Max Knoblauch

AI companies' content crawlers are saddling website owners with unexpected, crushing bandwidth charges

AI bots that crawl websites and scrape their content are becoming more difficult to block and it's costing websites thousands of dollars.

By looting large amounts of data without permission, AI bot crawlers can leave smaller sites (that make and write things) on the hook for big unexpected bandwidth charges.

One coding documentation site said a crawler accessed 73 terabytes of files in May, including 10 terabytes in a single day, costing it more than $5,000 in bandwidth fees.

The problem: AI-focused companies keep making new crawlers that companies haven't blocked yet, and some appear to be bypassing attempts to block their bots altogether. Last month, OpenAI and Anthropic were accused of ignoring attempts to block their crawlers. Wired reporting found that Jeff Bezos and Nvidia-backed AI search startup Perplexity AI has similarly ignored scraping blockers, prompting an investigation from Amazon's cloud division.

Websites are Blocking the Wrong AI Scrapers (Because AI Companies Keep Making New Ones)

Websites are Blocking the Wrong AI Scrapers (Because AI Companies Keep Making New Ones)

One coding documentation site said a crawler accessed 73 terabytes of files in May, including 10 terabytes in a single day, costing it more than $5,000 in bandwidth fees.

The problem: AI-focused companies keep making new crawlers that companies haven't blocked yet, and some appear to be bypassing attempts to block their bots altogether. Last month, OpenAI and Anthropic were accused of ignoring attempts to block their crawlers. Wired reporting found that Jeff Bezos and Nvidia-backed AI search startup Perplexity AI has similarly ignored scraping blockers, prompting an investigation from Amazon's cloud division.

More Tech

See all Tech
tech
Rani Molla

Salesforce falls as Anthropic debuts Cowork tool

Salesforce is on track for its worst trading day in nearly two years, with shares down more than 6% Tuesday afternoon. One potential contributor: Anthropic’s release of Cowork, an autonomous digital assistant for completing office tasks. Essentially, Cowork is an agent-based version of Anthropic’s Claude chatbot that can access and manipulate files, automate workflows, and execute tasks on a user’s behalf.

Salesforce watchers will recall that the SaaS giant has thrown its weight behind its own agent-based workplace AI, Agentforce, which CEO Marc Benioff recently described as one of the company’s two main “momentum drivers.” In December, Benioff said he would consider renaming the company "Agenforce."

Dancing robot gif
DO THE
ROBOT
Jon Keegan2h
Rani Molla
Reality Check
4h
Meta will suffer the most from shouldering higher data center costs

Meta is no Google or Microsoft, but it’s spending like it is.

Mark Zuckerberg robot
tech
Rani Molla

Google reaches record high and crosses $4 trillion market cap after major wins for Gemini

Google parent Alphabet closed yesterday at a record-high stock price of $331.86, giving the company a market capitalization just above $4 trillion, as investors reward a string of wins for its Gemini AI model, including high-profile partnerships with Apple and Walmart.

After months of speculation, Apple announced a multiyear partnership to use Gemini to power its AI assistant, Siri, a major endorsement of Google’s AI prowess. That same day, Walmart said it would partner with Google to let customers purchase products directly through the Gemini chatbot, a move that would put Gemini in front of millions of Walmart shoppers and test whether AI chatbots can drive real commerce at scale rather than isolated queries. (Amazon, OpenAI, and Microsoft are experimenting with similar AI shopping tools.)

The stock is up nearly 1% again in premarket trading today. While Microsoft and Apple have both crossed $4 trillion in the past, they’ve since dipped below it, leaving Google and Nvidia as the only companies currently valued above the threshold.

tech
Jon Keegan

Meta reportedly plans to cut about 10% of employees in Reality Labs metaverse business

Meta is planning to cut roughly 10% of its Reality Labs employees, according to a report from The New York Times. The division is home to products related to the namesake of the company — the metaverse — which includes virtual and augmented reality glasses and headsets. Employees working on the metaverse are the target of the cuts, per the report.

Reality Labs has been bleeding cash and struggling to build significant revenue, racking up losses of around $70 billion since the company started reporting its numbers in 2020. Other than Meta Ray-Ban glasses, the group’s products have not been popular with consumers, and the idea of the metaverse that CEO Mark Zuckerberg evangelized never took off.

The company has since pivoted to a focus on building AI “superintelligence.”

Meta Plans to Cut 10% to 15% of Employees in Reality Labs Business

Meta Plans to Cut 10% to 15% of Employees in Reality Labs Business

Reality Labs has been bleeding cash and struggling to build significant revenue, racking up losses of around $70 billion since the company started reporting its numbers in 2020. Other than Meta Ray-Ban glasses, the group’s products have not been popular with consumers, and the idea of the metaverse that CEO Mark Zuckerberg evangelized never took off.

The company has since pivoted to a focus on building AI “superintelligence.”

Latest Stories

Sherwood Media, LLC produces fresh and unique perspectives on topical financial news and is a fully owned subsidiary of Robinhood Markets, Inc., and any views expressed here do not necessarily reflect the views of any other Robinhood affiliate, including Robinhood Markets, Inc., Robinhood Financial LLC, Robinhood Securities, LLC, Robinhood Crypto, LLC, or Robinhood Money, LLC.
©2026 Sherwood Media, LLC