☢️ Apocalypse PBC

Bioweapons! Atomic war! Political skullduggery!

There’s seemingly no manner of calamity that AI won’t help plan, if only the people who make it bravely intercede. Last week saw the release of a number of reports about the capabilities of what frontier AI can do. The question on many minds is: can AI hurt us, or even kill us?

That’s a boring question. The better question is “why did all the major AI labs spend all of last week talking about this?” because the answer has material impacts for the regulation of AI in the United States. 

• First, the news. Anthropic reported three incidents where models broke containment and reached real targets. This comes two months after OpenAI reported its frontier model breached HuggingFace. Anthropic also said that it intercepted requests to design biological weapons, and caught Russian state media using the models to make propaganda. 

• Iran also allegedly used Claude to try to target U.S. Navy ships, which came after AI was used to plan the American attack on Iran.

• And while not official company news, the loud departure of an Anthropic staffer over safety concerns captured headlines — and an Anthropic data scientist responding with, “We really do earnestly believe AI could kill all humans! I personally think it is >10% within the next decade,” didn’t exactly assuage concerns. 

But again, the question is not why, but why now? What do they want?

• Well, for the large AI labs pushing these stories, the answer is somewhat self-interested. If things proceed as the polls indicate, the House of Representatives and possibly the Senate will be held by Democrats come January, breaking up a GOP trifecta. 

• There are several pieces of AI regulation under consideration, many of which are supported by the AI industry and their government relations teams in Washington. 

• If Democrats take one or more chambers, it’ll be harder for the AI labs to get the kind of regulation that they want done, especially given how opposition to data centers has been a motivating issue of the left. The tech industry has learned the advantages of regulation. When social media companies got regulated, only the ones that made a lot of money and could afford compliance departments managed to survive. 

If I’m Anthropic or OpenAI, my ideal regulation puts onerous requirements on AI labs that only the largest can afford, demanding closed models that are controlled tightly by a single entity, thereby killing off the open-source model scene and giving the federal government fiat to turn off AI that they don’t like in places they don’t like.

And golly, would you look at that, the bipartisan AI Kill Switch Act sure looks like it may have the votes, and does exactly that! Axios reported Friday of an effort to bring the House back and keep it in session until Congress passes AI regulation. What a coincidence of timing that this happens the week after all these scary reveals!

The Takeaway

There’s another side of these revelations too, and it’s got ramifications for American AI as well. While most of the eyes on the Anthropic threat assessment report were likely on the “biological misuse” and the “conventional weapons” headlines, the section about illicit distillation has pretty interesting implications for investors trying to assess the state of the art. 

The idea behind model distillation is that countries that don’t have access to American frontier AI (often U.S. adversaries) or want to get local AI labs to catch up by any means necessary (also often U.S. adversaries) will attempt to bombard Claude and other models with requests that reveal their reasoning abilities, so that they can train their own models. 

Anthropic’s report calls out a few Chinese labs — Alibaba (which they say trained Qwen on Claude), Moonshot (which they say silently forwarded customer requests for Kimi to Claude), DeepSeek (which Anthropic said did the same), and Z.ai (which they claim an a chain-of-thought extraction pipeline against Claude) — and questions whether their progress came as organically as believed. 

Why that’s relevant is that some American labs that appeared to underwhelm with recent demos may be worth a second look. For instance, if the initial models out of Thinking Machines — founded by ex-OpenAI CTO Mira Murati — seemed to underperform Chinese labs, well, if the latest out of those labs resulted from simply distilling Claude, perhaps Thinking Machines is worth another look. 

— Walt Hickey

Holiday travel may be getting more expensive as the war in Iran drags on. Last week the price of jet fuel was up 9% over the prior week and up 90% from this time last year according to the International Air Transport Association (IATA) Fuel Price Monitor. 

Airline executives say that this raises alarms for travel costs. Ryanair’s CEO Michael O'Leary said Thursday that airfare prices may see “significant” increases if oil prices continue to rise. 

Last week, as JetBlue raised its third quarter revenue outlook, it also flagged rising fuel costs as a headwind. The airline said it expects to pay $3.96 per gallon for jet fuel in Q3, up from its $3.49 estimates. Those prices are likely to get passed onto the consumer. 

“The average traveler may be priced out of the holiday season,” Katy Nastro, aviation and travel correspondent at Flighty, told Forbes.

— Paige Oamek

• 🏈 MNF: The Kansas City Chiefs are 56% favorites* to beat the Denver Broncos in the Monday Night Football season kickoff, in an AFC matchup that pits two strong playoff contenders against one another. 

• 🏛️ Fed: The probability that the Federal Reserve hikes rates by 25 basis points in September has soared to 75%, up from 45% a week ago and up from 27% as recently as the end of August. 

*Event contracts are offered through Robinhood Derivatives, LLC — probabilities referenced or sourced from KalshiEx LLC or ForecastEx LLC.

That’s the projected amount of U.S. cookie sales in 2026. 

• Monday: Morgan Stanley hosts a health care conference through Wednesday. 

• Tuesday: August retail sales report slated for release at 8:30 a.m. ET. 

• Wednesday: Federal Reserve interest rate decision and updated economic projections due out at 2:00 p.m. ET. Lennar slated to release quarterly results after the close.