(Arc Prize)

The toughest AI benchmark just got a whole lot tougher

ARC-AGI-3 is the latest version of a clever benchmark that challenges AI models to solve mini video games with no written instructions.

Jon Keegan

3/26/26 3:10PM

The flood of new AI models with increasingly advanced “reasoning” capabilities is forcing the AI industry to abandon early benchmark tests and invent new ones to test for many skills.

To watch the evolution of one such test — ARC-AGI — is to witness the huge technical leaps that today’s generative-AI models have made in a few short years. Tech CEOs brag about their models’ high scores on ARC-AGI, as it is widely considered one of the most unique and difficult AI benchmarks in use today.

Rather than testing how well a model can translate an inscription on an ancient Roman tombstone, or offer a diagnosis for a complex medical case, ARC-AGI challenges AI models to analyze abstract geometric puzzles and games without any written instructions. This ensures that the models are forced to create solutions to complex multistep problems, rather than regurgitate text from their training.

We created an in-house game studio and built 135 novel environments from scratch

No instructions, Core Knowledge Priors-only

In order to beat these games, AI must:
• Explore the environment
• Form hypotheses
• Execute a plan
• Learn and adapt pic.twitter.com/oaSVhut7Cp
— ARC Prize (@arcprize) March 25, 2026

The latest version that just launched, ARC-AGI-3, is basically a collection of mini games, which the user can play by moving simple shapes through a pixelated game board using directional arrows. As designed, the games are easy for humans to figure out after a few minutes of experimentation, but incredibly difficult for computers to solve.

François Chollet, the creator of ARC-AGI, told Sherwood News in an email:

“You can’t cram for the test. It requires you to explore and figure out each environment on the fly, on your own, instead of relying on extensive training data. Humans are really good at adapting to novelty, but AI systems are still fundamentally reliant on memorized templates.”

Chollet said that even after AI models have seen thousands of games, they struggle with ARC-AGI-3 games, since they are all unique.

One of the fascinating new features of the latest version is a replay mode that lets human observers read through AI models’ “chain of thought” transcript to see how a model breaks down the problem and attempts a solution.

Humans can play through these games on the project’s website. For now it seems humans don’t have much to worry about.

The most capable state-of-the-art models in the wild haven’t even cracked a 1% score (out of 100). The current leaderboard for ARC-AGI-3 shows OpenAI’s GPT-5.4 in the lead at 0.3%, and tied for second place are Anthropic’s Opus 4.6 and Google’s Gemini 3.1 Pro. xAI’s Grok 4.20 Reasoning model got a 0%.

Chollet says his team is already working on future versions of ARC-AGI:

“We are currently working on ARC-AGI-4 and ARC-AGI-5. We will release a new benchmark every year, each time asking the most important unsolved questions on the way to AGI. Three important topics we’re looking at for future versions are continual learning, open-endedness, and autonomous invention.”

Updated to include comments from François Chollet.

Chris Stokel-Walker

Bot bias

6/17/26

Companies are getting AI chatbots to smear their competitors

The race to influence AI chatbots is leading to some companies to adopt shady competitive tactics.

Tom Jones6/17/26

Prediction markets have, predictably, been given a boost by the summer of sports

Major platforms like Kalshi and Polymarket have seen huge upticks in users of late, thanks in no small part to what’s felt like a recent sporting smorgasbord, with major competitions across hockey, basketball, and soccer soaking up fans’ time (and spending, clearly) at the outset of summer.

While gaming industry groups may not like it, there’s been a huge change in the methods people are using to put money on the big games, with everyone from fortunate NYC bar owners, to a far less fortunate Spanish supporter, turning to prediction markets to try and turn their sports know-how into cold, hard cash.

According to a new report from Adam Blacker for apptopia, that shift might have been even more seismic than imagined in the wake of the NBA and NHL finals and around the 2026 World Cup kicking off.

2026 World Cup Reverses Seasonal Lull for Sports Betting Apps

According to a new report from Adam Blacker for apptopia, that shift might have been even more seismic than imagined in the wake of the NBA and NHL finals and around the 2026 World Cup kicking off.

South by Southwest Conference and Festivals

Gold Tesla Cybercabs are piling up, but they’re not picking up passengers yet

Low-volume production started in April. Now people are noticing them more and more in the wild.

Rani Molla6/15/26

Millie Giles

TAKING ACCOUNTS

6/15/26

Britain announces social media ban for under-16s starting early 2027

The UK government plans to use the same model for the restrictions as Australia — but how successful has that case study been so far?

UK Prime Minister Announces Under-16s Social Media Ban

Jon Keegan6/15/26

Anthropic pulls Fable and Mythos access worldwide after Trump administration bars their use by foreign nationals

Only days after releasing two versions of its next-gen AI model, Anthropic has disabled them for users worldwide.

Anthropic says it received a Friday night order from the Trump administration to suspend access to the models for any foreign national (anywhere in the world) — a group that included some Anthropic employees. In response, the company turned off access to everyone.

Last week, the company released to the public its much-anticipated Claude Fable 5 model (and its restricted version Claude Mythos 5, which is still being tested with trusted partners). Anthropic said in a blog post announcing the action that officials cited national security concerns with the new models, while offering few specific details.

The post said that the government gave the company “verbal evidence of a potential narrow, non-universal jailbreak” of the public Fable 5 model. A jailbreak is a means by which users can evade restrictions built into the code to unlock prohibited functionality. Anthropic downplayed the significance of the attack, and said other major models, such as OpenAI’s GPT-5.5, could also be affected by the technique described.

Fears of these first Mythos-class models being misused are running high, after Anthropic warned the cybersecurity world in May that the advanced cyber capabilities of Mythos have rapidly discovered thousands of vulnerabilities in ubiquitous software, leading to the decision to restrict the full version of the model to a close group of trusted partners for testing.

This morning, Axios reported that Anthropic technical staff have flown to Washington to meet with White House officials to resolve the issue.

The Wall Street Journal is reporting that the Trump administration’s decision to take action against Anthropic was prompted by discussions that Amazon CEO Andy Jassy had with officials, including Treasury Secretary Scott Bessent. According to the report, Amazon researchers said they had been able to evade some of Fable 5’s security restrictions using specific prompts. Amazon is a major investor in Anthropic.

Anthropic is currently suing the US government to fight the Pentagon’s blacklisting of the company on national security grounds.

Statement on the US government directive to suspend access to Fable 5 and Mythos 5