How do AI models stack up vs. humans on standardized benchmarks?

AI is getting good at a lot of different tasks

4/17/24 10:46AM

The state of AI

As AI has gone mainstream, companies haven’t been shy in deploying the technology — applying it to manual and repetitive tasks and even citing it as a reason for mass layoffs.

But how does AI compare to humans on technical tasks? A new report, Stanford University’s 2024 AI Index, summarizes where the burgeoning technology is at.

The headline is that recent breakthroughs have heralded an unprecedented improvement in the performance of AI models on benchmark tests. For a long time, AI has been able to tell what’s in a picture, even as websites ask us to endlessly prove we’re not a robot by clicking on images of traffic lights or stop signs.

But now, AI is doing visual reasoning and math — seriously hard math.

Indeed, the 2024 AI Index reports that models have gone from scoring less than 10% of the relative performance of humans to more than 90% in just 2 years in competition-level math. In more simple tasks, the AI models evaluated already outperform the relevant human benchmarks.

The good news for anyone worried about losing their job is that AI researchers are increasingly concerned about running out of high-quality data to train their models, with some predicting that the available supply will be exhausted by 2026. This shortage might force developers to depend increasingly on AI-generated, or 'synthetic', data for training new models. Adobe’s solution? Pay people $3 a minute for videos of them touching things.

Related reading: How ChatGPT broke the Turing test.

Rani Molla2h

Some 8% of kids ages 5-12 have interacted with AI chatbots like OpenAI’s ChatGPT or Google’s Gemini, according to a new Pew Research Center survey of their parents. While that’s nowhere near the usage rates of other devices like smartphones or even voice assistants, it’s still notable for a relatively new technology — especially one that’s already had devastating consequences for young people.

Rani Molla5h

Ives says he’s “relatively disappointed” in the price point of lower-cost Tesla models

On Tuesday, Tesla unveiled its long-awaited lower-cost cars, which turned out to be downgraded versions of the existing Model Y and Model 3. Tesla bull and Wedbush Securities analyst Dan Ives wasn’t particularly impressed with the price point, noting that it’s “still relatively high versus other vehicles on the market.”

The Model Y Standard and Model 3 Standard cost about $40,000 and $37,000, respectively. That’s more than the Model Y Premium and Model 3 Premium — what previous editions (or “trim levels”) are now called — cost last month, before the US federal government’s $7,500 tax credit expired. And the Standard models are missing a lot of Premium features, including Autopilot, second-row screens, and Tesla’s iconic glass roofs, among numerous other downgrades.

In other words, Tesla buyers will now be paying more for less, in what amounts to car-sized shrinkflation.

The stock closed down 4.5% yesterday on the news.

Ives doesn’t think it’s the end of the world but is “disappointed” in the price tag:

“We believe the launch of a lower cost model represents the first step to getting back to a ~500k quarterly delivery run-rate which will be important to stimulate demand for its fleet with the EV tax credit expiring at the end of September but we are relatively disappointed with this launch as the price point is only $5k lower than prior Model 3’s and Y’s.”

Rani Molla5h

Nvidia helps boost xAI funding round to $20 billion

xAI’s latest funding round has now doubled to $20 billion from $10 billion a month ago, thanks in part to backing from Nvidia, which invested $2 billion in the equity portion of the transaction, Bloomberg reports. In an interview with CNBC, Nvidia CEO Jensen Huang confirmed the investment, adding that his “only regret” was that he didn’t give xAI more money.

The mix of $7.5 billion in equity and $12.5 billion in debt will finance a special purpose vehicle that will purchase Nvidia chips that xAI will then rent. It’s one of many circular AI deals these days that’s contributing to chatter over an AI bubble by some, while being seen by others as a rational way for industry leaders to boost the potential size of the addressable market and lift their longer-term prospects in the process.

Investors in Elon Musk’s other company, Tesla, will vote next month at the company’s annual shareholder meeting on whether to invest in xAI as well — an outcome Musk has he said supports.

Musk’s xAI Nears $20 Billion Capital Raise Tied to Nvidia Chips

Investors in Elon Musk’s other company, Tesla, will vote next month at the company’s annual shareholder meeting on whether to invest in xAI as well — an outcome Musk has he said supports.

Rani Molla

Electric Slide

22h

Tesla unveils cheaper Model Y and Model 3

The company announced a $39,990 Model Y today.

$1T

Jon Keegan10/7/25

In the past few weeks, OpenAI has announced a flurry of massive deals with Oracle, Nvidia, CoreWeave, AMD, and others as hundreds of billions fly between technology partners racing to expand AI infrastructure at unprecedented scale. The Financial Times tallied it all up and found that the company has signed about $1 trillion worth of deals, and it isn’t clear at all that it will be able to fund them.

The “circular” nature of some of these arrangements is also one factor playing into fears that we’re in an AI bubble.

The state of AI

More Tech

Ives says he’s “relatively disappointed” in the price point of lower-cost Tesla models

Nvidia helps boost xAI funding round to $20 billion

Musk’s xAI Nears $20 Billion Capital Raise Tied to Nvidia Chips

Latest Stories