The state of AI

As AI has gone mainstream, companies haven’t been shy in deploying the technology — applying it to manual and repetitive tasks and even citing it as a reason for mass layoffs.

But how does AI compare to humans on technical tasks? A new report, Stanford University’s 2024 AI Index, summarizes where the burgeoning technology is at.

The headline is that recent breakthroughs have heralded an unprecedented improvement in the performance of AI models on benchmark tests. For a long time, AI has been able to tell what’s in a picture, even as websites ask us to endlessly prove we’re not a robot by clicking on images of traffic lights or stop signs.

But now, AI is doing visual reasoning and math — seriously hard math.

Indeed, the 2024 AI Index reports that models have gone from scoring less than 10% of the relative performance of humans to more than 90% in just 2 years in competition-level math. In more simple tasks, the AI models evaluated already outperform the relevant human benchmarks.

The good news for anyone worried about losing their job is that AI researchers are increasingly concerned about running out of high-quality data to train their models, with some predicting that the available supply will be exhausted by 2026. This shortage might force developers to depend increasingly on AI-generated, or 'synthetic', data for training new models. Adobe’s solution? Pay people $3 a minute for videos of them touching things.

Related reading: How ChatGPT broke the Turing test.