AI systems approaching month-long task automation in five years, new benchmark suggests

A team of computer scientists has developed a fresh approach to measuring AI capabilities against real-world work demands, moving beyond traditional benchmarking to evaluate how long it takes humans to complete tasks that AI systems finish successfully half the time.

A New Way to Measure AI Progress#

Rather than relying on abstract test scores, the researchers introduced the "50%-task-completion time horizon"—essentially asking: how long would a competent human expert typically spend on work that a given AI model can finish 50% of the time? Testing frontier models like Claude 3.7 Sonnet against a mix of established benchmarks and newly created shorter programming challenges, the team found that current leading systems operate at roughly a 50-minute time horizon. This translates to AI models now performing tasks that skilled developers would typically spend an hour on.

Doubling Every Seven Months#

The analysis reveals a striking trend in AI development velocity. Since 2019, the time horizon frontier has been expanding at an exponential rate—doubling approximately every seven months. Though the researchers note this trend may have accelerated further during 2024, the consistent pattern suggests a clear trajectory of increasing autonomy.

What's Driving the Progress#

The improvements stem from three core developments: AI systems are becoming more reliable and better at recovering from mistakes, they're making sharper logical inferences, and they're improving their ability to use tools effectively. These gains compound as models tackle longer sequences of reasoning and action.

Five-Year Outlook and Caveats#

If the current trend continues and generalizes to typical software engineering work, the researchers' projections suggest AI systems could automate many month-long programming tasks within five years. However, the team emphasizes important limitations: the external validity of their results remains uncertain, and questions about the real-world applicability of their benchmark tasks require further investigation. They also flag the safety implications of increasingly autonomous systems capable of handling complex, multi-stage work.