Is AI Stealing Freelance Jobs? This Study of 240 Real Projects Has the Answer

It feels like AI is claiming new territory every forty eight hours.

One day it is Google announcing a new Gemini update. The next day we are looking at something like "Gemini Nano Banana" or a new open source breakthrough. We look at the benchmarks and the graphs go up and to the right. It all looks like an exponential rocket ship.

But there is a problem.

The real world feels different. It is messy. It is complicated. It involves massive datasets, corrupted files, external systems, and clients who do not know exactly what they want.

Well, the team at Center for AI Safety and Scale AI just gave us one. They released a massive, comprehensive paper titled the "Remote Labor Index" and the results were staggering.

This Was Not a Multiple Choice Test

Most AI tests ask the model to answer a question or write a short script. This study did something much harder.

They went to Upwork.

They gathered 240 real, paid freelance projects. These were not synthetic tasks made up by researchers. These were actual jobs that humans had been paid to do. The average project cost was over $600 and they ranged from 3D animation and architectural blueprints to game development and complex data visualization.

You can explore the results here:

Crucially, the researchers are trying to avoid the "contamination" problem that plagues so many other benchmarks. To ensure future AI models don't just memorize the answers, they have only published 10 of these projects. The remaining 230 are kept strictly private in a secure test set.

Then they unleashed the heavy hitters of the AI world on these projects. They tested GPT-5, Claude Sonnet 4.5, Grok 4, and a specialized agent called Manus.

If you believe the hype on Twitter, you might expect the AI to crush these tasks.