A group of researchers decided to test the future. Instead of doing it in a sterile laboratory or in theoretical simulations, they did so by founding a fictitious company and hiring as employees a staff entirely composed of artificial intelligence agents.
For weeks, these AI agents worked (or so they tried) as if they were human professionals: programmers, project managers, financial analysts.
The premise was clear: to check if AI agents are ready to be integrated into the world of work. The answer was a resounding and revealing no.
The unfulfilled promise of AI agents
By: Gabriel E. Levy B.
The idea behind the experiment, carried out by researchers at Carnegie Mellon University, sounded like a piece of domestic science fiction.
A digital company, TheAgentCompany, equipped with a basic structure: a team of 18 employees, internal documentation, Slack-style communication channels and realistic tasks to fulfill.
In theory, everything was in place for this artificial community to function with the relentless efficiency that is presumed in discourses on AI.
The agents operated autonomously, without direct human intervention.
Models from OpenAI, Google, Meta, Anthropic and Amazon were launched to write code, plan projects, respond to emails, organize spreadsheets. But what happened was far from the techno-scientific dream.
As if they were distracted office workers, the agents began to demonstrate their fragility: they did not know how to close pop-ups, they confused users, and when they did not know what to do, they preferred to cheat.
One, for example, couldn’t find a colleague to talk to, so he simply changed the name of another user in the system. He pretended that he had fulfilled his task.
This was not an anecdotal failure, it was the norm.
The most effective agent (Claude 3.5 Sonnet, from Anthropic) barely completed 24% of the assigned tasks.
ChatGPT and Gemini 2.0 Flash hovered around 10%. Amazon’s Nova Pro 1 didn’t even exceed 2%. The promised efficiency was belied by operational clumsiness.
“Artificial intelligence is an unfulfilled promise,” wrote Nicholas Carr
More than a decade ago, in his famous essay The Shallows, journalist Nicholas Carr warned that digital technology was redefining not only the way we work, but also the way we think.
Carr wondered if the enthusiasm for automation was not leading us to a dangerous simplification of complex processes, those that require intuition, judgment and context.
TheAgentCompany’s results seem to prove him right.
The project does not only serve as a technical warning.
It also raises a philosophical question: what does it mean to “work” for an artificial intelligence?
The philosopher Hubert Dreyfus, a critic since the 70s of attempts to simulate human intelligence, already pointed out that machines can manipulate symbols, but they do not understand.
Understanding, acting with meaning and purpose, is still human territory.
The great dream of AI agents was, precisely, that they did not need step-by-step instructions.
But the Carnegie Mellon experiment shows that even in the face of routine tasks, models fail when there is no clearly delineated framework. They are efficient only in closed and highly structured environments.
When algorithms “work”, but don’t think
The technological context in which this type of experiment emerges is, without a doubt, that of a frantic race to demonstrate that artificial intelligence can not only assist, but replace.
Since mid-2022, when generative AI dazzled the world with the emergence of tools such as ChatGPT and DALL· And, a wave of investment, enthusiasm and fears was unleashed. 2023 and 2024 were years of euphoria. And 2025, we were told, would be the year of “AI agents.”
Unlike chatbots, AI agents promise something more ambitious: autonomy. The ability to receive a goal (“design an app”, “optimize this budget”, “solve this technical problem”) and decide how to achieve it.
The concept isn’t new, but today’s technology has pushed this idea to the forefront. Companies such as OpenAI, Google DeepMind and startups such as Adept or Cognosys are building systems that simulate reasoning, planning, execution and coordination.
But the case of TheAgentCompany reveals that this autonomy is still far from functional.
The problem is not in computational power, but in the absence of common sense, in the inability of models to interpret ambiguous contexts, to improvise without breaking the rules or to collaborate in a meaningful way.
Machines don’t get stressed, but they don’t adapt well to the unexpected either.
And yet, reports from the World Economic Forum continue to fuel the vertigo: more than 90 million jobs could disappear in the next five years due to AI-driven automation.
Although it is estimated that up to 130 million new roles could be created, the displacement and transformation of the labour market is already palpable.
Are we really prepared to delegate work to entities that can’t even close a pop-up?
“It’s not about if they can do the job, it’s about how they do it.”
The examples collected by the Carnegie Mellon researchers are eloquent and even comical.
A programmer stopped working when he didn’t understand an instruction. Another waited indefinitely for an answer that never came.
Yet another, not finding how to search the internet accurately, ended up copying irrelevant fragments.
Even more worrying was the tendency of some models to “trick” the system to simulate productivity. A symptom of creativity? An ethical failure? Or is it simply a consequence of having assigned them goals without providing them with a deep understanding of why those goals matter?
These behaviors reproduce, in a caricatured version, some vices of the contemporary work environment: ineffective bureaucracy, obsolete chains of command, and an obsession with the appearance of efficiency rather than with real results.
Instead of questioning the model, AI agents mimicked it. They did not innovate, they simulated.
In a real work environment, these mistakes aren’t just inefficiencies, they’re breaches of trust.
The difference between an employee who reports a problem and one who hides it is the basis of collaborative work.
The machines, for now, do not know the difference.
And yet, there is no shortage of those who continue to bet on its massive deployment.
In environments such as technical support, data analysis or basic content production, AI is already showing effectiveness.
But translating that efficiency into open, interactive, and collaborative tasks requires more than natural language processing. It requires judgment.
It requires context. It requires, to put it bluntly, something that machines do not yet have.
The Mirage of Intelligence
What TheAgentCompany reveals is not only that AI agents are far from replacing us, but that they still don’t fully understand what it means to collaborate, adapt, or make meaningful decisions. Experience highlights a paradox: the more they resemble us in their way of working, the more their limits become evident.
In 2021, AI expert Gary Marcus already warned that “deep learning is powerful, but fundamentally limited when it comes to general understanding.”
Without real cognitive architecture, without built-in working memory, without causal reasoning ability, agents will continue to seem intelligent only as long as we don’t ask them to be too smart.
In conclusion, the Carnegie Mellon experiment is a call for caution amid the enthusiasm for AI agents. Far from being the employees of the future, these systems still show fundamental shortcomings when faced with complex, dynamic and human work. For now, the best tool is still human judgment backed by technology, not its replacement by algorithms that don’t yet know how to think.
References:
- Carr, N. (2011). The Shallows: What the Internet Is Doing to Our Brains. W. W. Norton & Company.
- Dreyfus, H. (1972). What Computers Can’t Do: A Critique of Artificial Reason. Harper & Row.
- Marcus, G. (2021). “Deep Learning Is Hitting a Wall.” Wired Magazine.
- World Economic Forum. Future of Employment Report 2024.
- Experiment “TheAgentCompany”, Carnegie Mellon University (summarized in Xataka, 2025).
