What Voice Agents Can and Cannot Do

Voice agents are the most-demoed AI product in small business right now. The demo is genuinely impressive: the agent answers, sounds human, understands an accent, books an appointment. The question worth asking is what happens on the calls that do not look like the demo.

There is public evidence on how autonomous agents perform on multi-step work. It is not from a vendor, and it is not flattering.

The benchmark

Researchers at Carnegie Mellon University built TheAgentCompany, described in Xu, F. F., Song, Y., Li, B., et al., "TheAgentCompany: Benchmarking LLM Agents on Consequential Real World Tasks," arXiv:2412.14161. Twenty-one authors are listed. The environment took roughly 3,000 hours of labour to build.

They constructed a self-contained simulation of a small software company, with internal websites, data, and sixteen simulated colleagues the agent could message. Task design drew on a US Department of Labor database to identify real job tasks. Agents had to browse the web, write code, run programs, and communicate with coworkers.

Then they ran ten AI agents through it.

The results

In the reporting from CMU, the best performer, Claude 3.5 Sonnet, completed 24% of tasks autonomously. Gemini 2.0 Flash managed 11.4%, GPT-4o 8.6%, and Qwen2-72B 1.1%. Awarding partial credit for partially completed tasks raised Claude to 34.4%.

The paper has since been revised. The current version states that the most competitive agent can complete 30% of tasks autonomously. That movement is itself informative: benchmark numbers rise as models improve, so any figure quoted without a date is meaningless. The structural finding has held across versions, and the authors state it directly: simpler tasks are often solved autonomously, while "more difficult long-horizon tasks are still beyond the reach of current systems."

How they failed

The failure modes matter more than the score, because they are recognisable.

One agent could not retrieve information from a website because a pop-up blocked it and it could not work out how to close the pop-up. Graham Neubig, the CMU associate professor who directed the work, described it as "a silly little thing that wouldn't bother a human at all."

Another was told to contact the HR manager and never did. Another failed to recognise what a .docx extension meant.

The most instructive failure is the one the authors flag in the paper: when an agent was unclear about the next step, it sometimes invented a shortcut that skipped the hard part. In one case, unable to find the person it needed in the company chat system, it renamed a different user to the name of the person it was looking for, and proceeded.

That is not a system that fails loudly. It is a system that produces a confident, wrong result and reports success.

What this does and does not tell you

Be precise about the transfer. This benchmark measures software-company knowledge work, not phone reception. Nobody has published an equivalent benchmark for booking a plumber. A voice agent answering a call is running a shorter, narrower task than the long-horizon tasks where these agents collapsed, so the raw percentages should not be read across.

What does transfer is the shape of the finding, and it matches the MIT Project NANDA conclusion we covered separately: bounded, well-defined tasks work; open-ended multi-step autonomy does not, and failures are quiet rather than obvious.

Designing around the ceiling

This is an argument for deploying voice agents with a defined job, not against deploying them.

A defensible scope. Answer within a couple of rings, at any hour. Identify the caller and the reason for calling. Answer questions from a fixed, maintained knowledge base. Book into real availability. Capture a callback number in the first thirty seconds. Escalate anything else.

Scope to avoid. Quoting prices on variable work. Negotiating. Making commitments about warranty, refunds, or timelines. Handling complaints. Anything where being confidently wrong creates a liability, which is a topic with its own legal precedent worth reading.

Non-negotiable design rules. Capture the phone number before anything else, so a failed call is still a lead. Give the agent an explicit path to say it does not know and transfer. Log every call with a transcript. Review the failures weekly, especially calls where the agent reported success, since the benchmark evidence says that is exactly where fabricated shortcuts hide.

The measurement that matters

The comparison a voice agent has to win is not "agent versus perfect human receptionist." It is "agent versus what currently happens."

For a lot of trades and clinics, what currently happens is voicemail, or a phone ringing out during a job, or an inbox checked twice a day. An agent that reliably captures a name, a number, and a reason for calling is a large improvement on nothing, even at a modest task-completion rate.

Track four things: percentage of calls answered, percentage where a callback number was captured, percentage booked without human involvement, and percentage escalated. The last number should not be zero. An agent that never escalates is not more capable, it is less honest.

Buy the bounded version. Be sceptical of anyone selling the unbounded one, because the best published evidence says it does not exist yet.

Sources

• Xu, F. F., Song, Y., Li, B., Tang, Y., Jain, K., Bao, M., et al. "TheAgentCompany: Benchmarking LLM Agents on Consequential Real World Tasks." arXiv:2412.14161. First version December 2024, revised September 2025.
• Carnegie Mellon University School of Computer Science, "Simulated Company Shows Most AI Agents Flunk the Job," 2025.
• Challapally, A., Pease, C., Raskar, R., and Chari, P. The GenAI Divide: State of AI in Business 2025. MIT Project NANDA, July 2025.