In a blind test published in August 2026, people held live conversations with two hidden AI voice agents and chose which one they preferred. Google’s Gemini 3.1 Flash Live Preview in its Minimal configuration finished at the top of the preference ranking with an Elo score of 1,046, although it completed only 74.6% of the eligible tasks it was given.
Several agents that people preferred less often completed more tasks. OpenAI’s GPT Realtime model from August 2025 recorded a lower preference score of 944 and a task success rate of 89.4%, while Grok Voice Think Fast 2.0 High led task success at 94.7%.
The model people most enjoyed speaking with was therefore not the model most likely to finish the job. That result came from Artificial Analysis’s Speech Agent Arena, and the way the arena was constructed explains how both statements can be true.
One conversation produced two separate records
Each participant received a scenario and completed it separately with two hidden speech-to-speech systems. The arena contained 35 scenarios at launch. Twenty involved conversation without tools, such as asking about a swimming pool’s Sunday hours. Fifteen required an action through a tool, such as ordering food within a budget or booking the earliest available dental appointment.
After both calls, the participant chose the conversation they preferred. Those pairwise choices produced the preference ranking.
The tool trace produced another record because an eligible task counted as successful only when the agent made the correct final call, used the right tool, supplied the required arguments and avoided extra final actions that the participant did not request. The evaluator inspected what the system actually submitted instead of accepting a pleasant confirmation as proof.
Preference and task success were also calculated from different pools because preference covered agentic and non-agentic conversations, while task success applied only to eligible conversations that required tools. The two scores should therefore not be subtracted as though they measured the same event.
They can still disagree because one score belongs to the conversation the person experienced, while the other belongs to the action recorded behind it.
Artificial Analysis reported that highly preferred models in reviewed conversations tended to respond quickly, sound natural and produce fewer audio artefacts. Its highest-preference model had a time to first audio of 0.96 seconds. A quick, smooth reply can improve the call even when the final booking or order is wrong.
The arena researchers described cases in which a conversation sounded as though the requested action had been completed even though the required final tool call had failed, which meant the user heard a confident confirmation while the outside system still held the wrong state.
“Good” occupies more than one square
A voice agent can land in four different places:
| Conversation | Task result | What the user experiences |
|---|---|---|
| Pleasant | Correct | The agent feels good and finishes the job |
| Pleasant | Wrong | The agent sounds convincing while leaving the wrong result behind |
| Awkward | Correct | The task succeeds, but the call feels slow, stiff or difficult |
| Awkward | Wrong | Both the conversation and the action fail |
The second square is easy to miss in a product demonstration. A polished voice can pronounce the customer’s name correctly, pause at natural moments and confidently say that a reservation has been changed. None of those audible qualities proves that the booking tool received the right date.
The third square creates a different mistake. A developer hears an awkward confirmation and replaces the speech model even though the agent made the correct tool call. The conversation improves after the replacement, but the component responsible for successful action was already working.
Some voice agents make the separation visible in their architecture by using a speech recognizer to turn the caller’s audio into text, a language model to decide what to say and which tools to use, and a speech generator to read the answer aloud. Other systems process speech more directly and do not expose three clean stages, but the product still leaves evidence at different levels, including recorded audio, a transcript or internal representation, tool calls and the final state of the outside system.
Each level answers a different question, so the evidence should show whether the agent heard the request, chose the right action, sent the correct arguments to the tool, completed the action and described the result accurately in its spoken confirmation.
The call can fail at any one of those boundaries while sounding normal everywhere else.
Test an outcome that cannot hide behind the voice
Use a harmless test account and create a task with a result you can inspect. A calendar is useful because the final state is visible.
Put a meeting on Thursday, make Priya unavailable on Friday and tell the agent, “Move Thursday’s meeting to Friday, but only if Priya can attend.”
The correct result is to leave the meeting where it is and explain that the condition was not met. A fluent “Done, I moved it to Friday” is a failure, however natural the delivery sounds.
Then make Priya available and repeat the request, which should now move the meeting exactly once. Record the instruction, agent and model version, transcript where available, tool call, final calendar state and spoken confirmation, and run both cases more than once.
Listen to the same recordings separately after checking the calendar. Judge speed, interruptions, pronunciation, pacing and comfort without letting the correct task result decide whether the voice sounded good. Then inspect the task result without giving the pleasant voice extra credit.
This separation also helps when testing the speech itself. Cartesia’s guide to text-to-speech evaluation points out that a voice can sound natural while mishandling a name, number or abbreviation. Running the generated speech through a recognizer creates another possible error because the recognizer may mishear a correct generation or recover the intended words from poor pronunciation, so listening and transcription supply different evidence.
For a tool-using agent, keep one record of the human exchange and another of the system state after it. A restaurant agent should leave the right order, quantity, address and charge, a support agent should make the permitted account change, and a scheduling agent should leave the calendar in the promised state.
The benchmark has boundaries
Artificial Analysis used paid, screened participants, and at launch it kept most scenario prompts and tool schemas private to reduce overfitting. Participant deviations and unverifiable conversations were excluded from the task-success calculation, while preference scores carried publication thresholds and confidence intervals.
The leaderboard will also change. The August 2026 result describes specific systems, configurations and a particular arena version. It does not establish that Gemini is generally less capable, that users always prefer worse agents, or that natural speech causes tool failures.
The measurement problem remains after the leaderboard changes because a voice agent produces something you hear and something the outside system must do, and the product has to succeed at both.
When an agent says the reservation is confirmed, use the recording to judge how the conversation felt and then open the reservation system to check what happened.
If the agent also feels slow after you stop speaking, Rohkun’s guide to locating voice-agent delay follows the timing evidence through the same call. To understand what a reference recording contributes to a generated speaker, read what a voice-cloning model learns from one recording.
Sources
- Artificial Analysis: Announcing the Speech Agent Arena, 24 August 2026.
- Artificial Analysis: Speech-to-speech benchmarking methodology, accessed 7 September 2026.
- Artificial Analysis: Speech Agent Arena leaderboard, accessed 7 September 2026.
- Cartesia: “Is this TTS model good?”, 28 July 2026.