A European politician said that a committee had voted on more than 1,600 amendments. Researchers took the recording, silenced the words “one thousand six hundred” and gave the altered clip to several speech recognition models, some of which returned the exact number even though its sound was gone.
This was part of a new audit of 11 open speech recognition models, including Whisper Large v3, Cohere Transcribe, Canary Qwen, Granite Speech, Phi-4 Multimodal, Qwen3 ASR and Voxtral. On the familiar LibriSpeech benchmark, some of the strongest models recovered the exact hidden number in roughly 30 to 40 percent of the tested examples.
The surrounding sentence can help a model guess that a number belongs in the gap. It should not usually reveal whether the missing number was 16, 1,600 or 16,000. The researchers were therefore left with a much stranger possibility. A system could appear to hear a recording more accurately because it recognized the test and knew what the published transcript expected.
A benchmark contains two things
Speech recognition is often measured with word error rate. A model hears a collection of recordings, produces transcripts and receives a lower error rate when its words match the official reference transcripts.
This sounds straightforward, although each benchmark contains two different objects, which are the audio that the model is supposed to hear and a written answer key created by people.
The answer key is separate from the sound and may use a spelling convention such as “Mister” instead of “Mr.,” omit a polite phrase that was clearly spoken or contain an ordinary transcription mistake.
The new paper, Towards Quantifying Benchmark Optimization in ASR Models, uses these differences as traps. If a model follows the audio, it should disagree with a mistaken answer key. If it follows the answer key, it can earn a better score while producing a less faithful transcript.
The researchers call this benchmark optimization, while its informal name is benchmaxxing. Some of the apparent gain on a public test may come from behavior fitted to that test rather than a general improvement in hearing new speech.
The first trap used mistakes already in the answer key
The team focused on VoxPopuli, a widely used collection of European Parliament recordings. Its reference transcripts contain enough errors to create a useful natural experiment.
Four independent models first helped identify places where the written reference appeared to disagree with the audio. The researchers found 1,113 candidate edits across 745 test clips. A separate human-annotated version of the dataset contained 93 percent of the edits flagged by the model panel, which gave the team a check against relying entirely on machine judgment.
They then asked a simple question. When the recording and the answer key disagree, which one does each speech model follow?
The six models with the lowest word error rates on VoxPopuli were also the six most likely to reproduce its questionable reference text. They copied the reference in 18 to 30 percent of the flagged cases. Every model with a noticeably worse benchmark score did so in 10 percent or fewer.
The models with the cleanest scores were therefore also the ones most willing to depart from audible speech when the benchmark expected something else.
Then the researchers removed the answer from the sound
Existing transcription errors leave room for arguments about what a person truly said, so the second test created a cleaner disagreement.
The researchers selected numbers because the rest of a sentence rarely determines their exact value. They located a number in the recording and replaced its audio with silence. A faithful transcription system could omit the missing span, mark uncertainty or make a context-driven guess. It should rarely recover the exact number from the sound because that sound no longer exists.
Some leading models nevertheless returned the benchmark’s original number. The strongest recovery rates on LibriSpeech reached about 40 percent.
One example came from the European Parliament clip about the 1,600 amendments. Its official reference transcript was already wrong because it said “more than 1 amendments.” After the real number was silenced, several systems produced the reference version or supplied the missing year and other words that were also absent from the edited audio.
Recovery was stronger on the old public benchmark recordings and weakened for several models on newly collected recordings from the same kinds of sources, which makes a universally familiar quotation an unlikely explanation. The researchers used European Parliament audio recorded after the models’ training cutoffs and readings from newly active LibriVox narrators, keeping the subject matter similar while changing whether the exact recordings could have been encountered before.
Even spelling became a clue
The third test avoided missing words entirely. It looked at spellings that sound the same.
LibriSpeech contains both “anyone” and the older form “any one.” Other public datasets differ in whether they write “Mr.” or “Mister.” A listener cannot hear the space in “any one,” and both versions of the title sound the same.
If a model has one ordinary writing preference, it should use that preference consistently. If it changes spelling to match the convention of each benchmark, then it has identified more than the words being spoken.
Eight of the 11 tested models beat the chance baseline on the “anyone” switch. Six did so on the title switch. Some reached roughly 90 percent accuracy at selecting the spelling expected by the particular dataset.
This is a small behavior with a large implication. The model may infer which test it is hearing from a speaker, recording channel or other acoustic detail, and then adopt the test’s house style.
The audit cannot identify one cause
These results are consistent with memorization, but the paper does not establish one cause that cleanly.
A 2025 study called Evaluation of LLMs in Speech is Often Flawed found that substantial portions of the LibriSpeech and Common Voice evaluation transcripts appeared in public language model training corpora. In controlled comparisons, contaminated language models assigned higher probabilities to test transcripts they had already seen. However, the difference in final speech recognition error was subtle.
The newer audit finds benchmark-shaped behavior across several kinds of speech model. Its authors say that training data leakage, memorization, benchmark-guided model selection and other forms of optimization could all contribute. Most model makers do not publish enough detail about their training mixtures to separate those causes.
The researchers could test what triggered the behaviour. When they cut a clip down to a narrow window around the disputed words, several models stopped copying the benchmark and returned to the audible version. Adding eight seconds of ordinary conversational audio also reduced the behaviour in several systems, while adding more VoxPopuli audio often preserved it.
The model therefore seemed to recognize a local acoustic world. The voice, recording conditions and surrounding clip told it that this sounded like VoxPopuli. That cue could switch on a transcription policy fitted to VoxPopuli, even when the policy contradicted the words in one small part of the recording.
Using context is normally a strength. People also use the speaker, sentence and situation to resolve unclear speech. The failure appears when context makes a system prefer a known answer key over clearer acoustic evidence.
A leaderboard score now needs a second question
None of this makes public benchmarks useless. They give researchers a shared test, make progress visible and allow models to be compared under repeatable conditions. The audit itself was possible because the recordings and reference transcripts were open.
Public tests also invite repeated optimization. Teams inspect the same datasets, tune decoding rules, choose checkpoints and make product claims against scores that everyone can see. Even when nobody deliberately trains on a test set, years of decisions can bend toward it.
The Open ASR Leaderboard added private evaluation data in 2026 so its maintainers could test models on recordings that developers cannot inspect in advance, including scripted and conversational speech from several accents. They also added a benchmark-fitting view that tracks reference errors and spelling switches.
For someone choosing a transcription model, a leaderboard should start the evaluation. The next question is what happened after the leaderboard.
Take recordings from the actual microphones, rooms, accents and vocabulary your product will encounter, keep part of that set private and include names, numbers and phrases where one wrong word matters. Compare the output with the sound as well as a reference transcript because the reference can be wrong too.
A model that fills a silent gap with the correct benchmark answer has learned enough about the test to know what belongs there, but that knowledge has interfered with the job the benchmark was meant to measure.
Sources
- Towards Quantifying Benchmark Optimization in ASR Models, Hume AI Research, August 2026.
- Evaluation of LLMs in Speech is Often Flawed: Test Set Contamination in Large Language Models for Speech Recognition, May 2025.
- Adding Benchmaxxer Repellant to the Open ASR Leaderboard, Hugging Face, May 2026.