Record yourself saying, “The meeting starts at nine.” Give that clip to a voice-cloning system, type a sentence you never recorded, and the system may read it back in something recognizably close to your voice.
The reference clip contains your words, your voice, the way you delivered those words and the sound of the room, but none of those parts arrives with a label because they reach the model as one waveform.
So when the generated voice sounds right, what exactly did the system get right?
One recording carries several things at once
Start with the words. “The meeting starts at nine” has a sequence of sounds that a listener can recognize as language. Change nine to five and the linguistic content changes, even if the same person reads both versions.
The speaker contributes another set of clues. The size and shape of a person’s vocal tract, their habitual pronunciation and many learned speaking habits affect what listeners recognize as that person’s voice. Voice-cloning papers often compress some of these clues into a speaker representation.
Delivery changes within the same person, so read the sentence once as a warning and again as good news. The words and speaker remain stable while pitch, pace, stress, rhythm and pauses move, and speech researchers group many of these features under prosody. In ordinary language, prosody carries much of the difference between what you said and how you said it.
The room leaves its own mark. A phone held in a tiled bathroom records different reflections and background noise from the same phone inside a treated studio. A microphone, compression setting and distance from the mouth can also alter the sample.
These parts are related and their boundaries are imperfect because an accent can help identify a speaker while also shaping delivery, and a laugh can carry identity and emotion together. The four-part split gives us four things to listen for without claiming that speech arrives in four tidy boxes.
One route compresses the speaker into a small code
A useful early architecture came from a 2018 paper by Ye Jia and colleagues. Their system had three separately trained components.
The first component was a speaker encoder. It had learned to tell speakers apart from recordings of thousands of people. Give it a few seconds of a new person’s speech and it produced a fixed-length numerical representation, commonly called a speaker embedding.
The second component took written text and that embedding, then generated an intermediate acoustic description called a mel spectrogram. The final component, a vocoder, converted the spectrogram into audible speech.
In practical terms, one network squeezes the reference recording into a compact description of who seems to be speaking, and another uses that description while deciding how the requested words should sound.
This design puts a bottleneck between the reference clip and the generated audio because the speaker encoder is trained to preserve information useful for distinguishing speakers without copying every cough, pause or refrigerator hum in the sample. The encoder’s training determines which parts it treats as speaker information and carries forward.
Another route keeps the reference closer to the generation
VALL-E, introduced by Microsoft researchers in 2023, took a different route. It represented speech using discrete codes from a neural audio codec, then treated text-to-speech as a conditional language-modelling task. A three-second recording of an unseen speaker could be supplied as an acoustic prompt.
The authors reported that generated speech could preserve the prompt’s speaker identity, emotion and acoustic environment. That last result is especially useful here. A model that follows the room as well as the person is copying more than the everyday meaning of “voice.” A reverberant sample may produce reverberant output because the recording itself is part of the instruction.
Different prompt-based systems preserve different amounts of room sound, so a reference clip cannot be treated as a neutral identity card. Some architectures allow more of the recorded moment to influence what comes back.
Some systems split identity and delivery on purpose
OpenVoice makes the distinction unusually explicit. Its authors describe cloning the reference speaker’s “tone colour” while exposing separate controls for emotion, accent, rhythm, pauses and intonation. The style does not have to be copied directly from the reference clip.
That design answers a common practical problem. Suppose the only available recording of a speaker is calm and slow, but the output needs to sound excited. A system that binds identity and delivery tightly may struggle to change one without disturbing the other. A system built to separate them can keep the recognizable speaker while directing a different performance.
Other systems draw the boundary elsewhere. YourTTS, published in 2022, extended a multilingual speech model for zero-shot text-to-speech and voice conversion. F5-TTS, published at ACL in 2025, uses a flow-matching system trained to fill in speech from reference audio and text. The reference sample matters in each case, but it does not pass through one universal voice-cloning diagram.
That is why model names alone tell you less than a controlled listening test.
Change one part of the reference at a time
Choose a sentence that contains ordinary words and no names likely to cause a pronunciation failure. Record four short reference clips with the speaker’s permission.
For the first pair, keep the speaker, sentence and delivery similar while changing the room. Make one recording dry and close to the microphone and let the other contain obvious room reflection, then generate the same target text from both and listen for background texture and reverberation in the outputs.
For the second pair, keep the speaker and room stable, then change the delivery. Read the reference once in a neutral voice and once with clear excitement or irritation. If the output changes its pace, pitch or stress, the reference is carrying delivery into generation.
For the third comparison, use two speakers reading the same reference sentence in roughly the same manner. This is the cleanest of the four checks for speaker resemblance, although microphone and accent differences can still interfere.
Finally, keep one reference clip fixed and change the target text. Include words, numbers or a language switch the speaker never used in the reference. This reveals whether a convincing vocal resemblance survives when the linguistic job becomes harder.
Write down the exact model version and settings before testing. Then judge the outputs in separate columns:
| What to judge | What to listen for |
|---|---|
| Words | Missing, added or mispronounced content |
| Speaker | Whether the same person remains recognizable |
| Delivery | Pace, stress, pitch, pauses and emotion |
| Recording | Room reflection, noise and microphone character |
A clone can score well in one column and badly in another because a clean, intelligible voice may lose the speaker, a convincing speaker may flatten every sentence and a lively performance may carry unwanted room sound from the prompt.
Only use a voice you own or have clear permission to reproduce because consent determines whether the test should happen at all, regardless of the model’s audio quality.
When a demo sounds “exactly like” someone, change the room, mood and sentence, then write down which parts followed the reference and which remained under the model’s control.
If you are also building live conversation systems, Rohkun’s guide to finding where voice-agent delay begins uses the same method: split one vague impression into events you can inspect.
Sources
- Ye Jia et al., “Transfer Learning from Speaker Verification to Multispeaker Text-To-Speech Synthesis”, NeurIPS 2018.
- Chengyi Wang et al., “Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers”, Microsoft Research, 2023.
- Zengyi Qin et al., “OpenVoice: Versatile Instant Voice Cloning”, 2023, revised 2024.
- Edresson Casanova et al., “YourTTS”, ICML 2022.
- Yushen Chen et al., “F5-TTS”, ACL 2025.