AI voice & dubbing

What Is Voice Cloning? How AI Copies a Voice, and Where It's Used

Voice cloning is using AI to create speech that sounds like a specific person, generated from a recording of their voice. Give a modern text-to-speech model a short clip of someone speaking, and it can say new sentences, even in other languages, in a voice that resembles theirs. It powers automatic dubbing and accessibility tools, and it raises real questions about consent and misuse.

5 min read · Updated

A simple definition

mydubly text to audio voice aiden energetic with style sample card

A voice clone is a model's imitation of one person's voice: their pitch, timbre, accent and speaking style. The model doesn't store recordings of every word; it learns characteristics of the voice and applies them to any text. The result is new speech the person never actually said, in a voice that sounds like theirs.

How voice cloning works

mydubly text to audio voice aiden bright with style sample card

Modern systems build on text-to-speech (TTS). A TTS model turns text into speech; a cloning-capable model also takes a reference recording and conditions its output on that voice.

  1. The reference audio is analysed into a compact representation of the voice, often called a speaker embedding.
  2. The text to speak is converted into a sequence the model understands.
  3. The model generates audio that says the text with the characteristics of the reference voice.

What text-to-speech is and how AI voice generation works explain the underlying technology.

Short-sample cloning

mydubly text to audio voice ryan angry with style sample card

Older systems needed hours of studio recordings of one person to build a voice. Newer zero-shot models clone from a few seconds of reference audio, with no training on that specific person. Quality depends heavily on the sample: clean, clear speech without music or echo gives a much closer copy than a noisy phone recording. Cross-lingual cloning goes a step further, letting the voice speak a language the person never recorded, which is what makes cloned dubbing possible.

Where voice cloning is used

mydubly text to audio voice vivian angry with style sample card
  • Dubbing: translated videos in the original speaker's own voice, rather than a stranger's.
  • Accessibility: people who are losing their voice to illness can bank recordings and keep speaking in a voice like their own.
  • Content production: corrections to narration without re-recording a whole session.
  • Games and animation: generating variations of lines with the actor's permission.

Getting a good clone from your own video

mydubly text to audio voice ryan clear with style sample card

Because a cloned voice is only as good as its reference, the original recording matters. A few habits make a noticeable difference:

  • Record each speaker on a decent microphone close to their mouth, rather than a camera microphone across the room.
  • Keep music low or absent under speech where you can, especially in the first minutes.
  • Avoid echoey rooms; soft furnishings help.
  • Give each speaker some stretches of clear speech without others talking over them.

None of these are special requirements for cloning: they're the same habits that give better transcripts and better videos generally. Choosing a microphone for speech covers equipment, and the YouTube video dubbing tool page covers preparing videos for dubbing as a creator.

How mydubly uses voice cloning

mydubly dubs videos with Chatterbox Multilingual, an open-source text-to-speech model from Resemble AI that clones voices from short reference audio. For each speaker in your video, up to 8, mydubly takes about 10 seconds of their clean speech from the original recording as the reference, then voices that speaker's translated lines in a copy of their voice. It doesn't use stock voices and doesn't keep voice models: the audio is deleted with the results within 30 minutes of delivery. Upload a video to AI dubbing; a dub costs 120 credits per minute in 19 voice languages. Chatterbox TTS covers the model.

Limitations of voice cloning

  • A copy is close, not identical. Listeners who know the person well may notice differences.
  • Short or noisy reference audio gives a weaker copy; music under the voice, echo and overlapping speakers all hurt.
  • Emotion and emphasis are harder to clone than timbre; a cloned voice can sound flatter than the original performance.
  • Some languages and accents come out more natural than others.
  • Only use voices you have the right to use; mydubly is built for dubbing your own videos or videos you have permission to translate.

Frequently asked questions

What is voice cloning?

Using AI to generate new speech in a voice that sounds like a specific person, based on a recording of them speaking.

How much audio does voice cloning need?

Modern zero-shot models can work from a few seconds of clean speech. mydubly uses about 10 seconds per speaker from the original video.

Is voice cloning legal?

It depends on the use and the country. Cloning your own voice, or someone's with permission, for dubbing or accessibility is common; impersonation and fraud are illegal in many places.

Can a cloned voice speak another language?

Yes. Cross-lingual models can make a voice speak languages the person never recorded, which is how cloned dubbing works.

Does mydubly keep my voice?

No. Audio and results are deleted within 30 minutes of delivery (for a dub, 30 minutes after you last use its editor), and mydubly doesn't keep voice models or use your audio for training.

Which model does mydubly use for voices?

Chatterbox Multilingual, an open-source text-to-speech model from Resemble AI.