AI voice & dubbing

How One TTS Model Speaks Many Languages

Multilingual text to speech is a single synthesis model that can speak many languages, usually in the same voice, by learning shared representations of speech across languages and being told which language each input is in. It makes it practical to give one video the same voice in Japanese, German and Arabic. It also brings its own problems: accents leaking from one language into another, numbers and dates read the wrong way, and quality that varies from language to language.

7 min read · Updated

From one voice per language to one model for many

mydubly text to audio voice aiden energetic with style sample card

Traditional TTS products offered voices per language. Each was recorded by a native speaker, so switching from English to French meant switching to a different person. Neural models trained on many languages at once changed that. Because they share parameters across languages, patterns learned from data-rich languages help weaker ones, and a single voice identity can be rendered in languages its reference speaker never recorded.

How text enters the model matters. Some systems convert text to phonemes first, which needs a separate pronunciation module per language. Others read characters or subword tokens directly and learn pronunciation from data, which scales to more languages but relies on the model inferring how each spelling sounds. The background on neural synthesis is in what is text to speech.

Language tags: telling the model what it is reading

mydubly text to audio voice aiden bright with style sample card

A multilingual model needs to know which language it is speaking, because the same letters sound different in different languages. "Chat" is a conversation in English and a cat in French; "gift" is a present in English and poison in German. Most multilingual models therefore take a language identifier along with the text. Chatterbox Multilingual, the open-source model from Resemble AI, works this way: the caller specifies the target language for each request.

When the tag is wrong or missing, the model may read the text with the wrong pronunciation rules or a foreign accent. Mixed-language sentences are a related weak spot. An input usually carries one tag, so an English product name inside a German sentence tends to get German pronunciation. Whether that sounds right depends on how German speakers actually say that name.

Accent leakage between languages

mydubly text to audio voice ryan angry with style sample card

In training data, a speaker's voice and accent arrive bundled together. A model that learned a voice mostly from English recordings has learned English vowels, rhythm and intonation along with the timbre, and some of that can carry over when the voice speaks Spanish or Japanese. The same happens with voices conditioned on a reference clip: the reference's language colors the output.

Typical symptoms are an English-sounding r in languages that roll or tap it, vowels reduced where the target language keeps them full, stress on the wrong syllable and question contours borrowed from the source language. Mitigations include language-appropriate reference recordings, stronger language conditioning in training, and, for anyone choosing a voice, listening tests by native speakers in each target language.

Numbers, dates, units and abbreviations

mydubly text to audio voice ryan bedtime with style sample card

Before synthesis, text has to be normalized: digits, symbols and abbreviations expanded into words. That step is language-specific, and it is where many multilingual errors start. German and many other European languages use a dot to group thousands and a comma for decimals, the reverse of English. Dates written with slashes mean different things in the United States and most of Europe. Hindi writing often groups large numbers in lakhs and crores. In Russian, Polish and Czech, number words change form depending on the noun that follows.

One string, different readings

Suppose a German script reads "Das Update erscheint am 03/04 und kostet 1.250 Euro." A German-aware normalizer should say twelve hundred fifty euros, because the dot groups thousands, but an English-style reading would treat it as a decimal. The date could be the third of April or the fourth of March depending on the reader. Writing the date in words, "am 3. April", and keeping the amount in the target language's own conventions removes the guesswork.

Writing systems that hide pronunciation

mydubly text to audio voice ryan clear with style sample card

Some scripts leave pronunciation underspecified, so the model has to infer it from context:

  • Chinese characters can have more than one reading, and each syllable carries a tone that changes its meaning.
  • Japanese kanji often have several readings, and the language uses pitch accent to distinguish some words.
  • Arabic and Hebrew are usually written without short vowels, so the model must supply them from context.
  • Russian does not mark word stress, and stress placement can change the meaning of a word.
  • Hindi in Devanagari drops some inherent vowels in speech that the spelling does not show.

These languages are not worse for synthesis, but they reward a native listener's check more than languages whose spelling maps closely to sound.

Where multilingual TTS works well

mydubly text to audio voice ryan horror with style sample card
  • Brand consistency: every language version of a course or channel can carry the same voice.
  • Coverage: less-resourced languages benefit from what the model learned elsewhere.
  • Simplicity: one model and one review process instead of a separate vendor per language.
  • Speed: adding a language does not require casting or booking a studio.

Limitations of multilingual TTS

  • Quality is uneven across languages, generally tracking how much training data each had.
  • Code-switching, where a speaker moves between languages mid-sentence, is poorly served by one tag per input.
  • Names from a third language, such as a Polish surname in a Japanese sentence, are guessed.
  • Tones and pitch accent may be slightly off in ways native listeners notice even when meaning is clear.
  • Regional variants are covered selectively; a model may offer one accent per language or a small set.
  • Accent leakage can be reduced but rarely vanishes completely.

How mydubly applies the copied voice to the target language

In mydubly, each person in the dub is spoken in a copy of their own voice, taken from a short, clean sample of their speech in your file, in any of 19 languages: every supported language except Czech and Vietnamese, which the voice model does not cover and which are available as subtitles and translated transcripts. You pick a target language; the spoken language of your video is detected automatically, and there is no voice to select. The voice character is copied while pronunciation is natural for the target language, so the original accent stays with the original language. The default voice engine is Chatterbox Multilingual, which generates the translated sentences in the target language, with short fragments merged into whole sentences of up to about 220 characters first. Each person keeps the same copied voice throughout the video, and someone with too little clean speech to copy is voiced with the copied voice of the person they sound most like, or a stock voice matching theirs if nobody sounds alike.

Two languages offer regional variants: Spanish has Latin America and Spain, and Portuguese has Brazil and Portugal. The variant guides the written translation's vocabulary, grammar and spelling, and the voice keeps the model's standard pronunciation for the language, with no regional accent setting; translating video for regional audiences discusses what that means for a specific market. Chinese output is Simplified Chinese, Arabic output is Modern Standard Arabic and Norwegian output is Bokmål. Language pages such as Spanish and Japanese cover each language in more detail.

Testing a voice in a new language

  1. Pick a two-minute passage that contains numbers, a date, a name and a question.
  2. Dub it into the target language with AI dubbing; a two-minute file costs 100 credits, or 10 cents.
  3. Ask a native speaker to listen for accent, misread numbers and stress on the wrong syllable.
  4. Compare any problem with the translated transcript. If the number is wrong in the text, it is a translation issue, not a voice issue.
  5. If the voice sounds rough, try a passage where the speaker is recorded more cleanly, because the sample shapes the copy in every language.
  6. Where numbers or dates misread, say them more explicitly in the original recording, for example "the third of April" rather than a bare date.

Where to go next

Once the copied voice passes in each target language, record future videos the same way so every language version keeps one consistent presenter. Start with mydubly's AI dubbing, and if your videos switch between languages, read translating mixed-language videos before you upload.

Frequently asked questions

Does a multilingual TTS voice sound the same in every language?

The timbre usually stays recognizable, but rhythm, vowels and intonation adapt to each language, and some accent from the voice's training data can carry over. Native listeners notice the differences more than anyone else, so test each language separately.

What is accent leakage in text to speech?

It is when features of one language, usually the one a voice was learned from, show up in another: an English r in Spanish, or English stress patterns in Italian. It comes from voice and accent being learned together, and can be reduced but rarely removed entirely.

How do I stop a TTS voice from misreading numbers and dates?

Write them in the target language's own conventions, or spell them out in words when the form is ambiguous. In a dubbing workflow where you do not edit the script, say numbers and dates clearly in the original recording and check them in the translated transcript.

Can multilingual TTS handle two languages in one sentence?

Usually not well. Most models take one language tag per input, so a foreign word inside a sentence gets the main language's pronunciation. Videos that switch languages also complicate recognition, since the spoken language is detected from the audio.

What changes when I pick Spain instead of Latin America for Spanish?

In mydubly the choice guides the written translation toward peninsular vocabulary, grammar and forms of address, and the dub speaks that wording. The voice keeps the copied speaker's character and the model's standard Spanish pronunciation either way. A reviewer from your market can check the translated transcript for any regional terms that slipped through.