What happens to several speakers in an AI translation

Speech recognition hears a single mixed soundtrack. It transcribes whatever words it can make out, in order, with timestamps, but unless a separate speaker diarization step is added, nothing in the output records which person spoke which segment. Translation then works on that unlabeled text, and the voice model reads it. Without a diarization step, the result keeps the conversation's content and timing while flattening its cast. What is speaker diarization explains the technique tools use to tell speakers apart, and why it has its own error modes.
A voice per person: what viewers actually hear

In a mydubly dub, the host asks the question in a copy of the host's voice and the guest answers in a copy of the guest's. Each line lands where the original speaker's line was, so the rhythm of the conversation survives, and each copy keeps its owner's character: a deep voice stays deep, a bright one stays bright, and a line delivered loudly or quietly keeps that relative level. Viewers can follow turn-taking by ear, even in audio-led video podcasts, with voices off camera, or when two people share similar framing.
The copies are close to each person without being identical, especially across languages, and laughter and overlapping speech are not reproduced as they were. Telling people apart is automatic, so quick back-and-forth, very short interjections such as 'yeah' or 'mm', and two people who sound very alike are where a line can end up in the wrong voice. The single voice vs multi-voice dubbing article weighs those trade-offs in general terms.
Transcripts and subtitles without speaker labels

Unlabeled transcripts are fine for searching and quoting but awkward for reading a conversation. Subtitles are less affected, because they appear while the original audio plays and viewers can hear who is speaking. If you want labels, add them yourself: subtitle files are plain text, so a reviewer can prefix lines with names or initials, or use the common convention of starting a line with a dash when the speaker changes within a two-line cue.
Crosstalk, interruptions and overlapping speech

Overlap is the main accuracy problem in multi-speaker video. When two people talk at once, a recognizer typically follows the louder voice, drops the quieter one, or merges fragments of both into one sentence that neither person said. Short backchannel responses such as 'right' and 'mm-hm' are often dropped. Each of those problems carries into translation and then into the voice track, where a merged sentence is read as a single smooth line.
Original audio: the guest says 'The budget was cut in March, so we' as the host cuts in with 'Wait, March this year?' The transcript may come out as 'The budget was cut in March, so we wait, March this year?' Translated and voiced, it becomes one confusing sentence, read in a single person's voice. With subtitles over the original audio, a viewer at least hears the interruption and can work out what happened.
For new recordings, separate microphones and a moderator who discourages talking over each other make a big difference. For existing recordings, there is nothing to separate after the fact, so plan for review around the busiest exchanges.
When translated subtitles serve multi-speaker video better
Subtitles keep the original voices audible, so nothing depends on automatic speaker detection. They are the stronger choice for:
- Panels and roundtables with heavy crosstalk or frequent interruptions.
- Recordings where two participants sound very alike and quick exchanges are common.
- Documentary interviews, where a person's own voice and emotion are part of the story and viewers prefer to hear the original.
- Debates and Q&A sessions, where it matters who said what.
- Any content where a misattributed statement could cause harm.
Subtitles and transcripts are also the cheapest output, at 5 credits per minute in mydubly; the subtitle generator produces them without a voice track.
When a translated dub works, and how to help it
A dub with a voice per person works well for host-and-guest interviews, video podcasts, panels with clear turns, explainer videos where one person talks most of the time, and training videos with a presenter and occasional questions. A few habits help:
- Make sure each speaker has some clean, close-mic stretches with no one talking over them, because each copied voice is taken from a short sample of that person.
- Tell viewers at the start, in the description or a title card, that the translation uses AI copies of the participants' voices.
- Publish the translated subtitles alongside the dub, with speaker names added where turns are hard to follow.
- Keep the original-language version available, so viewers who want the real voices can find them.
Getting a good copied voice covers what makes a source recording copy well.
Sponsored · mydubly
AI video dubbing deal
Dub any video for just 12¢ a minute
Was 240 credits/min — now only 120. Cloned voices, timed to your picture, at half the list price.
A workaround for two similar voices: two runs, one edit
If two people in an interview sound very alike and automatic detection keeps mixing up their lines, you can force the split yourself in a video editor. It relies on the fact that each run copies the voices of whoever speaks in the file you upload.
- In your editor, export two versions of the interview at full length: one with the guest's turns silenced, so only the host speaks, and one with the host's turns silenced, so only the guest speaks.
- Translate both versions into the same language. Each run copies the voice of the one person left speaking in that file.
- Download the translated audio file from each run.
- Place the original video with both translated audio files on separate tracks underneath, aligned at the start, and cut between them at the pauses between turns.
- Listen at every switch. Both versions keep the original timing, so lines should land close to the original moments, but check that nothing was clipped where you silenced a turn. If one person speaks only briefly, their file may have too little clean speech to copy, and that run falls back to a stock voice matching the speaker.
- Export the result, and fix any wording differences in the subtitle file so it matches what viewers hear.
Suppose a journalist wants a Spanish version of a 30-minute interview between two siblings whose voices are hard to tell apart, and a first run puts several of their lines in the wrong voice. Two dubbed runs of the full-length versions cost 1,500 credits each, 3,000 credits or $3.00 in total. Silencing turns and cutting between the tracks at about 60 turn changes takes an editor perhaps a couple of hours, and puts each line in the right voice. For a panel of five, this approach stops being practical, and subtitles are the better route if detection struggles.
Where multi-speaker translation falls short
Even with care, AI translation of conversations has limits worth stating plainly. Overlapping speech loses words, and what survives overlaps in the dub by at most about two seconds. Automatic speaker detection can put a short interjection or a line from a similar-sounding person in the wrong voice. Grammatical gender is a quiet trap: in languages such as French, Spanish, Hebrew or Polish, a speaker's own sentences often mark their gender, and a translator working without speaker information may choose inconsistently, especially when a man and a woman are talking. Laughter and sighs, which carry a lot of meaning in conversation, are not reproduced by the voice track. And humor between speakers, the teasing and callbacks, is among the hardest content to translate well.
mydubly and multi-speaker video
In mydubly, each person is dubbed in a copy of their own voice: the file is analysed to find how many people speak (up to 8) and which lines each says, with a line split where the speaker changes mid-line, and each voice is taken automatically from a short, clean sample of that person of up to about 10 seconds with music and background noise removed. Transcripts do not label speakers. Someone with too little clean speech to copy, under about 3 seconds, is voiced with the copied voice of the person they sound most like, if they sound alike; otherwise a stock voice matching theirs (female or male) is used. The translated MP4 removes the original speech and keeps only the original music and effects underneath the new voices, and the voice samples are deleted with the rest of the job's files. What you do get from each run is everything the workarounds above need: the translated MP4, the translated audio file on its own, SRT and VTT subtitles you can label, and transcripts in both languages for review. For conversation-heavy formats, the podcast translation and conference talk translation pages describe typical setups.
Choosing your approach
Watch for overlap more than counting speakers. Clean turns, whether two people or a panel of up to 8: a dub with a voice per person is usually fine, and the two-run edit is there if two similar voices get mixed up. Lots of crosstalk: lead with translated subtitles. When you are ready to hear how the voices handle your conversation, try a short excerpt with AI dubbing before committing the full recording.
Sponsored · mydubly
240 free credits + 50% off
Try free, then buy minutes at half price
New accounts get 240 credits (enough for a 2-minute dub). When you top up, you keep these sale rates.
Hear your video in another language
Pick a target language; mydubly speaks each person's lines in a copy of their own voice, fits them to the original timing and rebuilds the video in your browser.
Frequently asked questions
Can AI give each speaker in my video a different translated voice?
Yes. mydubly does it automatically: it works out who speaks which line, up to 8 people per file, and dubs each person in a copy of their own voice. If two people sound very alike and their lines get mixed up, you can export two versions with only one person audible in each, translate both, and cut between the two audio tracks in an editor.
Why does my translated interview transcript not show who is speaking?
Speech recognition outputs words and timestamps, not identities. mydubly tells speakers apart to give each their own voice, but its transcripts and subtitles do not label who is speaking, so add names to the subtitle or transcript files during review if you need them.
What happens to the parts where people talk over each other?
The recognizer usually captures the louder voice and may drop or merge the other, so overlapping passages are where most errors appear. Review those moments first, and consider subtitles for content with frequent interruptions.
Does a dub work for a video podcast?
Yes, especially when hosts and guests take clear turns. Each person keeps their own copied voice, so listeners can tell them apart by ear even when they are off camera. For episodes full of fast banter and people talking over each other, translated subtitles over the original audio are easier to follow.
Whose voice reads the lines in a multi-speaker translation?
In mydubly, each person's lines are read in a copy of their own voice, taken from a short, clean sample of their speech in the file. Someone with too little clean speech is voiced with the copied voice of the person they sound most like, or a stock voice matching theirs if nobody sounds alike. Test a short excerpt with a busy exchange to hear how it sounds before translating the full recording.
Sponsored · mydubly
Transcripts & subtitles deal