Keeping a Translated Voice in Time With the Original Speaker

Why translated speech never fits the original slots

A speaker's timing is set by their language, their pace and their pauses. A translation inherits none of that. The same sentence can need noticeably more syllables in one language than another, a point explored in text expansion in translation. A synthetic voice also has its own natural pace, which may be faster or slower than the person on screen. And a single fluent sentence in the source can come back from translation with a different word order, so the emphasis lands somewhere else.

The upshot is that every translated clip arrives with a duration that is close to, but rarely equal to, the slot it should occupy. Some are shorter, some are longer, and a few are much longer. Syncing is the job of deciding where each clip goes and how much, if at all, to change its speed.

Three naive approaches and how each one fails

It helps to look at what does not work.

• Play the clips back to back. Every clip that runs long pushes all later clips later. After a few minutes the voice is describing a slide that disappeared long ago, and the error never recovers.
• Start every clip exactly at its source timestamp and ignore length. Sync is perfect at each start, but long clips overlap the next line or get cut off, and two voices talking at once is worse than being late.
• Time-stretch every clip to exactly fill its slot. Lines that are much longer than their slot get compressed hard and sound rushed or robotic, while short lines get dragged out and sound sleepy.

The working approach sits between these. Starts stay anchored to the original, so error cannot accumulate. Lengths are adjusted, but in a fixed order of preference that keeps speed changes as small as possible.

The tools a timing planner has to work with

A planner has only a few levers, each with a cost:

• Tempo change. Modern time-stretching changes speed without changing pitch. Small changes are effectively inaudible; larger ones start to sound hurried, and the threshold depends on the voice and the listener.
• Starting early. If the time just before a line is silent, the line can begin slightly before the speaker's mouth moves. Viewers tolerate a small lead far better than a large lag.
• Running late. A line can spill a little into the next line's time if the next pause will absorb it, so the next line still starts close to its speaker.
• Slowing down. A clip that is much shorter than its slot can be stretched a little so it ends closer to when the speaker does, which keeps the voice from finishing while the person is visibly still talking.

Anchors come from the speech recognizer's segment timestamps, the same ones used for subtitles; subtitle timestamp alignment describes how they are produced.

How mydubly places each translated line

mydubly's voice track is assembled on the server, one roughly 30-second chunk at a time, using the chunk's recognized segments as anchors. Before any voice is generated, short fragments are merged into whole sentences of up to about 220 characters, because the default voice engine, Chatterbox Multilingual, sounds more natural reading complete sentences. A fragment is judged by how long it takes to say rather than by its character count, so languages written compactly, such as Chinese or Japanese, are not misjudged.

Then each line is placed with this order of preference:

1. Anchor the line at the moment the original speaker started that segment.
2. If the clip is too long for its slot, first speed it up by up to about 1.08 times, a change treated as inaudible.
3. If that is not enough, start it up to 0.3 seconds early, provided the time before it is free.
4. Next, let it run up to 0.6 seconds into the following line's time, pushing that line slightly late.
5. Only then raise the tempo further, up to 1.15 times by default. If a whole chunk still cannot fit inside its window, the ceiling is raised in small steps, with a firm upper limit.
6. If the clip is shorter than the original speech, slow it slightly, never below 0.9 times, so it ends with the speaker.

The planner also looks ahead. Working backward from the end of the chunk, it calculates how late each line may finish so that every later line can still start within its own allowance. A comfortable line early in a chunk may therefore be kept a little tighter to leave room for a long one that follows. A tenth of a second of space is kept between consecutive lines so they never run together.

Anchor

The start time of the original spoken segment

Inaudible speed-up

Up to about 1.08 times

Early start

Up to 0.3 s before the speaker, if the preceding time is free

Late finish

Up to 0.6 s into the next line's time

Default tempo ceiling

1.15 times, raised in small steps only when a chunk cannot otherwise fit

Slowest stretch

0.9 times, for lines shorter than the original speech

Gap between lines

0.1 s

A worked example: three lines in one chunk

The numbers below follow the placement rules exactly, for a fictional 12-second chunk of a product tutorial.

The listener hears line A arrive a hair early, line B at its natural pace and line C slightly brisk. Nothing overlaps and the next chunk starts exactly on time.

Matching loudness and joining chunks without gaps

Timing is half of sync; the other half is making the track sound continuous. Each generated clip is first normalized: converted to mono at 24 kHz, trimmed of leading and trailing silence so its measured length reflects actual speech, and loudness-normalized to the same target so lines and chunks sit at a consistent level. Each placed clip gets a fade of a few milliseconds at both ends to prevent clicks, and the mixed chunk passes through a limiter so overlapping edges cannot clip.

Every chunk is then padded with silence or trimmed to exactly the length of its window. Because the windows were contiguous slices of the original audio, the chunks add up to precisely the original duration, and drift cannot build up from one chunk to the next.

Finally the chunks are joined into one AAC track that matches the video length. Joining separately encoded AAC files naively tends to leave tiny gaps, because each encoder run adds a short priming delay at its start. mydubly instead joins the uncompressed audio first and encodes the continuous track. To use several processor cores, it encodes frame-aligned sections in parallel, giving each a few frames of the preceding audio to warm up the encoder and then discarding them, so the joined stream decodes as if it had been encoded in one pass. The browser then swaps this track into the video, as described in browser video muxing.

Where elastic timing works well

• Narration, lectures and tutorials, where one person speaks at a steady pace with regular pauses.
• Language pairs where translations come out at broadly similar lengths, so most lines need little or no adjustment.
• Content with natural breathing room, since every pause is time the planner can borrow.
• Long videos, because anchoring each line to the original prevents error from accumulating over an hour.
• Viewers who need to follow along with the picture, such as screen recordings, where a voice that drifts minutes behind would be useless.

What timing fit cannot fix

• Lip movements. mydubly aligns timing only and never alters the picture, so close-ups will show mouths that do not match the translated words; the AI lip sync article explains what visual lip-sync models do instead.
• Very dense speech. When a fast speaker leaves no pauses and the translation is much longer, lines reach the tempo ceiling and sound noticeably brisk.
• A perfect soundtrack. The translated MP4 keeps the original music and effects under the new voice, but separating them from the original speech is not perfect, so faint traces of the original voice can remain in dense mixes; keeping background music when translating video covers the details.
• Multiple voices. One chosen voice reads every line, so timing is preserved but the sense of different speakers is not.
• Vocal reactions. Laughter or sighs from the original speakers are not voiced by the new voice; where the original had them, only the background continues.
• Small local lag. A line can land up to 0.6 seconds behind its speaker, which is rarely noticeable in narration but can show in fast back-and-forth dialogue.

Checking sync on your own dubbed video

1. Watch the first two minutes with sound on and note any line that starts clearly before or after the speaker.
2. Jump to the middle and the last two minutes. If those sound as well aligned as the start, the track is not drifting.
3. Listen for brisk passages. They mark places where the translation ran long; consider shortening those sentences in the source script if you plan to re-record.
4. Compare with the translated SRT file. Its cues use the original timing, so a voice line that trails its cue by a lot is worth a second look.
5. For content with frequent on-camera close-ups, decide whether translated subtitles would suit viewers better than a voice, using the guide to subtitles vs dubbing.

Make a timed dub of your own video

The quickest way to judge the result is to dub a short, representative clip with mydubly's AI dubbing and listen at the start, middle and end. A translated voice costs 50 credits per minute with a 2-minute minimum per file, so a 12-minute tutorial costs 600 credits, which is 60 cents. The step-by-step dubbing guide covers choosing a voice and reviewing the output.