Translating Song Lyrics in Videos: Subtitles, Singable Versions and Rights

Why speech recognition struggles with singing

Speech recognition models learn from speech, and singing breaks many of the patterns they rely on. Vowels are stretched over several notes, so one word can last two seconds. Melody overrides the natural stress of words. Consonants get softened or dropped to keep a line flowing. Instruments, backing vocals and reverb sit right on top of the voice, often louder than any background noise in an ordinary recording.

The results are predictable: misheard words, whole lines skipped, choruses transcribed once instead of three times, and sometimes the opposite, text that was never sung. During instrumental passages, Whisper-style models can produce invented or repeated lines, a failure mode explained in Whisper hallucinations. Songs also have fewer of the quiet gaps that long-audio processing uses to split recordings cleanly, so a cut can fall mid-phrase.

Treat any automatic transcript of a song as a rough guide to where the song is, not as the lyrics. If you need the words, find the published lyrics or ask the performer.

Lyrics usually have their own copyright

For commercial songs, the lyrics are normally protected by copyright separately from the recording, and often owned or administered by a music publisher. Translating lyrics and publishing the translation, including as subtitles on a public video, can amount to making a derivative work, which generally needs the rights holder's permission. Lyrics sites and streaming services display lyrics under licenses for that reason.

Traditional folk songs and very old songs may be in the public domain, but a particular arrangement or an existing translation can still be protected. Your own songs are yours to translate. Platforms may also detect the music itself and restrict a video regardless of what you do with the subtitles. Copyright rules differ between countries and exceptions such as fair use or fair dealing are uncertain, so this isn't legal advice: check with the rights holder or a qualified professional before publishing translated lyrics. The broader question of translating content you don't own is covered in translating someone else's video legally.

Three ways to translate a lyric

Translators distinguish between very different jobs that all get called "translating a song":

Literal gloss

Line-by-line meaning, no rhyme or rhythm; good for learners and for understanding a song in a documentary

Meaning translation

Natural, readable lines that keep imagery and tone; the usual choice for subtitles

Singable translation

Fits the melody, syllable count, stress and often rhyme; needed only if someone will sing it

Singable translation is a craft closer to songwriting. The translation scholar Peter Low described it as balancing singability, sense, naturalness, rhythm and rhyme, accepting losses in some to keep others. A machine translation is, at best, a literal gloss or a rough meaning translation, and it will almost never fit the music.

For subtitles, a meaning translation is usually right. Viewers hear the original singing and read what it means. Trying to rhyme in a subtitle often forces awkward wording that is slower to read than plain prose.

Should the song be subtitled at all?

Not every song needs translating. Ask what the song is doing in the video:

• It tells the story. In a musical, a character's song moves the plot or reveals feeling. Subtitle it.
• It is the subject. A cover, a music lesson, a performance review or a documentary about a song. Subtitle it, often with a closer, more literal translation so viewers can follow the analysis.
• It sets the mood. Background music in a vlog, a montage or an advert. Usually leave it unsubtitled, or use a short tag such as a music note with "upbeat pop song" for viewers who rely on captions.
• It is already in the viewers' language, or famous enough that the title says it all. A brief note may be enough.

For same-language captions aimed at deaf and hard-of-hearing viewers, the expectation is different: describe the music, and transcribe the lyrics when they matter to the content.

Formatting sung lines in subtitles

A common captioning convention marks sung lines with a music note symbol at the start and end, so viewers know the text is being sung rather than spoken. Some subtitlers use italics instead or as well, where the player supports them. Pick one style and use it consistently.

Time each subtitle to a musical phrase rather than to a fixed length. One phrase per cue is easier to follow than cues that split a line in the middle. Repeat the chorus subtitle each time it is sung, since viewers who just joined need it too. When a lyric repeats a word many times, the subtitle can show it once or twice without losing anything.

Automatic subtitle cues follow recognized speech segments, which may not line up with sung phrases at all. Expect to re-time and rewrite song cues by hand; how to edit an SRT file covers the mechanics.

Why a dubbed voice shouldn't replace songs

Speech synthesis speaks; it doesn't sing. If a dubbing pipeline treats sung lyrics as speech, the result is a narrator reading the translated lyrics in a flat voice, at roughly the times the song played. That is rarely what anyone wants.

There's a second problem. Even when a dub keeps the original music by separating the voice from it, as mydubly does, sung vocals count as voice: the song keeps its instruments but loses its singing. For a video where the song matters, the better approach is to dub only the speech, keep the original song audio in an editor, and subtitle the song instead. How the original music is kept under the translated speech, and where that has limits, is explained in background music in translated videos; this article only covers the sung parts.

Mixed-language videos raise their own detection issues, which translating mixed-language videos explains.

Steps for subtitling a song inside a translated video

1. Find the songs. Scrub the video and note the start and end time of every sung passage.
2. Decide for each one: subtitle, tag only, or leave alone, using the questions above.
3. Check the rights for any song you will subtitle that isn't yours or clearly in the public domain.
4. Get the real lyrics from the performer, the published source or a fluent listener. Don't build on an automatic transcript of singing.
5. Write a meaning translation, one cue per phrase, marked with your chosen convention.
6. Replace or insert those cues in the translated subtitle file, and remove any garbled automatic cues inside the song.
7. If you also make a dub, keep the original audio over the song sections in your editor.
8. Watch the song sections with subtitles on before publishing.

Limits of automatic tools with music

• Recognition of sung lyrics is unreliable, and instrumental passages can trigger invented text.
• Machine translation produces glosses, not lyrics that rhyme or fit a melody. Singable versions need a human lyricist.
• Separating a voice from its accompaniment is imperfect even with dedicated tools, so swapping only the vocals of a song isn't realistic for most projects.
• No automatic tool can clear the rights to lyrics for you.
• Karaoke-style word-by-word timing needs specialized subtitle tools.

How mydubly handles videos with songs

mydubly treats everything in the audio as speech. Whisper transcribes it, so sung passages may come out misheard, partial or skipped, and the spoken language is detected automatically, so a song in another language may be handled inconsistently. Subtitles come as SRT and VTT, one recognized segment per cue on a single line, and mydubly doesn't import an existing lyrics or subtitle file, so song cues are edited after download.

For dubbing, AI vocal separation removes the original speech and keeps the music and effects under the translated voice, but sung vocals are removed along with the speech, so songs keep their instruments and lose their singing. Keep the original audio for song sections by cutting from the downloaded translated audio to the original audio for those sections in an editor. Subtitling the 14-minute vlog in the example costs 14 credits (1.4¢); the subtitle generator page shows the formats, and the AI dubbing page explains what the translated audio contains.

Next step

List the songs in your video and make the subtitle, tag or skip decision for each before you translate anything else. That decision, plus a rights check, removes most of the work. If you are weighing subtitles against a dub for the rest of the video, subtitles vs dubbing covers the trade-offs.