Replacing a Video's Audio Track in the Browser Without Touching the Picture

Codecs, containers and what muxing actually does

Video files are easy to misunderstand because two separate things share the same name. The file extension names the container; the content inside is compressed by codecs. Muxing only deals with the first.

Codec

The compression method for one stream, such as H.264, HEVC, VP9 or AV1 for video and AAC or Opus for audio

Container

The file format that holds one or more streams plus timing and metadata, such as MP4, MOV, MKV or WebM

Track or stream

One sequence of media inside a container, for example the picture or one language of audio

Packet

One compressed unit of a stream, such as a video frame or a short block of audio, with a timestamp

Demuxing

Reading a container and pulling out each stream's packets

Muxing

Writing packets from one or more streams into a container, interleaved by time

Remuxing

Demuxing and muxing again, usually into a different container or with tracks added or removed

Transcoding

Decoding a stream to raw frames or samples and encoding it again, which costs time and quality

Replacing an audio track is a remux: the video packets are read from the original file and written to a new one next to packets from a different audio stream. In ffmpeg terms this is a stream copy, mapping the video from one input and the audio from another while copying both codecs unchanged.

Why swapping audio should never re-encode the picture

Re-encoding the video to change its audio is like retyping a book to replace its cover. It works, but it is slow and introduces errors.

• Quality. Every lossy encode discards some detail. Copying packets keeps the picture bit-for-bit identical to the original.
• Time. Encoding decodes and compresses every single frame, while copying packets only moves bytes. On a phone or an older laptop, inside a WebAssembly encoder, the difference is large.
• Battery and heat. Video encoding keeps the processor busy for the whole run; a remux is dominated by storage reads and writes.
• No decisions to get wrong. Re-encoding forces choices about bitrate, profile and frame rate. A copy inherits the original's choices, including variable frame rate from phones and screen recorders, because each packet keeps its own timestamp.

The only stream that has to be encoded is the new audio, and in a dubbing pipeline that has already happened before the browser receives it.

What a browser needs in order to mux a file

A few web platform features make this practical; the article on browser-based video processing covers them in more depth.

• The File API. A chosen file is a handle, not a copy. Code can read any byte range of it on demand, so a muxer can walk through a multi-gigabyte video slice by slice.
• A demuxer and muxer in JavaScript or WebAssembly. Libraries can parse MP4, MOV, WebM and MKV structures and write new containers; WebAssembly builds of ffmpeg can do the same with a different memory model.
• Codec metadata. Each video stream carries a small decoder configuration, for H.264 the parameter sets describing the stream, which must be written into the new container alongside the packets.
• Somewhere to put the output. Small results can live in memory. Large ones need the origin private file system, a sandboxed storage area where a page can stream a file to disk through a writable stream.

How mydubly swaps the audio track in the browser

mydubly never uploads the video. The browser extracts and uploads the audio, the server returns text and a single finished AAC audio track, with the translated voice mixed over the original background, and the final video is assembled on the device using the open-source mediabunny library. As implemented, the merge works like this:

1. Open both inputs as on-demand readers: the original video file and the translated AAC track the server joined to match the video's length.
2. Take the primary video track from the original and the audio track from the translated file. The original audio is simply not copied.
3. Choose the container. If the MP4 muxer accepts the video codec, the output is MP4; otherwise it is MKV. The audio codec is checked against the chosen container as well.
4. Write the video's decoder configuration with its first packet, and carry over the rotation flag so portrait phone videos stay upright.
5. Feed packets from both tracks interleaved by timestamp, always taking whichever is earlier next. The muxer then never has to hold one track in memory while waiting for the other.
6. Finalize the container and hand the result to you as a download named after the original file.

Progress during this step follows the video packets' timestamps, which is why the final percentage moves steadily rather than jumping. Because the new track was built to exactly the video's duration, the two streams start together and end together. How that track is timed is covered in syncing translated audio with video.

MP4 or MKV: what happens when a codec doesn't fit

MP4 is the safest container for sharing and playback, so it is the first choice. Each container, though, only defines how to store certain codecs, and a muxer refuses streams it cannot represent. mydubly asks the MP4 muxer whether it can hold the video codec before writing anything; if the answer is no, the output is written as MKV, the Matroska container, which can carry almost any codec.

In practice, the library's MP4 writer accepts every video codec it can read today, including H.264, HEVC, VP8, VP9, AV1 and ProRes, so MP4 is the normal result even for WebM inputs. MKV is the safety net. Two caveats are worth knowing:

• A codec being allowed in MP4 is not the same as every player supporting it. A VP8 or ProRes stream inside MP4 is valid for the muxer but may not play in every app; desktop players with broad codec support generally cope.
• MKV files download normally but may not play inside some browsers or on some phones. A desktop media player will usually open them, and a desktop tool can remux them to MP4 later without re-encoding.

If the library cannot read the input video at all, a WebAssembly ffmpeg fallback attempts the same stream copy into MP4. That path holds everything in memory, so it is only tried for smaller files.

Where the merged file is written

The output is as large as the video, so where it lives matters. mydubly picks one of three outcomes based on what the device can handle:

• Disk. Where the browser supports writable streams in the origin private file system and storage has room for the output plus a safety margin, the merged file streams to disk in 8 MB pieces. Memory use stays flat however large the video is.
• Memory. If disk streaming is unavailable or fails, for example because of a storage quota in private browsing, a smaller output can be assembled in memory instead. The size allowed depends on the device and is stricter on phones, which close heavy tabs early.
• Audio only. If neither fits, mydubly explains why and offers the translated audio file, which you can combine with the video in any editor.

Only one merged file is kept in that private storage at a time; the next merge replaces it. Memory limits and the techniques used to stay within them are the subject of processing large video files in the browser.

Advantages of muxing on the user's device

• The video never leaves the device. Only the audio is uploaded, so footage, faces, slides and screens stay local; on-device video privacy explains what that does and does not protect.
• The picture is untouched. Resolution, bitrate, color and frame timing are exactly as recorded.
• Nothing large travels over the network. A multi-gigabyte upload and download is replaced by a few megabytes of compressed audio up and one audio track down.
• Speed depends on storage, not encoding. Most of the work is reading the original and writing the result.

Limitations and edge cases of in-browser muxing

• Browser differences. Disk streaming relies on newer storage features that vary between browsers and versions; desktop Chrome and Edge are the most dependable for very large outputs.
• Phones have tighter limits. A long, high-resolution video may exceed what a phone browser allows, in which case you get the audio file rather than a merged video.
• Only two tracks survive. The output holds the original picture and the translated audio. Extra audio tracks and embedded subtitle tracks in the source are not carried over, and subtitles are delivered separately as SRT and VTT files.
• The original audio track is gone. The new track keeps the original music and effects under the translated voice, but it is the only audio in the file, so the original voices cannot be switched back on.
• The MP4 index is written at the end. Streaming to disk means the file's index is finalized last. Local playback and uploads to video platforms are unaffected, but if you self-host the file for progressive streaming, a fast-start pass in a desktop tool moves the index to the front.
• Unreadable codecs. Rare or legacy codecs the library cannot parse fall back to the smaller in-memory path, and very large files in such formats end with an audio-only download.

Translate a video without uploading it

To see browser muxing in action, run a short clip through private video translation and watch the final step: the merge runs locally after the voice track arrives. The same process powers the main video translator, and the guide to translating video on a phone covers what to expect on mobile devices.