Inside an MP4 File: Boxes, Sample Tables, Fast Start and Fragments

Boxes all the way down

MP4 is built on the ISO base media file format, which itself grew out of Apple's QuickTime format. Everything in the file is a box, also called an atom. Each box starts with a header giving its size in bytes and a four-character type code, followed by its contents. Some boxes hold data; others are containers holding more boxes. A program reads a box header, decides whether it cares about that type, and either descends into it or skips ahead by its size.

That design makes MP4 extensible: a reader that meets an unknown box type simply jumps over it. It also means the order of boxes in the file can vary, which turns out to matter a great deal for streaming. How MP4 compares with its QuickTime sibling as a format choice is covered in MP4 vs MOV.

The top-level boxes

ftyp

File type. Lists brands, such as isom, mp42 or M4A, that tell a reader which specifications the file follows

moov

Movie. The index and metadata for the whole presentation, containing one trak box per track

mdat

Media data. The compressed audio, video and subtitle samples, usually interleaved, with no structure of its own

free or skip

Padding that readers ignore, sometimes reserved so metadata can grow without rewriting the file

moof

Movie fragment. Found only in fragmented MP4, indexing the media data that follows it

mfra

Optional random-access index for fragmented files, usually at the end

In a typical file you see ftyp, then moov and mdat in one order or the other. The mdat box is almost the entire file; moov is small by comparison, though for a long recording its tables can reach several megabytes.

Inside moov: tracks and sample tables

The moov box starts with mvhd, the movie header, which gives an overall timescale and duration. Then comes one trak box for each track. Each trak contains:

• tkhd, the track header: a track ID, the display width and height, and a transformation matrix. Phones commonly record rotation here as a flag rather than rotating the pixels, which is why some tools show portrait video sideways.
• edts with elst, an optional edit list that maps the track's media timeline onto the presentation timeline, used for trims and start offsets.
• mdia, holding mdhd (the track's own timescale and language), hdlr (whether this is video, sound, subtitles or something else) and minf, which leads to the sample table.

The sample table, stbl, is the heart of the format. It describes every sample, meaning every video frame or audio packet, without storing any of them:

stsd

Sample description: the codec and its configuration, such as an avc1 entry with the H.264 settings a decoder needs

stts

Decode time to sample: how long each sample lasts, from which decode timestamps are computed

ctts

Composition offsets: the gap between decode and presentation time, needed when B-frames reorder frames

stss

Sync samples: which samples are keyframes. Absent when every sample is a keyframe, as in most audio

stsz

Sample sizes in bytes

stsc

Sample to chunk: how samples are grouped into chunks

stco or co64

Chunk offsets: where each chunk starts in the file. The 64-bit form handles files over 4 GB

To fetch frame 5,000 of the video, a player reads stts and ctts for its timing, stsc to learn which chunk it is in, stco for where that chunk starts, and stsz to add up the sizes of earlier samples in the chunk. It then reads exactly those bytes from mdat. The timing values these tables produce are explained in PTS and DTS.

Fast start: where moov sits matters

A player cannot do anything until it has read moov. If moov is at the start of the file, playback can begin as soon as the first few hundred kilobytes arrive. If it is at the end, the player must either download the whole file first or, on a web server that supports range requests, jump to the end, fetch moov, then come back to the media data.

Recorders and many encoders write moov last, because they do not know the sample sizes and offsets until the recording is finished. Moving it to the front afterwards is called fast start, or sometimes web optimization. It requires rewriting the file, because every chunk offset changes once moov is inserted before mdat. In ffmpeg this is the -movflags +faststart option, which performs a second pass at the end of the export. Many editors offer it as a checkbox labelled fast start, web optimized or progressive download.

For files that are only played locally or uploaded to a platform that re-encodes everything, fast start makes little difference. For files served directly from your own website, it lets playback start without waiting.

Fragmented MP4

A fragmented MP4 splits the index into pieces. The initial moov describes the tracks and codecs but contains no samples; it carries an mvex box announcing that fragments follow. The file then continues as pairs of moof and mdat, each moof indexing only the short stretch of media in the mdat after it, with a decode time from tfdt and per-sample details from trun.

• Adaptive streaming uses fragments: DASH and modern HLS deliver video as fragmented MP4 segments, and the CMAF specification standardizes this.
• Browsers' Media Source Extensions accept fragmented MP4, so web players can append segments as they arrive.
• Recording software can write fragments so that, if it is interrupted, everything up to the last completed fragment is still playable. Some recording tools offer fragmented MP4 as an option for exactly this reason.
• The cost is a slightly larger file and patchier support in some older editors and players, which is why fragmented recordings are often remuxed to ordinary MP4 afterwards.

Why an interrupted recording often will not open

A camera or screen recorder writing an ordinary MP4 appends compressed samples to mdat as it goes and keeps the sample tables in memory. Only when you press stop does it write moov. If power fails, the app crashes or the storage card is pulled first, the file contains gigabytes of valid media and no index. Its size looks right, but players and tools report an error, ffmpeg's being moov atom not found.

The media is usually still there, but without the tables nothing knows where one frame ends and the next begins, which codec settings to use, or how the audio interleaves with the video. Recovery tools try to rebuild moov by scanning mdat for frame boundaries, typically using a healthy reference file recorded by the same device with the same settings to supply the codec configuration. Results vary: sometimes the whole recording comes back, sometimes the audio is lost or out of step. Work on a copy, never the only original.

Common MP4 misconceptions and pitfalls

• The extension is not the structure. An .mp4, .m4v, .m4a and many .mov files share the same box layout; the ftyp brands and track types tell them apart.
• A large file size does not prove a recording is intact. An interrupted file can be full size and unreadable.
• Moving moov to the front does not change quality. Fast start only reorders boxes and rewrites offsets.
• Not every codec fits in MP4. A remux into MP4 fails if a track uses a codec the format or the tool does not support, which is one reason tools fall back to MKV. Our article on browser video muxing explains the options.
• Edit lists and rotation flags are honored unevenly. A file can look right in one player and trimmed differently or rotated in another.

How to look inside an MP4

You do not need to read hex to see a file's boxes. Useful tools include MP4Box from the GPAC project with its -info option, Bento4's mp4dump, and several browser-based box viewers that display the tree without uploading the file. ffprobe gives a higher-level view of the tracks, and running it with -v trace prints the boxes as ffmpeg parses them, which helps when diagnosing a damaged file. Looking at a healthy file from your own camera first makes it much easier to spot what is missing from a broken one.

mydubly and MP4 files

mydubly accepts MP4, MOV and M4V files alongside WebM and MKV, up to 2 hours each. Reading them is a demuxing job: the browser uses the file's index to find the audio, decodes the default audio track and leaves the picture on your device. A file whose moov is missing cannot be demuxed by mydubly or any other standard tool, so repair or re-record it first.

When you create a dubbed video, the server assembles the translated voice, mixed over the original music and effects, as AAC in an MP4-family M4A file. Your browser then builds a new file from your original picture, copied packet by packet without re-encoding, and that audio track, which replaces the original audio track. The result is MP4 when your video codec can go in MP4 and MKV otherwise. It contains no subtitle track; subtitles come as separate SRT and VTT files. The MP4 to text page covers transcripts from MP4 sources, and private video translation explains why the picture stays local.

Next step: open one file's box tree

Load a recent MP4 from your phone into a box viewer and find ftyp, moov and mdat, then note whether moov comes before or after mdat. If you serve that file from a website and moov is at the end, re-export with fast start; if you only need its speech translated or transcribed, it can go straight into the mydubly video translator as it is.