DEV Community

Super Funicular
Super Funicular

Posted on

Your Phone Camera's Frame Rate Never Reaches the File - What Android Actually Writes Into an MP4 Timeline

#android #kotlin #video #programming

Anyone who has left a phone recording overnight has done the arithmetic at some point. Thirty frames a second, times sixty, times sixty again, times eight hours. Eight hundred and sixty-four thousand frames. From there you can estimate a file size, predict how long a seek will take, work out how many frames a motion-detection pass has to chew through, or convert a scrubber position into a time of day.

Every one of those calculations rests on a number that is not in the file.

An MP4 recorded by an Android phone does not store a frame rate. It stores something else, and the difference is invisible until the moment it isn't — usually when you are trying to work out exactly when something happened.

Does an MP4 recorded on Android store a frame rate?

No. The container stores a timescale and a list of per-sample durations. The playback rate is a consequence of those numbers, not an input to them.

That is not a quirk of any particular app. It is how the ISO base media file format has always worked, and Android's own documentation is unusually explicit that the frame rate you configure does not travel with the media.

The frame rate is a hint, and the documentation says where it stops

When you configure a video encoder on Android you set MediaFormat.KEY_FRAME_RATE. It is required for encoders. It is also, in the platform's own words, advisory:

"For video encoders this value corresponds to the intended frame rate (the rate at which the application intends to send frames to the encoder, as calculated by the buffer timestamps, and not from the actual real-time rate that the frames are sent to the encoder). Encoders use this hint for rate control, specifically for the initial frames, as encoders are expected to support variable frame rate (for rate control) based on the actual buffer timestamps of subsequent frames."

Two things in that paragraph deserve slow reading. The configured rate is described as intended, and it is explicitly calculated from the buffer timestamps rather than from wall-clock arrival. And encoders are "expected to support variable frame rate" driven by "the actual buffer timestamps of subsequent frames." The timestamps are upstream. The rate is downstream of them.

Then the same page closes the loop with one short sentence:

"This key is not used in the MediaCodec input/output formats, nor by MediaMuxer."

The muxer — the component that actually writes the MP4 — is never told the frame rate. Look at the format keys MediaMuxer.addTrack() documents and there is no frame-rate key among them. There is width, height, bit rate, and a set of colour keys. The number you configured stopped at the encoder.

CamcorderProfile uses the same careful vocabulary for the higher-level path. Its field is not called the frame rate; it is:

"The target video frame rate in frames per second."

Target. The sibling fields on that page say "target" too, so this is the page's standing word for a profile value rather than a special caveat about timing. But it is the right word, and it is not "actual."

What the muxer receives instead: one timestamp per frame

What MediaMuxer gets is a buffer and a MediaCodec.BufferInfo. The relevant field is documented in a single line:

"The presentation timestamp in microseconds for the buffer. This is derived from the presentation timestamp passed in with the corresponding input buffer."

So the timeline is assembled one frame at a time, from a value that originates upstream of the encoder. writeSampleData() imposes exactly one ordering rule, and it is weaker than developers tend to assume:

"The application needs to make sure that the samples are written into the right tracks. Also, it needs to make sure the samples for each track are written in chronological order (e.g. in the order they are provided by the encoder.)"

Chronological, per track. That is the whole contract. Worth noting what is not on that page: it never uses the word "monotonic," never says the timestamps must be evenly spaced, and never states what happens if they are not. The documented exceptions for writeSampleData are about invalid arguments and wrong muxer state — neither is tied to timestamp spacing. Nothing in the API is checking that your frames are 33,333 microseconds apart, because nothing in the format requires them to be.

What the container actually stores

On the file side, the relevant structures are the media header and the decoding-time-to-sample table. ISO/IEC 14496-12 defines the timescale in the media header box as:

"timescale is an integer that specifies the time-scale for this media; this is the number of time units that pass in one second. For example, a time coordinate system that measures time in sixtieths of a second has a time scale of 60."

And the stts box, which is mandatory and appears exactly once per sample table:

"This box contains a compact version of a table that allows indexing from decoding time to sample number... Each entry in the table gives the number of consecutive samples with the same time delta, and the delta of those samples. By adding the deltas a complete time-to-sample map may be built."

Each entry carries a sample_count and a sample_delta. The semantics are plain: sample_count "counts the number of consecutive samples that have the given duration," and sample_delta "gives the delta of these samples in the time-scale of the media."

A perfectly regular recording compresses beautifully here: one entry, sample_count = 864,000, sample_delta = a constant. That is the run-length compression the spec means by "a compact version of a table." A recording whose spacing wandered produces more entries. Same box, same rules, no error condition — the format treats even spacing as a lucky case, not a requirement.

And the duration of the track is defined by summation, not by division:

"the sum of all deltas gives the length of the media in the track"

That sentence is the whole argument in miniature. The length of your overnight recording is the sum of how long each frame actually claimed to last. It is not frame count divided by frame rate, because the file never stored a frame rate to divide by.

Where the spacing is documented to move

There is a legitimate, documented mechanism by which capture spacing changes during a recording, and it sits in Camera2. CONTROL_AE_TARGET_FPS_RANGE is typed as a Range<Integer> and described as:

"Range over which the auto-exposure routine can adjust the capture frame rate to maintain good exposure."

The auto-exposure routine adjusts the capture frame rate. That is the documented purpose of the key, not a side effect. And variable ranges are not exotic — CONTROL_AE_AVAILABLE_TARGET_FPS_RANGES guarantees that for devices at the LIMITED level or above, other than those advertising an NIR colour filter arrangement, the list "will always include (min, max) and (max, max) where min <= 15."

So a device is required to offer you at least one range whose floor is 15 or below, alongside a fixed one. If your capture request carries the variable range, you have asked the auto-exposure routine for permission to move the spacing, and the container will faithfully record that it did.

Separately, the same field documents a case where the requested maximum simply cannot be met:

"Note that the actual achievable max framerate also depends on the minimum frame duration of the output streams. The max frame rate will be min(aeTargetFpsRange.maxFps, 1 / max(individual stream min durations)). For example, if the application sets this key to {60, 60}, but the maximum minFrameDuration among all configured streams is 33ms, the maximum framerate won't be 60fps, but will be 30fps."

That is a fixed request of {60, 60} being delivered at 30. Requested and achieved are documented as different quantities.

What the documentation does not say, stated plainly. I went looking for a first-party sentence asserting that frame rate drops in low light, and did not find one. Searches will confidently hand you a specific floor figure, attributed to Android's Low Light Boost AE page. That page contains no FPS figures, no lux figures, and no exposure times at all — no numbers of any kind. The word "light" does not appear in the CONTROL_AE_TARGET_FPS_RANGE field. What is documented is the mechanism ("adjust the capture frame rate to maintain good exposure") and the guaranteed availability of a low-floor range. The size of the effect on your specific handset in your specific hallway is not documented by anyone, and the honest thing to do is measure it rather than quote a number you read.

The last frame is a special case, and it is easy to miss

Buried in writeSampleData() is a rule with no analogue anywhere else in the file:

"For MPEG4 media format, the duration of the last sample in a track can be set by passing an additional empty buffer(bufferInfo.size = 0) with MediaCodec.BUFFER_FLAG_END_OF_STREAM flag and a suitable presentation timestamp set in bufferInfo parameter as the last sample of that track. This last sample's presentation timestamp shall be a sum of the presentation timestamp and the duration preferred for the original last sample. If no explicit END_OF_STREAM sample was passed, then the duration of the last sample would be the same as that of the sample before that."

Every frame's duration is derivable from the gap to the next one. The final frame has no next one, so the muxer guesses — it copies the previous frame's duration. On a clip that ends cleanly, the error is one frame and nobody notices. The reason to know the rule is that it tells you where the file's declared duration comes from at the boundary: it is a convention, not a measurement.

Why any of this matters outside a codec discussion

Three practical consequences, each of which touches something you may already have built.

Seeking is priced in deltas, not in frames. If you are serving recordings from the phone over HTTP, the player converts a scrubber position into a time, the time into a sample via the stts map, and the sample into a byte offset. I wrote about the transport half of that — why the browser's range requests and the position of the moov atom decide whether playback starts in one second or after the whole file downloads — in The Range Header Is the Whole Feature. The timeline described here is the map that request is indexing into.

A truncated recording has no map at all. The sample tables live in moov, and a muxer cannot write them until it knows every delta. That is why a recording interrupted by a crash or a power cut will not open, and why the fixes look the way they do — covered in Why an Interrupted Android Recording Won't Play.

A position in a file is not a time of day. This is the one that bites during an actual incident. The timeline inside the file is an elapsed-time axis with a zero origin — the spec is explicit that "The DT axis has a zero origin." It carries no information about when recording started, and nothing reconciles it with the wall clock afterwards. Converting "01:47:22 on the scrubber" into "the time someone was at the door" is a separate problem with its own failure modes, which I went through in Recording Is the Easy Half.

What to check on your own footage

None of this requires taking my word for it, and the measurement is more useful than the theory.

  1. Read the file's own numbers. ffprobe will report both an average frame rate and a real base frame rate for a stream; when a recording's spacing wandered, those two diverge. On-device, MediaMetadataRetriever exposes METADATA_KEY_DURATION and METADATA_KEY_VIDEO_FRAME_COUNT as separate values — divide one by the other and compare the result to what you configured.
  2. Test the boundary, not the middle. Record a two-minute clip in bright light and a two-minute clip in the dimmest conditions the camera will be working in, with the same settings. Compare duration ÷ frame count for each. That single comparison tells you more about your handset than any specification will.
  3. Decide whether you actually want fixed spacing. If a constant rate matters more to you than exposure, request a fixed range — the (max, max) entry is guaranteed to be on the list. You are trading image quality in dim conditions for a predictable timeline. That is a real trade, and it should be a decision rather than a default.
  4. Do the arithmetic on measured values. Storage estimates, motion-pass costs and retention windows built on a nominal frame rate inherit an error you never measured. Substitute your own numbers from step 1.

The short version

An Android phone does not record at a frame rate. It produces frames, stamps each one with a presentation time, and hands them to a muxer that writes down the gaps. The frame rate is a hint that helps the encoder budget bits and then stops travelling — documented, in so many words, as not reaching the muxer at all. The file that results is an honest record of when frames arrived, which is a more useful thing to have than a nominal rate, provided you know that is what you are holding.

If you are building this, measure the spacing before you build anything on top of it. If you are running a phone as a camera and only ever watch the footage, this is the explanation for why the duration sometimes looks slightly wrong, and it is not your card failing.

Background Camera RemoteStream records with the screen off, keeps footage on the device rather than in anyone's cloud, serves a live view from a built-in web server to any browser on your own network, and can stream to YouTube Live. No account, no subscription. Built in Kotlin on Camera2 with an embedded Ktor server. Free on Google Play — more at superfunicular.com.

Sources, all first-party:

Top comments (0)