DEV Community

Super Funicular
Super Funicular

Posted on

The National Archives Will Take Phone Video as a Permanent Record. Its List Names the Codec, Not the File Extension

Short answer: The U.S. National Archives publishes the file formats it accepts when a federal agency hands over records meant to be kept permanently. For digital video, the acceptable list includes the MPEG-4 wrapper with H.264 inside it, a combination many phone cameras can record. The list is written as wrapper plus codec, though, so a file ending in .mp4 is half an answer. HEVC, which many phones offer as a space-saving setting, does not appear in the digital video table at all. Beyond format, the same bulletin requires files that are valid against their specifications, no encryption standing between the archive and the content, and a retained copy until custody is confirmed. All three translate into checks anyone can run on a folder of recordings.

Nobody is going to accession your driveway footage. Still, the people whose job is keeping video readable for decades have written down, in plain tables, what they will take, and it is more specific than most advice about saving recordings. I read NARA Bulletin 2014-04, the appendix of tables it points to, and the Records Express post that announced the most recent change to the video rows. What follows is a close reading by someone who builds a camera app, aimed at developers and anyone else holding a large pile of phone video. I am not an archivist, and this is federal agency guidance, which governs agencies and nobody else.

What the bulletin is for

The bulletin defines the property it is screening for, and the definition is narrower than a general idea of a good format:

"Sustainability, as it relates to electronic file formats, is the suitability of a format to preserve encoded information over time."

Note what is absent. Picture quality, file size and popularity of a particular app are not in the sentence. The question is whether the bits can still be turned back into the picture by someone else, later, with software that exists then.

The appendix splits moving images into two groups. Digital cinema covers digitized film. Digital video is described as "including both video digitized from analogue sources and born digital video", and born digital video is what a phone produces.

The video table, read as a developer would

The bulletin explains why the table has more than one column, and this is the part I would put in front of anyone who has ever told someone to just save it as MP4:

"Many file formats, especially those used with digital audio and video, are composed of multiple parts, including a wrapper, which is the file format, and an embedded encoding stream or codec."

Then it attaches a requirement to both parts:

"Agencies must submit electronic records in files that are valid both according to the wrapper and any specified codec standards."

So the digital video table names a wrapper and a codec on each row. As fetched for this piece, its acceptable formats include uncompressed AVI and uncompressed QuickTime, Windows Media Video with VC-1, MPEG-2 video, MXF carrying losslessly compressed J2K, and the row that matters to phone owners: MPEG 4, with H.264 in the codec column.

That row is the good news in this reading. H.264 inside an MPEG-4 file is a common output of phone cameras and of most video tools, and it sits on the acceptable list for records the federal government intends to keep permanently. A phone owner may already be holding files in a format the archive would take. The way to know is to check a file, which the commands further down do.

What is not on the table is just as useful. I searched the fetched page for HEVC, H.265, VP9 and WebM and found none of them anywhere on the page. Many phones offer HEVC as a high-efficiency option, since it generally produces smaller files at similar quality. If that switch is on, the recordings are in a codec this table does not list.

Here is the honest limit of that observation. The table does not give a reason for any individual absence. The bulletin does say what NARA weighs:

"Criteria such as the availability of technical specifications and licensing restrictions will be used to determine whether a format is sustainable and appropriate for use with permanent electronic records."

I can tell you HEVC is missing from the list. I cannot tell you, from these pages, whether that reflects a judgment against it or a review that has not happened. Anyone who claims to know which, citing only these documents, is guessing.

The preferred row that looks like a typo

The single preferred format in the digital video table is listed as AV1 Image File Format, AVIF, with AV1 as the codec. AVIF is better known as a still image format, so the entry is worth a second look. The Records Express post that announced it gives one reason:

"To remain consistent with NARA's digitization and motion picture preservation labs, this update adds AVIF as a preferred format for Digital Video"

That is the whole stated rationale. It ties the choice to what NARA's own labs produce, not to anything about consumer recording. For a phone owner the practical reading is modest: the preferred row describes how an archive lab digitizes, and the acceptable rows are where ordinary camera output lands.

The tables also move. The bulletin says NARA "may add new formats as they are adopted for use with Federal records, or may remove those that become obsolete". A reading of this table is a reading of one date. The page I fetched says it was last reviewed in March.

Three requirements that are about files, not formats

Section eight of the bulletin lists steps agencies must take in all cases. Two of them apply cleanly to a home archive.

The first concerns locks:

"Deactivate passwords or other forms of file level encryption including digital rights management (DRM) technologies commonly used with audio, video, and some publications that impede access to record data"

The archive's position is that a lock on the file is a dependency on whoever holds the key. For a household this is a question to ask of any camera system: when footage leaves the device, is it a plain file any player opens, or is it encrypted so that it opens only inside the vendor's viewer and account? The rule is written against the second kind.

The second concerns copies:

"Retain a copy of all permanent electronic records transferred to NARA until receiving official notification that NARA has accepted legal custody of the records."

The rule has an end condition, and the end condition is confirmation from the receiving side, not completion on the sending side. Translated to a home setup: a recording copied to a laptop or a drive is not safe to delete from the phone because the copy finished. It is safe once the copy has been opened and played from where it landed.

The third requirement is the validity rule already quoted, and it is the one with a practical trap.

Checking a folder yourself

Two commands from FFmpeg cover most of this. The first prints the codec of the video stream, which tells you which side of the HEVC line a file sits on:

ffprobe -v error -select_streams v -show_entries stream=codec_name -of default=nw=1:nk=1 clip.mp4
Enter fullscreen mode Exit fullscreen mode

On a test clip encoded with x264 it prints h264. On a test clip encoded with x265 it prints hevc. To sweep a folder of files without spaces in their names:

for f in *.mp4; do printf '%s ' $f; ffprobe -v error -select_streams v -show_entries stream=codec_name -of default=nw=1:nk=1 $f; done
Enter fullscreen mode Exit fullscreen mode

The second command decodes every frame and throws the output away, which is the closest thing a household has to a validity check:

ffmpeg -v error -i clip.mp4 -f null - 2> clip.decode.log
Enter fullscreen mode Exit fullscreen mode

Here is the trap, found by testing rather than assumed. I made a short clip with its index at the front of the file, cut off the last quarter, and decoded it. FFmpeg printed errors about invalid NAL unit sizes and a partial file, and then exited with status zero. A script that only checks the exit code would have reported that broken file as fine. A clip with its index at the end, cut in half, failed differently: it refused to open and exited non-zero.

So the check that matters is whether the log is empty:

test -s clip.decode.log && echo 'decode errors, read clip.decode.log'
Enter fullscreen mode Exit fullscreen mode

Two honest limits apply. A clean decode shows that this build of FFmpeg could read every frame, which is a necessary condition for validity and not a proof of it. NARA's bulletin does not name a validation tool, and I will not invent one for it. And the behaviour I describe is from the FFmpeg build in my test environment; other builds may report differently, which is exactly why the log, not the exit code, is the thing to read.

What this reading does not cover

  • Audio. The bulletin has a separate audio table, and a video file's audio track is held to it. I did not check what codec phone recordings use for sound.
  • Metadata. NARA has a separate metadata bulletin. Timestamps and location data in a recording are outside this piece.
  • Whether anything here binds a household. It does not. The value is that an institution with a long time horizon wrote its criteria down where anyone can read them.
  • What any specific app writes. I am describing the published guidance and two FFmpeg commands, not the internals of my own product or anyone else's.

The tables were read as fetched for this piece, the bulletin as revised in its current online form, and the Records Express post as originally published. All three can change, and the bulletin says plainly that the tables will.


Try it: Background Camera RemoteStream on Google Play -- record with the screen off and keep the footage on the phone. Setup notes and other write-ups are at superfunicular.com.

Sources. NARA Bulletin 2014-04, Format Guidance for the Transfer of Permanent Electronic Records, Appendix A tables of file formats and the Records Express post on the updated transfer guidance tables. Quotations are verbatim from those pages as fetched. The archive asks for the same thing from every file it keeps: that a stranger with ordinary tools can open it later, and a folder of home video can be held to that standard without any agency involved.

Top comments (1)

Some comments may only be visible to logged-in visitors. Sign in to view all comments.