tar czf on a Folder of Camera Clips Saved 1.3%, and 96% of That Was One CSV Log

A nightly job turns a folder of camera clips into one archive with tar czf. The z sends every byte through gzip. I measured what that bought on a test folder: the archive came out 1.31% smaller than plain tar cf, and 96% of that saving came from the one CSV log sitting next to the video. The six clips together gave up 12,171 bytes out of 36.9 million.

That result is not a surprise once you see where the bytes go, and the reason changes what the archive line should say. Below are the folder, five ways of archiving it, where the few kilobytes of clip savings actually come from, and a zip option that quietly cancels the fix.

Disclosure: I make an Android camera app. Nothing below uses its code or describes how it records, encodes or stores anything. The clips are synthetic, made with ffmpeg on a Linux machine, and every output is pasted from the scripts shown. Versions:

Linux 6.8.0-138-generic
GNU bash, version 5.1.16(1)-release (aarch64-unknown-linux-gnu)
tar (GNU tar) 1.34
gzip 1.10
*** zstd command line interface 64-bits v1.4.8, by Yann Collet ***
xz (XZ Utils) 5.2.5
This is Zip 3.0 (July 5th 2008), by Info-ZIP.
ffmpeg version 4.4.2-0ubuntu0.22.04.1

The test folder

Real footage would be a better test, and I do not have a day of it that I can publish. The stand-in is six 10-second 720p H.264 clips of ffmpeg's built-in test pattern with moving grain on top, so the encoder cannot treat the frames as easy, plus a CSV log with one line every five seconds for a full day:

#!/bin/bash
# Six 10-second 720p H.264 clips with moving grain, plus a CSV event log,
# standing in for a day of camera footage. Nothing here comes from a phone.
set -e
mkdir -p clips
for i in 1 2 3 4 5 6; do
  ffmpeg -nostdin -loglevel error -y \
    -f lavfi -i "testsrc2=size=1280x720:rate=30:duration=10" \
    -vf "noise=alls=8:allf=t:all_seed=$i" \
    -c:v libx264 -preset veryfast -crf 23 -pix_fmt yuv420p "clips/clip_0$i.mp4"
done
python3 -c '
import random
random.seed(1)
with open("clips/events.csv", "w") as f:
    f.write("time,clip,event,score\n")
    for s in range(0, 86400, 5):
        f.write("%02d:%02d:%02d,clip_0%d.mp4,%s,%.2f\n" % (s // 3600, s // 60 % 60, s % 60,
                1 + s % 6, random.choice(["motion", "none", "none", "sound"]), random.random()))
'
ls -l clips | awk 'NR > 1 {print $5, $9}'
du -cb clips/*.mp4 | tail -1

5705602 clip_01.mp4
6682703 clip_02.mp4
6397985 clip_03.mp4
6045601 clip_04.mp4
5830938 clip_05.mp4
6227074 clip_06.mp4
548675 events.csv
36889903    total

That is about 6 MB per 10-second clip. ffprobe reports 4,560,848 bit/s for the first one. The CSV is 1.47% of the folder's bytes.

Five ways to archive the same folder

Each archive is timed with date before and after, which is crude but enough to see the gaps:

#!/bin/bash
# The same folder archived five ways. Size in bytes, then seconds of wall time.
set -e
run() {  # run NAME OUTPUT COMMAND...
  local name=$1 out=$2; shift 2
  local t0=$(date +%s.%N)
  "$@"
  local t1=$(date +%s.%N)
  printf '%-22s %10d bytes  %6.2f s\n' "$name" "$(stat -c %s "$out")" "$(echo "$t1 - $t0" | bc)"
}
rm -f clips.tar clips.tgz clips.tgz9 clips.tar.zst clips.txz
run "tar cf (no compression)" clips.tar   tar cf clips.tar clips
run "tar czf (gzip -6)"       clips.tgz   tar czf clips.tgz clips
run "gzip -9"                 clips.tgz9  sh -c 'tar cf - clips | gzip -9 > clips.tgz9'
run "zstd -3"                 clips.tar.zst sh -c 'tar cf - clips | zstd -q -3 -o clips.tar.zst'
run "tar cJf (xz -6)"         clips.txz   tar cJf clips.txz clips

tar cf (no compression)   37447680 bytes    0.06 s
tar czf (gzip -6)        36958847 bytes    1.63 s
gzip -9                  36958643 bytes    1.55 s
zstd -3                  36968055 bytes    0.08 s
tar cJf (xz -6)          36924128 bytes   15.71 s

Every compressor lands within 1.4% of plain tar cf. The timings move from run to run: in two back-to-back runs tar czf took 0.99 s and then 1.63 s. The pattern held in both. xz at its default level took between ten and sixteen times as long as gzip to save 34,719 more bytes, gzip -9 saved 204 bytes more than the default -6, and zstd at level 3 finished in about the time plain tar did.

On a 37 MB folder none of those seconds matter. I measured only this folder, so I will not put a number on a full day of footage, but the per-clip saving is the part that does not grow into anything.

Which bytes actually shrank

Running gzip over each file on its own separates the clips from the log:

#!/bin/bash
# Each file through gzip -6 on its own: original bytes, gzipped bytes, change.
for f in clips/*; do
  a=$(stat -c %s "$f"); b=$(gzip -6 -c "$f" | wc -c)
  printf '%-12s %9d -> %9d  %+7.2f%%\n' "$(basename "$f")" "$a" "$b" "$(echo "scale=4; ($b - $a) * 100 / $a" | bc)"
done

clip_01.mp4    5705602 ->   5703509    -0.04%
clip_02.mp4    6682703 ->   6680775    -0.03%
clip_03.mp4    6397985 ->   6396034    -0.03%
clip_04.mp4    6045601 ->   6043511    -0.03%
clip_05.mp4    5830938 ->   5828857    -0.04%
clip_06.mp4    6227074 ->   6225046    -0.03%
events.csv      548675 ->     78688   -85.66%

Each clip shrank by about 2 KB, between 0.03% and 0.04%. The CSV shrank by 85.66%. Of the 488,833 bytes that tar czf saved, 469,987 are the CSV.

The gzip manual explains both numbers in one sentence. gzip uses "Lempel-Ziv coding (LZ77)", and "The amount of compression obtained depends on the size of the input and the distribution of common substrings." The CSV is made of common substrings: clip_0, .mp4,, none and the same few event names, thousands of times over. The clips are the output of a video encoder, whose whole job is to remove repetition before the file is written, so gzip finds very little left to match.

The 2 KB is the index, not the picture

An MP4 file is a sequence of boxes. Splitting the first clip at its top-level boxes and running each one through gzip on its own shows where the 2 KB came from:

# Split an MP4 into its top-level boxes and run each one through gzip -6 on its own.
import struct, subprocess, sys

data = open(sys.argv[1], "rb").read()
pos = 0
while pos < len(data):
    size, kind = struct.unpack(">I4s", data[pos:pos + 8])
    if size == 1:                      # 64-bit size follows the type
        size = struct.unpack(">Q", data[pos + 8:pos + 16])[0]
    elif size == 0:                    # box runs to the end of the file
        size = len(data) - pos
    box = data[pos:pos + size]
    z = len(subprocess.run(["gzip", "-6", "-c"], input=box, capture_output=True).stdout)
    print("%s %9d -> %9d  change %+6d bytes" % (kind.decode(), size, z, z - size))
    pos += size

ftyp        32 ->        48  change    +16 bytes
free         8 ->        28  change    +20 bytes
mdat   5701068 ->   5701904  change   +836 bytes
moov      4494 ->      1306  change  -3188 bytes

The mdat box holds the encoded frames, and gzip made it 836 bytes larger. The moov box holds the file's metadata, including the tables that say where each frame sits in mdat. Those tables are rows of similar numbers, and they shrank from 4,494 bytes to 1,306. The small saving on the whole clip is the index shrinking by more than the picture grew.

The growth on the picture is not a bug either. The manual states it outright: "Compression is always performed, even if the compressed file is slightly larger than the original. The worst case expansion is a few bytes for the gzip file header, plus 5 bytes every 32K block, or an expansion ratio of 0.015% for large files." 836 bytes on 5,701,068 is 0.0147%. The picture data sat right at gzip's documented worst case, which is what data gzip cannot compress looks like.

zip has an option for this, and -9 cancels it

zip has a list of file types it stores without trying to compress, and video is not on it. From the Info-ZIP man page: "By default, zip does not compress files with extensions in the list .Z:.zip:.zoo:.arc:.lzh:.arj". The -n option adds to that: "Do not attempt to compress files named with the given suffixes. Such files are simply stored (0% compression) in the output zip file, so that zip doesn't waste its time trying to compress them."

The same man page has a sentence a few lines further down that is easy to skip: "The maximum compression option -9 also attempts compression on all files regardless of extension." I tried all three:

#!/bin/bash
# zip with and without -n .mp4, and with -9 added. unzip -v shows the method per member.
rm -f plain.zip stored.zip stored9.zip
zip -q -r plain.zip clips
zip -q -r -n .mp4 stored.zip clips
zip -q -r -9 -n .mp4 stored9.zip clips
for z in plain.zip stored.zip stored9.zip; do
  printf '%-12s %9d bytes  ' "$z" "$(stat -c %s "$z")"
  unzip -v "$z" | awk '$8 ~ /clip_01|events/ {printf "%s %s  ", $8, $2} END {print ""}'
done

plain.zip     36957505 bytes  clips/events.csv Defl:N  clips/clip_01.mp4 Defl:N  
stored.zip    36969856 bytes  clips/events.csv Defl:N  clips/clip_01.mp4 Stored  
stored9.zip   36956640 bytes  clips/events.csv Defl:X  clips/clip_01.mp4 Defl:X  

unzip -v prints the method for each member. In plain.zip the clips were deflated (Defl:N, normal). With -n .mp4 they were stored, and the archive came out 12,351 bytes larger, which is about the 12 KB the clips gave up under gzip. Adding -9 to the same command put Defl:X, maximum, back on the clips. Someone who adds -9 because they want the smallest archive turns off the one option that told zip to leave the video alone.

What I would change in the archive line

• If the folder is almost all video, drop the z. tar cf here was 9,102 bytes larger than the files it holds, which is tar's own headers and padding.
• If there are text logs worth shrinking, compress those and not the clips. Either gzip the logs on their own, or pipe the whole tar through zstd -3, which on this folder kept 479,625 of gzip's 488,833 saved bytes at close to plain tar's speed.
• With zip, use -n .mp4 and leave out -9. Check with unzip -v that the clips say Stored.
• Keep a checksum if you drop compression. One thing the z does give you besides size: the manual says that for gzip and zip files, "gunzip checks a 32 bit CRC" when it decompresses, so a damaged archive is noticed on extraction. Drop the z and that check goes with it, so keep a sha256sum of the archive next to it.

Things I did not test

• Real phone footage. The clips here are a test pattern with added grain, encoded by ffmpeg's libx264. A phone's hardware encoder, a different bitrate, or HEVC instead of H.264 will give different sizes. I expect the picture data to behave the same way and have not measured it.
• A large folder. Everything above is 37 MB. I did not test how the timings grow on a day of footage, or multi-threaded compressors.
• Where moov sits. These clips have moov at the end of the file. I did not test whether moving it to the front changes the 2 KB.
• Other tool versions. The quotes come from the gzip man page and the Info-ZIP zip 3.0 man page. Everything I ran matched them, on the versions listed at the top.

Sources. gzip quotes are from the gzip(1) manual page as published by Hurricane Electric. zip quotes are from the Info-ZIP zip 3.0 manual page source. Outputs are pasted unedited from the scripts above.

If you want an old Android phone recording in the background so there are clips for a job like this to archive, the app I make is Background Camera RemoteStream on Google Play. Nothing above depends on it.

On this folder tar czf spent its time on the video and got its savings from the log, so compress the logs and store the clips.