Three ffmpeg Filters Looked for a Person Crossing a Still Camera's View. One Found Them

Continuous recording has an obvious cost: hours of a picture in which nothing changes, with the few seconds you care about somewhere inside. The usual answer is motion detection in the camera app, decided at recording time. This piece is about the other answer: keep the continuous recording and ask a laptop afterwards where the picture changed.

To see which of ffmpeg's built-in filters can do that, I scored three of them against a ninety-second synthetic clip with a person-sized shape crossing the room at a known time. One could not tell the person from the sensor noise at any setting. Another caught the person only inside a narrow band of settings, where a slow dimming of the room looked just as busy. The third found the crossing to within a frame or two at each end, once its thresholds were raised well past the defaults.

I build a camera app for old Android phones, but nothing here depends on it, and I am not describing how my app or any other app detects motion. Every result below comes from the commands shown, so you can rerun them and then try them on your own footage.

The test clip

The clip is synthetic on purpose: I know exactly when each event happens, so I can score the filters against the truth. It has four things in it. A grey room with a darker shape in one corner. A dark block, about the size a person would be at that distance, entering at the left edge at 20 s and leaving at the right edge by 26 s. A slow dimming of the whole picture from 40 s to 70 s, the way a room darkens at dusk. And a light switched back on, in a single frame, at 80 s. Every frame also carries temporal noise, which is what a small sensor produces in low light and what makes this hard.

# 90 s synthetic "fixed camera" clip, 15 fps:
#   a person-sized dark block walks in from the left edge at 20 s and out the right edge by 26 s,
#   the room dims slowly from 40 s to 70 s and the light snaps back on at 80 s,
#   and every frame carries temporal noise, like a small sensor in low light.
ffmpeg -v error -y \
  -f lavfi -i "color=c=0x707070:s=480x270:r=15:d=90" \
  -f lavfi -i "color=c=0x202020:s=40x150:r=15:d=90" \
  -filter_complex "[0]drawbox=x=40:y=150:w=160:h=90:c=0x404040:t=fill[bg];
    [bg][1]overlay=eval=frame:y=60:x='if(between(t,20,26),(t-20)*90-40,-100)',
    eq=eval=frame:brightness='if(between(t,40,80),-0.2*min(1,(t-40)/30),0)',
    noise=alls=10:allf=t" \
  -c:v libx264 -preset veryfast -crf 23 -g 30 -pix_fmt yuv420p clip.mp4

I checked the clip by pulling single frames and reading pixel values: at 23 s the block is in frame where expected, at 60 s the room is dimmer, at 85 s it is back to full brightness. A test clip needs its own check before it can check anything else: a clip in which the block never appeared would make every detector look right for reporting nothing.

All runs below use ffmpeg 4.4.2 from Ubuntu 22.04. The current documentation describes options this build does not have, which I note where it matters.

freezedetect: built for the opposite question

freezedetect exists to find video that has stopped. The ffmpeg filter documentation says it "logs a message and sets frame metadata when it detects that the input video has no significant change in content during a specified duration." For a fixed camera, a stretch with no significant change is an empty room, so the gaps between freezes should be where something happened. The documentation also says how it decides: "Video freeze detection calculates the mean average absolute difference of all the components of video frames and compares it to a noise floor."

That word mean is the whole story, as it turned out. Here is a small script that turns the filter's log into intervals. A gap between two freezes is printed as changing; a freeze that ends and restarts on the same frame, which is what a sudden step looks like, is printed as cut.

# Same clip, three noise tolerances: the default, 0.02 and 0.03.
for n in 0.001 0.02 0.03; do
  echo "== freezedetect n=$n"
  ffmpeg -hide_banner -nostats -i clip.mp4 -vf "freezedetect=n=$n:d=2" -f null - 2>&1 \
    | python3 gaps.py 90
done

# Turn freezedetect's log into "something changed here" intervals.
# Usage: ffmpeg -i clip.mp4 -vf freezedetect=n=0.02:d=2 -f null - 2>&1 | python3 gaps.py 90
import re, sys

duration = float(sys.argv[1])
events = re.findall(r"freeze_(start|end): ([0-9.]+)", sys.stdin.read())
still = []
for kind, t in events:
    if kind == "start":
        still.append([float(t), duration])   # a freeze with no end runs to the end of the file
    else:
        still[-1][1] = float(t)

cursor = 0.0
for start, end in still:
    if start > cursor:
        print("changing  %6.2f -> %6.2f  (%.2f s)" % (cursor, start, start - cursor))
    elif start == cursor and cursor > 0:
        print("cut       %6.2f" % cursor)
    cursor = end
if cursor < duration:
    print("changing  %6.2f -> %6.2f  (%.2f s)" % (cursor, duration, duration - cursor))

== freezedetect n=0.001
changing    0.00 ->  90.00  (90.00 s)
== freezedetect n=0.02
changing   20.13 ->  25.80  (5.67 s)
cut        40.07
changing   42.53 ->  43.07  (0.53 s)
cut        46.07
cut        49.07
cut        52.07
changing   54.13 ->  56.53  (2.40 s)
cut        59.53
cut        62.53
cut        65.53
changing   67.67 ->  70.00  (2.33 s)
cut        80.07
== freezedetect n=0.03
cut        43.07
cut        49.07
cut        55.00
cut        61.00
cut        67.00
cut        80.07

At the default noise tolerance, the whole ninety seconds is one long change, because the sensor noise alone is larger than the floor. That is a useful warning on its own: on footage from a small sensor at night, the defaults will report activity everywhere.

At 0.02 the block appears, from 20.13 s to 25.80 s, which matches the stretch when any part of it is in the frame, give or take a frame or two. But the slow dimming produces three more change intervals and seven cuts, so a person and a sunset look alike in the output. At 0.03 the person disappears entirely, and only the dimming and the light switch remain. I swept the tolerance more finely: nothing froze at all up to 0.019, and the block was still caught at 0.025. On this clip the usable window is narrow, and a person-sized object is a small share of the frame, so a whole-frame average barely moves when one walks through.

The scene score: built for edits

The select filter can compute a scene score for every frame. The documentation describes it as a "value between 0 and 1 to indicate a new scene; a low value reflects a low probability for the current frame to introduce a new scene". That is a tool for finding cuts in edited video, and the numbers show it.

# Score every frame, then list the frames whose scene score clears 0.1.
ffmpeg -v error -i clip.mp4 \
  -vf "select='gte(scene,0)',metadata=print:key=lavfi.scene_score:file=scores.txt" -f null -
ffmpeg -hide_banner -nostats -i clip.mp4 -vf "select='gt(scene,0.1)',showinfo" -f null - 2>&1 \
  | grep -o "pts_time:[0-9.]*"

pts_time:80.0667

# Highest scene score inside each stretch of the clip, read from scores.txt.
import re

text = open("scores.txt").read()
pairs = re.findall(r"pts_time:([0-9.]+)\s+lavfi\.scene_score=([0-9.]+)", text)
frames = [(float(t), float(s)) for t, s in pairs]
stretches = [("empty room", 0.2, 19.9), ("person crossing", 19.9, 26.1),
             ("slow dimming", 40.0, 70.0), ("light switch", 80.0, 80.2)]
for name, a, b in stretches:
    print("%-16s max %.4f" % (name, max(s for t, s in frames if a <= t < b)))

empty room       max 0.0055
person crossing  max 0.0063
slow dimming     max 0.0123
light switch     max 0.5050

With a cut-off of 0.1, the one frame selected is the light switch. Lower cut-offs do not help: the highest score any frame reached while the person was crossing is barely above the highest score of the empty room, so no threshold separates the two. The slow dimming scored higher than the person did.

mpdecimate: built for something else, and it worked

mpdecimate drops frames that barely differ from the previous one. The documentation is candid that this is not its intended job: "The main use of this filter is for very-low-bitrate encoding (e.g. streaming over dialup modem)". What made it work here is how it measures difference. It looks at small blocks, not the whole frame: "A frame is a candidate for dropping if no 8x8 blocks differ by more than a threshold of hi, and if no more than frac blocks (1 meaning the whole image) differ by more than a threshold of lo." One block changing a lot is enough to keep a frame, so a person who fills one corner of the picture still counts.

# How many of the 1350 frames does mpdecimate keep, and when are they?
for opts in "" "hi=64*30:lo=64*15:frac=0.33"; do
  ffmpeg -hide_banner -nostats -i clip.mp4 -vf "mpdecimate=$opts,showinfo" -f null - 2>&1 \
    | grep -o "pts_time:[0-9.]*" | cut -d: -f2 > kept.txt
  echo "mpdecimate=${opts:-defaults}: kept $(wc -l < kept.txt) of 1350 frames"
done

mpdecimate=defaults: kept 1350 of 1350 frames
mpdecimate=hi=64*30:lo=64*15:frac=0.33: kept 92 of 1350 frames

With defaults it keeps every frame, for the same reason as freezedetect: the noise is larger than the thresholds. With hi and lo raised, it keeps a small fraction. This script groups the kept frames into intervals:

# Collapse kept-frame timestamps into intervals: frames less than 1 s apart belong together.
import sys

times = [float(line) for line in open(sys.argv[1]) if line.strip()]
start = prev = times[0]
for t in times[1:] + [None]:
    if t is None or t - prev > 1.0:
        n = sum(1 for x in times if start <= x <= prev)
        print("%7.2f -> %7.2f  %3d frames" % (start, prev, n))
        if t is not None:
            start = t
    if t is not None:
        prev = t

   0.00 ->    0.00    1 frames
  20.13 ->   25.80   86 frames
  46.07 ->   46.07    1 frames
  55.00 ->   55.00    1 frames
  62.73 ->   62.73    1 frames
  70.87 ->   70.87    1 frames
  80.07 ->   80.07    1 frames

The crossing comes out as one interval, 20.13 s to 25.80 s, holding almost all of the kept frames. The dimming leaves four isolated single frames, one just after it ends, the light switch leaves one, and the first frame of the file is kept too. I tried other threshold pairs too: lower values kept a few more dimming frames, and higher values started dropping frames from the crossing, but every raised setting I tried showed the person as a single obvious cluster.

The practical output is the list itself. Instead of watching ninety seconds, or nine hours, you get a short list of times to jump to, and single isolated frames can be read as light changes rather than visitors.

Calibrate on yourself before you trust it

None of these thresholds will transfer to your footage. They depend on the sensor, the resolution, the compression, the lighting and the size of a person in the frame. What does transfer is the method, and the method needs a positive control: a change you know happened. Walk through the frame yourself at a time you note down, then tune hi and lo until that walk comes out as a cluster and a stretch you know was empty does not. If you cannot make your own walk appear, a result of no activity means nothing, because the tool could not have found anything.

What I did not establish

• Real footage. My noise comes from ffmpeg's noise filter, not a phone sensor, and my person is a uniform dark block. Real people have texture, clothing that matches the wall, and shadows.
• Night mode and auto-exposure. Many phones adjust exposure as light changes. A whole-frame exposure step would look like my light switch, but I have not measured how often a real phone produces one.
• Moving backgrounds. Trees in wind, a television, or a curtain near a heater vent would keep frames in a block-based filter. I did not test any of them.
• Other builds. The current documentation lists mpdecimate options, such as a second mode, that the 4.4.2 help output does not show. Check ffmpeg -h filter=mpdecimate on your own build.
• Speed on long recordings. I ran these on a ninety-second clip. I have not timed them on a full night of video.

---

Try it: Background Camera RemoteStream on Google Play -- record with the screen off and keep the footage on the phone. Setup notes and other write-ups are at superfunicular.com.

Sources. The FFmpeg filters documentation, sections on freezedetect, mpdecimate and the select filter's scene variable. Quotations are verbatim from that page as fetched. Test results come from the scripts above, run as shown. A filter that averages the whole frame is measuring the room, and a person is a small part of the room.