I put one portrait test clip through two paths on the same page. In a <video> element it played upright at 1080×1920. Through my own pipeline every frame came out 1920×1080 with the picture lying on its side. That pipeline is mp4box.js 0.5.2 to demux, which means pulling the compressed samples out of the MP4, and then VideoDecoder, the WebCodecs interface that turns those samples into frames. I expected at least one browser to disagree with the others, and none did. Chromium 149, Chrome 154, Firefox 151 and WebKit 26.5 (macOS) all handed me lying-down frames from the decoder path, and all four played the same file upright.
The red and green blocks were painted into the top-left and top-right corners of the upright picture when the sample was made. In the rightmost panel they sit on the left edge, so the frame is lying with its top to the left. The footage is a wind turbine video by Grendelkhan on Wikimedia Commons, CC BY-SA 4.0.
Playback is upright and the decoder output is not
The clip is not phone footage. I took that Wikimedia clip and used ffmpeg 7.1 to store the pixels sideways at 1920×1080 and write a rotation tag, which is the common way portrait video gets stored. The direction is written on the packaging, not in the picture. <video> reads the packaging: videoWidth × videoHeight reported 1080×1920, and drawImage(video) onto a canvas was upright in all four browsers. A decoder that is fed raw samples only ever gets the picture.
My next hope was VideoFrame.rotation, a property on a frame that says how far to turn it. With new VideoFrame(video) in Chromium 149 and Chrome 154 it worked: coded size 1920×1080, display size 1080×1920, and rotation read 90, 270 or 180 to match the sample. Firefox 151 and WebKit 26.5 had no rotation property on the frame, and the Firefox frame drew sideways. Frames from my own decoder reported rotation = 0 in both Chromium-based browsers. So I can't lean on the property.
The other shortcut, createImageBitmap(video), which grabs the current frame as a bitmap, did not agree across browsers either.
Look at the third panel in each row. In Chromium 149 the bitmap is 1920×1080 with the picture the right way up but squashed: the 160×160 marker blocks came out at about 284×90. Chrome 154 ran the same code and returned 1080×1920 with square blocks. Firefox 151 returned 1920×1080 lying down.
The angle is in the track header matrix
In MP4 and MOV the rotation is a 3×3 matrix in the track header box, tkhd. mp4box gives it to you along with the track. For my 90-degree sample it read [0, 65536, 0, -65536, 0, 0, 0, 0, 1073741824], and the untagged upright sample gave the identity matrix. The numbers are fixed-point, so 65536 is 1.0, and the first two are enough to get the angle:
const [a, b] = trak.tkhd.matrix; // 65536 = 1.0
const turn = (Math.round(Math.atan2(b, a) * 180 / Math.PI) + 360) % 360;
// [0, 65536, ...] -> 90, identity -> 0
Two things caught me here. ffmpeg prints rotation of -90.00 degrees for that same file, so I check the result against the marker blocks instead of trusting a sign. And mp4box reports track_width × track_height as 1920×1080, the stored size, so a canvas sized from the track is landscape. For a quarter turn I swap width and height, then rotate the context before drawing the frame. The 180-degree sample is the sneaky one, because its size doesn't change and a width check will never catch an upside-down frame. I haven't tested the mirrored matrices a front camera might write, and the snippet above would ignore a flip.
Bake the rotation into the pixels on export
On the way out there are two options: write upright pixels, or keep the sideways pixels and write the tag again. I tried the tag in WebM first. ffmpeg with -noautorotate to VP9 kept the 1920×1080 lying pixels, and ffmpeg could still read the -90 rotation from the result. Chromium 149 and Chrome 154 played that file upright at 1080×1920. Firefox 151 and WebKit 26.5 played it lying down at 1920×1080. Upright pixels with no tag were upright on every path in every browser I tested, so I rotate at encode time and write no tag.
To check an exporter that takes this route I used ImgIng (https://imging.ai/) in Chromium 149 with the preset that favors smaller files. The samples tagged 90, 270 and 180 all came out as MP4 stored upright at 608×1080 with no rotation tag, and the 90-degree one did the same as WebM.
None of this covers phone originals, HEVC, iPhone Safari or Android; the tags here were all written by ffmpeg. If you run your own decode path, the check is cheap. Paint two different blocks into the top corners of a portrait clip, tag it with ffmpeg, and see where the blocks land.
Top comments (0)