I'm building a small clip-sharing site as a side project. You pick a video, the page grabs one frame as a thumbnail before uploading, and the list page shows a grid of those thumbnails. While testing in Firefox I opened a vertical clip and it played upright, the way it should. Then I went back to the list, and the thumbnail for that same clip was lying on its side. Same file both times.
It turns out the picture in a vertical video file is often not stored standing up. The pixels are saved lying down, 1920 wide and 1080 tall, and the file carries a small note next to them that says "turn this 90 degrees when you play it". The proper name for that note is the rotation flag, and in MP4 and MOV files it lives in the track header, outside the picture data. A player reads the note and turns the picture. My thumbnail code got its frame a different way, and that way didn't read the note. Photos have a similar orientation story, but I'm staying with video here.
To rule out my own app I made test clips from two example videos from Wikimedia Commons, a wind turbine and a snowy street, both vertical. With ffmpeg 7.1 I painted a red square in the top-left corner and a green one in the top-right, laid the pixels on their side, and wrote a rotation flag into the file. So these are stand-ins made with ffmpeg, not files straight off a phone. Then I loaded them in one test page in Chromium 149, Chrome 154, Firefox 151 and WebKit 26.5 on macOS. Playback was upright in all four, and videoWidth × videoHeight reported 1080×1920 in all four, so the <video> element already hands you the turned size.
This is the turbine clip in Firefox 151. Follow the red square. In the two panels on the left, the <video> element and a canvas drawImage, it sits in the top-left corner and the size is 1080×1920. In the other three it has dropped to the bottom-left and the size is 1920×1080, which means I got the stored pixels with the note ignored.
The middle panel was my bug. I had used createImageBitmap(video), which turns a video's current frame into a bitmap, because it was one line. In Firefox 151 it returns the sideways 1920×1080 picture. Chromium 149 did something stranger: the bitmap was also 1920×1080, but the picture inside was upright and squashed to fit, so my 160×160 squares came out at roughly 284×90. Chrome 154 and WebKit 26.5 returned a correct 1080×1920 bitmap. I don't know why the two Chromium-family versions differ, and I didn't dig into it.
// before: sideways in Firefox 151, squashed in Chromium 149
const thumb = await createImageBitmap(videoEl);
// after: upright in all four browsers I tried
const cnv = document.createElement('canvas');
cnv.width = videoEl.videoWidth; // 1080, already the turned size
cnv.height = videoEl.videoHeight; // 1920
cnv.getContext('2d').drawImage(videoEl, 0, 0);
I repeated it with the snow clip in case it was something about one file. Same pattern, with the sizes swapped to 720×1280 and 1280×720: the canvas frame is upright and the bitmap is sideways. For a moment I thought about keeping the one-liner and just comparing the bitmap's width with videoWidth. That catches the 90 and 270 degree clips, but a clip flagged 180 degrees keeps its size and simply comes out upside down in the paths that ignore the note. I kept the canvas.
The two right-hand panels in the first picture are WebCodecs paths. new VideoFrame(video) in the two Chromium-family browsers has a rotation property that read 90 for this clip, and the frame draws upright. In Firefox 151 there was no rotation property and it drew sideways. If you split the file yourself with mp4box.js and feed the pieces to VideoDecoder, you get the sideways picture in all four browsers. The decoder only receives the picture data. The note stays behind in the container, so turning the frame becomes your job.
My site also shrinks clips before upload, so I wanted to know which way the pixels sit after compression. I used ImgIng (https://imging.ai/) for that. I ran its video compressor in Chromium 149 on the size-first preset, and all three flagged turbine clips (90, 270 and 180 degrees) came out as 608×1080 files with the pixels stored upright and no rotation flag left. My unflagged upright samples were right in every path and every browser above, so a file like that doesn't depend on anyone reading a note. That is one browser, one preset and five-second clips, nothing wider.
I haven't tested files straight from a real phone, HEVC, Safari on an iPhone, or anything on Android. The thumbnail in my project now goes through a canvas, and the red-and-green turbine clip stays in my test folder. The turbine footage is by Grendelkhan and the snow footage by Canbel, both from Wikimedia Commons under CC BY-SA 4.0.
Top comments (0)