How to glitch video in the age of web

#webdev #showdev #javascript #programming

The tool described in this post is available at ezglitch.kopanko.com

For years I've been interested in datamoshing and glitch art, but mainly for the computer aspect of it, like, you know, you edit some parts of the file, and it plays differently? How cool is that, right?

But if you wanted to get into glitching, there's an obvious barrier! Most tutorials rely on old and buggy software or require you to download countless environments and tools onto your computer! Some people argue that if you don't do it with buggy software, it ain't glitch-art at all!

In the past, I have had made my own tools to break files for me, like glitchbox, which was basically a JavaScript interface to ffglitch (back when it had none), always trying to make things as easy as possible for the end-user.

So, one evening, I sat down and set on rewriting my go-to AVI glitching tool, tomato for the web. Let me start by explaining how the AVI file is actually constructed. AVI files consist of three basic parts:

hdrl buffer - a header of sorts that contains data on the total amount of frames, width, and height of the video, and so on.
movi buffer - this is the part we actually care about as it contains raw frame data.
idx1 buffer - holds the index.

Now, the frames in the movi buffer are arranged as they will be played by the player. Audio data starts with the string 01wb and compressed video with 00dc. They end just before the next such tag or just before the idx1 buffer tag.

For the fun part - if we rearrange or copy those frames around, the player will play them right as it sees them. We don't need to know the exact structure of the frame, its DCT coefficients, or some other complicated technical stuff - we just need to be able to move bytes around! Fortunately for us, that is entirely possible in modern browsers!

const buf = await file.arrayBuffer();
const moviBuffer = buf.slice(moviMarkerPos, idx1MarkerPos);

Now that we have the entire movi buffer, we need to construct a frame table. We use some string-search algorithm to find all occurrences of 00dc or 01wb in the buffer - they mark the beginning of every frame.

// this is just "00dc" in hexadecimal
const pattern = new Uint8Array([0x30, 0x30, 0x64, 0x63]);
const indices = new BoyerMoore(pattern).findIndexes(moviBuffer);
const bframes = indices.map(v => {return {type: 'video', index: v}});

We do the same thing to I-frames, combine the two, and sort them based on their index. Then, we need to get each frame's byte size (which will come in very handy in a moment):

const table = sorted.map((frame, index, arr) => {
  let size = -1;
  if (index + 1 < arr.length)
    size = arr[index + 1].index - frame.index;
  else
    size = moviBuffer.byteLength - frame.index;
  return {...frame, size}
})

This has been a pretty linear and dull process so far, but now we get to have some genuine fun - we get to come up with a function to mess with the frames! Let's do the simplest thing and just reverse the whole array.

let final = table;
final.reverse();

This will, obviously, make the video play backward, but since the frames encoding motion do not take this into account we effectively flipped the motion vectors inside them, which in turn leads to a very odd effect in playback. Keep in mind the frames are still valid, and their data hasn't changed - just their order inside the file.

OK, so that's it? Well, not yet. We still need to reconstruct the new movi buffer from the frame table and combine it with hdrl and idx1 buffers. How do we approach it?

The best way to do it is to get the final size of the movi buffer and allocate that much memory beforehand so that we don't ever have to resize our Uint8Array.

let expectedMoviSize = 4;
final.forEach(frame => expectedMoviSize+=frame.size);

Wait, why expectedMoviSize = 4? Well, now we initialize the TypedArray with the final size and set the first 4 bytes to the movi tag itself.

let finalMovi = new Uint8Array(expectedMoviSize);
finalMovi.set([0x6D, 0x6F, 0x76, 0x69]);

This is the final stretch - for every frame in the frame table, we read the data from the original file and write it at the correct offset in the final movi tag. We advance the head by the frame bytesize so that the frames are written sequentially.

let head = 4; // guess why we start at 4

for (const frame of final)) {
  if(frame.index != 0 && frame.size != 0) {
    const data = moviBuffer.slice(frame.index, frame.index + frame.size);
    finalMovi.set(new Uint8Array(data), head);
    head += frame.size;
  }
}

Now all there's left is to recombine it with the original hdrl and idx1 and we're done!

let out = new Uint8Array(hdrlBuffer.byteLength + finalMovi.byteLength + idx1Buffer.byteLength); 
out.set(new Uint8Array(hdrlBuffer));
out.set(finalMovi, moviMarkerPos);
out.set(new Uint8Array(idx1Buffer), hdrlBuffer.byteLength + finalMovi.byteLength);