DEV Community

Cover image for DeathMark: Programmatically scan videos for points of interest
Alex King for basal.

Posted on • Updated on

DeathMark: Programmatically scan videos for points of interest

Problem

In 2020 like most people our company transition to more remote work and I started to invest in my home office to prepare for the coming onslaught of zoom meetings. Once I had a comfortable setup, I had everything I needed to start streaming outside of work as well because why not right? I started playing around with twitch and cutting my videos down for youtube.

Most of my streams were 2+ hours and while playing 2+ hours of games with your friends is an investment in relaxation and friendship. Watching back 2+ hours wasn't a time investment I wanted to make.

Conception

My game of choice is Valorant at the moment but this is true of most games; In the game there is a visual queue to show you have scored a point.

Frame 7 (1)

Because they are UI elements they are usually visually bright, consistently placed, and distinct from the background view of the game.

The score indicator is kind of like a flashing light, if I had something like a Light-Dependent Resistor, I could record when it flashed.

Frame 8

Application

I have a concept "check for increases in white in the video"
MDN has a great example of how to check each frame of a video using HTML canvas. The example is set up like this:

Frame 9
ctx1 = Canvas 1 context
ctx2 = Canvas 2 context

We are going to borrow their function and focusing on the computeFrame section. You can see below the RGB values for each pixel in each frame.

 processor.computeFrame = function computeFrame() {
//drawing the full frame to canvas
    this.ctx1.drawImage(this.video, 0, 0, this.width, this.height);
//get the frame from canvas at 0 x and 0 y 
    let frame = this.ctx1.getImageData(0, 0, this.width, this.height);

    let l = frame.data.length / 4;

    for (let i = 0; i < l; i++) {
      let r = frame.data[i * 4 + 0];
      let g = frame.data[i * 4 + 1];
      let b = frame.data[i * 4 + 2];
      if (g > 100 && r > 100 && b < 43)
        frame.data[i * 4 + 3] = 0;
    }
    this.ctx2.putImageData(frame, 0, 0);
    return;
  }
Enter fullscreen mode Exit fullscreen mode

In the example, above it's checking this range of color and if it falls within the threshold it will make it alpha instead thus creating a green screen or in this case yellow screen.
Frame 10

Simple enough, i will just check for white pixel in the area.

//255,255,255 is white so 240 -> 255 is mostly white
 if (g > 240 && r > 240 && b < 240)
        //is white pixel
    }
Enter fullscreen mode Exit fullscreen mode

But the game has complex visuals and many elements would trigger just "white"

Frame 11

Every picture is made up of an almost unique amount of colors and shades so all I needed to do is get as close as possible to that unique number.

Frame 12

let skullFound = []
let white = []
let green = []
let red = []
for (let i = 0; i < l; i++) {
      let r = frame.data[i * 4 + 0];
      let g = frame.data[i * 4 + 1];
      let b = frame.data[i * 4 + 2];
       if (isWhite(r,g,b))
         white.push({r,g,b})
       }
      if (isGreen(r,g,b))
        green.push({r,g,b})
      }
      if (isRed(r,g,b))
        red.push({r,g,b})
      }
}

if(whiteThreshold(white.length) && greenThreshold(green.length) && redThreshold(red.length)) {
 skullFound.push(video.currentTime)
 white = []
 green = []
 red = []
}
Enter fullscreen mode Exit fullscreen mode

After 30ish minutes of trial and error, I was able to get about a 99% accuracy rate at 2x speed with the videos tested with 1 main exception being, if the Character Sage is within the cropped section when she is shot with a sniper rifle... which is kinda rare.

ng7h1c85uhhrl6n4kpwp

Conclusion

While the current system is not perfect, It's a simplistic solution to the problem I was facing that can be built upon later.

I believe methods like the one above can be applied to many game videos. I look forward to finding more fun techniques in this area in the future.

Free Download

https://ko-fi.com/post/DeathMark-W7W8381IO

Oldest comments (3)

Collapse
 
bias profile image
Tobias Nickel

this is cool, one question:

does it scan the video at play speed? or is it pissible to scan faster? maybe only one in 10 frames need to get analyzed.

Collapse
 
stagfoo profile image
Alex King

Sorry I didn't include it in the Article but yes It scans at 2x speed. I tried higher speeds or less frames but because the element is animated the colors vary based on the frame of the skull animation recognition went down. I did play with skipping frames but again if you miss the most accurately matched frame, it might not register. I am definitely looking at this because the faster the process the better.

Collapse
 
bias profile image
Tobias Nickel

nice, thanks, i understand that there is a trade off.