DEV Community: Alex King

DeathMark: Programmatically scan videos for points of interest

Alex King — Wed, 13 Jan 2021 03:19:49 +0000

Problem

In 2020 like most people our company transition to more remote work and I started to invest in my home office to prepare for the coming onslaught of zoom meetings. Once I had a comfortable setup, I had everything I needed to start streaming outside of work as well because why not right? I started playing around with twitch and cutting my videos down for youtube.

Most of my streams were 2+ hours and while playing 2+ hours of games with your friends is an investment in relaxation and friendship. Watching back 2+ hours wasn't a time investment I wanted to make.

Conception

My game of choice is Valorant at the moment but this is true of most games; In the game there is a visual queue to show you have scored a point.

Because they are UI elements they are usually visually bright, consistently placed, and distinct from the background view of the game.

The score indicator is kind of like a flashing light, if I had something like a Light-Dependent Resistor, I could record when it flashed.

Application

I have a concept "check for increases in white in the video"
MDN has a great example of how to check each frame of a video using HTML canvas. The example is set up like this:

ctx1 = Canvas 1 context
ctx2 = Canvas 2 context

We are going to borrow their function and focusing on the computeFrame section. You can see below the RGB values for each pixel in each frame.

 processor.computeFrame = function computeFrame() {
//drawing the full frame to canvas
    this.ctx1.drawImage(this.video, 0, 0, this.width, this.height);
//get the frame from canvas at 0 x and 0 y 
    let frame = this.ctx1.getImageData(0, 0, this.width, this.height);

    let l = frame.data.length / 4;

    for (let i = 0; i < l; i++) {
      let r = frame.data[i * 4 + 0];
      let g = frame.data[i * 4 + 1];
      let b = frame.data[i * 4 + 2];
      if (g > 100 && r > 100 && b < 43)
        frame.data[i * 4 + 3] = 0;
    }
    this.ctx2.putImageData(frame, 0, 0);
    return;
  }

In the example, above it's checking this range of color and if it falls within the threshold it will make it alpha instead thus creating a green screen or in this case yellow screen.

Simple enough, i will just check for white pixel in the area.

//255,255,255 is white so 240 -> 255 is mostly white
 if (g > 240 && r > 240 && b < 240)
        //is white pixel
    }

But the game has complex visuals and many elements would trigger just "white"

Every picture is made up of an almost unique amount of colors and shades so all I needed to do is get as close as possible to that unique number.

let skullFound = []
let white = []
let green = []
let red = []
for (let i = 0; i < l; i++) {
      let r = frame.data[i * 4 + 0];
      let g = frame.data[i * 4 + 1];
      let b = frame.data[i * 4 + 2];
       if (isWhite(r,g,b))
         white.push({r,g,b})
       }
      if (isGreen(r,g,b))
        green.push({r,g,b})
      }
      if (isRed(r,g,b))
        red.push({r,g,b})
      }
}

if(whiteThreshold(white.length) && greenThreshold(green.length) && redThreshold(red.length)) {
 skullFound.push(video.currentTime)
 white = []
 green = []
 red = []
}

After 30ish minutes of trial and error, I was able to get about a 99% accuracy rate at 2x speed with the videos tested with 1 main exception being, if the Character Sage is within the cropped section when she is shot with a sniper rifle... which is kinda rare.

Conclusion

While the current system is not perfect, It's a simplistic solution to the problem I was facing that can be built upon later.

I believe methods like the one above can be applied to many game videos. I look forward to finding more fun techniques in this area in the future.

Free Download

https://ko-fi.com/post/DeathMark-W7W8381IO

Using Figma for Online Dungeons and Dragons

Alex King — Mon, 04 May 2020 15:22:32 +0000

I recently started dungeon mastering for a group. we tried to video but being spread across the world; video isn't great all the time so we started looking for different solutions

First try: Google Sheets

while it was functional, it was hard to find good colors to represent the players and enemies and got confusing.

I'm the pink one

Figma

is a free, easy to use vector editing tool in the browser.

Figma has a built-in layout grid system so you can overlay it on top of your dnd map for the distance the free dnd maps I used are here

The grid lines work for you but they don't work for people viewing an embed. There is a great Figma plugin called Layout Grid Visualiser that creates a copy of your grid as a layer and boom, you have a great grid.

To allow people to call out the number I used a mono spaces font and set the spacing to 60% so one character per square.

People can then say Pink A Green 1 or PAG1

this looks great but I didn't want everyone to have to sign up and they could see all the maps and layers for the campaign and spoil it for themselves;

Sidenote: I also made some hero markers using just a circle and clipping mask.
Artwork by E.M. Engel's

Glitch

this web app lets you whip up experiments of HTML and js so I made a very basic landing page.

I added links to an HTML page with full-screen single Figma embed limited to 1 frame.

Now I can plan out all my maps in secret for the campaign, list them on the page and hide and show them when needed!

When I move a character on the embed it will move in real-time without video streaming.

Finally, for voice, we are using discord but you could use anything, you could even just type it!

Web Components: Understanding Attributes and Properties

Alex King — Wed, 06 Feb 2019 02:40:44 +0000

When working with component libraries there is a blending of Javascript Properties and HTML Attributes that can make it confusing to understand what they really are and how they work. This can be a problem later on so here is a breakdown.

HTML elements have attributes. Theses are strings that can be passed into an HTML Tag.

    <img src="https://picsum.photos/200/300" />

This is because a HTML Document is only a string until it is coupled with a DOM.

The Document Object Model (DOM) represents that same document ... The DOM is an object-oriented representation of the web page, which can be modified with a scripting language such as JavaScript. - MDN

When you request a tag using Javascript, it will return an Element Object from the DOM; with this Javascript object, you can change the attributes of the HTML element but you can also set properties of the object itself which don't have visible representation in the document.

Properties can be complex data like arrays, functions or other objects; This is especially useful when storing data related to a node.

Try it yourself in your browser console.

$0 //will return the selected element in chrome/firefox/safari
$0.coolBeans = ['baked', 'black', 'refried'];
$0.coolBeans; // return a real array, not a string ['baked', 'black', 'refried']

Now lets do the same with attributes

$0 //will return the selected element in chrome/firefox/safari
$0.setAttribute('coolBeans', ['baked', 'black', 'refried']);
$0.coolBeans; // return undefined
$0.getAttribute('coolBeans'); // return a string "baked,black,refried"

Things get a little more complex when you start using a property that is already defined as an attribute for that tag.

const imgTag = document.createElement('img'); //non rendered img tag
imgTag.srcset = ['https://picsum.photos/320/320 320w',
                                 'https://picsum.photos/480/320 480w'
                                 'https://picsum.photos/640/320 640w']
// this is going to return a real array right?
// nope
imgTag.srcset; //"https://picsum.photos/320/320 320w,https://picsum.photos/480/320 480w,https://picsum.photos/640/320 640w"

This Image Element's Class has a property handlers for srcset which reflect it has attributes and set the property as well.

For the most part this isn't a problem but it's helpful to understand this relationship because it will help you build better more reusable Web Components.

We will be using the library Lit Element from Google in this Article. Lit Element includes a nice base structure and lets you create HTML structure for your component using template literals.

Below is a basic example.

class BeanList extends LitElement {
  constructor() {
    super();
  }
  render() {
    return html`
       <ul>
            <li>Baked<li>
            <li>Black</li>
          <li>Refried</li>
         </ul>
    `
  }
}
customElements.define('cool-beans', BeanList);
// You can now add <cool-beans></cool-beans> to your html

This will now render a beautiful list of cool beans, but it's a static string so let's move them to an array.

class BeanList extends LitElement {
  constructor() {
    super();
    this.list = [
            'Baked',
            'Black',
            'Refried'
        ]
  }
  render() {
    return html`
      <ul>
        ${this.list.map((beanName)=> { return html`<li>${beanName}</li>`})}
      </ul>
        `
  }
}

Now we have set the .list of the object, we can access this data using Javascript.

$0 //will return the selected element in chrome/firefox/safari
$0.list // its return a real array ['Baked','Black','Refried']
$0.list = ['Baked','Black','Refried','Pino'];
$0.list // ['Baked','Black','Refried','Pino']

const x = document.createElement('cool-beans');
x.list = ['Baked','Black','Refried','Pino'];
document.body.appendChild(x);
// This WILL render the <li>Pino</li> because the list was updated before
// it rendered the tag

But you will notice the HTML doesn't update. The Object property is not connected to the Render cycle and it won't re-render if this property is changed. So let's make it listen to our new list.

class BeanList extends LitElement {
  constructor() {
    super();
    this.list = [
            'Baked',
            'Black',
            'Refried'
        ]
  }
    static get properties() {
    return {
      list
    };
  }
  render() {
    return html`
      <ul>
        ${this.list.map((beanName)=> { return html`<li>${beanName}</li>`})}
      </ul>
        `
  }
}

Now we have added a property listener, when this property is updated, it will re-render.

$0 //will return the selected element in chrome/firefox/safari
$0.list // its return a real array ['Baked','Black','Refried']
$0.list = ['Baked','Black','Refried','Pino'];
$0.list // It will re-render and you still have
              // a real ['Baked','Black','Refried','Pino']

So now our example from before works perfectly. Now the final piece of the puzzle is connecting out properties to our attributes.

class BeanList extends LitElement {
    constructor() {
    super();
    this.list = [
            'Baked',
            'Black',
            'Refried'
        ]
  }
  static get properties() {
   return {
      list: {
      reflect: true,
      type: {
        fromAttribute: x => x.split(','),
        toAttribute: x => x.join(),
      }
    },
   }
  }
  render() {
    return html`
      <ul>
        ${this.list.map((beanName)=> { return html`<li>${beanName}</li>`})}
      </ul>
        `
  }
}

Hooray with this above example, you can now do both!

$0 //will return the selected element in chrome/firefox/safari
$0.setAttribute('list', ['Baked','Black','Refried','Pino'];);
$0.list = ['Baked','Black','Refried','Pino'];

Both statements will update the Attributes, Property and HTML of the component.

fromAttribute and toAttribute are very powerful interfaces to have but lit-element comes with some pre-made converters like Array the full list can be found here.

If you want to use more complex structures, you are unlikely to reflect those as attributes so you don't need the converters.

class BeanList extends LitElement {
  static get properties() {
   return {
      list: { attribute: false },
   }
  }
  constructor() {
    super();
    this.list = [];
  }
  render() {
    return html`
      <ul>
        ${this.list.map((bean)=> { return html`<li><caption><img src="${bean.img}" />${bean.name}</caption></li>`})}
      </ul>
        `
  }
}
customElements.define('cool-beans', BeanList);

In the above example you, no attribute is created on the element only properties and you a set them using a range of framework including native HTML. Have a play.

One Caveat is that some frameworks don't pass properties correctly to Web Components yet. A list of compatible frameworks can be found here