DEV Community: Will Blaschko

Falling Blocks - "Tetris" On Alexa (APL)

Will Blaschko — Fri, 17 May 2019 22:10:16 +0000

TL;DR

I created a fully-working Tetris clone (named Falling Blocks) for Alexa using Alexa Presentation Language (APL) that allows you to play in real-time, controlled entirely by voice. Building on the techniques in my last post, the gameplay area is comprised of a 8x12 grid of Pagers controlled by SetPage commands for forced animation. The biggest challenge was syncing the frontend with the backend, which was solved by abusing ControlMedia (Play/Rewind) commands (in Parallel) and listening for onPlay events for each tick of gameplay. Big thanks to Mark S. (an APL genius) for suggesting this solution!

UK Skill - US Skill

Watch the gameplay video

See the Gists

Introduction

In this post, I will talk about the Alexa (response size) and device (performance) constraints for the Falling Blocks project, share the math I used to calculate viability of the project, and cover the key elements of the solution (including gists).

During my most recent trip to Italy, I had an informal brainstorming session over dinner with my Spanish and Italian colleagues who were amused by my last post. With a mutual interest in games (classic and modern) we started discussing what other games could be APL-ified. It is with only a little shame that I admit that I literally jumped up and down with excitement (multiple times) when we stumbled upon Tetris.

I have a couple other projects in the queue, but the ability to execute on real-time voice gameplay was too tempting to pass up, so I bumped Tetris to top priority. Fast forward two weeks and my finger is hovering over the button to submit for certification.

Of the ~20 hours I spent on this project, about 8 were fighting with the frontend/backend sync challenge. After a very quick chat with Mark, I was able to completely resolve the issue in about an hour, reusing most of the code I had already written (the advantages of working with talented people).

As always, everything here could break tomorrow. Caveat emptor.

Constraints

There are two primary constraints that I had to keep in mind for this project: 1) the response size to Alexa needs to be under 24kb (~24,000 characters) and 2) the more components, the longer an Echo device takes to render and then execute commands. I found that around 300 components, my Echo Show starts to noticeably slow. Originally, my plan was to have a 12x20 grid, but I had to scale this back due to 10+ second render times (on device, more in the emulator).

For this project, I was able to trade one constraint for the other and I experimented in a few different ways before returning to my initial design:

Define grid groupings, so that one Pager controls 2+ tiles (for 3 * 3 combinations) to halve the number of SetPage commands required. This would have allowed for twice as many gameplay commands but 864 (9 * 8 * 12) components was too many for the device (and completely froze the emulator).
Pre-calculate coordinates and sizes and render the tiles in absolute positions resulting in a much larger APL Document. Despite no calculations for dimensions being required at runtime (relative/absolute percentages), I saw a huge performance hit. In the future, I may try to create more simple benchmarks to figure out the root cause because this seems backwards to me.
Reload the document. Even with the goal of a zero-load gameplay, this crossed my mind as I was trying to figure out how to reset the board quickly. The updated document and data source (without optimization) were about the same total size as the commands required to reset all the Pagers. This was not an option: load time killed this.
Calculate the delta after a voice commands and change only those tiles. With a limited response size, I wanted to minimize the number of Pagers I sent SetPage commands. There were some syncing issues (even after delta calculation) leading to ghost tiles. It was more prudent to completely reset the board once after a voice command and then do delta calculations for each tick.

Quick Math

(Please check my math!)

Assuming we follow the original design (which I did), one tile equals one Pager. For the below, active is defined as: the tiles that comprise the falling shape; inactive: the settled structure tiles (former active), and empty: unused grid tiles.

Our grid is going to be of the dimensions X by Y. Each grid tile has 3 states (active, inactive, empty) displayed as Frames within a parent Pager component. Our number of grid tile components is: 4XY (Pager plus Frames). With our Container components used to wrap them, we have an additional Y + 1 (one Container to hold everything and one Container per grid row).

With this design, our minimum number of components is therefore: 4XY + Y + 1.

Our number of SetPage commands will be broken down into: 1 full reset (XY) to all Pagers and one delta calculation per tick. The delta is a minimum of 4 SetPage commands (the O-Shape moving) and a maximum of 4X (clearing 4 rows with the I-Shape). More about shapes.

The minimum number of characters I found per SetPage command is 47 (where the componentId is a two-character grid location):

{"type":"SetPage","componentId":"aa","value":0}

Ignoring all the other response data required (session attributes, speak/reprompt, other commands), that leaves us with the following equation:

Minimum Number of Commands (I-Shape) = (24000/47 - XY)/4X

Maximum Number of Commands (O-Shape) = (24000/47 - XY)/4

As stated above in Constraints, if you can decrease X or Y (by either reducing grid size or by spanning multiple tiles with a single Pager), you can greatly increase the number of SetPage commands you can send. Therefore, lengthier autonomous gameplay is inversely related to higher device performance requirements.

In the case of my 8x12 grid, we have:

Min Components: 397

Min Commands: 12

Max Commands: 103

An intelligent solution could calculate available space left after each tick to see if there’s room for another but this calculation would need to be done as the last step. For this project, I picked a fixed number with enough headroom (27 minus user commands, so about two full shape falls), since I have other content in the response payload.

Solution

Since I already covered the basics in my last post, I will only cover here what is unique to this project.

Gists here.

Audio Marker (Hack)

Despite seeming like there should be, there is almost no way to track what has already happened to your components on the device when you send multiple commands with staggered delays. The request payload appears to have status information on component children, visibility, and state for each component but I did not see these being updated reliably. Pager and Sequence do not have an onChange event. I originally tried to do the math to estimate latency and calculate the current state, but the game was not playable (or at least not enjoyable).

In my conversation with Mark, he suggested I find a way to mark each batched set of commands (tick using Parallel commands). The only commands he knew of that sent a callback to the Lambda were the Media Commands. Long story short, onEnd wasn’t reliable enough (due to timing, it didn’t always fire), but onPlay fired every time.

Listening for the onPlay UserEvent on the Lambda, I was able to shift my server-side state queue and keep track of the current gameplay position.

Event Handler Gist

Layout Gist

InitialPage Attribute of Pager

This is one of the “read the documentation” moments for me. I was trying to figure out how to display the current game state in the best way on initial load with minimal performance overhead (add a firstItem attribute, send a set of reset SetPage commands, etc.). Nope, what I wanted was initialPage.

By passing in a 2D array which conveyed both X by Y dimensions and the starting values, I was able to calculate/display the initialPage (starting tile state) with no additional overhead.

Values Gist

Layout Gist

Multiple Moves Per Interaction

More as a quality of life implementation, I allow my users to send multiple commands at a time. With the most permutations, I get utterances that look like this (see gist):

"{DIRECTION_A} {DIRECTION_B} {DIRECTION_C} {DIRECTION_D} {DIRECTION_E} {DIRECTION_F} {DIRECTION_G} {DIRECTION_H} {DIRECTION_I} {DIRECTION_J}"

Instead of needing to say each “Right” individually (“Alexa, right. Alexa, right, Alexa, right”), the user can say all three (“Alexa, right, right, right.”). In testing, this greatly reduced friction. If I wanted to make the game more difficult, I would figure out the right timing to interleave these between the normal fall speed movements. I think the game is difficult enough (and fun enough) as it is for a proof of concept.

Intent Gist

Saving to Persistent vs Session Attributes

Most of the time it can make sense to save everything in your Session attributes to your Persistent storage and vice versa. If you do so at the start/finish of a session instead of each turn, you save yourself IO latency and cost.

Because I had to pre-calculate the future possible states, and because I was already constrained by my response size, I decided to split my storage logic. All game state queue information is in Persistent storage 100% of the time, which means I hit my DynamoDB for every conversation turn (and every gameplay tick). This is stored as a queue of states, one per gameplay tick. I then use variables in the Session attributes to extract what I need from the persisted information. This is the first legitimate need I’ve had in my own projects to split my user data in this way.

They latency cost of reading from and saving to DynamoDB isn’t high in the grand scheme of things (generally under 150ms). In gameplay, it’s not explicitly noticeable, but everything adds up.

Conclusion

One of my colleagues asked me how I have time and energy for projects like this. It’s a great question. Pushing the boundaries of what technology can do excites me, the creation process relaxes me, and completing projects/milestones is satisfying. There were parts of this project that were painful and frustrating, but that’s part of learning and experimentation (worth it in the end).

Like 2048, I see this game as another compelling reason for voice-controlled devices with screens. The possibilities and experience will only continue to get better as technology improves and developers become more creative.

If you’re new to Alexa or APL, I would not suggest starting with a project like this. In addition to the complexities, APL is still limited in functionality; Pager/Sequence animation is a hack, UI audio markers are a hack. I did a lot of math and design before I wrote my first line of code, even then I wasn’t sure the project was even practical.

That said, if you’re like me, go out there and see what you can break and then what you can create out of those broken pieces. Enjoy!

Disclaimer

This is me representing my personal night/weekend projects and has no association with my employer. All methods discussed here are based on publicly available features, just some used in unintended ways.

Everything here could break tomorrow. Caveat emptor.

https://youtu.be/XdgwUCVAOV4

Alexa Game and Animation Experiments (APL)

Will Blaschko — Fri, 17 May 2019 22:03:49 +0000

The purpose of this article is to share what I’ve done over the last two months by using (and misusing) Alexa Presentation Language (APL) to create visual games for Alexa. I will highlight animation, as I’ve spent the majority of my time making that work.

Background

I’ve been exploring the Alexa game space for fun and just launched my first personal Alexa game skill: a voice-controlled version of the popular game 2048 (UK version, US version, others coming soon) built with screen devices in mind (Spot, Show, FireTV). This game has a million clones across mobile and web, but I didn’t find an Alexa version. Demo video.

Although I have used APL previously, for these games, I dove deep into the limits of functionality, response size, and performance. One of the big blockers I ran into was the lack of game-type animation available through existing APL components and commands.

Faced with this challenge, I figured out how to make animation work (X and/or Y axis movement for world and objects; sprite-based animation). If you’ve seen some of my other posts or followed some of my projects, you know I don’t let convention or intended use limit me. For this, I broke from APL convention to get the results I needed.

TLDR; What’s Possible?

If you don’t care about my projects and want to get to the meat of the content, here you go! There are likely more possibilities, but here’s what I’ve made work so far:

X or Y axis world animation: using a Sequence (or nested Sequences) you can break up the background into tiles (in pixels or percents of viewport) and use Scroll commands Sequentially to move the world. Example background animation here (video).

Gist here

X or Y axis item animation: using a Sequence with small, hidden items, you can animate the primary item in the Sequence around the screen by using Scroll commands Sequentially. I used 1vh square empty Frames to accomplish this. Example item animation here (video).

Gist here

Frame-based sprite animation: using a Pager you can approximate a sprite animation by flipping between the different pages using SetPage commands Sequentially. Example sprite animation here - “bird” (video).

Gist here

200+ element APL documents: By being smart about reuse of Layouts, you can fit hundreds of APL components into the response sent to Alexa. This allows for complex scenes to be built and minimizes RenderDocument directives.

Gist here

Combine these together and you could get a pretty decent top-down 16-bit style game.

Projects

I will touch on 2048 and two other projects I currently have underway (both of which may never be seen by the public). I wanted to experiment with a variety of game types so I created a board game, a first person RPG, and a 2D side scroller/platformer. My personal goals for all these projects were 1) use RenderDocument directives only when absolutely necessary (a scene change) 2) animate everything possible and 3) create games that were enjoyable and sticky.

I will keep the below relatively code-light, see the gists above or the related videos for more details.

2048 (Board Game)

As the only skill for this piece launched so far, this one is the most polished and playable. I’d suggest trying it out to get a feel for how it works. You can play it on headless (no screen) devices, but it’s not the best experience. Video here.

Challenges

There were two primary challenges for this project: 1) adapting to different screen sizes and 2) rendering and controlling all the tiles and keeping it within performance and response size limitations.

1) I solved through some ternary operations based on viewport shape and width/height ratio. These calculations are used to place the X/Y grid of objects (Containers within a Container) in the correct location based on screen size/shape.

${@viewportShape == @viewportShapeRound ? viewport.width / 6 : (viewport.width > viewport.height ? viewport.height / 4 : viewport.width / 4)}

2) I created backgrounds and pagers for each tile (4x4 = 16) and created a custom Tile layout, which eliminated repeated code and greatly reduced response size. By simplifying these Tiles, I also reduced the number of rendered elements.

...{

    "type": "Tile",

    "name": "pos1"

}...

Goals

The only time the game receives a new RenderDocument is if you customize the tiles or background (“Alexa, customize”). Everything else is controlled by Pager commands, including the scoreboard. In theory, both the tiles and scoreboard should animate, I may play with the timing of the commands a little to see if I can get this to work. Stickiness: I personally quite enjoy this game and played it well beyond what was required to test it myself.

RPG Name TBD (First Person RPG)

My long-term goal for this game is to create a procedurally-generated RPG game where you fight monsters, complete quests, and gather treasure. If you want to play a game that has already done this well for voice (in my opinion), check out Skyrim.

I wanted visuals and animation. What’s the point of “taking a mighty swing at the troll” if all you have is a static image on the screen? Exactly. See this video for an early POC.

Unrelated, for this project, I needed to be able to return a sequence of Speak responses to Alexa, so I created a modified RequestHandler (gist) to be used with a RequestInterceptor.

Challenges

1) How can I animate the weapon and the monster and 2) create a layered environment that gives a sense of depth?

For #1, it was time to break Sequences. As the only moving component (Pagers are limited in that they only display one item), I had to figure out how to control the speed and direction of movement granularly. The solution was invisible components, pixels or percents of view height/width in size, that I could Scroll to using a timed Sequential command. With this, I have dancing creatures, attacking weapons, and about any other animation I could want on the X or Y axis.

For #2, this was easily solved with layered elements inside a container using position: absolute (more here). This allowed me to create multiple foregrounds and backgrounds.

Goals

I want to see if it’s possible to use less RenderDocument directives, currently one is shown every scene change (movement), or when a monster dies. I could likely use Pagers to switch between the different backgrounds/foregrounds, but the load isn’t as abrupt here as it was for 2048. The monsters and attacks are animated, as I develop this more, I expect to add additional animations. This game needs a lot of polish but both the APL and voice experience have a solid base. With more, better content and direction (and a better name), this game has potential to be sticky on both screen and voice-only devices.

Flappy Alexa (2D Side Scroller/Platformer)

Flappy Bird came and went in popularity but after the success with animating the RPG game, I wondered if it would be possible to use the same strategy to create side-scrollers and platformers. Short answer: yes, but only if you want to invest the time. I was able to throw together a quick Flappy Alexa POC in about 4 hours, it’s about as rough as you’d expect and probably won’t get much further attention. The video shows blocks (use your imagination) simulating the basic gameplay, all rendered and, theoretically, playable on a real device.

With the right design and math, there’s no reason this couldn’t be used to create a 2D map or platformer that automatically animates as the character “moves” around. The character could also move on the X or Y axis as required for storytelling using the same technique. 16-bit tile based games, anyone? Turn-based would be even easier.

The bird itself is a Pager component, if you watch the video, you see the transition. This same behavior could be used with an animation sprite to do rough frame-based animation.

Challenges

I quickly hit the hard limit for the Alexa response size between the map information and the animation commands. If this were not a pre-rendered sequence, you’d have more flexibility in design and animation. The payload is the full map information (we only see about 30% of it in the video) and about 500 distinct commands to animate the map and the bird. If this were real-time, you’d only need ~100 commands between user interactions for most use cases.

Goals

There are zero RenderDocument directives after the initial map load, everything is done through Pager or Sequence commands. Everything is animated. And the game could be sticky, but I wouldn’t suggest a fast twitch game like this using voice, maybe with Echo Buttons. Make it turn-based with a good voice experience and you have something here.

Future Projects

I have a couple fun ideas in mind. Hopefully, they become part of the Alexa skill selection in the not-too-distant future.

One game is currently in scoping and design will be a best-on-screen experience. The challenge here is I’ll need ~1000 APL elements displayed simultaneously if I want to be able to animate and avoid the RenderDocument directive load disruption. I’ve developed a few POCs that make me believe this is possible on real devices despite overwhelming the emulator.

The other game is still in ideation but will be a scripted, themed exploration game. This will be turn-based, with an animated X/Y map where the player will be able to interact with on-screen content through either voice or touch. This will also have a rich voice experience, so those of you without screen devices will also be able to enjoy.

Conclusion

Fellow tinkerers, I encourage you to go out and see how you can break things in new and exciting ways, beyond the intended use, to get the end experience you want. Let me know if you’ve been able to tame APL to create something new and different.

In my opinion, the end result created using these techniques is a more smooth and engaging experience for users. In a world where Alexa becomes available on more and more devices with screens, a visual interface needs to be as polished as the voice experience. Developers need to take risks and try something new.

Word of warning, this is all based on the public, beta version of APL as of writing, any hacks could stop working tomorrow. Also, remember to use adaptive or responsive design for all screen shapes and sizes.

Disclaimer

Alexa APL Performance Benchmark

Will Blaschko — Fri, 17 May 2019 21:43:48 +0000

(The above is a layout used to stress-test APL)

Rhetorical question: have you ever wanted to benchmark the load times for different Alexa APL documents? To be able to A/B/C/… test them to understand which implementation would give the quickest render time for your customers?

Hopefully the answer is: “Absolutely!”

Here’s some code you can use to benchmark both in the browser and on a physical device.

Requirements:

NodeJS script configured (GIST)
Local folder where you can persist Puppeteer user data (login cookies)
Fire TV Stick with Alexa (optional)
ADB installed (if benchmarking on Fire TV)

How it works:
Puppeteer opens the Displays Simulator and renders the layouts/data repeatedly. Each time the browser renders the layout, a command is also (optionally) sent to the Fire TV through ADB. Listening to web console logs (simulator) and ADB logs (Fire TV), the script calculates the time of each render.

How to Use:

Configure index.js to meet your needs, currently everything is defined in code, but it could be modified to be command line arguments.
If using Fire TV, adb connect … to the target device.
Copy a valid Chromium user directory to an accessible folder and use as a parameter (or change benchmark.js to log in automatically).

What This Is:
This is meant to be a proof-of-concept experiment as I optimize my own documents. I hope someone can use this as a starting point for a more comprehensive tool.

Files:
index.js - Launch benchmark and output performance
benchmark.js - Benchmark library
package.json - Node package
layout_base.json - Default layout used to reset the simulator
layout_example_*.json - Different layouts to benchmark
data_example.json - Data used for all layouts

Sample Output:
Running benchmark for Layout 0 (Browser)
Benchmark ran 100 times.
Average render: 1723ms
Max render: 3989ms
Min render: 236ms

Running benchmark for Layout 1 (Browser)
Benchmark ran 100 times.
Average render: 1786ms
Max render: 3326ms
Min render: 224ms

DEV Community: Will Blaschko

Falling Blocks - "Tetris"​ On Alexa (APL)

TL;DR

Introduction

Constraints

Quick Math

Solution

Audio Marker (Hack)

InitialPage Attribute of Pager

Multiple Moves Per Interaction

Saving to Persistent vs Session Attributes

Conclusion

Disclaimer

Alexa Game and Animation Experiments (APL)

Background

TLDR; What’s Possible?

Projects

2048 (Board Game)

Challenges

Goals

RPG Name TBD (First Person RPG)

Challenges

Goals

Flappy Alexa (2D Side Scroller/Platformer)

Challenges

Goals

Future Projects

Conclusion

Disclaimer

Alexa APL Performance Benchmark

Falling Blocks - "Tetris" On Alexa (APL)