The purpose of this article is to share what I’ve done over the last two months by using (and misusing) Alexa Presentation Language (APL) to create visual games for Alexa. I will highlight animation, as I’ve spent the majority of my time making that work.
Background
I’ve been exploring the Alexa game space for fun and just launched my first personal Alexa game skill: a voice-controlled version of the popular game 2048 (UK version, US version, others coming soon) built with screen devices in mind (Spot, Show, FireTV). This game has a million clones across mobile and web, but I didn’t find an Alexa version. Demo video.
Although I have used APL previously, for these games, I dove deep into the limits of functionality, response size, and performance. One of the big blockers I ran into was the lack of game-type animation available through existing APL components and commands.
Faced with this challenge, I figured out how to make animation work (X and/or Y axis movement for world and objects; sprite-based animation). If you’ve seen some of my other posts or followed some of my projects, you know I don’t let convention or intended use limit me. For this, I broke from APL convention to get the results I needed.
TLDR; What’s Possible?
If you don’t care about my projects and want to get to the meat of the content, here you go! There are likely more possibilities, but here’s what I’ve made work so far:
X or Y axis world animation: using a Sequence (or nested Sequences) you can break up the background into tiles (in pixels or percents of viewport) and use Scroll commands Sequentially to move the world. Example background animation here (video).
X or Y axis item animation: using a Sequence with small, hidden items, you can animate the primary item in the Sequence around the screen by using Scroll commands Sequentially. I used 1vh square empty Frames to accomplish this. Example item animation here (video).
Frame-based sprite animation: using a Pager you can approximate a sprite animation by flipping between the different pages using SetPage commands Sequentially. Example sprite animation here - “bird” (video).
200+ element APL documents: By being smart about reuse of Layouts, you can fit hundreds of APL components into the response sent to Alexa. This allows for complex scenes to be built and minimizes RenderDocument directives.
Combine these together and you could get a pretty decent top-down 16-bit style game.
Projects
I will touch on 2048 and two other projects I currently have underway (both of which may never be seen by the public). I wanted to experiment with a variety of game types so I created a board game, a first person RPG, and a 2D side scroller/platformer. My personal goals for all these projects were 1) use RenderDocument directives only when absolutely necessary (a scene change) 2) animate everything possible and 3) create games that were enjoyable and sticky.
I will keep the below relatively code-light, see the gists above or the related videos for more details.
2048 (Board Game)
As the only skill for this piece launched so far, this one is the most polished and playable. I’d suggest trying it out to get a feel for how it works. You can play it on headless (no screen) devices, but it’s not the best experience. Video here.
Challenges
There were two primary challenges for this project: 1) adapting to different screen sizes and 2) rendering and controlling all the tiles and keeping it within performance and response size limitations.
1) I solved through some ternary operations based on viewport shape and width/height ratio. These calculations are used to place the X/Y grid of objects (Containers within a Container) in the correct location based on screen size/shape.
${@viewportShape == @viewportShapeRound ? viewport.width / 6 : (viewport.width > viewport.height ? viewport.height / 4 : viewport.width / 4)}
2) I created backgrounds and pagers for each tile (4x4 = 16) and created a custom Tile layout, which eliminated repeated code and greatly reduced response size. By simplifying these Tiles, I also reduced the number of rendered elements.
...{
"type": "Tile",
"name": "pos1"
}...
Goals
The only time the game receives a new RenderDocument is if you customize the tiles or background (“Alexa, customize”). Everything else is controlled by Pager commands, including the scoreboard. In theory, both the tiles and scoreboard should animate, I may play with the timing of the commands a little to see if I can get this to work. Stickiness: I personally quite enjoy this game and played it well beyond what was required to test it myself.
RPG Name TBD (First Person RPG)
My long-term goal for this game is to create a procedurally-generated RPG game where you fight monsters, complete quests, and gather treasure. If you want to play a game that has already done this well for voice (in my opinion), check out Skyrim.
I wanted visuals and animation. What’s the point of “taking a mighty swing at the troll” if all you have is a static image on the screen? Exactly. See this video for an early POC.
Unrelated, for this project, I needed to be able to return a sequence of Speak responses to Alexa, so I created a modified RequestHandler (gist) to be used with a RequestInterceptor.
Challenges
1) How can I animate the weapon and the monster and 2) create a layered environment that gives a sense of depth?
For #1, it was time to break Sequences. As the only moving component (Pagers are limited in that they only display one item), I had to figure out how to control the speed and direction of movement granularly. The solution was invisible components, pixels or percents of view height/width in size, that I could Scroll to using a timed Sequential command. With this, I have dancing creatures, attacking weapons, and about any other animation I could want on the X or Y axis.
For #2, this was easily solved with layered elements inside a container using position: absolute (more here). This allowed me to create multiple foregrounds and backgrounds.
Goals
I want to see if it’s possible to use less RenderDocument directives, currently one is shown every scene change (movement), or when a monster dies. I could likely use Pagers to switch between the different backgrounds/foregrounds, but the load isn’t as abrupt here as it was for 2048. The monsters and attacks are animated, as I develop this more, I expect to add additional animations. This game needs a lot of polish but both the APL and voice experience have a solid base. With more, better content and direction (and a better name), this game has potential to be sticky on both screen and voice-only devices.
Flappy Alexa (2D Side Scroller/Platformer)
Flappy Bird came and went in popularity but after the success with animating the RPG game, I wondered if it would be possible to use the same strategy to create side-scrollers and platformers. Short answer: yes, but only if you want to invest the time. I was able to throw together a quick Flappy Alexa POC in about 4 hours, it’s about as rough as you’d expect and probably won’t get much further attention. The video shows blocks (use your imagination) simulating the basic gameplay, all rendered and, theoretically, playable on a real device.
With the right design and math, there’s no reason this couldn’t be used to create a 2D map or platformer that automatically animates as the character “moves” around. The character could also move on the X or Y axis as required for storytelling using the same technique. 16-bit tile based games, anyone? Turn-based would be even easier.
The bird itself is a Pager component, if you watch the video, you see the transition. This same behavior could be used with an animation sprite to do rough frame-based animation.
Challenges
I quickly hit the hard limit for the Alexa response size between the map information and the animation commands. If this were not a pre-rendered sequence, you’d have more flexibility in design and animation. The payload is the full map information (we only see about 30% of it in the video) and about 500 distinct commands to animate the map and the bird. If this were real-time, you’d only need ~100 commands between user interactions for most use cases.
Goals
There are zero RenderDocument directives after the initial map load, everything is done through Pager or Sequence commands. Everything is animated. And the game could be sticky, but I wouldn’t suggest a fast twitch game like this using voice, maybe with Echo Buttons. Make it turn-based with a good voice experience and you have something here.
Future Projects
I have a couple fun ideas in mind. Hopefully, they become part of the Alexa skill selection in the not-too-distant future.
One game is currently in scoping and design will be a best-on-screen experience. The challenge here is I’ll need ~1000 APL elements displayed simultaneously if I want to be able to animate and avoid the RenderDocument directive load disruption. I’ve developed a few POCs that make me believe this is possible on real devices despite overwhelming the emulator.
The other game is still in ideation but will be a scripted, themed exploration game. This will be turn-based, with an animated X/Y map where the player will be able to interact with on-screen content through either voice or touch. This will also have a rich voice experience, so those of you without screen devices will also be able to enjoy.
Conclusion
Fellow tinkerers, I encourage you to go out and see how you can break things in new and exciting ways, beyond the intended use, to get the end experience you want. Let me know if you’ve been able to tame APL to create something new and different.
In my opinion, the end result created using these techniques is a more smooth and engaging experience for users. In a world where Alexa becomes available on more and more devices with screens, a visual interface needs to be as polished as the voice experience. Developers need to take risks and try something new.
Word of warning, this is all based on the public, beta version of APL as of writing, any hacks could stop working tomorrow. Also, remember to use adaptive or responsive design for all screen shapes and sizes.
Disclaimer
This is me representing my personal night/weekend projects and has no association with my employer. All methods discussed here are based on publicly available features, just some used in unintended ways.
Top comments (0)