DEV Community: Lee Martin

State of Spotify Web API Report 2025

Lee Martin — Tue, 14 Oct 2025 12:31:37 +0000

Understanding Spotify's new API restrictions and finding practical alternatives.

Intro

On May 15, 2025, Spotify updated the criteria for Web API extended access, the process which determines if a Spotify app built on the Web API can come out of development mode (limited to 25 test users) and be offered publicly to a larger group of users. If you do not meet this steep new criteria, the Web API is now mostly limited to experimentation and personal use. As someone who has developed many artist campaigns and software projects using the Web API, I wanted to take a moment to talk about these changes. Naturally, I thought about complaining but... that doesn't really help anyone. Instead, I've decided to speak to how this platform has inspired my work in the past and share potential alternatives as we look to the future. This report is specifically about the Spotify Web API, Web Playback SDK, and Embeds. It does not cover any other Spotify Platform products. In addition to suggestions for you, I have a few suggestions for Spotify and other companies offering Music APIs. Anyway, I hope you enjoy this report. Please share it if you find it useful and if you have any feedback, please get in touch.

— Lee Martin

Disclaimer: I do not work for Spotify and all opinions and suggestions expressed are my own.

Brief History

We, the computer geeks at Spotify, also love cool software. To that end, we’re trying to open up our platform to enable third party application development on top of the Spotify platform. These pages are our first baby steps toward that goal.

I had to use the Way Back Machine to find this quote on the Spotify Developer site from 2009. Through my participation in Music Hack Day, I had the chance to meet some of the people who would eventually build this platform. I mention this to acknowledge that this API's existence was never guaranteed, it happened because people believed in API enablement and inspiration. The reason we mourn these changes is because it was really well done.

My first major project using the Spotify Web API was a playlist generator for the band Khruangbin. The idea behind the app was to help users generate playlists for an upcoming flight. Khruangbin provided a pool of funky tracks and the user provided their flight details and preferences. Coffee or tea? Aisle or window? These preferences were mapped to Spotify audio features and helped determine which songs from the pool should be used. For example, coffee drinkers got high energy tracks, while tea drinkers got low energy tracks. It was fun, useful, and only something the Spotify Web API could pull off.

Since then, I've developed numerous activations and apps using the Web API and have had a lot of fun doing so. In fact, a case study from a Shania Twain campaign back in 2023 is the last thing the Spotify Developer platform's Twitter account ever shared. Things were looking up then but the times, they are a-changin'.

New Requirements

In order to transition an app from development mode to extended quota mode, developers must submit their application for review. Previously, this review process was open to any individual but now, Spotify only accepts applications from established business entities, whose implementation meets a very specific set of requirements:

The business entity must be a legally registered business or organization.
The business must be operating an active and launched service.
The service must be maintaining a minimum of 250,000 monthly active users.
The service must be available in key Spotify markets
The service must prove commercial viability
The service must adhere to Terms

Anything jump out at you? 250,000 MAUs. This means that you need to have a massively successful app before seeking an integration with the Spotify Web API. In plain terms, that means your service should be successful without Spotify and only then should you be aiming to integrate Spotify with it. In my opinion, if you're thinking about building a new service which relies on Spotify, you should think twice. Spotify continues:

Eligibility criteria may change over time. Spotify makes no promise or guarantee that extended access requests will get approved.

We strongly advise that you start integrating the Web API into an application intended for an audience of more than 25 authenticated Spotify users only after Spotify has agreed to this application in writing.

Spotify doesn't mince words. They have final say on approvals. Read the Developer Policy thoroughly before making any investment.

I've built Spotify powered activations for major artists but even they can't prove 250k MAU. Under these requirements, I would no longer be able to build Spotify Web API powered artist campaigns which require user authentication. And therein lies the issue: Spotify is ending support for those apps that live somewhere between a personal project and an enterprise project.

I should mention that if your app doesn't require user authentication, you should be able to continue to use some aspects of the Spotify Web API using client credentials but you should still be aware of the development mode limitations.

Feature Alternatives

If your application requires user authentication and is affected by the new requirements, you'll need to get creative or look elsewhere. Let me walk through key Web API features, how they inspired me, and what you can do instead.

Search

Web API search requires authentication. That means if you're looking for a track, album, artist, or playlist, you'll need to be authenticated. Since you won't be able to search on behalf of a user, you can look towards using client credentials instead. In order to combat development mode rate limits, you'll also want to look at caching data.

One viable alternative for music searching is the ancient iTunes Search API. This API does not require any authentication and will return a lot of the same basic metadata you'd expect from Spotify. In addition, most tracks will have an ISRC, which you can use to cross-reference with the Spotify platform and other music APIs. Also, Apple Music still provides audio previews which Spotify deprecated last year.

I once used the Web API search endpoint to develop a search feature into an early 2000s inspired mixtape generator for the Plain White T's. By adjusting the year parameter to 2000-2010, the search would only return tracks from this era, leading to very nostalgic results.

Likes and Follows

Authentication is required to like a track or follow an artist using the Web API. Without it, your best recourse is simply linking to the associated pages on Spotify. Previously, Spotify had a follow button which was pre-authenticated but they deprecated it back in 2021. Since the Web API was suggested as an alternative, Spotify should reconsider embed solutions for these touchpoints.

Audio Features

I wrote about the deprecation of Audio Features last year. This was a massive blow to what made the Spotify Web API truly unique and now we know it was the sign of things to come. I used Audio Features for so many things including a Sadboi Detector I developed for Illenium. That application analyzed a user's listening history or a provided playlist and determined how sad it was by examining each track's Valence. Spotify deprecated this feature, alongside Recommendations and Related Artists, because they were concerned developers would use it to train AI models.

While I haven't done much building around audio classification this year, I've been eyeing Cyanite as a potential alternative to Audio Features. Cyanite's business is analyzing the emotions of music and offers all of the measurements the Spotify Web API previously provided alongside even more data. Depending on your use case, audio classification in general is worth exploring as it becomes more mainstream with AI.

Playlist Creation

I already mentioned one playlist generator for Khruangbin but I created multiple with the Web API, including another for Khruangbin during the pandemic called Shelter in Space. Playlist generators work particularly well when an artist or curator can bring a particular vibe to the pool of potential songs.

Since creating a playlist and adding tracks requires authentication, we need to look elsewhere for a solution. Depending on your use case, you may be able to manually create the playlist or a series of playlists and serve them up thematically. The Apple Music API is still very good at creating playlists and that can be accomplished using MusicKit on the web.

If still you want to create an application that builds playlists dynamically and saves them to Spotify, you may need a 3rd party solution. I haven't used or tested TuneMyMusic, but this service allows users to create a playlist from a text file. If your service can generate a text file of tracks, your users could import it via a 3rd party solution (or recreate it manually). I kinda recall building a text file export or playlist tracks into one of my client apps. Maybe it was the Tycho Weather playlist generator?

This raises an interesting point: there may be existing Spotify apps with extended quota access that can bridge the gap. However, thoroughly vet any 3rd party service before sending your users there.

Web Playback

Being a web developer for over two decades means I was able to see a lot of new features adopted by browsers and as soon as feature parity was met, I was able to roll them into my client projects. I felt the same way about the Music APIs and I kept a little spreadsheet which tracked when Spotify, Apple Music, and Deezer inched closer to providing full track streaming via their API to authenticated users. It was a great day when all of these finally turned green.

With this functionality, I built a Spotify and Apple Music powered A/B player for R.E.M., which allowed fans to jump between the original and new mix of Monster seamlessly. I also built an app called Listening Party which allows artists to throw parties around their music.

Deezer has deprecated their custom player library and with these new Spotify requirements, that leaves Apple Music as the only viable option of the three for authenticated playback. However, you can also look into SoundCloud or just rolling something custom.

Spotify also has an embed (potentially with an authenticated Spotify user) and a complimentary iFrame API, similar to YouTube. However, unlike YouTube, the iFrame API isn't very customizable. With the Web Playback SDK now being out of reach, I recommend Spotify consider a chromeless iFrame API for more customizable integrations.

Listening History

One of the most popular features of the Spotify Web API is the ability to read the users top tracks or artists in three different time ranges: last 4 weeks, last 6 months, and all time. It's sort of a simplified version of Spotify Wrapped. I used this feature to generate Waffle House receipts for Jonas Brothers, mail postcards to Noah Kahan fans, and grow a virtual plant for Joy Oladokun, among countless other activations. Not having the ability to integrate this feature into an application that has more than 25 users is hard to stomach but there are ways forward.

Apple Music offers an alternative. While not as detailed as Spotify's data, you can check what's in heavy rotation on a user's account. I also noticed a new endpoint called replay data which summarizes the data from Apple Music Replay, their answer to Spotify Wrapped. That's something the Spotify Web API never provided!

What we really need is a platform-agnostic way to keep track of plays. Did someone say scrobbling? God, I miss last.fm. By the way, did you know you can export your extended streaming history from Spotify? It takes up to 30 days but you can download your data in case you ever switch services or simply want to build something interesting around your listening history.

Future Possibilities

Unless Spotify changes their mind, we'll need to adapt to these limitations. I haven't developed any Spotify Web API powered apps this year because my clients didn't ask for any. Sign of the times? However, I did build an activation for Lil Poppa which suggests a song off his new album based on the results of an AI-powered therapy session. On this app we simply linked to the song on Spotify. No integration needed. I wrote a little bit about this and using AI to bridge the gaps of these missing features.

I've been using my YouTube channel to research new technologies like AI and automation for creative marketing in music. While researching, I noticed something interesting: Spotify has granted extended quota access to some AI and agentic platforms. This means you can develop agentic flows that interact with your Spotify account. This doesn't necessarily solve the issues outlined in this report but it may offer a glimpse at a near future where Spotify and other DSPs are integrated with these platforms, allowing for custom applications.

Check out my video introduction to AI Agents to see how I use Spotify in a custom flow, and take a look at OpenAI's new Apps SDK which integrates Spotify. I'll be keeping an eye on these developments so please subscribe if you're interested.

Conclusion

Relying on and investing ourselves into other platforms to achieve things online has always been a risky endeavor. Platforms change, adjust their terms, shut down, and all you're left with is whatever value you were able to extract and make defensible elsewhere. To be a web developer is to be malleable. You need to be constantly looking for new ways to achieve solutions to your problems. These changes by Spotify might be hard to process at the moment, but I see them as the latest in a series of hard lessons about web development. Instead of letting it get me down, I embrace the web's evolution and accept the challenge of looking for new solutions. I hope this report helps you, and I can't wait to see what you build. Thanks for reading.

About

Lee Martin is a creative developer who has largely worked in music for the last two decades. He previously worked at Silva Artist Management, SoundCloud, and Songkick. Currently he is a freelancer who builds creative marketing campaigns for some of the biggest and smallest names in music. In addition, he maintains several software projects including Turn, Vinyl Mockup, and Listening Party. He publishes a newsletter called Netmaker and is publicly exploring the world of AI/Automation on his YouTube channel.

Developing The Cosmic Selector Jukebox for Lord Huron

Lee Martin — Fri, 18 Jul 2025 12:29:47 +0000

With our Cosmic Coins produced and in fans’ hands, it was time to turn our focus to The Cosmic Selector jukebox. Just to recap, we created physical NFC powered coins and it was our intention to let fans spend these coins to play clips from Lord Huron’s new album, The Cosmic Selector Vol. 1, on a virtual jukebox. This jukebox would live at cosmicselector.com and similar to a jukebox in a bar, any fan currently on the site could hear what was being played. In addition to creating a visual for the jukebox itself, this work would include some bespoke database design and determining how the queue and audio playback would function. Let’s start by taking a look at the Cosmic Selector visual.

3D Modeling

There was a goal to have the final visual of the jukebox exist as a sort of CCTV visual and there were many ways to get there. First, we had some existing teaser videos of the jukebox in various locations and we also had a photo shoot of the jukebox in various locales. However, both of these assets were flat 2D and that doesn’t pair very nicely with the responsive web, nor does it leave much room for dynamic choices. In my mind, if I could somehow create a 3D model of the jukebox, I would then be able to better control it dynamically and place it into various 3D environments. It’s just a couple of boxy lines, how hard could it be? Well, it turned out to be VERY hard to model.

The Cosmic Selector’s foundation is a Wurlitzer 2710 from 1963. Yes, the 60s. That means there are some wild angles to it which makes understanding the shape and scale of it very difficult. At one point, I was tempted to buy the actual jukebox on Ebay but it was pickup only, and in Florida. After a few false starts in Blender, I ended up purchasing the original Wurlitzer 2710 manual (instead of the whole jukebox,) which included a schematic of the jukebox, and slowly started sketching the overall shape of it in my sketchbook. I also referenced many photos of the original jukebox and the band’s adjustments. After spending a lot of time with all these assets, I built up the confidence to get back into Blender and just slowly steered the design into the model you see above. I wasn’t trying to get it super detailed. In fact, I wanted the model footprint to be quite small (low poly) so the website continued to run quickly.
Once I had the model to the place I wanted, I started working up the textures. Most of the jukebox is reflective metal, which you can accomplish in 3D design with the right material and an environment texture. For the other elements, I hand-crafted a few textures and pulled others straight from photos of the Cosmic Selector.

Three.js Scene

With the jukebox modeled in Blender, I could now bring it into the website using Three.js. First, I exported the model in GLTF format. Then, I used the GLTFLoader to load it into my scene. However, it was void of an environment. I had all sorts of ideas of how to handle environments for this project, including using AI, which the band and I agreed to avoid. That’s when I saw this example for a groundprojected skybox and I thought maybe I could ground the jukebox in an HDRI environment. So, I grabbed a series of HDRI textures from Polyhaven and it ended up working nicely.

In order to achieve the CCTV aesthetic, I used a series of post processing passes. First, a saturation pass was used to desaturate the scene. Next, I applied a subtle blur to lower the overall quality of the image. Then I added scanlines and a rolling TV effect to make it feel like a camera broadcast with a bad signal. Finally, I sprinkled a bit of static and vignette on top to complete the visual. This drastically changed the vibe of the scene but without sacrificing the performance of the site much.

The final aspect I needed to work out was making some elements of the jukebox glow when it was playing. This was done by using another post processing pass: UnrealBloom. Through a process of selective blooming, I could make some elements of the jukebox glow and others not. In addition to this selective glow, I used Greensock to add a bit of flickering.

Database Modeling

Technically, as part of the coin work, I had already established a database for creating and storing coins. This was done using a Serverless instance and a DynamoDB database. In addition to a unique ID, we stored things like the coin color, medium, and how many plays were left on the coin. In addition to coins, we needed to store a database of songs that were playable on the jukebox. These song items included the name of the song, duration of the clip, and whether or not the song was currently visible on the jukebox’s list of selections.

The final table was the selections table. If you think of an actual jukebox, a selection is the union of a payment and a song. I am choosing to spend my coin in order to add this particular selection to the jukebox. So, in general, a selection would include a reference to the associated song and coin. That’s easy enough but it really gets complicated when you begin to establish how to connect these selections to an ever evolving queue of selections. More on that later.

Title Strips

Back on the website, I needed to visualize this list of songs in a thematic but responsive manner. I sorta knew from the beginning that I didn’t want users to be able to touch the jukebox. In my mind, the jukebox should always exist at a distance (it’s too mysterious) and we’d use an external panel to interact with it. This led to the two panel layout we landed on, which also solved some responsive UI problems. For the list of selections, I was inspired by how jukeboxes displayed songs and their associated codes. This motif is called a title strip and it typically references the A and B side of a record in the jukebox. Through a bit of careful logic, I was able to take an evolving list of songs from my database and display them as a list of title strips and their associated letter and number code. First, we needed a list of letters and numbers.

// Letters
const letters = ['A','B','C','D']

// Numbers
const numbers = ['1','2','3','4']

Then, we could establish an array of potential selector codes by combining the two.

// Selector codes
const selectorCodes = computed<string[]>(() => {
  // Initialize codes
  const codes: string[] = []

  // Loop through letters
  letters.forEach(letter => {
    // Loop through numbers
    numbers.forEach(number => {
      // Push code
      codes.push(`${letter}${number}`)

    })

  })

  // Return codes
  return codes

})

This provided us with an array of codes such as A1, A2, A3, A4, B1, etc. We could then map our jukebox songs with a potential jukebox code.

// Selector songs
const selectorSongs = computed<CosmicTitle[]>(() => {
  // Return empty if no songs
  if (!songs.value) return []

  // Map songs with associated selector code
  return songs.value?.map((song, i) => {
    return {
      songArtist: song.songArtist,
      songId: song.songId,
      songCode: selectorCodes.value[i],
      songTitle: song.songTitle
    }
  })

})

And, finally, pair off these songs as title strips which could be visualized by our app.

// Selector titles
const selectorTitles = computed(() => {
  // Return empty if no selector songs
  if (!selectorSongs.value) return []

  // Titles
  const titles = []

  // Loop through selector songs by 2
  for (let i = 0; i < selectorSongs.value.length; i += 2) {
    // Initialize pair of titles
    const pair: (CosmicTitle | null)[] = [selectorSongs.value[i]]

    // If second song exists add it, otherwise add null
    if (i + 1 < selectorSongs.value.length) {
      // Push song
      pair.push(selectorSongs.value[i + 1])

    } else {
      // Push null
      pair.push(null)

    }

    // Push pair
    titles.push(pair)

  }

  // Return titles
  return titles

})

I wrote a custom Vue component to visualize these but they are mostly just a little custom HTML element which displays the song title and associated code. When a user punches in their code to the selector, I use a little helper method to then determine which song was chosen.

// Find song by code
function findSongByCode(code: string) {
  // Find first song that matches the code
  return selectorSongs.value.find(song => song.songCode === code)

}

It’s now time to make a cosmic selection.

Making a Selection

As we determined previously, a selection is a union between a coin and a song. However, in the case of our app, we also need to determine the timing of each selection’s playback. This is because while our app will playback all selections for all users currently sitting on the site, it doesn’t do so with a real-time stream. Instead, I use a technique I developed for Listening Party, which utilizes time to determine which song should be playing when and at what position. But first, we get the coin from the database and make sure it has any remaining plays. Then, we make sure the song the user chose also exists. If both of those things are valid, we can create our new selection.

To create a new selection, we must first determine if there are any existing selections in the queue and we’re particularly interested in the last selection of the current queue because this new selection would begin when it is finished. We can query for this last selection using the DynamoDB API.

// Query selections command
const querySelectionsCommand = new QueryCommand({
  TableName: `cosmic-selector-selections`,
  IndexName: "SelectionsEndsAtIndex",
  KeyConditionExpression: "selectorId = :selectorId AND endsAt > :dateValue",
  ExpressionAttributeValues: marshall({
    ":selectorId": "1",
    ":dateValue": Date.now()
  }),
  Limit: 1,
  ScanIndexForward: false
})

Now, if an existing queue of selections doesn’t exist, we can simply determine that the new selection should start now and end at the duration of the song.

// Selection starts now
selection.startsAt = Date.now()

// Selection ends now plus the song duration
selection.endsAt = Date.now() + song.songDuration

However, if a queue of selections does exist, our new selection should instead begin at the end of the last selection of the queue and once again end at the duration of the song

// Selection starts at the end of last selection
selection.startsAt = lastSelection.endsAt

// Selection ends at the end of last selection plus the song duration
selection.endsAt = lastSelection.endsAt + song.songDuration

We can now store our new selection in the selections database and alert all users of this new selection using a Pusher integration. We’ll talk about playback in a bit but first there’s one more thing to do. We need to decrement the plays of our coin using an UpdateItemCommand.

// Update coin command
const updateCoinCommand = new UpdateItemCommand({
  TableName: `cosmic-selector-coins`,
  Key: marshall({
    coinId: coin.coinId
  }),
  UpdateExpression: "SET updatedAt = :updatedAt, coinPlays = coinPlays - :decrement",
  ExpressionAttributeValues: marshall({
    ":decrement": 1,
    ":updatedAt": Date.now()
   }),
  ReturnValues: "ALL_NEW"
})

And, that’s it for new selections.

Oh, wait, there’s one very important thing. Say there are 50 people at a bar and all of them want to play the jukebox. How would this manifest itself? They would line up and each would get a chance to make a selection. Now, I didn’t want users to have to queue up to make selections, but I wanted to make sure these selections were created one at a time so the queue wasn’t corrupted. We can accomplish this with Lambda using reserved concurrency. This makes sure our create selection function can be called once and must resolve before it is called again. Basically, this is a cheap way of doing a queue and making sure each new selection is handled one at a time. Ok, let’s playback some audio.

Playing Audio

The final piece of this experience involves playing back the audio itself. When users initially get to the site, we can ask the database for the queue of current selections. If a selection exists on the queue, we can determine which selection should currently be playing based on those startsAt and endsAt properties we established earlier. Then, depending on the current time, we can determine at what position of the song the jukebox should be playing from. For example, if a 60 second song was played 30 seconds ago, I should be listening from the 30 second mark. It’s a little smoke and mirrors but works nicely.

What about selections being added to the queue in real-time? Our website is notified of these additions thanks to an integration of Pusher from the server side to the client side. As these selections come through, they are added to the end of the local queue and are played as selections complete. There is a chance the database queue and local queue will be slightly out of sync but as they say, it’s close enough for rock and roll. An added bonus of the Pusher integration is that we can add a real-time ticker of current listeners.

For actual audio playback, we’re using short lived S3 urls which get signed when they are needed. These MP3 urls are then played back using my audio library of choice: Howler.js.

Epilogue

This project is an amalgamation of everything I try to do as a creative developer. It bridges the gap between the real and digital world, engages the fanbase, expands the band’s lore, and challenges what is possible on the digital web. It is fiercely simple in concept but requires complex thinking to achieve. A challenge for sure but also with a realistic approach. I learned a lot and through all the ups and downs, I’m happy with the outcome. Thanks again to Mercury Records and Loyal T Management for collaborating with me on this project. Thanks to Ben and the rest of Lord Huron for entrusting me with another chapter of their story. Special thanks to some of the best fans in music who not only participated in this campaign but also celebrate these types of projects. The Cosmic Selector Vol 1. is out now.

Minting Cosmic Coins for Lord Huron's Cosmic Selector

Lee Martin — Wed, 16 Jul 2025 21:00:59 +0000

Having worked with Lord Huron three times now on the Follow The Emerald Star geofenced listening campaign, the Long Lost Seance teaser, and the Your Other Life player, I couldn’t help but notice a teaser video for their new album, The Cosmic Selector Vol. 1, go up at the beginning of the year. This video featured an otherworldly jukebox named the Cosmic Selector and, by habit, I immediately registered the domain cosmicselector.com. Now, I wasn’t sure if they were going to hire me again (and I wasn’t trying to hold the domain for ransom) but I had a feeling this might be the name of the album and the domain could prove to be a valuable asset for an upcoming marketing campaign. I would give it up, if asked.

Flash forward to April 1, of all days, and a call was scheduled to discuss the “Cosmic Selector.” I quickly dropped the secret: I had already registered cosmicselector.com and we had a good laugh about it. Naturally, we discussed the jukebox (more on the later) and there was an existing thought to potentially make some physical coins but the purpose of said coins wasn’t clear. The answer was to not look too far. If the website was going to be the “jukebox,” the coins should be the currency to play it. So, we would set out to produce unique physical coins which could be used to play selections from the new album via a digital jukebox.

Now, I usually try to do one thing I haven’t done before on each project (this is how I self-educate) but this experience required many new “firsts” for me:

Producing a physical asset
Using NFC technology so a mobile device could interact with the coin
Minting coins to a database and writing unique coin URLs to NFCs
Figuring out simple coin packaging

And that’s just the coins. The jukebox would also have a series of new problems I would need to traverse. For the sake of this dev blog, we’ll stick to the coins for now.

3D Printed Coins

Once we knew the direction of the campaign, we jumped on a quick follow up call about potential physical coin solutions. This included super pricey metal NFC challenge coins but… our goal was to find an affordable solution that would allow us to get these into as many fans’ hands as possible. I recalled that my local barista, Josh, had recently purchased a 3D printer he was raving about and while I’m not a fan of plastic, I wondered if it would be a viable solution for our NFC coins. So, I left that call with some homework. I would order a 3D printer and look into plastic as a potential affordable solution. We also set a deadline. We would distribute these coins at the end of May at the back to back Red Rock shows, which has turned into a fan pilgrimage, and the accompanying merch bazaar.

48 Hours of Prototyping

I knew our time was limited to get this done so as soon as my 3D printer showed up I got to work learning how it functioned and attempting to prototype a coin. I went with a Bambu Studio A1 mini because it was affordable and I read it was user friendly. I began by sizing my Blender workspace so I was modeling in millimeters and then I made sure the printer was printing at the right size. An easy experiment was to create a quarter shaped disc in Blender, print it on the Bambu, and then compare it to a real quarter. All good.

I think a good prototype shows off what is possible visually so I aimed to create a design which had a simple arcade token motif on one side and a natural scene relief on the other. That’s when I learned about nozzles.

A 3D printer kinda works like frosting a cake. You have nozzles of different diameters which can create designs of various levels of detail by extruding plastic. The higher the detail, the longer and more expensive the print. We found ourselves in a rather complicated position because our coin design needed to be both detailed and small. The prototype I created was a bit larger and modeled with a 0.4mm nozzle but I knew we would probably end up using 0.2mm. In addition to getting the details of a two-sided design right, we also needed a spot for the NFC sticker which would live within. I experimented with all sorts of constructions and learned a lot about the pros and cons of different printing strategies. Naturally, with every decision I made, I had to also consider how a 3rd party printer vendor might handle the print. It’s a lot to unpack over one weekend.

One thing I like about plastic is that it comes in a great range of colors (kinda like vinyl variations) and I started thinking about how coins could be different colors and carry stories from different dimensions. Should a coin look like the emerald star? Maybe a coin might have a bit of moss on it because it was buried in the swamp? Maybe it should be void of color? Maybe we sent them to fans unpainted and encouraged them to design their own coins. My fiancé, Anne Blenker, hand-painted one of the prototypes to simulate this idea.

Finally, I did a bit of thinking about simple packaging which would secure the coin and also provide a bit of instructions on how it worked. I went with a simple coin envelope with The Cosmic Selector Vol. 1 logo printed on the front.

Needless to say, it was a lot of thinking for 48 hours and all of this was presented in the most verbose client email ever. Seriously. We all jumped on another call, and rightfully so, set our sights on avoiding plastic for a higher quality material. So, I turned off my 3D printer and got back to work on the jukebox. Until…

Printing Returns

Two weeks before Red Rocks, we realized any alternate solutions we were planning would not arrive on time and 3D printing was officially back on. Panic. I stopped working on the jukebox and jumped right back into the coins.

Since my initial prototyping session, Tony Wilson had come up with a lovely design for the coin but it was in 2D. So, I quickly got to turning it into a 3D relief design. Tony’s design was a lot smaller at 40mm x 40mm so I had to consider which elements could stay and which had to go based on 0.2mm nozzle detail. I also finalized the coin construction to use a bowl and lid method. The bottom of the design and edge side would be printed like a bowl which the NFC could sit within. The top of the design would be designed as a lid which would be fit into the bowl and glued on top. This should help speed up the assembly process. I also placed an order for a Cosmic Selector logo stamp at StampMaker.com as well as an order for simple coin envelopes from Amazon.

It then came down to getting the coins printed. We found a partner in Scott from Stand It Up. He said the best way to print the quantity we were aiming for in the time we had remaining was to use resin. So, that’s what we did. Scott’s team printed it quickly and the coins were set to arrive the Saturday before the Wednesday show via Fedex for assembly. However, my weekend Fedex guy is notorious for not attempting to get in my building and, sure enough, the delivery was missed. Of course, Monday was Memorial Day, which means the coins would not arrive until Tuesday, the day before the first Red Rocks show. I began to realize I may in fact be hand delivering these coins to Denver.

Minting Coins and Writing NFCs

While waiting for those coins to arrive, I developed a workflow for minting coins and writing the unique coin URL to an NFC using the Web NFC API and an Android phone. Unlike most NFC campaigns which use a single URL, our NFCs required a different unique URL for every coin because we wanted to track the type of coin it was and whether or not it was “spent” to play a selection on the jukebox. Using the Web NFC API, I could check to see if a NFC already had a Cosmic Coin URL associated with it and if not, mint a new coin in our coin database and write the new Cosmic Coin URL to the waiting NFC. Check out this separate dev blog for more on that workflow.

Spray Paint

On Tuesday, the coins finally arrived and we quickly realized the gold resin was much too transparent. (Something we could have remedied with a bit more time.) That’s when Anne made the call to spray paint them gold, which is a common solution in 3D printing. For the Red Rocks coins, we just grabbed a can of gold spray paint from Ace Hardware but in the future we would use Montana Goldchrome for a nice metallic shine. So, there we were, spray painting these coins in our apartment parking lot 24 hours before they would be handed to fans in Denver. A lot of neighbors asked what we were up to but they just sorta figured we were creating next year’s Mardi Gras throws really early.

To The Mountains

With no time left to ship these coins, Anne and I woke up at 3am in New Orleans to catch a 5:40am flight to Denver. That flight got us to Denver with enough time to deliver these to the Tubb’s Bazaar near downtown Denver. We made it to the bazaar and were stamping the final coin envelopes an hour or two before the doors opened. Management had the genius idea to hide these throughout the bazaar and merch offerings. Soon, word started to trickle out that fans were finding coins in their hoodies, at the bar, and behind plants. Then, someone realized the coin could be scanned. That’s when something interesting happened. I had added a statement telling fans not to share their coin url and they were such rule followers that they also didn’t share the cosmicselector.com domain. Honestly, I was pretty proud of how secretive they were and figured it was only a matter of time before the domain leaked.

Coin Security

As soon as the coins were in fans’ hands, my brain started to worry about the security of the coin URLs. Even though the fans were being very secretive about their unique coin URLs, I was worried about someone else screwing it up: Google. Google Search is VERY good about indexing somewhat hidden URLs and I had this concern that all of the unique coin URLs would show up on Google and bad actors could play other fans’ coins. So, I quickly turned off search engine indexing in robots.txt for the entire website and that seemed to do the trick.

A Night at Red Rocks

Feeling like we did everything we could, we got ready for a night at Red Rocks and enjoyed some concessions on the terrace. We had the loveliest night hearing Strange Trails from front to back. Then, the band performed some songs from the new album. At the end of one of the songs, Ben reached into his pocket and threw some coins out to the fans. It was surreal.

I got to see Ben and management after the show and we celebrated the fact that we got it done. Henry (from management) told me he dropped all the coins in a puddle at some point, adding to the chaos of the coins’ journey to fans’ hands. I then turned my attention to the jukebox.

Oh yeh, we have to build a jukebox…

How I Designed a Red Dead Redemption 2 Inspired Map in Mapbox Studio

Lee Martin — Thu, 06 Feb 2025 22:40:21 +0000

This is a project I did with Mapbox back in 2018 while we were all playing Red Dead Redemption 2 for the first time. This rather lengthy how-to blog was always hidden away on mapbox.com but the link has been dead for years. So, I've gone ahead and reposted it here for archival purposes along with a working link to the map. Note: Mapbox Studio has evolved since I wrote this so some things may have changed technically but the approach stays the same. I hope you enjoy. - Lee Martin

Like millions of others, I have been enjoying Rockstar Game’s latest installment in the Red Dead Redemption series: Red Dead Redemption 2. Apart from the deep narrative, incredible visuals, and deadeye gameplay, I was struck by all the energy put into the user interface and in particular the game map. Rockstar’s fictional United States spans five states and many major cities including one inspired by my home: New Orleans (or in the game, Saint Denis.) Being an avid fan of Mapbox and an amateur at their Studio product, I thought it would be fun to redesign the actual world map in the style of the game map. Herein lies a very in depth tutorial on my design journey.

Explore the map and read on to find out how it was done.

Preparation

Setup

By far the hardest part of building this map, is simply that you have to take a break from this incredible game. However, it helps to have it running in the background so you can explore the map in game or from the companion iPad app. It was helpful for me to use this app to take screenshots and really understand how the map fidelity changes as the user zooms in and out. Another piece of advice is to try not to explore too much while you’re building the map and rather focus on each task. Maps, by nature, are fun to explore and I found myself using up a lot of my time simply typing in random locations.

Colors

One of the first things I did was recreate the game map’s color palette in my favorite design tool, Figma. To do this, I simply used the eyedropper tool to sample some colors from a series of game screenshots. If you don’t count things like red routes and yellow mission circles, there are really only five colors on the permanent map. Take note of these because we’ll be using them a lot later.

Land #DEC29B — The base of the map is a canvas color
Ink #40423D — The next prominent color is a dark black ink color
Water #9E9985 — Water is somewhere between ink and canvas with a hint of blue/green
Contours #C8B28D — The contour lines and fills look like stains on the canvas
Pencil #716454 — The viewpoints on the map are printed in a burnt tone

With your color palette built, it’s time to dig into typography.

Typography

The game map uses several different typefaces to label places of interest from massive slab serifs to handwritten cursive. Rather than spending a single dime on paid fonts, I headed over to Google Fonts and quickly found decent alternatives to each type. Here’s the breakdown:

State Names — Merriweather Black
City Names — Raleway Black
Major Bodies of Water — Noto Serif
Minor Bodies of Water and Rivers — Crimson Text Bold Italic
Station Names — Raleway Black
Natural Features — Chau Philomene One Italic
Viewpoints — Homemade Apple
Forts — Lato Black

Go ahead and grab those typefaces or any better alternative you find from Google Fonts. We’ll load them into Mapbox Studio soon.

Iconography

I turned to the Mapbox Maki Icon editor in order to recreate the game map’s icon set by simply setting their background to black, foreground color to white, and making them round. I then referenced the OpenStreetMap tags Mapbox makes available to build out a list of alternate categories. The final list of RDR2 places and their equivalent Mapbox place type looks something like this.

Bait Shop — Aquarium
Barber — Hairdresser
Butcher — Butcher
Doctor — Doctor, Hospital
General Store — General
Hotel
Photo Studio
Post Office
Saloon — Bar
Show — Cinema, Theater
Stable — Bicycle Rental
Stagecoach — Bus Station
Tailor — Clothes
Trapper — Fabric

Once you’re happy with your icon set, you can download them from the editor as one zip.

Scale

It’s helpful to wrap your head around the difference in scale between the world map and the game map as Mapbox will allow you to style every one of 22! different zoom scales. By comparing the geographical sizing of an area of similar size, I estimated that the Red Dead Redemption Map starts around 11 and goes up to 18 or so. So, if you zoom out to 11, your map features should have similar sizing. Conversely, if you zoom into 18, features should be of similar sizing to a completely zoomed in RDR2 map.

Mapbox Studio

Start Blank

I like building up my map from a blank canvas rather than using an existing template so I don’t get overwhelmed with layers and rather try to make the exact decisions necessary to recreate the style. In order to create a new blank map, head to Studio and click the “Pick a template of upload a style” button. This will bring up a new dialogue which includes a “Start Blank” button you can click to create a new blank map.

Interface

I think the Mapbox Studio interface is pretty intuitive and certainly easier to use than the item selector in RDR2. The left panel will include all of the graphical layers you’ll create similar to a layer panel in Figma or Photoshop. The open area on the right will display your map as it is designed. The top right corner includes another set of useful features and indicators including the map’s current zoom level and position. In addition, it includes a search box that allows you to quickly jump around to different locations around the world thanks to Mapbox’s geocoding technology.

Upload Assets

Click on the “Fonts” button in the top right corner (or press t on your keyboard) to bring up your account fonts. From here you can click the “Upload new font” button in order to upload each of the fonts you downloaded from Google Fonts into your account. Next click on the “Images” button (or press i) to bring up your images manager and you can then upload your Maki icon set.

Finally, let’s click the background layer in the left panel and click the trash icon (or press ⌘ + Delete) to delete the layer so we really have a blank map to start with. It’s now time to design our map.

Land and Water

Land

Every feature on our map begins as a new layer in Mapbox Studio. Let’s recreate our first layer (the background layer) by clicking the “Add layer” button or simply pressing n on your keyboard. When a new layer is created, Mapbox immediately opens that layers data source selection screen. We will be doing amazing things in this tab later but for now select “Background layer” as the source and then click “Create background layer.” Mapbox will then jump to the style tab of the layer panel which will allow us to adjust the aesthetic of our layer. In the case of our land layer, let’s simply change the hex color to be the same as our land palette color. Finally, we can hover over the name background on this panel and click to edit it. Let’s call it “Land.”

Good job creating your first layer. Yes it’s bare but let’s pretend it is a completely unexplored map at the moment.

Water

Fill

As someone who actually grew up down the bayou, I love how RDR2 handles water and makes it another crucial part of your exploration. Let’s add some water to our map by creating another new layer. Again, this will redirect you to our new layer’s data source tab. Instead of choosing background layer again, open “Mapbox Streets v7” in the active sources and click the “water” source. Make sure your map is completely zoomed out (you can use your mousewheel) and you should see green everywhere there is water in our world. Mapbox provides this sort of blueprint view of all of their data to help you understand what features will be visible depending on your source and filtering selections. We’ll take a closer look at this later. For now, you can click “Style” at the top of the layer panel to jump to the styling tab. From here, we can change the color of our water to reflect what we decided on earlier. I think that you’ll agree that this one simple addition almost immediately allows our app to begin to reflect some of that RDR2 map aesthetic. Now you may think that the next logical step is to simply style the 1px stroke provided by this same map layer but we’ll want to do something a bit more sophisticated. Namely, we’re going to style our water outlines to resize themselves depending on the current zoom level. For this, we’ll need to create another layer for the water outlines. Before that, let’s go ahead and make the 1px stroke transparent by setting the color’s alpha property to zero. Finally, let’s rename this layer “Water Fill.”

Line

Click on “Add layer” and again select “water” from the “Mapbox Streets v7” source. In the left column of the data tab, you’ll see a list of additional features including Type, Filter, and Zoom extent. Click Type and change the type to Line. You will then see our blueprint of visible features turn into a line around the water. Yes, this is awesome. Let’s jump over to the styling tab. First, let’s change the color of our line to be the ink color from our palette. Now a 1px ink colored line is fine but wouldn’t it be nice if the width of our line grew in size as we zoomed into our map like it does on the RDR2 map? Let’s do that. In the same style tab, click “Width” in the left column. From here you’ll see the default width of 1px and a couple of additional options:

Style across zoom range
Style across data range
Style with data conditions
Use a formula

You’re beginning to realize just how powerful this tool is right? Let’s click “Style across zoom range.” Mapbox Studio allows you to style pretty much anything based on the map’s current zoom level. In the case of our water lines, we would like them to grow in size as our map’s zoom level increases. A definition of style at zoom level is called a “stop” and by default Mapbox provides a stop for the highest and lowest levels (0 and 22.) As we discussed earlier in the “Scale” section, we’re going to be focused on zoom levels that more closely represent that of the RDR2 map. Now what I did to achieve this is not scientific at all, I simply zoomed in and out of the game map to decide when water lines should appear and the general sizing of the line depending on the zoom level. Let’s adjust the first stop’s zoom level from 0 to 15. You can leave the line width at 1px. Let’s then click the 2nd stop. You can leave the zoom level at 22 but we’ll want to adjust the line width to be 36px. Zooming in and out near a coast line will show that this looks pretty good but we can do better. Right above the two stop’s you just adjusted is another feature called “Rate of change.” Let’s change our rate of change from linear to exponential and set the base to 1.3. You just created your first dynamically sized line. Nice one! One more thing. By default our water lines are joined by sharp angled corners. Wouldn’t it be nice if these could be more rounded like the RDR2 map? Well click “Join” in the left column and then choose the “Round” line join option. That should smooth things out a bit. Let’s rename this layer “Water Line.”

Since we now have two layers for our water, let’s group them in a folder in the layers panel. To do this, simply click the Water Fill layer and then shift-click to also select the Water Line layer. From here you can click the little folder icon or simple press g on your keyboard to group your water layers. Finally, if you highlight the new Group at the top of the stack, you should see a pencil you can click to rename the group to “Water.”

If you made it this far, be proud of yourself. You’ve already learned one of the most powerful features of Mapbox Studio and we’ll make good use of zoom level based styling in the sections to come. This is also a good time to mention that it helped me a lot to focus on one small area while designing this map rather than jumping all over the world and getting lost in exploration. A good choice is a nice city park like beautiful Stanley Park in Vancouver. Simply search “Stanley Park” in the top right corner to jump to that location or try one that has a nice mix of features.

Contours

Line

Another key feature of the RDR2 map is its use of contour lines to give you an idea of what sort of topography you can expect from different regions of the map. Can Mapbox recreate this feature? Of course it can! Let’s add another new layer and select “Mapbox Terrain V2” in unused sources to select the “contour” data source. Then change the Type to Line just like you did with the Water Line layer. The blueprint view should now show green outlines that follow the contours of the earth. This really is a beautiful way to represent the slopes and valleys of our planet.

Let’s jump to “Style” for this layer to adjust the color and width to our contour lines. First, simply adjust the hex color by using the appropriate swatch from our palette. Then, we’ll want to again style the line width according to the zoom level. Our first stop will be at the zoom level 13 with a line width of 1px. Our second stop will be at the zoom level of 22 with a line width of 36px. We’ll again adjust the line to use exponential sizing and in this instance change the base to 1.4. Finally, let’s rename this layer to “Contour Line.”

Fill

In addition to contour lines, the game map includes an indication of steeper changes by using simple hill shading. We can mimic this effect in Mapbox Studio also. Add another layer and again select “Mapbox Terrain V2” from the sources section. Instead of choosing “contour” again, we’ll instead select “hillshade.” Now hillshade will provide polygons that include features of several different levels. I decided to filter this layer to only include highlights at the highest levels. In order to filter the data, we’ll click “Filter” in the left column within our new layer’s panel. From here we’ll click “Add filter” and choose “class” from the available fields. Studio will then ask us which class we’d like to show. Click “Empty” in the class selection dialogue and then click “highlight” in the list of filters. Next, let’s add another filter to only show highlights of a certain level. Click “Add filter” again and choose “level.” From here, you can click the dropdown which says “Is any of” to change the operator to “is greater than or equal to.” You can then click “Empty” again and select the filter of 90. Well done. You just created your first layer filter.

Styling for this layer couldn’t be any easier, simply change the color to the same one you used for your contour lines. This will allow the hills to blend into the contour lines the same way it does on the RDR2 map. Let’s again rename this layer “Contour Fill” and then group it along with “Contour Line” to create a new group called, you guessed it, “Contour.”

Now you might notice a bit of clashing between the water outlines and contour lines. This is because the water lines should appear above the contour lines to look best. In addition to organizing our layers, the layer panel also provides a built in hierarchy that simply stacks layers on top of each other from the bottom most layer to the top most layer. So, in order to make the Water group appear above the Contour group, we simply click and drag the Water group on top. Let’s do this and we’ll be ready to move onto more lines!

Paths, Roads, and Borders

From what I can tell, the RDR2 map consists of a few different line types: major roads, minor roads, paths (or trails), state borders, and railway lines. Let’s first look at these roads, paths, and borders as we can build off of our zoom level styling work.

Major Roads

Thinking graphically, a highway line is about 3 times the width of the water line we declared earlier and also the same ink color. It helps to use these simple measurements as we continue to pile more layers onto our map. Let’s again add a new layer and choose “road” from the “Mapbox Streets v7” data source. Mapbox will immediately highlight every single road in the world but let’s filter this down to motorways only. Click “Filter,” “Add filter,” and choose “class” as the filter field. Then click “Empty” and select “motorway” as the filter. If you’re still looking at Stanley Park, you will not see a visible line because there is no Interstate running through the park. :D Simply zoom out to find the nearest motorway to the north. Once you do, let’s jump over to styling. Again, we’ll make the line our map’s ink color. We’ll then create two stops. The first at the zoom level of 13 with a line width of 1px and the second at zoom level 22 with a line width of 72px. Again, we’ll set an exponential rate of change with a base of 1.4. Finally, let’s call this layer “Major Roads.”

Minor Roads

Minor roads are about twice as wide as our water lines but again use the same ink color. We’ll add the same “road” data source as the “Major Roads” layer but this time add more filtering to bring in additional road types. Click “Filter,” “Add filter,” and choose “class” once again. Then click “Empty” once again and choose the “motorway_link” filter. Then instead of leaving this dialogue, we’ll click “Add new filter value” and select “primary.” Repeat this step until “secondary,” “street,” “tertiary,” and “trunk” are added to the list of filters. This will make a nice set of minor roads visible on our map. Let’s get this layer styled also. First, change the color to Ink again. Then your width should be:

Zoom 13, line width: 1px
Zoom 22, line width: 48px
Exponential Change, base: 1.4

Let’s make sure the join for our minor and major roads are round like our “Water Line” and then rename this layer to “Minor Roads.” Now let’s learn about dashed paths while adding trails to our map.

Paths

The RDR2 map includes dashed lines that represent walking paths and trails. We can add this in a similar fashion to previous lines with one addition to styling. You know the drill: “Add layer,” select “Mapbox Streets v7,” choose “road,” and then click “Filter.” From here, we’ll filter by the class “path” this time. Jump back over to styling and turn the color Ink once again. Nice one. Round out that join and then set up your line width to the following zoom styling:

Zoom 15, line width: 1px
Zoom 22, line width: 24px
Exponential change, base: 1.4

Now in the same left column you find color and width, you’ll see a function called “Dash array” which allows you to declare sizes for dashes (and gaps between those dashes.) Click this to begin adjusting the line’s dash array. Click “Add dash with gap” and set the dash to 3 and the gap to 4. Pretty cool, right? If you zoom in, you’ll notice our path dashes are very square even though they are round in the game. Well, we can adjust this also. 💪 Similar to joins, Studio will allow us to adjust the caps of lines which by default are squared off. Click “Cap” in the left column and choose “round” as your cap type. Let then rename this layer “Paths.”

Borders

The thickest line on the game map represents the borders between each major area of the map. I like to think that RDR2 wanted these to represent state borders so let’s style that data source in Studio as well. In addition to begin thicker than the lines we’ve already added, the border lines include a rather interesting dash array which we can achieve by evolving on the “Path” work we just did. First add a new layer and choose the “admin” data source from “Mapbox Streets v7.” You may need to zoom out to see the lines that become visible as this will only major borders. When you do that, you might notice that in addition to lines between states, we’ll also see lines representing maritime borders. To have our border lines end abruptly when they run into the water like on the game map, let’s filter maritime borders off. Click “Filter,” “Add filter,” and “maritime” as your filter field. Then click “Empty” and set the filter to 0 to hide the maritime lines. Nice work, now let’s style it.

First things first, you guessed it: turn that line Ink colored. Let’s then set up our border’s width to be styled across a zoom range like before. Here’s the parameters that work nicely:

Zoom 6, line width: 1px
Zoom 22, line width: 144px
Exponential change, base: 1.4

Very nice. Now just round off both the “Join” and “Cap.” We can now add a dash array which is similar to the game’s map. If you look at the game map, you’ll see that the line is solid and then there is a gap, dash, gap, dash, gap of equal spacing. Let’s setup our dash array exactly that way. Click “Dash Array” and set the first dash to 10 and the gap to 1. Then click the handy “Add dash” button and set it as 1. Repeat this for another gap, dash, and gap. Setting them all to 1. That’s it! Rename this layer to “Border” and pat yourself on the back for unlocking a rare achievement: Dash Arrays.

Since our roads and borders are very similar, let’s group all of these in a group called “Lines.” Congratulations, this is really starting to feel like the game map. While I’m tempted to jump straight to the Railways work with you since it is similar to roads and borders, let’s quickly add some buildings first.

Buildings

The buildings on the game map couldn’t be simpler. They have a stroke that’s identical in width to our path lines (and also the same ink color.) In addition they have a fill which is the same as the contour color. Similar to water, we’ll want to break buildings into two layers (fill and lines) so we have better control over the stroke width. Let’s start by adding the fill since it should appear under the stroke.

Fill

Add a new layer and select the data “building” from the “Mapbox Streets v7” source. Jump over to Style and change the color to be the same as our contour color. Rename this layer to “Building Fill” and you’re done.

Line

Go ahead duplicate the “Building Fill” layer by select it and clicking the duplicate button or pressing d on your keyboard. Click the new layer and rename it “Building Line” then click “Select data” to bring up the data source selection for this layer. Our data source is still right because we’re still styling a building but we’ll want to change the Type from “Fill” to “Line” so we’re styling the lines instead. Once you’ve changed that, jump back to the Style tab. First, change the color of the line to Ink. Then, set up the same zoom range styling as “Path” for the width:

Zoom 15, line width: 1px
Zoom 22, line width: 24px
Exponential change, base: 1.4

Make sure that join is rounded and take pride in all the buildings around the world you just built. Finally, group both building layers in a new group called “Buildings.” Now let’s add our rail system.

Railways

Railways are a great way to get around in the game (and real life.) I still think about taking the Rocky Mountaineer from Vancouver to Whistler a few years back. It was incredible and I spent the whole ride hanging off a viewing platform enjoying views of Howe Sound. The railways on the game map consist of three elements. First there is the overall line which is the same width as a major road. Next comes a series of dots along the path which are the same color as our Land. Finally, each station has a donut shaped ring around it. Let’s create each of these in studio. Search the following coordinates to position the map near the station I stepped out of in Whistler: -122.994,50.096.

Line

Add a new layer and select “road” from “Mapbox Streets v7” as the data source. Then filter this data source by class of “major_rail” to only show major railway lines. The style of this line is similar to that of a major roadway so let’s just copy what we have there. Turn the line an Ink color and create the following zoom styling for width:

Zoom 13, line width: 1px
Zoom 22, line width: 72px
Exponential change, base: 1.4

Finally, set the “Join” to round and rename the layer “Railway Line.”

Dots

Duplicate the “Railway Line” and rename it “Railway Dot” then click “Select data” to navigate to the layer’s data source selector. Instead of a “Line” “Type,” let’s change to “Circle.” Mapbox will warn you that this change will drop any previous styling for this layer which is okay because this is a new layer. Click “Okay.” If you’re still positioned around the Whistler station, you should see a series of green dots following the path of the railway line. Let’s style these. Head over to styling and change the color to be the same as the land. This is our first “Circle” layer but it works the same ways as fills or lines. We’re going to simply use zoom styling to adjust the size of our circle radius so it fits nicely within our railway line. Setup the following zoom styling for your “Radius:”

Zoom 15, line width: 1px
Zoom 22, line width: 28px
Exponential change, base: 1.3

Nice work. You just conquered circles, no problem.

Stations

Add a new layer and select “rail_station_label” from “Mapbox Streets v7” as the data source. You should see a single green dot light up for the Whistler station. Let’s style it. First, we’ll give it a similar styling to the “Railway Dot” layer and then we’ll add a zoom styling for it’s stroke width. Let’s start by adjusting the color to be our land color and adding the same “Radius” zoom styling as the railway dots. You can quickly add an existing layer’s styling by choosing “Apply existing value,” filtering by the name of the desired layer, and selecting the style. Once you’ve done that, you can change the “Stroke Color” to be our Ink color. Finally, let’s add some zoom styling for the “Stroke Width” property so it creates a responsive donut around our circle. These values should work nicely:

Zoom 15, line width: 1px
Zoom 22, line width: 64px
Exponential change, base: 1.3

Rename this layer “Railway Station” and group all of your railway layers into a new group called “Railway.” Well done. Your world is now connected with an RDR2 inspired railway. You’re a regular Leviticus Cornwall.

Typography

As covered earlier in the “Prepare” section, the RDR2 map uses over 8 different typefaces to label everything from cities and rivers to natural features and famous forts. Luckily, Mapbox Studio has data sources that cover all of these subjects. In order to add these labels to our map, we must simple choose the correct data source, set our layer type to “Symbol,” “Filter” accordingly, and then style. Let’s start with major city names.

Cities

The game map has a few major cities which are labeled including the jewel of the south Saint Denis. You do know I live in the city it was inspired by… New Orleans? Yes, I do. Naturally that makes me love this game even more. Let’s label it. Add a new layer, select “place_label” from “Mapbox Streets v7” as your data source, and then change your layer “Type” to “Symbol.” This should light up all sorts of cities on your map. Let’s filter down by major cities only. Click “Filter” and add a filter for type, adding values for both “town” and “city.” Let’s another filter for “localrank” and set the value to 1. This will greatly decrease the label density, leaving only major cities. Once you’ve done this jump over to styling and don’t panic if you don’t see a label. “Text field” should be preselected and we’ll need to tell Studio that we wish to use the “name” property from our data sources as the text on the label. In order to do this, click the little icon under “Insert a data field” and choose name. You should then see the names show up. Let’s style them.

First, set the color to Ink like our previous labels and then select “Raleway Black” as your typeface. Click the “Transform” function and choose Uppercase as the transform. Then set your letter spacing to 0.2em, giving it a bit more spacing like in the game. In general, this works pretty well but we’ll want to add some zoom styling for the text size to make it perform a bit more like the game map. Click “Size” and add the following:

Zoom 8, line width: 8px
Zoom 22, line width:144px
Exponential change, base: 1.2

That should do nicely. Rename your layer “City Name” and let’s move on.

Viewpoints

By far my favorite typeface on the map, may not actually be a typeface at all. It seems like RDR2 had someone hand write some of the more interesting viewpoints and locations on the map. Luckily Google Font had a typeface called Homemade Apple which comes very close to emulating the effect. Search “Ferguson Point” using the “Search places” box and we’ll work on adding a label for it. Create a new layer and choose “poi_label” from “Mapbox Streets v7” as your data source. Let’s make this layer the “Type” of “Symbol” as well. The poi data source of Mapbox is incredibly rich with data and we’ll be putting this to good use in the Iconography section later. For now, let’s “Filter” the data by the type of “viewpoint.” This should highlight Ferguson Point in Stanley Park. Let’s style it.

Again, let’s make sure the “Text field” is using the data source’s name field. Instead of Ink, we’ll change the color of this label to the Pencil color we defined earlier. Let’s then change the typeface to “Homemade Apple.” Very nice. Now, I noticed on the map that a lot of the labels tend to be on two lines so let’s reduce the “Max width” to 5em in order to simulate this. Now we simply need to add some zoom styling once again for the text size. Here’s what works:

Zoom 8, line width: 8px
Zoom 22, line width: 72px
Exponential change, base: 1.2

That’s probably my favorite layer of the map. Let’s name it “Viewpoints” and work on some mountain peaks.

Natural Features

The game map labels a bunch of natural features like Tempest Rim, Window Rock, and the Three Sisters in an all caps italic typeface. We can add a similar set of labels to our map by styling the mountain peak labels data source from Mapbox Studio. Add a new layer and choose “mountain_peak_label” from “Mapbox Street v7,” making sure the layer “Type” is set to “Symbol” once again. This layer by default will label anything from a peak to a hill so let’s add a “Filter” of “elevation_ft” and set it to “greater than” “10000.”

Search “Pikes Peak” to reposition the map near that particular peak. We’ll style this one in a similar manner, setting the “Text field” to “name,” changing the color to Ink, and transforming the text to uppercase. We’ll once again change the word spacing to 5em in order to force the name to fall one 2 or more lines. (Pike’s Peak is small enough to stay on one line.) Finally, we can add some zoom styling for the text size. Since it is close to the same size as viewpoints, we can reuse the styling from that label. Click “Size” and choose “Apply existing value” and then select the “Viewpoints” text sizing from the list. Nice work. Rename this layer “Natural Features” and we’ll move onto some Forts.

Forts

Forts are a common place for adventure in the RDR series and their prominence in the story is made apparent by their permanent labeling on the game map. Let’s make sure to label the forts (and castles) on our map as well. Create a new layer and choose “poi_label” from “Mapbox Streets v7,” making sure the layer “Type” is set to “Symbol.” We’ll then “Filter” the data by type of “fort” and “castle.” Search for “Fort Independence” and jump over to styling. Set the “Text field” to “name,” change the color to Ink, transform the type to uppercase, set the max width to 5em, and adjust the letter spacing to 0.1em. You can then change the typeface to “Lato Black.” Once again, you can use the same zoom styling for size as “Natural Features” and “Viewpoints” by simply clicking “Size” and choosing “Apply existing value.” Rename this layer “Forts” and we’ll add some train stations next.

Stations

In addition to adding a circle for each railway station, the game map chooses to label some of the stations that appear outside of town. For the sake of our map, let’s go ahead and add similar labels to the major rail stations on our map. Add a new layer and choose “rail_station_label” from “Mapbox Streets v7,” making sure the layer “Type” is set to “Symbol.” Then create a “Filter” to make sure the “network” is “rail.” “Whistler” is once again a good location to jump to because it has a station right on the line. Let’s search for that location and then jump to styling. You know the drill: set the “Text field” to name, color to Ink, transform to uppercase, and max width to 5em. We’re also going to set the “Letter spacing” to be 0.2em and the font to be “Raleway Black” like the city names. We can once again reuse the same size styling from “Natural Features.” Nice.

Now the name of the station should appear right on top the line. However, on the game map, the label for Bacchus Station and Benedict Point station is actually positioned off to the side. We can create a similar layout by adjusting the symbol position. Click the “Position” tab to jump to this function. First, let’s set the “Text anchor” to be “Left” so that the label is anchored, you guessed it, to the left. Now there is probably still a bit of overlap between “Whistler” and the donut station, we’ll need to offset the text slightly to the right in order to remedy this. This will work best as a simple zoom styling. We’ll keep the rate of change to linear this time and setup the following stops:

Zoom 12, x: 0em, y: 0em
Zoom 22, x: 2em, y; 0em

Nice, the label should now be positioned slightly to the right. Rename this layer “Stations” and we’ll start taking a look at water labels.

Major Water Body

The largest body of water on the game map is Flat Iron Lake, followed by the San Luis and Lannahechee rivers. These are very simply styled as large labels. Water is rather tricky to style simply because it comes in all shapes and sizes. However, Mapbox will actually allow us to filter by the area of water to distinguish between large and small water sources. Let’s give that a whirl. “Mille Lacs Lake” is a hefty lake in Minnesota so let’s search for that one to position our map. Zoom out to make sure the whole lake is in view. Then create a new layer and choose “water_label” from “Mapbox Streets v7” as our data source. Once again, set the layer “Type” to “Symbol.” If you click on the Mille Lacs Lake feature on the blueprint map, you’ll see that the lake has an area of 1082656512. Let’s use that number as a starting point to distinguish large bodies of water. Simple add a “Filter” for “area” that is “greater than or equal to” 1000000000. With that filter setup, let’s get into styling.

Once again, set your “Text field” to use the data’s name, adjust the color to Ink, and transform the text to uppercase. We’ll also want to set the letter spacing to 1em and adjust the “Line height” property to be 2em. You can then change the font to “Noto Serif Bold Italic.” You can then apply the same styling for “Size” as the “City Name” layer. Go ahead and rename that layer “Major Water Body” and we’ll work on minor water bodies next.

Minor Water Body

For all smaller water bodies (but not too small) we’re going to apply a different label treatment. This is found on the game map in the form of ponds, pools, lagoons, and smaller lakes. Let’s start by duplicating the “Major Water Body” label you just created and renaming it “Minor Water Body.” Then jump back to your data source by clicking “Select Data.” From here we’ll adjust the “Filter” to areas of water “less than” 1000000000 and also “greater than” 10000. (I noticed a couple of fountains gaining this distinction when not adding this threshold.) You may adjust as needed. Now search the map for a smaller lake you’re familiar with and open up the style tab.

Since the style is still that of a “Major Water Body,” let’s adjust it. First, change the font to “Crimson Text Bold Italic.” Reset the text transform back to “None” so that it is no longer in all caps. Bring the letter spacing down to 0.1em and the line height down to 0.9em, making the text much more snug. You should also change the max width to 5em. Finally, let’s apply the same “Size” styling as our Forts and Stations. It’s now time to look at rivers!

Rivers

RDR2 will have you fording plenty of rivers from the Kamassa to the Dakota. These are set in the same type and sizing as minor water bodies. What’s unique about their labeling is how it follows the shape of the actual river. Mapbox Studio, once again, has a distinction for this type of label: Waterway Label. Let’s add this layer to label our map’s rivers as well. Create a new layer and choose “waterway_label” from “Mapbox Streets v7” as your data source and again set the layer type to “Symbol.” Let’s go ahead and then “Filter” the data by the “class” of “river” so we’re not styling any canals or streams in this manner. You can then jump over to styling.

This should now be second nature to you but let’s it anyway: adjust the “Text field” property so it uses the data’s name field, set the color to Ink, change the font to “Crimson Text Bold Italic,” and the letter spacing to 0.1em. You can then give it the same “Size” styling as the “Minor Water Bodies.” Sweet, now let’s make it follow the curvature of the river. Click the “Placement” tab and choose “Line center” as the “Placement” instead of circle. Your labels should now follow the line of the river. Good work! Let’s rename this layer “Rivers.”

Before we move on, let’s group all of our typography layers in a new group called “Typography.”

Icons

The game map uses many icons and markers to help you navigate. For the sake of our map, we’ll be focused on emulating the permanent place icons you can find in each town such as Saloons and Butchers. Adding an Icon to your Mapbox map is similar to adding a text label. It starts with picking the appropriate data source, making sure the layer is of type “Symbol,” and filtering down to the appropriate category. Make sure you have your icon images we created earlier uploaded to Studio and let’s quickly add them to our map.

First Icon

We’ll start by creating an icon for Post Offices. It’s here you can mail in your legendary fish, read letters from love ones, and get hot tips about possible stagecoach robberies. Start by adding a new layer to your map and choosing “poi_label” from the “Mapbox Streets v7” data source. Make sure your layer “Type” is set to “Symbol.” Then add a “Filter” of “type” and filter it by the value of “Post Office.” Make sure your map is positioned near a Post Office and let’s jump to Style once again.

Woah, wait. Before you start adding “name” as your “Text field…” Don’t. You don’t need to anymore. Instead, click on the “Icon” tab as that’s where we’ll be spending our time. From here, simply clicking the mail icon to establish it as your symbol’s icon. I would then set the size to 2. That’s it! Rename the layer “Post Offices” and let’s create the others.

Other Icons

For each other icon, you simply need to start by duplicating the “Post Office” and then changing the data source filter to reflect your new category. Finally, make sure to change the icon to reflect the new category and rename the layer accordingly. Reference the category list from preparation section to make this easier. Once you get all of these icons added to your map, group all of the layers in a new group called Iconography.

Texture

Our map, thus far, has been composed of solid colors but a closer look at the game map reveals a subtle texture. We can add this easily to our map by creating a new background layer, giving it a pattern, and moving it to the top of our layer stack. Mapbox Studio expects an SVG vector texture so I designed a very simple grain texture in Illustrator and followed Mapbox’s instructions on putting it on the right format. Go ahead and download the texture I created here and upload it to your map images. Once you’ve done that, add a new layer and instead of adding a data source, choose “Background layer” and click “Create background layer.” This will cause your map to black out but we can fix this by simply adjusting the “Pattern.” Click it and choose your texture from the images. Then adjust the layer opacity down to 0.01. Nice work!

Game Over?

Not quite. If you made it this far, congratulate yourself. You now have an incredible map which you can use to start blurring the lines between your game life and the real world. There are plenty of elements and features to the game map that I’ve left out which you should try to tackle. Here’s a few additional achievements you can unlock:

Drawing an illustration of a fish, uploading it as a map icon, and adding it to your favorite lake as a legendary fish location
Adding yourself as a main quest icon. (Homemade Apple works great for this)
Importing your newly created map into Mapbox GL JS to recreate the game’s compass
Using the Mapbox Directions API to build red line routing
Importing a geojson of your state or county’s boundaries and styling it to make yourself Wanted in that area

I just want to thank Rockstar Games for the inspiration. You really are building worlds there. Finally, thanks to Mapbox for supporting this adventure and building the tools that make it possible.

Thank you for coming along on this cattle drive.

Building an AI Powered Camera for David Bowie

Lee Martin — Sat, 01 Feb 2025 16:41:23 +0000

Last year, when Universal Music Group announced they were taking their music off TikTok, I immediately started thinking about ways you could recreate some of the TikTok user experience on the web.

I began with the feed. Pretty simple to emulate the snappy vertical scrolling the app is known for. Then, I set my sights on the camera. Naturally, I’ve built many web based camera concepts over the years but TikTok is known for its AI filters. So, I wanted to try and develop a web based, AI powered camera. The outcome of that (unofficial) experiment is called Heroes Camera and it uses AI to face swap you and David Bowie so you can become a parody of the “Heroes” album cover. Honestly, my focus was less on AI and more on user experience. I wanted the app to feel simple, familiar, and magic. Become a hero, today:

www.heroes.camera

When I originally put this app up, it garnered a few 100 usages and a chuckle from friends and followers. Then, I noticed a swift uptick of uses… 1000, 10000, 50000…? What was happening? I dived into the analytics only to discover that the app went viral in Japan. As it turns out, David Bowie had an incredible relationship with Japan and Japanese style. A lot of this was thanks to his Japanese costume designer Kansai Yamamoto. Apparently the application really resonated with Japanese users and it ended up being covered on Yahoo Japan. Since then, I visited Japan for 3 months and can say I also have a deep affection for this country and its people. I can’t wait to go back. To all of you from Japan who used the app: ありがとうございます!

Read on to learn how this app came together in a day.

Development

This simple app is built on Nuxt and uses a face swap model from Replicate in order to swap David Bowie’s face with your own. All we need to do is capture a new photo of the user's face or allow them to submit an existing photo, then we’ll pass that photo alongside the Heroes cover to swap faces. Like most AI stuff, it is fearfully simple. Let’s start by getting a photo from the user.

Capture Photo or Image Upload

I recently discussed in a dev blog for The Black Keys how I integrate MediaDevices to access a user’s camera and capture a photo of them (or their surroundings.) Check out that dev blog for more info. In order to obtain an existing image from a user’s photo library or computer, we just need a file picker.

<input type=”file” accept=”image/*” @change=”fileSelected” />

And a bit of Javascript. I actually read the image and place it onto an input canvas as I’ll be passing a data url to the prediction function later.

// File selected
// —-------
function fileSelected(e) {
  // File
  const file = e.target.files[0]

  // Create url
  const url = URL.createObjectURL(file)

  // New image
  const img = new Image()

  // On load
  img.onload = () => {
    // Prepare image
    prepareImage(img, img.width, img.height)

  }

  // Image src
  img.src = url

}

// Prepare image
// —-------
function prepareImage(image, width, height) {
  // Create canvas
  const canvas = document.createElement(“canvas”)

  // Resize
  canvas.height = height
  canvas.width = width
  // Context
  const context = canvas.getContext(“2d”)

  // Draw image
  context.drawImage(image, 0, 0)

  // Update input
  input.value = canvas

}

Now that our image has been drawn on a canvas, we can send it through to Replicate so it may predict the face swap.

Using Replicate to Swap Faces

Let’s first set up a Nuxt server route which will run the face swap model on Replicate based on a target image (the Heroes cover) and a source image (the user’s photo.) We’ll integrate the official Replicate Node.js client in order to do this. This code is pretty straightforward but it is worth noting that I store my replicate config details using the Nuxt composable useRuntimeConfig.

import Replicate from “replicate”

export default defineEventHandler(async (event) => {
  // Body
  const body = await readBody(event)

  // Config
  const config = useRuntimeConfig()

  // Client
  const replicate = new Replicate({
    auth: config.replicateApiToken
  })

  // Model
  const model = config.replicateModel

  // Run model
  const output = await replicate.run(model, {
    input: {
      target_image: “heroes.jpg”,
      source_image: body.source_image
    }
  })

  // Return
  return output

})

Finally, in order to call this from the client, we merely need to post the input photo as a Base64 to this new API endpoint.

// Predict
// —-------
async function predict() {
  // Source image as base64
  const sourceImage = input.value.toDataURL(“image/jpeg”)

  // Predict
  const data = await $fetch(“/api/predict”, {
    method: “POST”,
    body: {
      source_image: sourceImage
    }
  })

 // Image
 // data.image

})

We can now decide how we want to display the data.image result back to our user. You can simply throw it up in an <img> tag or generate a reveal video on the fly like I’ve done using Pixi.JS and MediaRecorder. Perhaps a topic for another dev blog.

Conclusion

The web can literally do anything these days and it is a free sandbox to explore and challenge the artist/fan relationship. When it comes to relying on social platforms like TikTok, I give all artists the same advice:

Use everyone.
Trust no one.

Social platforms come and go and typically leave a lot of carnage in their wake. It’s important to grab attention where you can but do not be surprised if that community you’re building disappears overnight. Focus as much attention on channels you own, like email newsletter, sms, past purchasers of your merch store, etc. Only then, will we be kings and queens.

Tattooing The Black Keys Fans Using MediaDevices and PixiJS

Lee Martin — Thu, 30 Jan 2025 15:07:08 +0000

One of the classic ways a fan can declare their loyalty for a band is by getting a tattoo of the band’s logo inked somewhere on their body. Whether that’s the Foo Fighters “FF,” the Slipknot tribal “S,” or yes, the Black Eyed Peas “?.” In fact, I have a sneaking suspicion that some of you reading this blog have some music related ink on you.

The aesthetic of The Black Keys new release, No Rain No Flowers, is based around the world of American traditional tattoos. I’m talking roses, hearts, scrolls, and sailor jerry inspired typography. As an interactive extra to the album and tour announcement, we’ve developed a new web app that allows fans to virtually tattoo themselves to show their allegiance and then encourage them to join the band's fan club, The Lonely Boys & Girls Club. I researched so many ways to accomplish this (AR, skin segmentation, marker tracking, etc) but landed on something simple and functional: 2D sticker tech. Using either your front or back camera, you can position, rotate, and scale the tattoo on your body and snap a photo of the result. We use a bit of blending and opacity to allow it to bleed into your skin. It’s certainly easier to give someone else a tattoo than give yourself one but I think that’s true to form.

Are you ready to be inked? Join the club today.

tattoo.theblackkeys.com

Read on to find out how this activation was developed using PixiJS.

Development

For this application, we gain access to the user’s camera using MediaDevices and then place it onto a PixiJS canvas as a video sprite. Then, we load the tattoo as an additional sprite and give it a bit of opacity and blending to bleed it into the user’s skin. Simple controls are added to allow the user to rotate, scale, and position the tattoo for the perfect inking.

Accessing Camera

I’ve written about accessing a user’s camera using the getUserMedia method of MediaDevices many times before. Not much has changed with this API implementation over time but I have personally started using a custom Typescript written Vue composable to integrate this technology into the experiences I'm building. I’ve made this app’s implementation available as a gist here. As long as I have a <video> tag with the id “cameraVideo” somewhere in my dom, all I need to do is call the start() method to gain access to the user’s camera. I wrap this method in a promise and fire a resolve only when the onloadedmetadata event is complete, signaling the camera stream is ready for action. I’ve also gone ahead and caught the primary errors and have provided real sentences I can display for my users. For this particular app, I’ve added the flip() method to flip the user’s camera from back to front. I also track which way the camera is facing using useState so my app can react appropriately (usually involves mirroring the camera image horizontally when it is facing the user.)

Pixi Setup

One tip from the start is that since we’re going to be capturing the Pixi canvas as an image later, we’ll need to preserve the drawing buffer when initializing the Pixi app. You can do this by adding the appropriate setting.

// Pixi app
const app = new Application()

// Init
app.init({
  canvas: document.getElementById(“pixiCanvas”),
  height: 1920,
  preserveDrawingBuffer: true,
  width: 1080
})

Since our camera stream is being displayed in a <video> element on our page, we can easily add it as a sprite on Pixi by using the from() method.

// Sprite from video
const videoSprite = Sprite.from(videoEl)

Sometimes you’re dealing with a canvas size that is different from the camera stream size. To handle this I use a little utility library called intrinsic-scale to calculate the dimensions and position which would resize the camera in a way that “covers” the canvas. It’s extremely helpful when I’m building these camera apps.

// Calculate covering
const { width, height, x, y } = resizeToFit('cover', { width: video.videoWidth, height: video.videoHeight }, { width: canvas.width, height: canvas.height })

// Size
videoSprite.height = height
videoSprite.width = width

// Position
videoSprite.x = x
videoSprite.y = y

For the tattoo, we just use the Assets load method of PixiJS to load the image texture and then create a new tattoo sprite using it. In order to blend the tattoo into the user’s skin, we can lower the opacity slightly and also apply a multiply blend.

// Load tattoo texture
const tattooTexture = await Assets.load(‘tattoo.jpg’)

// Create tattoo sprite
const tattooSprite = new Sprite(tattooTexture)

// Apply multiply blend
tattooSprite.blendMode = ‘multiply’

// Adjust opacity
tattooSprite.alpha = 0.8

The user is given control to reposition, rotate, and scale the sprite. PixiJS has a great example regarding dragging here. For rotation and scaling, I decided to wire up a HTML range slider using Vue reactivity. When the user changes the value via the slider, we can adjust the property on the tattoo sprite. Here’s the HTML:

<input type=”range” min=”0” max=”2” v-model=”scale” />

And the Javascript:

// Reactive
const scale = ref(1.0)

// Watch scale
watch(scale, (newVal, oldVal) => {
  tattooSprite.scale.set(newVal)
}

Originally, I used HammerJS to allow the user to use their fingers to do this but I found it rather difficult to use two fingers and try to capture a photo of myself. It’s a bit easier to simply hold my phone firmly and use the sliders to adjust.

Inking Tattoo

In order to ink the user aka take the photo, we simply need to use the toDataURL() and/or toBlob() method of HTML canvas. I use toDataURL to capture a preview url which I can display back to the user in an <img> tag on the outro page.

// Canvas to data url
const url = pixiCanvas.toDataURL(“image/jpeg”)

<img :src=”url” />

The blob is used to provide the user with either a download or we can use it alongside the Web Share API to prompt the user to share the image via one of their installed apps.

// Canvas to blob
const blob = await new Promise(resolve => pixiCanvas.toBlob(resolve, “image/jpeg”))

// File from blob
const file = new File([blob], “tattoo.jpg”, {
  type: blob.type
})

// Web share
navigator.share({
  files: [file]
})

Acknowledgements

Side story here. I always wanted to get my father’s American traditional style tattoo of a rose and scroll adorned with the timeless phrase “Mom.” Or as he calls it, “The worst decision of my life.” 😅 However, my older sister, Maria, beat me to it! However, I still plan on joining the The Loving Sons & Daughters Club soon.

Special thanks to Amber Nagle, Nina Schollnick, and David Adcock for dragging me to the tattoo shop with them. Be on the lookout for new tattoo design options at The Lonely Boys & Girls Tattoo Shop soon.

Finding Stars and Affirmations in the Sky with Three.js for Ayra Starr

Lee Martin — Mon, 01 Apr 2024 15:00:00 +0000

The Afrobeats revolution was well underway at Mavin Records long before Universal Music Group purchased a majority stake in the label, thanks to the meteoric rise of artists like Rema and Ayra Starr.

Ayra describes herself as a “celestial being” and her music as “heavenly.” Fans describe her lyrics as a series of affirmations meant to uplift and provide resiliency in the face of life’s hardships. I had the opportunity to build upon these themes for Ayra, alongside her new single “Commas,” and the outcome is a unique affirmations app for her fans.

Fans are encouraged to visit affirmations.ayrastarr.com daily to receive words of encouragement. Leaning into the celestial aspects of Ayra’s being, these daily affirmations are revealed in the user’s sky, like a message from heaven. As the reveal occurs, it is also recorded as a video, and the user is then provided a unique shareable piece of content consisting of Ayra’s words and their beautiful sky.

Here's how we used Three.js to place and find affirmations in the sky.

Development

I've developed a lot of web apps which use Three.js to create a light AR experience in the user's sky. From the Jack White Twilight Receiver to the Pop Smoke Tracklist Reveal, it's a simple but powerful mechanic. Each time I've developed one of these, I dreamed of a little wayfinder component (inspired by Sky Guide) which helps the user find the things we've placed in the sky but never actually got around to developing it. Since this particular app for Ayra was straight-forward, I put in the extra work to unlock this new piece of UX for this and future projects. I'm just going to focus on this core UX but please check out previous dev blogs on web AR projects for a more indepth look at how I build these apps.

Placing Star in Sky

The star in our sky is simply a Three.js Sprite with a star image texture. We place it in a random place in the sky using setFromSphericalCoords by choosing any random degree higher than the horizon and any degree around the user. I've adjusted the position a little so it isn't too close to the horizon.

// Texture
const texture = new THREE.TextureLoader().load("/images/star.png")

// Color space
texture.colorSpace = THREE.SRGBColorSpace

// Material
const material = new THREE.SpriteMaterial({
  depthTest: false,
  map: texture,
  opacity: 0.5,
  transparent: true
})

// Star
star = new THREE.Sprite(material)

// Spherical position
let vector = new THREE.Vector3().setFromSphericalCoords(10, THREE.MathUtils.degToRad(Math.random() * 90), THREE.MathUtils.degToRad(Math.random()* 360))

// Copy position
star.position.copy(vector)

// Add to scene
scene.add(star)

Finding Star in Sky

In order to allow users to use their device as a controller to adjust the position of the camera and find stars, I use the depreciated DeviceOrientationControls by patching it back into Three. In order for DeviceOrientationControls to function, we need access the user to grant access to their device's orientation. I attempt to gain access to this, alongside their camera, during a previous step of the UX using a custom composable I wrote for this purpose. You can see that permission step in the mockup video above. Once this permission is granted, we can initialize our DeviceOrienationControls with a single line.

// Device Orientation Controls
const controls = new DeviceOrientationControls(camera)

Just make sure to update the controls in your render step using controls.update(). Now the user can point their device at the sky and find the star but how do we know they are pointing at it? That's where a Raycaster comes in.

Targeting Star

On our app, we want to prevent users from revealing an affirmation unless they find and target the star in the sky. In order to determine if the user is currently pointing at the star, we can use a Raycaster. The raycaster, as it sounds, points a ray from the camera to a point on the screen. In our case, we're just interested in the center of the screen so a default (0, 0) point should work fine. Let's initialize both.

// Raycaster
const raycaster = new THREE.Raycaster()

// Pointer
const pointer = new THREE.Vector3()

Now, back in our render step. We can update the raycaster with the latest camera position and use the intersectObject method to determine if the raycaster intersects our star. Then you can do "something." In our case, we activate the ability to reveal the associated affirmation in the sky.

// Update ray
raycaster.setFromCamera(pointer, camera)

// Check for intersections
const intersects = raycaster.intersectObject(star)

// If intersects
if (intersects.length) {
  // Targeting star

} else {
  // Not targeting star

}

Now, let's look at the new piece of UX: the wayfinder.

Star Wayfinder

As I mentioned, I've always wanted to add a little 2D wayfinder to help direct users to the position of objects in the sky but I couldn't quite wrap my head around turning a 3D direction into a 2D element. After a lot of research and trial and error, I landed on a rather simple solution that works well. What we do is clone the star's position and then use the camera to convert it to local space. We can then calculate the directional angle in degrees using atan2 and adjust it by -90°. We then have a degree angle we can use with CSS to rotate the wayfinder element.

// Clone star position
let local = star.position.clone()

// Convert from world space to camera's local space
camera.worldToLocal(local)

// Calculate angle in degrees
let angleDeg = Math.atan2(-local.y - 0, local.x - 0) * 180 / Math.PI + 180

// Adjust by 90 degrees
angleDeg -= 90

// Rotate compass
document.querySelector("#wayfinder").style.transform = `rotate(${angleDeg}deg)`

Be on the lookout for an evolution of this component in future projects.

Using Vue Computed Properties to Count Down to New Armin van Burren

Lee Martin — Sun, 24 Mar 2024 17:07:11 +0000

I’ve always thought of the countdown clock as a mainstay digital marketing tactic and I’ve developed a lot of them in my career. Maybe 50? A slow ticking clock builds anticipation but also helps establish “when” something will happen. And, isn’t that what we’re trying to do? Increase hype and awareness through thematic means and reward fans for their attention.

A few weeks back Janice Renée Wendel and Lucas Wijkhuizen from Armada Music contacted me about an upcoming Armin van Buuren and Gryffin track “What Took You So Long.” The Joris van Meegen designed single visual includes a sort of ancient timepiece on top of a geometric vector design. While the first thing that came to mind was a rather complicated “sundial” concept I’ve been shopping around since the Daniel Ceasar campaign, the limited time we had led us to a simpler, countdown “clock” concept. An obvious start would be animating the hand/needle/gnomon on the timepiece so it makes one revolution from the start of our campaign to the track releasing. However, once you have a start and end time, you also gain a “percentage” of the time elapsed. What if we also included a preview clip of the song and the amount of audio you get to listen to is connected to how much of the countdown has elapsed? Using the outer edge of the geometric design, we established a series of time “segments” which unlock over time and in turn unlock more of the audio preview. It’s a simple tactic which will hopefully increase anticipation.

In an effort to generate some pre-saves, this activation is hidden behind a Feature.fm presave, which provides a post pre-save redirect url. So, once a user successfully presaves the track via their DSP of choice, Feature redirects them to our clock. Presave “What Took You So Long” today to enjoy an evolving preview of the upcoming release and read on to learn how it came together.

Development

A Vue computed property is defined as a data property which depends on another property. It is a surprisingly simple and powerful expression which allows you to compute all sorts of useful properties based on the values of another. In the case of this app, "time" drives the computation of all of the properties that make our clock tick. Let's start from the beginning.

Countdown

Every countdown clock has a current (now) time and the time we are counting down to. Our clock also requires a start time so we can determine the percentage time passed since the start of our campaign. We can define these easily using the ref function and use Date.UTC to make sure we're counting down to the same time regardless of the user's timezone.

// Start
const start = ref(new Date(Date.UTC(2024, 2, 15, 8, 0, 0)))

// Now
const now = ref(new Date())

// End
const end = ref(new Date(Date.UTC(2024, 2, 21, 23, 0, 0)))

Now, let's declare the most obvious computed property: finished. This will help us determine if the countdown has completed. All we need to do is check to see if the current time is greater than or equal to the countdown ending time.

// Finished
const finished = computed(() => {
  // If now greater than end
  return now.value >= end.value

})

In order to tick the clock forward, we simply need to start an interval which ticks every second (or sooner, depending on your use case.) Within this interval, the now value is updated and if the countdown is finished, we can stop ticking.

// Start tick
function startTick() {
  // Set tick interval
  tick = setInterval(() => {
    // Update now
    now.value = new Date()

    // If finished
    if (finished.value) {
      // Stop tick
      stopTick()

    }

  }, 1000)

}

// Stop tick
function stopTick() {
  // Clear interval
  clearInterval(tick)

}

Now, let's compute the properties which drive our clock visual.

Clock

As I mentioned in the intro, the clock image has a series of 32 segments near the outer circular edge. I thought it would be nice to fill these up visually as time passed. In order to do that, we'll need to compute some properties. First, let's computed the time span of the entire countdown. This is the overall time from the start of activation to its completion.

// Time span
const timeSpan = computed(() => {
  // End minus start
  return end.value - start.value

})

We'll also need the time elapsed, which is how much time has elapsed since the countdown begun.

// Time elapsed
const timeElapsed = computed(() => {
  // Now minus start
  return now.value - start.value

})

It will also be helpful to know the percentage of time elapsed for visual purposes. Note how this property is deriving from other computed properties. This is where computed properties get super powerful.

// Time percent
const timePercent = computed(() => {
  // Elapsed time divided by timespan
  return timeElapsed.value / timeSpan.value

})

There are 32 time segments on the clock image so let's determine how much time each of these segments take. In order to do that, we simply divide the entire span of time by the amount of segments.

// Time segment
const timeSegment = computed(() => {
  // One segment of timespan
  return timeSpan.value / segments.value

})

Now that we know how much time makes up a time segment, we can determine which segment we're on by getting the remainder of the elapsed time and the end time. Then, dividing that value by the time segment value. Finally, we'll use floor to round down to the largest integer.

// Current segment
const currentSegment = computed(() => {
  // Current unlocked segment
  return Math.floor((timeElapsed.value % end.value) / timeSegment.value)

})

You can visualize the filling of segments many ways, depending on your user case. In the end, I created a little component which uses a CSS conic gradient to create a sort of donut chart. You could also do this using canvas, especially if you're keen to create a dynamic image of the countdown for sharing.

The last bit of the clock visual is rotating the clock hand. We can do this by simply multiplying the percentage of time elapsed by 360 (degrees.) Don't worry about seek. We'll get into that next when talking about the audio player.

// Clock rotation
const clockRotation = computed(() => {
  // If seek is 0
  if (seek.value == 0) {
    // Time angle
    return timePercent.value * 360

  } else {
    // Seeked angle
    return seekPercent.value * 360

  }

})

Then, all we need to do is use CSS to rotate the clock hand image. Shout out to Vue for style bindings also.

<div id="hand" :style="{ transform: `rotate(${clockRotation}deg)` }"></div>

Now that we have a ticking clock. Let's bring some of this countdown logic into a dynamic audio player.

Player

In addition to the clock, it was our ambition to add an audio player which allowed users to play a preview of the upcoming song and the amount of preview they could listen to was connected to the amount of time elapsed on the countdown. So, as the countdown ticked closer to release, they were able to hear more of the track.

First, let's load the track with our audio library of choice: Howler.js. Keeping track of the track's duration is important. We'll get into the startListening and stopListening methods soon.

sound = new Howl({
  src: ["/what-took-you-so-long.mp3"],
  html5: true,
  onload: () => {
    // Update duration
    duration.value = sound.duration()

  },
  onplay: () => {
    // Start listening
    startListening()

  },
  onend: () => {
    // Stop listening
    stopListening()

  },
  onpause: () => {
    // Stop listening
    stopListening()

  },
  onstop: () => {
    // Stop listening
    stopListening()

  }
})

Now that we have the track's duration, we can computed the audio segment equivalent by diving the duration by the total amount of segments (32.) We'll use this to visualize the preview playback on the clock also.

// Audio segment
const audioSegment = computed(() => {
  // One segment of audio
  return duration.value / segments.value

})

As the track plays, a seek property is updated to represent the current position in time. (We'll set this in a different method.) We can use seek and duration to determine which segment the playback is on. Again, this is the audio player equivalent to the clock segment visual.

// Seek segment
const seekSegment = computed(() => {
  // Current seeked segment
  return Math.floor((seek.value % duration.value) / audioSegment.value)

})

Another computed property we'll need is determining how much of the audio track is unlocked for listening. This is determined simply by multiplying the duration of the track by the percentage of time elapsed.

// Audio unlocked
const audioUnlocked = computed(() => {
  // Time percent times duration
  return duration.value * timePercent.value

})

With all these properties in place, we'll need a method that is constantly updating the seek time when a track is playing and checks to see if the current seek is greater than the unlocked time. If so, we'll stop the track. Let's just call it listen().

// Listen
function listen() {
  // Request animation frame
  frame = requestAnimationFrame(listen)

  // Update
  seek.value = sound.seek()

  // If seek is greater than unlocked
  if (seek.value >= audioUnlocked.value) {
    // Stop sound
    sound.stop()

  }

}

And back in the Howler.js setup, we start and stop listening depending on if the track is playing. We'll use requestAnimationFrame to update things as fast as the browser wishes for us to.

// Start listening
function startListening() {
  // Request animation frame
  frame = requestAnimationFrame(listen)

}

// Stop listening
function stopListening() {
  // Cancel animation frame
  cancelAnimationFrame(frame)

}

Now, when a user plays the track, the listener will start and as soon as the seek meets the unlocked value, the track will stop.

Acknowledgements

Thanks again to Janice Renée Wendel, Lucas Wijkhuizen and their team at Armada Music for this opportunity. Their attention to detail is unrivaled and they entered this project with a great sense of excitement to do something special for Armin’s fans. I’ll gladly make time for them.

Wishing Upon A Star with Web AR for Disney’s Wish

Lee Martin — Sat, 25 Nov 2023 13:51:00 +0000

Reposted from my dev blog: leemartin.com/wish-star-finder

I’ve had the opportunity to work with Disney a couple of times in the past few years on a number of Instagram filters but I’ve never been hired to build a web app for the company. So, imagine my delight when Dani Ratliff from Disney Music contacted me about their new film Wish and was wondering if I’d be interested in taking the celestial learnings I’ve applied to other client projects for Jack White, Eddie Vedder, Shinedown, etc and attempt to build something unique for Wish on… the web. Finally, my wish came true.

Wish is the culmination of 100 years of Disney, a company that has been a creative inspiration to me since I was a child. I’ve always admired Disney’s ability to weave R&D, engineering, and artistry to tell stories in new ways. From them, I learned that technology should fade into the background and allow the magic of the interaction to shine through in simple and accessible ways. So how does one take these learnings and apply it to marketing a film like Wish?

In the film, our main character Asha makes a wish on a star. A scene we’ve seen countless times in Disney films. However, in this instance, the star answers and appears to her as a celestial being aptly called, “Star.”

What if we developed an AR experience in the browser which plots a field of wishing stars in the user’s sky. Then, when the user points their device at one of these stars, it unlocks a piece of content from the upcoming film. Users then have the option to wish on that star and watch as the star listens to their request. Once the user finishes wishing, the wishing star bursts in response and reveals the user, themselves, only to be shortly joined by the character “Star.” This entire magical experience is recorded and then offered back to the user as a shareable video.

Make a wish on a star today at wish.disneymusic.co and read on to learn how it all came together.

Wireframes

Once a scope is agreed upon and a proposal is approved, I provide my clients with a series of designless wireframes which illustrate the entire user journey. This allows everyone to understand how the application will function without getting sidetracked by the distraction of visual design. Here’s a look at some of those wireframes.

Once these wireframes are approved, I will begin developing a prototype version of the application which also does not include design. This allows me to get buy-in on the UX before we delve into the opinionated world of design.

Permissions

Since our application is a web AR app, we’ll need the user to agree to a few permissions in order to gain access to the technology required to augment their reality. Access to their camera is required so the application may stream an image of their reality. We also require access to their device’s motion and orientation so that they may use their device like a controller and point it at stars in the sky. Finally, we’ll need access to their microphone to receive and record their spoken wish.

Camera & Microphone

We use WebRTC to gain access to a user’s camera and microphone using the getUserMedia method. Typically, I would gain access to both of these from the same call. However, our experience requires the camera to flip from facing the environment to facing the user and I noticed that the small period of time the flip occurred (and microphone wasn’t available) contributed to a bit of audio lagging in the final recorded video. This was one of the nastier bugs I faced in development. So, we’ll just access each of these on their own media streams so that the camera can flip independently from the microphone.

I decided to write simple composables for both the camera and the microphone for maximum reusability. Here’s what the microphone looks like.

export const useMicrophone = () => {
  // State
  const microphoneStream = useState('microphoneStream')

  // Start microphone
  // ----------
  const start = async () => {
    // Promise
    return new Promise(async (resolve, revoke) => {
      try {
        // Get microphone
        microphoneStream.value = await navigator.mediaDevices.getUserMedia({
          audio: true
        })

        // Resolve
        resolve()

      } catch (e) {
        // Microphone error
        switch (e.name) {
          case "NotFoundError":
            revoke("Please enable the microphone on your device.")
            break
          case "NotAllowedError":
            revoke("Please allow access to your microphone.")
            break
          case "NotReadableError":
            revoke("Please close all other tabs which are using your microphone.")
            break
          default:
            revoke("This experience requires access to your microphone.")
        }

      }

    })

  }

  // Stop microphone
  // ----------
  const stop = () => {
    // If stream exists
    if (microphoneStream.value) {
      // Stop all streams
      microphoneStream.value.getTracks().forEach(track => track.stop())

      // Set stream to null
      microphoneStream.value = null

    }    

  }

  // Return
  return {
    start,
    stop
  }

}

I can then import the start and stop methods from this composable on any required page or component.

const { start, stop } = useMicrophone()

I use a state variable to keep track of the microphone stream because it is used in a few places throughout the app.

The camera composable works in a similar way with a few differences. First, the view state is passed to the camera start method so the camera is facing the appropriate direction. Next, I manually update a video tag with the camera stream and wait for its metadata to load since we’ll be rendering the visual from this video tag in our AR scene.

Motion

The API for gaining access to a user’s device motion and orientation is a bit different from WebRTC but not too complicated once you get the hang of it. To keep my code concise, I created a composable for this permission also. So, even though the permission APIs between media devices and device motion are different, at least they can seem similar in my code. The trick to accessing motion is making sure DeviceMotionEvent and DeviceOrientationEvent exist on the window and then using the requestPermission method to gain access to it. If everything was successful, the method will respond with granted. Here’s what my composable looks like.

export const useOrientation = () => {
  // Start orientation
  // ----------
  const start = async () => {
    // Promise
    return new Promise((resolve, revoke) => {
      // If request permission exists
      if (window.DeviceMotionEvent && typeof window.DeviceOrientationEvent.requestPermission === 'function') {
        // Request permission
        window.DeviceOrientationEvent.requestPermission()
        .then(response => {
          // Log
          console.log('orientation permission: ', response)

          // If granted
          if (response == "granted") {
            // Resolve
            resolve()

          } else {
            // Revoke
            revoke()

          }

        })
        .catch(e => {
          // Log
          console.log('orientation error: ', e)

          // Revoke
          revoke("This experience requires access to your device's motion.")

        })

      } else {
        // Log
        console.log('orientation does not exist')

        // Resolve
        resolve()

      }

    })

  }

  // Return
  return {
    start
  }
}

Access Granted

I keep track of the status of all of these permission requests on the same screen and compute whether or not they are successful. Once the user has enabled their camera, microphone, and motion, they may continue the experience.

const grantedAccess = computed(() => {
  // All access granted
  return cameraStream.value && microphoneStream.value && orientationEnabled.value
})

Now, let’s create a sky of wishing stars.

Wishing Stars

Our application augments a Three.js powered scene of wishing stars on top of a user’s sky. The user may then point their device at one of these stars to reveal the content associated with that particular star. Contentful is used by the client to manage these stars and a new one is added every Wednesday at the wishing time of 11:11. Let’s talk a little bit about plotting these stars and targeting them with the user’s device.

Managing Stars

As I mentioned, Contentful is used by the client to manage the wishing stars in the sky and their associated content. Content can be an image, video, or audio. Once the content is added to Contentful, we can use their Publishing API to fetch them for plotting. To do this, I first set up a Contentful plugin in Nuxt and set up the associated environment variables in Netlify.

// Imports
import * as contentful from 'contentful'

// Define plugin
export default defineNuxtPlugin(nuxtApp => {
  // Use config
  const config = useRuntimeConfig().public

  // Provide contentful client as helper
  return {
    provide: {
      contentful: contentful.createClient({
        space: config.contentfulSpaceId,
        accessToken: config.contentfulAccessToken,
        host: config.contentfulHost
      })
    }
  }

})

I can then use this client to fetch all the star entries.

const { items } = await $contentful.getEntries()

Now that we have our wishing stars, let’s plot them in the sky.

Plotting Stars

As mentioned, we’re using Three.js to power the AR visual of our web app. I’ll save you the basics of building with Three.js since they have wonderful docs and instead focus on the specifics of our application. A wishing star in our sky is one that the user can interact with. So, even though there are many stars visible on the app, most of these are what I call “idle stars” and they cannot be interacted with. A wishing star is actually a Three.js Group which includes several Sprites (or a plane that always faces the user.)

The body of the star is the graphic of the iconic 8-point wishing star we’ve seen in many Disney films.
The glow of the star is the glow of light surrounding the 8-point star. This will animate as the user speaks their wish.
The flare of the star is the star burst graphic that appears when the wishing star begins to answer a user’s wish.
The flash is a flat white Sprite that is used to envelop the user’s screen in bright white light after the star bursts.

Both the idle and wish stars need to be plotted in the upper hemisphere of the user’s reality, aka their sky. In order to do this, we come up with a random spherical positioning 60 degrees and below the sky directly above the user and then position the star there.

// Random position
let position = new THREE.Vector3().setFromSphericalCoords(
  100, 
  THREE.MathUtils.degToRad(Math.random() * 60),
  THREE.MathUtils.degToRad(Math.random() * 360)
)

// Position star
star.position.copy(position)

All of these stars are added to another wishStars group so they can be easily targeted later.

In order to make the sky itself a bit more interesting, we plot a Sphere in the scene and give it a Disney Wish texture. I use a gradient fade so the texture is more noticeable the higher the user points up in the sky. This helps differentiate the user’s reality with the Wish reality.

Targeting Stars

With these wishing star groups and their associated sprite layers plotted in our textured sky, we can add DeviceOrientationControls and a Raycaster to determine which star the user is currently pointing at. Check out the Eddie Vedder “Earthling” dev blog for more context on resurrecting the defunct DeviceOrientationControls using patch-package. Once the user’s device orientation is controlling the camera, we can use a Raycaster to determine if they are intersecting with a wish star. First, let’s set up our Raycaster and a 2d Pointer.

// Raycaster
raycaster = new THREE.Raycaster()

// Pointer
pointer = new THREE.Vector2()

Now, in our render loop, we can update the raycaster and check for any intersections with wish stars.

// Update ray
raycaster.setFromCamera(pointer, camera)

// Check for intersections
const intersectObjects(wishStars.children)

This combination of DeviceOrientationControls and Raycaster works really well. However, I didn’t want the associated content to unlock immediately but rather when the user paused over the wishing star for a moment.

Unlocking Stars

When a user intersects with a wishing star, I use Greensock to tween a progress variable from 0.0 to 1.0 over 3 seconds. If the user stops pointing at that wishing star before the tween completes, the tween is killed and the wishing star content is not revealed. However, if the animation completes, the star content will reveal itself as a <dialog> element (more on that soon.) Here’s a look at that Greensock tween.

starUnlock = gsap.to(starProgress, {
  duration: 3,
  ease: ‘linear’,
  value: 1,
  onComplete: () => {
    // Star visible
    starVisible.value = true
  }
})

A tween can be killed with the kill method.

starUnlock.kill()

In order for the user to know this is happening, we need a clear visual of the unlock progressing. To achieve this, I designed a custom SVG scope in the Wish aesthetic which receives a starProgress prop and animates a glowing white circle progress. Here’s a CodePen of that component.

Revealing Content

When the user successfully unlocks a wishing star’s content, it is revealed in a Dialog element. I love this standardized method of bringing up a modal dialog box that must be interacted with in some manner in order for it to be closed. It can also be nicely customized in CSS and I did my best through the use of pseudo elements to give it a Disney Wish flare.

We can show a dialog by calling the showModal method and hide it by using the close method.

// Show modal
document.querySelector(“dialog”).showModal()

// Close modal
document.querySelector(“dialog”).close()

In the case of our modal, users have the ability to close it and return to exploring the sky or close it and begin wishing on the currently targeted star. Before we talk about wishing, let’s discuss our setup for recording the wishing experience as a video.

Wish Recorder

Before we allow the user to begin their wish, we need to set up the infrastructure that will record it. Rather than directly record the Three.js renderer canvas, I set up a separate recording canvas with the story-friendly dimensions of 1080px by 1920px. We can then overlay additional images and videos which are not visible in the user experience but should be present on the final shareable video. All of this visual content is then recorded with MediaRecorder but we can’t forget to manage and record the audio of the experience also. Namely, the audio coming from the microphone and “Star” character video.

Recording Canvas

The recording canvas itself is simply a blank 1080 x 1920 canvas.

// Recording canvas
recordingCanvas = document.createElement(‘canvas’)

// Size
recordingCanvas.height = 1920
recordingCanvas.width = 1080

// Recording context
recordingContext = recordingCanvas.getContext(‘2d’)

We can scale, position, and draw the Three.js renderer canvas onto this recording canvas by calculating a cover size using the intrinsic-scale helper library.

// Get cover
let { x, y, height, width } = cover(recordingCanvas.width, recordingCanvas.height, threeCanvas.width, threeCanvas.height)

// Draw three
recordingContext.drawImage(threeCanvas, 0, 0, threeCanvas.width, threeCanvas.height, x, y, width, height)

In addition to drawing the Three.js canvas, we can draw image and video layers to add a bit of a story and more visual elements to the final recorded video.

For example, I could draw the Wish logo and a little quote from the film over the Three.js scene. What about the moment the user’s screen goes white? Instead of that simply being a blank white screen in the final video, we’ve made it a short video clip which shows the “Star” character zipping past the white screen before appearing to the user. To manage this, I use a shot variable to determine which additional layer to draw when rendering the recording canvas.

if (shot.value == “wish” || shot.value == “star”) {
  // Draw overlay image
  recordingContext.drawImage(overlayImage, 0, 0)

} else if (shot.value == “flyby”) {
  // Draw video
  recordingContext.drawImage(document.getElementById(“flybyVideo”), 0, 0)

} else if (shot.value == “end”) {
  // Draw end card
  recordingContext.drawImage(endImage, 0, 0)

}

We can then adjust the current “shot” as the user progresses through the interaction. Now that we’re rendering the canvas we’d like to record, let’s set up our MediaRecorder to record it and all of the audio happening in our experience.

MediaRecorder

The MediaRecorder interface is a fascinating piece of standard browser technology that allows us to record a group of visual and audio streams as new video or audio files. In the case of our app, we’re interested in recording the “recording canvas” we’re rendering with all of our visual content. In addition, we’re interested in recording the user’s microphone when they are wishing and the “Star” characters sound effects when it is on screen.

Video

To begin with, capturing a visual stream and associated video track of the recording canvas couldn’t be simpler.

// Get canvas stream
const canvasStream = recordingCanvas.captureStream(30)

// Get video tracks
const [videoTrack] = canvasStream.getVideoTracks()

Audio

As for the audio… Here's what I settled on after a lot of engineering work. The audio in our experience is coming from three different places. First, from the microphone stream when the user is wishing. Second, from the “Star” flyby video which is rendered onto the recording canvas secretly when the user’s screen is white. And lastly, from the “Star” appearance video which is part of the Three.js scene featuring the user’s video. One of these is an existing media stream and the other two are existing media elements (video tags on in the HTML.) In order to manage this variety of sources and their associated volume, we can use the Web Audio API.

First, we’ll set up a new media stream destination to receive all of these sources.

// Create audio context
const context = new AudioContext()

// Create destination
const destination = context.createMediaStreamDestination()

For our microphone, we’ll create a new stream source and connect it to a gain node so we may dynamically control its volume when recording. We then connect the gain node to our destination stream.

// Microphone source
const microphoneSource = context.createMediaStreamSource(microphoneStream.value)

// Microphone volume
microphoneVolume = context.createGain()

// Adjust volume
microphoneVolume.gain.value = 1.0

// Connect to volume
microphoneSource.connect(microphoneVolume)

// Connect to destination
microphoneVolume.connect(destination)

For our “Star” videos, we’ll create a pair of media element sources and also connect them to individual gain nodes so we can adjust their volume as needed. Unlike the microphone, we’ll want the user to hear these videos through the app so we’ll connect their gain nodes to the primary audio context destination in addition to the destination stream. Here’s an example.

// Star video source
const starSource = context.createMediaElementSource(document.getElementById("starVideo"))

// Star volume
starVolume = context.createGain()

// Adjust volume
starVolume.gain.value = 0.0

// Connect to volume
starSource.connect(starVolume)

// Connect to destination
starVolume.connect(destination)

// Connect star volume to context destination
starVolume.connect(context.destination)

With all of these sources now connected to the destination stream, we can access the stream and get its associated audio track.

// Get audio stream
const audioStream = destination.stream

// Get audio track
const [audioTrack] = audioStream.getAudioTracks()

We can then combine this audio track with the recording canvas video track to create our final video stream which is what will be used to finally initialize our MediaRecorder.

// Create video stream
const videoStream = new MediaStream([videoTrack, audioTrack])

// Recorder
recorder = new MediaRecorder(videoStream)

Since this is already starting to feel like a MediaRecorder dev blog, I’m going to stop short of how to use MediaRecorder and instead direct you to the excellent documentation on the subject. The most important bits are storing video chunks as they are available and creating the video blob once recording is complete.

// On data available
recorder.ondataavailable = e => {
  // If size
  if (e.data.size > 0) {
    // Push new chunks
    recorderChunks.push(e.data)

  }
}

// On stop
recorder.onstop = e => {
  // Create blob
  let blob = new Blob(recorderChunks, {
    type: recorder.mimeType
  })

  // Store blob
  videoBlob.value = blob

}

Analyzer

In addition to using Web Audio to wrangle our audio sources for recording, we’ll use it to set up an Analyzer node to receive volume levels from a user’s microphone as they speak. Then, we’ll use these levels to adjust the glow of the wishing star so it seems like the star is glowing in response. Here’s how you set up a very simple analyser and connect it to the microphone source.

// Create analyser
analyser = context.createAnalyser()

// Set fft size
analyser.fftSize = 64

// Connect microphone to analyser
microphoneSource.connect(analyser)

// Initialize analyser data
analyserData = new Float32Array(analyser.fftSize)

We’ll discuss how to use this analyser next.

Wishing On A Star

With the recorder setup and ready to receive wishes, it is time to let the user make their wish. As the user speaks into their microphone, our Web Audio analyzer will adjust the glow of the wishing star so it seems to be listening. Once the user has finished wishing or the wishing time elapses, the wishing star will react by bursting and flashing the screen white. This is very similar to what Asha faces in the film when she makes her own wish.

Listening Star

Within our renderer loop, we’ll analyze the microphone to get an average volume level when the user is wishing. Then we’ll use this level to adjust the wishing star glow. First, we’ll use the getFloatTimeDomainData to get the current waveform array of magnitude data. Then we’ll determine an average of this data and adjust it with Math.min and Math.max until it is giving a pronounced value between 0.5 and 1.0. Finally, we’ll adjust the opacity of the glow.

// Get time domain data
analyser.getFloatTimeDomainData(analyserData)

// Sum
let sum = 0.0

// Loop through data
analyserData.forEach(d => sum += d * d)

// Get average
let avg = Math.sqrt(sum / analyserData.length) / 0.1

// Get level
let level = Math.min(1.0, Math.max(0.5, avg))

// Adjust glow opacity
glow.material.opacity = level

Now the wishing star will glow as the user speaks.

Star Reaction

Once the user finishes their wish, the wishing star will burst until the entire screen goes white. This effect is achieved by using a Greensock timeline to tween the scale and opacity of the flare and flash sprites within the wishing star group. Here’s that setup.

// Timeline
let tl = gsap.timeline()

// Fade in flare
tl.to(flare.material, {
  duration: 1,
  ease: "power4.in",
  opacity: 1
})

// Scale flare
tl.to(flare.scale, {
  duration: 1,
  ease: "power4.in",
  x: 200,
  y: 200, 
}, "<")

// Fade in flash
tl.to(flash.material, {
  duration: .5,
  opacity: 1
})

As the flare fades in, it is also scaled to 200 x and y. Then, the flash appears and the screen goes white.

Wishing Time

In an attempt to keep the final recorded video on the shorter side, users are given a maximum of 15 seconds to make their wish. The solution here is somewhat similar to the Greensock tween powered star progress scope we used to unlock wishing stars. However, instead of using an elaborate SVG scope, I’m just using a simple progress bar in the header of the experience. Here’s the Greensock tween.

wishTimeout = gsap.to(wishProgress, {
  duration: 15,
  ease: ‘linear’,
  value: 1.0,
  onComplete: () => {
    // Stop wishing
  }
})

Users also have the ability to click a “Finish Wish” button if they finish wishing before the 15 seconds are up. At which point, the wishTimeout tween would be killed.

Star Answers

With the screen white and our wisher waiting patiently, we use this opportunity to flip their device camera so it is now facing them. In addition, we no longer record their microphone audio and instead focus on recording the “Star” character audio. When the user finally sees themselves, they are not alone. They are accompanied by “Star” who has heard and answered their wish.

Camera Flip

Since we put together that nice composable for accessing the user’s camera, in order to flip it, we simply need to stop the camera, adjust the view state, and start the camera again. Again, we do this independently from the microphone to help fight audio lag on the final recorded video.

// Stop camera
stopCamera()

// View user
view.value = ‘user’

// Start camera
startCamera()

Now let’s adjust the audio.

Audio Flip

While the user is wishing, we’re interested in recording their microphone audio but when they’re done, we’re interested in recording the “Star” character SFX. Remember those Web Audio gain nodes we set up earlier? Here’s where they come into play. By adjusting the gains, we can mute the microphone audio and unmute the “Star” audio for recording.

// Mute microphone
microphoneVolume.gain.volume = 0.0

// Unmute flyby
flybyVolume.gain.volume = 1.0

// Unmute star
starVolume.gain.volume = 1.0

I love to see Web Audio and MediaRecorder work together in this manner.

Star Video Sprite

The final magical element of our experience is when the “Star” character appears to the user to sprinkle some stardust on them. We’re all stars after all. The star video is also a Three.js Sprite, which is resized to cover the full dimensions of the Three.js scene. Transparency is achieved by creating a star video source which has both color and alpha channels and using both separately as maps for the sprite material. Let’s start with initializing the video textures.

Video Textures

Here’s an example video showing the stacking of both color and alpha channels. Luckily, Disney provided excellent video toolkit assets which had these channels available to me. What we want to do is use the top for our full color texture map and the bottom for our alpha map. That way, areas which are black on the alpha map texture will be transparent. This involves creating new VideoTextures from the startVideo video tag and using offset and repeat to isolate each specific channel/area.

// Star texture
starTexture = new THREE.VideoTexture(document.getElementById("starVideo"))

// Wrap
starTexture.wrapS = THREE.RepeatWrapping
starTexture.wrapT = THREE.RepeatWrapping

// Offset
starTexture.offset.set(0.0, 0.5)

// Repeat
starTexture.repeat.set(1.0, 0.5)

// Star alpha
starAlpha = new THREE.VideoTexture(document.getElementById("starVideo"))

// Wrap
starAlpha.wrapS = THREE.RepeatWrapping
starAlpha.wrapT = THREE.RepeatWrapping

// Offset
starAlpha.offset.set(0.0, 0.0)

// Repeat
starAlpha.repeat.set(1.0, 0.5)

We can then use these when initializing our sprite material.

const material = new THREE.SpriteMaterial({
  alphaMap: starAlpha,
  depthTest: false,
  map: starTexture,
  side: THREE.DoubleSide,
  transparent: true
})

Resizing Sprite

In order to have the video sprite fill the screen, we’ll calculate the vertical field of view and then the screen height based on how far the camera is from our sprite. It’s two lines of fancy math that effectively resizes the sprite so it fills the entire vertical space of the screen.

// Vertical field of view
let vFOV = THREE.MathUtils.degToRad(camera.fov)

// Height
let height = 2 * Math.tan(vFOV / 2) * (camera.position.z - starVideo.position.z)

// Scale
starVideo.scale.set(height, height, 1)

Playing Sprite

With our star video sprite resized to fit the view and properly textured, we simply need to play the starVideo tag to have the “Star” character appear in front of our user. I’ll also listen for the “ended” event so I know when to conclude the recording experience.

// Play star video
document.getElementById(“starVideo”).play()

// Wait until ended
document.getElementById("starVideo").addEventListener("ended", () => {
  // Stop recording

}, {
  once: true
})

Sharing

Once the video is created with MediaRecorder, it can be shared directly to any social app using the Web Share API. This involves creating a file out of the video blob and passing it to the Web Share API. I like using this mime library to determine the precise extension and type from the video blob.

// Extension
let extension = mime.getExtension(videoBlob.value.type)

// Type
let type = mime.getType(extension)

// File name
let fileName = `wish.${extension}`

// File
let file = new File([videoBlob.value], fileName, {
  type: type
})

// Share file
navigator.share({
  files: [file]
})

Design

While I was provided a ton of Disney Wish toolkit assets, there wasn’t a whole lot of web friendly graphic design associated with this project. For example, Disney Wish does not have a custom built website, just a standardized page on the Disney movies domain. So, in order to create a responsive design system for this project, I had to look further. I found inspiration by exploring the full range of merchandise associated with the film, from activity books to backpacks and gel pens to UNO cards. I even purchased The Art of Wish off Amazon. This allowed me to get a birds eye view of how a bunch of different stakeholders were handling design and translate some of their choices into a responsive design system of colors, typography, and buttons. Again, I try not to go overboard here. The design should feel familiar but also work well on a simple web UI. One highlight is simply the primary button which uses before and after pseudo elements to create ornamental edges in the style of the architecture seen in Rosas. Here’s a CodePen of that button construction.

Acknowledgements

Thanks so much to Dani Ratliff, Weston Lyon, Natalia Castillo, and their entire team Disney Music for this opportunity. Special thanks to the privacy, legal, and technology teams at Disney who helped me pull off my first web app for the company. I truly hope we can build more magical things together.

Vue Sci-Fi Scanner Transition

Lee Martin — Mon, 18 May 2020 15:28:16 +0000

I recently had the opportunity to launch Shelter In Space for the band Khruangbin which allows users to generate a Spotify or Apple Music playlist (curated by them) for a home activity of their choosing. The design of this app was inspired by the guide animations from the 1980's BBC television adaption of The Hitchhiker's Guide to the Galaxy. One of the key components I tried to recreate was the vertical scanning effect which transitioned between content. Click the image in the Codepen below to see the final solution and read on to better understand how it was developed.

First, the structure. .scanner holds two divs which both have background images. One is a pair of touching hands and another is Buddah. Both of these are absolutely positioned on top of each other. There is a single Vue property of state which allows toggling between the two via a click event on the .scanner div.

Let's first talk about the clipping transition which clips one image out of frame while showing the other. Vue has great documentation on ways to apply enter/leave transitions which an item is removed or added from the DOM. In our case, we'll be using the transition classes which Vue applies automatically. The clip-path CSS property allows you to clip an element in all sorts of shapes and sizes. We're only interested in an inset rectangular shape.

For example, if you wanted to clip 50% of a div from the top.

clip-path: inset(50% 0 0 0)

What about 75% from the bottom?

clip-path: inset(0 0 75% 0)

The clip-path property is also animatable. With this knowledge, we can setup our transition classes accordingly.

/* Clip 100% from bottom before shown */
.scan-enter{
  clip-path: inset(0 0 100% 0);
}

/* Transition to no clipping from bottom */
.scan-enter-to{
  clip-path: inset(0 0 0 0);
}

/* Start from no clipping */
.scan-leave{
  clip-path: inset(0 0 0 0);
}

/* Transition to 100% clipping from top */
.scan-leave-to{
  clip-path: inset(100% 0 0 0);
}

/* Set time and easing */
.scan-enter-active, .scan-leave-action{
  transition: clip-path 2s linear;
}

Make sure to add key attributes to your content and clicking should show the clip-path transition. Now, let's discuss the scanner line itself. In the source material, the scanner lines sits over both the incoming and outgoing content at the point of transition and turns content beneath it white without affecting the black background. CSS has a property called backdrop-filter which allows you to apply CSS filters such as blur or brightness to elements beneath the styled div. While most folks might use this to create blurred overlays, let's use it to brighten the illustrations. Rather than create a new div for the line, we'll add a ::before pseudo element to each image div and position it absolutely.

.scanner div::before{
  backdrop-filter: blur(1px) brightness(1000%) grayscale(100%);
  -webkit-backdrop-filter: blur(1px) brightness(1000%) grayscale(100%);
  content: "";
  display: block;
  height: 100%;
  position: absolute;
  transform: translateY(-50%);
  width: 100%;
}

Note: I'm using the transform here to make sure the line isn't present before or after transition. There's probably a smarter way to do this but it works.

With the pseudo element in place, we can expand on our transition classes to simply move the position of the line.

/* Start at the top */
.scan-enter::before, .scan-leave::before{
  top: 0%;
}

/* Transition to the bottom */
.scan-enter-to::before, .scan-leave-to::before{
  top: 100%;
}

/* Set timing and ease */
.scan-enter-active::before, .scan-leave-active::before{
  transition: top 2s linear;
}

And that's about it. As a Vue beginner, I was very happy with this result and I think it adds a lot of magic to our project without adding much complexity.