Crashing Into Creative Coding

When a friend of mine approached me to make an interactive audio website for her album project, I was terrified of pursuing this creative project I could easily envision the concept in my mind, but hardly knew how to execute.

In climbing/bouldering, we call each climbing route a ‘problem’; as you climb more regularly and understand your own strengths and limitations, you’re able to look at a problem and see where the most challenging part of it will be for you-- the crux of the problem. Climbers will break the project into sequences measured either by momentum, personal strengths, cadences, and where they think their crux is.

This post outlines how I approached this project, the vision, and broke down the challenge so I could work through taking apart a creative concept into simpler, manageable logistics. I write this in hopes it helps or encourages someone else to pursue new and unfamiliar ideas with a little more confidence and excitement, clearing clouds of doubt so that they may focus on the wonderful and infinite possibilities in the art of creation.

The Idea

A is producing an album for Lin Qiang( Lim Giong), a famous Taiwanese pop star turned experimental musician. A is a forward-thinking millennial, and decides the album release needs a type of digital presence to signal boost the experience and concept of the album into the 21st century world; something like Holodec & Bureau Cool's very cool image-melting AI-powered website Holodec.world but less 'extra'; something that mirrored the physical album cover design but played with the soundscape of each track.

We collaborate and decide on a type of audio visualization: wouldn't it be cool if the audio of each track was visualized through the ripple of each track's image?

The Research

Step One | Identify Your General/Current Unknowns

Once I've gotten the basic non-interactive functions features out of the way using Gatsby.js I'm faced with the following unknowns:

• How do I access data from each track?

• How do I begin learning what I need to know about animating without drowning in all the things I don't know (WebGL and learning C#)?

• How do I adapt these solutions to React hooks?

Step Two | Give Myself General Guidelines as I Begin Research

Because I feel like there's so much I don't know, I try and give myself guidelines.

• Find examples/similar concepts and see what tools they use

• Find and follow creative coders and see what tools they use

• Focus on focusing (Is this something worth reviewing, or do I feel like I'm getting too off track?)

Step Three | Break down Questions Into Smaller Specifics

As you conduct your research, be sure to go back to those initial questions as a way to map out your progress.

• How do I access data from each track?

  • Web audio API allows you to access audio data, and read the sampling of the sound waves in sequences so you can do something with it. You can adjust the rate at which you sample your audio data through AudioBuffers.
  • "Audio visualizations are achieved by accessing an output of audio data over time, usually gain or frequency data... the Web Audio API has an AnalyzerNode that doesn't alter" but instead reads the audio signal passing through it... "can be passed to a visualization technology such as "*
  • To port available vanilla javascript solutions into my React hooks project, I'll have to see which variables need to change, be constant, or be available to pass between audio and animation functions

• How do I begin learning what I need to know about animating without drowning in all the things I don't know (WebGL and learning C#)?

  • CurtainsJS provides a great, narrower scope of what I need in terms of animating images on a canvas element
  • Fundamentals of WebGL/ Curtains include understanding canvases, planes, textures, vertex and shader code
  • I will probably need to take audio data passed through an AnalyzerNode and push it into an animation effect per audio track

• How do I adapt available vanilla javascript solutions into React hooks?

  • There are multiple ways to store shader and vertex code data-- inline and in variables
  • Audio contexts don't need to change on rerenders/track changes, but their sources do.

Don't reinvent the wheel, find a wheel and use it

Let's go back to my guidelines for research to organize and orient myself. I can create a tangible, logical workspace for me to visualize a kind of execution and what it takes to get there: which tools I'll look into learning and using, and which I can ignore.

Step Four | Compile My Resources & Workspace

• Find examples/similar concepts and see which tools they use (part of this is not only figuring out how to execute, but honing your vision: discover standards of interactivity, improvements, or clarifications on how you'd like the animation to perform and interact with viewers).

  • CodePen: Sound Visualization
  • Animate Images and Videos With CurtainsJs (CSS Tricks)
  • Visualizing Music Through Code

• Find and follow creative coders and see what tools they use

  • Zach Lieberman (OpenProcessing)
  • Cassie Evans ( CSS, SVGs)
  • Bureau Cool
  • Martin Laxenaire (CurtainsJs)
  • Greensock
    • Could potentially be something I use, but so general/broad in use cases and animation effects/elements
  • Three.js
    • Originally what I thought I'd end up using, but very steep learning curve
  • OpenProcessing
    • Cool generative art framework, but not quite the fit for my use case/vision
  • WebGL
    • Like ThreeJS, my assumption was I'd use something WebGL, but the learning curve is very steep

• Focus on Focusing: terms, concepts, and tools I've found that may or may not help me; I have to ask 'is this something worth reviewing, or do I feel like I'm getting too off track?'

  • On music/sound: Fourier Transform, Frequency
  • WebGL Fundamentals

Step Five | Focus Your Resources

Examples I'd like to replicate or try (or use pieces of):

• Voice-Change-O-Matic
• A Github repository for a project that distorts your voice and visualizes the effects of audio distortion
• CSS Tricks: Animate Images and Videos with CurtainsJs
  • I keep on coming back to this article-- I like the effects and execution, and it seems to offer a good balance of learning curve + narrower scope of effects
• CurtainsJS Example: A Simple Plane
  • How To: Music Visualizer(Web Audio API) byGina Lee

Helpful References:

• CurtainsJS
• Visualizations with Web Audio API
• Basic Concepts Behind Web Audio API
• Web Audio API

Experimentation

Step Six | Create a Testing Environment

A safe space to experiment, make mistakes, and isolate your problem-solving makes for easier troubleshooting. You don't want outlying variables to block your learning and understanding how different code or frameworks work.

I created a /test page for my project; while most of the static code functionalities (i.e. play list, switching images and tracks when clicking an artist) was up and working on the index page, I didn't want to entangle myself in that existing code + logic.

If you don't want to add things to your project files, go ahead and try in-browser sandboxes like Codepen, Repl.it, or Glitch.

Step Seven | Pre-Map your Progress

Understanding the pieces of code/frameworks I need to learn and comprehend, it's unlikely I'll solve it all at once. Referencing those general initial questions I formulated in Step One, I'll create general milestones and goal markers to measure the progress of my experiments (and to keep me on track).

• Figure out how to use Audio Web API to play/pause a track + extract any type of data from a track/audio element

• Run the CurtainsJS simple plane project on the test page

• Access the audio data output from passing the audio track through the AudioAnalyzer; find what needs to be passed through to the canvas.

Once I've familiarized myself and completed these tasks:

• Successfully connect audio data and animation: have the animation respond to the track as it plays

• Learn and understand how to customize/ tweak the animation effect

And the last step of experimentation, once I've completed all of the above:

• Integrate the audio + animation effects into the rest of the code

If any of these steps become more overwhelming/complicated than you can bear or focus on, I suggest breaking things down even further. This may or may not happen as you continue experimenting!

Execution

And there you have it: a roadmap to execution. You started from nothing, and now you have a solid plan to attempt what you've never attempted before.

The road to success will be a bumpy rollercoaster of emotions: there will be mighty successes followed by a squall of bugs and what feels like failure: but just remember that until you decisively give up, you're just not done yet.

Light for you in dark places when all other lights go out

Looking back, here are a few things that have helped me through challenges:

• When you hit a wall, deep read the code (and re-read it again):
  • If you don't understand a detail, Google and read up. Allow yourself to get lost in a concept for a little bit-- with a time limit.
  • For code you've copied, make sure what you copy is relevant to your project.
  • What does each framework method aim to do? What does it return? How is it called later, and why?
    • A good ol' console.log never hurts anybody
• Compare examples, solutions, and the different ways to skin a cat
  • Specific to my project, I had to sort between different ways to load images onto a plane, or store shader code. What works for the environment/code you're using? Which is a simpler solution?
  • Keep in mind different solutions and examples from different coders are influenced by their own strengths, weaknesses, and backgrounds. Play to your strengths, reuse your experiences, and freak what you got.
• Take breaks!!!
  • Being able to step away and absorb new learnings/bugs/mistakes and concepts helps digest and organize your mental workspace (and re-up your enthusiasm).
  • Talk to someone about it; rubber ducky your challenges out loud.

---

This is a post on the de-abstraction of my process. I'll be updating this post with links to a more technical article on how I executed these steps with code.

In the meantime, check out my completed project at A Pure Person.