Dancing Pixels: Building an Immersive Audio-Reactive 3D Web Experience with React Three Fiber

This article was co-authored by @shnnkwhr

---

We are currently living through one of the most exciting eras of browser graphics in history.

With the real-time rendering tools available today, a solo developer can build interactive 3D experiences that once required an entire studio. We can stream audio, analyze frequencies, animate characters, and run GPU shaders, all inside a single tab. The barrier between imagination and execution has almost disappeared.

But most web experiences are still static. We scroll. We click. We consume. What if the browser didn’t just display content?

What if it sparked? it listened?

Meet LowCortiSparcs, a real-time audio visualizer built using today’s browser graphics technologies, removing the barrier between imagination and execution.

This project demonstrates a modern, end-to-end pipeline for creating interactive, audio-driven 3D web experiences. The workflow begins in Blender, where base character models are created, rigged, cleaned up, and optimized for web performance. These are then exported to Adobe Mixamo for automated skeletal rigging and the application of motion-capture animations. Once the animated characters are brought back into Blender for final web preparation (exporting as lightweight .glb files), they are integrated into a NextJS/React frontend architecture powered by Three.js. At the core of the experience is the Web Audio API, which captures sound frequencies in real-time through an AnalyserNode, feeding that data directly into the React render loop (useFrame). This real-time data dynamically modulates the character's animation speed and mesh scaling, creating a music-reactive visualizer directly in the browser.

Understanding the WebGL Stack (Three.JS)

To bring these 3D assets to the browser, we rely on Three.js. While web browsers natively use WebGL (a JavaScript API) to render 2D and 3D graphics, writing raw WebGL requires complex math and verbose code. Three.js perfectly bridges this gap, since it is a powerful JavaScript library that abstracts away the heavy lifting, allowing developers to create 3D scenes with significantly less code. A standard Three.js project is built upon 3 core components:

• the Scene, which acts as a master container operating on a 3D Cartesian coordinate system to hold all objects, lights, and backgrounds;
• the Camera, which serves as the viewer's "eyes" and determines what is visible using specific projections (like perspective or orthographic); and
• the Renderer, which compiles the scene and camera perspectives to draw the final result onto an HTML <canvas> element.

To actually display objects within this scene, Three.js uses a Mesh, which pairs two essential properties:

• the Geometry, which defines its physical mathematical shape; and
• the Material, which defines its surface appearance, color, and reactions to light.

Step-by-Step Guide on Building the Project

1. Prerequisites

Before we dive into the code, here is what you need to understand the stack:

• React & Hooks: Familiarity with useState , useEffect , and useRef .
• Three.js Basics: A conceptual understanding of scenes, cameras, meshes, and materials.
• Node.js: Installed on your machine (to run Vite and handle dependencies).

Our main libraries:

• @react-three/fiber : A React reconciler for Three.js.
• @react-three/drei : Useful helpers for R3F (like useFBX and Environment maps).
• zustand : For lightweight global state management

2. Project Setup and System Architecture

3D projects can get messy fast. To keep things lightweight and maintainable, we separate our 3D elements from our audio logic and global state.

3. Building the 3D Model

In order to bring a 3D character to life, we first need the 3D character itself.

Do not worry if you’re not a 3D artist or if you don’t have Blender! For this guide, we’ve provided the files so you can jump straight into coding the visualizer.

Modeling the Mesh

Open Blender and enter “Edit Mode” to start creating our model. Transform a simple cube by using tools such as extrude to create limbs, scale to edit proportions, and loop cuts to add joints or you can simply find models with rigs from Adobe Mixamo.

Texturing and Materials

A 3D model is flat and gray by default. To give it color and depth, we have to assign materials and textures to its faces, giving it visual identity.

The Rigging Process

At this point, we already have the “skin” of our model, but it is still static and unable to move. To fix that, we have to give it a skeleton first. Add an armature, which is a digital skeleton, to the parts that you want to move, such as the model’s arms, legs, and body. A critical step after adding armature is weight painting. This defines which parts and how much of the “skin” should move whenever a “bone” in the model moves.

With the mesh complete and armature in place, our 3D model is now ready to be animated in Adobe Mixamo.

You can get access to the assets here: Asset

4. Sparking up the Heartbeat

Before we can make anything react to sound, we need the sound data itself. Browsers provide a powerful, native Web Audio API for exactly this purpose.

We wrap this logic in a singleton class: AudioController.ts

Creating the Analyzer
To get audio data, we need two main parts: an AudioContext and an AnalyserNode.

// src/core/audio/AudioController.ts

class AudioController {
  context: AudioContext | null = null;
  analyser: AnalyserNode | null = null;
  source: AudioBufferSourceNode | null = null;
  frequencyData: Uint8Array;

  init() {
    // Initialize the Web Audio API context
    this.context = new (
      window.AudioContext || (window as any).webkitAudioContext
    )();

    // Create an AnalyserNode to extract frequency data
    this.analyser = this.context.createAnalyser();

    // FFT (Fast Fourier Transform) size determines the frequency resolution.
    // Higher = more detailed data, but heavier processing.
    this.analyser.fftSize = 256;

    const bufferLength = this.analyser.frequencyBinCount;

    // An array to hold the live frequency data
    this.frequencyData = new Uint8Array(bufferLength);
  }
}

The AnalyserNode acts as a pass-through node that extracts time and frequency data from the audio signal without altering the output.

Extracting the Data
Next, we need a method where the 3D scene can poll 60 times a second to get the current audio levels. This is to ensure that the 3D scene is in sync with the audio, reacting to every vibration of the track without missing a beat.

// src/core/audio/AudioController.ts

getAudioData() {
  if (this.analyser && this.frequencyData) {
    // Populate the array with current frequency data (values 0-255)
    this.analyser.getByteFrequencyData(this.frequencyData);

    // Calculate the average volume of all frequencies combined
    let sum = 0;

    for (let i = 0; i < this.frequencyData.length; i++) {
      sum += this.frequencyData[i];
    }

    const average = sum / this.frequencyData.length;

    return {
      frequencyData: this.frequencyData, // Array for specific bands
      average: average // Single numeric value for overall volume
    };
  }

  return { frequencyData: new Uint8Array(0), average: 0 };
}

By returning average , we have a unified metric (0-255) representing how "loud" the song is at any exact millisecond!

5. Setting up the Low Cortisol Environment

Now that we have audio data, we need a place to put our 3D objects. VisualizerCanvas.tsx is where the declarative magic of React Three Fiber shines.

// src/components/visualizer/VisualizerCanvas.tsx

import { Canvas } from '@react-three/fiber';
import { OrbitControls, Environment, ContactShadows } from '@react-three/drei';
import { Suspense } from 'react';
import FBXVisualizer from '../../features/visualizers/FBXVisualizer';
import FloatingFlowers from '../../features/visualizers/FloatingFlowers';

const VisualizerCanvas = () => {
  return (
    <div className="w-full h-screen bg-black">
      <Canvas camera={{ position: [0, 5, 20], fov: 45 }}>
        {/* Scene Setup */}
        <color attach="background" args={['#050505']} />

        {/* Cyberpunk Lighting */}
        <ambientLight intensity={1.5} />
        <directionalLight position={[10, 20, 10]} intensity={2} />
        <pointLight
          position={[-10, -10, -10]}
          intensity={1}
          color="#ff0080"
        /> {/* Pink */}
        <pointLight
          position={[10, 10, 10]}
          intensity={1}
          color="#00ffff"
        /> {/* Cyan */}

        {/* 3D Elements */}
        <Suspense fallback={null}>
          <FloatingFlowers count={150} />
          <FBXVisualizer />

          {/* Realistic Reflections and grounded shadows */}
          <Environment preset="city" />
          <ContactShadows
            position={[0, -5, 0]}
            opacity={0.5}
            scale={50}
            blur={2}
          />
        </Suspense>

        {/* Allow the user to drag the camera */}
        <OrbitControls makeDefault />
      </Canvas>
    </div>
  );
};

export default VisualizerCanvas;

Using <Suspense> is critical here. Since loading heavy 3D models (useFBX) is asynchronous, Suspense ensures React waits for the assets to download and parse before trying to render them.

6. Giving the 3D Model the LowCortiSpark

This is where the magic happens. We load a baked skeletal animation and tie its speed directly to the audio average we built earlier.

// src/features/visualizers/FBXVisualizer.tsx

import { useRef, useEffect } from 'react';
import { useFrame } from '@react-three/fiber';
import { useFBX } from '@react-three/drei';
import * as THREE from 'three';
import { audioController } from '../../core/audio/AudioController';
import { useAudioStore } from '../../store/useAudioStore';

const FBXVisualizer = () => {
  // Load the FBX model
  const fbx = useFBX('/Hip Hop Dancing (1).fbx');
  const groupRef = useRef<THREE.Group>(null);
  const mixerRef = useRef<THREE.AnimationMixer | null>(null);
  const isPlaying = useAudioStore((state) => state.isPlaying);

  // Setup animation mixer
  useEffect(() => {
    if (!fbx) return;

    if (fbx.animations && fbx.animations.length > 0) {
      const mixer = new THREE.AnimationMixer(fbx);
      const action = mixer.clipAction(fbx.animations[0]); // play first animation
      action.play();
      mixerRef.current = mixer;
    }

    return () => mixerRef.current?.stopAllAction();
  }, [fbx]);

  // Audio-reactive loop
  useFrame((_, delta) => {
    if (!groupRef.current) return;

    const { average } = audioController.getAudioData();

    // Scale dancer based on volume
    const baseScale = 0.02;
    const targetScale = baseScale + (average / 255) * 0.01;
    groupRef.current.scale.lerp(new THREE.Vector3(targetScale, targetScale, targetScale), 0.1);

    // Adjust animation speed with volume
    if (mixerRef.current) {
      let targetSpeed = 0;
      if (isPlaying) {
        targetSpeed = 1.0 + (average / 255) * 0.5; // louder → faster
      }
      mixerRef.current.timeScale = THREE.MathUtils.lerp(mixerRef.current.timeScale, targetSpeed, 0.1);
      mixerRef.current.update(delta);
    }
  });

  return (
    <group ref={groupRef} position={[0, -2, 0]}>
      <primitive object={fbx} />
    </group>
  );
};

Breaking down the useFrame hook
In React Three Fiber, useFrame executes roughly 60 times a second (the requestAnimationFrame loop). By fetching getAudioData() inside this loop, we are constantly getting the newest amplitude data. We then manipulate mixerRef.current.timeScale . By altering timeScale , we are literally speeding up or slowing down the playback of the skeletal animation based on how loud the song is.

7. Creating Infinite Ambience: Instanced Particles

To truly sell the immersive experience, we need background elements. Instead of rendering 150 individual mesh components (which would destroy framerates), we use THREE.InstancedMesh .

// src/features/visualizers/FloatingFlowers.tsx

import { useRef, useMemo } from 'react';
import { useFrame } from '@react-three/fiber';
import * as THREE from 'three';
import { audioController } from '../../core/audio/AudioController';

const FloatingFlowers = ({ count = 150 }) => {
  const meshRef = useRef<THREE.InstancedMesh>(null);
  const dummy = useMemo(() => new THREE.Object3D(), []);

  // Generate random particles behind the dancer
  const particles = useMemo(() => {
    const temp = [];
    for (let i = 0; i < count; i++) {
      temp.push({
        x: (Math.random() - 0.5) * 60,
        y: (Math.random() - 0.5) * 60,
        z: -15 - Math.random() * 30, // far in the background
        scale: Math.random() * 0.8 + 0.2,
        speed: Math.random() * 0.01 + 0.005,
      });
    }
    return temp;
  }, [count]);

  // Animate particles
  useFrame(() => {
    if (!meshRef.current) return;

    const { average } = audioController.getAudioData();
    const audioBoost = (average / 255) * 0.05; // boost speed on beat

    particles.forEach((particle, i) => {
      // Float upwards, faster on louder beats
      particle.y += particle.speed + audioBoost;

      // Reset to bottom if too high
      if (particle.y > 30) particle.y = -30;

      dummy.position.set(particle.x, particle.y, particle.z);

      // Pulse size with music
      const audioScale = particle.scale + (average / 255) * 0.5;
      dummy.scale.set(audioScale, audioScale, audioScale);

      dummy.updateMatrix();
      meshRef.current.setMatrixAt(i, dummy.matrix); // apply transform
    });

    meshRef.current.instanceMatrix.needsUpdate = true; // notify Three.js
  });

  return (
    <instancedMesh ref={meshRef} args={[null, null, count]}>
      <torusKnotGeometry args={[0.5, 0.15, 64, 8]} />
      <meshStandardMaterial
        color="#ff0080"
        emissive="#ff0080"
        emissiveIntensity={0.5}
        roughness={0.2}
        metalness={0.8}
      />
    </instancedMesh>
  );
};

This single <instancedMesh> draws 150 highly-detailed Torus Knots in one batch. Just like the dancer, their vertical velocity ( audioBoost ) and their physical size ( audioScale ) are mathematically tethered to the audio average.

Conclusion

Congratulations! By following this guide, you've successfully transformed a static 3D model into a digital performer through React Three Fiber and the Web Audio API, bridging the gap between raw sound data and 3D geometry.

But, this is just the first step. The real-time audio visualizer we built in this tutorial can still be taken further, whether you want to change the colors of the particles based on the audio frequency, or shake the camera when the sound reaches a certain threshold, the possibilities are all up to you!

Now that you've stepped into the realm of creative coding, make the web browser your playground and see what else you can bring to life.

If you want to follow through the Github repository or analyze the spark of the code itself, you may do so here! LowCortiSparcs