DEV Community: Ben Follington

High Performance 3D Animation with React + rxjs

Ben Follington — Fri, 10 Sep 2021 05:02:06 +0000

You know when you load a website and it has a bunch of fancy visualisations that respond to mouse and scroll position with animation? For most of the web's history creating experiences like these has either been impossible or required masochistic determination.

Trying to replicate an apple.com product page is an extreme test of sanity

It used to be difficult to create pretty much any interface in the browser. Efforts like React, Vue, Svelte, Solid and friends have trivialised the jQuery battles of the past. Now we can express our interfaces declaratively, as a function of state -> view.

In fact React has even let us port this idea to the third dimension with react-three-fiber; a wonderful library that uses three.js as a custom React render target.

const ColoredBox = () => {
  const [toggled, setToggled] = useState(false)

  return (
    <mesh onClick={() => setToggled(!toggled)}>
      <boxGeometry args={[1, 1]} />
      <meshStandardMaterial
        color={toggled ? 'blue' : 'red'}
      />
    </mesh>
  )
}

Try panning, scrolling and clicking

This is, in my opinion, staggeringly little code to implement in-browser 3D. We get a lot for free here courtesy of React's Virtual DOM (VDOM) diffing, suspense and state management. However, there is a sticking point.

VDOM style renderers are surprisingly fast given what they do, but sadly they are not quite fast enough for state that changes 60 times a second; the standard for "smooth" animation.

The Future of User Interaction on the Web

I've been wondering, with libraries like react and react-three-fiber combined with rising support for webgl, wasm and wgpu, are we on the path to far richer interactions in the browser? As a game developer I work with a few common game engines and none of them can be considered "declarative". In a typical game the graph of data dependency is far wider and denser than a typical web app and as a result most game engines prioritise performance over clarity. So, how can we get the best of both worlds? Declarative, composable animation logic that reacts to 60hz (minimum!) state updates.

Programmatic animation is a whole sub-specialty within user interface development: tweens, timelines, easing functions, springs, cancellation, the FLIP method... There's a lot of jargon 😵‍💫.

In turn, it's common for us developers to lean on existing libraries to get the job done. framer-motion, react-spring and GSAP are great libraries, but we can learn a lot more about animation by implementing our own approach. What's more almost all animation libraries require us to work with someone else's conceptual model of user input. They provide extension APIs of course but we tend to implement each of these as closed-box concepts, you can consume them but not compose them freely.

A contrived but illustrative example: animating positions of 3D objects that are derived from one another and play sound effects based on their progress. This is difficult to implement in many libraries because of built-in assumptions about the kinds of animation we'd like to do and how to compose them.

Recently I came across samsarajs, a library designed for continuous user interfaces. That is, interfaces that may never be "at rest" and are constantly reacting to changes in data. The project is rooted in functional reactive programming or FRP.

Briefly, FRP is focused on one main concept: the data stream.

Stream (striːm)
A sequence of values distributed over some amount of time

What values? How much time? Those are up to the specific instance. Libraries like rxjs provide an algebra for working with streams, letting us mix them together, pluck out select elements and aggregate data over time. Others have explained streams far better than I can.

In my experience reactions to FRP are mixed. Many people are scared away by its abstract nature, some fear it encourages tightly wound spaghetti code and a dedicated few believe it is the future of programming. I think it's all of the above, FRP is powerful and like any powerful abstraction it is open to abuse. When you have a nuclear-powered ultra-hammer everything looks like an ultra-nail.

Regardless, samsarajs's fundamental insight is that the layout of an application can be modelled as a stream^[ref]. Selfishly, I immediately wondered if I could apply this to my problem.

Animation can also easily be modelled as a stream^[ref], it's almost in the definition:

Animation anɪˈmeɪʃ(ə)n
A series of frames shown in succession over time, creating the illusion of movement.

Combining this with input streams from the user we can create a unified model of user intention -> data mutation -> animated visualisation.

The "dialogue" abstraction, image provided by the cycle.js docs.

This model is heavily inspired by cycle.js which is one of the most mindblowing frameworks around even after 7+ years of development. The cycle described by cycle.js from sources to sinks is a conceptual model that I find myself using in every interface, generative artwork or game I create.

So with all that said, is there a way to use FRP and react-three-fiber to create performant, declarative animations? Let's find out.

Implementation

Alright, here's the meaty part. I'm using React and react-three-fiber for rendering and rxjs to provide our streams. My implementation focuses on a three core concepts:

useObservable: values to animate
interpolator: how to transition between values
useAnimation: performant rendering of animations

useObservable

You might've heard of observables before, the base concept is simple:

Observable əbˈzəːvəbl
A variable which notifies subscribed listeners when the internal value is changed. Listeners can subscribe and unsubscribe from change notifications on demand.

const scale = useObservable(1)

In this case, calling scale.set(2) or scale.swap(x => x + 1) will change the underlying value and send an update event down the scale.changes stream.

const scale = useObservable(1)

scale.changes
     .pipe(filter(x => x > 1))
     .subscribe(x => console.log(`it's ${x}!`));

scale.set(2);
// => it's 2!
scale.set(1);
//
scale.swap(x => x + 1.5);
// => it's 2.5!

In ReactiveX terminology, this is a Subject<T> wrapped up for easy consumption from React.

interpolator

type Interpolator = {
  end: number,
  sample: (t: number) => number
}

const demo: Interpolator = 
  interpolator(0, 1, 'easeOutCubic')

An interpolator acts as a translation layer between different numerical ranges. They typically take the form of functions accepting take a value, t, from 0...1 and output a value of t from 0...1. This might sound familiar if you've heard of easing functions, which are almost ubiquitous in programmatic animation:

Comparison of various common easing functions, via Noisecrime on the Unity Forms

Our interpolators are almost identical except for two important properties:

1. Remapping

const linear = interpolator(0, 1, 'linear')
console.log(linear(0), linear(0.5), linear(1))
// => 0, 0.5, 1

const warped = mapInterpolator(linear, -2, 4)
console.log(warped(0), warped(0.5), warped(1))
// => -2, 1, 4

This is important when we apply an animation. We'll animate values with certain curves between 0...1 but in practice we want to translate that into whatever range is relevant. We might want to animate a box's width between 32px and 400px but until the point of actually applying the animation we can preserve our sanity by using the normalised 0...1 range.

2. Composition

You can combine interpolators in many useful ways. We might want to add them together, subtract them, multiply them or sequence them one after the other.

Currently I've only written the sequence composition, but it demonstrates the principle.

const bounce = sequence(
  interpolator(0, 1.2, 'easeOutCubic'),
  interpolator(1.2, 1, 'easeOutCubic')
)
console.log(bounce(0), bounce(0.5), bounce(1))
// => 0, 1.2, 1

useAnimation

Finally, the hook that connects it all together. useAnimation takes an observable value, an interpolator, the duration in milliseconds and a function to apply the animated value.

useAnimation(scale, bounce, 500, value => {
  mesh.scale.x = mesh.scale.y = value;
})

The value => {} callback is the application site of our side effects, in FRP terms this is known as a sink. Before this function is called all we are doing is changing some numbers in memory over time using an animation curve defined by our interpolator, but our sink is where we connect to our output.

This may feel a little "bare metal" on first inspection, but I would argue this approach is vital for practical usage. A simple adjustment allows us to use this same animation with react-three-fiber or react-dom, retargeting only the binding layer.

const bounce = sequence(
  interpolator(0, 1.2, 'easeOutCubic'),
  interpolator(1.2, 1, 'easeOutCubic')
)
const scale = useObservable(1);

// react-three-fiber
const mesh = useRef();
useAnimation(scale, bounce, 500, value => {
  mesh.current.scale.x = mesh.current.scale.y = value;
});

// react-dom
const element = useRef();
useAnimation(scale, bounce, 500, value => {
    element.current.style.transform = `scale(${value})`;
});

This approach gives us maximum control and flexibility without compromising on performance. You can imagine packaging these value => {} callbacks into common pieces scaleDom, rotateDom, updateShaderUniform etc.

const scaleDom = (el, v) => el.current.style.transform = `scale(${value})`;
const rotateDom = (el, v) => el.current.style.transform = `rotateZ(${value})`;
const setShaderUniform = (shader, uniform, value) => shader.current.uniforms[uniform].value = value;

Here's an example sketch I made using this API (try moving your mouse around, panning, zooming etc.):

The source for this entire article including the above sketch is public on github.

How does useAnimation work?

I'm not ready to publish useAnimation as a library on npm just yet, I'd like to explore the API surface more and put together documentation / examples. That said, you can poke around the sourecode yourself on github and come back if you're confused / curious to know more.

I started with, "what happens when a value we want to animate changes?" Well, we emit a change event on our .changes stream. Okay, so then from that change event we need to start an animation from the current value to the new value. As expressed earlier, an animation is a stream of frames... So we need to get one of those.

Thankfully Subject<T> from rxjs has us covered yet again. If we create a new Subject, we can call .next() on it to emit a new value whenever we want. So, if we combine a Subject with requestAnimationFrame we will have a new value published on every renderable frame the browser gives us.

This is a little gnarly in practice, but luckily I found an example from learnrxjs.com that worked perfectly. My version is in frameStream.ts and is identical except I don't clamp the framerate to 30.

The implementation for react-three-fiber turned out to be more challenging, I ran into issues asking for multiple requestAnimationFrame loops. So, instead, I built on top of useFrame to construct a stream held in a React MutableRef<T> in a similar way:

export const useFrameStream = () => {
  const s = useRef<Subject<number>>(new Subject<number>())
  useFrame(({ clock }) => {
    s.current.next(clock.getDelta())
  })
  return s
}

Okay, so we've got our framestream. Let's look at useAnimation and break it down piece by piece.
We'll start by let's identifying some familiar concepts:

source is the return value of useObservable()
source.changes is the update stream to the underlying value
frame$ is the stream of requestAnimationFrames

export const useAnimation = (
    source: ObservableSource,
    interpolator: Interpolator,
    duration: number,
    sink: (v: Animatable) => void
  ) => {

  // first, store the current animation state seperate to the observed value
  const underlying = React.useRef(source.value())

  React.useEffect(() => {
    // update the render target upon mounting the component
    sink(underlying.current)

    // listen to the update stream from our observable value
    const sub = source.changes
      .pipe(
        // switchMap: the magic operator that enables cancellation
        // our value might change AGAIN mid-animation and
        // we need to cut over to target the updated value
        //
        // switchMap has exactly these semantics, it'll cancel
        // an old stream and replace it with a new one whenever
        // it recieves a value
        switchMap((v) => {
          // capture the time when the animation started
          const baseTime = Date.now()

          return concat(
            // take our frame stream at ~60Hz
            frames$.pipe(
              share(),
              // calculate the % into the total duration we are at
              map((dt) => (Date.now() - baseTime) / duration),
              // only animate while are < 100%
              takeWhile((t) => t < 1),
            ),
            // we append 1 to ensure we send an explicit frame at 100%
            of(1),
            // mapInterpolator warps an interpolator's domain from 0...1
            // to whatever we want
            // here we map [0<->1] to [prev<->current]
          ).pipe(
            map(mapInterpolator(interpolator, underlying.current, v).sample)
          )
        }),
      )
      .subscribe((v) => {
        // finally we store the current value and call
        // the supplied update callback
        underlying.current = v
        sink(v)
      })

    return () => {
      // stop listening for changes when the component unmounts
      sub.unsubscribe()
    }
  }, [duration, source, sink, interpolator])
}

Wrapping Up

As stated above, all the code for this experiment is available on github with an MIT license.

If you want to go deeper again then check out the project README and samsarajs. I'd like to try @most/core instead of rxjs here since it boasts impressive performance^[ref]. To me, this seems like a promising area for further investigation. I've begun to experiment with a similar approach in Unity3d, hopefully more to report soon!

// Detect dark theme var iframe = document.getElementById('tweet-1430922583088799753-302'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1430922583088799753&theme=dark" }

This is the first post from my new project ⊙ fundamental.sh where I'm attempting to document my favourite abstractions and programming patterns. Please don't hesitate to get in touch with me with feedback, ideas for extension or questions. You can find me on twitter, discord (ben#6177) or around the web.

If you'd like to be notified of the next time I write about programming subscribe to the mailing list. I only post when I have something worth saying.

Event-Driven Game Development in Unity

Ben Follington — Wed, 05 Jun 2019 00:00:00 +0000

I’m going to spend a little time introducing a different way of looking at structuring your games in Unity. My code examples will apply to Unity specifically but this pattern can be applied in any engine or language and applies in almost any context, not just game development.

So, you’ve read the title and you might be wondering: what is event-driven programming? In brief, it is taking the perspective that what happens in an application is a series of events and structuring the code around that belief. These events act as inputs and outputs between your various systems, i.e. Jump Button Pressed, Player Landed On Ground, Sound Effect Played. In regular old unity code this is what your method calls usually represent:

void Update() {
  if (JumpButtonPressed()) {
    Jump();
  }
}

void LandedOnGround() {
  LandSound.Play();
}

This is simple & readable when everything can be captured in the same GameObject. What about a more complex example? Let’s say we have an Enemy who fires Bullets at the Player. If a Bullet hits a Player then we want the Player to lose HP (and their health bar to update) and a sound to play. We also want the Enemy to celebrate if they successfully shoot the player.

// Enemy.cs

private float _timer = 0;

void Update() {
  _timer += Time.deltaTime;

  if (_timer > 2) {
    var bulletObject = GameObject.Instantiate(BulletPrefab, transform.position, Quaternion.identity);
    bulletObject.GetComponent<Bullet>().SetSourceEnemy(this);
    _timer = 0;
  }
}

public void BulletHit() {
  Animator.Play("Celebrate");
}

// Bullet.cs

void OnCollisionEnter2D(Collision2D col) {
  var player = col.gameObject.GetComponent<Player>();
  if (player) {
    player.Hit();
    sourceEnemy.BulletHit();
  }
}

// Player.cs

void Update() {
  if (JumpButtonPressed()) {
    Jump();
  }
}

public void Hit() {
  Hp -= 2;
  HealthBar.Update(Hp);

  HurtSound.Play();
}

void LandedOnGround() {
  LandSound.Play();
}

This is alright code but we’re starting to introduce some very tight coupling between our scripts. Our bullet has to know all about both our player and the enemy. Worse still, our player class has to know about the UI! This is pretty much the opposite of the Single Responsibility Principle (the S in SOLID). There are a lot of different modules here that need splitting up:

Bullets - a common concern, should be handled by a BulletManager that can handle object pooling
UI - user interfaces should not be directly linked to game entities
Sounds - a PlayerSoundEffects manager should control how sound effects are played, whether they interrupt one another and global volume settings
Enemy - just needs to shoot and be notified if the shot hit something
Player - jump in response to jump button and react when hurt

Instead what if we introduced a mediator here? Something that could buffer the communication between the different systems in our game. I’m going to call this an EventManager. Our EventManager is going to let us define different event types and allow different components in our game to subscribe to be notified of those events. Have a look at my implementation if you’re interested, this will eventually be published as part of some open-source TwoPM branded Unity toolkit. If you open that link and feel slightly overwhelmed, don’t worry so did I when I wrote it and the internal workings of it aren’t important for this article.

// Events.cs

public class PingEvent {}
public class PongEvent {}

// Pinger.cs

void Start() {
  EventManager.AddListener<PongEvent>(
    (ev) => {
      Debug.Log("Pong");
    }
  );
}

void OnClicked() {
  EventManager.Dispatch<PingEvent>();
}

// Ponger.cs

void Start() {
  EventManager.AddListener<PingEvent>(
    (ev) => {
      Debug.Log("Ping");
      EventManager.Dispatch<PongEvent>();
    }
  );
}

When we run this and click on the Pinger GameObject, we’ll see the following output:

> "Ping"
> "Pong"

So, how would this look for our bullet example above?

Woah, that just got a lot more complicated. There are lots more moving parts and, as you might expect, this is going to translate into more lines of code. This pattern is a great example of the fact that lines of code has almost no relationship to the difficulty of understanding and working on a program.

Think of it like this: our default way of setting things out is a big network of objects that all know about one another and talk directly to each other. But, with our EventManager we can instead focus on one little sub-problem at a time. I don’t want to think about sound effects when I’m trying to redesign my user interface etc. Instead of thinking in objects, try thinking in behaviours. Our player is just a collection of behaviours:

// Behaviour: Update HealthBar
PlayerHitByBullet → PlayerHpUpdated → UiHealthBarUpdated

// Behaviour: Play Sounds
PlayerHitByBullet → PlayerHurtSoundPlayed 
LandedOnGround → PlayerLandedOnGroundSoundPlayed

// Behaviour: Jump
JumpPressed → PlayerJumped

Whether these behaviour exist all on one object or scattered around the scene, it doesn’t matter. This lets us split up our code by what it actually does. So what does the our example actually look like? I’ll focus on just Player, HealthBar, Enemy, BulletManager and Bullet for the example.

// Player.cs

void Start() {
  EventManager.AddListener<JumpPressed>(
    (ev) => {
      Jump(); // Let's pretend this exists
      EventManager.Dispatch<PlayerJumped>();
    }
  );

  EventManager.AddListener<PlayerHitByBullet>(OnHit);
}

private void OnHit(PlayerHitByBullet e) {
  Hp -= 2;
  EventManager.Dispatch(new PlayerHpUpdated(Hp));
}

// HealthBar.cs

void Start() {
  EventManager.AddListener<PlayerHpUpdated>(
    (ev) => {
      HealthBarDisplay.Value = ev.Hp;
      EventManager.Dispatch<HealthBarUpdated>();
    }
  );
}

// Enemy.cs

private float _timer = 0;

void Start() {
  EventManager.AddListener<PlayerHitByBullet>(
    (ev) => {
      // Did the bullet THIS ENEMY fired hit?
      if (ev.Source == this) {
        Animator.Play("Celebrate");
      }
    }
  );
}

void Update() {
  _timer += Time.deltaTime;

  if (_timer > 2) {
    EventManager.Dispatch(new BulletFired(this));
    _timer = 0;
  }
}

// BulletManager.cs

void Start() {
  EventManager.AddListener<BulletFired>(OnBulletFired);
}

private void OnBulletFired(BulletFired e) {
  var bulletObject = GameObject.Instantiate(BulletPrefab);
  bulletObject.GetComponent<Bullet>().SetSourceEnemy(e.Source);
}

// Bullet.cs
private Enemy _source;

void OnCollisionEnter2D(Collision2D col) {
  var player = col.gameObject.GetComponent<Player>();
  if (player) {
    EventManager.Dispatch(new PlayerHitByBullet(_source));
  }
}

Phew. We’re starting to get the picture now! While this is a decent amount of code it’s actually pretty easy to consume. One of the best things about this architecture is that if you take any one of those files and read it, you don’t actually need to check any other files to understand the logic. We can also easily extend this system without modifying any of our original code! If we take the above as-is and want to add sound effects we can do all that in a new component.

// PlayerSoundEffects.cs

void Start() {
  EventManager.AddListener<PlayerHitByBullet>(
    (ev) => PlayerHurtSound.Play()
  );
}

That’s it. No other changes and we get to stay focused on just our current small task.

Hopefully after that small demonstration you’re feeling at least a little positive about this way of doing things. Let’s recap the benefits we’ve seen and touch on some others.

Takeaways

Loose Coupling

I don’t think I need to say much about why loose coupling is good. This one of the most fundamental goals of software architecture: keep modules as isolated as possible. This, in turn, allows us to engage the wisdom of “Favour composition over inheritance”. Unity already pushes us into thinking compositionally due to its entity-component structure. We can drop new components on to GameObjects and augment their behaviour in a piecemeal way.

By adopting event-driven architecture we can take this pattern further, stepping outside the restriction of composition only within one GameObject. We can add, change and remove behaviours & restructure our scene graph freely, without having to rework code and battle broken object references.

Extensibility

Let’s say you’re going to get your game on Steam. You’ve decided you want to add achievements! One of those achievements is going to be for jumping 10 times. Now, we could add a jumpCounter to Player and increment it each time, then unlock the achievement:

// Player.cs

private int _jumpCounter = 0;

private void Jump() {
  // ... Jump logic ...

  _jumpCounter++;

  if (_jumpCounter >= 10) {
    SteamAchievements.Unlock("JumpedALot");
  }
}

But this is really ugly. Now our Player class has to know about Steam achievements! What about if we need to disable achievements if the game isn’t being run from Steam? What if we want to add logic to persist the _jumpCounter between play sessions?

Instead, with events, we can just add a new JumpAchievementTracker.

// JumpAchievementTracker.cs

private float _counter = 0;

void Start() {
  EventManager.AddListener<PlayerJumped>(
    (ev) => {
      _counter++;
      CheckAchievement();
    }
  );
}

private void CheckAchievement() {
  if (_counter > 10) {
    SteamAchievements.Unlock("JumpedALot");
  }
}

This leaves our Player code totally isolated and gives us a convenient spot to add any persistence logic. We can also easily enable and disable this behaviour by toggling JumpAchievementTracker on and off!

Testing

Games are notoriously hard to write tests for. We often end up with so many interlaced dependencies that tests become a mess of mocking code. Now, to easily and effectively test Unity code it should not be part of a MonoBehaviour (since you cannot easily run MonoBehaviours outside of Unity). I chose to keep everything in MonoBehaviour form above for simplicity but there are better approaches (involving Inversion of Control and Dependency Injection) which I’ll touch on in the future. Let’s take a modified example for testing:

// Player.cs

void Start() {
  PlayerController.Init(EventManager);
}

// PlayerController.cs 

public class PlayerController {
  private bool _jumping = false;
  public bool IsJumping => _jumping;

  public void Init(EventManager eventManager) {
    eventManager.AddListener<JumpPressed>(
      (ev) => {
        Jump();
        eventManager.Dispatch<PlayerJumped>();
      }
    );
  }

  public void Jump() {
    _jumping = true;
  }
}

// PlayerControllerTests.cs

public class PlayerControllerTests {

  [Test]
  public void ItJumpsWhenInputPressed() {
    // Arrange
    var em = new EventManager();
    var controller = new PlayerController();

    controller.Init(em);

    var called = false;

    // Act
    em.AddListener<PlayerJumped>((ev) => called = true);
    em.Dispatch<JumpPressed>();

    // Assert
    Assert.IsTrue(called);
    Assert.IsTrue(controller.IsJumping);
  }

}

What’s Next?

That’s enough for one day! In the future I plan to cover how this mixes with:

Inversion of Control & Dependency Injection with Zenject
Coroutines & Cutscenes
Observables (via UniRx)
Data Aggregation and Persistence

Resources

EventManager.cs
Taking this further with Event Sourcing