DEV Community

Ayush Shah
Ayush Shah

Posted on

I Built an Adaptive Hand Cricket Game Using React, MediaPipe, and TensorFlow.js

#gamedev #javascript #machinelearning #react

I wanted to build something fun, not another CRUD app or chatbot.

So I built Adaptive Hand Cricket: a browser-based game where you play hand cricket through your webcam, your hand gestures are recognized in real time, and the computer gradually becomes harder by combining randomness, live gesture reading, and machine learning.

How it works

The game uses:

  • React + Vite for the frontend
  • MediaPipe for real-time hand landmark detection
  • TensorFlow.js for learning player patterns
  • Webcam input through getUserMedia
  • localStorage to remember previous batting behavior

The supported gestures are:

1 -> index
2 -> index + middle
3 -> index + middle + ring
4 -> index + middle + ring + pinky
5 -> all fingers
6 -> thumb
7 -> thumb + index
8 -> thumb + index + middle
9 -> thumb + index + middle + ring
Enter fullscreen mode Exit fullscreen mode

Instead of simply counting fingers, I classify the exact finger pattern.

The game loop

Every delivery follows:

Computer commits to a move
        ↓
        3
        2
        1
        ↓
      SHOW
        ↓
Human gesture is locked
        ↓
Both moves are revealed
        ↓
Score / wicket is calculated
Enter fullscreen mode Exit fullscreen mode

This keeps the reveal synchronized like actual hand cricket.

Making the opponent adaptive

A completely random computer got boring quickly, so I added three decision strategies:

RANDOM
ML
PEEK
Enter fullscreen mode Exit fullscreen mode

Difficulty automatically increases with the player's score:

0-39   -> EASY
40-99  -> MEDIUM
100+   -> HARD
Enter fullscreen mode Exit fullscreen mode

Each level changes how often the opponent uses each strategy.

For example:

EASY: {
  peekChance: 0.10,
  mlChance: 0.30,
  randomChance: 0.60
}
Enter fullscreen mode Exit fullscreen mode

Peek strategy

Humans often form their gesture before the countdown reaches zero.

On some deliveries, the computer is secretly allowed to inspect the gesture at a random point during:

3... 2... 1...
Enter fullscreen mode Exit fullscreen mode

If it gets a valid reading, it can use that number.

Slightly evil. Very effective.

Learning player patterns

The more interesting strategy uses TensorFlow.js.

Every batting move is stored:

7, 4, 2, 7, 8, 4, 7...
Enter fullscreen mode Exit fullscreen mode

The model learns from:

  • the player's last 5 moves
  • overall frequency of numbers 1-9
  • recent frequency of numbers 1-9

This produces 63 input features.

The network is intentionally small:

63 inputs
   ↓
Dense 64
   ↓
Dense 32
   ↓
9-way Softmax
Enter fullscreen mode Exit fullscreen mode

The output might look like:

4 -> 18%
7 -> 42%
8 -> 12%
...
Enter fullscreen mode Exit fullscreen mode

So the model predicts 7 as the player's next move.

Training starts after enough moves have been collected and retrains periodically rather than after every ball.

What surprised me

The machine learning was not actually the hardest part.

The trickier problems were:

  • thumb detection across different hand orientations
  • synchronizing countdown and capture timing
  • preventing invalid camera frames from affecting gameplay
  • managing game transitions cleanly
  • keeping browser ML from interfering with real-time video processing

A small game ended up touching computer vision, React state management, probability, feature engineering, browser ML, and real-time interaction design.

What I'd improve next

I'd like to add:

  • a short multi-frame voting window instead of using one camera frame
  • proper ML-vs-random prediction accuracy tracking
  • player profiles
  • a comparison between neural networks and simpler approaches like Markov chains
  • better gesture classification for rotated hands

Tech Stack 

React
Vite
MediaPipe Tasks Vision
TensorFlow.js
WebRTC / getUserMedia
localStorage
Enter fullscreen mode Exit fullscreen mode

It started as a simple question:

Can I play hand cricket against my webcam?

It turned into:

Can the computer learn how predictable I am?

Top comments (0)