Zille Rehman

Posted on Aug 6

Building a Production-Ready Pakistan Sign Language Recognition Model (89% Test Accuracy with EfficientNetB0)

#deeplearning #flutter #machinelearning #mobile

Most ML tutorials stop at "here's my accuracy score." That number means very little if the model was never tested against how it will actually be used. While building the alphabet-recognition model for my Pakistan Sign Language (PSL) Flutter app, I learned this the hard way — my first "95% accurate" version was quietly cheating, because near-duplicate images from the same recording session were leaking between my train and test sets.

This post walks through how I rebuilt the pipeline properly: honest data splitting, transfer learning with EfficientNetB0, per-class evaluation, and exporting a model that's actually ready for a mobile app.

TL;DR

40 PSL Urdu alphabet classes, ~22.8K images (originals + augmented + video-extracted frames)
Group-based train/val/test split to eliminate data leakage
EfficientNetB0 transfer learning, fine-tuned in two stages
89.2% test accuracy / 93.0% balanced accuracy on a genuinely unseen test set
Exported straight to .tflite + label maps for a Flutter app

The Goal

Recognize Pakistan Sign Language Urdu alphabets from images and ship a model a Flutter app can actually run on a phone. That meant optimizing for three things beyond raw accuracy:

Reliability on unseen data — not just the validation set
Deployability — small, fast, TFLite-compatible
Debuggability — clear per-class metrics so weak signs can be targeted with more data later

Dataset Preparation

The raw dataset combined three sources: original photographed images, augmented (generated) images, and frames extracted from short videos — 40 normalized alphabet classes, roughly 22,700 raw rows before frame extraction.

The trap here is that these sources aren't independent. An augmented image and its original are near-duplicates. Frames pulled from the same 2-second video clip are nearly identical to each other. If any of that ends up split across train and test, the model isn't being tested on unseen data — it's being tested on a slightly blurred copy of something it already memorized.

So instead of a random image-level split, I split by group_id = label + source sample, keeping every image that traces back to the same physical sample together on one side of the split:

def split_groups_within_label(group_df: pd.DataFrame, val_ratio: float = 0.15, test_ratio: float = 0.15):
    groups = sorted(group_df['group_id'].drop_duplicates().tolist())
    random.Random(SEED).shuffle(groups)
    total = len(groups)
    n_test = max(1, round(total * test_ratio)) if total >= 3 else 0
    n_val = max(1, round(total * val_ratio)) if total >= 3 else 0
    # ...assign whole groups to train/val/test, never split within a group

Preventing Data Leakage

Three rules made the difference between an optimistic score and a trustworthy one:

Split before extracting video frames — frames are generated after the split decision, using the split their parent video was already assigned to.
Augmented images stay in train only if their source sample is also in train — otherwise they're dropped, since an augmented twin of a test image is still leakage.
Whole groups move together — no group is ever split across train/val/test.

This approach costs you a few points of headline accuracy compared to a naive random split. It's worth it: the resulting number reflects how the model behaves on a sign it has genuinely never seen, which is the only number that matters once the model is in someone's phone camera.

Model: EfficientNetB0 with Two-Stage Transfer Learning

Rather than training a CNN from scratch, I used EfficientNetB0 pretrained on ImageNet as a frozen feature extractor first, then unfroze the top of the network for fine-tuning:

base_model = tf.keras.applications.EfficientNetB0(
    include_top=False, weights='imagenet', input_shape=IMG_SIZE + (3,)
)
base_model.trainable = False

inputs = tf.keras.Input(shape=IMG_SIZE + (3,))
x = data_augmentation(inputs)
x = tf.keras.applications.efficientnet.preprocess_input(x * 255.0)
x = base_model(x, training=False)
x = tf.keras.layers.GlobalAveragePooling2D()(x)
x = tf.keras.layers.Dropout(0.35)(x)
x = tf.keras.layers.Dense(256, activation='swish')(x)
x = tf.keras.layers.Dropout(0.25)(x)
outputs = tf.keras.layers.Dense(num_classes, activation='softmax')(x)

Training happened in two stages:

Head-only training — base frozen, only the new dense layers learn, AdamW at 3e-4.
Fine-tuning — unfreeze all but the first ~80 layers of EfficientNetB0, drop the learning rate to 5e-5, and continue with label smoothing, EarlyStopping, and ReduceLROnPlateau.

Class weights were computed and passed into both stages to counter the class imbalance baked into the dataset (some letters had 12 test samples, others had 180+).

Results

On the held-out, leakage-free test set:

Metric	Score
Test accuracy	89.22%
Balanced accuracy	93.04%

Balanced accuracy sitting above raw accuracy is a good sign here — it means the model isn't just riding the majority classes; it's doing well even on the alphabets with few samples.

Per-class results told a more useful story than the headline number: most letters hit perfect or near-perfect precision/recall (ح, ب, ے, چ, ھ, ء, ق, ل all at 1.00), while a handful — like د at 0.50 precision — stood out as weak and in need of more real (non-augmented) samples. That's the actual output of this pipeline: not just a score, but a punch list of which signs to collect more data for next.

Reliable Label Mapping (English folders → Urdu display)

The dataset folders use English names (Bay, Zuey, 1-Hay), but the app needs to display Urdu characters. A silent mistake here — two folders mapping to the same Urdu symbol, or a missing entry — would fail quietly and confuse users. So the mapping is validated before export ever runs, and export is blocked if two class names map to the same symbol:

LABEL_MAP = {
    '1-Hay': 'ح', '2-Hay': 'ہ', 'Ain': 'ع', 'Alif': 'ا',
    'Alifmad': 'آ', 'Aray': 'ڑ', 'Bay': 'ب', 'Byeh': 'ے',
    # ...
}
assert_unique_label_map(class_names)  # export stops if this fails

Preparing for Flutter Deployment

Training the model was step one; the notebook also produces everything the Flutter app needs directly:

psl_urdu_alphabet_model.tflite — the exported model
labels.txt / labels_map.json — predicted class index → Urdu character
training_metadata.json, per_class_metrics.csv, top_confusions.csv — for debugging weak predictions later

Treating deployment artifacts as a first-class output of the notebook — not something figured out after the fact — meant the Flutter integration step didn't need to touch the training code at all.

Lessons Learned

A clean, leakage-free dataset beats a bigger, dirtier one.
Balanced accuracy told me more than raw accuracy about whether the model was actually generalizing.
Two-stage transfer learning (frozen head → careful fine-tune) is a big time saver over training from scratch, even on a modest dataset.
Validating the label map before export catches a whole category of "why is the app showing the wrong letter" bugs before they happen.
Deployment artifacts should come out of the same notebook that produces the model, not a separate afterthought step.

What's Next

I'm now working on real-time PSL recognition and integrating this alongside a word-level (BiLSTM + attention) model into the same Flutter app, so it can recognize both individual alphabets and full signed words.

If you're working on sign language recognition, accessibility tooling, or just fighting your own data-leakage bugs — I'd love to hear how you handled it. Drop a comment below!

DEV Community