DEV Community: Wlad Radchenko

Why lip-sync blacks out the bottom half of the face before the model sees it

Wlad Radchenko — Wed, 15 Jul 2026 14:00:00 +0000

A lip-sync model has one job: draw a mouth that matches the audio. So the obvious input is a face and a sound. Feed in the picture, feed in the speech, get back a talking face.

That is not what the code feeds it. Before a single frame reaches the model, the bottom half of every face is set to zero. Pure black. The model never sees the real mouth it is supposed to reproduce.

The first time I read this in the data generator, I assumed it was a bug or some leftover debug code. It is neither. Blanking the lower face is the whole reason the model learns to lip-sync instead of learning to copy. This post walks through the few lines that build that input, in datagen, the generator that batches frames and audio for Wav2Lip.

Give the model the lit half and ask it to invent the rest. Photo: Unsplash.

The setup: frames in, mel chunks in

datagen is a generator. It takes a list of frame file paths, a list of mel-spectrogram chunks (the audio, sliced into short pieces), the target face size, and a batch size. It yields batches the model can run.

def datagen(self, frame_files, mels, img_size, wav2lip_batch_size, crop=None):
    img_batch, mel_batch, frame_batch, coords_batch = [], [], [], []
    face_det_results = self.face_detect_with_alignment_crop(frame_files, crop)

First it runs face detection once over all the frames, so it knows where every face sits. Then it walks the audio, not the video:

for i, m in enumerate(mels):
    idx = i % len(frame_files)

That idx = i % len(frame_files) is small but it sets the rhythm of the whole loop. The driver is the audio. There is one mel chunk per output frame, and each chunk needs a face to draw on. The modulo just wraps the frame index back to the start when the audio runs longer than the video. Twenty seconds of speech over a five-second clip? The clip loops. You never run out of faces, because the frame list is treated as circular.

Be precise about which list leads. If you have more audio than video, you get more output frames than input frames and the source clip repeats. If you have more video than audio, the loop stops when the mel chunks stop, and the tail of the clip is unused. The mel list is the clock.

Crop the face, resize to a square

For each audio chunk, grab the matching frame and the detected face box:

frame_to_save = cv2.imread(frame_files[idx])
dets = face_det_results[idx] if idx < len(face_det_results) else face_det_results[i % len(face_det_results)]
coords = dets[0].get("bbox") if len(dets) > 0 and dets[0] is not None and isinstance(dets[0], dict) else (0, 0, 0, 0)
x1, y1, x2, y2 = map(int, coords)

Clamp the box to the image, then either cut out the face and resize it to a square, or, if no face was found, resize the whole frame as a fallback:

if int(x2 - x1) == 0 and int(y2 - y1) == 0:
    img_batch.append(cv2.resize(frame_to_save, (img_size, img_size)))
else:
    face = frame_to_save[int(y1): int(y2), int(x1):int(x2)]
    img_batch.append(cv2.resize(face, (img_size, img_size)))

mel_batch.append(m)
frame_batch.append(frame_to_save)
coords_batch.append((y1, y2, x1, x2))

Notice it keeps two parallel things. img_batch holds the resized face crops the model will work on. frame_batch holds the original full frames, untouched, and coords_batch holds where each face was. The full frame and the coordinates are not for the model. They are for later, when the predicted mouth has to be pasted back into the real picture. The model only ever sees the square crop.

The trick: blank the lower half, then stack the original on top

Here is the part that matters. Once a batch is full, it gets assembled:

if len(img_batch) >= wav2lip_batch_size:
    img_batch, mel_batch = np.asarray(img_batch), np.asarray(mel_batch)

    img_masked = img_batch.copy()
    img_masked[:, img_size // 2:] = 0

    img_batch = np.concatenate((img_masked, img_batch), axis=3) / 255.
    mel_batch = np.reshape(mel_batch, [len(mel_batch), mel_batch.shape[1], mel_batch.shape[2], 1])

    yield img_batch, mel_batch, frame_batch, coords_batch
    img_batch, mel_batch, frame_batch, coords_batch = [], [], [], []

Three lines do the real work. Take them one at a time.

Copy the faces.

img_masked = img_batch.copy()

We need two versions of the same faces, so we copy first. The original stays intact.

Black out the bottom half of the copy.

img_masked[:, img_size // 2:] = 0

img_batch has shape (batch, height, width, channels). The slice [:, img_size//2:] is "every image, every row from the vertical middle downward." Setting it to 0 paints those rows black. The face is square and roughly centered, so the middle row falls around the nose. Everything below, the mouth, the jaw, the chin, goes dark. What survives is the top half: eyes, brow, hairline, head angle.

Think of it like a witness sketch. You give the artist the eyes and the shape of the head and say "the rest matched this voice, draw it." The model gets identity and pose from the top half, and it has to invent the mouth from the audio. It cannot cheat by reading the answer off the bottom of the picture, because the bottom of the picture is gone.

Stack the untouched face behind the masked one.

img_batch = np.concatenate((img_masked, img_batch), axis=3) / 255.

This is the line people miss. axis=3 is the channel axis. A normal image has 3 channels (BGR). Concatenating the masked face and the original face along that axis makes a 6-channel image: channels 0 to 2 are the masked face (top half only), channels 3 to 5 are the full, unmasked face.

So the model gets both at once, pixel-aligned:

The masked half tells it where to draw and whose face this is.
The full face is a reference. It is the same person, in this lighting, at this resolution, with this skin tone, just from a different moment in the clip.

Why hand over the full face if the point was to hide the mouth? Because there is a difference between hiding the answer for this frame and hiding what this person looks like. The masked input removes the mouth for the audio chunk being generated. The reference shows the model the subject's appearance so the painted mouth comes out the right color and sharpness and clearly belongs to that face. Wav2Lip (Prajwal et al., ACM Multimedia 2020) is built around this: the reference is a different timestep than the masked frame, so the network cannot just copy pixels straight across. It has to actually paint a new mouth in the right style. Same person, wrong moment. Learn to paint, not to copy.

The whole thing is divided by 255. at the end to put pixels in [0, 1].

The mel chunk gets a trailing channel dimension so it reads as a single-channel image too:

mel_batch = np.reshape(mel_batch, [len(mel_batch), mel_batch.shape[1], mel_batch.shape[2], 1])

Now both inputs are image-shaped tensors, and the batch is yielded.

The tail batch

The same masking and concatenation are repeated once more after the loop, for whatever is left over when the audio does not divide evenly into full batches:

if len(img_batch) > 0:
    img_batch, mel_batch = np.asarray(img_batch), np.asarray(mel_batch)
    img_masked = img_batch.copy()
    img_masked[:, img_size // 2:] = 0
    img_batch = np.concatenate((img_masked, img_batch), axis=3) / 255.
    ...
    yield img_batch, mel_batch, frame_batch, coords_batch

Duplicated logic, easy to overlook if you only read the main loop. If you ever change the masking, change it in both places, or your last partial batch will be built differently from every full one.

Gotchas you will actually hit

A few things bit me, or would have:

Six channels, not three. If you write your own model loader or swap checkpoints, the first conv layer expects 6 input channels. Hand it a 3-channel image and the shapes will not match. The doubling happens here, not in the model.
The masked half is exactly half, by a hardcoded fraction. img_size // 2 assumes the face is roughly centered and upright in the square crop. If your face detector returns a loose or tilted box, the cut can land on the lip line or the nose. The face detection and the masking are coupled even though they live in different functions.
Channel order. Crops are read with cv2.imread, so they are BGR. Both halves of the 6-channel tensor are BGR. Keep that consistent with whatever the checkpoint was trained on.
Audio length decides output length. Because the loop iterates mels and wraps the frame index with %, the number of output frames equals the number of mel chunks, not the number of input frames. Short video, long audio: the clip loops. That is by design, but it surprises people who expect output length to match the source video.
The full frame is not model input. frame_batch and coords_batch ride along only so the predicted mouth can be placed back later. Do not feed them to the network.

About the author. I'm Wlad Radchenko, a software engineer. The code in this article comes from Wunjo Make (open source), local software for video makers, and Wunjo Design, an offline PWA for designers. Get in touch to find more on GitHub and LinkedIn.

Wrap-up

The interesting decision in lip-sync input is not "give the model a face and some audio." It is "give the model a face with the mouth deleted, plus a clean reference of the same person, and make the audio fill the gap." The masking forces the model to generate from sound. The reference keeps the result looking like the right person. Two faces stacked into six channels, one of them half-erased.

The pasting-back and seam-hiding is a separate problem with its own tricks, covered in another post. This one stops at the input, because the input is where the lip-sync actually gets taught.

Code: portable/src/visual_processing/lip_sync/sync.py in Wunjo Make, the datagen method.

References

Prajwal, Mukhopadhyay, Namboodiri, Jawahar. "A Lip Sync Expert Is All You Need for Speech to Lip Generation In the Wild." ACM Multimedia 2020. arXiv:2008.10010. https://arxiv.org/abs/2008.10010
Chung, Zisserman. "Out of Time: Automated Lip Sync in the Wild." ACCV 2016 Workshops (SyncNet). https://www.robots.ox.ac.uk/~vgg/publications/2016/Chung16a/

Hiding the seam: making a lip-synced mouth stop looking pasted on

Wlad Radchenko — Tue, 14 Jul 2026 12:22:00 +0000

Lip-sync models like Wav2Lip give you a small, correct mouth. Then you paste that mouth back onto the face and the illusion collapses. You can see the rectangle. The skin tone is off by a shade, the lighting does not match, and the jaw flickers between frames. The network did its job. The compositing gave it away.

The interesting code in a lip-sync pipeline is not the model. It is everything that happens after the model, in the few dozen lines that make a generated patch belong to a real face. This is a walkthrough of that part.

The eye goes straight to the mouth on a talking face, which is exactly why a sloppy paste there is so obvious. Photo: Unsplash.

Where the seam comes from

Wav2Lip (Prajwal et al., ACM Multimedia 2020) takes a masked face plus an audio chunk and predicts the lower face for that audio. You get back a crop the size of the mouth box. The model never saw the full frame, the exact skin tone of the cheeks, or the lighting on the chin. So when you drop the crop in:

ff = f.copy()
ff[y1:y2, x1:x2] = p   # p is the predicted mouth, f is the original frame

you get a hard edge where the crop meets the face, a color step across that edge, and detail that does not match the resolution of the rest of the head. Three separate problems. The fix is three separate tools, stacked.

Tool 1: restore the detail (GFPGAN)

The raw prediction is a little soft. Before blending anything, the patched frame goes through GFPGAN (Wang et al., CVPR 2021), a face restoration GAN, to bring the mouth detail up to the same sharpness as the surrounding skin:

_, _, restored_img = self.gfpgan.enhance(
    ff, has_aligned=False, only_center_face=True, paste_back=True
)

only_center_face=True keeps it from wasting work on faces in the background. Now we have a sharp version of the whole face, but pasting that whole face back would change the eyes and cheeks too. We only want its mouth.

Tool 2: take only the mouth, with a soft mask

This is the step that separates a clean result from a visibly fake one. Instead of a rectangle, use a face-parsing network to cut out only the mouth and lips, then feather the edge.

# select the mouth / lip classes from the face parser
mm = [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 255, 255, 255, 0, 0, 0, 0, 0, 0]
mouth_mask = np.zeros_like(restored_img)
tmp_mask = self.face_enhancement.faceparser.process(restored_img[y1:y2, x1:x2], mm)[0]
tmp_mask_resized = cv2.resize(tmp_mask, (x2 - x1, y2 - y1))[:, :, np.newaxis] / 255.
mouth_mask[y1:y2, x1:x2] = tmp_mask_resized

That mm list is a class selector. The face parser labels every pixel (skin, eyes, nose, lips, and so on), and setting only a few entries to 255 keeps just the mouth region. The result is a mask shaped like an actual mouth, not a box. That shape is what lets the blend follow the lips instead of cutting across the cheek.

Tool 3: blend at every scale (Laplacian pyramid)

A single alpha blend along the mask edge still shows a line, because skin has detail at many scales and a hard cut mixes them all at once. The classic answer is a Laplacian pyramid blend (Burt and Adelson, 1983): split both images into frequency bands, blend each band with a correspondingly blurred mask, then add the bands back. Low frequencies merge over a wide area, high frequencies merge over a narrow one, and the seam disappears.

def laplacian_pyramid_blending_with_mask(A, B, m, num_levels=6):
    GA, GB, GM = A.copy(), B.copy(), m.copy()
    gpA, gpB, gpM = [GA], [GB], [GM]
    for _ in range(num_levels):                 # build Gaussian pyramids
        GA, GB, GM = cv2.pyrDown(GA), cv2.pyrDown(GB), cv2.pyrDown(GM)
        gpA.append(np.float32(GA)); gpB.append(np.float32(GB)); gpM.append(np.float32(GM))

    lpA, lpB, gpMr = [gpA[-1]], [gpB[-1]], [gpM[-1]]
    for i in range(num_levels, 0, -1):          # Laplacian = level minus upscaled next level
        lpA.append(np.subtract(gpA[i-1], cv2.pyrUp(gpA[i])))
        lpB.append(np.subtract(gpB[i-1], cv2.pyrUp(gpB[i])))
        gpMr.append(gpM[i-1])

    LS = [la * gm[:, :, None] + lb * (1 - gm[:, :, None])   # blend each band by the mask
          for la, lb, gm in zip(lpA, lpB, gpMr)]

    out = LS[0]                                  # reconstruct
    for i in range(1, num_levels):
        out = cv2.add(cv2.pyrUp(out), LS[i])
    return out

The pipeline runs this at num_levels=10 on 512x512 crops. Ten bands is more than a still photo needs, but video punishes any visible edge frame after frame, so it is worth the extra levels.

Tool 4: match the light (Poisson blending)

The pyramid blend fixes texture. It does not fix a global color or brightness difference between the patch and the face. For that the final step is Poisson blending (Pérez, Gangnet, Blake, SIGGRAPH 2003), which pastes by matching gradients instead of pixels. It keeps the detail of the mouth but reads the color and lighting from the surrounding face, so the patch takes on the skin tone around it.

f, _, _ = self.face_enhancement.process(pp, f, bbox=c, possion_blending=True)

After this, there is no patch. There is a face that happens to be saying something new.

The bonus: a running mean that kills jaw jitter

One more problem only shows up in motion. Face detection wobbles a pixel or two every frame, so the mouth box jumps, and a jumping box makes the chin vibrate even when the blend is perfect.

The fix is cheap. Keep a short history of recent mouth centers and reject any new detection that jumps too far from the running average:

center_current = np.array([(x1 + x2) / 2, (y1 + y2) / 2])
len_coords_mouth = 3      # history length
max_distance_mouth = 20   # pixels

if len(coords_mouth) < len_coords_mouth:
    coords_mouth.append(center_current)
    # ... do the full blend for this frame ...
else:
    distances = [calculate_distance(center_current, c) for c in coords_mouth]
    if np.mean(distances) < max_distance_mouth:
        # close enough: accept and blend
        ...
    coords_mouth.pop(0)               # slide the window
    coords_mouth.append(center_current)

Three frames of memory and a 20-pixel threshold. That is the entire anti-jitter system, and it removes most of the chin shake that makes cheap lip-sync look cheap.

How the model is even asked to invent a mouth

Worth one paragraph, because it explains why the blend has anything to blend. Before inference, the bottom half of the reference face is zeroed out and stacked with the unmasked face as extra channels:

img_masked = img_batch.copy()
img_masked[:, img_size // 2:] = 0
img_batch = np.concatenate((img_masked, img_batch), axis=3) / 255.

The model sees the top of the face for identity and pose, and a blank lower half it must fill from the audio. That is why the output is a plausible new mouth rather than a copy of the original one. It also deserves its own post, which is coming.

Takeaway

The model gives you a correct mouth. Making it a believable mouth is four steps of classical image work plus three frames of memory: restore detail, cut to a mouth-shaped soft mask, blend across scales with a Laplacian pyramid, match the light with Poisson, and stabilize the box over time. None of it is deep learning. All of it is the reason the result holds up.

The full function is in lip_sync/sync.py in the Wunjo Make repo. If you have been fighting a visible patch in your own pipeline, the mask shape and the Poisson step are where I would start.

References

Prajwal, Mukhopadhyay, Namboodiri, Jawahar. "A Lip Sync Expert Is All You Need for Speech to Lip Generation In the Wild." ACM MM 2020. arXiv:2008.10010
Wang, Li, Zhang, Shan. "Towards Real-World Blind Face Restoration with Generative Facial Prior." CVPR 2021. arXiv:2101.04061
Pérez, Gangnet, Blake. "Poisson Image Editing." ACM SIGGRAPH 2003.
Burt, Adelson. "The Laplacian Pyramid as a Compact Image Code." IEEE Trans. Communications, 1983.

Offline-first in the browser. How I keeping PWA and AI jobs alive across a refresh

Wlad Radchenko — Wed, 08 Jul 2026 11:24:00 +0000

A user types a prompt, clicks generate, and while the model is working they close the tab by accident. Or the wifi drops. Or they just hit refresh out of habit. In a lot of web apps, that is the end of the job and sometimes the end of their unsaved work too. The request was living in a variable in memory, and memory is gone.

The browser is a hostile place to keep state. Tabs close, networks flap, the page reloads on a whim. An editor that loses work on a refresh feels broken even when every other part is polished. This is how I think about durability in Wunjo Design, where the editor itself runs offline and only AI generation and cloud sync need the network. The snippets below are minimal and generic, just enough to show the shape.

The refresh is the cheapest test of whether your app respects the user's work. Photo: Unsplash.

Two things have to survive

When you say "offline-first," people hear "cache the assets." That is the small part. The real work is keeping two kinds of state alive across a reload:

The working document, so the canvas comes back exactly as the user left it. And in-flight AI jobs, so a generation that was running before the refresh is still running, or already done, when the page comes back. Lose either and the user feels the floor move.

The mental model I use is a kitchen ticket rail. When an order comes in, you clip the ticket to the rail before you start cooking. If the cook walks away, the ticket is still there for whoever comes back. The order does not live in the cook's head. It lives on the rail.

The rail is IndexedDB, not localStorage

localStorage is the wrong tool here: it is synchronous, small, and string-only, so it blocks the main thread and chokes on real documents. IndexedDB is the right one: asynchronous, large, and it stores structured data and blobs. A thin wrapper keeps it readable.

import { openDB } from 'idb'

const db = await openDB('editor', 1, {
  upgrade(db) {
    db.createObjectStore('docs')                 // the working document
    db.createObjectStore('jobs', { keyPath: 'id' }) // AI generation jobs
  },
})

Autosave the document on a throttle, not on every keystroke, so you are not hammering the disk:

const save = throttle(async () => {
  await db.put('docs', encodeDoc(), 'current')
}, 1000)

On startup, the first thing the app does is read docs/current back and rehydrate the canvas. The user never sees a blank page after a reload.

The job has to exist before the network call

This is the part people skip, and it is the whole point. If you call the AI endpoint first and plan to record the job when it answers, a refresh in between leaves you with a charge and no record. So you write the job down first, in IndexedDB, as a stub, and only then talk to the network.

// 1. record intent BEFORE the request
const job = { id: uuid(), prompt, status: 'queued', createdAt: Date.now() }
await db.put('jobs', job)

// 2. submit, then persist the remote handle
const { remoteId } = await api.submit(prompt)
await db.put('jobs', { ...job, status: 'running', remoteId })

Now the job is durable. The browser can die at any line and the worst case is a job stuck in queued or running, which is recoverable, instead of a job that silently vanished.

Resume is just reading the rail on boot

Because every job is a row in IndexedDB with a status, resuming after a reload is not special logic. It is a query.

async function resumeJobs() {
  const jobs = await db.getAll('jobs')
  for (const job of jobs) {
    if (job.status === 'running') pollUntilDone(job)   // pick up where we left off
    if (job.status === 'queued')  submitAndRun(job)    // never actually started
  }
}

One observer watches job state and updates the UI, so the canvas does not care how a job finishes, only that its status changed:

class JobObserver {
  constructor() { this.listeners = new Set() }
  subscribe(fn) { this.listeners.add(fn); return () => this.listeners.delete(fn) }
  async update(job) { await db.put('jobs', job); this.listeners.forEach(fn => fn(job)) }
}

That decoupling matters. "Is the job done" lives in one place; "what the canvas shows" lives in another; they talk through status changes. You can close the panel, reopen it, refresh the page, and the UI rebuilds itself from the persisted jobs every time.

Keep the main thread for drawing

Persisting, encoding, exporting, and diffing are not free, and the one thread you cannot afford to block is the one painting the canvas at 60 frames a second. Push the heavy work to a worker.

const worker = new Worker('./export.worker.js', { type: 'module' })
worker.postMessage({ type: 'export', doc: encodeDoc() })
worker.onmessage = (e) => downloadBlob(e.data.blob)

The editor stays responsive while a large export or a re-encode runs off to the side. Offline-first and smooth are not in tension; they both come from treating the main thread as sacred and everything else as background work.

About the author. I'm Wlad Radchenko, a software engineer. This article draws on building Wunjo Design, a browser-based AI vector editor that works offline, and the open-source Wunjo Make. Get in touch to find more on GitHub and LinkedIn.

Things that bite you in practice

Make resume idempotent. A generation costs money, so resuming a running job must re-attach to the existing remote job by its remoteId, never submit a fresh one. The "write the stub first" rule is what makes this possible: you always have the handle.

Respect the quota. IndexedDB has limits and the browser can evict your data under pressure. Store images as Blobs, not base64 data URLs, which inflate size by about a third, and prune finished jobs on a schedule.

Version your schema. The upgrade callback runs when you bump the database version. Plan migrations now, because shipping a change that silently drops a user's stored documents is the one bug you cannot apologize your way out of.

Treat the network as optional. Writes go to local storage first and sync when a connection exists. That is the local-first stance Kleppmann and colleagues argued for in 2019, and in a creative tool it is the difference between "my work is safe" and "I hope the save went through."

What this buys the product

The editor in Wunjo Design opens and works with no connection at all; only AI generation and cloud storage reach for the network. Your document survives a refresh, and a generation you kicked off is still there when you come back. That reliability is not a feature on the pricing page. It is the absence of a specific bad feeling, the one where you close a tab and your stomach drops. Building for the refresh is how you make sure that feeling never happens.

References

Kleppmann, Wiggins, van Hardenberg, McGranaghan. "Local-first software: you own your data, in spite of the cloud." ACM Onward! 2019. PDF
MDN Web Docs. "IndexedDB API" and "Using Web Workers." developer.mozilla.org

The sliding window that stops clicks in audio source separation

Wlad Radchenko — Tue, 07 Jul 2026 13:21:00 +0000

A vocal-separation model cannot eat a whole song at once. The network in Wunjo Make takes a fixed-size block of audio, runs an STFT, predicts a mask, runs an inverse STFT, and hands you back a block of the same size. So a four-minute track has to be cut into blocks, processed one at a time, and glued back together.

The naive way to glue blocks back is to lay them end to end. That clicks. Every seam becomes an audible tick, and across a song you get a rhythmic stutter that no listener will forgive.

This article is about the fix, which lives entirely inside one method, demix(), in portable/src/sound_processing/separator/mdx/model.py. The fix is a sliding window with overlap and a weighted average. I want to walk through it line by line with the real constants from the code, because the trick is small and the payoff is large.

A long take has to be cut into pieces before a model can touch it. Photo: Unsplash

Why cutting at hard edges clicks

Picture cutting a long ribbon into sections, processing each section, then taping them back in order. If the cuts are clean and the sections line up perfectly, fine. But a neural net does not return perfectly continuous audio at a block boundary. The sample at the very end of block 1 and the sample at the very start of block 2 were predicted by two separate forward passes that never saw each other. The waveform value can jump.

A jump in a waveform is a step. A step is a click. Your ear is very good at hearing steps, because a sudden discontinuity contains energy across the whole frequency spectrum. That is the tick you hear at every seam.

So you have two problems at once. The model output near a block edge is less reliable (the STFT has less context there), and butting two blocks together creates a discontinuity. Overlap-add solves both with the same move.

The constants, with real numbers

Before the loop, the model settings get computed in initialize_model_settings():

self.n_bins = self.n_fft // 2 + 1
self.trim = self.n_fft // 2
self.chunk_size = self.hop_length * (self.segment_size - 1)
self.gen_size = self.chunk_size - 2 * self.trim

The fixed knobs in the MDX separator are these:

self.segment_size = 256
self.overlap = 0.25
self.hop_length = 1024
self.batch_size = 1

Plug in the two that do not depend on the model file. The chunk size is hop_length * (segment_size - 1), which is 1024 * 255 = 261120 samples. At a 44100 Hz sample rate that is about 5.9 seconds of audio per block. So the model chews through the song roughly six seconds at a time.

trim is n_fft // 2, where n_fft comes from the model file (mdx_n_fft_scale_set). It is the padding the STFT needs on each side so the transform has context. gen_size is the useful interior of a chunk, chunk_size - 2 * trim, the part you actually keep after throwing away the padded edges.

Hold on to one number: chunk size is 261120 samples. Everything else is relative to it.

The window: a fade-in and fade-out in one array

Here is the plain version. Instead of taping the ribbon sections edge to edge, you let each section fade in at its start and fade out at its end, and you let neighboring sections overlap. Where two sections overlap, you add them. The fades are shaped so that the overlapping fades sum back to one. No seam, because there is no single point where one block stops and the next starts cold.

The shape that does this is a Hann window, np.hanning. It is a smooth bell that is zero at both ends and one in the middle.

window = None
if overlap != 0:
    window = np.hanning(chunk_size_actual)
    window = np.tile(window[None, None, :], (1, 2, 1))

chunk_size_actual is the length of the current chunk, which equals chunk_size for every chunk except the last one (more on that below). np.tile just copies the one-dimensional bell across both stereo channels so it lines up with the (1, 2, samples) shape of the audio.

The window is created fresh per chunk, keyed on the actual length, which matters for the short final chunk.

The step: how far the window slides

step = int((1 - overlap) * chunk_size)

With overlap = 0.25 and chunk_size = 261120, the step is int(0.75 * 261120) = 195840 samples. So each chunk starts 195840 samples after the previous one, but each chunk is 261120 samples long. The difference, 261120 - 195840 = 65280 samples, is the overlap region. At 44100 Hz that is about 1.5 seconds where two consecutive windowed chunks both contribute.

The loop walks the whole padded mixture in steps:

for i in tqdm(range(0, mixture.shape[-1], step)):
    start = i
    end = min(i + chunk_size, mixture.shape[-1])

start advances by step, end is start + chunk_size clamped to the array length. That clamp is what makes the last chunk shorter, and it is why the window is sized from chunk_size_actual = end - start rather than a constant.

The accumulators: result and divider

This is the heart of it. Two arrays, both zeroed and both the full length of the padded mixture:

result = np.zeros((1, 2, mixture.shape[-1]), dtype=np.float32)
divider = np.zeros((1, 2, mixture.shape[-1]), dtype=np.float32)

Think of result as the running sum of all the windowed, predicted audio, and divider as the running sum of all the window weights at each sample position. They grow together, chunk by chunk.

Inside the loop, after the model runs:

if window is not None:
    tar_waves[..., :chunk_size_actual] *= window
    divider[..., start:end] += window
else:
    divider[..., start:end] += 1

result[..., start:end] += tar_waves[..., : end - start]

So for each chunk: multiply the model output by the bell (fade in, fade out), add that into result at this position, and add the same bell into divider at this position. The fade makes the chunk contribute almost nothing at its edges and full weight in its middle. Where two chunks overlap, both their bells land in divider, so divider records exactly how much total weight piled up at every sample.

The else branch (divider += 1) is the no-overlap fallback. If overlap were zero, every sample gets weight one and you are back to plain concatenation. The code keeps that path, but the default overlap = 0.25 never takes it.

The normalization: divide by the weight you actually applied

After the loop:

epsilon = 1e-8  # A small constant to avoid division by zero
tar_waves = result / (divider + epsilon)

This one line is the whole point. You summed up weighted predictions in result. You summed up the weights in divider. Dividing gives you a weighted average at every single sample. A sample that only one chunk covered gets that chunk's value. A sample in the overlap region gets a blend of two chunks, weighted by how confident each window was there (the fading-out tail of the left chunk, the fading-in head of the right chunk).

Because the blend is continuous, the waveform is continuous. No step, no click.

The epsilon is a guard. At the very ends of the padded array the window can taper to zero, so divider can be zero there, and 0 / 0 is a NaN that would poison the audio. Adding 1e-8 first turns 0 / 0 into 0 / 1e-8 = 0, which is silence, which is exactly what you want at a sample no chunk really covered.

This pattern, sum weighted signal over sum of weights, is the classic weighted overlap-add (WOLA). It is the same family of idea that makes the STFT itself invertible. Here it is applied one level up, across model-sized blocks instead of FFT frames.

The padding and the trim, so the edges line up

Two more pieces make the bookkeeping exact. First, padding before the loop:

pad = gen_size + self.trim - ((mix.shape[-1]) % gen_size)
mixture = np.concatenate(
    (np.zeros((2, self.trim), dtype="float32"), mix, np.zeros((2, pad), dtype="float32")), 1)

Silence of length trim is prepended, and enough silence is appended to round the length up so the chunking covers everything. The leading trim of silence gives the very first real sample the same STFT context every interior sample gets, so the model is not guessing at the start.

Second, after normalization, the trim comes back off:

tar_waves_ = np.vstack(tar_waves_)[:, :, self.trim: -self.trim]
tar_waves = np.concatenate(tar_waves_, axis=-1)[:, : mix.shape[-1]]

[:, :, self.trim: -self.trim] drops the padded edges, and [:, : mix.shape[-1]] cuts the result back to the exact length of the input you handed in. You get out exactly as many samples as you put in, click-free.

Gotchas you will actually hit

A few things that bit me or would have:

The last chunk is shorter, and that is fine. Because end is clamped with min(...), the final chunk's length is whatever is left. The window is sized from chunk_size_actual = end - start, so the bell still fits. There is also a zero-pad just before the tensor conversion to make the model see a full-size block:

mix_part_ = mixture[:, start:end]
if end != i + chunk_size:
    pad_size = (i + chunk_size) - end
    mix_part_ = np.concatenate((mix_part_, np.zeros((2, pad_size), dtype="float32")), axis=-1)

but only the first end - start samples of the model output are accumulated (tar_waves[..., : end - start]), so the padding never leaks into the result.

Do not forget the epsilon. If you reimplement this and divide by divider directly, you will get NaNs at the tapered ends and your exported WAV will be full of garbage or silence-with-pops. The + 1e-8 is not decoration.

Overlap is a quality-versus-time dial, not free. With overlap = 0.25 the step is 75 percent of a chunk, so consecutive chunks share 25 percent. Push overlap to 0.5 and you process roughly twice as many chunks for smoother seams. The match-mix pass in the same method deliberately uses a tiny overlap = 0.02, because that pass is reconstructing the mixture, not the delicate vocal, and does not need the expensive smoothing:

if is_match_mix:
    chunk_size = self.hop_length * (self.segment_size - 1)
    overlap = 0.02

The window must match the chunk slice exactly. tar_waves[..., :chunk_size_actual] *= window and divider[..., start:end] += window both use the same length. If those two ever disagree, the divide stops being a true average and the seams come back. Keeping both keyed on chunk_size_actual is what keeps it honest.

Try it on your own audio

The whole mechanism is one loop and one divide. If you want to see it work, separate a track in Wunjo Make and listen to a long sustained note that crosses a chunk boundary. With overlap on it is smooth. Set overlap = 0 in your own copy and you will hear the ticks come back roughly every 4.4 seconds (the step length), which is the clearest proof that the window is doing the work.

The code is in portable/src/sound_processing/separator/mdx/model.py in the Wunjo Make repo. The MDX separation approach this builds on comes from KUIELab-MDX-Net.

References

Kim, Lee, Lee, Lee. "KUIELab-MDX-Net: A Two-Stream Neural Network for Music Demixing." MDX Workshop / ISMIR 2021. arXiv:2111.12203. https://arxiv.org/abs/2111.12203
Mitsufuji et al. "Music Demixing Challenge 2021." Frontiers in Signal Processing, 2022. arXiv:2108.13559. https://arxiv.org/abs/2108.13559
Défossez. "Hybrid Spectrogram and Waveform Source Separation." (Demucs v3.) 2021. arXiv:2111.03600. https://arxiv.org/abs/2111.03600

One model, two stems: how a vocal remover gets the instrumental for free

Wlad Radchenko — Wed, 01 Jul 2026 11:10:00 +0000

You feed a song in. You get two files back: one with only the singer, one with only the band. The obvious way to build that is to train two models, one per stem. The audio separator in my open-source app does something cheaper and, in practice, cleaner: it runs one model, then gets the second stem by subtraction.

This post walks through the actual code that does it. It is short, it is specific, and once you see the subtraction trick you will use it in other places too.

A mixing desk keeps every instrument on its own channel. Source separation tries to recover those channels after they have already been bounced down to one. Photo: Unsplash.

The mental model: a song is a stack of transparent sheets

Picture the final track as a stack of clear plastic sheets pressed together. One sheet has the vocals printed on it, the others have drums, bass, guitar. When you hear an MP3, you are looking down through the whole stack at once.

Separation tries to lift one sheet off. Here is the part people miss: if you can lift the vocals sheet cleanly, you do not need a separate model to read the instrumental. Whatever is left on the table after the vocals are gone is the instrumental. Lift one, subtract, keep both.

That is the whole idea. The rest is making "lift the vocals" and "subtract" precise enough to sound good.

Step 1. Stop working with audio, start working with a picture

You cannot un-mix a waveform by staring at the wiggly line. So the first move is a Short-Time Fourier Transform (STFT), which turns the audio into a spectrogram: a heatmap where one axis is time, the other is pitch, and brightness is how loud that pitch is at that moment.

The useful framing: a spectrogram is an editable image. A vocal sits in a recognizable band with recognizable shapes. A model that edits images can learn to erase it.

class STFT:
    def __init__(self, n_fft, hop_length, dim_f, device):
        self.n_fft = n_fft            # 6144 here: window length
        self.hop_length = hop_length  # 1024: how far the window slides
        self.dim_f = dim_f            # 3072: how many frequency bins we keep
        self.hann_window = torch.hann_window(window_length=n_fft, periodic=True)

    def __call__(self, x):
        spec = torch.stft(x, n_fft=self.n_fft, hop_length=self.hop_length,
                          window=self.hann_window.to(x.device),
                          center=True, return_complex=True)
        spec = torch.view_as_real(spec)            # split into real + imaginary
        spec = spec.permute(...).reshape(...)      # pack as image-like channels
        return spec[..., : self.dim_f, :]          # keep only the first dim_f bins

Two numbers there matter more than they look.

n_fft = 6144 sets how many pitch bins exist: n_fft // 2 + 1 = 3073. hop_length = 1024 is the stride, the same idea as stride in a CNN, just along time. Smaller hop means more overlap and a smoother picture at the cost of more compute.

The frequency cut-off (the part that is not obvious)

Look at the last line: return spec[..., : self.dim_f, :]. The STFT produces 3073 frequency bins, but the model only ever sees 3072 of them. The top of the spectrum is thrown away before inference and padded back with zeros on the way out.

This is not a bug, it is a deliberate trick from the KUIELab-MDX-Net paper (Kim et al., ISMIR 2021). You cut the frequency range to the band where the target source actually lives, which lets you spend the model's resolution where it counts instead of on bins that carry almost no energy. The inverse transform pads the missing bins back so the shapes line up:

def pad_frequency_dimension(self, x, ...):
    freq_padding = torch.zeros([..., num_freq_bins - freq_dim, time_dim]).to(x.device)
    return torch.cat([x, freq_padding], -2)   # put the cut bins back as zeros

One more small thing the code does right before inference:

spek = self.stft(mix)
spek[:, :, :3, :] *= 0   # zero the lowest 3 bins

That kills the bottom three frequency bins, the sub-bass rumble and DC offset that add nothing to a vocal and only confuse the model. Three lines, real quality difference.

Step 2. Run the model on the picture

The model is shipped as ONNX, so inference is one call. It takes the spectrogram and returns a predicted spectrogram for the primary stem.

def run_model(self, mix):
    spek = self.stft(mix)
    spek[:, :, :3, :] *= 0
    spec_pred = self.model_run(spek)                 # ONNX forward pass
    return self.stft.inverse(spec_pred)              # back to a waveform

self.model_run is just the ONNX session wrapped in a lambda, which is the entire reason this runs fast on a CPU when no GPU is around:

session = ort.InferenceSession(model_path, providers=providers, sess_options=opts)
self.model_run = lambda spek: session.run(None, {"input": spek.cpu().numpy()})[0]

Step 3. The subtraction that hands you the second stem

Now the payoff. The model gave us one stem. The other one is mixture minus that stem. The code offers two ways to do it.

The plain way is waveform subtraction:

# source = the stem the model predicted
self.secondary_source = mix.T - source.T

The careful way is spectral subtraction, used when you want to suppress bleed more aggressively. It subtracts in the frequency domain and keeps the phase of the original mix:

def invert_audio(specs, invert_p=True):
    X_mag = np.abs(specs[0])                  # mixture magnitude
    y_mag = np.abs(specs[1])                  # predicted stem magnitude
    max_mag = np.where(X_mag >= y_mag, X_mag, y_mag)
    return specs[1] - max_mag * np.exp(1.0j * np.angle(specs[0]))

Both work. Waveform subtraction is the default because it is exact and fast.

The subtlety that makes subtraction sound clean

Here is the detail I am proud of, and it is easy to get wrong. If you subtract the model's clean output from the raw input, the reconstruction artifacts from the model's own STFT and inverse STFT do not cancel. You hear them as a faint warble in the second stem.

The fix is to subtract through the same lossy pipeline. Before subtracting, the code re-runs the mix through the transform in a pass-through mode that applies STFT and inverse STFT but skips the model:

raw_mix = self.demix(mix, is_match_mix=True)   # STFT -> ISTFT, no model

if self.invert_using_spec:
    self.secondary_source = invert_stem(raw_mix, source)
else:
    self.secondary_source = mix.T - source.T

In that pass-through mode, run_model just returns the spectrogram untouched:

if is_match_mix:
    spec_pred = spek.cpu().numpy()   # identity: no model, only transform round-trip

So both sides of the subtraction carry the same transform artifacts, and the artifacts cancel instead of leaking into the result. Small idea, audible difference.

The 1.022 fudge factor

One last constant worth knowing about:

if not is_match_mix:
    source *= self.compensate   # compensate = 1.022

The model very slightly under-predicts magnitude, so the output is multiplied by 1.022 before it is written. It is a calibration constant that ships with the model weights, not a magic number I picked. If your separated stem sounds a hair quiet, this is where it lives.

Gotchas if you build this yourself

A few things that cost me time:

The audio must be stereo before it hits the model. Mono gets duplicated into two channels (np.asfortranarray([mix, mix])), otherwise the channel math downstream breaks.
Normalize before and after. The code normalizes to a peak of 0.9 so the int16 export does not clip.
The ONNX path here assumes segment_size == dim_t (256 in this model). If they differ you have to fall back to an onnx2torch path, which the code guards against with an explicit error rather than failing silently.

I left out one piece on purpose: how the song is cut into overlapping chunks and stitched back without clicks at the seams. That sliding-window-and-Hann-window dance deserves its own walkthrough, and it is next in this series.

Takeaway

You do not need one model per stem. Train one to lift the hardest source, get the rest by subtraction, and spend your effort on making the subtraction honest: same transform on both sides, phase preserved, magnitude calibrated. The full file is separator/mdx/model.py in the repo if you want to read past the snippets.

If you try this on your own audio, tell me what the second stem sounds like before and after the match-mix trick. That difference is the whole point.

References

Kim, Lee, Lee, Lee. "KUIELab-MDX-Net: A Two-Stream Neural Network for Music Demixing." ISMIR 2021. arXiv:2111.12203
Mitsufuji et al. "Music Demixing Challenge 2021." Frontiers in Signal Processing, 2022. arXiv:2108.13559
Défossez. "Hybrid Spectrogram and Waveform Source Separation." (Demucs v3.) 2021. arXiv:2111.03600

Version history and live collaboration in a browser editor, without a server lock

Wlad Radchenko — Tue, 30 Jun 2026 12:15:00 +0000

Two people open the same canvas. One drags a rectangle, the other recolors a circle, both while a third person is offline on a train editing the same file. When everyone syncs, the document has to agree on one result, nobody's work can vanish, and you still want a history you can scrub back through. The lazy answer is a server lock: one editor at a time, everyone else read-only. It is easy to build and miserable to use.

I want to walk through the data model that makes the hard version work, the one behind real-time collaboration, offline editing, and version history in Wunjo Design. None of this is framework magic. It is one idea applied carefully, and you can reason about all of it from first principles. I will keep the code to small, generic snippets so the shape is clear.

Multiplayer editing looks like a UI feature. It is really a question about how two histories merge into one. Photo: Unsplash.

The real problem: merging two edits with no referee

Picture the document as a plain object: shapes keyed by id. Now two clients change it at the same time, each against their own copy. You receive both changes out of order, maybe minutes apart. What is the final state?

If you just replay "last write wins" on whole objects, the recolor clobbers the move, or the other way around, depending on which packet arrived last. Work disappears. If you funnel every edit through a central server that serializes them, you are back to a lock with extra steps, and offline editing is impossible because there is no server to ask.

The question underneath multiplayer is not "how do I send changes over a socket." It is "given two divergent versions of the same document, how do I merge them so that every client lands on the identical result, no matter what order the changes arrive in."

Two families of answer

There are two well-known approaches.

Operational Transformation (OT) is what classic Google Docs uses. Each edit is an operation, and when operations conflict you transform one against the other so they still compose. It works, but the transformation functions are notoriously fiddly, and most designs lean on a central server to order operations.

Conflict-free Replicated Data Types (CRDTs) take a different route. You design the data structure so that merging is commutative, associative, and idempotent by construction. Apply the same set of changes in any order, any number of times, and every replica converges to the same state. Shapiro and colleagues formalized this as strong eventual consistency in 2011. The practical upshot: no central referee is required. Any peer can merge any other peer's changes and they all end up equal.

I went with the CRDT route because it gives offline editing and history for free instead of as extra subsystems. The algorithm behind the JavaScript CRDT libraries in this space (Yjs, via the YATA algorithm) is built exactly for editing arbitrary structured data in a browser.

Modeling a canvas as shared types

The trick is to stop thinking of the document as one big blob and model it as nested CRDT containers: a map of shapes, where each shape is itself a map of properties. A minimal sketch with the Yjs API:

import * as Y from 'yjs'

const doc = new Y.Doc()
const shapes = doc.getMap('shapes')         // the whole scene

function addRect(id) {
  const rect = new Y.Map()
  rect.set('type', 'rect')
  rect.set('x', 10); rect.set('y', 20)
  rect.set('w', 100); rect.set('h', 60)
  rect.set('fill', '#3366ff')
  shapes.set(id, rect)
}

Granularity is the whole design decision here. Because each shape is its own map, two people editing different shapes never touch the same data, so there is nothing to conflict. And because each property is its own key, one person moving a rectangle (x, y) while another recolors it (fill) merges cleanly: the move and the recolor are independent keys, both survive. You only get a real conflict when two people set the same property of the same shape at the same instant, and there the CRDT picks a deterministic winner that every client agrees on.

Sync is just bytes you can apply in any order

Once the document is a CRDT, "sync" stops being a protocol you design and becomes two function calls. A client serializes its changes to a compact binary update and ships it. Any peer applies it. Order does not matter.

// on the client that made changes
const update = Y.encodeStateAsUpdate(doc)   // a small binary diff

// on any other client, whenever it arrives
Y.applyUpdate(otherDoc, update)             // merge; commutative and idempotent

That applyUpdate is the entire merge engine. Apply the same update twice, it is a no-op. Apply two clients' updates in opposite orders on two machines, both machines match. This is the property that makes the network boring, which is what you want a network to be.

Offline falls out for free

Here is the part I find satisfying. Offline editing is not a separate feature you bolt on. When a client is offline, it keeps producing updates against its local document. When it reconnects, those queued updates merge with everyone else's through the same applyUpdate call. There is no special "sync conflict resolution" screen, because the data type cannot produce an unresolvable conflict. This is the local-first idea that Kleppmann and colleagues argued for in 2019: the network is an enhancement, not a requirement, and your data lives on your device first.

History without saving fifty copies

Version history sounds like it needs a pile of saved files. It does not. A CRDT document already carries the information needed to reconstruct earlier states, because it is fundamentally a log of changes plus enough metadata to order them. You take lightweight snapshots at points worth remembering, and you can reconstruct or restore the document as of any snapshot.

const checkpoint = Y.snapshot(doc)   // a cheap marker, not a full copy
// later: rebuild the document state as of `checkpoint` for preview or restore

So "version history with rollback" becomes: keep a list of snapshots with labels, render a preview from any of them, and to restore, apply the delta that takes you back. You are storing operations and markers, not duplicate documents, which is why the history can be deep without bloating storage.

About the author. I'm Wlad Radchenko, a software engineer. This article draws on building Wunjo Design, a browser-based AI vector editor with collaboration and version history, and the open-source Wunjo Make. Get in touch to find more on GitHub and LinkedIn.

Things that bite you in practice

A few lessons that cost me time.

Keep transient state out of the document. Cursors, selections, and who-is-online change many times a second and mean nothing tomorrow. Put them on an ephemeral presence channel (Yjs calls it awareness), not in the CRDT, or your history fills with mouse twitches.

Do not store images inside the CRDT. Big binary blobs make every sync and every snapshot heavier. Store the blob separately, keyed by id, and keep only the reference in the document.

Undo has to be per-user and merge-aware. In a shared document, your undo must reverse your last action, not your collaborator's. A naive global undo stack will happily undo someone else's work. The CRDT libraries track the origin of each change so undo can be scoped, and that is its own topic, which I will take apart next.

Compact the history. An append-only log grows. Periodic snapshots plus garbage collection of superseded operations keep it bounded.

What this buys the product

One data model gives Wunjo Design three features that usually cost three subsystems: real-time collaboration, offline editing, and version history you can roll back. They are not separate engineering efforts stapled together. They are consequences of choosing a data structure that merges by construction. That is the kind of decision that does not show up in the UI but determines whether the UI is even possible.

If you are building anything collaborative in the browser, start from the data model, not the socket. Pick a structure where merge is total, and the network, the offline support, and the history mostly write themselves.

References

Shapiro, Preguiça, Baquero, Zawirski. "Conflict-free Replicated Data Types." SSS 2011. HAL
Nicolaescu, Jahns, Derntl, Klamma. "Near Real-Time Peer-to-Peer Shared Editing on Extensible Data Types." ACM GROUP 2016. (The YATA algorithm behind Yjs.)
Kleppmann, Wiggins, van Hardenberg, McGranaghan. "Local-first software: you own your data, in spite of the cloud." ACM Onward! 2019. PDF

Stopping the flicker when you restyle a video frame by frame

Wlad Radchenko — Sat, 27 Jun 2026 14:01:21 +0000

Run a diffusion restyle on every frame of a clip, one frame at a time, and the still images look great. Then you play them back and the whole thing boils. Textures crawl, colors pulse, a brick wall shifts its grout lines every frame. The model did nothing wrong on any single frame. It just made a slightly different choice each time, and at 24 frames a second your eye reads those differences as flicker.

This is a walkthrough of the code that kills that flicker. The trick is to stop restyling every frame. Stylize a few frames, then carry the style to the rest by following the motion. The interesting part is the bookkeeping and the blending that make the carry invisible, so I will spend most of the post there.

Each frame is a separate sample from the model, so each frame lands in a slightly different place. Played in sequence, those small differences become flicker. Photo: Unsplash.

Why frame-by-frame flickers

A diffusion model starts from noise and walks toward an image. Two frames of a video that look almost identical to you are still two different starting points and two different walks. The model has no memory of what it drew last frame, so it picks a fresh interpretation of "oil painting" or "anime" each time. On a still you never notice. In motion you see the model changing its mind 24 times a second.

You can lower the denoise strength so the model stays close to the input, but then you barely restyle anything. You can feed the previous frame back in, which helps a little and drifts a lot. The cleaner answer is structural: restyle only a sparse set of frames, and fill the gaps by warping a real stylized frame into place. A warped frame cannot disagree with itself between neighbors, because it is the same pixels pushed along the motion. This is the idea behind Rerender A Video (Yang et al., SIGGRAPH Asia 2023) and behind EbSynth (Jamriška et al., ACM ToG 2019), and it is what the code below implements.

Step 1. Pick the keyframes

Keyframes come from scene detection plus a fixed interval. Inside the scene detector:

if interval:
    scene_frames.extend(range(start_frame, end_frame, interval))
    scene_frames.append(end_frame - 1)

The default interval is 10. So inside each detected scene you take every tenth frame, plus the last frame of the scene, as keyframes. Those are the only frames the diffusion model ever touches. Everything between two keyframes is going to be synthesized by warping, not by the model. Pick the interval too large and motion outruns the warp; pick it too small and you pay for diffusion you did not need. Ten is a reasonable middle for most footage.

Step 2. The bookkeeping that nobody talks about

Once you commit to "stylize keyframes, propagate the rest," you inherit a filing problem. For every gap between keyframe i and keyframe i+1 you need the right input frames, the right output paths, the right optical-flow files, and the right guide images, in both the forward and backward direction. Get one index off and a frame lands in the wrong folder. VideoSequence in video_sequence.py is the class that does this filing.

It is constructed with the list of keyframes (here called frame_files_with_interval) and it makes one output subdirectory per keyframe:

self.__frame_files_with_interval = [f for f in frame_files_with_interval if ".png" in f]
self.__n_seq = len(self.__frame_files_with_interval)
# ...
out_subdir = self.__get_out_subdir(frame_file)   # out_<keyframe-name>

The core method is get_input_sequence. Given a gap index i, it returns the list of input frame paths in that gap:

def get_input_sequence(self, i, is_forward=True):
    if i + 1 > len(self.__frame_files_with_interval) - 1:
        # last gap: run from the final keyframe to the true last frame of the video
        last_input_frame = self.__input_frames[-1]
        last_interval_frame = self.__frame_files_with_interval[i]
        if last_input_frame == last_interval_frame:
            return None
        else:
            beg_id = int(last_interval_frame.split(".")[0])
            end_id = int(last_input_frame.split(".")[0])
    else:
        beg_id = self.get_sequence_beg_id(i)
        end_id = self.get_sequence_beg_id(i + 1)
    if is_forward:
        id_list = list(range(beg_id, end_id))
    else:
        id_list = list(range(end_id, beg_id, -1))
    return [os.path.join(self.__input_dir, self.__input_format % id)
            for id in id_list if self.__input_format % id in self.__input_frames]

Two things to notice. First, beg_id and end_id come straight from the keyframe filenames, which are named by frame number (%04d.jpg). The filename is the index. That is why the whole class can do its math on int(name.split(".")[0]) instead of carrying a separate table. Second, the is_forward flag reverses the range. The pipeline propagates style from the left keyframe rightward, and from the right keyframe leftward, then meets in the middle. The backward pass needs the same frames in reverse, and this one flag gives both.

The same shape repeats for every artifact the propagation needs:

get_output_sequence builds the destination paths inside the keyframe's out_ folder.
get_flow_sequence builds flow_f_%04d.npy for forward and flow_b_%04d.npy for backward.
get_edge_sequence, get_temporal_sequence, get_pos_sequence build the per-frame guide paths in the keyframe's tmp folder.

One detail worth flagging: the flow lists are one element shorter than the frame lists. There are N frames but only N-1 motions between them:

if is_forward:
    id_list = list(range(beg_id, end_id - 1))      # forward flows: N-1
else:
    id_list = list(range(end_id, beg_id + 1, -1))  # backward flows: N-1

If you ever zip flows against frames and get an off-by-one, this is where it comes from. The class is built so the flow list and the warp loop line up.

The last gap is the awkward one. The final keyframe is rarely the literal last frame of the video, so the code special-cases it: when i+1 runs past the keyframe list, it uses self.__input_frames[-1], the true last frame, as the end of the gap. Without that branch the tail of every clip would go unstyled.

Step 3. The guides that steer the propagation

Propagation here is done EbSynth-style: you give EbSynth a source stylized image and a set of guide channels, and it synthesizes the target frame so it matches the style of the source while respecting the guides. The guides live in guide.py. Each one answers a different question for the synthesizer.

The positional guide answers "where did this pixel come from?" It starts from a synthetic image where each pixel encodes its own coordinate as color, then warps that image along the optical flow, frame after frame:

@staticmethod
def __generate_first_img(H, W):
    Hs = np.linspace(0, 1, H)
    Ws = np.linspace(0, 1, W)
    i, j = np.meshgrid(Hs, Ws, indexing='ij')
    r = (i * 255).astype(np.uint8)   # row -> red
    g = (j * 255).astype(np.uint8)   # col -> green
    b = np.zeros(r.shape)
    return np.stack((b, g, r), 2)

Red is the row, green is the column. After you warp this map by the flow, a pixel's color tells you which original pixel ended up there. That is a dense, smooth correspondence field, which is exactly what a synthesizer wants so it does not invent new texture in moving regions.

The temporal guide answers "what did the previous stylized frame look like, moved to here?" It takes the previous stylized frame and warps it forward by the flow:

def get_cmd(self, i, weight) -> str:
    if i == 0:
        warped_img = self.stylized_imgs[0]
    else:
        prev_img = cv2.imread(self.stylized_imgs[i - 1])
        warped_img = self.flow_calc.warp(prev_img, self.flows[i - 1], 'nearest').astype(np.uint8)
        warped_img = cv2.inpaint(warped_img, self.masks[i - 1], 30, cv2.INPAINT_TELEA)
        cv2.imwrite(self.imgs[i], warped_img)
    return super().get_cmd(i, weight)

This is the anti-flicker guide. It pushes the synthesizer to make frame i look like frame i-1 carried along the motion, so the style stays put on a surface as it moves instead of re-rolling every frame.

Both warps leave holes. Where motion uncovers a region the camera could not see last frame, the warp has no data, and the optical-flow mask marks those pixels. The fix is the same in both guides:

cur_img = cv2.inpaint(cur_img, mask, 30, cv2.INPAINT_TELEA)

cv2.INPAINT_TELEA fills the disoccluded holes from their surroundings so the guide has no black gaps. A radius of 30 pixels is generous, which suits the smooth guide maps; you do not need sharp inpainting here, just plausible filler.

The other two guides are simpler. The edge guide runs a Laplacian-style filter so the synthesizer keeps structure aligned to the input:

filter = np.array([[0, -1, 0], [-1, 4, -1], [0, -1, 0]])
res = cv2.filter2D(img, -1, filter)

And the color guide is just the raw frames, so the synthesizer has the original colors to refer to. Each guide carries a -weight, so you can dial how strongly the synthesizer listens to motion versus structure versus color.

Step 4. Where forward and backward meet, blend the colors

Now you have two stylized versions of every in-between frame: one propagated forward from the left keyframe, one propagated backward from the right keyframe. They agree on geometry but rarely on color, because each picked up the tint of a different keyframe along the way. Stack them naively and you get a visible color step. histogram_blend.py fixes the color before the seam gets stitched.

It works in Lab color space, which separates lightness from color so you can match tone without muddying brightness:

a = cv2.cvtColor(a, cv2.COLOR_BGR2Lab)
b = cv2.cvtColor(b, cv2.COLOR_BGR2Lab)
# normalize each to a common mean/std
t_mean_val = 0.5 * 256
t_std_val = (1 / 36) * 256
a = histogram_transform(a, a_mean, a_std, t_mean, t_std)
b = histogram_transform(b, b_mean, b_std, t_mean, t_std)
# average them, then re-key to the reference frame's statistics
ab = (a * weight1 + b * weight2 - t_mean_val) / 0.5 + t_mean_val
ab = histogram_transform(ab, ab_mean, ab_std, min_error_mean, min_error_std)

The shape is: push both images to the same neutral mean and standard deviation, average them, then push the average to the statistics of min_error, the frame the pipeline trusts most for this position. The two odd constants are a target mean of 0.5 * 256 (mid-gray) and a target std of (1/36) * 256. They are just a stable common ground to average in; the final re-keying is what makes the result match a real frame rather than a washed-out midpoint. This is per-channel mean/std transfer, the same idea as classic Reinhard color transfer, done twice.

Step 5. Hide the seam with Poisson fusion

Color matching makes the two halves agree on average. It does not hide the actual boundary where forward meets backward. For that the pipeline pastes in the gradient domain, the same Poisson Image Editing idea from Pérez, Gangnet and Blake (SIGGRAPH 2003) that photo tools use to drop an object into a new background without a halo.

The principle: do not copy pixels, copy differences between pixels. Build a target gradient field by taking gradients from image 1 outside the mask and image 2 inside it, then solve for the image whose gradients match that field. Seams disappear because you never enforce an absolute pixel value at the boundary, only the slope across it.

def poisson_fusion(blendI, I1, I2, mask, grad_weight=[2.5, 0.5, 0.5]):
    Iab = cv2.cvtColor(blendI, cv2.COLOR_BGR2LAB).astype(float)
    Ia  = cv2.cvtColor(I1, cv2.COLOR_BGR2LAB).astype(float)
    Ib  = cv2.cvtColor(I2, cv2.COLOR_BGR2LAB).astype(float)
    m = (mask > 0).astype(float)[:, :, np.newaxis]

    # gradient from I1 outside the mask, from I2 inside it
    gx[:-1] = (Ia[:-1] - Ia[1:]) * (1 - m[:-1]) + (Ib[:-1] - Ib[1:]) * m[:-1]
    gy[:, :-1] = (Ia[:, :-1] - Ia[:, 1:]) * (1 - m[:, :-1]) + (Ib[:, :-1] - Ib[:, 1:]) * m[:, :-1]

Then for each channel it solves a least-squares system Ax = b where A stacks the gradient operators and an identity term, and b stacks the target gradients and the original intensities:

A = As[i]
b = np.vstack([im_dx * weight, im_dy * weight, im])
out = scipy.sparse.linalg.lsqr(A, b)

Two things make this practical. First, grad_weight=[2.5, 0.5, 0.5] weights the L (lightness) channel five times harder than the two color channels. Lightness carries the structure your eye locks onto, so the solver is told to preserve lightness gradients tightly and let color relax. Second, the big sparse matrix A depends only on image size and weights, not on pixel values, so it is built once and cached:

crt_states = (h, w, grad_weight)
if As is None or crt_states != prev_states:
    As = construct_A(*crt_states)
    prev_states = crt_states

Building A walks every pixel to wire up the gradient operators, which is slow. For a video you run poisson_fusion on hundreds of frames at the same resolution, so caching it across calls turns a per-frame cost into a one-time cost. That global cache is the difference between a restyle that finishes and one you abandon.

About the author. I'm Wlad Radchenko, a software engineer. The code in this article comes from Wunjo Make (open source) is local software for video makers, and Wunjo Design as offline PWA for designers. Get in touch to find more GitHub and LinkedIn

Gotchas you will actually hit

The interval is the dial that matters most. With fast motion or a moving camera, optical flow gets unreliable and a wide interval lets the warp smear. Drop the interval so keyframes are closer together and the propagation has less work to do per gap.

The flow list is N-1, not N. If you write your own warp loop against get_flow_sequence, remember there is one fewer flow than frame, and the temporal guide already accounts for it by special-casing i == 0.

Inpaint radius is forgiving here. The TELEA radius of 30 looks large, but it fills guide maps, not final pixels, so a soft fill is fine. The real frame quality comes from the synthesizer, the histogram match, and the Poisson solve downstream.

Watch the last gap. The final keyframe is almost never the last frame of the clip. The i + 1 > len(...) branch in every VideoSequence method exists to run that tail against the real last frame. If you reimplement the bookkeeping and skip it, your output will be a few frames short and the cut will be obvious.

Wrap-up

Per-frame restyle flickers because the model re-decides the look on every frame. The fix is to decide rarely and propagate. Stylize sparse keyframes, warp them across the gaps with EbSynth-style positional and temporal guides, match colors in Lab with a double histogram transfer, and stitch the forward and backward halves with a gradient-domain Poisson solve that weights lightness heavily and caches its matrix. None of the temporal-coherence work is diffusion. It is flow, inpainting, and two classic blends, sequenced carefully.

The code is in visual_generation/restyle/blender/ in the Wunjo Make repo: video_sequence.py for the bookkeeping, guide.py for the guides, histogram_blend.py and poisson_fusion.py for the blends. If your own restyle boils, start by cutting the per-frame diffusion down to keyframes.

References

Yang, Zhou, Liu, Loy. "Rerender A Video: Zero-Shot Text-Guided Video-to-Video Translation." SIGGRAPH Asia 2023. arXiv:2306.07954
Jamriška et al. "Stylizing Video by Example." (EbSynth.) ACM ToG 2019.
Xu, Zhang, Cai, Rezatofighi, Tao. "GMFlow: Learning Optical Flow via Global Matching." CVPR 2022. arXiv:2111.13680
Zhang, Rao, Agrawala. "Adding Conditional Control to Text-to-Image Diffusion Models." (ControlNet.) ICCV 2023. arXiv:2302.05543
Pérez, Gangnet, Blake. "Poisson Image Editing." ACM SIGGRAPH 2003.

Optimizing and Running Neural Networks on React Native: A Grass Case Study

Wlad Radchenko — Thu, 08 Jan 2026 11:53:22 +0000

During the long New Year holidays, I decided to start a new pet project. I wanted to experiment with something interesting involving neural networks, and the idea of a mobile app that could recognize plants — identifying species, age, and even diseases — directly on a mobile device without a server, while also providing recommendations on care and treatment, seemed exciting and new to me.

As I dug in, I realized that there are already quite a few similar apps. However, that didn’t diminish the value of the experiment: I wanted to understand how neural networks can be optimized for mobile devices and run directly on my phone. In this article, I’ll share what I managed to achieve — from selecting models to optimizing them and integrating them into React Native — as well as my experience with various methods for plant classification, disease detection, and estimating age and leaf count.

This is GitHub code and Hugging Face models.

SCOLD Model for Cross-Modal Disease Search

The first tool I decided to experiment with was the SCOLD model from Hugging Face. It was designed for cross-modal search, meaning it can match an image with a textual description. For example, you can feed in a photo of a corn leaf and a text description of a disease, and the model will estimate how well the text matches the image. SCOLD was trained on the LeafNet dataset.

Despite its interesting concept, I wasn’t entirely sure how to apply it in practice. On top of that, the model itself was about 1GB in size, which immediately makes running it on a mobile device challenging. I decided to modify the model slightly so that it could identify the plant disease directly.

Since I was only interested in working with images, I removed the Roberta text encoder and kept only the image processing module based on the Swin Transformer [source code]. As a result, I ended up with a much smaller model that outputs a 512-dimensional feature vector for each image.

Originally, this vector was meant to be compared with Roberta’s output for textual descriptions, but for mobile deployment, I decided on a different approach: instead of working with text, I compare the image vectors with vectors precomputed for the entire dataset.

This setup also introduced two additional files: a 248MB binary file with the embeddings and an 18MB JSON file containing disease labels. This approach preserves the original task of disease identification without needing to process text, while significantly reducing the model size for mobile use.

from timm import create_model
import torch
import torch.nn as nn
import numpy as np
from transformers import RobertaModel

EMBEDDING_DIM = 512


class ImageEncoder(nn.Module):
    def __init__(self):
        super(ImageEncoder, self).__init__()
        # Load the Swin Transformer with features_only=True
        self.swin = create_model("swin_base_patch4_window7_224.ms_in22k", pretrained=True, features_only=True)
        for param in self.swin.parameters():
            param.requires_grad = True

        # Get the feature size of the final stage
        self.swin_output_dim = self.swin.feature_info.channels()[-1]  # Last stage: 1024 channels

        # Define FC layer
        self.fc1 = nn.Linear(self.swin_output_dim * 7 * 7, EMBEDDING_DIM)  # Flattened input size
        nn.init.xavier_uniform_(self.fc1.weight)
        nn.init.zeros_(self.fc1.bias)
        for param in self.fc1.parameters():
            param.requires_grad = True

    def forward(self, x):
        # Extract features from Swin
        swin_features = self.swin(x)[-1]  # Use the last stage feature map (e.g., [B, 1024, 7, 7])

        # Flatten feature map
        swin_features = swin_features.view(swin_features.size(0), -1)  # Shape: (B, 1024*7*7)

        # Pass through FC layer
        output = self.fc1(swin_features)  # Shape: (B, embedding_dim)
        return output


class LVL(nn.Module):
    def __init__(self):
        super(LVL, self).__init__()
        self.image_encoder = ImageEncoder()
        self.text_encoder = nn.Identity()
        self.t_prime = nn.Parameter(torch.ones([]) * np.log(0.07))
        self.b = nn.Parameter(torch.ones([]) * 0)

    def get_images_features(self, images):
        image_embeddings = self.image_encoder(images)  # (batch_size, EMBEDDING_DIM)
        image_embeddings = nn.functional.normalize(image_embeddings, p=2, dim=-1)
        return image_embeddings

    def get_texts_feature(self, input_ids=None, attention_mask=None):
        """
        Plug
        :param input_ids: Tensor of shape (batch_size, seq_length)
        :param attention_mask: Tensor of shape (batch_size, seq_length)
        :return:
        """
        return None

    def forward(self, images, input_ids=None, attention_mask=None):
        """
        Args:
            images: Tensor of shape (batch_size, 3, 224, 224)
            input_ids: Tensor of shape (batch_size, seq_length)
            attention_mask: Tensor of shape (batch_size, seq_length)

        Returns:
            Image and text embeddings normalized for similarity calculation
        """

        image_embeddings = self.get_images_features(images)
        return image_embeddings

The next step was converting the model to the ONNX format, which is well-supported on mobile and allows for integrating neural networks directly into React Native. During this process, I had to account for the specifics of each model: setting up the correct dummy input to verify the input shape, specifying input and output names, defining dynamic_axes to support batch sizes, and sometimes adjusting the opset version to ensure compatibility with ONNX Runtime.

import torch
# pip install onnxscript onnxruntime onnxruntime-tools
from model import LVL


def export(model_path: str, output_name: str):
    device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
    # Create a dummy input with the same dimensions as the real data.
    dummy_input = torch.randn(1, 3, 512, 512, device=device)

    # Load model
    model = LVL()
    model.to(device)
    model.eval()

    #  git clone https://huggingface.co/enalis/scold
    state_dict = torch.load(model_path, map_location=device)

    # Leave only the keys that are in the new model (only image_encoder)
    model_dict = model.state_dict()
    filtered_dict = {k: v for k, v in state_dict.items() if k in model_dict}

    # Load only image_encoder weights
    model.load_state_dict(filtered_dict, strict=False)  # <- strict=False important!

    # Export
    torch.onnx.export(
        model,                      # Model
        dummy_input,                # Input
        output_name,           # Name
        export_params=True,         # Save weights
        opset_version=18,           # Version
        do_constant_folding=True,   # Optim
        input_names=['images'],     # Name of inputs
        output_names=['image_embeddings'],  # Name of outputs
        dynamic_axes={'images': {0: 'batch_size'},  # Dynamic support batch size
                      'image_embeddings': {0: 'batch_size'}}
    )
    print("Finish!")


if __name__ == "__main__":
    from argparse import ArgumentParser

    parser = ArgumentParser()
    parser.add_argument("--model_path", type=str, required=True, help="Path to model pth")
    parser.add_argument("--output_name", type=str, default="disease_detector.onnx", help="Path to save onnx model with name model.onnx")
    args = parser.parse_args()

    export(model_path=args.model_path, output_name=args.output_name)

After that, I further reduced the model size by converting it from float32 to float16, while keeping the input format as float32. As a result, the model size dropped from 1 GB to 231 MB, making it much more suitable for a mobile application.

import onnx
from onnxconverter_common import float16


def export(model_path: str, output_name: str):
    # Load model
    model_fp32 = onnx.load(model_path)
    # Export to float16
    model_fp16 = float16.convert_float_to_float16(model_fp32, keep_io_types=True)
    # Save
    onnx.save(model_fp16, output_name)


if __name__ == "__main__":
    from argparse import ArgumentParser

    parser = ArgumentParser()
    parser.add_argument("--model_path", type=str, required=True, help="Path model onnx")
    parser.add_argument("--output_name", type=str, default="fp16.onnx", help="Path to save fp16 mode with name model_fp16.onnx")
    args = parser.parse_args()

    export(model_path=args.model_path, output_name=args.output_name)

The disease_detection.onnx model itself and the code for running it can be found on GitHub.

Plant Classification from Images

It would be hard to imagine this project without a neural network for recognizing plants from photos, so the next step was to tackle exactly that. While looking for suitable data and models, I came across the PlantCLEF 2024 competition, which led me to the Pl@ntNet dataset. The dataset itself is huge, and training a model on the full dataset would require a lot of time and resources. Fortunately, instead of overloading my laptop, I found that a pre-trained model along with the code to run it had already been released on Zenodo.

The model takes a plant image as input and outputs a set of probabilities corresponding to specific plant species. Essentially, this is a standard image classification task without any extra complications. For my purposes, this was more than enough, so I followed the familiar path: exporting the model to ONNX and then converting it to float16 to reduce its size and make it more suitable for mobile deployment. The ready-to-use model and the code for running it are available in the project repository.

import torch
import timm
# pip install onnxscript onnxruntime onnxruntime-tools
import argparse


def load_class_mapping(class_list_file):
    with open(class_list_file) as f:
        class_index_to_class_name = {i: line.strip() for i, line in enumerate(f)}
    return class_index_to_class_name


def export(model_path: str, output_name: str):
    torch.serialization.add_safe_globals([argparse.Namespace])
    device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

    # Class and model after train.py
    class_mapping = load_class_mapping(args.class_mapping)

    # Load model
    model = timm.create_model('vit_base_patch14_reg4_dinov2.lvd142m', pretrained=False, num_classes=len(class_mapping))
    checkpoint = torch.load(model_path, map_location=device)
    model.load_state_dict(checkpoint['state_dict'])
    model.eval()

    # Dummy input (ViT Base 224x224 RGB)
    dummy_input = torch.randn(1, 3, 518, 518)

    # Export
    torch.onnx.export(
        model,                      # Model
        dummy_input,                # Input
        output_name,           # Name
        export_params=True,         # Save weights
        opset_version=18,           # Version
        input_names=['input'],      # Name of inputs
        output_names=['output'],    # Name of outputs
        dynamic_axes={"input": {0: "batch_size"}, "output": {0: "batch_size"}},
    )
    print("Finished!")


if __name__ == "__main__":
    from argparse import ArgumentParser

    parser = ArgumentParser()
    parser.add_argument("--class_mapping", type=str, required=True, help="Path to class mapping")
    parser.add_argument("--model_path", type=str, required=True, help="Path species mapping")
    parser.add_argument("--output_name", type=str, default="plant_classificator.onnx", help="Path to save onnx model with name model.onnx")
    args = parser.parse_args()

    export(model_path=args.model_path, output_name=args.output_name)

Later, after studying the dataset more closely, I realized that it mostly contains rare and exotic plants. It barely includes common ornamental flowers, vegetables, or fruits that one might expect in such an app—or at least typical bouquet flowers.

Because of this, I decided to go further and implement code for fine-tuning the model or training it from scratch. In practice, training from scratch turned out to be the preferable option, as it allows creating a smaller model, which is critical for mobile devices. Additionally, this approach gives users the flexibility to choose which model to load depending on their specific needs and the available resources on their device.

I moved the training process into a separate module and documented the dataset structure and the steps for running training in the project documentation. Training can be run with a batch size of 8 on a GPU with 8 GB of VRAM, or the batch size can be reduced if resources are limited. The dataset must be assembled manually by combining data from Kaggle and other open sources. This approach proved to be more flexible and aligns well with the concept of an experimental mobile ML project.

Regression Model for Estimating Plant Age and Leaf Count

The task of estimating a plant’s age and the number of its leaves may seem trivial at first, but in practice, I couldn’t find any ready-to-use models. There is a scientific work, GroMo: Plant Growth Modeling with Multiview Images, along with its corresponding repository, but the code there is somewhat messy and not easily scalable for practical use. Therefore, I decided not to adapt it, but to rewrite the solution from scratch, tailored to my own constraints.

In the original work, plants of different species, such as mustard, radish, wheat, and okra, were photographed from multiple angles and at various growth stages, with annotations indicating the plant’s age and the number of leaves. An important point is that visual features strongly depend on the species: leaf shape, size, and even the overall plant structure can vary significantly, and sometimes leaves are visually almost indistinguishable from stems.

In the original model, four images from different angles were fed into the network at once, whereas in my case, the input was simplified to a single photo, which is more aligned with a real mobile user scenario. There was also an idea to check whether the current number of leaves or stems is appropriate for the plant’s age—or if it’s below average—but I decided to leave that for future work.

At the core of this model is MobileNetV3 Large, used as a universal visual encoder. From the pretrained network, only the convolutional part is taken, which extracts compact and informative features from a leaf image. These features are then aggregated using Adaptive Average Pooling into a fixed-size vector of 960 dimensions, making the model independent of the input image size.

This shared feature vector is then used for two regression tasks simultaneously: estimating the number of leaves and the plant’s age. To accomplish this, two separate “heads” are added, each implemented as a small fully connected block. I named this model LeafNet.

import torch.nn as nn
from torchvision.models import mobilenet_v3_large

class LeafNet(nn.Module):
    def __init__(self):
        super().__init__()
        backbone = mobilenet_v3_large(weights="DEFAULT")
        self.encoder = backbone.features

        self.pool = nn.AdaptiveAvgPool2d(1)

        self.count_head = nn.Sequential(
            nn.Linear(960, 128),
            nn.ReLU(inplace=True),
            nn.Linear(128, 1)
        )

        self.age_head = nn.Sequential(
            nn.Linear(960, 128),
            nn.ReLU(inplace=True),
            nn.Linear(128, 1)
        )

    def forward(self, x):
        feat = self.encoder(x)
        pooled = self.pool(feat).flatten(1)
        count = self.count_head(pooled)
        age = self.age_head(pooled)
        return feat, count, age

Training was performed on a combined dataset without separating by plant species, allowing the model to learn from a wider variety of visual patterns. The total dataset size was around 380 GB. Full training took approximately three days, and during training, the model is automatically exported to ONNX format for use on mobile devices. The training documentation, along with all the code for running experiments, is available in the project repository.

Integrating and Running Neural Networks in React Native

In this part of the article, I’ll discuss running neural networks directly on a mobile device and the code required to make it work. I intentionally skip basic setup steps like installing Node.js, running an Android emulator, or setting up the development environment. Those are well-documented elsewhere and not directly relevant to the task at hand. Here, we’ll focus on the code and the limitations encountered when working with neural networks on mobile platforms.

For convenience, the project repository includes a ready-to-use APK for those who don’t want to build the app themselves (it’s large because it includes the models). However, for development and experimentation, I highly recommend building the project manually. Testing was performed on a Pixel 8 Pro emulator, so behavior on other devices may vary. Detailed build and run instructions are available in the repository under the mobile folder.

The first major problem, which cost me about a day of development, was choosing the right mobile stack. I initially started with Expo since it allows quickly spinning up a prototype. However, I discovered that running ONNX models in this environment is either unstable or completely impossible. Even building a custom Expo client didn’t help. Eventually, I abandoned Expo and switched to pure React Native, which proved to be the correct choice for further native integration and neural network usage.

Next, I ran into a fundamental limitation of mobile platforms: the JavaScript engine on Android and iOS is not designed to handle large binary files. Files like captions.json and especially embeddings.bin (248 MB) cannot simply be loaded into memory from JavaScript without risking OutOfMemory errors or extremely slow performance due to constant data copying. Yet, these files are essential for the disease detection model, where searches are performed over precomputed embedding vectors.

To solve this, all work with large data and vector search is moved to the native layer on Android by implementing a custom Kotlin native module, called FaissSearchModule. Its main purpose is to load the embedding file, store the data outside the JavaScript heap, and perform vector search entirely in native memory. Instead of loading embeddings.bin into RAM, a memory-mapped file is used via MappedByteBuffer. This allows the operating system to manage memory and load data on demand, without holding the entire file in the app’s memory. Essentially, the embedding file becomes part of the process’s virtual memory, which is crucial for handling large datasets on mobile devices.

The nearest-vector search is also performed entirely in native code. Only the query embedding is sent from JavaScript, and the results returned are compact: indexes, distances, and text labels for diseases. This minimizes the load on the JavaScript-to-native bridge and avoids transferring large arrays back and forth. A similar approach is used for captions.json: instead of standard JSON parsing, the file is read streamingly, preventing the need to load the entire file into memory. These files are located in the Android project directory for proper access.

package com.berkano

import com.facebook.react.bridge.*
import org.json.JSONArray
import java.io.*
import java.nio.ByteBuffer
import java.nio.ByteOrder
import kotlin.math.sqrt

/**
 * Native module for vector search using Faiss
 * Data is stored in native memory and does not enter the JavaScript heap
 */
class FaissSearchModule(
    private val reactContext: ReactApplicationContext
) : ReactContextBaseJavaModule(reactContext) {

    // Store data in native memory
    // Use MappedByteBuffer instead of FloatArray for large files
    // This allows avoiding loading all data into RAM at once
    private var embeddingsBuffer: java.nio.MappedByteBuffer? = null
    private var captions: List<String>? = null
    private var numVectors: Int = 0
    private var embeddingSize: Int = 512
    private var embeddingsFile: File? = null

    override fun getName(): String {
        return "FaissSearch"
    }

    /**
     * Loads embeddings from a binary file using a memory-mapped file
     * Does NOT load all data into RAM, reads on demand
     */
    @ReactMethod
    fun loadEmbeddings(embeddingsPath: String, promise: Promise) {
        try {
            val file = File(embeddingsPath)
            if (!file.exists()) {
                promise.reject("FILE_NOT_FOUND", "Embeddings file not found: $embeddingsPath")
                return
            }

            val fileSize = file.length()
            numVectors = (fileSize / 4 / embeddingSize).toInt() // 4 bytes per float

            // Use MappedByteBuffer — this does NOT fully load the file into RAM
            // The operating system manages memory and loads data on demand
            FileInputStream(file).use { fis ->
                val channel = fis.channel
                val byteBuffer = channel.map(
                    java.nio.channels.FileChannel.MapMode.READ_ONLY,
                    0,
                    fileSize
                )
                byteBuffer.order(ByteOrder.LITTLE_ENDIAN)
                embeddingsBuffer = byteBuffer
            }

            embeddingsFile = file

            val result = Arguments.createMap()
            result.putInt("numVectors", numVectors)
            result.putInt("embeddingSize", embeddingSize)
            result.putLong("fileSize", fileSize)

            promise.resolve(result)
            println("Mapped $numVectors vectors of size $embeddingSize from file (${fileSize / 1024 / 1024}MB)")
        } catch (e: OutOfMemoryError) {
            promise.reject(
                "OUT_OF_MEMORY",
                "Not enough memory to map embeddings file. File too large: ${e.message}",
                e
            )
        } catch (e: Exception) {
            promise.reject("LOAD_ERROR", "Failed to load embeddings: ${e.message}", e)
        }
    }

    /**
     * Reads a vector by index from the memory-mapped file
     * Does not load all data into memory
     */
    private fun getVector(index: Int): FloatArray {
        if (embeddingsBuffer == null) {
            throw IllegalStateException("Embeddings not loaded")
        }

        val startByte = index * embeddingSize * 4 // 4 bytes per float
        val vector = FloatArray(embeddingSize)

        embeddingsBuffer!!.position(startByte)
        embeddingsBuffer!!.asFloatBuffer().get(vector)

        return vector
    }

    /**
     * Loads captions from a JSON file
     * Uses streaming parsing for large files
     */
    @ReactMethod
    fun loadCaptions(captionsPath: String, promise: Promise) {
        try {
            val file = File(captionsPath)
            if (!file.exists()) {
                promise.reject("FILE_NOT_FOUND", "Captions file not found: $captionsPath")
                return
            }

            // Use BufferedReader for efficient reading of large files
            val captionsList = mutableListOf<String>()
            var buffer = StringBuilder()
            var inString = false
            var escapeNext = false

            BufferedReader(FileReader(file), 8192).use { reader ->
                var char: Int
                while (reader.read().also { char = it } != -1) {
                    when {
                        escapeNext -> {
                            buffer.append(char.toChar())
                            escapeNext = false
                        }
                        char == '\\'.code -> {
                            escapeNext = true
                        }
                        char == '"'.code -> {
                            if (inString) {
                                // End of string
                                captionsList.add(buffer.toString())
                                buffer = StringBuilder()
                                inString = false
                            } else {
                                // Start of string
                                inString = true
                            }
                        }
                        inString -> {
                            buffer.append(char.toChar())
                        }
                        char == ']'.code -> {
                            // End of array
                            break
                        }
                    }
                }
            }

            captions = captionsList
            val result = Arguments.createMap()
            result.putInt("count", captionsList.size)

            promise.resolve(result)
            println("Loaded ${captionsList.size} captions")
        } catch (e: OutOfMemoryError) {
            promise.reject(
                "OUT_OF_MEMORY",
                "Not enough memory to load captions. File too large: ${e.message}",
                e
            )
        } catch (e: Exception) {
            promise.reject("LOAD_ERROR", "Failed to load captions: ${e.message}", e)
        }
    }

    /**
     * Computes cosine distance between two vectors
     */
    private fun cosineDistance(vecA: FloatArray, vecB: FloatArray): Float {
        var dotProduct = 0f
        var normA = 0f
        var normB = 0f

        for (i in vecA.indices) {
            dotProduct += vecA[i] * vecB[i]
            normA += vecA[i] * vecA[i]
            normB += vecB[i] * vecB[i]
        }

        normA = sqrt(normA)
        normB = sqrt(normB)

        if (normA == 0f || normB == 0f) {
            return 1f
        }

        val similarity = dotProduct / (normA * normB)
        return 1f - similarity
    }

    /**
     * Finds top-K nearest vectors to the query
     * Search is performed in native memory, only results are sent to JS
     */
    @ReactMethod
    fun search(
        queryEmbedding: ReadableArray,
        topK: Int,
        promise: Promise
    ) {
        try {
            if (embeddingsBuffer == null) {
                promise.reject("NOT_LOADED", "Embeddings not loaded. Call loadEmbeddings first.")
                return
            }

            if (captions == null) {
                promise.reject("NOT_LOADED", "Captions not loaded. Call loadCaptions first.")
                return
            }

            // Convert query from ReadableArray to FloatArray
            val query = FloatArray(queryEmbedding.size())
            for (i in 0 until queryEmbedding.size()) {
                query[i] = queryEmbedding.getDouble(i).toFloat()
            }

            if (query.size != embeddingSize) {
                promise.reject(
                    "INVALID_SIZE",
                    "Query embedding size ${query.size} != $embeddingSize"
                )
                return
            }

            // Compute distances for all vectors
            // Vectors are read on demand from the memory-mapped file
            val distances = mutableListOf<Pair<Int, Float>>()

            for (i in 0 until numVectors) {
                // Read vector from file on demand
                val vector = getVector(i)
                val distance = cosineDistance(query, vector)
                distances.add(Pair(i, distance))
            }

            // Sort and take top-K
            distances.sortBy { it.second }
            val topResults = distances.take(topK)

            // Prepare results to send to JS
            val results = Arguments.createArray()
            for ((index, distance) in topResults) {
                val resultItem = Arguments.createMap()
                resultItem.putInt("index", index)
                resultItem.putDouble("distance", distance.toDouble())
                resultItem.putString("caption", captions!![index])
                results.pushMap(resultItem)
            }

            promise.resolve(results)
        } catch (e: Exception) {
            promise.reject("SEARCH_ERROR", "Search failed: ${e.message}", e)
        }
    }

    /**
     * Clears loaded data from memory
     */
    @ReactMethod
    fun clearCache(promise: Promise) {
        embeddingsBuffer = null
        embeddingsFile = null
        captions = null
        numVectors = 0
        promise.resolve(null)
    }

    /**
     * Checks whether data is loaded
     */
    @ReactMethod
    fun isLoaded(promise: Promise) {
        val result = Arguments.createMap()
        result.putBoolean("embeddingsLoaded", embeddingsBuffer != null)
        result.putBoolean("captionsLoaded", captions != null)
        result.putInt("numVectors", numVectors)
        promise.resolve(result)
    }
}

The FaissSearchPackage plays a supporting role in this setup. Its purpose is to register the native module in React Native and inform the JavaScript part of the app that FaissSearchModule is available. All the actual logic and computation happens inside the module itself; the package serves purely as an infrastructure layer to connect it to the React Native framework.

package com.berkano

import com.facebook.react.ReactPackage
import com.facebook.react.bridge.NativeModule
import com.facebook.react.bridge.ReactApplicationContext
import com.facebook.react.uimanager.ViewManager

class FaissSearchPackage : ReactPackage {
    override fun createNativeModules(
        reactContext: ReactApplicationContext
    ): List<NativeModule> {
        return listOf(FaissSearchModule(reactContext))
    }

    override fun createViewManagers(
        reactContext: ReactApplicationContext
    ): List<ViewManager<*, *>> {
        return emptyList()
    }
}

Besides working with the embedding files, there’s another problem that can’t be solved purely in JavaScript. All ONNX models in the project expect a tensor of fixed shape and format as input, whereas in React Native an image usually comes as a URI or a base64 string. Converting an image to a numeric tensor in JavaScript is possible, but in practice it either runs slowly or consumes a lot of memory, especially for camera images. For this reason, input preprocessing is also moved to the native layer.

For this, a separate native module, ImageDecoderModule, was implemented. Its role is to take an image URI, load it via ContentResolver, decode it into a Bitmap, resize it to the required dimensions, and convert it into a CHW-format tensor expected by the PyTorch-trained models exported to ONNX. At this stage, pixel values are normalized to the [0,1] range, and RGB channels are explicitly separated so that the resulting array exactly matches the model input.

Importantly, all of this logic runs entirely on the Android side before passing the data to JavaScript. React Native cannot directly handle FloatArray, so the tensor is converted into a WritableArray, which is then safely passed across the bridge. This approach ensures the array is created once in native code without intermediate copies or unnecessary allocations in the JS engine, significantly reducing memory usage and speeding up data preparation for inference.

package com.berkano;

import android.graphics.Bitmap
import android.graphics.BitmapFactory
import android.net.Uri
import com.facebook.react.bridge.*
import java.io.InputStream

class ImageDecoderModule(
  private val reactContext: ReactApplicationContext
) : ReactContextBaseJavaModule(reactContext) {

  override fun getName(): String {
    return "ImageDecoder"
  }

  @ReactMethod
  fun decodeToTensor(
    uriString: String,
    targetWidth: Int,
    targetHeight: Int,
    promise: Promise
  ) {
    try {
      val uri = Uri.parse(uriString)
      val inputStream: InputStream? =
        reactContext.contentResolver.openInputStream(uri)

      if (inputStream == null) {
        promise.reject("ERROR", "Cannot open image stream")
        return
      }

      var bitmap: Bitmap? = null
      try {
        bitmap = BitmapFactory.decodeStream(inputStream)
        if (bitmap == null) {
          promise.reject("ERROR", "Cannot decode image")
          return
        }
        bitmap = Bitmap.createScaledBitmap(bitmap, targetWidth, targetHeight, true)
      } finally {
        inputStream.close()
      }

      val width = bitmap.width
      val height = bitmap.height

      val pixels = IntArray(width * height)
      bitmap.getPixels(pixels, 0, width, 0, 0, width, height)

      // CHW: [3, H, W] — format used for passing data to JavaScript
      val tensor = FloatArray(3 * width * height)

      for (y in 0 until height) {
        for (x in 0 until width) {
          val color = pixels[y * width + x]

          val r = (color shr 16 and 0xFF) / 255f
          val g = (color shr 8 and 0xFF) / 255f
          val b = (color and 0xFF) / 255f

          val idx = y * width + x
          tensor[idx] = r
          tensor[width * height + idx] = g
          tensor[2 * width * height + idx] = b
        }
      }

      // Convert FloatArray to WritableArray for proper transfer to JavaScript
      // React Native cannot directly convert FloatArray, so WritableArray is used
      val writableArray = Arguments.createArray()
      for (value in tensor) {
        // pushDouble accepts double, so convert float to double
        writableArray.pushDouble(value.toDouble())
      }

      promise.resolve(writableArray)
    } catch (e: Exception) {
      promise.reject("ERROR_DECODING_IMAGE", e)
    }
  }
}

Just like with the vector search module, ImageDecoderModule is registered via a separate ImageDecoderPackage, which is then included in MainApplication.kt. Without this, React Native won’t recognize the module. Additionally, the required permissions must be explicitly declared in AndroidManifest.xml; otherwise, the app won’t be able to access images. All these details are available in the project repository, in the Android part of the mobile application.

package com.berkano;

import com.facebook.react.ReactPackage
import com.facebook.react.bridge.NativeModule
import com.facebook.react.bridge.ReactApplicationContext
import com.facebook.react.uimanager.ViewManager

class ImageDecoderPackage : ReactPackage {

  override fun createNativeModules(
    reactContext: ReactApplicationContext
  ): List<NativeModule> {
    return listOf(ImageDecoderModule(reactContext))
  }

  override fun createViewManagers(
    reactContext: ReactApplicationContext
  ): List<ViewManager<*, *>> {
    return emptyList()
  }
}

In the end, all heavy and memory-sensitive operations remain on the native side, while JavaScript only works with preprocessed data and inference results.

Now, for running the models on a mobile device, I created a separate file that handles preparing, copying, initializing models, and running inference. On the first run, the code checks whether the model files and associated assets exist in the local filesystem. If not, they are copied from the assets folder (on Android). The logic is implemented in the copyAssetIfNeeded function. On Android, it’s recommended to copy large files using copyFileAssets.

The prepareModelAssets function gathers all necessary paths to models, embeddings, and auxiliary files, returning an object with these paths for later use. ONNX models are initialized via initializeModel, which creates an ONNX session with NNAPI priority for GPU/NPU, falling back to CPU if needed.

Inference is performed in runInference, where a Float32Array containing the image tensor (prepared by the native modules, as described in the ImageDecoderModule) is passed in. The function constructs a tensor in the correct format for the model, e.g., [1, 3, 224, 224] for disease detection, runs the model, and returns a normalized embedding.

import * as ort from 'onnxruntime-react-native';
import RNFS from 'react-native-fs';
import { Platform } from 'react-native';

async function copyAssetIfNeeded(assetName: string, folder: string): Promise<string> {
  const localPath = `${RNFS.DocumentDirectoryPath}/${assetName}`;

  const exists = await RNFS.exists(localPath);
  if (!exists) {
    try {
      if (Platform.OS === 'android') {
        // Android: read from assets
        // Path must be relative to the assets folder: folder/assetName
        const assetPath = `${folder}/${assetName}`;
        console.log(`Attempting to copy from assets: ${assetPath} to ${localPath}`);
        await RNFS.copyFileAssets(assetPath, localPath);
        console.log(`Successfully copied ${assetName} to local FS: ${localPath}`);
      } else {
        // iOS: read from the main bundle
        const source = `${RNFS.MainBundlePath}/${folder}/${assetName}`;
        console.log(`Attempting to copy from bundle: ${source} to ${localPath}`);
        await RNFS.copyFile(source, localPath);
        console.log(`Successfully copied ${assetName} to local FS: ${localPath}`);
      }
    } catch (err: any) {
      console.error(`Failed to copy ${assetName} from ${folder}/${assetName}:`, err);
      console.error(`Error details:`, {
        message: err?.message,
        code: err?.code,
        platform: Platform.OS,
        localPath,
        assetPath: `${folder}/${assetName}`,
      });
      // Try an alternative path without the folder (in case the file is in the root of assets)
      if (Platform.OS === 'android') {
        try {
          console.log(`Trying alternative path: ${assetName}`);
          await RNFS.copyFileAssets(assetName, localPath);
          console.log(`Successfully copied using alternative path`);
        } catch (altErr) {
          console.error(`Alternative path also failed:`, altErr);
          throw new Error(
            `Failed to copy ${assetName}. Please verify that the file is located in assets/${folder}/`
          );
        }
      } else {
        throw err;
      }
    }
  } else {
    console.log(`${assetName} already exists at ${localPath}`);
  }

  return localPath;
}

export async function prepareModelAssets() {
  const modelPath = await copyAssetIfNeeded('disease_detection.onnx', 'models');
  const embeddingsPath = await copyAssetIfNeeded('embeddings.bin', 'files');
  const captionsPath = await copyAssetIfNeeded('captions.json', 'files');
  const classMappingPath = await copyAssetIfNeeded('class_mapping.txt', 'files');
  const speciesMappingPath = await copyAssetIfNeeded('species_id_to_name.txt', 'files');

  return { 
    modelPath, 
    embeddingsPath, 
    captionsPath,
    classMappingPath,
    speciesMappingPath,
  };
}

export interface ModelSession {
  session: ort.InferenceSession;
  inputName: string;
  outputName: string;
}

let modelSession: ModelSession | null = null;

/**
 * Initializes the ONNX model with GPU/NPU priority via NNAPI
 */
export async function initializeModel(): Promise<ModelSession> {
  if (modelSession) {
    return modelSession;
  }

  try {
    // Use the path obtained from prepareModelAssets
    const { modelPath } = await prepareModelAssets();

    // Use the local model path
    const modelUri = modelPath;

    // Session options with NNAPI priority (GPU/NPU)
    const sessionOptions: ort.InferenceSession.SessionOptions = {
      executionProviders: ['nnapi', 'cpu'], // NNAPI for GPU/NPU, CPU as fallback
      graphOptimizationLevel: 'all',
    };

    // Create the inference session
    // onnxruntime-react-native can work directly with file paths
    const session = await ort.InferenceSession.create(modelUri, sessionOptions);

    // Retrieve input and output names
    const inputNames = session.inputNames;
    const outputNames = session.outputNames;

    if (inputNames.length === 0 || outputNames.length === 0) {
      throw new Error('The model has no inputs or outputs');
    }

    modelSession = {
      session,
      inputName: inputNames[0],  // Assume the first input
      outputName: outputNames[0], // Assume the first output
    };

    console.log('Model successfully loaded');
    console.log('Input name:', modelSession.inputName);
    console.log('Output name:', modelSession.outputName);

    return modelSession;
  } catch (error) {
    console.error('Error while loading the model:', error);
    throw error;
  }
}

/**
 * Runs model inference on an image
 * @param imageTensor Image tensor in [1, 3, 224, 224] format
 * @returns Image embedding
 */
export async function runInference(
  imageTensor: Float32Array,
): Promise<Float32Array> {
  if (!modelSession) {
    throw new Error('Model is not initialized. Call initializeModel() first.');
  }

  try {
    // Create input tensor
    // Format: [batch, channels, height, width] = [1, 3, 224, 224]
    const inputTensor = new ort.Tensor('float32', imageTensor, [1, 3, 224, 224]);

    // Run inference
    const feeds = { [modelSession.inputName]: inputTensor };
    const results = await modelSession.session.run(feeds);

    // Retrieve output tensor
    const outputTensor = results[modelSession.outputName];
    const embedding = outputTensor.data as Float32Array;

    // Normalize the embedding
    const norm = Math.sqrt(
      Array.from(embedding).reduce((sum, val) => sum + val * val, 0),
    );
    const normalizedEmbedding = new Float32Array(embedding.length);
    for (let i = 0; i < embedding.length; i++) {
      normalizedEmbedding[i] = embedding[i] / norm;
    }

    return normalizedEmbedding;
  } catch (error) {
    console.error('Error during inference:', error);
    throw error;
  }
}

/**
 * Releases model resources
 */
export function disposeModel(): void {
  if (modelSession) {
    modelSession.session.release();
    modelSession = null;
  }
}

To manage multiple models, I created modelManager.ts. It allows registering different models (disease, plant, age) and managing their loading into memory. Only two models are kept in memory at the same time to avoid overloading the device. If the limit is reached, the least recently used model is unloaded.

import * as ort from 'onnxruntime-react-native';
import RNFS from 'react-native-fs';
import { Platform } from 'react-native';

export type ModelType = 'disease' | 'plant' | 'age';

export interface ModelInfo {
  type: ModelType;
  path: string;
  session: ort.InferenceSession | null;
  inputName: string;
  outputName: string;
  isLoaded: boolean;
  lastUsed: number;
}

// Maximum number of models allowed in memory at the same time
const MAX_MODELS_IN_MEMORY = 2;

// Model cache
const modelCache: Map<ModelType, ModelInfo> = new Map();

/**
 * Copies a file from assets to the local file system
 */
async function copyAssetIfNeeded(assetName: string, folder: string): Promise<string> {
  const localPath = `${RNFS.DocumentDirectoryPath}/${assetName}`;

  const exists = await RNFS.exists(localPath);
  if (!exists) {
    try {
      if (Platform.OS === 'android') {
        const assetPath = `${folder}/${assetName}`;
        console.log(`Copying ${assetName} from assets...`);
        await RNFS.copyFileAssets(assetPath, localPath);
        console.log(`Successfully copied ${assetName}`);
      } else {
        const source = `${RNFS.MainBundlePath}/${folder}/${assetName}`;
        await RNFS.copyFile(source, localPath);
      }
    } catch (err: any) {
      console.error(`Failed to copy ${assetName}:`, err);
      if (Platform.OS === 'android') {
        try {
          await RNFS.copyFileAssets(assetName, localPath);
        } catch (altErr) {
          throw new Error(`Failed to copy ${assetName}`);
        }
      } else {
        throw err;
      }
    }
  }

  return localPath;
}

/**
 * Unloads the least recently used model from memory
 */
function unloadLeastUsedModel(): void {
  let leastUsed: ModelInfo | null = null;
  let leastUsedTime = Date.now();

  for (const model of modelCache.values()) {
    if (model.isLoaded && model.lastUsed < leastUsedTime) {
      leastUsed = model;
      leastUsedTime = model.lastUsed;
    }
  }

  if (leastUsed) {
    console.log(`Unloading model: ${leastUsed.type}`);
    if (leastUsed.session) {
      leastUsed.session.release();
      leastUsed.session = null;
      leastUsed.isLoaded = false;
    }
  }
}

/**
 * Loads a model into memory
 */
async function loadModel(modelInfo: ModelInfo): Promise<void> {
  if (modelInfo.isLoaded && modelInfo.session) {
    modelInfo.lastUsed = Date.now();
    return;
  }

  // Check if the memory limit for loaded models has been reached
  const loadedModels = Array.from(modelCache.values()).filter(m => m.isLoaded);
  if (loadedModels.length >= MAX_MODELS_IN_MEMORY) {
    console.log('Memory limit reached, unloading least recently used model...');
    unloadLeastUsedModel();
  }

  try {
    console.log(`Loading model: ${modelInfo.type}`);

    const sessionOptions: ort.InferenceSession.SessionOptions = {
      executionProviders: ['nnapi', 'cpu'],
      graphOptimizationLevel: 'all',
    };

    const session = await ort.InferenceSession.create(modelInfo.path, sessionOptions);

    const inputNames = session.inputNames;
    const outputNames = session.outputNames;

    if (inputNames.length === 0 || outputNames.length === 0) {
      throw new Error('The model has no inputs or outputs');
    }

    modelInfo.session = session;
    modelInfo.inputName = inputNames[0];
    modelInfo.outputName = outputNames[0];
    modelInfo.isLoaded = true;
    modelInfo.lastUsed = Date.now();

    console.log(`Model ${modelInfo.type} loaded successfully`);
  } catch (error) {
    console.error(`Error loading model ${modelInfo.type}:`, error);
    throw error;
  }
}

/**
 * Initializes the model management system
 */
export async function initializeModelManager(): Promise<void> {
  // Prepare model paths
  const diseaseModelPath = await copyAssetIfNeeded('disease_detection.onnx', 'models');
  const plantModelPath = await copyAssetIfNeeded('plant_classification.onnx', 'models');
  const ageModelPath = await copyAssetIfNeeded('plant_analysis.onnx', 'models');

  // Register models
  modelCache.set('disease', {
    type: 'disease',
    path: diseaseModelPath,
    session: null,
    inputName: '',
    outputName: '',
    isLoaded: false,
    lastUsed: 0,
  });

  modelCache.set('plant', {
    type: 'plant',
    path: plantModelPath,
    session: null,
    inputName: '',
    outputName: '',
    isLoaded: false,
    lastUsed: 0,
  });

  modelCache.set('age', {
    type: 'age',
    path: ageModelPath,
    session: null,
    inputName: '',
    outputName: '',
    isLoaded: false,
    lastUsed: 0,
  });

  console.log('Model manager initialized');
}

/**
 * Retrieves a model (loads it if necessary)
 */
export async function getModel(modelType: ModelType): Promise<ModelInfo> {
  const modelInfo = modelCache.get(modelType);
  if (!modelInfo) {
    throw new Error(`Model ${modelType} not found`);
  }

  await loadModel(modelInfo);
  return modelInfo;
}

/**
 * Unloads a model from memory
 */
export function unloadModel(modelType: ModelType): void {
  const modelInfo = modelCache.get(modelType);
  if (modelInfo && modelInfo.session) {
    console.log(`Unloading model: ${modelType}`);
    modelInfo.session.release();
    modelInfo.session = null;
    modelInfo.isLoaded = false;
  }
}

/**
 * Unloads all models from memory
 */
export function unloadAllModels(): void {
  for (const modelType of modelCache.keys()) {
    unloadModel(modelType);
  }
}

/**
 * Checks whether a model is loaded
 */
export function isModelLoaded(modelType: ModelType): boolean {
  const modelInfo = modelCache.get(modelType);
  return modelInfo?.isLoaded ?? false;
}

For loading and running the models, I use the same approach with onnxruntime-react-native. After obtaining the modelInfo object via getModel:

const modelInfo = await getModel('age');

an image tensor is created:

const inputTensor = new ort.Tensor('float32', imageTensor, [1, 3, n, n]);

This tensor is fed into the model, and the output tensors are returned immediately.

const feeds = { [modelInfo.inputName]: inputTensor };
const results = await modelInfo.session.run(feeds);

The examples available in plantClassificationService.ts and diseaseSearchService.ts demonstrate how to feed embeddings, normalize them, and perform vector search using Faiss. This approach is useful not only for this app but also for any mobile project that deals with large models and vector search.

A few words about the project itself and its purpose. At first glance, it might seem like just a collection of neural networks for plants, running on mobile devices with some optimizations. In reality, it’s an experiment in transferring complex models to resource-constrained devices, managing memory efficiently, and building unified code for different types of tasks. I wrote it not only for readers but also as notes for myself.

If you’ve made it to the end of this article, it means you enjoy diving into the details and aren’t afraid of challenges. The project is open-source under the MIT license, and all code is available on GitHub, so anyone can try it out, fine-tune models, optimize performance, or suggest new features. Finally, I invite everyone interested to join as contributors: every idea, optimization, or small feature makes the project better and more interesting. Sometimes a small experiment leads to unexpected discoveries, and this project is all about experimenting and finding those discoveries.

How I Tried to Save My Google Play Developer Account — and Ended Up Making a App for Crypto

Wlad Radchenko — Mon, 19 May 2025 10:32:32 +0000

I got a surprise email from Google recently:

“Your developer account is inactive and may be closed…”

I created this account back when I was still a student, mainly to publish a couple of small side projects. Then I kind of forgot about it. Years passed. Now Google warned me I had 60 days to publish something new — or the account would be deleted. And honestly? I didn’t want to lose it. It held a little piece of my dev journey.

"Okay," I thought. "I’ll whip something up real quick. Should take, like, 10 minutes!"

And that’s how I ended up building a full open-source crypto portfolio app — with donuts and LLM generated prompts. Here’s how it all spiraled out of control.

From Quick Fix to Full-on Crypto App

My first plan? Just update an old app. But I had forgotten how to React Native. Then I figured, hey, maybe I’ll add some features. A bit of design polish. A few UX tweaks…

One weekend later, I’d rebuilt the UI from scratch, rewritten all the logic, and ended up with a neat little open-source app that builds randomized crypto portfolios and generates ChatGPT prompts for each coin.

How It Works

The app fetches crypto data from the CoinGecko API
Random coins are selected and diversified based on how much money you have
Each coin is displayed as a donut chart
You tap a button and get a prompt for ChatGPT or any LLM, like:
- What's the investment risk level for this token (low/medium/high) and why?
- Does it have good profit potential?
- Is now a good time to buy, or should I wait?
- How reliable/safe is the token based on its market cap, volume, and supply structure?
- Give a short, actionable recommendation.

And yep, the app saves your portfolio history and lets you share it with friends. I was inspired during the creation by this picture:

Under the Hood

⚠️ Warning: there's a lot of stream-of-consciousness code below. If you want to know what came out of it, scroll to the end of the article.

What was going through my head while coding:

I’ll walk you through how I built it, the traps I ran into with Expo + React Native, and whether I actually managed to save my developer account in the end.

Requirements

You’ll need:

Node.js (v14+)
Android Studio (for Android) or Xcode (for iOS)
React Native with Expo
And EAS Build for creating Android bundles
The full source is on GitHub.

If you’re a total npm newbie, don’t worry — just run:

git clone <project>
сd <project>
npm install
npm run android or npm run ios

Goodbye Old Splash Screen

The first thing I ditched was the outdated splash screen. I replaced it with a clean background and a login screen. Installed the needed packages and created a new SplashScreen.tsx.

npm install react-native-responsive-screen

And code in SplashScreen.tsx:

import { useRef } from 'react';
import { View, Text, Image, StyleSheet, TouchableOpacity, Animated } from 'react-native';
import { widthPercentageToDP as wp, heightPercentageToDP as hp } from 'react-native-responsive-screen';

const SplashScreen = ({ onHide }) => {
  const fadeAnim = useRef(new Animated.Value(1)).current;
  const scaleAnim = useRef(new Animated.Value(1)).current;

  // Animation after clicking a button
  const handleGetStarted = () => {
    Animated.parallel([
        Animated.timing(fadeAnim, {
        toValue: 0,
        duration: 500,
        useNativeDriver: true,
        }),
        Animated.timing(scaleAnim, {
        toValue: 2, // you can reduce it to 0.5 or 0.1 if you want a collapse, but I have an increase
        duration: 500,
        useNativeDriver: true,
        }),
    ]).start(() => onHide());
   };

  // Draw View  
  return (
    <Animated.View style={[styles.container, { opacity: fadeAnim, transform: [{ scale: scaleAnim }] }]}>
      <View>
        <Image 
            source={require('./assets/donuts/background.png')} 
            style={{ marginTop: hp('4.76%'), width: wp('109%'), height: hp('57.14%') }}
            resizeMode="contain">
        </Image>
      </View>
      <View style={styles.content}>
        <Text style={styles.title}>Doughfolio</Text>
        <Text style={styles.description}>
          Visualize your crypto portfolio with delicious donut charts
        </Text>
      </View>
      <TouchableOpacity style={styles.button} onPress={handleGetStarted}>
        <Text style={styles.buttonText}>Get Started</Text>
      </TouchableOpacity>
    </Animated.View>
  );
};


const styles = StyleSheet.create({
  container: {
    position: 'absolute',
    top: 0,
    left: 0,
    right: 0,
    bottom: 0,
    backgroundColor: '#FFD8DF',
    justifyContent: 'center',
    alignItems: 'center',
    zIndex: 1000,
  },
  content: {
    flex: 1,
    justifyContent: 'flex-end',
    alignItems: 'flex-start',
    paddingHorizontal: wp('7.27%'),
    marginBottom: hp('7.61%'),
  },
  title: {
    fontSize: wp('14%'),
    fontWeight: 'bold',
    color: '#FF6E76',
    marginBottom: hp('1.9%'),
    textAlign: 'left'
  },
  description: {
    fontSize: wp('5.09%'),
    color: '#FF6E76',
    textAlign: 'left',
  },
  button: {
    backgroundColor: 'white',
    borderRadius: 28,

    // Shadow for Android
    elevation: 10,

    // Shadow for iOS
    shadowColor: '#9B8084',
    shadowOffset: { width: 5, height: 5 },
    shadowOpacity: 0.1,
    shadowRadius: 10,

    paddingHorizontal: wp('14.72%'),
    paddingVertical: hp('2%'),
    width: '80%',
    marginBottom: hp('5.71%'),
    alignItems: 'center',
    justifyContent: 'center'
  },
  buttonText: {
    color: 'black',
    textTransform: 'uppercase',
    fontSize: wp('4.72%'),
    fontWeight: 'bold',
  },
});

// We set export to export SplashScreen to App.tsx
export default SplashScreen;

Feeling fancy? You can even add Lottie animations. Just grab a JSON animation from LottieFiles (you'll need an account), install Lottie, and embed the animation in your view.

npm install lottie-react-native

And integrate LottieView:

<LottieView source={require('./assets/animation.json')} autoPlay loop />

In App.tsx, I layered the SplashScreen over the main content and used splashVisible state to toggle its visibility.

export default function App() {
  const [splashVisible, setSplashVisible] = useState(true);

  const [fontsLoaded] = useFonts({
    'Roboto-Bold': require('./src/assets/fonts/Roboto-Bold.ttf'),
    'Roboto-Light': require('./src/assets/fonts/Roboto-Light.ttf'),
  });

  return (
    <View style={{ flex: 1 }}>
      <DonutChartContainer />
      {splashVisible && (
        <SplashScreen onHide={() => setSplashVisible(false)} />
      )}
    </View>
  );
}

Coin Data + Donuts

The interesting part starts in App.tsx, where the app fetches data:

// Function for getting data from CoinGecko API
async function fetchCryptoData() {
  const response = await fetch('https://api.coingecko.com/api/v3/coins/markets?vs_currency=usd&order=market_cap_desc&per_page=200&page=1');
  const data = await response.json();
  return data;
}

// Function to randomly select 10 cryptocurrencies
function getRandomCryptos(data, count = 10, maxIndex = 200) {
  const minSlice = Math.floor(Math.random() * (maxIndex - count + 1));  // from 0 to 90
  const maxSlice = minSlice + count;
  return data.slice(minSlice, maxSlice);
}

 const handleHistorySelect = (item: any) => {
  setData(item.data);
  // Updating totalValue via withTiming for smooth animation
  totalValue.value = withTiming(item.totalValue, { duration: 500 });
  // Recalculating percentages
  decimals.value = item.data.map(crypto => crypto.percentage / 100);
};

  ////////////////////////////
 // BREAKING THE TEMPLATE //
///////////////////////////

try {
  // Mix up the donut pictures
  setImages(getShuffledDonutImages());

  // Step 1: Get data from the API
  const cryptoData = await fetchCryptoData();

  // Step 2: Randomly select 10 cryptocurrencies
  const selectedCryptos = getRandomCryptos(cryptoData, 10);

  // Step 3: Generate random numbers for weight distribution
  const generateNumbers = generateRandomNumbers(n, amount);

  // Calculate the total sum of these numbers
  const total = generateNumbers.reduce((acc, currentValue) => acc + currentValue, 0);

  // Calculate percentages for each number
  const generatePercentages = calculatePercentage(generateNumbers, total);

  // Round off percentages and make them in the format 0.00
  const generateDecimals = generatePercentages.map((number) => {
    if (number != null && !isNaN(number)) {
      return Number(number.toFixed(0)) / 100;
    }
    return 0; // Yes, there can be null values from the API
  });

  totalValue.value = withTiming(total, { duration: 1000 });

  decimals.value = [...generateDecimals];

  // Generate an array of objects with data
  const arrayOfObjects = generateNumbers.map((value, index) => ({
    name: selectedCryptos[index].name,
    image: selectedCryptos[index].image,
    symbol: selectedCryptos[index].symbol,
    minPrice: selectedCryptos[index].ath,
    maxPrice: selectedCryptos[index].atl,
    price: selectedCryptos[index].current_price,
    marketCap: selectedCryptos[index].market_cap,
    marketCapChangePercentage24h: selectedCryptos[index].market_cap_change_percentage_24h,
    priceChangePercentage24h: selectedCryptos[index].price_change_percentage_24h,
    circulatingSupply: selectedCryptos[index].circulating_supply,
    maxSupply: selectedCryptos[index].max_supply,
    totalVolume: selectedCryptos[index].total_volume,
    value,
    percentage: generatePercentages[index],
    decimals: generateDecimals[index] / 100,
    color: colors[index], // Random color generation
    url: 'https://www.coingecko.com/en/coins/' + selectedCryptos[index].id,
  }));

  // Output data to the console
  setData(arrayOfObjects);
  await addToHistory(arrayOfObjects); // Save data + total amount
} catch (error) {
  console.error('Failed to data generation:', error);
}

When setData is called, we’re updating the data state inside the DonutChartContainer — which then redraws the chart.

const [data, setData] = useState<Data[]>([]);

Bonus: there’s also a useHistory.ts hook that saves every generated portfolio to AsyncStorage under the key cryptoDonutHistory. Firstly installation:

npm install @react-native-async-storage/async-storage

And code of useHistory.ts:

// src/hooks/useHistory.ts
import AsyncStorage from '@react-native-async-storage/async-storage';
import { useState, useEffect } from 'react';

interface HistoryItem {
  date: string;
  data: any[];
  totalValue: number; 
}

export const useHistory = () => {
  const [history, setHistory] = useState<HistoryItem[]>([]);

  const loadHistory = async () => {
    try {
      const saved = await AsyncStorage.getItem('cryptoDonutHistory');
      if (saved) {
        const parsed = JSON.parse(saved);
        setHistory(parsed);
      }
    } catch (e) {
      console.error('Failed to load history', e);
    }
  };

  interface HistoryItem {
    date: string;
    data: {
      name: string; 
      value: number; // Price in $
      percentage: number;
      color: string;
      image: string;  // Image for coin
      symbol: string; //  Symbol of coin
    }[];
    totalValue: number;
  }

  const clearHistory = async () => {  // Clear AsyncStorage for cryptoDonutHistory
    try {
      await AsyncStorage.removeItem('cryptoDonutHistory');
      setHistory([]);
      return true;
    } catch (e) {
      console.error('Cleaning error:', e);
      return false;
    }
  };

  const addToHistory = async (newData: any[]) => {  // Added data in AsyncStorage
    const totalValue = newData.reduce((sum, item) => sum + item.value, 0);
    try {
      const newItem = { 
        date: new Date().toLocaleString(),
        data: newData,
        totalValue,
      };
      const updatedHistory = [newItem, ...history];
      await AsyncStorage.setItem('cryptoDonutHistory', JSON.stringify(updatedHistory));
      setHistory(updatedHistory);
    } catch (e) {
      console.error('Failed to save history', e);
    }
  };

  useEffect(() => {
    loadHistory();
  }, []);

  return { history, addToHistory, clearHistory };
};

Localized Number Formatting

In the US, we write numbers like $324,654,765. But in many other countries, the comma is a decimal separator. Which means your users might read that as $324 and some cents. Not ideal.

To fix this, I formatted all numbers using the device’s locale — so users see values in the format they expect.

export const formatNumber = (value, { isCurrency = false, currency = 'USD', minimumFractionDigits = 0, maximumFractionDigits = 2 } = {}) => {
  if (value == null || isNaN(value)) return '∞';

  return value.toLocaleString(undefined, {
    style: isCurrency ? 'currency' : 'decimal',
    currency,
    minimumFractionDigits,
    maximumFractionDigits,
  });
};

// Also uses safeToFixed to safely round numbers.
export const safeToFixed = (value, decimals = 2) => {
  if (isNaN(value) || value == null) {
    // If the value is not a number or null, return a string with the default value
    return '0.00';
  }
  return value.toFixed(decimals);
}

The Finishing Touches

With 4 hours left before midnight, I focused on the visuals. I found the tastiest-looking donut images on Vecteezy (free license), designed some Play Store slides in Corel Vector, and uploaded everything.

I hit “Submit” just after midnight, watched the build go through, and went to bed thinking I was safe from the next inactivity purge for a while. Here’s the app on Google Play if you're curious.

But Then, Google Strikes Again

Next morning?

“Your developer account may be at risk due to inactivity.”

Come on, Google! I just published an app! I followed the rules!

In the Play Console, the message basically said:

Confirm your email and phone number (✅ done)
Publish a new app or release an update (✅ also done)

So yeah. Either they didn’t register my app as “active” yet, or I’ll need to wait and see if the red banner disappears. But hey — I now have:

An open-source project I’m proud of
A maybe-doomed dev account
And a ton of ideas for the next version

Thinking of adding a map with donut shops, built-in LLM chat, and even more data in the future. I'm planning how I'll implement new features:

Let’s Trade Stories

Got your own horror story about Google Play or dev accounts going rogue? Drop it in the comments. Share your thoughts, join the open-source repo — and if your story’s spicy enough, I owe you a donut.

GitHub repo: https://github.com/wladradchenko/doughfolio.wladradchenko.ru
Google Play: https://play.google.com/store/apps/details?id=com.wladradchenko.donut
Made with: ☕, 🍩, and 😅

Why My Open Source Project Wunjo Can’t Reach 1K Stars? 😢

Wlad Radchenko — Tue, 25 Mar 2025 08:09:50 +0000

"Build it, and they will come..." – They said.
"Release an awesome AI-powered tool, and the stars will rain down on you!" – They said.

P.S. Now there is no trust in articles and posts about how to earn a lot of stars on GitHub and that a beautiful README matters a lot. All of this is fiction that does not work today. This myth was 💥 Busted!

Well, here I am, stuck at 979 stars 😩, and that number has barely moved for weeks.

I’ve been building Wunjo, an Open Source AI-powered video editing tool that can today automatically cut, highlight, and transform videos with a simple text prompt. Sounds cool, right? Yet, getting to 1K stars on GitHub feels like an endless grind. This is a set of tools in software to optimization process of video, photo editing and API (API Docs) inside for other pet-projects.

I don’t enjoy manual video editing—it takes forever. Finding the right cuts, trimming, syncing audio… it’s exhausting. That’s why I built Wunjo:

✅ Upload any video (even hours long!)
✅ Describe what you want: "Make a highlights reel," "Cut out dull moments," "Find all goals & red cards"
✅ AI does the rest – detects key moments, edits them together, and delivers a polished video. You also can settings duuration, for example to get shorts for YouTube or TikTok.

Wunjo can even analyze specific fragments to answer:
❓ "What’s happening in this scene?"
❓ "What are people wearing?"
❓ "How is the light falling?"

Future updates might allow editing videos based on sound, for making interview shorts, news clips, and more.

But… Where Are My Stars?

I’m constantly improving the project, adding features, optimizing performance, and sharing updates. But that 1K star milestone just keeps teasing me. Only 21 stars left, yet it’s been like this for WEEKS.

So, fellow devs, I need your wisdom:

💬 How do you promote your Open Source projects?
💬 Have you struggled with GitHub star plateaus?
💬 Any tips for getting more eyes on a project?

Let’s talk! And, well… if Wunjo sounds useful, maybe drop a star? ⭐😉

👉 GitHub Repo and Patreon.

P.S. Also, check out the project pages on Product Hunt, Product Radar, and DevHunt.

Your coding year in review

Wlad Radchenko — Mon, 09 Dec 2024 18:03:50 +0000

Exploring the internet, I stumbled upon GitHub Unwrapped—a fascinating tool that automatically creates a video based on your GitHub activity over the past year. It showcases your most-used languages, those spontaneous late-night coding sessions, stars earned, and more. Plus, it’s open source.

This gem is built using Remotion, another open-source project that enables automating video creation with React for web. The documentation is good, but you need to figure it out. I thought it was so cool that I had to share it n dev.to!

At the end of the video, I added a little extra—an update about my own open-source pet project, which contributed heavily to my stats and the video itself. Fun fact: Wunjo was recently named Product of the Week on Product Radar with record number votes from Makers!

DevHunt Weekly: Unfair Rankings or Just a Glitch?

Wlad Radchenko — Tue, 26 Nov 2024 07:40:44 +0000

Last week, I launched my open source project, Wunjo Community Edition CE, on DevHunt.

I’m reflecting on the results of this week’s DevHunt in this post, as it’s a platform where open-source projects and tools for developers are launched and compete. Since Dev.to is a place where people often share about development, useful resources, and tools for IT professionals, I believe many of you might consider launching your projects on DevHunt. I want to share my disappointing experience so you can be aware of potential pitfalls.

I already wrote thoughts about it, the positive and negative sides that I felt as a user, you can read them on dev.to.

The voting lasted all week, and I kept a close eye on the rankings. With just 12 hours to go (Voting ends on Monday), I received an email stating that my project was in second place, while another project, WebDraw on DevHunt, was down at 10th place. The position of the project in the letter indicates its place in the current voting by upvote.

Two hours before the voting ended, I checked again—Wunjo was on second rank, and WebDraw wasn’t even in the top five. However, when the results were announced, I was shocked to find that WebDraw had surged into third place, while Wunjo was bumped to fourth.

Here’s where things don’t add up

Votes and Impressions Tell a Different Story

Both Wunjo and WebDraw have the same number of upvotes, but the impressions for Wunjo were 2896, compared to WebDraw's 1362. Impressions, as I understand them, are based on project views and retention in the top list, meaning they reflect actual engagement. This discrepancy raises serious questions about the legitimacy of WebDraw’s sudden surge.

Timing of the Shift

The bulk of WebDraw’s movement happened in the last two hours of the voting period. For a project that had little activity throughout the week to suddenly leapfrog multiple projects at the finish line—it just doesn’t feel organic.

The Notification That Confused Me Further

After the results were announced, I received an email with the subject "Celebrating Your Remarkable Third Place Win on DevHunt!" (screenshot attached). But when I checked the platform, there was no badge, no acknowledgment of third place—just disappointment.

I even reached out to DevHunt founder John Rush on X (formerly Twitter) but haven’t received a response yet.

Disclaimer: This post is not a call to avoid launching your projects on DevHunt—I’m simply reflecting on my experience and sharing what happened during my recent launch. I hope this sparks a discussion and maybe even some much-needed change.

Why This Matters

This experience left me disheartened, not just because Wunjo didn’t receive the recognition it deserved, but because this undermines the transparency and trust that platforms like DevHunt rely on. Even if the situation gets corrected, the moment for social sharing, celebrating, and engaging with the community has already passed. Those who followed the voting will only remember the unfair results.

It’s not about winning—it’s about fairness.

My Questions to the Community

Have you ever experienced something similar? How did it make you feel?
What do you think about platforms like DevHunt? Should they improve transparency in their voting processes?

Let’s discuss! I’m curious to hear your thoughts and ideas.

For me, this has been a lesson in resilience. Despite the disappointment, I still believe in Wunjo and its potential. I hope this post helps bring awareness to the importance of fair competition, especially in spaces meant to uplift and celebrate creators.

Looking forward to your thoughts and stories!

Update: Justice Restored!

I’m happy to share that fairness has prevailed—Wunjo is now officially among the top three projects on DevHunt! 🏆🎉

Thank you to the Dev.to and DevHunt communities, as well as John Rush, for addressing this issue. Your support means a lot, and it gives me hope that transparency and fairness are valued on platforms like this.

Let’s continue building a positive and supportive environment for creators!

How Else Can You Promote Your Open Source Product?

After launching on Product Hunt, I saw my first sales and got a wave of new users — definitely worth it. But surprisingly, Product Radar brought in even more traffic and conversions, and it was way simpler to launch there — no waiting for a feature spot or special scheduling. Beyond that, platforms like BetaList and AppSumo can also give your product strong early exposure and help you grow your user base fast.