DEV Community: Yuvraj Raghuvanshi

The Website That Looked Like It Needed Selenium (But Didn’t)

Yuvraj Raghuvanshi — Thu, 30 Apr 2026 10:33:32 +0000

For my thesis I needed a large corpus of Hindi poetry. Hindwi is one of the better maintained Hindi literature archives on the internet. Thousands of poems, hundreds of poets, content spanning from the 8th century to contemporary writers. It had everything I needed.

I didn’t plan to spend much time on the scraper. Collect the data, move on.

That didn’t happen.

The Obvious Problem

Visit hindwi.org/poets and you’ll see a listing of poets. Scroll down and more appear. Visit an individual poet’s page and the same thing happens — poems load as you scroll. This is the pattern that makes every scraper writer reach for Selenium almost reflexively. The content isn’t in the initial HTML. JavaScript is loading it dynamically. You need a browser.

So I set up Selenium. Headless Chrome, scroll simulation, wait for elements to appear, extract content. It worked. It was also agonizingly slow.

The real problem wasn’t just speed — it was that Selenium is fundamentally impractical to parallelize. You can’t easily spin up ten browser instances and scrape ten poets simultaneously the way you can with threads making HTTP requests. Each browser instance carries its own rendering engine, memory space, and JavaScript runtime. The resource cost compounds quickly, and the coordination between instances is a nightmare. Even with aggressive parallelism, back-of-envelope math on 25,000+ poems made it clear this would take days, not hours.

There had to be a better way.

Ten Minutes in DevTools

Before writing any more Selenium code, I opened the browser DevTools Network tab and watched what actually happened when the page loaded more content.

This is always worth doing before committing to browser automation. Dynamic-looking behavior on the frontend is still, at the network level, just HTTP requests. The browser has to get the data from somewhere. The question is whether that somewhere is directly reachable.

On Hindwi, when you scroll to the bottom of the poets listing, the browser fires a request like this:

https://www.hindwi.org/PoetCollection?lang=2&pageNumber=2&Info=poet
&StartsWith=&keyword=&typeID=659186cb-44e7-4d94-8b1a-fc70f939a733
&TypeSlug=poets&contentFilter=&_=1777462454692

Plain GET request. No authentication tokens in the body, no encrypted signatures, no WebSocket handshake. Just query parameters. The _=1777462454692 at the end is a cache-busting timestamp the browser adds automatically - the server doesn't validate it, so scrapers can ignore it entirely.

The response that came back was raw HTML — not JSON, not XML. Just HTML cards containing poet names, dates, and profile links, ready to be injected into the DOM. So the website wasn’t serving a proper API, but it was serving something structured, paginated, and directly reachable over plain HTTP.

Screenshot: DevTools Network tab showing the /PoetCollection request and its HTML response body

The next question was: how does the browser know what URL to request for page 3, page 4, page 5? Where does that information come from?

The URL Was Sitting Right There

I looked at the page source. And there they were — all of them, already embedded in the initial HTML response:

<div class="contentLoadMore">
    <div class="contentLoadMorePaging" 
         data-url="/PoetCollection?lang=2&pageNumber=3&Info=poet
                  &StartsWith=&keyword=&typeID=659186cb-44e7-4d94-8b1a-fc70f939a733
                  &TypeSlug=poets&contentFilter=">
        <svg class="screenLoader" ...></svg>
    </div>
</div>

The site pre-embeds the URLs for every subsequent page inside data-url attributes on div.contentLoadMorePaging elements. The JavaScript reads these attributes and fires the requests when you scroll into view. But from a scraper's perspective, the URLs are already there in the first response - you don't need to scroll anything. You just parse them out and fetch them directly.

This was the moment Selenium became irrelevant.

What looked like dynamic JavaScript-driven content was really just a simple pattern: fetch the initial page, extract the hidden data-url values, make those HTTP requests directly. No browser. No scroll simulation. No waiting for DOM mutations.

Screenshot: page source with the contentLoadMorePaging div and data-url attribute visible

The Same Pattern, Everywhere

Once I knew what to look for, I checked the individual poet pages. Same pattern. A poet with more than 50 poems (Mona Gulati, for example) has this in her initial page response:

<div class="contentLoadMore">
    <div class="contentLoadMorePaging" 
         data-url="/PoetCollection?lang=2&pageNumber=2&info=ghazals
                  &SEO_Slug=kavita&Id=34074990-5be7-43e9-8a85-6aaa0be4833c
                  &Info=ghazal&StartsWith=a&typeID=659186cb-...
                  &contentType=kavita&sort=popularity-desc&filter=">
    </div>
    <div class="contentLoadMorePaging" 
         data-url="/PoetCollection?lang=2&pageNumber=3...">
    </div>
</div>

Both page 2 and page 3 are listed upfront in the first response. The site hands you the complete roadmap immediately. Fetch once, and you know exactly what to fetch next — no interaction, no scrolling, no waiting.

This held for dohas, quotes, and every other content type on the site. The contentLoadMorePaging pattern was consistent across all of Hindwi. Understanding it once meant the whole site was open.

Turning the Insight Into Code

The scraper that came out of this is conceptually simple. For the poet listing, hit the /PoetCollection endpoint and keep incrementing pageNumber until you get an empty response:

def _get_paginated_poet_cards(self, info, extra_params=None):
    page = 1
    while True:
        params = {"lang": 2, "pageNumber": page, "Info": info}
        if extra_params:
            params.update(extra_params)

        soup = get_soup(POETS_ENDPOINT, params=params)
        cards = soup.select("div.poetColumn")
        if not cards:
            break
        yield from cards
        page += 1

For poem lists, fetch the poet’s kavita page, parse whatever poems are already in the initial HTML, then extract and follow every data-url:

def _extract_poem_metadata(self, kavita_url):
    soup = get_soup(kavita_url)
    poems = self._parse_poem_list(soup)

    pagination_divs = soup.select("div.contentLoadMorePaging[data-url]")
    seen_urls = set()
    for div in pagination_divs:
        data_url = div.get("data-url")
        if not data_url or data_url in seen_urls:
            continue
        seen_urls.add(data_url)
        full_url = urljoin("https://www.hindwi.org", data_url)
        paginated_soup = get_soup(full_url)
        poems.extend(self._parse_poem_list(paginated_soup))
    return poems

No browser. No scroll events. Two BeautifulSoup calls per paginated poet.

One thing worth mentioning about _parse_poem_list: the initial page and the dynamically loaded fragment pages use different CSS classes for their poem cards. The initial listing uses div.rt_contentBodyListItems, while the paginated HTML fragments come back using div.contentListItems.nwPoetListBody. I caught this when certain poets were returning suspiciously fewer poems than their profile pages suggested - the paginated content was being silently skipped because the selector only matched the first class. A multi-selector handles both:

cards = soup.select(
    "div.rt_contentBodyListItems, div.contentListItems.nwPoetListBody"
)

This is exactly the kind of thing that produces wrong results silently. No error, no exception — just a poem count that’s quietly lower than it should be.

Screenshot: terminal output showing a poet being processed with their correct poem count

Extracting the Poems

Each poem lives on its own URL. The page serves the text in Devanagari and, for many poems, a Romanized transliteration toggled by a button. In the HTML, both versions are already present — just hidden or shown depending on which toggle is active:

# Devanagari
hindi_div = soup.find("div", {"class": "pMC", "data-roman": "off"})

# Romanized
roman_div = soup.find("div", id="HindwiRoman")
roman_pmc = roman_div.find("div", {"class": "pMC", "data-roman": "on"})

The text itself is structured as

tags containing tags per word or phrase. Joining the spans within each paragraph gives one line:

for p in hindi_div.find_all("p"):
    line = " ".join(span.get_text(strip=True) for span in p.find_all("span"))
    if line.strip():
        hindi_lines.append(line)

Both versions get saved as separate plain text files. Not every poem has a Romanized version, so the code returns None for the roman field when it doesn't exist rather than an empty list - preserving the distinction between "no Roman version" and "Roman version is blank."

Screenshot: a poem page on Hindwi showing the Devanagari text alongside the Roman toggle

Concurrency — The Real Payoff

With Selenium out of the picture, threading became trivial. The poem scraper processes all poets concurrently with a thread pool:

def scrape_poems(self, max_workers=10):
    with ThreadPoolExecutor(max_workers=10) as executor:
        futures = [executor.submit(self._process_poet, poet, i, total)
                   for i, poet in enumerate(self.poets)]
        for future in as_completed(futures):
            future.result()

Ten threads making lightweight HTTP requests is nothing. This is what was completely impractical with Selenium — ten browser instances would have needed a dedicated server to run without thrashing. Ten requests threads ran fine on a laptop, barely registering on the CPU.

Every request goes through a shared get_soup wrapper that enforces a 1-second politeness delay and retries with exponential backoff on failures. Errors at any level - a single poem, an entire poet - get logged and skipped rather than crashing the thread. The run completed cleanly over about two hours. A small number of URLs consistently returned server errors and landed in the log; everything else went through without issue.

The Result

Two hours. 25,000+ poems across hundreds of poets. Devanagari and Romanized versions where available. Structured metadata including titles, URLs, slugs, and categories per poem. Around 300MB of text in total.

The dependency list tells the whole story:

beautifulsoup4==4.13.4
requests==2.32.4

No Selenium, no browser drivers, no Playwright, no headless Chrome. Just HTTP requests and HTML parsing.

What I Took From This

The instinct to reach for Selenium when you see dynamic content is understandable — it’s the safe default that definitely works. But dynamic content loading just means the browser is making HTTP requests after the initial page load. Those requests go somewhere, return something, and in most cases can be replicated directly.

The contentLoadMorePaging pattern on Hindwi is a good illustration of how often websites like this are more accessible than they appear. The site wasn't hiding anything. It was handing out pagination URLs in plain HTML, sitting in data-url attributes, ready to be read. JavaScript happened to be the first thing reading them - until a scraper was.

Ten minutes in the Network tab before writing any scraping code is almost always worth it. In this case, it was the difference between days of Selenium pain and a two-hour requests script that finished before lunch.

This article is for educational purposes — all ethical considerations have been addressed, including measures such as rate limiting and conducting scraping during periods of low website traffic.

This article is rewritten using AI chatbots.

April 30, 2026

Training a Classifier on Huge Dataset When RAM Is Not Your Friend

Yuvraj Raghuvanshi — Mon, 13 Apr 2026 18:05:03 +0000

I didn’t set out to build a custom data loader. I set out to train a model on the Quick, Draw! dataset.

The data pipeline was supposed to be the boring part — the few lines you write before the interesting work starts. It ended up being most of the work, the source of the most frustrating bugs, and, in retrospect, the most interesting engineering decision of the whole project.

This is the story of why I ended up with a directory containing millions of individual .npy files, and why that turned out to be the right call.

What Quick, Draw! Actually Is

Quick, Draw! is a Google dataset of human drawings collected from a browser game where players had 20 seconds to draw a prompted word. It has 345 categories — cats, airplanes, zigzags, The Eiffel Tower — with up to 100,000 drawings per class. That’s about 50 million drawings in total.

What makes it interesting for ML, and annoying for data pipelines, is that each drawing has two representations:

Raster images — each drawing rendered as a 28×28 grayscale bitmap, stored as a flat array of 784 values. These come in .npy files where a single file for one class contains an array of shape (N, 784). For 100,000 samples, that's 100,000 rows of 784 floats per file.

Stroke sequences — the original drawing data: a sequence of (dx, dy, pen_state) triplets representing how the pen moved. These come in .npz files, split into train, val, and test keys. The stroke data varies in length per drawing - a simple zigzag might have 10 points, a detailed drawing of The Great Wall of China might have hundreds.

The model I wanted to build was multimodal: it would take both representations as input simultaneously, letting a CNN process the image and an LSTM process the stroke sequence, then merge their outputs for classification. Which meant the pipeline had to serve both modalities in sync, for every sample, across 345 classes.

Screenshot: a sample drawings from Quick, Draw! — both the raster image and stroke visualization side by side

The Naive Approach and Why It Dies

The obvious first attempt is the one-liner:

data = np.load("cat.npy") # shape: (~100000, 784)

That loads fine for one class. You run it for a few classes, you’re still fine. Then somewhere around class 20 or 30 your process gets killed by the OOM killer, or your Jupyter kernel crashes silently, or the remote server you’ve SSH’d into drops your connection and takes your training run with it.

With 345 classes at 30,000 samples each (my chosen limit) — we’re talking about loading roughly 10 million samples into RAM at startup. At around 11% of a 128GB server’s memory for 10,000 samples per class, the math on 30,000 samples gets uncomfortable fast. And that’s before you account for the stroke data.

The real problem isn’t just peak RAM usage. It’s that loading everything upfront means you can’t start training until loading finishes, the loaded arrays stay resident for the entire run, and any shuffle operation has to work over the full dataset in memory. All of this compounds.

There’s also a subtlety with the stroke files: they come pre-split into train/val/test partitions. If you want to do your own splits (which you do, so you can control the ratio and the random seed), you need to recombine them first and re-split yourself.

So before we get to the loader itself, there are three preprocessing steps to run.

Step 1: Downloading the Data

The download script fetches both file types from Google’s Cloud Storage. The listing endpoint returns XML, which the script parses to find the URLs for the classes you’ve defined in base_classes. Downloads run in parallel using a thread pool:

with concurrent.futures.ThreadPoolExecutor(max_workers=10) as executor:
    executor.map(lambda url: download_file(url, download_folder), file_urls)

Two separate calls — one for .npy raster files, one for .npz stroke files, filtered to the sketchrnn/ prefix:

download_quickdraw_files(..., file_type="npy")
download_quickdraw_files(..., file_type="npz", prefix_filter="sketchrnn/")

Files that already exist are skipped, which matters when you’re running this on a remote server where connections drop and you have to restart.

Screenshot: terminal output during download — the [DOWNLOAD] and [SKIP] lines showing parallel fetching

Step 2: Recombining the Stroke Splits

Each stroke .npz file has three keys: train, val, and test. Left as-is, you're working with a subset of the available data. The fix is to concatenate them:

combined = np.concatenate([data["train"], data["val"], data["test"]], axis=0)
np.savez_compressed(out_path, strokes=combined)

This runs in parallel across all classes using ProcessPoolExecutor. One thing worth noting: after combining, gc.collect() is called explicitly. This is a multiprocessing context and Python's garbage collector doesn't always release memory between processes the way you'd expect. Without this, a machine with moderate RAM will start sweating as dozens of processes hold combined arrays simultaneously.

Step 3: The Key Idea — One File Per Sample

This is the decision everything else depends on.

Instead of keeping each class as a single large .npy file, we explode every sample out into its own file:

dataset/processed/
  images/
    cat/
      000001.npy ← shape: (28, 28, 1)
      000002.npy
      ...
  strokes/
    cat/
      000001.npy ← shape: (130, 3)
      000002.npy
      ...

The conversion script loops over every class, loads the class-level arrays, preprocesses each sample, and saves them individually. The index is global across all classes — not per-class — which is what keeps image and stroke files aligned:

global_idx = 0
max_samples_per_class=100_000

for label_name, _ in LABEL_MAP.items():
    images = np.load(img_path, mmap_mode="r") # note: memory-mapped
    strokes = np.load(
                stroke_path, allow_pickle=True, encoding="latin1"
              )["strokes"]

    N = min(len(images), len(strokes), max_samples_per_class)

    for i in range(N):
        idx = global_idx + i
        np.save(
          f"images/{label_name}/{idx:06d}.npy", 
          preprocess_image(images[i])
        )
        np.save(
          f"strokes/{label_name}/{idx:06d}.npy", 
          preprocess_strokes(strokes[i])
        )

    global_idx += N

The image loading uses mmap_mode="r" - memory-mapped, so NumPy doesn't load the entire (100000, 784) array into RAM just to iterate over it row by row. The preprocessing happens at this stage, not at training time, so the generator later is just doing file reads.

This step takes a while to run. On the upside, it runs once.

Screenshot: the processed/ directory structure — showing the per-class subdirectories with numbered .npy files

What Preprocessing Actually Does

Images are straightforward. Reshape (784,) to (28, 28), divide by 255 to get [0, 1] floats, expand the channel dimension:

img = flat_img.reshape(28, 28).astype(np.float32) / 255.0
return np.expand_dims(img, axis=-1) # (28, 28, 1)

Strokes are more involved. The raw data uses relative coordinates — each (dx, dy) is an offset from the previous point, not an absolute position. This makes sense for how drawings are recorded but not for how a model should see them. The preprocessing converts to absolute, centers the drawing at the origin, then scales to a fixed [-100, 100] range:

# Relative -> absolute
strokes[:, 0] = np.cumsum(strokes[:, 0])
strokes[:, 1] = np.cumsum(strokes[:, 1])

# Center at origin
strokes[:, 0] -= strokes[:, 0].mean()
strokes[:, 1] -= strokes[:, 1].mean()

# Scale to [-100, 100]
max_coord = max(np.abs(strokes[:, 0]).max(), np.abs(strokes[:, 1]).max())
if max_coord > 0:
    strokes[:, 0] *= 100.0 / max_coord
    strokes[:, 1] *= 100.0 / max_coord

Stroke sequences are variable length. To get a fixed-size tensor for the LSTM, sequences are either truncated or zero-padded to 130 points. Why 130? Empirically, that covers the vast majority of drawings in the dataset without wasting too many zeros on the short ones.

The pen state column (the third feature) is left as-is — it’s already a binary indicator of whether the pen is lifted.

Screenshot: before/after visualization of a stroke — raw relative coordinates as a mess of lines, then the centered/normalized version looking like the actual drawing

The Loader

After preprocessing, the index step is fast. We walk the processed directory and collect all file paths:

for cls, label in LABEL_MAP.items():
    image_files = sorted(glob(f"{PROCESSED_DATA_DIR}/images/{cls}/*.npy"))
    stroke_files = sorted(glob(f"{PROCESSED_DATA_DIR}/strokes/{cls}/*.npy"))

N = min(len(image_files), len(stroke_files), SAMPLES_PER_CLASS)
    for i in range(N):
        images.append(image_files[i])
        strokes.append(stroke_files[i])
        labels.append(label)

At this point, images and strokes are just lists of strings. Nothing has been loaded into memory. The total dataset - 345 classes × 30,000 samples - indexes in a few seconds.

There’s also a threshold in the config: IN_MEMORY_THRESHOLD = 30_000. If SAMPLES_PER_CLASS is below that number, the loader will actually call np.load() during indexing and store the arrays directly. For quick experiments on a subset of data, this avoids the per-sample I/O overhead at training time. For large runs, it streams from disk instead.

USE_IN_MEMORY = USE_INDIVIDUAL and (SAMPLES_PER_CLASS <= IN_MEMORY_THRESHOLD)

Both paths feed into the same generator interface, which is a nice property — you can switch between them by changing one number.

The Generator and the tf.data Pipeline

The generator is a Python function that yields (image, stroke, one_hot_label) tuples:

def data_generator(images, strokes, labels):
    if USE_IN_MEMORY:
        for image, stroke, label in zip(images, strokes, labels):
            yield image, stroke, tf.one_hot(label, depth=NUM_CLASSES)
    elif USE_INDIVIDUAL:
        for img_path, str_path, label in zip(images, strokes, labels):
            yield (
                    np.load(img_path), 
                    np.load(str_path), 
                    tf.one_hot(label, depth=NUM_CLASSES)
                  )

This feeds into a tf.data.Dataset via from_generator, which requires explicit output signatures - TensorFlow needs to know shapes and dtypes upfront since it can't infer them from a Python generator:

output_signature = (
    tf.TensorSpec(shape=(28, 28, 1), dtype=tf.float32),
    tf.TensorSpec(shape=(130, 3), dtype=tf.float32),
    tf.TensorSpec(shape=(NUM_CLASSES,), dtype=tf.int32),
)

ds = tf.data.Dataset.from_generator(gen, output_signature=output_signature)

The full pipeline adds shuffling (shuffles a buffer of 10× the batch size rather than the entire dataset), repeating, batching at 512, and prefetching:

def build_dataset(images, strokes, labels, is_shuffle=False):
    ds = tf.data.Dataset.from_generator(gen, output_signature=output_signature)
    if is_shuffle:
        ds = ds.shuffle(BATCH_SIZE * 10)
    ds = ds.repeat()
    ds = ds.map(format_sample, num_parallel_calls=tf.data.AUTOTUNE)
    ds = ds.batch(BATCH_SIZE)
    return ds.prefetch(tf.data.AUTOTUNE)

The format_sample step reformats the yielded tuple into the dictionary format Keras expects for multi-input models:

def format_sample(img, stroke, label):
    return {"stroke_input": stroke, "image_input": img}, label

Shuffling indices, not files, is important here. The file layout on disk stays sequential — images for cat are in one directory, images for airplane in another. The shuffle happens in the data pipeline as it reads, which avoids random I/O seeks across the disk. Sequential reads are substantially faster than random ones, and the OS page cache will warm up the recently accessed files naturally.

Screenshot: htop showing RAM usage during training — relatively flat, not growing with training time

Splitting the Dataset

The split is index-based. We shuffle a global index array once with a fixed seed, then slice it:

indices = np.arange(total)
np.random.seed(42)
np.random.shuffle(indices)

train_end = int(0.8 * total)
val_end = train_end + int(0.1 * total)
train_idx = indices[:train_end]
val_idx = indices[train_end:val_end]
test_idx = indices[val_end:]

The 80/10/10 ratio applies across all classes since the indexing step already interleaved everything. There’s no risk of a class being entirely in the training set and absent from validation.

Validation and test datasets use .take() to consume a fixed number of batches - computed from the split sizes - since the generator repeats indefinitely:

val_ds = build_dataset(
            val_images, val_strokes, val_labels
          ).take(math.ceil(len(val_labels) / BATCH_SIZE))
test_ds = build_dataset(
            test_images, test_strokes, test_labels
          ).take(math.ceil(len(test_labels) / BATCH_SIZE))

What Went Wrong Along the Way

File count. The processed dataset ends up with roughly 345 × 30,000 × 2 = 20.7 million files. Some filesystems handle this poorly. If you're on a filesystem with inode limits or slow directory listing (common with some HPC storage systems), the sorted(glob(...)) calls at index time can take several minutes. Structured subdirectories (one per class) help, but it's still a lot of files.

Index alignment. The global index scheme — where file names reflect position across all classes, not within a class — exists entirely to prevent a specific bug. An earlier version used per-class indices, which caused a silent alignment failure: image cat/000001.npy and stroke cat/000001.npy were always aligned, but after shuffling, the code was pulling from globally-indexed lists and the class-local numbering didn't correspond. The {idx:06d} naming ensures that whatever index you retrieve from the lists, the image and stroke file names will match.

Training on a remote server with an unstable SSH connection. The training history in the notebook has a gap. BackupAndRestore meant the model weights survived; the history object didn't. TensorBoard logs were the fallback, and the actual metrics are there - the notebook's loss and accuracy plots just show what was available from the Python history object after reconnecting. If you're doing long training runs remotely, save the history separately and frequently, not just at the end.

Memory growth with TensorFlow’s GPU allocator. By default, TensorFlow pre-allocates the entire GPU memory. For a machine shared with other users, or one running other processes, this is a problem. The fix is:

for gpu in gpus:
    tf.config.experimental.set_memory_growth(gpu, True)

This makes TensorFlow allocate GPU memory incrementally as needed. It’s not set by default because it can slightly reduce performance in some scenarios, but for shared environments it’s basically always the right call.

What I’d Do Differently

The main thing I’d want to add is parallel file loading. Right now the generator is single-threaded — it loads one sample at a time, yields it, repeats. tf.data.AUTOTUNE on the prefetch helps by trying to keep the pipeline filled ahead of the model's consumption, but the actual I/O is sequential. Adding multiple generator workers (like PyTorch's num_workers) would reduce the time the GPU spends waiting for data.

LMDB would also be worth experimenting with. The advantage over millions of small files is that it’s a single file that supports fast key-value lookup, sequential reading, and doesn’t suffer from filesystem overhead per-entry. The disadvantage is that it complicates the setup and makes debugging harder. For this project the small-files approach was fast enough, but at larger scale it would start to matter.

A smarter caching strategy (keeping recently accessed samples in a bounded RAM buffer) would also help with the “warm up” problem. The first epoch is always slower than subsequent ones because the OS page cache starts cold. A pre-warmed in-memory buffer for the most frequently accessed samples would smooth that out.

The Part That Surprised Me

When I first sketched this out, my expectation was that disk-based loading would be noticeably slower than loading everything to RAM — enough to be a real bottleneck. It wasn’t, for a reason that only became clear after thinking about it: individual .npy file loads are fast. A (28, 28, 1) array at float32 is 3,136 bytes. A (130, 3) stroke array is 1,560 bytes. These are tiny files. The actual read time per sample is in the low microseconds, and the OS cache handles repeat accesses to recently-read files transparently.

What you trade away compared to pure in-memory loading is predictability. With everything in RAM, access time is constant. With disk loading, you’re occasionally hitting a file that isn’t cached, and that read takes longer. In practice, the prefetch buffer absorbs most of this variance. The GPU never actually sat idle waiting for data in my runs — the bottleneck was always computation, not I/O.

The other thing that surprised me was how much the single-file-per-class approach had been hiding. When everything for cat is one big (100000, 784) array, you have no choice but to load the whole thing before you can access any of it. That's a loading cost you pay every time. With individual files, you pay per sample - and you only pay for the samples you actually use.

The Notebook Setup (in case it’s useful)

One thing worth mentioning for anyone running this on a remote server: the port forwarding setup for Jupyter. If you’re SSH-ing into a machine and want to run notebooks rather than pulling .py files and running them in screen sessions, you forward the Jupyter port to localhost:

ssh -L 8888:localhost:8888 user@server_ip

# On the server:
jupyter notebook --no-browser --port=8888

If you’re going through two layers of SSH (e.g. a department gateway server that routes to a compute node), you just carry the forwarding through:

ssh -L 8888:localhost:8888 user@gateway
# On gateway:
ssh -L 8888:localhost:8888 user@compute_node

And for full control over Python and library versions, running the kernel inside a virtual environment is worth the setup time:

python3.12 -m venv .tens
source .tens/bin/activate
pip install jupyter ipykernel tensorflow numpy tqdm matplotlib
python -m ipykernel install --user --name=.tens

Then you can select .tens as the kernel in Jupyter and know exactly what Python version and library versions are running - which matters if you're planning to later quantize the model and deploy it somewhere like a Raspberry Pi, where the environment constraints are much stricter.

The pipeline ended up being more engineered than I originally wanted. But it runs, it doesn’t crash, and it’ll scale to more classes or more samples per class without changes. For a dataset this size on a memory-constrained machine, that’s the bar.

The code is all in the repository if you want to look at the actual implementation rather than the edited excerpts here. I’ll make this public once the paper is accepted.

This article is rewritten using AI chatbots.

April 14, 2026

Reverse Engineering SmartLock by Parivahan: What I Found Inside a Python Proctoring App

Yuvraj Raghuvanshi — Tue, 07 Apr 2026 12:14:33 +0000

I didn’t plan to reverse engineer a proctoring application. I just wanted to understand why a page kept refreshing in an infinite loop.

That one puzzling symptom ended up pulling me down a rabbit hole that took days to climb out of — involving PyInstaller internals, broken decompilers, browser automation quirks, and a race condition that convincingly pretended to be tamper detection. The journey was longer than I expected, and honestly more interesting. So I figured I might as well write it up.

The application in question is SmartLock by Parivahan , a proctoring system used for government driver’s learning license exams in India, and yes, I am getting a driving license at the age of 25. We all start somewhere.

It’s a Python desktop app that locks down your machine, watches your screen, monitors USB ports, controls your browser, and talks to a remote server — all at the same time. Understanding how it does all of this, and how the pieces fit together, is what this article is about.

Phase 1: Opening the Box

The first thing I did was look at the installation directory. This usually tells you a lot before you write a single line of code.

_internal/
browser/
config/
log/
Pictures/
Smartlock.exe

That _internal/ folder was the giveaway. It's a classic PyInstaller signature. The folder contains Python libraries and compiled bytecode - essentially, a self-contained Python runtime bundled into a single executable.

Screenshot: Installation directory structure showing _internal/ folder and Smartlock.exe

So the application was built in Python and packaged using PyInstaller. That meant extraction was possible using pyinstxtractor:

python pyinstxtractor.py Smartlock.exe

After extraction, the structure looked like this:

Smartlock.exe_extracted/
├── PYZ-00.pyz_extracted/
│ ├── asyncio/
│ ├── psutil/
│ ├── pydivert/
│ ├── selenium/
│ ├── websockets/
│ ├── win32com/
│ ├── yaml/
│ ├── controller.pyc
│ ├── registry_edit.pyc
│ └── ...
├── core.pyc
└── ...

Most of what you see here is noise — third-party libraries. The signal is in the handful of .pyc files: core.pyc, controller.pyc, registry_edit.pyc. These contain the actual application logic. Everything else is plumbing.

Phase 2: Decompilation and Why It’s Always Messier Than It Sounds

This is where things got annoying.

I tried the standard tools first:

uncompyle6 core.pyc
decompyle3 core.pyc

Both failed. Version mismatch — the bytecode was compiled with a Python version these tools didn’t fully support. I eventually got somewhere using pychaos, but I want to be honest about what “decompiled code” actually looks like in practice. It’s not clean. Comments are gone (they’re never stored in bytecode). Control flow gets reconstructed heuristically and is often wrong. You get artifacts like this:

while __CHAOS_PY_TEST_NOT_INIT_ERR__ :
    message = yield None

When the actual code probably looked something like:

async for message in websocket:
    msg = json.loads(message)

The decompiler is doing its best, but it’s guessing. Reverse engineering at this level is less about reading code and more about reconstructing intent from imperfect evidence. You develop a feel for what the code is trying to do, even when the syntax is broken.

The three key files broke down roughly like this:

core.pyc Main orchestrator - startup, thread management

controller.pyc Enforcement logic - monitoring, detection

registry_edit.pyc OS-level restrictions - registry modifications

Phase 3: Reconstructing the Architecture

Once I had a working (if imperfect) picture of the code, the overall architecture became clear:

Screenshot: detailed diagram showcasing the connections between SmartLock application, bundled Chrome application, and remote server

It’s a tightly coupled system. The desktop app and the browser aren’t independent — they’re in constant communication. And both of them are talking to the remote server. Remove any one of these connections and the whole thing breaks.

Phase 4: The Browser That Kept Redirecting

After reconstructing and running the application, I ran into something strange.

The exam webpage was redirecting continuously to a 403.jsp webpage and in a split second back to exam login webpage. Every few seconds — reload, reload, reload. My first instinct was that this was intentional tamper detection. After all, the whole point of a proctoring system is to detect when something isn’t right. Maybe it had detected something about my environment and was punishing me with an infinite loop.

That turned out to be wrong. But figuring out why it was wrong took a while.

The browser bundled with SmartLock isn’t a standard Chrome installation. It’s a portable Chromium build with a preconfigured user profile:

browser/
├── App/
│ └── Chrome-bin/
├── Data/
│ └── profile/
│ └── Default/

I inspected the stored cookies, session tokens, cached scripts, and extensions looking for some kind of tamper detection artifact. Nothing useful. The refreshing wasn’t coming from stored state.

Screenshot: browser/ directory structure

Phase 5: SmartSocket.js — Where It All Connected

The actual cause was in the exam webpage itself. Buried in the page source was this:

<script src="SmartLock/SmartSocket.js"></script>

This script establishes a WebSocket connection to the local application:

socket = new WebSocket('ws://localhost:8000/');

This is the critical link. The browser doesn’t just display the exam — it actively depends on the local application being alive and reachable. As soon as the connection is established, the browser authenticates:

reqOb.type = "Authentication";
reqOb.token = '1234';
reqOb.userid = appl_no;
socket.send(JSON.stringify(reqOb));

And if the connection fails — even momentarily — the page clears the session and reloads:

// On connection failure:
// → Clear session
// → Redirect or reload
// → UI resets to initial state

This is not tamper detection. It’s a strict runtime dependency. The browser requires the local WebSocket server to be up before it finishes loading. If it isn’t, you get an infinite reload loop.

Screenshot: SmartSocket.js connection code in browser devtools

Phase 6: The Race Condition

With that understanding, the root cause became obvious. The application starts the WebSocket server and the browser in parallel:

thread1 = start_websocket()
thread2 = launch_browser()

The problem with this is that “starting” the WebSocket server takes a moment. The browser, however, is fast — it loads the page, runs the script, and tries to connect to localhost:8000 before the server is actually ready. Connection fails. Page reloads. Tries again. Same thing. Infinite loop.

The sequence of events looked like this:

Browser loads page
 → SmartSocket.js executes immediately
 → Attempts WebSocket connection to localhost:8000
 → Server not ready yet
 → Connection refused
 → Page session cleared
 → Page reloads
 → Same thing happens again
 → ...forever

It perfectly mimicked tamper detection behavior, which is why I assumed that’s what it was. But it was just a timing issue.

The fix is simple: wait for the server to be ready before launching the browser.

def browser_start(self):
    try:
        import socket as sock
        import time

        for _ in range(50):
            try:
                s = sock.create_connection(("localhost", 8000), timeout=0.1)
                s.close()
                break
            except OSError:
                time.sleep(0.1)

        # Now launch browser

With this in place, the correct sequence is:

Start App
 → Start WebSocket server
 → Poll until server is accepting connections
 → Launch browser
 → WebSocket connects successfully
 → Authentication succeeds
 → Exam proceeds normally

Screenshot: Before — infinite reload loop vs successful startup

Phase 7: What the App Is Actually Doing Under the Hood

Once the startup problem was solved, I could look more carefully at all the enforcement mechanisms running in the background. There’s quite a lot.

OS-Level Lockdown

The registry editor modifies Windows to disable the usual escape routes:

DisableTaskMgr = 1
DisableLockWorkstation = 1
NoLogoff = 1

This disables Task Manager, the lock screen, and the ability to log off. The Ctrl+Alt+Del menu effectively becomes useless.

Process Monitoring

The monitoring engine maintains a list of software that shouldn’t be running during an exam. Screen recording and virtual camera tools are specifically targeted:

OBS Studio
ManyCam
XSplit

If any of these processes are detected, a violation is flagged.

USB Monitoring

The app takes a snapshot of connected USB devices at startup and watches for changes:

if current_usb != initial_usb:
    flag_violation()

Plugging in a USB drive during the exam is treated as a potential integrity violation.

Multi-Monitor and VM Detection

Multiple displays are blocked. The app also checks whether it’s running inside a virtual machine — which would make it easier to manipulate the environment without being detected.

Network Filtering via pydivert

This was the most interesting piece. The app uses pydivert - a Python wrapper around WinDivert - to implement packet-level network filtering. During an exam, only certain destinations are allowed. Everything else is dropped at the kernel level.

Phase 8: Two WebSockets, Not One

I initially assumed there was a single WebSocket connection: browser to local app. There are actually two:

Local WebSocket (localhost:8000) Browser ↔ Desktop App

Remote WebSocket Desktop App ↔ Remote Server

The local one handles session management for the browser. The remote one is for continuous telemetry — the app regularly sends status updates to the server:

{ "type": "USB", "status": true }
{ "type": "ProcessCheck", "detected": false }

The server isn’t passive. It’s continuously validating that the client is behaving correctly. If the telemetry stops or reports a violation, the server can terminate the session.

Configuration

All the connection details live in a YAML config file:

ExamIp: 164.100.69.5
ExamUrl: https://sarathi.parivahan.gov.in/sarathiservice/authenticationaction.do?authtype=Anugyna
SocketPort: 8000
SocketServerPort: 3000
SocketUrl: ws://sarathi.parivahan.gov.in
StatusApiUrl: https://sarathi.parivahan.gov.in/sarathiWS/rsServices/smartLockCheck/smartLockCheck
ViolationApiUrl: https://sarathicov.nic.in:8443/sarathiWS/rsServices/smartLockCheck/examViolation
primaryServerIPV4: 10.172.31.33
primaryServerIPV6: 2001:4408:7204:8:5d93:8239:8876:d238
secondaryServerIPV4: 10.172.31.30
secondaryServerIPV6: 2001:4408:7204:9::aac:2033

This means the behavior is somewhat server-controlled. The exam URL, the socket address, the session logic — it’s all configured externally, which makes the server the real authority over how the session runs.

Phase 9: The Firewall

One more thing worth mentioning: the app appears to interact with the Windows Firewall directly.

fw.pbox_fw_backup("pbox_bkp.wfw")

This exports the current firewall rules to a backup file before modifying them. The behavior is that the app replaces your firewall rules with its own restricted ruleset for the duration of the exam, then restores the backup afterward. It may also hash the backup to detect if someone has tampered with the rules mid-session.

Phase 10: Why You Can’t Just Rebuild It

One last I want to address directly: extracting the PyInstaller binary does not give you a working copy of the application. There’s a common misconception that extraction = reconstruction. It doesn’t.

The workflow for actually rebuilding would be:

Decompile .pyc files to .py
Manually correct the decompilation errors
Rebuild with PyInstaller

Steps 1 and 2 are where it falls apart in practice. The decompiled code has inaccuracies that aren’t always obvious. Some of the control flow is wrong in subtle ways that only become apparent at runtime. There are also timing dependencies baked into the threading model, and the server-side validation means you’d need a cooperating server to test anything properly.

The security here doesn’t come from any single mechanism being unbreakable. It comes from the combination:

The local app monitors the OS
The browser depends on the local app
The server monitors the local app
The network is filtered at the kernel level
The firewall is replaced during the session

Each layer on its own is probably defeatable. Together, they create a system where defeating one layer doesn’t help much because the others remain intact.

What I Took Away From This

A few things stuck with me after this investigation.

The infinite reload loop was genuinely convincing as tamper detection. I spent more time than I’d like to admit looking for a security mechanism that wasn’t there. The lesson is that emergent behavior from a race condition can look exactly like intentional defensive behavior. Don’t assume intent before you’ve traced the actual execution path.

The browser is doing real security work here, not just displaying a UI. SmartSocket.js is the link between the exam session and the local enforcement system. If that connection breaks, the exam can't proceed. That's a deliberate architectural choice, not an accident.

And PyInstaller extraction, while possible, is just the beginning. The hard part isn’t getting the bytecode out. It’s making sense of what the decompiler gives you and reconstructing what the developer originally meant to write.

What Could Come Next

If I were going to continue this investigation, the natural directions would be:

Mapping the full WebSocket protocol between browser and app, and between app and server
Tracing the telemetry payloads to understand exactly what data gets sent and when
Building a sequence diagram for the full session lifecycle, from startup through exam completion
Looking more carefully at the firewall manipulation and how (or whether) it detects tampering with the backed-up rules

None of that fits in one article. But the architecture is now clear enough that any of those threads could be pulled independently.

Screenshot: Final — application running normally with exam loaded, WebSocket connection established

This article is for educational purposes — understanding how production security systems are architected and why they’re difficult to tamper with. The focus throughout has been on the design and behavior of the system, not on defeating it.

This article is rewritten using AI chatbots.

April 07, 2026