DEV Community: abtahi-tajwar

My Research Journal: I Thought Web Scraping Was the Hard Part

abtahi-tajwar — Thu, 10 Sep 2026 00:03:51 +0000

The modern web is honestly astonishing. There are layers upon layers of security mechanisms working behind the scenes, and some of these systems are so sophisticated that even web developers might never notice them, let alone regular users. For those who are new here, this is a series of blogs where I document my research journey as an independent researcher, starting out in this field and figuring things out in real time. I started my research journey with web information retrieval, and the context behind that can be found in my previous post: https://dev.to/abtahitajwar/my-research-journal-why-i-decided-to-start-doing-research-what-i-learned-4a1g

Initially, I started building a scraper. My goal was to establish a foundation for scraping websites using techniques similar to legacy scrapers, with the help of wrappers. Eventually, I wanted to work my way up, bit by bit, toward building something more autonomous and agentic. My thinking was simple: if I don’t understand the implementation firsthand, reading hundreds of papers probably isn’t going to help me much. So I wanted to get my hands dirty first. Then I noticed something I wasn’t really expecting: my scraper was getting blocked. A lot. In one particular case, I found that whenever I switched to a headless browser, I would immediately get flagged. Naturally, I started digging.

I went through several recent papers on web bot detection and anti-bot mechanisms, trying to understand what was actually happening behind the scenes. Boy, I wasn’t ready for this. I initially thought bot detection would be something relatively straightforward, like checking the User-Agent, looking at request frequency, or maybe detecting Selenium. Nope. Some systems inspect an absolutely ridiculous number of browser properties. They can look at things like your browser environment, WebGL, Canvas, fonts, permissions, audio APIs, and even things that made me stop and ask, “Why the hell are they checking that?” Some detectors even probe things like the battery API. Battery percentage. And the crazy part isn’t any individual check. It’s that all of these tiny signals can be combined to build a picture of whether the browser behind the request looks like a real person or an automated system.

The more I dug into it, the more I realized that there is an entire security ecosystem operating underneath the websites we casually browse every day. So I decided this could actually be a really interesting direction for my research. It also happens to sit right at the intersection of two areas I already have a background in: cybersecurity and software engineering. But before I start trying to come up with some grand research idea, I want to understand how these systems actually work. My next step is probably going to be studying an existing open-source implementation, perhaps a JavaScript bot-detection package, and getting my hands dirty with the actual code. Then, hopefully, after I understand the existing approaches well enough, I’ll have the knowledge to start brainstorming something meaningful that I can contribute to this area.

One rabbit hole at a time.

My Research Journal: Why I Decided to Start Doing Research & What I Learned

abtahi-tajwar — Wed, 02 Sep 2026 23:42:07 +0000

Why I Decided to Start Doing Research

So I decided I will start doing research, and in this blog series I will try to share my research journey in real time. Before jumping into what I am doing, I decided to share my story of what led me to go down this boring path. Well, I’ll eventually talk about it, but even if it seems boring, it is not really.

When I graduated, I wanted to get into a PhD. I applied to about 20 universities and didn’t get a call from a single one. At that time, I thought my background was great. I knew programming, worked in real industries, and published one paper, so I should technically be able to get in, right? Well, I was dead wrong.

It was such a humbling realization of how surface-level my knowledge of computer science actually was. I started realizing that many of the applications most of us programmers are building today, and even many of the ideas we are generating, were already researched 10 years ago. So I decided I would have to specialize in something.

The thought of getting cutting-edge knowledge about something that very few people in the world possess really excited me. I decided to pursue my Master’s in Cybersecurity, and I came to Canada. Well, I did not enjoy cybersecurity as much as I thought I would. Mostly because I had this wrong idea that it was mostly technical, and I am a technical person. I thought it would involve breaching complex codebases and doing highly technical work, while the reality is that many jobs in the field involve management, compliance, and other areas that I wasn’t as interested in.

At the same time, AI started getting more and more powerful and the programming job market started becoming increasingly difficult. For me, it was something like 100 job applications and 0 interview calls. Well, I knew I had to "git gud". Being a basic programmer is not going to cut it anymore.

What the world increasingly seems to be looking for is specialization. BUT YOU DON’T GET SPECIALIZED KNOWLEDGE FROM YOUTUBE TUTORIALS!!

So I started reading research papers, and honestly, it was really fun to mine knowledge from other people’s work. It’s kind of like those Sherlock Holmes movies where detectives go into libraries, dig out old information, and connect things together. After some digging, I found out that I am actually interested in Web Information Retrieval and Extraction research.

Primarily because I was building a SaaS, and whenever I come up with some kind of startup idea, it almost always revolves around collecting data from the web and using that data to create value for people. Then this thing crossed my mind: since AI is becoming so powerful, I should be able to do some crazy things with scraped data. Hence, I started my independent research journey.

I have been working on it for about a week now. I dug up some old foundational research papers to learn how this domain actually started. The thing that amazed me was that ideas I was thinking about myself, like creating some kind of central ruleset that would allow scrapers to automatically retrieve data from websites, were already invented in the 90s. They called them wrappers.

This is another great example of how Google Scholar is living in 2040 while we are still in 2026.

So I decided to read some papers from 2026 and gather an understanding of the current research trends and the direction the field is progressing toward. Well, I kind of cheated. I mostly listed out the research directions that current authors themselves have already mentioned. I do not yet have enough technical depth to analyze everything myself honestly. I know, I know. Criticism in the comments is most welcome.

Based on my research, I found the following research trends. This is completely my personal opinion and I would not recommend taking it as fact.

Designing systems that can work around common anti-bot protections and restrictions.
Framework design incorporating LLMs.
How to use small language models to achieve similar levels of efficiency.
How to actually scale systems that use LLM models.
Creating datasets that can help LLMs learn web extraction and scraping tasks better.

Some future research directions that seem interesting are:

How to test the faithfulness of AI systems and make sure they are not hallucinating information.
How to defend systems against autonomous LLM agents.
Domain-specialized LLM agents.
Generation of scraping and extraction logic on the fly in the most cost-efficient manner possible.

Again, I am just getting started, so this is not supposed to be some authoritative analysis of the field. This is more of a snapshot of what I have understood so far.

The goal of this blog series is to document the journey in real time. I will share what I learn, what papers I read, what ideas I find interesting, and hopefully, at some point, what I actually manage to build or contribute.

Let’s see where this rabbit hole goes.

How I Accidentally Came Up With a Reactive JS UI Library

abtahi-tajwar — Sun, 15 Jun 2025 01:03:34 +0000

Okay okay — I get it. Please don’t roll your eyes 🙏

Yes, I built yet another JavaScript UI library. I know it seems like every day there’s a new one popping up — Svelte, Solid, Alpine, HTMX… and now, VaneJS.

But hear me out — this wasn’t the plan.

🚧 The Accidental Start

It all began with a basic personal project. I was building my portfolio site using a raw HTML/CSS template. My goal was simple: load content from a JSON file dynamically — no fuss.

Naturally, I thought of using Svelte. It has that clean, HTML-like syntax and is pretty powerful. But it didn’t feel worth the migration headache just for a small site. So I fell back to vanilla JS.

Now here’s the problem: I kept spamming document.querySelector, innerHTML, and manual state updates just to reflect data. Tedious. So I thought:

"Why not just write a simple templating engine?"

I just wanted to dump data into the HTML cleanly. But... it spiraled. 😅

Before I knew it, I had:

Reactive state management
Loops and conditional rendering
Store syncing between pages
Event bindings with inline params
And yes, even an API call example

Then I thought: "Okay, this is officially something."

I picked a name — VaneJS — in like 10 seconds.

👉 Check the documentation

✨ Very Basic Usage

Here’s the hello world of VaneJS. No CLI, no build step, no npm — just plug and play.

<!-- index.html -->
<script type="module" src="vanjejs.module.js"></script>

<h1 data-vn-bind="user.name">Hello</h1>

<script type="module">
  import { $setState } from './vanjejs.module.js';

  $setState("user", { name: "Abtahi" });
</script>

Want to add loops?

<ul data-vn-repeat="user.skills as skill">
  <li data-vn-ritem="{skill}.label"></li>
</ul>

Update with:

$setState("user", {
  name: "Abtahi",
  skills: [{ label: "JS" }, { label: "C++" }]
});

And conditionally render blocks:

<div data-vn-if="{theme} === 'dark'">Dark Mode</div>
<div data-vn-else>Light Mode</div>

🧠 What Makes VaneJS Different?

✅ Zero Build Step: Just add a single static .js file to your project. No bundlers, no compilers.
✅ Backend Friendly: Works out-of-the-box with PHP, Laravel, Django, Express, ASP.NET — whatever you want.
✅ No Paradigm Shift: Feels like HTML, behaves like a reactive system.
✅ Minimal Mental Overhead: You don’t need to “learn a framework.” If you know HTML and JS, you’re good.
✅ Reactive Stores Between Pages: Yes, you can store values and retrieve them across HTML pages.

<!-- about.html -->
<p data-vn-bind="mystore.name"></p>

// main.js
$setStore("mystore", { name: "SharedValue" });

✅ Smart Inline Events: Attach multiple events inline:

<button data-vn-on="click:myFunc({user.name}); mouseover:logHover()">Click me</button>

$event("myFunc", ({ params }) => console.log(params[0]));
$event("logHover", () => console.log("hovered!"));

🚀 What’s Next

I’m actively building more around the zero-config, zero-setup philosophy:

🔨 Templating System: You’ll be able to define reusable templates without any build tools. Like:

export default () => \`
  <h1>Hello</h1>
  <p data-vn-bind="user.bio"></p>
\`

🧠 Language Server (LSP): I’m planning to write an LSP for VaneJS so your editor can offer intellisense, diagnostics, and autocompletion just like in React or Svelte.
🌱 Plugin System (planned): Think mini-addons for things like forms, routers, or animation — optional and simple.

👋 Final Thoughts

I didn’t set out to build a JavaScript UI library. I was just trying to get my site done faster. But sometimes, the shortcuts become the road.

If you’re a fullstack dev, or even a backend-first developer who hates writing repetitive DOM updates — try VaneJS. It might just save you a few headaches.

🔗 Docs: vanejs.netlify.app

📦 Demo / GitHub: https://github.com/abtahi-tajwar/vanejs

Type-Safe Theming in Tailwind CSS Using CSS Variables and TypeScript

abtahi-tajwar — Thu, 08 May 2025 03:16:16 +0000

Modern frontend development demands scalable theming, responsive design tokens, and runtime flexibility. With the recent enhancements in Tailwind CSS v3.4+, namely @theme and @custom-variant, developers can now implement fully dynamic, type-safe theming using CSS variables, TypeScript, and zero runtime JS logic. This article outlines a theming strategy I came up with, which I believe is a clean and scalable approach built around those tools. It enables easy support for light/dark modes and beyond (daylight, retro, etc.) with minimal complexity.

Why I Think `tailwind.config.js` Isn't Enough for Theming

Traditionally, developers define colors like primary, accent, or background directly in tailwind.config.js:

module.exports = {
  theme: {
    colors: {
      primary: '#571089',
      accent: '#ea698b',
    }
  }
}

But in my experience, this introduces several limitations:

No runtime access from JavaScript/TypeScript
- If you want to use the same primary color in a charting library, animation logic, or conditional rendering, you must hardcode or duplicate it.
No direct support for runtime theming
- You can't change colors dynamically (e.g., toggle between light/dark) without rebuilding Tailwind or using custom hacks.
Theme values scattered across multiple places
- You may have tailwind.config.js, SASS/SCSS maps, and JS constants — causing desynchronization and maintenance headaches.

My Solution: Centralized Theme + CSS Variable Generation

I addressed these issues by creating a single, typed theme.ts file like this:

export const theme = {
  mode: 'light',
  light: {
    color: {
      primary: '#571089',
      primary_light: ['#6411ad', '#6d23b6'],
      primary_deep: ['#47126b'],
      background: '#ffffff',
    },
    font: {
      primary: 'ui-sans-serif, system-ui, sans-serif',
    }
  },
  dark: {
    color: {
      primary: '#571089',
      primary_light: ['#6411ad', '#6d23b6'],
      primary_deep: ['#47126b'],
      background: '#0a0f1c',
    },
    font: {
      primary: 'ui-sans-serif, system-ui, sans-serif',
    }
  }
} as const;

This provides TypeScript type safety, autocompletion, and full control over the structure.

Then, using a simple build script with esbuild, I extract this object and generate a tailwind.css file with native Tailwind CSS support:

// theme-generator.ts
import { buildSync } from 'esbuild';
import fs from 'fs';
import path from 'path';

const result = buildSync({
  entryPoints: ['src/config/theme/theme.ts'],
  bundle: true,
  platform: 'node',
  write: false,
  format: 'cjs',
});

const compiled = result.outputFiles[0].text;
const module = { exports: {} };
const func = new Function('module', 'exports', compiled);
func(module, module.exports);
const theme = module.exports.default || module.exports.theme || module.exports;

if (!theme || !theme.light || !theme.dark) {
  console.error('Theme format invalid or missing colors');
  process.exit(1);
}

let css = '/* Do not edit this file, this will be automatically replaced by system */\n@import "tailwindcss";\n@custom-variant dark (&:where(.dark, .dark *));\n\n@theme {\n';

const flatten = (prefix, value, isDark = false) => {
  const convertedPrefix = prefix.split("_").join("-");
  if (Array.isArray(value)) {
    value.forEach((v, i) => {
      css += `    --${convertedPrefix}-${i}${isDark ? '-dark' : ''}: ${v};\n`;
    });
  } else if (typeof value === 'object') {
    Object.entries(value).forEach(([k, v]) => {
      flatten(`${convertedPrefix}-${k}`, v, isDark);
    });
  } else {
    css += `    --${convertedPrefix}${isDark ? '-dark' : ''}: ${value};\n`;
  }
};

Object.entries(theme.light).forEach(([key, value]) => flatten(key, value));
Object.entries(theme.dark).forEach(([key, value]) => flatten(key, value, true));

css += '}';

fs.mkdirSync(path.join('src'), { recursive: true });
fs.writeFileSync(path.join('src', 'tailwind.css'), css);
console.log('✅ Tailwind theme generated at: src/tailwind.css');

Generated tailwind.css file

/* Do not edit this file, this will be automatically replaced by system */
@import "tailwindcss";
@custom-variant dark (&:where(.dark, .dark *));

@theme {
    --color-primary: #571089;
    --color-primary-light-0: #6411ad;
    --color-primary-light-1: #6d23b6;
    --color-primary-light-2: #822faf;
    --color-primary-light-3: #973aa8;
    --color-primary-light-4: #ac46a1;
    --color-primary-deep-0: #47126b;
    --color-accent: #ea698b;
    --color-accent-deep-0: #d55d92;
    --color-accent-deep-1: #c05299;
    --color-accent-deep-2: #c05299;
    --color-background: #ffffff;
    --font-primary: ui-sans-serif, system-ui, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol", "Noto Color Emoji";
    --color-primary-dark: #571089;
    --color-primary-light-0-dark: #6411ad;
    --color-primary-light-1-dark: #6d23b6;
    --color-primary-light-2-dark: #822faf;
    --color-primary-light-3-dark: #973aa8;
    --color-primary-light-4-dark: #ac46a1;
    --color-primary-deep-0-dark: #47126b;
    --color-accent-dark: #ea698b;
    --color-accent-deep-0-dark: #d55d92;
    --color-accent-deep-1-dark: #c05299;
    --color-accent-deep-2-dark: #c05299;
    --color-background-dark: #0a0f1c;
    --font-primary-dark: ui-sans-serif, system-ui, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol", "Noto Color Emoji";
}

This file is auto-generated and imported into the app.

Tailwind + Runtime CSS Variables

With this setup, usage becomes extremely clean:

<div class="bg-primary dark:bg-primary-deep text-accent dark:text-accent-deep">
  Hello themed world!
</div>

No need to hardcode Tailwind color names or rebuild config — everything is centralized and flexible.

Easy Dark Mode and Beyond

Thanks to @custom-variant dark, this structure supports .dark selectors out of the box. But more importantly, it’s easy to add other modes:

Add daylight, retro, solarized, or any custom modes to theme.ts
Extend the script to generate --color-*--retro variables
Use class="retro" or similar to toggle themes

Dark/light modes are simple and future-friendly.

Why Keeping Theme Centralized Matters

I think it’s extremely important to define all theme values in one place — not scattered across Tailwind config, SCSS files, and JS constants. This setup ensures:

Type safety in app logic
No duplication or drift
Theme switching at runtime with zero JS updates
Full visibility into how your design system behaves

In Summary

I avoid embedding colors directly in tailwind.config.js
I define a typed theme.ts file and generate CSS variables from it
I use Tailwind’s native @theme and @custom-variant support
This allows scaling to many theme modes (dark, daylight, retro, etc.) effortlessly

✅ Bonus: Where This Helps

Dashboard-driven theme customization
Runtime A/B testing of styles
Building Figma-style token systems with Tailwind

I’ve found this setup modern, scalable, and type-safe. If you're using Tailwind 3.4+, I highly recommend trying it.

Let me know if you want a GitHub repo or CLI template for this approach!