The modern web is honestly astonishing. There are layers upon layers of security mechanisms working behind the scenes, and some of these systems are so sophisticated that even web developers might never notice them, let alone regular users. For those who are new here, this is a series of blogs where I document my research journey as an independent researcher, starting out in this field and figuring things out in real time. I started my research journey with web information retrieval, and the context behind that can be found in my previous post: https://dev.to/abtahitajwar/my-research-journal-why-i-decided-to-start-doing-research-what-i-learned-4a1g
Initially, I started building a scraper. My goal was to establish a foundation for scraping websites using techniques similar to legacy scrapers, with the help of wrappers. Eventually, I wanted to work my way up, bit by bit, toward building something more autonomous and agentic. My thinking was simple: if I don’t understand the implementation firsthand, reading hundreds of papers probably isn’t going to help me much. So I wanted to get my hands dirty first. Then I noticed something I wasn’t really expecting: my scraper was getting blocked. A lot. In one particular case, I found that whenever I switched to a headless browser, I would immediately get flagged. Naturally, I started digging.
I went through several recent papers on web bot detection and anti-bot mechanisms, trying to understand what was actually happening behind the scenes. Boy, I wasn’t ready for this. I initially thought bot detection would be something relatively straightforward, like checking the User-Agent, looking at request frequency, or maybe detecting Selenium. Nope. Some systems inspect an absolutely ridiculous number of browser properties. They can look at things like your browser environment, WebGL, Canvas, fonts, permissions, audio APIs, and even things that made me stop and ask, “Why the hell are they checking that?” Some detectors even probe things like the battery API. Battery percentage. And the crazy part isn’t any individual check. It’s that all of these tiny signals can be combined to build a picture of whether the browser behind the request looks like a real person or an automated system.
The more I dug into it, the more I realized that there is an entire security ecosystem operating underneath the websites we casually browse every day. So I decided this could actually be a really interesting direction for my research. It also happens to sit right at the intersection of two areas I already have a background in: cybersecurity and software engineering. But before I start trying to come up with some grand research idea, I want to understand how these systems actually work. My next step is probably going to be studying an existing open-source implementation, perhaps a JavaScript bot-detection package, and getting my hands dirty with the actual code. Then, hopefully, after I understand the existing approaches well enough, I’ll have the knowledge to start brainstorming something meaningful that I can contribute to this area.
One rabbit hole at a time.
Top comments (0)