Why I Decided to Start Doing Research
So I decided I will start doing research, and in this blog series I will try to share my research journey in real time. Before jumping into what I am doing, I decided to share my story of what led me to go down this boring path. Well, I’ll eventually talk about it, but even if it seems boring, it is not really.
When I graduated, I wanted to get into a PhD. I applied to about 20 universities and didn’t get a call from a single one. At that time, I thought my background was great. I knew programming, worked in real industries, and published one paper, so I should technically be able to get in, right? Well, I was dead wrong.
It was such a humbling realization of how surface-level my knowledge of computer science actually was. I started realizing that many of the applications most of us programmers are building today, and even many of the ideas we are generating, were already researched 10 years ago. So I decided I would have to specialize in something.
The thought of getting cutting-edge knowledge about something that very few people in the world possess really excited me. I decided to pursue my Master’s in Cybersecurity, and I came to Canada. Well, I did not enjoy cybersecurity as much as I thought I would. Mostly because I had this wrong idea that it was mostly technical, and I am a technical person. I thought it would involve breaching complex codebases and doing highly technical work, while the reality is that many jobs in the field involve management, compliance, and other areas that I wasn’t as interested in.
At the same time, AI started getting more and more powerful and the programming job market started becoming increasingly difficult. For me, it was something like 100 job applications and 0 interview calls. Well, I knew I had to "git gud". Being a basic programmer is not going to cut it anymore.
What the world increasingly seems to be looking for is specialization. BUT YOU DON’T GET SPECIALIZED KNOWLEDGE FROM YOUTUBE TUTORIALS!!
So I started reading research papers, and honestly, it was really fun to mine knowledge from other people’s work. It’s kind of like those Sherlock Holmes movies where detectives go into libraries, dig out old information, and connect things together. After some digging, I found out that I am actually interested in Web Information Retrieval and Extraction research.
Primarily because I was building a SaaS, and whenever I come up with some kind of startup idea, it almost always revolves around collecting data from the web and using that data to create value for people. Then this thing crossed my mind: since AI is becoming so powerful, I should be able to do some crazy things with scraped data. Hence, I started my independent research journey.
I have been working on it for about a week now. I dug up some old foundational research papers to learn how this domain actually started. The thing that amazed me was that ideas I was thinking about myself, like creating some kind of central ruleset that would allow scrapers to automatically retrieve data from websites, were already invented in the 90s. They called them wrappers.
This is another great example of how Google Scholar is living in 2040 while we are still in 2026.
So I decided to read some papers from 2026 and gather an understanding of the current research trends and the direction the field is progressing toward. Well, I kind of cheated. I mostly listed out the research directions that current authors themselves have already mentioned. I do not yet have enough technical depth to analyze everything myself honestly. I know, I know. Criticism in the comments is most welcome.
Based on my research, I found the following research trends. This is completely my personal opinion and I would not recommend taking it as fact.
- Designing systems that can work around common anti-bot protections and restrictions.
- Framework design incorporating LLMs.
- How to use small language models to achieve similar levels of efficiency.
- How to actually scale systems that use LLM models.
- Creating datasets that can help LLMs learn web extraction and scraping tasks better.
Some future research directions that seem interesting are:
- How to test the faithfulness of AI systems and make sure they are not hallucinating information.
- How to defend systems against autonomous LLM agents.
- Domain-specialized LLM agents.
- Generation of scraping and extraction logic on the fly in the most cost-efficient manner possible.
Again, I am just getting started, so this is not supposed to be some authoritative analysis of the field. This is more of a snapshot of what I have understood so far.
The goal of this blog series is to document the journey in real time. I will share what I learn, what papers I read, what ideas I find interesting, and hopefully, at some point, what I actually manage to build or contribute.
Let’s see where this rabbit hole goes.
Top comments (0)