Research Paper AI: Building an Academic Search Engine with Bright Data

Anudeep Adiraju — Tue, 17 Dec 2024 05:26:58 +0000

This is a submission for the Bright Data Web Scraping Challenge: Most Creative Use of Web Data for AI Models

What I Built

As someone deeply interested in academic research, I often found myself juggling between different websites to find relevant papers and research work. I wanted a tool that could simplify this process, so I built Research Paper AI.

Research Paper AI is a streamlined academic search engine that scrapes papers from various sources like arXiv and Google Scholar in real-time. What makes it special is how it uses Bright Data's infrastructure to reliably access these academic sources, which are typically challenging to scrape due to their anti-bot measures and complex structures.

The magic happens when you type in a search query - the app simultaneously searches across multiple academic sources, handling all the complexities of web scraping behind the scenes, and presents you with clean, unified results. Think of it as your personal research assistant that knows how to navigate the academic web.

Here's what it looks like in action:

Main search interface
Search results with papers

How I Used Bright Data

This is where things get interesting! Academic websites are notoriously tricky to scrape - they have CAPTCHAs, rate limits, and sometimes require complex JavaScript rendering. Bright Data's tools made these challenges much more manageable.

Here's how I leveraged Bright Data:

Scraping Browser Integration: I used Bright Data's Scraping Browser to handle JavaScript-heavy pages and bypass anti-bot measures. Here's a snippet of how it works:

class BrightScraper:
    def __init__(self, config: Dict):
        self.username = config['username']
        self.password = config['password']
        self.proxy_url = f"http://{self.username}:{self.password}@{self.host}"

    async def scrape_arxiv(self, query: str):
        async with self.session.get(search_url, proxy=self.proxy_url) as response:
            # Bright Data handles all the complex stuff behind the scenes
            html = await response.text()
            return self._parse_results(html)

Proxy Management: Instead of dealing with proxy rotation and management myself, Bright Data's infrastructure handles it automatically. This means:
- No more blocked requests
- Reliable access to academic sources
- Clean, consistent data collection

The real breakthrough came when dealing with Google Scholar, which is typically very difficult to scrape. Bright Data's tools made it feel almost trivial!

Tech Stack

Frontend: React + TailwindCSS (because life's too short for bad UIs)
Backend: FastAPI (because async is awesome)
Scraping: Bright Data's Scraping Browser
Data Sources: arXiv, Google Scholar

Interesting Challenges

The CAPTCHA Conundrum: Academic sites love their CAPTCHAs. Bright Data's Scraping Browser handled these seamlessly.
Rate Limiting: Initially, I was getting blocked after a few requests. Switching to Bright Data's proxy network solved this instantly.
JavaScript Rendering: Some sites needed full JavaScript execution to load content. The Scraping Browser handled this without breaking a sweat.

Future Plans

I'm excited to add more features:

More academic sources (IEEE, Semantic Scholar)
Citation network visualization
Paper similarity analysis
PDF content extraction

Want to Try It Yourself?

Check out the GitHub repo and give it a spin! The setup is pretty straightforward:

Clone the repo
Add your Bright Data credentials
Run the backend and frontend
Start discovering papers!

Conclusion

Building Research Paper AI was a fantastic learning experience. Bright Data's tools turned what could have been a complex scraping nightmare into a manageable and fun project. The best part? It actually solves a real problem that researchers face daily.

brightdatachallenge #webdev #api #ai

FinWise: Your AI-Powered Financial Advisor 🌟

Anudeep Adiraju — Wed, 01 May 2024 18:26:00 +0000

This is a submission for the Coze AI Bot Challenge: Bot Innovator.

What I Built

FinWise is an AI-powered bot designed to revolutionize personal finance management. It offers personalized financial advice and guidance, helping users create budgets, set financial goals, and make informed decisions about investments, savings, and debt management. FinWise analyzes a user's unique financial situation and provides actionable recommendations tailored to their specific needs.

Demo

Try out FinWise in the Coze Bot Store: FinWise Bot

Your Configuration

Here's a look at FinWise's configuration, including prompts and skills used:

Journey

Building FinWise was an incredible learning experience. I started by researching the most common financial challenges people face and how AI could help solve them. I then broke down the bot's functionality into key features like budgeting, goal setting, debt management, and investment advice.

One of the biggest challenges was creating a user-friendly interface that could gather the necessary financial information without overwhelming the user. I spent a lot of time refining the prompts and conversation flow to make it as intuitive as possible.

I'm particularly proud of FinWise's ability to provide personalized recommendations based on each user's unique financial situation. By leveraging AI, the bot can analyze a user's data and offer tailored advice that would typically require a human financial advisor.

Moving forward, I hope to expand FinWise's capabilities to include features like real-time market updates, retirement planning, and integration with popular budgeting apps. I also plan to gather user feedback to continuously improve the bot's functionality and user experience.

Overall, participating in the Coze AI Bot Challenge has been a fantastic opportunity to push the boundaries of what's possible with AI and create a bot that has the potential to make a real difference in people's financial lives.

DEV Community: Anudeep Adiraju

Research Paper AI: Building an Academic Search Engine with Bright Data

What I Built

How I Used Bright Data

Tech Stack

Interesting Challenges

Future Plans

Want to Try It Yourself?

Conclusion

brightdatachallenge #webdev #api #ai

FinWise: Your AI-Powered Financial Advisor 🌟

What I Built

Demo

Your Configuration

Journey