DEV Community: Aashish Karki

Building the Academic Research Copilot: From ArXiv to Semantic Search in Minutes

Aashish Karki — Tue, 28 Oct 2025 17:47:09 +0000

Finding the right paper shouldn’t feel like searching for a needle in a haystack. Keyword search misses context, titles can be misleading, and abstracts use different vocabulary for the same idea. The Academic Research Copilot solves this with hybrid semantic search over ArXiv—combining vector embeddings with simple filters—so you can ask real questions and get relevant papers fast.

This post walks through the problem, the architecture, it was built, and how you can run or extend it yourself.

TL;DR

Ingest ArXiv metadata → DuckDB
Create a MindsDB Knowledge Base → generate embeddings for title + summary
Query with natural language → get semantically similar papers
Serve via FastAPI and a Streamlit UI
Run locally with Docker, Gemini embeddings

Repo: https://github.com/aashish079/academic-research-copilot

Repo folders to peek at:

src/data/fetch_papers.py – ArXiv → DuckDB
src/knowledge_base/kb_manager.py – KB creation + ingestion
src/knowledge_base/queries.py – semantic/hybrid queries (with a DuckDB fallback)
src/api/routes.py, src/app.py – FastAPI endpoints
src/ui/streamlit_app.py – Streamlit UI

The use case: question-first research

Researchers often start with a question, not a keyword. For example:

“privacy in federated learning”
“diffusion models for medical imaging”
“efficient attention variants in transformers”

Traditional search requires exactly the right words. Semantic search uses embeddings to find conceptually similar content, not just textual matches. Hybrid search then refines results with lightweight metadata filters (authors, year, categories) when needed.

Outcome: faster discovery, better recall, less time sifting PDFs.

Architecture

At a high level:

1) Fetch papers from ArXiv and store them in a local DuckDB papers table.
2) Register that DuckDB file inside MindsDB.
3) Create a Knowledge Base (KB) configured to embed title + summary.
4) Populate the KB using INSERT … SELECT (MindsDB generates embeddings automatically).
5) Expose clean HTTP APIs via FastAPI; Streamlit calls the API and renders results.

Data ingestion: ArXiv → DuckDB

We use the arxiv Python package to fetch results by topic and store them into DuckDB. Each paper is normalized to a consistent schema.

Key fields:

entry_id (primary key)
title
summary (abstract)
authors (comma-separated)
published_date
pdf_url
categories

Snippet (from src/data/fetch_papers.py):

papers = []
for result in search.results():
    papers.append({
        "entry_id": result.entry_id,
        "title": result.title,
        "summary": result.summary.replace("\n", " "),
        "authors": ", ".join([a.name for a in result.authors]),
        "published_date": result.published.date(),
        "pdf_url": result.pdf_url,
        "categories": ", ".join(result.categories),
    })

The script creates the DuckDB table if needed and upserts rows to avoid duplicates.

Knowledge Base: embeddings with MindsDB

We connect the DuckDB file inside MindsDB and create a KB named academic_kb whose embeddings are built over title and summary. In this project, we use Google Gemini text-embedding-004 by setting GEMINI_API_KEY.

Create KB (SQL idea shown in README):

CREATE KNOWLEDGE_BASE academic_kb
USING
  embedding_model = {
    "provider": "google",
    "model_name": "text-embedding-004",
    "api_key": "${GEMINI_API_KEY}"
  },
  content_columns = ['title','summary'],
  id_column = 'entry_id';

Populate the KB from DuckDB:

INSERT INTO academic_kb
SELECT entry_id, title, summary, authors, published_date, pdf_url, categories
FROM duckdb_papers.papers;

The KB stores vector embeddings and metadata. Query-time fields such as relevance and distance help rank results.

Fallback behavior: if the KB isn’t available, we degrade to DuckDB text search (LIKE over title/summary/categories) so the app keeps working.

Querying: semantic and hybrid search

All query logic is centralized in src/knowledge_base/queries.py. The app supports:

Basic semantic search (WHERE content = 'your query')
Thresholded semantic search (filter by relevance)
Hybrid search (semantic + metadata filters like author/year/category)

Example semantic search (SQL):

SELECT entry_id, title, summary, authors, published_date, pdf_url, categories,
       distance, relevance
FROM academic_kb
WHERE content = 'privacy in federated learning'
ORDER BY relevance DESC
LIMIT 10;

Python usage via the SDK (simplified idea):

from src.knowledge_base.queries import query_academic_papers
results = await query_academic_papers("privacy in federated learning", limit=10)

Hybrid constraints are applied either in SQL or post-filtered in Python, depending on the field.

Serving: FastAPI + Streamlit

FastAPI endpoints (see src/api/routes.py):
- POST /api/search
- POST /api/search/semantic
- POST /api/search/hybrid
- GET /api/papers/{entry_id}
- GET /api/health
Streamlit UI (src/ui/streamlit_app.py) calls those APIs and renders the results list with titles, authors, abstracts, links, and relevance scores.

This separation keeps the UI thin and the backend reusable.

Running it locally

Choose Docker or bare metal.

Docker (recommended):

# 1) Configure env
cp .env.docker.example .env
# (edit GEMINI_API_KEY)

# 2) Start services
docker compose up --build

# 3) Populate KB (first run)
docker compose exec academic_research_copilot python scripts/populate_kb.py

# 4) Open apps
# UI       → http://localhost:8501
# API Docs → http://localhost:8000/api/docs

Bare metal:

# 1) Configure env
cp .env.example .env
# (edit GEMINI_API_KEY)

# 2) Create & activate venv (zsh/bash)
python -m venv venv
source venv/bin/activate

# 3) Install deps
pip install -r requirements.txt

# 4) Populate KB (first run)
python scripts/populate_kb.py

# 5) Start apps (two terminals)
uvicorn src.app:app --reload
streamlit run src/ui/streamlit_app.py

Tip: The first populate run can take several minutes (fetching papers + generating embeddings). Subsequent runs are much faster.

What makes this effective

Semantic recall: finds conceptually similar papers even when vocabulary differs.
Hybrid control: tighten the net with author/year/category filters as needed.
Local-first: DuckDB + MindsDB, easy to containerize, embedding model.
Clean API surface: a small set of focused endpoints for the UI or other clients.

What’s next

Reranking: add a cross-encoder or LLM reranker on the top 20 candidates.
Query understanding: expand to multi-query rewriting or synonyms.
Summarization: on-demand TL;DR and key-takeaways for each paper.
Collections: let users save, label, and export reading lists.

Closing

The Academic Research Copilot is a pragmatic, local-first way to bring semantic search to your research workflow. It’s lightweight, fast to run, and easy to extend. If you’re curious about a new topic or deep into a literature review, this setup will save you time and surface better results.

Happy researching! 🎓

The Fascinating Journey of Database Management Systems: From Punch Cards to Cloud and AI-Powered Data Intelligence

Aashish Karki — Fri, 06 Dec 2024 17:30:45 +0000

Introduction:

Only a few domains in the world of technology have transformed as dramatically as database systems. What began as a simple method of recording census data has evolved into a complex ecosystem that powers virtually every digital interaction we experience today. Let us dive deep into the remarkable history of database systems—a tale of innovation, challenges, and relentless human creativity that now reaches the frontier of artificial intelligence.

The Early Days: Punched Cards and Magnetic Tapes(1950s-1960s):

The story begins in the early 20th century with Herman Hollerith's punched cards. Imagine a world where data was manually recorded on physical cards, it was actually the case in 1890 United States Census. These cards were the first step towards automated information processing, predating modern computers by decades.

In the 1950s and early 1960s, magnetic tapes revolutionized data storage. Businesses started automating processes like payroll, but data processing was incredibly rigid. Imagine having to sort punch cards and tapes in exact synchronization just to update employee salaries! Programmers had to meticulously order data, as tapes could only be read sequentially.

Breaking Free: The Disk Revolution(1960s-1970s):

The late 1960s and early 1970s marked a pivotal moment with the widespread adoption of hard disks. Suddenly, data wasn't confined to sequential access. Any piece of information could be accessed in milliseconds, freeing programmers from the "tyranny of sequentiality."

This era saw the birth of hierarchical and network data models. The hierarchical model organized data in a strict tree-like structure, similar to a family tree, with each record having a single parent. While excellent for representing simple, structured organizational relationships, it lacked flexibility for complex data interactions. The network model improved upon this limitation by allowing multiple relationships between records, creating an interconnected web of data that more closely resembled real-world complexity. Programmers could now construct and manipulate these structures with unprecedented flexibility.

The Relational Database: A Groundbreaking Paradigm(1970s-1980s):

In 1970, Edgar Codd published a landmark paper "A Relational Model of Data for Large Shared Data Banks," [ Link to the Paper ] that would change everything. The relational model introduced a revolutionary concept: a non-procedural way of querying data. Its simplicity was its strength – implementation details could be completely hidden from programmers.

Initially, relational databases were considered academically interesting but impractical. That was changed with IBM's System R project, which developed techniques for efficient relational database systems. Concurrent developments like the Ingres system at UC Berkeley and the first version of Oracle proved that relational databases could compete with existing models. Database performance optimization was a critical area of research and development at this period of time.

The magic of this era was the abstraction layer that relational databases provided. Programmers were liberated from low-level implementation details, allowing them to focus on logical data design rather than intricate performance optimization.

The Web Explosion: From Back-Office Tools to Global Powerhouses (1990s):

The 1990s marked a seismic shift in the role of databases, evolving them from back-office storage systems into the dynamic engines powering the global information age. With the explosive growth of the World Wide Web, databases became essential tools for scaling and democratizing data access. What was once a specialized technology now transformed into a flexible communication hub, connecting users worldwide. The era brought about groundbreaking changes, from web-based applications and transaction processing systems capable of managing massive, concurrent user loads to intuitive web interfaces that empowered non-technical users to access and interact with data. Suddenly, databases weren’t just about storing information—they became platforms for dynamic exchange and decision-making.

To keep up with the demands of a connected world, databases had to undergo rapid evolution. High-speed transaction processing, complex querying capabilities, and 24/7 availability became non-negotiable. Maintenance downtime was no longer an option, and systems had to meet the rising need for decision support and data analysis tools. It was like turning a specialized factory machine into a versatile, always-on global communication network. The result? Databases became the beating heart of the digital revolution, laying the groundwork for the modern web-driven world we take for granted today.

Transformations in Data Management: Innovations and Trends of the 2000s:

The 2000s marked an era of unprecedented diversification in data systems and formats. While the previous decades focused on standardization, this period revolved around specialization and flexibility. Data management solutions were no longer "one size fits all"—they evolved to handle new types of information and meet specific business needs.

Key Innovations:

The rise of semi-structured data formats like XML and JSON
Emergence of spatial and geographic databases for mapping and location-based services
Growth of open-source database systems such as MySQL and PostgreSQL
Development of specialized databases tailored to specific use cases

Social networks and web platforms fundamentally changed the game. Traditional tabular data structures, designed for rows and columns, struggled to represent the intricate relationships between users, posts, likes, and interactions. This led to the birth of graph databases—an entirely new paradigm designed for storing and analyzing interconnected data.

During this decade, the data analytics revolution kicked into high gear. Businesses started to see data not just as a byproduct of operations but as a strategic asset for driving decisions and growth. This shift gave rise to column-store databases, which excelled at rapidly analyzing massive datasets, providing the foundation for modern business intelligence and big data tools. The 2000s laid the groundwork for the explosion of big data technologies that would dominate the next decade.

The 2010s: Cloud, NoSQL, and Distributed Systems

The 2010s revolutionized data management, introducing unprecedented scale, distributed computing, and the emergence of cloud services that reshaped the landscape. Businesses enthusiastically adopted cloud storage and "Software as a Service" (SaaS) models, fundamentally changing their strategies for data storage and management. This decade marked the rise of NoSQL databases, challenging traditional approaches and offering flexible schema designs that prioritized scalability and performance, breaking free from rigid structures and strict consistency. Big data processing frameworks like Hadoop and Spark became essential tools, empowering organizations to analyze and derive insights from colossal datasets with remarkable efficiency. As companies increasingly shifted to cloud solutions, entrusting not just data storage but entire applications to third-party providers, the importance of data privacy, ownership, and regulatory compliance emerged as critical concerns. The NoSQL movement represented a transformative shift in database design philosophy, allowing systems to accommodate diverse and evolving data types without predefined schemas. This focus on scalability and eventual consistency enabled businesses to manage increasing workloads effectively across distributed systems. Amid this wave of innovation, the heightened emphasis on data security and privacy illuminated the challenges posed by the rapidly evolving data landscape in a cloud-driven era.

The 2020s: AI, Machine Learning, and Intelligent Databases

The 2020s have introduced a groundbreaking era for database technologies, driven by the integration of artificial intelligence (AI) and machine learning (ML). Databases are no longer passive storage systems—they’ve evolved into active, intelligent platforms capable of extracting insights, automating predictions, and interacting with users in ways that were once unimaginable. Innovations like MindDB exemplify this transformation. With features such as natural language querying, automated machine learning for generating insights and predictive models, and seamless data integration across multiple sources, these intelligent databases empower users to unearth hidden answers within complex data using simple, conversational language. This marks a profound leap forward in how we interact with and derive value from data.

The decade has also seen significant advancements in distributed and edge computing databases. From quantum databases leveraging quantum computing principles to edge computing databases processing data closer to its source, the focus has been on decentralization and efficiency. Blockchain-based databases have introduced unparalleled levels of transparency and security, while federated learning databases enable collaborative model training without requiring centralized data storage. At the same time, vector databases have emerged as a pivotal technology for AI and ML applications. Designed to handle embedding vectors—numerical representations of complex data like images, text, and audio—these systems power semantic search, recommendation engines, NLP applications, and machine learning model training at scale.

However, as databases become increasingly intelligent and interconnected, new ethical and regulatory challenges have come to the forefront. Privacy concerns have driven the need for stringent data regulations, while issues like bias detection in machine learning models and transparent AI decision-making demand greater accountability. Ethical guidelines for data collection, usage, and governance have become more critical than ever, ensuring that the innovations of the 2020s are aligned with society's values and expectations. Together, these advancements and challenges are redefining the very fabric of data management and intelligence in the modern era.

Getting Started with the Minds JavaScript SDK

Aashish Karki — Wed, 30 Oct 2024 03:22:49 +0000

Ever wanted to integrate AI models with your databases seamlessly? The Minds JavaScript SDK provides exactly that—a bridge between your data sources and AI models. In this comprehensive guide, we'll explore how to use this powerful tool to create and manage "minds" (AI models) and data sources.

What is Minds?

Minds is an AI platform designed to simplify the creation, deployment, and management of AI models—referred to as "minds". It allows developers to build AI-powered applications that can interact with data sources to provide intelligent responses or automate tasks. The JavaScript SDK makes this integration smooth and straightforward.

Note: This article focuses on the Minds platform and its JavaScript SDK. It is different from MindsDB, which is an open-source AI layer for existing databases. For more details on the differences, refer to the Minds vs. MindsDB section at the end.

Getting Started

Installation

First, let's install the SDK:

npm install minds_js_sdk

Basic Configuration

The SDK offers flexible configuration options:

import MindsClient from 'minds_js_sdk';

// Using environment variables
const client = new MindsClient();

// Or with custom configuration
const client = new MindsClient('your_api_key', 'https://api.minds.com');

Working with Data Sources

Let's dive into managing your data sources.

Creating a Data Source

Here's how to connect your PostgreSQL database:

const datasourceConfig = {
  name: 'my_postgres_db',
  engine: 'postgres',
  description: 'My PostgreSQL database',
  connection_data: {
    host: 'localhost',
    port: 5432,
    database: 'mydb',
    user: 'user',
    password: 'password',
  },
  tables: ['table1', 'table2'],
};

try {
  const datasource = await client.datasources.create(datasourceConfig);
  console.log('Datasource created:', datasource);
} catch (error) {
  console.error('Error:', error);
}

Managing Data Sources

The SDK provides simple methods for data source management:

// List all data sources
const datasources = await client.datasources.list();

// Get details of a specific data source
const datasource = await client.datasources.get('my_postgres_db');

// Delete a data source
await client.datasources.drop('my_postgres_db');

Working with Minds (AI Models)

Now comes the exciting part—creating and managing AI models!

Creating a Mind

Here's how to create an AI model connected to your data source:

const mindConfig = {
  name: 'my_mind',
  model_name: 'gpt-4',
  provider: 'openai',
  prompt_template: 'Use your database tools to answer the user\'s question: {{question}}',
  datasources: ['my_postgres_db'],
  parameters: {
    // Additional model parameters
  },
};

try {
  const mind = await client.minds.create('my_mind', mindConfig);
  console.log('Mind created:', mind);
} catch (error) {
  console.error('Error:', error);
}

Managing Minds

Similar to data sources, minds can be managed easily:

// List all minds
const minds = await client.minds.list();

// Get a specific mind
const mind = await client.minds.get('my_mind');

// Delete a mind
await client.minds.drop('my_mind');

Using a Mind for Completions

Here's where the magic happens—using your AI model:

try {
  // Regular completion
  const result = await mind.completion('What was our revenue last month?');
  console.log('Answer:', result);

  // Streaming completion
  await mind.completion('Analyze our sales trend', true);
  // Results will stream to stdout
} catch (error) {
  console.error('Error:', error);
}

Error Handling Best Practices

The SDK includes custom error classes for better error handling:

import { ObjectNotFound, ObjectNotSupported } from 'minds_js_sdk/exception';

try {
  const mind = await client.minds.get('non_existent_mind');
} catch (error) {
  if (error instanceof ObjectNotFound) {
    console.error('Mind not found');
  } else if (error instanceof ObjectNotSupported) {
    console.error('Unsupported operation');
  } else {
    console.error('Unexpected error:', error);
  }
}

Pro Tips

Default Prompt Template: The SDK comes with a default prompt template: "Use your database tools to answer the user's question: {{question}}"
OpenAI Integration: The SDK uses OpenAI's client for completions, making it compatible with popular AI models.
Project Structure: All minds are created under the 'minds' project by default.

Conclusion

The Minds JavaScript SDK provides a powerful interface to integrate AI capabilities with your databases. Whether you're building a data analytics tool, a chatbot, or any application that needs AI-powered database interactions, this SDK makes the process straightforward and developer-friendly.

Remember to check out the official documentation for more advanced features and updates.

Minds vs. MindsDB

It's important to note the difference between Minds and MindsDB:

Minds: An AI platform that simplifies the creation and deployment of AI models (minds) that interact with data sources via APIs and SDKs. Ideal for developers building AI-powered applications.
MindsDB: An open-source AI layer for existing databases, allowing machine learning models to be trained and deployed inside the database environment using SQL. Suited for data scientists and database administrators.

Understanding these differences ensures you choose the right tool for your specific needs.

Resources

Have you tried integrating AI with your databases before? What challenges did you face? Let's discuss in the comments below! 👇

Note: This article is based on the latest version of the **Minds JavaScript SDK. Features and APIs might change in future versions.