DEV Community: Aarav Rana

GraphRAG Pipeline: Airbyte Ingestion to Neo4j Knowledge Graph

Aarav Rana — Sat, 17 Jan 2026 16:33:33 +0000

Introduction

I’ve spent the last year building RAG systems, and I’ve hit a hard ceiling.

Vector databases are fantastic at finding similar text. If you ask, "How do I reset my password?", a vector search finds the password reset doc immediately.

But they are terrible at relationships.

If I ask my internal dev bot: "Who is currently working on the ticket related to the billing API latency?", a vector DB fails. It finds documents with "billing" and "latency," but it has no concept of "Assigned To" or "Created By." It sees words; it doesn't see the graph.

To fix this, I stopped trying to force everything into vectors and moved to GraphRAG.

This is the architecture I built to solve it: Airbyte handles the messy API extraction, Neo4j stores the topology, and Gemini 3.0 translates my English questions into Cypher queries.

View the Full Source Code on GitHub

The Architecture

We aren't just moving tables; we are building a semantic network.

Ingestion (Airbyte): It pulls raw data (Issues, Pull Requests, Users) from GitHub/Jira/Salesforce instead of writing custom scrapers.
Storage (Neo4j): We need a native graph database to traverse deep relationships (e.g., User -> Team -> Ticket -> PR).
Reasoning (Gemini 3.0 Pro): We need an LLM that is strictly logical to write accurate database queries. The 3.0 Pro model is significantly better at SQL/Cypher generation than previous iterations.

Why I ditched Vector-Only RAG

In my previous iteration of this project, I used a standard vector database. It worked fine for questions like "What is an error?" but failed miserably at "Who is responsible for the error?".

I found that Vector DBs are great at Similarity but terrible at Topology. If the relationship between "User" and "Issue" isn't explicitly written in text (e.g., "John created issue #4"), the vector search misses it. Neo4j was the only way to make those invisible links visible to the LLM.

Prerequisites

Docker & Docker Compose installed.
Airbyte (Open Source version running locally).
Neo4j Database (Neo4j Aura Free Tier or local Docker).
Python 3.10+.

Step 1: Local Infrastructure

I run Neo4j in Docker to keep my host machine clean. We need the apoc plugins enabled to handle some of the JSON parsing later.

docker-compose.yml

version: '3.8'
services:
  neo4j:
    image: neo4j:5.15.0
    container_name: graphrag-neo4j
    ports:
      - "7474:7474"
      - "7687:7687"
    environment:
      NEO4J_AUTH: neo4j/password
      NEO4J_PLUGINS: '["apoc"]'

Start the instance:

docker-compose up -d

Step 2: The "Airbyte" Data Dump

When Airbyte syncs data into a database, it doesn't magically create a graph. It dumps "Raw Tables." For example, a GitHub Issue becomes a node labeled _AirbyteRawGitHubIssues containing a giant JSON blob.

To make this guide reproducible, I wrote a script that mimics this exact "Raw Destination" format.

src/seed_airbyte_data.py

tx.run("""
    CREATE (n:_AirbyteRawGitHubIssues {
        _airbyte_data: $data,
        _airbyte_ab_id: randomUUID()
    })
""", data=json.dumps(issue))

Run the seeder:

python src/seed_airbyte_data.py

Now we have the data, but it's messy. It's just disconnected JSON files. This is where normal RAG faces limits.

Step 3: Transforming JSON to Graph

This is the most critical part of the pipeline. We need to take those JSON files and "explode" them into entities.

We need to tell the database: "Take the user.login field from the Issue JSON, find the User node with that name, and draw a :CREATED line between them."

src/build_graph.py

def transform_graph(tx):
    tx.run("""
        MATCH (r:_AirbyteRawGitHubUsers)
        WITH r, apoc.convert.fromJsonMap(r._airbyte_data) AS data
        MERGE (u:User {login: data.login})
    """)

    tx.run("""
        MATCH (r:_AirbyteRawGitHubIssues)
        WITH r, apoc.convert.fromJsonMap(r._airbyte_data) AS data
        MERGE (i:Issue {id: data.id})

        WITH i, data
        MATCH (u:User {login: data.user.login})

        MERGE (u)-[:CREATED]->(i)
    """)

Run the build script:

python src/build_graph.py

If you check the Neo4j Browser now, you won't see isolated records. You'll see a web.

Step 4: The Agent (Gemini 3.0)

Standard RAG searches for keywords. GraphRAG searches for paths.

To make this work, we use LangChain to pipe our schema into Gemini 3.0 Pro. We have to be very strict with the prompt. LLMs hallucinate column names (guessing username instead of login).

src/query_graph.py

llm = ChatGoogleGenerativeAI(model="gemini-3.0-pro", temperature=0)

CYPHER_GENERATION_TEMPLATE = """
Schema:
{schema}

CRITICAL RULES:
1. For User nodes, ALWAYS use the property 'login'.
2. The relationship is [:CREATED], do NOT guess [:AUTHORED].
3. Use 'CONTAINS' for partial string matching.

The question is: {question}
"""

chain = GraphCypherQAChain.from_llm(
    llm, 
    graph=graph, 
    cypher_prompt=CYPHER_PROMPT,
    verbose=True
)

Step 5: Result

Let's try a query that requires hopping from a User to an Issue based on a partial title match.

Query: "Who created the issue about 'Memory Leak'?"

Generated Cypher:

MATCH (u:User)-[:CREATED]->(i:Issue)
WHERE toLower(i.title) CONTAINS toLower('Memory Leak')
RETURN u.login

Result:

💡 AI: ai_wizard created the issue.

Final Thoughts

Building this pipeline made me realize that data topology is just as important as data content.

By using Airbyte for the plumbing and Neo4j for the structure, we can build AI agents that actually understand context, not just keywords. And by leveraging Gemini 3.0 Pro, we get high-accuracy query generation without the massive cost overhead of older models.

View the Full Source Code on GitHub

Serverless AI Agents on Civo: Replacing Docker with WebAssembly (Spin) and Rust

Aarav Rana — Sat, 17 Jan 2026 16:33:11 +0000

Introduction

Performance benchmarks for AI agents often overlook the impact of cold-start latency. While Docker remains the industry standard for containerization, its resource overhead is a significant bottleneck for ephemeral AI workloads. Wrapping a simple inference agent in a 2GB Linux container is inefficient, especially when scaling hundreds of instances on a Kubernetes cluster.

WebAssembly (Wasm) offers a high-performance alternative. By compiling Rust-based agents into Wasm modules, we can achieve binary sizes under 5MB and near-instant execution. This guide demonstrates how to deploy these modules on a Civo K3s cluster using SpinKube to optimize both cost and infrastructure density.

View the Full Source Code on GitHub

Real-World Bottleneck

I started this experiment because I was hitting a wall with my Docker-based agents. On a standard K3s node, pulling a 2GB PyTorch container took about 14 seconds on a cold start. For a chatbot that needs to reply instantly, that's unacceptable.

By switching to this Rust/Wasm architecture, I didn't just "optimize" code—I changed the physics of the deployment.

Docker Cold Start: ~14.2s
Wasm Cold Start: ~0.04s

Why WASM on Civo?

Civo’s K3s clusters are optimized for lightweight workloads. By pairing Civo with the Spin framework, we gain three distinct advantages:

Sub-millisecond Startups: WASM modules bypass the heavy initialization of the Linux kernel required by Docker.
Resource Efficiency: High pod density is achievable because Wasm modules share the host's runtime resources more efficiently than containers.
Native AI Integration: The Spin SDK allows Wasm modules to offload LLM inference to the host nodes, removing the need to bundle heavy libraries like PyTorch or TensorFlow within the application.

Prerequisites

Rust 1.84+
Spin CLI v3.0+.
Civo CLI
Helm

Step 1: The Infrastructure (Civo + SpinKube)

Standard Kubernetes nodes are designed to execute OCI containers. To run Wasm binaries, we must install the SpinKube operator and a runtime shim.

Install the SpinKube Operator: Use Helm to deploy the operator and the runtime class.

helm repo add kwasm http://kwasm.sh/kwasm-operator/
helm install kwasm-operator kwasm/kwasm-operator \
  --namespace kwasm --create-namespace \
  --set kwasmOperator.installer.type=image

Annotate K3s Nodes: Label your Civo nodes to enable the WASM runtime shim.

kubectl annotate node --all kwasm.sh/kwasm-node=true

Step 2: Developing the Inference Agent

We will build an agent that performs sentiment analysis using Llama 2. The Rust code utilizes the spin_sdk::llm module to perform inference without managing model weights in memory.

Logic (src/lib.rs):

use spin_sdk::http::{IntoResponse, Request, Response};
use spin_sdk::http_component;
use spin_sdk::llm;
use std::str;

const MODEL: llm::InferencingModel = llm::InferencingModel::Llama2Chat;

fn handle_request(req: Request) -> anyhow::Result<impl IntoResponse> {
    let body = req.body();
    let input_text = str::from_utf8(body).unwrap_or("");

    let prompt = format!(
        "Analyze the sentiment of this text and reply with only 'Positive', 'Negative', or 'Neutral': '{}'",
        input_text
    );

    let result = llm::infer(MODEL, &prompt)?;

    Ok(Response::builder()
        .status(200)
        .header("Content-Type", "application/json")
        .body(format!(r#"{{"sentiment": "{}"}}"#, result.text.trim()))
        .build())
}

Step 3: Compiling to WebAssembly

We target the WebAssembly System Interface (WASI) to ensure the binary is portable across any infrastructure running a compatible shim.

Configure the Build:

In your spin.toml, ensure the build command points to the wasip1 target.

[component.civo-spin-ai-agent]
source = "target/wasm32-wasip1/release/civo_spin_ai_agent.wasm"
build = { command = "cargo build --target wasm32-wasip1 --release" }

Execute the Build:

spin build

Upon completion, the resulting .wasm file should be approximately 2.5MB. This is a massive reduction in deployment size compared to a 2GB Python-based Docker image.

Step 4: Production Deployment

We package the Wasm binary as an OCI artifact and push it to a registry.

Push the Artifact:

spin registry login ghcr.io

spin registry push ghcr.io/RanaAarav/CIVO-Spin-AI-Agent

Deploy to Kubernetes:

We don't need to write a YAML file manually. Spin creates the "Runtime Class" configuration for us.

spin kube scaffold --from ghcr.io/RanaAarav/CIVO-Spin-AI-Agent/deploy/k8s-deployment.yaml

Apply it:

kubectl apply -f k8s-deployment.yaml

Step 5: Testing the Agent

Now let's hit the endpoint. Since we didn't set up an Ingress for this demo, we'll port-forward.

kubectl port-forward svc/civo-agent 8000:80

Run the Inference:

curl -X POST http://localhost:8000 \
     -d "Civo Kubernetes is incredibly fast and easy to use."

Response:

Positive

Conclusion

By moving away from the "Docker-for-everything" mindset, we can build AI agents that are significantly leaner and faster. Using Rust and Wasm on Civo K3s reduces binary sizes by 99% and eliminates the cold-start latencies typically associated with containerized AI. As the serverless AI landscape evolves, this architecture provides a cost-effective and scalable foundation for real-time agentic applications.

View the Source Code on GitHub

Beyond Git: Features Every Young Professional Should Know in 2025

Aarav Rana — Sun, 21 Sep 2025 14:48:36 +0000

Everyone knows git init, git commit, git push. They’re foundational.

But if you're building skills that stand out, there are several new GitHub practices & features that are becoming essential in 2025.

Here are some features & best practices that fresh grads, interns, or early-career devs should be aware of, not just for doing the job, but doing it professionally.

1. Advanced Security with GitHub Advanced Security (GHAS)

GitHub isn’t just a hosting platform anymore, it now offers more built-in protection for your code and dependencies than a team of testers.
Official GHAS docs

Key updates in 2025:

Unbundled security products: As of September 2025, GHAS is split into two offerings:
1. GitHub Secret Protection → catches leaked secrets (API keys, credentials), includes push protection using AI-powered detection.
2. GitHub Code Security → includes code scanning (via CodeQL), dependency review, Copilot Autofix, and security campaigns.

Public vs Private repos: Many security features are free for public repositories; for private/internal ones, you’ll often need GHAS. Using these tools avoids vulnerabilities, supply chain risks, and shows safety in your workflow.

2. Immutable Actions & Supply Chain Safety

Supply chain security is a huge deal in 2025. GitHub now offers features to lock things down.

Immutable Actions (preview) Instead of depending on mutable references like this:

  uses: actions/checkout@v3

You pin it to a commit hash:

  uses: actions/checkout@93ea4e...

These are distributed as OCI artifacts with provenance info.
Immutable Releases (public preview)

Once a release is marked immutable, its assets and Git tag cannot be changed. Attestations are generated automatically so you can verify provenance.
Immutable Releases docs

This prevents tampering with actions or releases, protecting your builds and users.

3. GitHub Previews (Spark, Copilot Spaces, etc.)

AI is becoming a daily part of dev workflows.

Copilot Agents & Mission Control GitHub Agents can now fix bugs, generate PRs, or update docs automatically. The Agents Panel (“Mission Control”) lets you start tasks from anywhere and track progress.
Previews worth trying
- Immutable Actions
- Copilot Autofix and deeper IDE integrations
- GitHub Spark (lightweight collaboration tool)

GitHub Previews

4. Workflow & CI/CD Best Practices

Beyond “Hello World” workflows, code needs maintainability and reusability.

Reusable workflows / templates

  jobs:
    build:
      uses: ./.github/workflows/build.yml

Cache & split jobs Speed up builds by caching dependencies and breaking workflows into parallel jobs.

Self-hosted runners Required in orgs with sensitive infra. Know how to set them up & secure them.
Networking requirements With Immutable Actions, self-hosted runners must allow traffic to pkg.actions.githubusercontent.com and ghcr.io.
Breaking changes notice
Branch protection & reviews Configure required reviews, status checks, and security rules. These reflect professionalism.

5. Essentials Freshers Often Overlook 🧭

These aren’t fancy, but they separate juniors from pros.

SSH vs HTTPS: HTTPS now requires PATs. SSH is smoother daily. Worth setting up.
Managing multiple accounts: Use ~/.ssh/config to handle work + personal + college accounts cleanly:

Host github-work
  HostName github.com
  User git
  IdentityFile ~/.ssh/id_rsa_work

README.md = Your Project’s Resume Include: intro, setup, usage, license, screenshots, badges.
CONTRIBUTING.md Document contribution flow, coding style, PR etiquette. Maintainers do notice.
Commit hygiene: Small, clear commits with meaningful messages. Recruiters and reviewers value this more than people think.

6. Why This Matters in 2025 (and Beyond)

Security threats are more sophisticated. Immutable releases/actions and GHAS are direct defenses.
AI tools are embedded in workflows, not optional extras. Early adopters gain an edge.
Professional Styling (docs, workflows, secure pipelines) is expected in modern dev environments.
Employers and open source maintainers notice these things before they even look at your code.

Final Takeaway

GitHub in 2025 is not just about git push.

Security first → GHAS, immutable actions/releases.
AI productivity → Copilot Agents, previews, automation.
Professional polish → SSH, READMEs, CONTRIBUTING.md, commit hygiene.

This is the Github knowledge you'd want to beat 2025.