DEV Community: Prathamesh Anvekar

Building a Secure LLM Proxy with Fine-Tuned Gemma 3 for Production Applications

Prathamesh Anvekar — Sun, 18 Oct 2026 07:46:58 +0000

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 2: Side Hustle

What I Built

Over the past year, many freelance developers, small agencies, and independent builders have integrated LLM capabilities into their client work. Whether creating customer intake assistants, contract summarizers, or invoice extractors, clients increasingly expect AI-driven workflows.

However, deploying generative models in client-facing environments introduces several practical operational and compliance challenges. Clients and end users frequently paste sensitive personal data - including customer contact information, phone numbers, and production API credentials - directly into prompt inputs. Input fields are also vulnerable to adversarial prompt injections designed to bypass system instructions. In addition, document analysis workflows can be targeted with hidden or low-contrast text embedded within PDFs and images, potentially hijacking downstream reasoning.

While enterprise AI security platforms exist, they are typically aimed at large organizations with corresponding price points and complex onboarding processes. For independent developers and small software studios delivering custom applications on modest budgets, these enterprise solutions are often impractical.

I built AEGIS as a lightweight, drop-in security proxy that developers can place in front of their LLM endpoints. It acts as an intermediary gateway that sanitizes personal data in real time, inspects and OCRs uploaded documents, evaluates prompt safety using an open-source fine-tuned Gemma 3 model, and provides sanitized web retrieval through SerpApi. For agencies and independent contractors, this provides a straightforward security layer that can be offered to clients as a paid compliance add-on or hosted as a managed micro-service.

Incoming Untrusted Payload (Text / PDF / Image / Web Search)
                        │
                        ▼
   [ Layer 1: Magika Deep MIME & Polyglot Verification ]
                        │
                        ▼
   [ Layer 2: Dual-Stream OCR & Hidden Text Extractor ]
                        │
                        ▼
   [ Layer 3: Context-Aware Presidio + Regex PII Scrubber ]
           (Transforms PII -> <EMAIL>, <PHONE>, <API_KEY>)
                        │
                        ▼
   [ Layer 4: Fine-Tuned Gemma 3 / LoRA Safety Classifier ]
           (Detects Prompt Injections & Malicious Payloads)
                        │
                        ▼
   [ Layer 5: Sanitized Web Retrieval via SerpApi ]
           (Filters Poisoned Knowledge Blocks & Injections)
                        │
                        ▼
      Clean, Verified Context -> Target LLM -> User

The gateway handles inbound requests across multiple input types. Plain text prompts are stripped of credentials and personal identifiers before being evaluated by the safety classifier. File uploads undergo deep MIME verification and visual OCR to detect obscured text before text extraction proceeds. When web search augmentation is enabled, SerpApi results are retrieved and scrubbed before being appended to the model context. A web dashboard provides real-time visibility into each verification step, showing exactly what data was modified and why.

Demo

The application provides an interactive interface built to demonstrate each stage of the proxy pipeline. Users can enter raw prompts, upload documents, or toggle web augmentation while observing how the system handles the input.

When a request is submitted, the interface displays the immediate verdict from the fine-tuned Gemma 3 classifier alongside a detailed breakdown of the sanitization process. Original text and redacted text are presented side by side, highlighting replaced entity tags such as email addresses, phone numbers, and API tokens. For file submissions, the system displays the extracted text alongside confidence scores from the OCR and file analysis stages. The dashboard also includes reference summaries of recent real-world security incidents, such as source code leaks and exposed credentials, to provide practical context for why these protective layers are necessary in production software.

Code

The complete source code is available on GitHub at

prathamanvekar / secure-llm-proxy

AEGIS 🛡️

Guardian for your LLMs - A secure, production-ready AI gateway with multi-layer defensive sanitization.

An open-weight, privacy-first proxy that intercepts sensitive PII, neutralizes prompt injections, de-weaponizes uploaded documents, and filters web context using a fine-tuned Gemma 3 model.

Overview • Architecture • Key Features • Tech Stack • Quickstart • Verification

Overview

As generative AI transitions into production workflows, organizations face several critical operational risks: accidental exposure of personal identifiable information (PII), adversarial prompt injections designed to hijack model instructions, and malicious payloads hidden within uploaded files or untrusted web context.

AEGIS provides an automated security proxy that sits between client applications and downstream models. It cleans inbound prompts, redacts sensitive credentials and contact data, inspects multi-format documents using dual-stream OCR, classifies injection threats using an open-source fine-tuned Gemma 3 model, and sanitizes web knowledge retrieved through SerpApi.

Zero Trust AI Gateway: Raw client prompts, uploaded documents…

View on GitHub

. The repository includes the FastAPI gateway implementation in backend.py, the training and inference notebook for fine-tuning Gemma 3 in backend notebook/aegis-server-wow-verse-mmcoe.ipynb, and the frontend interface in the src/ directory.

How I Built It

AEGIS is structured around an asynchronous multi-layer pipeline combining open-source models, parsing utilities, and structured validation routines.

At the core of the system is a fine-tuned Google Gemma 3 open-weight model (gemma-safety-classifier). Using open weights was central to this project because safety classification needs to be fast, deterministic, and free of per-token API charges. I fine-tuned Gemma 3 using Low-Rank Adaptation (LoRA) through the PEFT library, training it specifically to evaluate user prompts and classify them into discrete categories including safe prompts, prompt injections, cybercrime or malware payloads, and general policy violations. To keep memory utilization minimal while maintaining classification accuracy, the model runs with 4-bit NormalFloat quantization through BitsAndBytes:

# Quantized loading of the fine-tuned Gemma 3 safety classifier
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_use_double_quant=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16
)

peft_config = PeftConfig.from_pretrained(model_path)
base_model = AutoModelForCausalLM.from_pretrained(
    peft_config.base_model_name_or_path,
    quantization_config=bnb_config,
    torch_dtype=torch.bfloat16,
    trust_remote_code=True,
    low_cpu_mem_usage=True
)
self.model = PeftModel.from_pretrained(base_model, model_path)

By deploying the fine-tuned Gemma 3 model directly within the proxy, the system evaluates inputs deterministically without routing them to third-party providers. The model is paired with a lightweight heuristic check for common benign drafting requests, ensuring that standard creative writing tasks pass through without unnecessary latency.

File uploads introduce an attack surface where malicious instructions can be hidden inside documents. To handle this, AEGIS uses Google Magika to inspect byte headers and confirm that files match their claimed extensions. When processing PDFs, the system uses a dual-stream parsing approach: pdfplumber extracts standard textual content, while pdf2image renders each page into an image for OCR evaluation using Tesseract and EasyOCR. This allows the system to catch adversarial techniques such as zero-opacity fonts, off-canvas text, or white-on-white text that bypass typical text parsers.

Personal data sanitization runs through Microsoft Presidio combined with targeted regular expressions. The pipeline identifies emails, phone numbers, credit card numbers, US Social Security numbers, IBANs, and common API key patterns, replacing them with standardized placeholders such as <EMAIL> or <API_KEY> before text reaches any downstream LLM:

def sanitize_input(user_text: str) -> Dict[str, Any]:
    regex_cleaned = regex_sanitize(user_text)
    HIGH_RISK = {"EMAIL_ADDRESS", "PHONE_NUMBER", "CREDIT_CARD", "API_KEY", "IBAN", "US_SSN"}
    final_text, presidio_findings = presidio_redact(regex_cleaned, redact_types=HIGH_RISK)
    return {
        "original": user_text,
        "final_sanitized": final_text,
        "presidio_findings": presidio_findings
    }

For applications requiring external knowledge retrieval, AEGIS integrates SerpApi to query Google Search. Rather than passing raw search results directly into the prompt context, AEGIS extracts organic search snippets, question summaries, and overview blocks, running each text fragment through the fine-tuned Gemma 3 classifier and PII sanitizer before assembling the final reference block. This mitigates indirect prompt injection from untrusted external websites:

def secure_serp_knowledge(results: Dict) -> List[str]:
    raw_blocks = extract_knowledge_from_serp(results)
    safe_blocks = []

    for text in raw_blocks:
        threat = detect_prompt_injection(text)
        if threat != "SAFE":
            continue
        if len(text.strip()) >= 20:
            safe_blocks.append(text)

    return safe_blocks

Why Does Open Innovation Matter?

For a developer tool designed as a sustainable side project, using open-source AI is what makes the architecture viable both technically and commercially.

First, open weights make the unit economics predictable. If a security proxy had to invoke a closed commercial API to evaluate every incoming prompt, inference costs would scale directly with volume. A client sending hundreds of thousands of requests per month would generate significant API bills, eroding the margins of a flat-rate subscription. By running a fine-tuned Gemma 3 model in 4-bit precision, inference runs on accessible consumer or cloud GPU hardware with zero marginal cost per request.

Second, open models uphold genuine data privacy. Routing unvetted user prompts containing confidential customer information to a third-party proprietary API simply to check for sensitive data creates the very exposure problem the tool is meant to solve. Running an open-source model locally or in a dedicated private environment ensures that raw customer data never leaves the developer's controlled infrastructure.

Finally, open innovation enables task-specific fine-tuning. General-purpose proprietary models are designed to balance broad conversational capabilities, making them prone to either over-refusing benign technical prompts or missing specialized adversarial injection techniques. Having direct access to Gemma 3 allowed me to fine-tune the weights on dedicated security and prompt injection datasets, resulting in a compact, highly focused classifier that performs its specific role reliably.

My First User

To test AEGIS in a real-world workflow, I shared an instance with an independent developer who builds intake chatbots and document summarization tools for local accounting clients. His primary concern was handling client tax documents where individuals frequently paste sensitive identity numbers or confidential notes into unstructured form fields.

We configured his development environment to route test requests through AEGIS. During testing, he submitted sample tax summaries containing mock Social Security numbers and phone numbers, as well as test documents with hidden prompt injection overrides. AEGIS redacted the personal data into standardized tokens before passing the text to the summarizer, and successfully flagged the injection attempts without disrupting normal queries.

Based on the testing, he noted that having an auditable sanitization layer reduced the compliance risk of offering AI features to his clients. He indicated he would pay a monthly fee of $45 to route his production client applications through a hosted instance of the proxy, and suggested adding webhook alerts to notify his team via Slack whenever a prompt injection attempt is intercepted.

My Agent Session

Building Feels: Local AI Soundtrack Matcher for Outdoor Spaces

You

Hacktoberfest Dev Challenge: Touch Grass. Build an open-source AI project that gets people off the screen and into the real world, running open-weight models locally with privacy and grounded discovery.

Agent

Designing Feels: a local, privacy-first web application that turns recent Spotify listening history into curated outdoor nature destinations. Architecture uses local Qwen 2.5 Coder 7B via Ollama for mood synthesis, SerpApi for grounded Google Maps venue discovery, SQLite WAL mode for deterministic caching and rate limiting, and editorial neobrutalism UI.

▸ run_tests python -m pytest tests -v

Input

python -m pytest tests -v

Output

21 passed in 2.75s

You

Separate the Landing Page and the App. After connecting to Spotify, redirect to the main app with a 4-stage data processing indicator before beautifully revealing the outdoor recommendations. Rename the app to Feels.

Agent

Segregated editorial landing page at / and application dashboard at /app. Integrated real-time 4-stage processing progress bar (track reading, local Qwen inference, SerpApi discovery, and grounding), updated branding to Feels, migrated SQLite to feels.db, and verified 15/day quota enforcement.

Prize Categories

Best Use of Gemma

AEGIS uses a fine-tuned Google Gemma 3 open-weight model as its central security classifier. Trained with LoRA adapters and running with 4-bit quantization, Gemma 3 provides deterministic, low-latency classification for prompt injections, malware payloads, and unsafe requests without relying on closed cloud moderation APIs.

Feels: Translating Your Spotify Soundtrack into Real Outdoor Places with Local AI and SerpApi

Prathamesh Anvekar — Fri, 09 Oct 2026 21:49:14 +0000

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 1: Touch Grass

What I Built

Like many developers, I spend hours locked in a sensory bubble at my desk with noise-canceling headphones on, cycling through music while deep in code. Over time, I noticed something undeniable about my daily Spotify queue: my listening history serves as an unfiltered mirror of my mental state and fatigue. When hunting subtle bugs or sprinting against deadlines, my queue naturally shifts to driving, high-tempo electronica. When unwinding after a long refactoring session or mapping system architecture, it settles into minimalist acoustic or atmospheric ambient soundscapes:

Session Window (Last 10 Tracks):
- M83: Midnight City -> 105 BPM, expansive synth wave, high kinetic momentum
- Justice: Genesis -> 117 BPM, distorted analog bass, aggressive forward drive
- The xx: Intro -> 100 BPM, minimalist reverberation, steady contemplative pace
- The Prodigy: Breathe -> 130 BPM, breakbeat rhythm, peak adrenaline burst
- Woodkid: Run Boy Run -> 132 BPM, marching orchestral drums, surging forward drive

Streaming algorithms take this behavioral data and use it to feed more screen addiction, building autoplay queues engineered to keep you in your chair. The moment you push your commits, you close your editor only to open social feeds or video streams instead of getting fresh air.

I built Feels to turn that dynamic around: using your soundtrack as a natural bridge to step away from your desk and reconnect with the outdoors.

Feels is a web application that takes your recent Spotify listening session and interprets its acoustic cadence, energy density, and emotional texture using an open-weight model running locally on your hardware. Rather than trapping you in a conversational chatbot loop, it acts as an environmental scout. It maps your sonic state to geographic search intents, then uses SerpApi's Google Maps engine to find three verified real-world outdoor destinations - such as elevated ridgeline trails, shaded forest loops, or tranquil lakeside parks. Each spot includes verified user ratings, review counts, photos, a grounded explanation of why it fits your current soundtrack, and one-click Google Maps directions.

To ensure the screen stays the shortest part of the experience, the application enforces a strict cap of 15 refreshes per day. You check your curated spots, pick a trail, shut the laptop, and head outside.

Demo

Here is a walkthrough of Feels in action, showing the Spotify authorization, the live four-stage processing indicator, and the resulting outdoor trail cards:

Code

prathamanvekar / feels

Feels 🌿

Turn your music into a reason to get outside.

An open-weight, privacy-first web application that translates your recent Spotify listening history into curated outdoor destinations using on-device AI and real-time map discovery.

Live Demo • Architecture • Configuration • Deployment

Overview

Most streaming apps use your listening history to feed more screen time, generating algorithmic queues engineered to keep you in your chair.

Feels flips that dynamic around.

Instead of another chatbot or infinite playlist, Feels treats your recent soundtrack as an unfiltered mirror of your mental headspace. It reads your 10 most recent tracks once per day, runs an open-weight model (Qwen 2.5 Coder 7B) locally on your machine to analyze tempo, energy density, and emotional cadence, and grounds that mood in real-world outdoor spaces using SerpApi's Google Maps engine.

You get three curated physical destinations - elevated ridge trails, shaded forest loops, or…

View on GitHub

How I Built It

The system is designed as an end-to-end pipeline connecting audio streaming telemetry, private on-device reasoning, and grounded spatial exploration. When users open the app, a real-time four-stage status pipeline visualizes the progress as tracks are fetched, analyzed on-device, queried spatially, and curated:

Data ingestion begins with the Spotify Web API, authenticating users via OAuth 2.0 with the user-read-recently-played scope. Rather than polling continuously, Feels retrieves your ten most recently played tracks at most once per calendar day. The backend extracts essential track attributes, artist metadata, and timestamps to evaluate pacing and acoustic density while keeping external requests to a minimum. These tracks are hashed and stored in an embedded SQLite database (feels.db), ensuring that subsequent refreshes throughout the day never spam Spotify servers.

To translate raw music tracks into an outdoor setting, Feels runs Qwen 2.5 Coder 7B locally through Ollama on localhost port 11434. Instead of open-ended conversational completions, the model is constrained to output a strict JSON schema containing acoustic energy levels, dominant emotional descriptors, and targeted spatial search queries:

# Querying local Ollama with JSON schema enforcement and defensive extraction
payload = {
    "model": "qwen2.5-coder:7b",
    "prompt": f"{ENVIRONMENTAL_PSYCHOLOGY_PROMPT}\n\nRecent listening session:\n{tracklist_text}",
    "stream": False,
    "format": "json",
    "options": {"temperature": 0.35, "top_p": 0.9, "num_predict": 700},
}
async with httpx.AsyncClient(timeout=60.0) as client:
    response = await client.post("http://127.0.0.1:11434/api/generate", json=payload)
    analysis = extract_and_validate_json(response.json().get("response", ""))

Those environmental queries feed directly into SerpApi's Google Maps engine to retrieve genuine physical places with verified ratings, reviews, and geographic coordinates. The SQLite cache deduplicates visited spots so each refresh rotates to fresh outdoor locations. For cloud hosting and continuous deployment, Feels includes a native render.yaml blueprint, allowing it to deploy seamlessly to Render as a Python web service with automatic dependency installation, persistent environment variables, and live health monitoring via /api/health.

Why Does Open Innovation Matter?

Listening history is personal behavioral telemetry. What someone plays during late-night debugging or high-pressure sprints reflects their emotional state, stress levels, and daily habits. Routing raw listening history to third-party commercial AI providers exposes private behavioral patterns to corporate tracking pipelines. Running an open-weight model locally ensures that private listening data never leaves the developer's laptop, transmitting only generalized, non-identifying outdoor search terms to map services.

Open innovation also eliminates recurring token anxiety. When small personal utilities depend on metered cloud APIs, every casual refresh incurs a monetary penalty. Running open-weight models on consumer hardware provides predictable, deterministic inference at zero marginal cost, enabling sustainable tools that cost nothing to operate.

Finally, outdoor exploration often takes place near hill ridgelines, nature sanctuaries, and forest paths where mobile reception is weak or unavailable. Because Feels combines local Ollama inference with persistent SQLite caching, the core mood profiling and trail guidance continue to function smoothly even when disconnected from the network.

My Agent Session

Building Feels: Local AI Soundtrack Matcher for Outdoor Spaces

You

Agent

▸ run_tests python -m pytest tests -v

Input

python -m pytest tests -v

Output

21 passed in 2.75s

You

Agent

Prize Categories

Best Use of Render

Turning metacognition notes to your competitive programming coach

Prathamesh Anvekar — Mon, 05 Oct 2026 00:11:51 +0000

This is a submission for the Hacktoberfest Weekend Challenge: Build for a Friend

What I Built

My friend Ved and I have been grinding competitive programming together for months. After discussing Colin Galen's video on how top competitors dissect their practice, we started doing something specific: keeping a text file open while solving LeetCode and Codeforces problems to record timestamped "metacognition notes" of our thoughts:

0:00 - read problem statement. contiguous subarrays with k distinct elements.
0:03 - thought about standard sliding window with two pointers.
0:06 - issue: how to count *exact* k without missing smaller left bounds?
0:12 - stuck trying to rewind left pointer. count gets messy with duplicates.
0:18 - breakthrough: exact(k) = atMost(k) - atMost(k - 1)!
0:25 - wrote helper, tested edge cases, submitted -> Accepted.

These notes capture our real-time cognitive blindspots - the exact minute we got confused, the false paths we went down, and the habits that cost us time, like jumping straight into code before proving invariants. But the moment a problem was accepted, we always discarded them.

I built cpmeta for Ved and myself to stop that hard-earned data from going to waste.

cpmeta is a project that analyzes raw thinking notes alongside the problem statement using an open-weight LLM. Instead of just handing you an algorithmic solution, it acts like an ICPC coach reviewing your thought process. It calculates exactly how long you stalled and diagnoses the root cause, such as missing a monotonic reduction trick. It also detects recurring cognitive traps, like abandoning valid approaches too early or overlooking problem constraints. Over time, it aggregates data across all your logged sessions to generate a prioritized study plan that highlights your persistent topic weaknesses and proven strengths.

When I showed it to Ved, he was stunned: "Brochacho we're actually using this for our contest prep now."

Demo

Below is the full demo of the project, you may have to wait good 2 sec for it to load!

Code

prathamanvekar / cpmeta

Feeding your chaotic scratchpad notes to a local LLM so it can professionally diagnose your skill issues.

cpmeta 🧠

A thinking-process analyzer for competitive programmers. Takes raw, timestamped scratchpad notes from your practice sessions and turns them into actionable algorithmic insights, stall root causes, and targeted study plans using local LLMs.

Why cpmeta?

When solving LeetCode or Codeforces problems, developers often keep scratchpad notes:

0:00 - read problem statement. contiguous subarrays with k distinct elements.
0:03 - thought about standard sliding window with two pointers.
0:07 - stuck: how to count *exact* k without missing smaller left bounds?
0:15 - breakthrough: exact(k) = atMost(k) - atMost(k - 1)
0:22 - tested edge cases and submitted -> Accepted.

These notes capture your real-time cognitive blindspots, but usually get discarded. cpmeta parses your notes alongside the problem statement with a local LLM to figure out:

Where and why you got stuck (root cause analysis with exact durations).
Behavioral patterns (e.g. abandoning correct approaches too early, missing invariants).
Cumulative weaknesses…

View on GitHub

How I Built It

The engine is powered by Ollama running open-weight coding models like qwen2.5-coder:7b locally. It connects directly to the local Ollama instance on port 11434, while also supporting remote endpoints via Cloudflare tunnels or Ngrok when needed.

To make local models output consistent coaching data instead of unstructured chat, we constrain inference to a strict JSON schema and pair it with a multi-stage parser that heals trailing truncations:

# Querying local Ollama with JSON enforcement and defensive extraction
response = requests.post(
    f"{base_url}/api/generate",
    json={
        "model": model_name,
        "system": SINGLE_PROBLEM_SYSTEM_PROMPT,
        "prompt": user_prompt,
        "stream": False,
        "format": "json",
        "options": {"temperature": 0.2, "top_p": 0.9},
    },
    timeout=120,
)
analysis = extract_clean_json(response.json().get("response", ""))

The interface is built with Python and Streamlit.

All sessions, raw notes, and coach metrics persist in an embedded SQLite database (cpmeta.db). For longitudinal synthesis, a custom aggregator condenses weeks of practice history into high-signal JSON summaries that fit comfortably within local LLM context windows without blowing past token limits. To ensure stable UI rendering, a multi-stage parser sanitizes raw model responses and auto-recovers valid JSON schemas.

Why Does Open Innovation Matter?

Competitive programmers solve hundreds of problems every month. Paying metered cloud API bills for every quick thought note or practice session is unrealistic for students and independent builders. Open-weight models running on consumer hardware cost nothing to run, making deliberate practice truly unlimited.

Problem-solving scratchpads are also raw, unfiltered mental dumps. Keeping inference entirely on-device ensures that no personal notes, contest thoughts, or thinking habits ever leave the machine.

Finally, practice happens anywhere - on commutes, in libraries with restrictive firewalls, or during offline study sprints. Because cpmeta relies on local Ollama inference, the entire diagnostic suite works seamlessly without internet access, accounts, or third-party API dependencies.

My Agent Session

Building cpmeta: Local AI Metacognition Engine for Competitive Programming

You

Build cpmeta: a Competitive Programming Metacognition Notes Analyzer using local Ollama LLMs, Streamlit, and SQLite.

Agent

Designed the Minimal Neobrutalist UI, implemented Ollama API client with defensive multi-stage JSON parsing, SQLite schema and session seeder, single-problem ICPC coach analysis, and cross-session longitudinal pattern synthesis.

You

Fix the card rendering issue where HTML tags were displaying as raw text inside divs, rename the project to cpmeta across all files, and create clean gitignore and documentation.

Agent

Identified CommonMark 4-space indentation bug where markdown engines treat indented HTML as code blocks. Implemented robust line-stripping render_html helper, refactored dynamic item templates, renamed all branding and database schema to cpmeta with automatic migration, and created production-ready .gitignore and minimal README.

Prize Categories

Render