DEV Community: Muhammad Zulqarnain

How I Structure Every Full Stack Project in 2025

Muhammad Zulqarnain — Wed, 20 May 2026 06:00:00 +0000

I build a lot of full-stack projects. The first 50 lines of code set the pattern for the next 50,000.

Here's the structure I use for every project. Opinionated. Works.

The Monorepo Structure

project/
├── packages/
│   ├── web/                    # Next.js frontend
│   │   └── src/
│   │       ├── app/
│   │       ├── components/
│   │       ├── hooks/
│   │       └── lib/
│   ├── api/                    # Node.js backend
│   │   └── src/
│   │       ├── routes/
│   │       ├── controllers/
│   │       ├── services/
│   │       ├── middleware/
│   │       └── models/
│   └── shared/                 # Shared TypeScript types
│       └── src/
│           ├── types.ts
│           ├── constants.ts
│           └── validators.ts
├── docker-compose.yml
├── turbo.json
└── package.json

Root package.json

{
  "workspaces": ["packages/*"],
  "scripts": {
    "dev": "turbo run dev --parallel",
    "build": "turbo run build",
    "test": "turbo run test"
  }
}

Environment Management

// packages/shared/src/env.ts
const requiredEnvVars = ['DATABASE_URL', 'JWT_SECRET', 'REDIS_URL'];

export const env = {
  databaseUrl: process.env.DATABASE_URL!,
  jwtSecret: process.env.JWT_SECRET!,
  redisUrl: process.env.REDIS_URL!,
};

requiredEnvVars.forEach((key) => {
  if (!process.env[key]) throw new Error(`Missing required env var: ${key}`);
});

Type Sharing

// packages/shared/src/types.ts
export type User = { id: string; email: string; name: string; createdAt: Date };
export type ApiResponse<T> = { success: boolean; data?: T; error?: string };

Backend:

import { User, ApiResponse } from '@shared/types';
router.post('/users', async (req, res) => {
  const user: User = await db.users.create(req.body);
  res.json<ApiResponse<User>>({ success: true, data: user });
});

Frontend uses same types — no mismatches, no runtime surprises.

Docker for Local Dev

version: '3'
services:
  postgres:
    image: postgres:16
    environment:
      POSTGRES_USER: dev
      POSTGRES_PASSWORD: dev
      POSTGRES_DB: mydb
    ports: ["5432:5432"]
  redis:
    image: redis:7
    ports: ["6379:6379"]

docker-compose up -d and the environment is ready.

Testing Structure

tests/
├── unit/
│   └── services/
├── integration/
│   └── routes/
└── fixtures/

CI/CD (GitHub Actions)

jobs:
  test:
    runs-on: ubuntu-latest
    services:
      postgres:
        image: postgres:16
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with: { node-version: '20' }
      - run: npm ci
      - run: npm run lint
      - run: npm run test
      - run: npm run build

Why This Stack

Monorepo: One repo, one CI/CD, type sharing, easier refactoring
Three packages: web deploys to Vercel, api deploys anywhere, shared is the contract between them
TypeScript everywhere: Catches errors at build time, not runtime
Turbo: Parallel builds, smart caching
Docker locally: Dev matches prod

Day 1 setup: monorepo + docker + shared types + CI/CD + 5 integration tests. Costs one week. Saves months.

zunain.com

Building Inference-Optimized Emaibeddings: Why Smaller Models Beat Bigger Ones

Muhammad Zulqarnain — Sun, 17 May 2026 20:57:34 +0000

How to Build a RAG Evaluation Framework That Catches Real Problems

Muhammad Zulqarnain — Sun, 17 May 2026 19:38:58 +0000

Six months into running a production RAG system, I had a problem: my users kept complaining about wrong answers, but my evaluation metrics looked fine. Retrieval accuracy: 87%. User satisfaction: 82%. Everything looked good on paper.

Then I sat with users for a week and watched them interact with the system.

The real problems were invisible to my metrics.

The Metrics You're Probably Using (And Why They're Not Enough)

Most RAG evaluation setups check:

Retrieval recall: "Did we get the relevant documents?"
Answer faithfulness: "Is the answer based on the retrieved content?"
Answer relevance: "Does the answer address the question?"

These are fine starting points. But they miss the failure modes that actually hurt users.

The Failure Modes Nobody Talks About

1. Temporal Confusion

Your system retrieved a correct document from 8 months ago. The policy changed 3 months ago. Your answer is confidently wrong.

Standard metrics: pass. Users: furious.

Fix: Track document age in your evaluation. Flag answers based on documents older than 30 days for manual review. Update your vector store on a schedule.

2. Partial Retrieval

User asks a compound question: "What are the refund limits for premium vs standard accounts?"

Your system retrieves the premium refund policy perfectly. Misses the standard account policy entirely.

Your retrieval recall: 50%. Your answer: half wrong. Your metrics: don't catch this.

Fix: Decompose compound questions before evaluation. Each sub-question needs its own retrieval check.

3. Confidence Calibration

Your model says "Based on the policy, the limit is $500." The document actually says "approximately $400-600 depending on account history."

The model collapsed a range into a specific number. Confidently. Wrongly.

Fix: Add a confidence calibration metric. Compare model confidence signals against actual document specificity.

4. The Hallucination in Context

This one is subtle. The document says X. The model retrieves it correctly. But the model's response adds context from its training data that contradicts X.

Faithfulness metrics typically catch obvious hallucinations. They miss when the model supplements retrieved content with training data.

Fix: Run a "context sufficiency" check. If the retrieved documents don't fully answer the question, flag it rather than letting the model fill gaps.

Building the Evaluation Framework

Here's what actually works in production:

Layer 1: Automated Metrics (Fast, Cheap)

Retrieval recall against a golden dataset
BERTScore for semantic similarity
Answer length relative to question complexity
Response time (slow = bad UX = user churn)

Cost: ~$0.002 per query to evaluate

Layer 2: LLM-as-Judge (Medium Cost, Medium Accuracy)

Have GPT-4 evaluate faithfulness and relevance
Check for temporal issues by passing document metadata
Flag compound question handling

Cost: ~$0.08 per query to evaluate (use on 10% of traffic)

Layer 3: Human Review (Expensive, Ground Truth)

Weekly review of 50 random flagged queries
Monthly review of edge cases identified by Layer 2
Quarterly user satisfaction interviews

Cost: $800-2,000/month in person-hours

The Feedback Loop That Matters

Every failed evaluation goes into a "failure taxonomy." Not just "wrong answer" — but why it was wrong.

After 3 months of this taxonomy, our top failure modes were:

Temporal confusion: 34% of failures
Compound question partial retrieval: 28%
Confidence miscalibration: 19%
Hallucination in context: 14%
Other: 5%

This told us exactly where to invest engineering time.

The Numbers

Before framework: Users reported wrong answers 3-4 times per week.

After 3 months with the framework:

Temporal failures: down 71%
Compound question failures: down 58%
Overall wrong answer reports: down 64%

The evaluation framework cost us 40 engineering hours to build and $800/month to run. The reduction in support tickets saved $3,000/month.

What to Build First

If you're starting from scratch: build Layer 1 first. Get automated metrics running on every query. Create a golden dataset of 200 question-answer pairs that represent your most common use cases.

Run that for 30 days. The failure patterns will tell you what to build next.

Don't let perfect be the enemy of good. An imperfect evaluation framework running consistently beats a perfect one you're still designing.

Build the thing. Measure. Iterate.

Embedding Quality Over Speed: Why Your Search Results Are Mediocre

Muhammad Zulqarnain — Sun, 17 May 2026 05:28:42 +0000

The Node.js Patterns I Wish I Knew 3 Years Ago

Muhammad Zulqarnain — Fri, 08 May 2026 06:00:00 +0000

Node.js is amazing for shipping fast. It's also amazing at letting you ship broken things fast.

I've built a lot of broken things. Here are the patterns that make Node.js systems not break.

Pattern 1: EventEmitter for Internal Pub/Sub

import { EventEmitter } from 'events';
const orderEvents = new EventEmitter();

async function createOrder(userId, items) {
  const order = await db.orders.create({ userId, items });
  orderEvents.emit('order:created', order);
  return order;
}

orderEvents.on('order:created', async (order) => {
  await sendConfirmationEmail(order).catch(err => logger.error('Email failed', err));
});

orderEvents.on('order:created', async (order) => {
  await logAnalytics('order_created', order).catch(err => logger.error('Analytics failed', err));
});

Result: Order creation returns in 50ms. Side effects async. If email fails, order still succeeded.

Pattern 2: Worker Threads for CPU-Bound Tasks

import { Worker } from 'worker_threads';
const workers = Array(os.cpus().length).fill(null).map(() => new Worker('./worker.js'));
let currentWorker = 0;

function processImage(imageBuffer) {
  return new Promise((resolve, reject) => {
    const worker = workers[currentWorker];
    currentWorker = (currentWorker + 1) % workers.length;
    const timer = setTimeout(() => reject(new Error('Worker timeout')), 30000);
    worker.once('message', (result) => { clearTimeout(timer); resolve(result); });
    worker.once('error', reject);
    worker.postMessage({ imageBuffer });
  });
}

Result: Heavy computation doesn't block the main thread.

Pattern 3: Domain-Specific Error Classes

class AppError extends Error {
  constructor(message, statusCode, code) {
    super(message);
    this.statusCode = statusCode;
    this.code = code;
  }
}

class NotFoundError extends AppError {
  constructor(message) { super(message, 404, 'NOT_FOUND'); }
}

class ValidationError extends AppError {
  constructor(message) { super(message, 400, 'VALIDATION_ERROR'); }
}

app.get('/users/:id', async (req, res, next) => {
  try {
    const user = await getUser(req.params.id);
    res.json(user);
  } catch (err) {
    if (err instanceof AppError) {
      res.status(err.statusCode).json({ error: err.message });
    } else {
      res.status(500).json({ error: 'Internal server error' });
    }
  }
});

Pattern 4: Connection Pooling

const pool = new pg.Pool({
  max: 20,
  idleTimeoutMillis: 30000,
  connectionTimeoutMillis: 2000,
});

async function getUser(id) {
  const result = await pool.query('SELECT * FROM users WHERE id = $1', [id]);
  return result.rows[0];
}

Pattern 5: Circuit Breaker

class CircuitBreaker {
  constructor(fn, options = {}) {
    this.fn = fn;
    this.failureThreshold = options.failureThreshold || 5;
    this.resetTimeout = options.resetTimeout || 60000;
    this.state = 'CLOSED';
    this.failureCount = 0;
    this.lastFailureTime = null;
  }

  async call(...args) {
    if (this.state === 'OPEN') {
      if (Date.now() - this.lastFailureTime > this.resetTimeout) {
        this.state = 'HALF_OPEN';
      } else {
        throw new Error('Circuit breaker is OPEN');
      }
    }
    try {
      const result = await this.fn(...args);
      this.failureCount = 0;
      this.state = 'CLOSED';
      return result;
    } catch (err) {
      this.failureCount++;
      this.lastFailureTime = Date.now();
      if (this.failureCount >= this.failureThreshold) this.state = 'OPEN';
      throw err;
    }
  }
}

Pattern 6: Graceful Shutdown

async function gracefulShutdown() {
  server.close(async () => {
    logger.info('Server closed');
  });
  const forceShutdown = setTimeout(() => process.exit(1), 30000);
  try {
    await db.close();
    await redis.quit();
    clearTimeout(forceShutdown);
    process.exit(0);
  } catch (err) {
    process.exit(1);
  }
}
process.on('SIGTERM', gracefulShutdown);
process.on('SIGINT', gracefulShutdown);

The Meta Pattern: Async Error Handling

function asyncHandler(fn) {
  return (req, res, next) => Promise.resolve(fn(req, res, next)).catch(next);
}

app.get('/users/:id', asyncHandler(async (req, res) => {
  const user = await db.getUser(req.params.id);
  res.json(user);
}));

app.use((err, req, res, next) => {
  logger.error('Unhandled error', err);
  res.status(500).json({ error: 'Internal server error' });
});

These aren't fancy patterns. They're the ones that keep systems running at 3am without pages.

zunain.com

The Node.js Patterns I Wish I Knew 3 Years Ago

Muhammad Zulqarnain — Fri, 08 May 2026 05:06:33 +0000

Event loop mastery, streams, worker threads, graceful shutdown, and the design patterns that separate senior Node.js engineers from the rest.

Building Production-Ready RAG: A Complete Architecture Guide

Muhammad Zulqarnain — Mon, 27 Apr 2026 16:39:13 +0000

Building Production-Ready RAG: A Complete Architecture Guide

Most RAG tutorials show you how to stuff documents into a vector database and query them with an LLM. That gets you a demo. It does not get you a production system.

I've built RAG pipelines that serve real users. The gap between "it works in a notebook" and "it works at 2 AM when the on-call gets paged" is massive. This guide covers what actually matters when you ship RAG to production.

Why Most RAG Demos Fail

The standard RAG tutorial looks like this: load PDFs, chunk them, embed them, store in Pinecone, query with OpenAI. Five lines of LangChain. Done.

Then you deploy it and discover:

The chunking strategy matters more than the model. Bad chunks produce bad retrieval. Bad retrieval produces hallucinated answers. Users lose trust fast.

Latency kills user experience. A 6-second response time is fine in a demo. Real users expect sub-2 seconds. You need caching, streaming, and careful orchestration.

Cost scales faster than you expect. Every query hits an embedding model and an LLM. At 10K queries per day, your monthly bill will surprise you.

Evaluation is almost impossible without infrastructure. "Does this answer look right?" is not a test strategy.

Let's fix each of these.

The Ingestion Pipeline

Ingestion is where most RAG systems quietly break. Bad data in, bad answers out.

Document Processing

Before you chunk anything, clean your documents. Strip headers, footers, navigation elements, and boilerplate. Extract tables separately because they chunk terribly as plain text.

interface ProcessedDocument {
  id: string;
  content: string;
  metadata: {
    source: string;
    title: string;
    section: string;
    lastUpdated: Date;
    documentType: 'article' | 'api-doc' | 'tutorial' | 'faq';
  };
}

async function processDocument(raw: RawDocument): Promise<ProcessedDocument[]> {
  // Split by semantic sections, not arbitrary character counts
  const sections = extractSections(raw.html);

  return sections.map((section, index) => ({
    id: `${raw.id}-section-${index}`,
    content: cleanMarkdown(section.content),
    metadata: {
      source: raw.url,
      title: raw.title,
      section: section.heading || `Section ${index + 1}`,
      lastUpdated: raw.updatedAt,
      documentType: classifyDocument(raw),
    },
  }));
}

Chunking Strategy

Forget fixed-size chunking. It splits sentences mid-thought and produces chunks that lack context.

Use semantic chunking: split on paragraph boundaries, heading boundaries, and logical breaks. Each chunk should be a self-contained thought that makes sense in isolation.

function semanticChunk(
  document: ProcessedDocument,
  options: { maxTokens: number; overlap: number }
): Chunk[] {
  const paragraphs = document.content.split(/\n\n+/);
  const chunks: Chunk[] = [];
  let currentChunk: string[] = [];
  let currentTokens = 0;

  for (const paragraph of paragraphs) {
    const paragraphTokens = estimateTokens(paragraph);

    if (currentTokens + paragraphTokens > options.maxTokens && currentChunk.length > 0) {
      chunks.push({
        content: currentChunk.join('\n\n'),
        tokenCount: currentTokens,
        metadata: document.metadata,
      });

      // Keep last paragraph for overlap/context continuity
      const overlapParagraphs = currentChunk.slice(-1);
      currentChunk = [...overlapParagraphs, paragraph];
      currentTokens = estimateTokens(overlapParagraphs.join('\n\n')) + paragraphTokens;
    } else {
      currentChunk.push(paragraph);
      currentTokens += paragraphTokens;
    }
  }

  if (currentChunk.length > 0) {
    chunks.push({
      content: currentChunk.join('\n\n'),
      tokenCount: currentTokens,
      metadata: document.metadata,
    });
  }

  return chunks;
}

Target chunk size: 200-500 tokens. Smaller chunks give more precise retrieval. Larger chunks give more context. I've found 300-400 tokens to be the sweet spot for most documentation use cases.

Embedding Models

For production, I use OpenAI's text-embedding-3-small for most cases. It's cheap ($0.02 per 1M tokens), fast, and the quality is good enough for 90% of use cases.

If you need better retrieval quality and can afford the latency, text-embedding-3-large with dimension reduction to 1024 gives excellent results at a reasonable cost.

async function embedChunks(chunks: Chunk[]): Promise<EmbeddedChunk[]> {
  // Batch embeddings for efficiency (max 2048 per request)
  const batchSize = 2048;
  const results: EmbeddedChunk[] = [];

  for (let i = 0; i < chunks.length; i += batchSize) {
    const batch = chunks.slice(i, i + batchSize);
    const response = await openai.embeddings.create({
      model: 'text-embedding-3-small',
      input: batch.map(c => c.content),
    });

    for (let j = 0; j < batch.length; j++) {
      results.push({
        ...batch[j],
        embedding: response.data[j].embedding,
      });
    }
  }

  return results;
}

The Retrieval Layer

Vector similarity search alone is not enough. You need hybrid search and re-ranking.

Hybrid Search with pgvector

If you're already running PostgreSQL, use pgvector. One less service to manage. Combine vector similarity with full-text search for hybrid retrieval.

async function hybridSearch(
  query: string,
  queryEmbedding: number[],
  options: { limit: number; minScore: number }
): Promise<SearchResult[]> {
  const results = await db.query(`
    WITH vector_results AS (
      SELECT id, content, metadata,
             1 - (embedding <=> $1::vector) AS vector_score
      FROM chunks
      WHERE 1 - (embedding <=> $1::vector) > $3
      ORDER BY embedding <=> $1::vector
      LIMIT $2 * 2
    ),
    text_results AS (
      SELECT id, content, metadata,
             ts_rank(search_vector, plainto_tsquery('english', $4)) AS text_score
      FROM chunks
      WHERE search_vector @@ plainto_tsquery('english', $4)
      ORDER BY text_score DESC
      LIMIT $2 * 2
    )
    SELECT COALESCE(v.id, t.id) AS id,
           COALESCE(v.content, t.content) AS content,
           COALESCE(v.metadata, t.metadata) AS metadata,
           COALESCE(v.vector_score, 0) * 0.7 + COALESCE(t.text_score, 0) * 0.3 AS combined_score
    FROM vector_results v
    FULL OUTER JOIN text_results t ON v.id = t.id
    ORDER BY combined_score DESC
    LIMIT $2
  `, [queryEmbedding, options.limit, options.minScore, query]);

  return results.rows;
}

The 0.7/0.3 weighting between vector and text scores works well as a starting point. Tune it based on your evaluation results.

Re-ranking

After retrieval, re-rank the results. A cross-encoder model gives much better relevance scores than cosine similarity alone. Cohere's re-rank API is the easiest option.

async function rerankResults(
  query: string,
  results: SearchResult[],
  topK: number = 5
): Promise<SearchResult[]> {
  const response = await cohere.rerank({
    model: 'rerank-english-v3.0',
    query,
    documents: results.map(r => r.content),
    topN: topK,
  });

  return response.results.map(r => ({
    ...results[r.index],
    relevanceScore: r.relevanceScore,
  }));
}

This step alone improved our answer quality by about 15% in our evaluations.

The Generation Layer

Prompt Design

Your system prompt should be specific about the task, the constraints, and the expected format. Vague prompts produce vague answers.

function buildPrompt(query: string, context: SearchResult[]): string {
  const contextBlock = context
    .map((c, i) => `[Source ${i + 1}: ${c.metadata.title}]\n${c.content}`)
    .join('\n\n---\n\n');

  return `You are a technical documentation assistant. Answer the user's question using ONLY the provided context. If the context doesn't contain enough information to answer fully, say so clearly.

Rules:
- Cite sources using [Source N] format
- If you're unsure, say "I'm not confident about this" rather than guessing
- Keep answers concise and technical
- Include code examples when relevant

Context:
${contextBlock}

Question: ${query}`;
}

Streaming Responses

Users should see the answer forming in real time. Stream tokens as they arrive.

async function* generateAnswer(
  query: string,
  context: SearchResult[]
): AsyncGenerator<string> {
  const prompt = buildPrompt(query, context);

  const stream = await openai.chat.completions.create({
    model: 'gpt-4o-mini',
    messages: [{ role: 'user', content: prompt }],
    stream: true,
    temperature: 0.1, // Low temperature for factual answers
    max_tokens: 1000,
  });

  for await (const chunk of stream) {
    const content = chunk.choices[0]?.delta?.content;
    if (content) yield content;
  }
}

Hallucination Prevention

Three layers of defense:

Confidence scoring: If no retrieved chunk scores above 0.75 relevance, don't generate an answer. Tell the user you couldn't find relevant information.
Source grounding: The prompt explicitly limits answers to the provided context. This doesn't eliminate hallucinations but reduces them significantly.
Citation verification: Post-process the response. If the model cites [Source 3] but Source 3 doesn't contain the claimed information, flag it.

function validateCitations(
  answer: string,
  sources: SearchResult[]
): ValidationResult {
  const citationPattern = /\[Source (\d+)\]/g;
  const citations = [...answer.matchAll(citationPattern)];
  const issues: string[] = [];

  for (const citation of citations) {
    const sourceIndex = parseInt(citation[1]) - 1;
    if (sourceIndex >= sources.length) {
      issues.push(`Citation [Source ${citation[1]}] references non-existent source`);
    }
  }

  // Check if answer contains claims without any citations
  const sentences = answer.split(/[.!?]+/).filter(s => s.trim().length > 20);
  const uncitedClaims = sentences.filter(
    s => !s.includes('[Source') && looksLikeFactualClaim(s)
  );

  if (uncitedClaims.length > sentences.length * 0.5) {
    issues.push('More than half of factual claims lack citations');
  }

  return {
    valid: issues.length === 0,
    issues,
    citationCount: citations.length,
  };
}

Evaluation

You cannot improve what you cannot measure. Build an evaluation pipeline from day one.

Test Set Construction

Create a test set of 50-100 question/answer pairs. Include edge cases: questions with no answer in the docs, ambiguous questions, questions requiring information from multiple sources.

interface EvalCase {
  query: string;
  expectedAnswer: string;
  relevantDocIds: string[];
  difficulty: 'easy' | 'medium' | 'hard';
  category: 'factual' | 'procedural' | 'conceptual' | 'unanswerable';
}

async function evaluateRAG(testCases: EvalCase[]): Promise<EvalReport> {
  const results = await Promise.all(
    testCases.map(async (tc) => {
      const retrieved = await hybridSearch(tc.query, await embed(tc.query), { limit: 5, minScore: 0.3 });
      const answer = await generateFullAnswer(tc.query, retrieved);

      return {
        query: tc.query,
        retrievalRecall: calculateRecall(retrieved, tc.relevantDocIds),
        answerRelevance: await scoreRelevance(answer, tc.expectedAnswer),
        latencyMs: answer.latencyMs,
        tokenCost: answer.totalTokens,
      };
    })
  );

  return {
    avgRetrievalRecall: avg(results.map(r => r.retrievalRecall)),
    avgAnswerRelevance: avg(results.map(r => r.answerRelevance)),
    avgLatencyMs: avg(results.map(r => r.latencyMs)),
    totalCost: sum(results.map(r => r.tokenCost)),
    results,
  };
}

Key Metrics

Track these in production:

Retrieval recall@5: What percentage of relevant documents appear in the top 5 results? Target: >85%.
Answer relevance: LLM-as-judge scoring against expected answers. Target: >4.0/5.0.
Latency P95: Time from query to first streamed token. Target: <2 seconds.
User feedback rate: Thumbs up/down ratio. Target: >80% positive.
Hallucination rate: Flagged by citation validation. Target: <5%.

Cost Optimization

At scale, RAG costs add up fast. Three strategies that made the biggest difference for us:

Semantic Caching

Cache answers for semantically similar queries. If someone asks "how do I deploy to production?" and another person asks "what's the deployment process?", they should get the same cached answer.

async function getCachedOrGenerate(query: string): Promise<RAGResponse> {
  const queryEmbedding = await embed(query);

  // Check cache for semantically similar queries
  const cached = await db.query(`
    SELECT response, query
    FROM response_cache
    WHERE 1 - (query_embedding <=> $1::vector) > 0.95
    AND created_at > NOW() - INTERVAL '24 hours'
    ORDER BY query_embedding <=> $1::vector
    LIMIT 1
  `, [queryEmbedding]);

  if (cached.rows.length > 0) {
    return { ...cached.rows[0].response, fromCache: true };
  }

  // Generate fresh response
  const response = await generateRAGResponse(query, queryEmbedding);

  // Cache it
  await db.query(
    `INSERT INTO response_cache (query, query_embedding, response) VALUES ($1, $2, $3)`,
    [query, queryEmbedding, response]
  );

  return response;
}

This alone cut our LLM costs by about 40%.

Model Selection

Use gpt-4o-mini for most queries. It's 15x cheaper than gpt-4o and handles straightforward documentation questions just fine. Route complex queries (multi-hop reasoning, comparison questions) to gpt-4o.

Embedding Cost

Pre-compute and store embeddings. Never re-embed a document that hasn't changed. Use a hash of the content to detect changes during re-indexing.

Real-World Performance Numbers

From our production system serving ~5K queries per day:

Average retrieval latency: 45ms (pgvector with HNSW index)
Average generation latency: 1.2s (gpt-4o-mini, streaming)
Time to first token: 380ms
Retrieval recall@5: 88%
User satisfaction (thumbs up): 84%
Monthly cost: ~$280 (embeddings + LLM + infrastructure)
Cache hit rate: 38%

These numbers are realistic for a mid-scale documentation search system. Your numbers will vary based on document corpus size, query complexity, and model choice.

Wrapping Up

Production RAG is not about picking the right model. It's about building a system with good data pipelines, smart retrieval, proper evaluation, and cost controls.

Start with the simplest version that works. Measure everything. Improve the pieces that matter most based on your metrics.

The code examples in this article are simplified but functional. Adapt them to your stack and your data.

If you're building RAG right now, I'd love to hear what challenges you're hitting. Drop a comment or reach out.

Stop Building AI Wrappers. Start Building AI Products.

Muhammad Zulqarnain — Fri, 24 Apr 2026 06:00:00 +0000

Everyone and their brother is building an AI wrapper right now.

You know the type: "I built a ChatGPT UI with a prompt." "I wrapped Claude and added a database." "I made an AI that does X by just calling the API."

These are not products. They're proofs of concept that will be dead in 6 months.

Here's the difference between an AI wrapper and an AI product, and why it matters.

What's an AI Wrapper?

Wrapper = LLM API + UI/UX

"Prompt builder for GPT" — still just calling GPT
"AI email writer" — just Claude, but for emails
"Chat interface that remembers context" — ChatGPT but persistent

These have zero defensibility. When OpenAI releases ChatGPT Plus with the same features, your wrapper dies.

What's an AI Product?

AI Product = (Proprietary Data + Specialized Model + Workflow Integration + User Loop)

Cursor: Code editor that understands your specific codebase. Remove AI, product breaks.
Perplexity: Web search + AI reasoning over sources + citations. The synthesis is the product.
Replit Agent: AI that executes code, sees errors, iterates. The feedback loop is the product.

The 5 Differences

1. Proprietary Data

Wrapper: Uses public information
Product: Has a data moat

2. Specialized vs. General

Wrapper: Uses a general LLM
Product: Fine-tunes for specific task

3. Workflow Integration

Wrapper: Standalone tool
Product: Integrated into how users work

4. Feedback Loop

Wrapper: Fire and forget
Product: Learns from user behavior

5. Defensibility

Wrapper: Dead when the LLM vendor ships the same feature
Product: Moat that gets wider with users

How to Build an AI Product

Start with a specific, narrow problem
Identify your data advantage — if the answer is "none," you're building a wrapper
Build the feedback loop from day 1 — capture accept/reject/edit signals
Integrate into user workflow
Fine-tune or specialize your model
Think about the data flywheel

The Investment Thesis

Wrapper: "We built a UI for Claude." — VCs pass
Product: "We trained a model on your domain data and it handles 60% of your support tickets." — VCs listen

Build for defensibility. Not demo impressiveness.

More at zunain.com.

From $200/Month to Free: Running OpenClaw with Local AI Models

Muhammad Zulqarnain — Sun, 19 Apr 2026 18:15:45 +0000

This is a submission for the OpenClaw Writing Challenge

The Problem: AI Assistant Costs Are Skyrocketing

If you're running OpenClaw with cloud-hosted LLMs like Claude or GPT-4, you know the pain. Premium API access can easily cost $200/month or more, and that's assuming moderate usage. For developers, founders, or anyone automating workflows extensively, those costs compound fast.

But here's the thing: OpenClaw doesn't require cloud AI. You can run it entirely locally with open-source models—and in many cases, get comparable results for $0/month in API fees.

This guide walks through three deployment tiers, from completely free to budget-friendly, showing you how to cut your OpenClaw costs to zero while maintaining functionality.

Understanding Your Options

Tier 1: Completely Free (Ollama + Local Models)

Cost: $0/month

Hardware: Any spare laptop/desktop with 8GB+ RAM

Best For: Personal automation, learning, experimentation

How it works:

Ollama lets you run powerful open-source models like Qwen 2.5 (7B/14B), Llama 3, or Mistral locally. These models are surprisingly capable for most automation tasks—code generation, data extraction, text summarization, and workflow orchestration.

OpenClaw connects to Ollama as a model provider, treating your local instance like any cloud API.

Setup Steps:

Install Ollama (Mac/Linux/Windows):

   curl -fsSL https://ollama.com/install.sh | sh

Pull a capable model:

   ollama pull qwen2.5:14b
   # or for lower-end hardware:
   ollama pull qwen2.5:7b

Configure OpenClaw:

In your OpenClaw settings, switch the model provider to ollama and point it to http://localhost:11434.
Test your setup:

Create a simple skill (e.g., "Summarize my emails") and verify it works with your local model.

Tradeoffs:

Your device needs to stay on 24/7 for skills to run
Slightly slower inference than cloud APIs
Smaller context windows (typically 8K-32K tokens vs 128K+ for cloud models)

Real savings: If you were paying $200/month for Claude API access, that's $2,400/year saved.

Tier 2: Budget Cloud ($10-30/month)

Cost: $10-30/month

Hardware: None (cloud-hosted)

Best For: Production workflows, team usage, 24/7 availability

How it works:

If running a local device 24/7 isn't practical, you can deploy Ollama on a cheap VPS (Virtual Private Server) and point OpenClaw to it remotely.

Alternatively, use budget-friendly cloud APIs like:

Minimax API: ~$0.001 per 1K tokens (~$20-30/month for heavy use)
Groq: Fast inference, generous free tier
Together AI: Competitive pricing on open models

VPS Setup Example (DigitalOcean/Hetzner):

Spin up a VPS (~$10-15/month for 8GB RAM):

   # SSH into your VPS
   ssh user@your-vps-ip

Install Ollama:

   curl -fsSL https://ollama.com/install.sh | sh
   ollama pull qwen2.5:14b

Expose Ollama (use a reverse proxy like ngrok or Tailscale for secure access):

   ollama serve --host 0.0.0.0

Point OpenClaw to http://your-vps-ip:11434

Tradeoffs:

Small monthly cost but still 10x cheaper than Claude Max
Requires basic VPS management skills
Latency depends on VPS location

Real savings: Instead of $200/month on cloud APIs, you're paying $15-30/month—saving $170-185/month or $2,040-2,220/year.

Tier 3: Hybrid Approach (Best of Both Worlds)

Cost: Variable ($0-50/month depending on usage)

Strategy: Use local models for routine tasks, cloud APIs for complex reasoning

How it works:

OpenClaw supports multiple model providers simultaneously. You can configure different skills to use different models:

Routine automation (email filtering, data extraction) → Ollama (free)
Complex reasoning (code review, strategic planning) → Claude/GPT-4 (pay-per-use)

This hybrid approach optimizes for both cost and capability.

Configuration Example:

skills:
  email_summarizer:
    model: ollama/qwen2.5:14b

  code_reviewer:
    model: anthropic/claude-3-opus

Real savings: If 80% of your tasks run locally and 20% use cloud APIs, you're looking at ~$40/month instead of $200—saving $160/month or $1,920/year.

Choosing the Right Model

Not all models are created equal. Here's what works well for OpenClaw:

Model	Size	Best For	Context Window
Qwen 2.5	7B-14B	General automation, coding	32K tokens
Llama 3.1	8B-70B	Reasoning, chat	128K tokens
Mistral	7B-22B	Fast inference, multilingual	32K tokens
DeepSeek Coder	6.7B	Code generation, debugging	16K tokens

For most users, Qwen 2.5 14B offers the best balance of capability and resource requirements.

Real-World Example: My 5-Agent Setup

I run 5 OpenClaw agents entirely on Ollama using a spare MacBook Air (16GB RAM):

Email Assistant: Filters, summarizes, drafts replies
Code Helper: Generates boilerplate, reviews PRs
Research Agent: Monitors RSS feeds, summarizes articles
Data Extractor: Pulls structured data from websites
Task Scheduler: Manages my Notion workspace

Total monthly cost: $0 (minus electricity, ~$2-3/month)

Previous cloud API cost: ~$180/month

Annual savings: $2,160

The MacBook runs 24/7, but I was going to keep it plugged in anyway. The agents paid for themselves in week one.

Getting Started: Your First Local OpenClaw Agent

Here's a step-by-step walkthrough to create your first cost-free OpenClaw skill:

1. Install Prerequisites

# Install Ollama
curl -fsSL https://ollama.com/install.sh | sh

# Pull a model
ollama pull qwen2.5:14b

# Verify it's running
ollama list

2. Configure OpenClaw

In your OpenClaw instance:

Navigate to Settings → Model Providers
Add a new provider: Ollama
Set endpoint: http://localhost:11434
Test connection

3. Create a Simple Skill

Let's build an Email Summarizer:

# Example skill configuration
name: "Daily Email Summary"
trigger: "cron: 0 8 * * *"  # Run at 8 AM daily
model: "ollama/qwen2.5:14b"

prompt: |
  Summarize these emails into a concise bullet-point list.
  Focus on action items and key information.

  {email_content}

output_format: "markdown"
notification: "slack"

4. Test & Iterate

Run the skill manually first:

openclaw run email-summarizer --test

Once it works, let it run on schedule. Monitor performance and adjust the prompt as needed.

Tips for Optimizing Local Model Performance

Use quantized models: GGUF 4-bit quantization runs 2-3x faster with minimal quality loss
Batch requests: Process multiple items together to maximize throughput
Cache responses: For repetitive tasks, cache and reuse model outputs
Monitor resources: Use htop or Activity Monitor to track CPU/GPU usage
Upgrade RAM if needed: 16GB is the sweet spot for running 14B models comfortably

When Cloud APIs Still Make Sense

Local models aren't always the answer. Stick with cloud APIs when:

You need cutting-edge reasoning (GPT-4o, Claude Opus for complex tasks)
Context windows matter (analyzing 100K+ token documents)
Latency is critical (sub-second response times)
You don't have suitable hardware (less than 8GB RAM)

The hybrid approach (local for most tasks, cloud for special cases) often delivers the best ROI.

Conclusion: Take Control of Your AI Costs

OpenClaw's flexibility means you're not locked into expensive cloud APIs. Whether you go fully local with Ollama, deploy a budget VPS, or use a hybrid strategy, you can dramatically reduce costs without sacrificing functionality.

Key takeaways:

✅ Local models (Ollama + Qwen/Llama) work for 80%+ of automation tasks
✅ VPS deployment costs $10-30/month vs $200+ for cloud APIs
✅ Hybrid approach balances cost and capability
✅ Annual savings of $1,920-2,400 are realistic

If you're spending over $100/month on AI API access, it's time to evaluate local options. OpenClaw makes it easy.

Resources

OpenClaw Docs: docs.openclaw.ai
Ollama: ollama.com
Qwen 2.5: huggingface.co/Qwen
Budget VPS Providers: DigitalOcean, Hetzner, Vultr

Have you switched to local models for OpenClaw? What's your setup? Drop a comment below!

EcoMind: AI Carbon Footprint Analyzer for Earth Day 2026

Muhammad Zulqarnain — Sun, 19 Apr 2026 16:05:04 +0000

What I Built

EcoMind is an AI-powered carbon footprint analyzer that helps individuals understand their environmental impact and receive personalized, actionable recommendations to reduce their carbon emissions.

Live Demo: https://ecomind-coral.vercel.app
GitHub: https://github.com/mzunain/ecomind

Demo

EcoMind provides:

🌍 Instant carbon footprint analysis based on your lifestyle
📊 Sustainability score (1-10) with detailed breakdown
💡 Top 5 personalized reduction actions ranked by impact
🌱 Region-specific advice (optimized for Nordic countries)
⚡ Fast, beautiful dark theme UI

Journey

Why Google Gemini?

I chose Google Gemini for its structured JSON output capability, which is perfect for generating consistent, type-safe environmental data. The model excels at understanding complex lifestyle patterns and providing nuanced, regional advice.

Technical Implementation

Using Next.js 15 with the App Router and Google's @google/generative-ai SDK, I implemented:

Structured Schema: Defined a TypeScript schema for carbon analysis
Regional Context: Added location-aware recommendations
Difficulty Scoring: Categorized actions by implementation difficulty
Premium UI: Modern glassmorphism design with Tailwind CSS

Challenges Faced

Model naming: Gemini API model versions required experimentation
Structured output validation: Ensuring consistent JSON responses
Regional accuracy: Balancing global averages with local context (e.g., Finland's clean energy grid)

Category Submission

Best Use of Google Gemini - EcoMind showcases Gemini's strength in structured data generation and contextual understanding for environmental impact analysis.

Additional Prize Categories

N/A

EcoMind: AI-Powered Carbon Footprint Analyzer Using Google Gemini

Muhammad Zulqarnain — Sun, 19 Apr 2026 15:31:31 +0000

This is a submission for Weekend Challenge: Earth Day Edition

What I Built

EcoMind is a Next.js web application powered by the Google Gemini API that helps users understand and reduce their personal carbon footprint. You describe your daily habits — commute, diet, home energy use, shopping patterns — and Gemini analyzes your lifestyle, estimates your CO₂ impact, and returns a personalized action plan with concrete steps ranked by impact.

The goal was to make environmental awareness feel personal and actionable rather than abstract. Most carbon calculators spit out a number and leave you wondering "so what?" EcoMind closes that gap by combining Gemini's reasoning capabilities with structured output parsing to give context-aware recommendations tailored to your actual situation.

Key features:

Conversational input: Describe your habits in plain language — no rigid forms
Gemini-powered analysis: Uses gemini-1.5-flash to parse lifestyle data, estimate emissions, and generate recommendations
Structured JSON output: Forced JSON schema response from Gemini for reliable rendering
Actionable breakdown: Each recommendation shows estimated CO₂ savings per year
Finland-aware context: Recommendations factor in northern European energy mix and public transit infrastructure

Demo

🔗 Live demo: https://ecomind-coral.vercel.app

You can also run it locally — instructions in the README.

Code

mzunain / ecomind

AI-powered carbon footprint analyzer using Google Gemini - DEV Weekend Challenge Earth Day Edition

EcoMind 🌍

AI-powered carbon footprint analyzer using Google Gemini

Built for the DEV Weekend Challenge: Earth Day Edition

Live Demo

🔗 https://ecomind-gamma.vercel.app

What It Does

EcoMind lets you describe your daily lifestyle in plain language and returns a personalised carbon footprint analysis powered by the Google Gemini API. No rigid forms — just describe your commute, diet, home heating, and travel habits, and get back:

Estimated annual CO₂ in kg
Sustainability score (1–10)
Top 5 reduction actions ranked by impact with CO₂ savings per year
Regional context — advice tailored to your location (Finland-aware by default)

Tech Stack

Next.js 14 (App Router) + TypeScript
Google Gemini API (gemini-1.5-flash) via @google/generative-ai
Structured JSON output using Gemini's responseSchema feature
Tailwind CSS for styling
Vercel for deployment

Getting Started

# Clone the repo
git clone https://github.com/mzunain/ecomind.git
cd ecomind
# Install dependencies
npm install

# Add your Gemini API key
echo '

…

View on GitHub

How I Built It

Tech Stack:

Next.js 14 (App Router) + TypeScript
Google Gemini API (gemini-1.5-flash) via @google/generative-ai SDK
Tailwind CSS for styling
Vercel for deployment

How Gemini is integrated:

The core of EcoMind is a single API route (/api/analyze) that takes a user's lifestyle description and feeds it to Gemini with a carefully engineered system prompt:

const model = genAI.getGenerativeModel({
  model: "gemini-1.5-flash",
  generationConfig: {
    responseMimeType: "application/json",
    responseSchema: ecoAnalysisSchema,
  },
});

const prompt = `
You are an environmental impact expert. Analyze the following lifestyle description
and return a structured carbon footprint assessment.
Lifestyle: ${userInput}
Provide: estimated annual CO2 in kg, top 5 reduction actions with CO2 savings,
and an overall sustainability score (1-10).
`;

const result = await model.generateContent(prompt);

I used Gemini's structured output / JSON schema enforcement feature to guarantee parseable responses — this was the most interesting technical decision. Instead of prompt-engineering my way to consistent JSON, I defined a Zod-like schema and passed it as responseSchema. This made the frontend rendering completely reliable.

Structured output schema:

const ecoAnalysisSchema = {
  type: "object",
  properties: {
    annualCO2Kg: { type: "number" },
    sustainabilityScore: { type: "number" },
    topActions: {
      type: "array",
      items: {
        type: "object",
        properties: {
          action: { type: "string" },
          annualSavingKg: { type: "number" },
          difficulty: { type: "string", enum: ["easy", "medium", "hard"] },
          category: { type: "string" },
        },
      },
    },
    summary: { type: "string" },
  },
};

Why this matters for Earth Day:
Carbon literacy is one of the biggest blockers to individual climate action. People don't act because the data feels distant. By making it conversational and immediate — type a few sentences about your week and get back a personalized plan — EcoMind lowers that barrier significantly.

I also added a Finland-specific context layer because I'm based in Turku and the sustainability advice for someone in the Nordics (excellent public transit, clean energy grid, cold climate heating costs) is genuinely different from generic global advice.

Prize Categories

Best Use of Google Gemini — Gemini is the core engine of EcoMind. The project uses:

gemini-1.5-flash for fast, cost-effective inference
Structured JSON output with response schema for reliable parsing
A multi-part system prompt that grounds Gemini in environmental science context
Dynamic prompt construction based on user input

5 Next.js Performance Patterns That Cut Our Load Time by 60%

Muhammad Zulqarnain — Mon, 13 Apr 2026 06:00:00 +0000

Performance is a feature.

At Quran.com, we serve 50 million users across 200+ countries on varying networks. A 1-second delay for users on slow connections meant people gave up. Every millisecond mattered.

Here are 5 patterns we implemented that cut load time by 60%. With code.

Pattern 1: Font Subsetting

The Problem: We served the full Amiri Arabic font (600KB) on every page. Users on 3G in Southeast Asia waited 4 seconds for the first render.

The Pattern: Subset fonts to only the Unicode ranges actually used on each page.

@font-face {
  font-family: 'Amiri';
  src: url('/fonts/amiri-subset.woff2') format('woff2');
  font-display: swap;
  unicode-range: U+0600-06FF, U+200C, U+200D;
}

The Result: Font size: 600KB → 48KB. Load time improvement: 40% on mobile.

pyftsubset Amiri-Regular.ttf --unicodes=0600-06FF,200C,200D --output-file=amiri-subset.woff2

Pattern 2: Incremental Static Regeneration (ISR)

The Problem: 114 surahs with 200+ translations. Full site generation: 45 minutes. Deploys blocked.

export async function getStaticProps(context) {
  const data = await fetchSurahData(context.params.surah);
  return {
    props: { data },
    revalidate: 3600,
  };
}

export async function getStaticPaths() {
  const paths = Array.from({ length: 114 }, (_, i) => ({
    params: { surah: String(i + 1) },
  }));
  return { paths, fallback: 'blocking' };
}

On-demand revalidation:

export default async function handler(req, res) {
  if (req.query.secret !== process.env.REVALIDATE_SECRET) {
    return res.status(401).json({ message: 'Invalid token' });
  }
  await res.revalidate(`/surah/${req.query.surah}`);
  return res.json({ revalidated: true });
}

The Result: Pages cached on CDN, served in <100ms globally.

Pattern 3: Dynamic Imports for Heavy Components

The Problem: Quran player: 350KB bundled. Every user downloads it even if they never use audio.

import dynamic from 'next/dynamic';

const QuranPlayer = dynamic(() => import('@/components/QuranPlayer'), {
  loading: () => <div>Loading player...</div>,
  ssr: false,
});

export default function SurahPage({ data }) {
  const [showPlayer, setShowPlayer] = useState(false);
  return (
    <div>
      <SurahText data={data} />
      {showPlayer && <QuranPlayer surah={data.surah} />}
      <button onClick={() => setShowPlayer(true)}>Play Audio</button>
    </div>
  );
}

Result: 68% of users never click play. They saved the download entirely.

Pattern 4: Image Optimization Pipeline

The Problem: Users upload 3-5MB JPEGs displayed at 120x120px.

import sharp from 'sharp';

export default async function handler(req, res) {
  const buffer = Buffer.from(req.body);
  const webpBuffer = await sharp(buffer).resize(400, 400, { fit: 'cover' }).webp({ quality: 80 }).toBuffer();
  const pngBuffer = await sharp(buffer).resize(400, 400, { fit: 'cover' }).png({ quality: 80 }).toBuffer();
  const webpUrl = await uploadToS3(webpBuffer, 'image.webp');
  const pngUrl = await uploadToS3(pngBuffer, 'image.png');
  res.json({ webpUrl, pngUrl });
}

<picture>
  <source srcSet={image.webpUrl} type="image/webp" />
  <img src={image.pngUrl} alt="User avatar" width={120} height={120} />
</picture>

Result: Images: 3-5MB → 45-80KB.

Pattern 5: Intelligent Prefetching

export function SurahNavigation({ currentSurah }) {
  const router = useRouter();
  const handlePrefetch = () => router.prefetch(`/surah/${currentSurah + 1}`);
  return (
    <Link href={`/surah/${currentSurah + 1}`} onMouseEnter={handlePrefetch} onTouchStart={handlePrefetch}>
      Next Surah
    </Link>
  );
}

Result: Users feel instant navigation. Pages load in <200ms instead of 1-2s.

Before/After

Metric	Before	After	Improvement
First Contentful Paint	3.2s	1.8s	44%
Largest Contentful Paint	4.8s	1.9s	60%
Time to Interactive	6.2s	2.4s	61%
Page Size	2.8MB	1.1MB	61%
Lighthouse Score	42	86	+44 pts

Performance work isn't about micro-optimizations. It's about architectural decisions that compound.

Find me at zunain.com if you're optimizing at scale.