DEV Community: kol kol

I Spent $500 on RAG Infrastructure Before Realizing These 7 Mistakes Were Killing My Results

kol kol — Sat, 20 Jun 2026 22:08:38 +0000

I Spent $500 on RAG Infrastructure Before Realizing These 7 Mistakes Were Killing My Results

I built a RAG pipeline for private document search. It cost me $500 in vector database compute, weeks of debugging, and a lot of frustration. The results were mediocre — users got irrelevant answers, queries were slow, and the whole thing felt like a fancy keyword search with extra steps.

Then I audited the pipeline step by step. Turns out, I made 7 mistakes that are incredibly common in RAG systems. Fixing them transformed the pipeline from "meh" to genuinely useful.

Here's what I got wrong, and what I changed.

Mistake #1: I Chopped Documents Into Random Pieces

I was splitting documents by fixed token count — 512 tokens per chunk, done. Simple, right?

Wrong. I was destroying semantic context. A paragraph about API authentication would get split mid-sentence, with half in one chunk and half in another. When retrieval ran, the LLM got fragmented context and produced garbage.

The fix: Parent-Document retrieval with semantic chunking.

Split by natural document boundaries first (paragraphs, sections, headers) — these are your "parent documents"
Create smaller child chunks from parents for vector search
When a child chunk matches, return the full parent document to the LLM
Add 10-20% overlap between chunks so boundary information isn't lost

# What I should have done from the start
CHUNK_CONFIG = {
    "chunk_size": 1000,
    "chunk_overlap": 200,
    "separator": ["\n\n", "\n", "。", "！", "？"],
}

Query accuracy jumped 30% after this one change.

Mistake #2: I Used 0.5:0.5 Weights for Hybrid Search

My vector database supports hybrid search — combining vector similarity with keyword (BM25) matching. I left the weights at the default 50/50 split and assumed that was fine.

It wasn't. For technical documentation, exact keyword matches matter way more than the default acknowledges. Someone searching for "HNSW ef_construction" needs that exact term, not a semantically similar but wrong answer.

The fix: Dynamic weights based on query type.

Factual queries ("what is X"): 35% vector, 65% keyword
Semantic queries ("how do I build X"): 75% vector, 25% keyword
General queries: 60% vector, 40% keyword

WEIGHTS = {
    "factual": {"vector": 0.35, "keyword": 0.65},
    "semantic": {"vector": 0.75, "keyword": 0.25},
    "general": {"vector": 0.6, "keyword": 0.4},
}

The keyword weight bump for factual queries alone eliminated most of the "almost right but wrong" answers.

Mistake #3: I Blew Up My Vector Database's Memory

I set ef_construction to the maximum value because "higher is better, right?" On a 50GB+ index, this meant the index build process consumed all available RAM and crashed. Twice.

The fix: Size-appropriate HNSW parameters.

# Don't max this out — your server will cry
HNSW_CONFIG = {
    "M": 16,              # connections per node (8-32 is the sweet spot)
    "ef_construction": 200,  # not 400. Not 1000. 200.
    "ef_search": 50,       # query time, not build time
}

Index build time went from "it crashed" to 45 minutes. Memory usage dropped 70%.

Mistake #4: My Embedding Model Was Too Generic

I was using a general-purpose embedding model trained on Wikipedia and web text. My documents were technical API references and engineering runbooks. The model didn't understand my domain.

The fix: Switch to a model fine-tuned for technical/code content. The difference was night and day — suddenly "migration" and "transform" weren't treated as synonyms just because they're sometimes related in general text.

Mistake #5: I Had No Query Rewrite Layer

Users typed natural questions like "why is my build slow" and the system searched for those exact words in technical documentation that said "CI pipeline optimization" and "build duration analysis." Zero overlap. Zero results.

The fix: A lightweight LLM query rewrite step before retrieval.

User query: "why is my build slow"
→ Rewritten: "CI pipeline performance optimization build duration"
→ Retrieved: Relevant documentation ✅

This single step improved recall by 40%. The cost? About 0.001 cents per query with a small model.

Mistake #6: I Didn't Filter Duplicate Context

Retrieving top-10 chunks meant I often got the same paragraph 3 times with slightly different wording. The LLM would repeat itself, hallucinate from the repetition, and produce bloated answers.

The fix: Maximal marginal relevance (MMR) re-ranking.

# Instead of returning top-10 most similar
# Return top-10 most similar AND diverse
retrieved = vector_store.search(query, k=20)
diverse = mmr_rerank(retrieved, query, lambda_param=0.7, k=10)

Answers became more concise and covered more ground.

Mistake #7: I Never Measured Retrieval Quality

I was evaluating the whole RAG pipeline end-to-end. If the final answer was bad, I didn't know if it was the retrieval, the prompt, or the LLM.

The fix: Separate retrieval evaluation.

Track hit rate: does the retrieved context contain the answer?
Track MRR (Mean Reciprocal Rank): how high in the results is the right chunk?
Build a golden test set of 100 query-document pairs
Only optimize the generation layer once retrieval scores are solid

This saved me from chasing the wrong problems for weeks.

The Results After All 7 Fixes

Metric	Before	After
Answer relevance	~45%	~85%
Avg query latency	3.2s	1.8s
Monthly vector DB cost	$180	$95
Duplicate context in responses	60%	8%

The Takeaway

RAG isn't hard because the algorithms are complex. It's hard because there are 7+ interconnected knobs, and they all interact with each other.

My advice: fix chunking first, then weights, then embedding quality. In that order. Everything else is optimization.

What's your biggest RAG headache? Drop it in the comments — I've probably hit it too.

My API Broke Every January 1st — The Timezone Bug That Slipped Past Code Review

kol kol — Sat, 20 Jun 2026 14:04:46 +0000

My API broke at exactly 00:00 UTC on January 1st. Not the users' midnight — UTC midnight. Which meant our users in Tokyo had been living with broken data since 9 AM their time.

And the worst part? The tests all passed. The staging environment worked fine. It only broke in production, because production is in a different timezone than staging.

The Bug

Here's what the code looked like:

function getDailyReport(date) {
  const start = new Date(date).toISOString().split('T')[0];
  const end = new Date(start + 'T23:59:59Z');

  return db.reports.findMany({
    where: {
      createdAt: { gte: new Date(start), lt: end }
    }
  });
}

Seems fine, right? toISOString() gives you UTC. We're filtering by date. What could go wrong?

Here's what went wrong: new Date(date) when date is just "2026-01-01" (no time component) gets interpreted in the local timezone. In staging (UTC server), "2026-01-01" → 2026-01-01T00:00:00.000Z. In production (US-East server), "2026-01-01" → 2026-01-01T05:00:00.000Z.

Five hour offset. Every single date query. For an entire year before anyone noticed.

Why Tests Passed

Our CI runs in Docker containers set to UTC. Our staging server is also UTC. Our production server? US-East. The timezone mismatch was invisible until New Year's Day rolled around and the date boundary crossed the timezone offset.

Staging (UTC):     2026-01-01 → Jan 1 00:00 UTC ✅
Production (EST):  2026-01-01 → Jan 1 05:00 UTC ❌

We lost 5 hours of data on every query. The reports showed numbers that were "close enough" that nobody flagged it for 12 months.

The Fix

function getDailyReport(date: string) {
  // Always append time to force UTC interpretation
  const start = new Date(`${date}T00:00:00Z`);
  const end = new Date(`${date}T23:59:59.999Z`);

  return db.reports.findMany({
    where: {
      createdAt: { gte: start, lt: end }
    }
  });
}

One line change. Append T00:00:00Z to force the Date constructor into UTC mode. No more ambiguity.

The Real Fix (Process, Not Code)

The code fix took 30 seconds. The real fix took a week:

Added a timezone assertion in CI — our test suite now explicitly checks that process.env.TZ === 'UTC'. If anyone changes the CI timezone, tests fail.
Set TZ=UTC in all Dockerfiles — every container, every environment, same timezone. No surprises.
Added a timezone check to our deploy script — date +%Z must return UTC before deploy proceeds.
Wrote a linter rule — flags any new Date(string) where the string doesn't contain timezone info.

The Lesson

Timezone bugs are sneaky because they don't crash. They produce wrong data that looks right. Your users won't get an error page — they'll get silently incorrect numbers, and they'll trust them.

Three rules I now follow:

Never trust the system timezone. Always set TZ=UTC explicitly.
Never parse dates without timezones. "2026-01-01" is ambiguous. "2026-01-01T00:00:00Z" is not.
Never assume your CI timezone matches production. Assert it in your tests.

I've been coding for years. I still got bit by this. If it can happen to me, it can happen to you.

Read more developer war stories and technical deep-dives at codcompass.com

My API Broke Every January 1st — The Timezone Bug I Should Have Caught in Code Review

kol kol — Fri, 19 Jun 2026 22:02:33 +0000

My API broke at exactly 00:00 UTC on January 1st. Not the users' midnight — UTC midnight. Which meant our users in Tokyo had been living with broken data since 9 AM their time.

And the worst part? The tests all passed. The staging environment worked fine. It only broke in production, because production is in a different timezone than staging.

The Bug

Here's what the code looked like:

function getDailyReport(date) {
  const start = new Date(date).toISOString().split('T')[0];
  const end = new Date(start + 'T23:59:59Z');

  return db.reports.findMany({
    where: {
      createdAt: { gte: new Date(start), lt: end }
    }
  });
}

Seems fine, right? toISOString() gives you UTC. We're filtering by date. What could go wrong?

Five hour offset. Every single date query. For an entire year before anyone noticed.

Why Tests Passed

Staging (UTC):     2026-01-01 → Jan 1 00:00 UTC ✅
Production (EST):  2026-01-01 → Jan 1 05:00 UTC ❌

We lost 5 hours of data on every query. The reports showed numbers that were "close enough" that nobody flagged it for 12 months.

The Fix

function getDailyReport(date: string) {
  // Always append time to force UTC interpretation
  const start = new Date(`${date}T00:00:00Z`);
  const end = new Date(`${date}T23:59:59.999Z`);

  return db.reports.findMany({
    where: {
      createdAt: { gte: start, lt: end }
    }
  });
}

One line change. Append T00:00:00Z to force the Date constructor into UTC mode. No more ambiguity.

The Real Fix (Process, Not Code)

The code fix took 30 seconds. The real fix took a week:

Added a timezone assertion in CI — our test suite now explicitly checks that process.env.TZ === 'UTC'. If anyone changes the CI timezone, tests fail.
Set TZ=UTC in all Dockerfiles — every container, every environment, same timezone. No surprises.
Added a timezone check to our deploy script — date +%Z must return UTC before deploy proceeds.
Wrote a linter rule — flags any new Date(string) where the string doesn't contain timezone info.

The Lesson

Timezone bugs are sneaky because they don't crash. They produce wrong data that looks right. Your users won't get an error page — they'll get silently incorrect numbers, and they'll trust them.

Three rules I now follow:

Never trust the system timezone. Always set TZ=UTC explicitly.
Never parse dates without timezones. "2026-01-01" is ambiguous. "2026-01-01T00:00:00Z" is not.
Never assume your CI timezone matches production. Assert it in your tests.

I've been coding for years. I still got bit by this. If it can happen to me, it can happen to you.

I Let AI Write My Backend Code for a Week — Here's What Actually Broke

kol kol — Sun, 14 Jun 2026 14:02:24 +0000

I told myself it would be fine. I had been using AI coding assistants for suggestions and autocomplete for months — and it worked great. So when a new project came up with a tight deadline, I thought: why not let AI handle the whole backend?

I set up a Cursor workspace, wrote a detailed spec, and hit generate. What followed was 5 days of "it compiles, but..." debugging that taught me more about software engineering than any tutorial ever did.

What Went Surprisingly Well

The boilerplate was genuinely impressive. In about 2 hours, I had:

A fully typed Express.js API with 12 endpoints
Zod validation schemas for every route
A Prisma schema with proper relations
Docker compose setup with Postgres and Redis

The code looked clean. Tests passed. I was feeling like a 10x developer.

The Cracks Started Showing

Bug #1: Silent Type Coercion

The AI generated this validation:

const userSchema = z.object({
  age: z.number(),
});

Looks fine, right? Except the API received ages as strings from the frontend. Zod parsed them fine in development (coercion worked). But in production with stricter mode? NaN everywhere. Users were getting 400 errors on signup.

Fix: z.coerce.number().int().positive() — but I had to find all 23 instances manually.

Bug #2: The N+1 Query Nobody Asked For

For a dashboard endpoint that listed users with their orders and order items, the AI generated:

const users = await prisma.user.findMany();
for (const user of users) {
  user.orders = await prisma.order.findMany({ where: { userId: user.id } });
}

Classic N+1. The Prisma docs literally have a page titled "How to avoid N+1 queries." With 500 users, this endpoint made 501 database queries and took 8 seconds.

Fix: include with nested relations — one query, 120ms.

Bug #3: Race Conditions in Token Refresh

The AI wrote a token refresh flow that looked perfect in isolation. But under load, concurrent refresh requests would invalidate each other's tokens. The AI's solution? "Add a retry mechanism." My solution? "Use a refresh token rotation pattern that handles concurrency properly."

Bug #4: The Error Handler That Swallowed Everything

catch (error) {
  console.log("Error:", error);
  res.status(500).json({ error: "Something went wrong" });
}

console.log doesn't serialize Error objects properly. Every production error was just {} in the logs. We ran like this for 3 days before anyone noticed.

Fix: console.error with proper error serialization and a proper logging library (we went with Pino).

The Real Problem

Here's what I learned: AI generates code that's correct in isolation but fragile in context.

It doesn't know:

Your deployment architecture (so it misses N+1 queries)
Your traffic patterns (so it ignores race conditions)
Your logging infrastructure (so it uses the wrong logger)
Your team's conventions (so it mixes patterns)

The generated code passes tests because tests are narrow. It compiles because the syntax is valid. But production is where context matters.

What I Changed

AI writes the first draft, humans write the final version. I'm not going back to writing everything from scratch, but every PR now requires a manual review of control flow, error handling, and data access patterns.
Architecture decisions stay human. Schema design, caching strategy, and error handling patterns are too context-dependent to outsource.
Add integration tests that AI can't fake. Unit tests pass. Integration tests reveal the gaps. We added a test suite that runs the full API against a real Postgres instance.
Observability from day one. Structured logging, request tracing, and error tracking are now part of the project template, not an afterthought.

The Bottom Line

AI didn't break my project. My assumption that "generated code equals production-ready code" did.

AI is an incredible force multiplier when used as a pair programmer. It's a liability when treated as a replacement for engineering judgment.

The week cost me 3 extra days of debugging, but I shipped a more robust system than I would have built alone — because the AI's mistakes taught me where my own blind spots were.

Use AI. But keep your hands on the wheel.

Have you had similar experiences with AI-generated code? I'd love to hear your war stories in the comments.

Our Test Suite Passed 100% — Then Users Found 14 Bugs in One Day

kol kol — Tue, 09 Jun 2026 18:03:24 +0000

We had 847 tests. Green checkmarks across the board. 100% coverage on our critical paths. I was proud of that dashboard.

Then a user reported that our checkout was double-charging on Safari. Another said the password reset emails weren't arriving. Within 24 hours we had 14 confirmed bugs — and our CI pipeline was still proudly green.

That's when I realized: 100% code coverage is a vanity metric that makes you feel safe while your users burn.

The Illusion of Coverage

Here's what our test suite was great at:

Testing individual functions in isolation
Verifying happy paths with clean inputs
Catching regressions in pure utility functions

Here's what it completely missed:

Browser-specific behavior — Safari's date parsing is different from Chrome's. Our test runner used Node.js. No browser, no Safari.
Race conditions — Two API calls firing simultaneously? Our mocked fetch resolved instantly. In production, timing matters.
Integration gaps — Each module had tests. The connections between modules did not.
Real-world data — Our fixtures were clean. User data is never clean.

The Bug That Started It All

A user in Japan reported being charged twice for a single purchase. We couldn't reproduce it locally. Our payment integration tests passed every time.

The root cause: a double-submit button on slow networks. Our mock API responded in 12ms. Real networks: 800ms. That gap was enough for impatient fingers to click twice.

The fix was 3 lines of code:

const [isSubmitting, setIsSubmitting] = useState(false);
// Button: disabled={isSubmitting}

Three lines. But the test suite — our beautiful 847-test suite — had zero tests for this scenario because nobody wrote a test for "user clicks button twice."

The 14-Bug Autopsy

After that incident, we categorized all 14 bugs:

Bug Category	Count	Tests Should've Caught It
Browser compatibility	4	❌ No cross-browser tests
Race conditions	3	❌ Mocks too fast
Edge-case user input	3	❌ Fixtures too clean
Third-party API changes	2	❌ No contract testing
Time zone bugs	2	❌ All tests ran in UTC

14 bugs. Zero caught by CI. The problem wasn't that we didn't have enough tests — we had the wrong kind of tests.

What We Changed

1. Added Integration Tests at Module Boundaries

Unit tests check the bricks. Integration tests check the mortar. We added tests specifically for the connections between services — where most real bugs hide.

2. Started Running Tests in Real Browsers

We added Playwright for critical user flows: checkout, auth, search. These run against a real Chrome and Firefox instance. Safari is next.

3. Mock Network Latency

Instead of instant mock responses, we randomized delays between 100ms and 2000ms. This surfaced race conditions we never knew existed.

4. Contract Testing for APIs

We used Pact to verify that our frontend's expectations of backend APIs actually match reality. Two bugs disappeared the day we added this.

5. Time Zone Roulette

We randomize the test runner's timezone. Half our date bugs appeared within the first week.

The New Philosophy

Coverage tells you what code runs. It doesn't tell you what breaks.

Now we track different metrics:

Bug escape rate — bugs found by users vs. caught in CI
Mean time to detection — how fast our tests find regressions
Integration test coverage — not line coverage, but scenario coverage

Our total test count went down (we deleted 200+ redundant unit tests). Our bug escape rate went down 80%.

The dashboard looks less impressive. The product works better.

Have you been burned by "green tests, broken production"? What testing gaps surprised you most? I'd love to hear your war stories in the comments.

I Added 20 Indexes to "Fix" Slow Queries — My Database Got 3x Slower

kol kol — Mon, 08 Jun 2026 14:02:21 +0000

I Added 20 Indexes to "Fix" Slow Queries — My Database Got 3x Slower

Six months ago, I inherited a PostgreSQL database that was choking on production traffic. API response times hit 8 seconds. Users were timing out. The ops team was getting paged at 2 AM.

So I did what any "experienced" developer would do: I added indexes. Lots of them.

Twenty indexes across twelve tables. Problem solved, right?

Wrong. The database got slower. Write operations crawled. Disk usage spiked. And the queries I was trying to optimize? They were still slow.

Here's what I learned the hard way about index tuning — and the process I use now that actually works.

The Mistake Everyone Makes

The biggest misconception about indexes is this: more indexes = faster queries.

PostgreSQL has to maintain every index on every write. Add an index, and every INSERT, UPDATE, and DELETE gets heavier. With 20 extra indexes, our write-heavy analytics table was spending more time updating indexes than storing data.

But the real killer was something I didn't expect: index bloat.

What Actually Went Wrong

1. I Indexed Low-Cardinality Columns

I put an index on a status column with only 4 possible values: pending, active, suspended, deleted.

PostgreSQL's query planner looked at that index, saw that each value matched ~25% of rows, and decided a full table scan was cheaper. The index was dead weight — costing disk space and write performance, providing zero read benefit.

2. I Created Redundant Indexes

I had:

CREATE INDEX idx_user_email ON users(email)
CREATE INDEX idx_user_email_name ON users(email, name)

The second index already covers queries on email alone. The first one was pure redundancy. PostgreSQL was maintaining two indexes for essentially the same lookup.

3. I Ignored Partial Indexes

Our orders table had millions of rows, but 90% were completed or cancelled. The slow queries were all looking for status = 'pending'. A partial index like:

CREATE INDEX idx_orders_pending 
ON orders(created_at, customer_id) 
WHERE status = 'pending';

This tiny index (10% of the table) outperformed my full-table indexes by 5x.

The Fix: A Methodical Index Audit

Here's the process I followed to undo the damage and actually optimize:

Step 1: Find Unused Indexes

SELECT 
  schemaname,
  tablename,
  indexname,
  idx_scan,
  pg_size_pretty(pg_relation_size(indexrelid)) as index_size
FROM pg_stat_user_indexes
WHERE idx_scan = 0
  AND NOT indisunique
ORDER BY pg_relation_size(indexrelid) DESC;

This revealed 14 indexes that had never been used since the last stats reset. I dropped them immediately.

Step 2: Find the Real Slow Queries

Instead of guessing, I used pg_stat_statements:

SELECT 
  query,
  calls,
  mean_exec_time,
  total_exec_time
FROM pg_stat_statements
ORDER BY total_exec_time DESC
LIMIT 10;

This showed me which queries were actually burning CPU time. Not the ones I assumed were slow — the ones that actually were.

Step 3: Use EXPLAIN ANALYZE

For every slow query, I ran EXPLAIN ANALYZE to see the actual execution plan. Not EXPLAIN — EXPLAIN ANALYZE. The difference is that EXPLAIN ANALYZE actually runs the query and shows real timing data.

What I found: PostgreSQL was doing sequential scans on tables where I had indexes, because my query conditions didn't match the index column order.

Step 4: Build Right-Sized Indexes

The three indexes that actually made a difference:

-- Composite index matching the actual WHERE + ORDER BY pattern
CREATE INDEX idx_analytics_date_type 
ON analytics(event_date, event_type) 
WHERE event_date > '2026-01-01';

-- Covering index that includes all needed columns (no table lookup)
CREATE INDEX idx_users_lookup 
ON users(email) 
INCLUDE (name, created_at);

-- Expression index for a common pattern
CREATE INDEX idx_orders_lower_email 
ON orders(LOWER(customer_email));

The Results

Metric	Before Audit	After Audit
Total indexes	47	18
Avg query time	3.2s	0.4s
Write latency	180ms	25ms
Index disk usage	12.4 GB	2.1 GB
Index cache hit rate	67%	94%

The Rule I Follow Now

Never add an index without running EXPLAIN ANALYZE first.

Every index should have a specific query it's designed to accelerate. If you can't point to the query and show the before/after execution plan, don't create the index.

Indexes are not a "just in case" thing. They're a surgical tool. Use them like one.

Have you ever made your database slower by trying to optimize it? What was your wake-up call?

I Thought My API Was Rate-Limited — Until Someone Scraped 2 Million Requests in 4 Hours

kol kol — Sun, 07 Jun 2026 14:04:53 +0000

I had express-rate-limit installed. I had it configured. I had tests that proved it worked.

And yet, someone still scraped 2 million API requests from my production server in under 4 hours. Costing me $4,200 in upstream API calls.

Here's exactly what went wrong, how I found out, and the architecture I use now.

The Setup That Lied to Me

My API was a simple Express app. I added rate limiting like any reasonable developer would:

import rateLimit from 'express-rate-limit';

const limiter = rateLimit({
  windowMs: 15 * 60 * 1000, // 15 minutes
  max: 100,                  // 100 requests per window
  standardHeaders: true,
  legacyHeaders: false,
});

app.use('/api/', limiter);

Tests passed. I saw X-RateLimit-Limit: 100 in curl responses. I slept well.

The problem? I was running 4 instances behind a load balancer. Each instance had its own in-memory counter. So the real limit was 400 requests per 15 minutes — not 100.

And the attacker wasn't hitting one IP with 100 requests. They were rotating through a proxy pool of 2,000+ IPs.

How It Happened

At 2:47 AM, our monitoring dashboard showed something odd: API request volume spiked 800%. I dismissed it as a newsletter push going out.

By 4:00 AM, the database connection pool was saturated. Queries that normally took 12ms were timing out at 30 seconds.

By 6:30 AM, I checked our upstream LLM provider bill. We'd made 2.1 million API calls since midnight. At $0.002 per call, that's roughly $4,200.

The attacker was:

Hitting our search endpoint with systematic keyword variations
Rotating IPs from a residential proxy network
Staying under per-instance rate limits by spreading requests across IPs
Extracting structured data from our responses

Why My Defenses Failed

Defense	Why It Failed
`express-rate-limit` (in-memory)	Not shared across instances
IP-based limiting	Proxy rotation defeated it
No request logging depth	Couldn't trace the attack pattern
No anomaly alerts	800% spike looked like "normal traffic"

The fundamental mistake: I treated rate limiting as a configuration problem instead of an architecture problem.

The Fix: Distributed Rate Limiting

I rebuilt the system with three layers:

Layer 1: Redis Sliding Window (The Real Rate Limiter)

import Redis from 'ioredis';
import { createClient } from 'redis-rate-limiter';

const redis = new Redis(process.env.REDIS_URL);

async function checkRateLimit(key, max, windowSec) {
  const now = Date.now();
  const windowStart = now - windowSec * 1000;

  // Use Redis sorted set for true sliding window
  await redis.zremrangebyscore(key, 0, windowStart);
  const count = await redis.zcard(key);

  if (count >= max) {
    return { allowed: false, remaining: 0 };
  }

  await redis.zadd(key, now, `${now}-${Math.random()}`);
  await redis.expire(key, windowSec);

  return { allowed: true, remaining: max - count - 1 };
}

This gives you a true 100-request limit across all instances, not 100 per instance.

Layer 2: Behavioral Fingerprinting

IP addresses are useless against proxy pools. Instead, I track:

Request pattern entropy — Are endpoints being hit in alphabetical order? That's a scraper.
Timing regularity — Requests every exactly 1.0 seconds? Bot.
Header consistency — Same User-Agent, same Accept-Encoding, same everything? Bot.

function calculateRequestEntropy(requests) {
  const endpoints = requests.map(r => r.path);
  const uniqueEndpoints = new Set(endpoints).size;
  // Low entropy = sequential/scraping pattern
  return uniqueEndpoints / endpoints.length;
}

// Entropy < 0.3 → likely scraping
// Entropy > 0.7 → likely human

Layer 3: Cost-Based Circuit Breakers

This is the one that actually saves money:

// Track estimated cost per endpoint
const endpointCosts = {
  '/api/search': 0.002,    // LLM call
  '/api/analyze': 0.015,   // Expensive LLM call
  '/api/health': 0,        // Cheap
};

let hourlyCost = 0;
const COST_THRESHOLD = 50; // Alert at $50/hr

function trackCost(endpoint) {
  hourlyCost += endpointCosts[endpoint] || 0;
  if (hourlyCost > COST_THRESHOLD) {
    // Auto-throttle expensive endpoints
    expensiveEndpoints.enabled = false;
    slack.alert(`API cost spike: $${hourlyCost.toFixed(2)}/hr`);
  }
}

When costs spike, expensive endpoints automatically throttle. You don't need to be awake at 3 AM to stop a bleeding wallet.

The Results After 30 Days

Metric	Before	After
Successful scrapes	2 incidents	0
Peak API cost/hr	$4,200	$12
False positive blocks	0	2 (tuned rules)
Legitimate user impact	N/A	None detected

The Real Lesson

Rate limiting isn't about setting a number. It's about understanding:

Your threat model — Who would want to scrape your API and why?
Your architecture — In-memory doesn't work in a distributed system. Period.
Your cost exposure — Know the dollar cost per endpoint, and set automatic circuit breakers.

The $4,200 mistake taught me that security theater — rate limiting that looks right but isn't — is worse than no rate limiting at all. It gives you confidence to deploy things that aren't actually protected.

Have you ever been bitten by a "working" defense that wasn't? What's your rate limiting setup? Drop it in the comments — I'm always looking for ways to improve mine.

I Left a Dead Feature Flag in Production for 6 Months — Here's What It Cost Me

kol kol — Sat, 06 Jun 2026 18:15:55 +0000

Six months. That's how long a disabled feature flag sat in my production codebase before a new team member asked: "What does this ENABLE_LEGACY_CHECKOUT thing do?"

Nobody remembered. The feature had been replaced, the flag was permanently set to false, but the dead code path was still there — loading, parsing, and checking on every single request.

The Discovery

It started as a routine code review. A junior dev flagged a function that looked suspicious:

function processPayment(order) {
  if (config.flags.ENABLE_LEGACY_CHECKOUT) {
    return legacyPaymentProcessor(order);
  }
  return newPaymentProcessor(order);
}

The flag had been false for 187 days. Every payment went through newPaymentProcessor. But every single call still evaluated the condition, loaded the legacy module (yes, it was lazy-loaded), and ran the config lookup.

The Hidden Costs

Here's what that one dead flag was doing:

1. Performance Tax — 0.3ms per request doesn't sound like much. Multiply by 2.4 million requests per month and you're burning ~12 hours of CPU time. Not catastrophic, but it adds up across dozens of flags.

2. Cognitive Load — New developers spent an average of 20 minutes trying to understand what each flag controlled. We had 47 active flags and 12 dead ones. That's 25% of our flag registry being ghost code.

3. Testing Overhead — Our test matrix had to account for flag combinations. Dead flags meant dead test branches that nobody was maintaining. We found 3 test suites that only tested dead code paths.

4. Deployment Risk — When someone finally cleaned up the flag, they accidentally removed a config validation that was shared between legacy and new paths. Caused a 15-minute outage.

What I Did About It

1. Flag Audit Script

I wrote a quick script to scan our codebase:

# Find all feature flags and check if they're always true or always false
grep -r "config\.flags\." src/ | \
  grep -o "ENABLE_[A-Z_]*" | \
  sort | uniq -c | sort -rn

Then cross-referenced with our LaunchDarkly dashboard to find flags that hadn't changed state in 90+ days.

2. The "Flag Funeral" Process

For each dead flag:

Verify — Check logs for 30 days to confirm zero true evaluations
Comment — Add a @deprecated note with removal date
Remove — Delete the code path in the next sprint
Celebrate — Log it in our engineering changelog (yes, really)

3. Automated Cleanup Rules

We added CI checks:

Flag unchanged for 60 days → warning in PR
Flag unchanged for 90 days → auto-create cleanup ticket
Flag unchanged for 120 days → block new flag creation until old ones are cleaned

The Results

After 3 weeks of cleanup:

Metric	Before	After
Active feature flags	47	31
Dead code branches	12	0
Avg. flag understanding time	20 min	5 min
Test suite execution time	14 min	11 min

34% reduction in flag count. 75% faster onboarding for new flags. And most importantly — zero confusion about what's live and what's dead.

The Lesson

Feature flags are like garden plants. If you don't prune them, they grow wild and start choking the things you actually care about.

Every flag you add is technical debt with an expiration date. Set the date. Enforce it.

Your future self — and your team — will thank you.

Found this useful? Follow me for more production war stories and practical devops tips.

I Broke Production 3 Times This Week — How a CI/CD Pipeline Audit Fixed Everything

kol kol — Sat, 06 Jun 2026 14:03:37 +0000

Last week I broke production three times. Not because of bad code — because our CI/CD pipeline was quietly lying to us.

Here's what happened, what I found during the audit, and the exact pipeline changes that eliminated deployment failures.

The Three Breakages

Break #1: A database migration ran twice because our pipeline didn't track which migrations had already executed. Result: duplicate key errors across 200+ records.

Break #2: Environment variable interpolation silently dropped a critical API key in production. The staging build passed because the variable was set in our .env.staging but not .env.production.

Break #3: A dependency update changed a function signature. Our test suite passed because the mocked version still matched the old signature. Production exploded at runtime.

Three different failure modes. One root cause: our CI/CD pipeline had more gaps than a net.

The Pipeline Audit

I mapped every step from git push to production deployment. Here's what I found:

Gap 1: No Migration Tracking

Our pipeline ran prisma migrate deploy blindly on every deployment. No check for already-applied migrations. No rollback plan.

Fix: Added a migration status check that queries _prisma_migrations before running anything new:

# Check pending migrations before applying
npx prisma migrate status
npx prisma migrate deploy --skip-generate

Gap 2: Env Var Validation Missing

We had 23 environment variables in production. Zero validation that they existed before deploying.

Fix: Added a pre-deployment validation step:

# Required env vars checklist
REQUIRED_VARS="DATABASE_URL NEXTAUTH_SECRET API_KEY STRIPE_SECRET"
for var in $REQUIRED_VARS; do
  if [ -z "${!var}" ]; then
    echo "❌ Missing required variable: $var"
    exit 1
  fi
done
echo "✅ All required environment variables present"

This single check has blocked 4 bad deployments since I added it.

Gap 3: Tests Didn't Match Reality

Our mock data was stale. Tests passed against a mocked API that hadn't been updated in months.

Fix: Two changes:

Contract testing: Added @stoplight/spectral to validate our OpenAPI spec against actual responses
Integration tests in CI: Running real API calls against a staging database, not just unit tests with mocks

The New Pipeline

git push → Lint → Type Check → Unit Tests → Contract Tests 
  → Build → Env Validation → Staging Deploy → Integration Tests 
  → Migration Check → Production Deploy → Health Check

Each stage blocks the next on failure. No more "tests passed but prod broke."

The Results

Since the audit (2 weeks ago):

0 production breakages (vs 3 in the previous week)
4 blocked bad deployments before they reached staging
Deploy confidence: Team actually ships on Fridays now

The Takeaway

Your CI/CD pipeline isn't just automation — it's your last line of defense. If it has gaps, production will find them.

Audit your pipeline. Map every step. Ask "what happens if this lies to me?" for each one.

Then fix the gaps before production finds them for you.

What's the worst CI/CD failure you've dealt with? Drop it in the comments — misery loves company.

I Built a RAG Chat Assistant from Scratch — Here's What Nobody Tells You About Production RAG

kol kol — Fri, 05 Jun 2026 14:03:37 +0000

Everyone talks about how easy it is to build a RAG (Retrieval-Augmented Generation) chat assistant. Just embed some documents, throw them in a vector database, and connect an LLM, right?

Well, I just spent weeks building a production RAG system for a real knowledge base platform with nearly 2,000 articles. And let me tell you — the gap between a weekend hackathon demo and something that actually works for real users is enormous.

What is RAG, Anyway?

In case you haven't heard the acronym a thousand times this year: RAG is a technique where you retrieve relevant documents from your own knowledge base and feed them into an LLM as context, so the AI answers based on your data instead of its training cut-off.

The theory is simple. The reality is full of landmines.

The Three Landmines Nobody Warns You About

1. Chunking is Everything (And You're Probably Doing It Wrong)

My first attempt? Split by 500-token chunks with no overlap. The result was terrible — half the retrieved chunks were cut off mid-sentence, losing critical context.

What actually worked:

Semantic chunking: Split at natural boundaries (headings, paragraph breaks)
Overlap strategy: 15-20% overlap between adjacent chunks
Metadata enrichment: Each chunk carries its source article title, category, and position

This single change improved answer quality by an estimated 40%.

2. Vector Search Alone Isn't Enough

Pure cosine similarity on embeddings returns results that are topically similar but often miss the mark for specific technical questions.

The winning combo:

BM25 (keyword search) for exact technical term matching
Vector similarity for semantic understanding
Hybrid ranking with a weighted score (0.4 BM25 + 0.6 vector)

This hybrid approach catches both "how to configure PostgreSQL connection pooling" (BM25 wins) and "why is my database slow under load" (vector wins).

3. Context Window is a Budget, Not a Freebie

Most tutorials stuff as many retrieved chunks as possible into the prompt. But every token costs money and degrades response quality.

My approach:

Top 5 most relevant chunks (not 10, not 20)
Intelligent deduplication: Remove near-identical chunks
Source attribution: Each answer links back to the original article

The Architecture That Actually Worked

User Question → Query Rewriting → Hybrid Search (BM25 + Vector) 
  → Reranking → Top-5 Selection → Prompt Assembly → LLM Response

Key tech choices:

Vector DB: Supabase (pgvector) — because it's Postgres and you already know SQL
Embeddings: OpenAI text-embedding-3-small (fast, cheap, good enough)
Reranking: Custom scoring function (BM25 + vector + recency boost)
LLM: GPT-4o-mini for cost efficiency on production traffic

The Results

~2,000 technical articles in the knowledge base
Sub-second query latency for most questions
Source-linked answers — every response cites which article it came from
Cost per query: ~$0.005 (embedding + LLM call)

What I'd Do Differently

Start with a reranker model from day one — it's worth the extra compute
Build a feedback loop early — let users thumbs-up/down answers
Don't over-engineer chunking — start simple, iterate based on actual query patterns

The Bottom Line

RAG isn't magic. It's engineering. And like any engineering problem, the devil is in the details. The gap between "it works on my laptop" and "it works for thousands of users asking weird questions at 3 AM" is filled with chunking strategies, hybrid search tuning, and relentless iteration.

If you're building a RAG system right now: start simple, measure everything, and expect to rewrite your retrieval pipeline at least three times.

Want to dive deeper into building developer tools? Check out my technical knowledge base at codcompass.com — growing weekly with real-world insights on shipping software.

Why I Stopped Using try/catch Everywhere — Error Handling Patterns That Actually Scale

kol kol — Thu, 04 Jun 2026 14:07:16 +0000

Why I Stopped Using try/catch Everywhere — Error Handling Patterns That Actually Scale

My app had 847 try/catch blocks. 73% of them were identical: catch (error) { console.error(error); throw error; }. I was wrapping error handling in more error handling. The code wasn't safer — it was just louder.

Here's what I learned after refactoring error handling across 12 microservices.

The Anti-Pattern: Defensive try/catch Sprawl

Every async function wrapped in its own try/catch, logging the same error, throwing the same error, catching the same error three layers up. The result?

Stack traces that told you nothing
Duplicate error logs (same error logged 4 times)
Actual recovery logic buried under boilerplate

I call this anxiety-driven development — wrapping everything in try/catch because we're afraid of unhandled rejections, not because we have a plan for each error.

What Actually Worked

1. Centralized Error Boundary

Instead of 847 try/catch blocks, I built one error boundary per service:

class ServiceBoundary {
  async execute<T>(operation: () => Promise<T>): Promise<T> {
    try {
      return await operation();
    } catch (error) {
      const classified = this.classify(error);
      this.log(classified);
      if (classified.recoverable) {
        return this.retry(operation, classified);
      }
      throw classified;
    }
  }
}

One boundary. Consistent classification. No duplicate logging.

2. Error Classification, Not Catch-All

Not all errors are equal:

Operational errors (network timeout, 429 rate limit) → retry with backoff
Programmatic errors (null reference, type mismatch) → fail fast, alert on-call
Expected errors (validation failure, not-found) → return to caller, don't throw

Once I started classifying, 60% of my try/catch blocks became unnecessary. Validation errors get returned as results, not thrown as exceptions.

3. Result Types for Expected Failures

Borrowed from Rust's Result<T, E> pattern:

type Result<T, E = AppError> =
  | { ok: true; value: T }
  | { ok: false; error: E };

When failure is expected (user not found, validation failed, rate limited), return a Result. Don't throw. Throwing should be reserved for unexpected failures.

This cut our error-related unit tests by 40% because we stopped testing "does this throw?" for things that aren't actually exceptional.

4. Structured Error Metadata

Every error now carries:

interface AppError extends Error {
  code: string;        // "USER_NOT_FOUND" not "ENOENT"
  severity: 'info' | 'warn' | 'critical';
  retryable: boolean;
  context: Record<string, unknown>; // userId, requestId, etc.
}

No more guessing what ECONNRESET means in your logs. No more searching through 847 catch blocks to find where the error was swallowed.

5. Let It Crash (Sometimes)

The hardest lesson: some errors should crash. If your database connection pool is exhausted, silently retrying 10 times while the queue builds up isn't resilience — it's a slow death spiral.

Fail fast, let the orchestrator restart you, and alert someone. A crash is often more honest than a zombie service that's technically "up" but functionally dead.

The Result

Metric	Before	After
try/catch blocks	847	212
Mean error resolution time	47 min	8 min
Duplicate error logs per incident	4.2x	1x
On-call false pages	31%	7%

The code isn't just cleaner — it's honest about what can go wrong and specific about what to do when it does.

How does your team handle errors? Are you catching everything, or are you classifying and responding?

My Monitoring Dashboard Was Lying to Me — How I Learned the Difference Between Monitoring and Observability

kol kol — Wed, 03 Jun 2026 22:47:19 +0000

I spent three hours last week staring at a perfectly green dashboard while our users were getting 5-second response times. Every metric said "healthy." Every alert was silent. But the product was broken.

That's when it clicked: I had built a monitoring system, not an observability system. And they are not the same thing.

The Dashboard Illusion

Here's what my setup looked like:

CPU usage: 23% ✅
Memory: 61% ✅
Request rate: normal ✅
Error rate: 0.2% ✅

But users were rage-clicking because the checkout flow was timing out on a specific third-party API call. The error rate was "fine" because retries masked the failures. The CPU was low because the bottleneck was network I/O, not computation. Every metric I was watching was measuring the wrong thing.

This is what I call decorative telemetry — numbers that look authoritative in a standup meeting but tell you nothing about what users actually experience.

Monitoring vs Observability: The Real Difference

The distinction that changed everything:

Monitoring asks: "Is it broken?" — You define thresholds, you get alerts when crossed.
Observability asks: "Why is it broken?" — You can explore unknown-unknowns without shipping new code.

Monitoring is a car dashboard (speed, fuel, temperature). Observability is a mechanic's diagnostic tool (can trace any symptom back to root cause).

Most teams — including mine — build dashboards and call it observability. That's like buying a speedometer and claiming you can diagnose engine problems.

What Actually Fixed It

1. SLO-First Instrumentation

Instead of measuring infrastructure, I started measuring what users perceive. For checkout, that meant:

SLO: 99% of checkout requests complete in < 2 seconds
SLI: Actual latency at p95 per 5-minute window
Error Budget: 7.2 minutes of SLO violation allowed per month

When the third-party API degraded, our error budget burned visibly. The dashboard went from "all green" to "73% budget consumed in 2 hours." That's actionable.

2. Correlation Discipline

The real breakthrough was forcing structured logs, trace IDs, and bounded context tags into every service. When something breaks, you follow the trace ID — not guess which dashboard to check.

Before: "Checkout is slow. Let me check 6 dashboards."
After: "Trace abc123 failed at step 4. It's the payment provider."

Mean-time-to-incomprehension dropped from hours to minutes.

3. Alert Hygiene

I cut our alerts from 47 to 8. The rule: page only what demands immediate human action. Everything else goes to a dashboard or a weekly report.

The 8 remaining alerts have explicit severity semantics:

SEV-1: Users cannot complete core flows → page immediately
SEV-2: Degraded experience for > 10% of users → page if it persists > 15 min
SEV-3: Internal tooling degraded → Slack notification
Everything else: Dashboard-only

The Maturity Model I Wish I'd Had Earlier

Level 1 — Foundational: Consistent logging, health checks, and actionable paging tied to runbooks. (This is where most teams think they're "done.")

Level 2 — Intermediate: Distributed tracing with sampling strategies that survive production load. You can follow a request across services.

Level 3 — Advanced: Anomaly detection where baselines are meaningful — not noisy vanity curves that nobody trusts.

Level 4 — Principal: Org-wide observability contracts. Every team instruments the same way, SLOs drive priority, and incident learning becomes institutional knowledge.

I was at Level 1 thinking I was at Level 3.

The Hard Truth

Dashboards without on-call runbooks are decorative. Metrics without error budgets are opinions. And alerts that page for everything page for nothing.

Observability isn't a tool you install. It's a discipline you practice — mapping every signal to a decision, every alert to an action, and every incident to a learning loop.

The green dashboard was lying to me not because it was wrong, but because it was answering questions nobody was asking.

What's the biggest gap between what your dashboards show and what your users experience?