DEV Community: vigneshwar

I benchmarked 7 LLMs on 100 identical prompts. The cost gap shocked me.

vigneshwar — Mon, 08 Jun 2026 05:43:29 +0000

Everyone asks: which LLM is the best?

Wrong question.

The right question: which LLM is best for your use case, at your scale, at your budget?

I ran 100 identical prompts across 7 major LLMs. Here's what the data actually showed.

The Numbers Nobody Shows You

Model	Accuracy	Cost/1K	Latency
GPT-4o	88.2%	$0.0080	892ms
Claude 3.5 Sonnet	87.6%	$0.0090	1240ms
GPT-4o-mini	78.4%	$0.0003	432ms
Gemini 1.5 Flash	76.8%	$0.0001	380ms
Claude 3 Haiku	74.2%	$0.0010	410ms
Mistral Small	71.0%	$0.0010	520ms
Llama 3 8B	64.4%	$0.0002	680ms

GPT-4o vs Gemini Flash:

11% accuracy gap
80x cost gap
2.3x speed gap

For 90% of production apps — Gemini wins.

Why I Built This

Every AI leaderboard ranks by accuracy.

Your production AWS bill ranks by cost per request.

They are not the same list.

I built an open source LLM Evaluation Framework that benchmarks any model across all 5 dimensions simultaneously:

5 Metrics in One Run

1. Accuracy
Four-strategy cascade scorer:

Exact string match (case-normalized)
Prefix normalization (strips "The answer is...")
Multiple-choice letter extraction
Fuzzy Levenshtein match at 0.85 threshold

2. Latency (full percentile breakdown)

p50, p75, p90, p95, p99
SLA violation rate against configurable threshold
Async parallel evaluation via LiteLLM

3. Cost per 1K tokens

From real token counts in API responses
Not estimates — actual billing data
Supports 15+ model providers

4. Hallucination Rate

Linguistic signal analysis
Detects hedging phrases, uncertainty markers, ungrounded claims vs grounding signals
Runs entirely locally, zero extra API cost
Score: 0.0 (grounded) to 1.0 (heavily hallucinating)

5. Reasoning Quality

Chain-of-thought depth scoring
Counts reasoning markers, grounding signals, response calibration
Score: 1 (one-word answer) to 10 (structured multi-step reasoning)

Quick Start

pip install llm-evaluation-framework

export OPENAI_API_KEY=your-key
export GEMINI_API_KEY=your-key

llm-eval compare --models gpt-4o-mini --models gemini/gemini-1.5-flash --benchmark mmlu --samples 100

Output:

Model              Accuracy  Latency p95  Cost/1K    Hallucination  Reasoning
gpt-4o-mini        78.4%     891ms        $0.000300  0.12           7.2
gemini-1.5-flash   76.8%     743ms        $0.000100  0.15           6.8

What's Under the Hood

Async parallel evaluation — runs all samples concurrently with configurable semaphore
MMLU benchmark — 57 subjects, ~14K questions (Massive Multitask Language Understanding)
TruthfulQA benchmark — 817 questions designed to expose common misconceptions
Custom benchmarks — bring your own JSON: [{"prompt": "...", "expected": "..."}]
FastAPI REST API — 12 endpoints, OpenAPI docs
Streamlit dashboard — radar charts, scatter plots, histograms
CLI — 7 subcommands
SQLite persistence — all runs stored, queryable
PDF report generation — shareable evaluation reports

The Uncomfortable Truth About AI Benchmarks

Leaderboards rank models by accuracy on standardized tests.

Production systems rank models by accuracy per dollar.

Those are very different rankings.

GPT-4o vs Gemini Flash:

On a leaderboard: GPT-4o wins by 11%
At 10M requests/month: $80,000 vs $1,000

For most apps, the correct answer 11% more often is NOT worth $79,000/month.

Stop picking LLMs from leaderboards. Start picking them from your data.

Live Demo

Try the accuracy scorer and hallucination detector live — no API key needed:

https://huggingface.co/spaces/vigneshwar234/llm-eval-demo

I Spent 4 Months Building a RAG System That Actually Understands Causality — Here's What I Learned (and the Math Behind It)

vigneshwar — Sun, 07 Jun 2026 11:29:58 +0000

"I spent 4 months building something the entire ML community said was already solved. Turns out, it wasn't."

I need to tell you something uncomfortable about every RAG system running in production today.

They're all broken in the same way. And almost nobody is talking about it.

I'm not saying they don't work. They do — most of the time. But there are two silent failure modes that cause them to hallucinate even when they retrieve the correct document. After months of banging my head against this problem, I built VORTEXRAG to fix both of them. This is that story.

The Day I Realized Something Was Deeply Wrong

I was building a financial Q&A system. Simple enough — index a corpus of SEC filings, answer questions about why companies performed the way they did.

The query: "Why did Company X's revenue drop in Q3?"

My RAG pipeline retrieved the right document. I could see it in the logs — the actual earnings call transcript where the CEO explained the supply chain disruption. Cosine similarity: 0.91. Perfect.

But the LLM's answer was completely wrong. It talked about macroeconomic conditions, interest rate sensitivity, sector-wide headwinds. All factually true things — about the industry. None of them the actual cause.

I dug into the context window. The correct chunk was there. But it was surrounded by 7 other chunks:

The company's 10-K risk factors (similarity: 0.87)
An analyst report on sector performance (similarity: 0.84)
Fed reserve commentary on the quarter (similarity: 0.82)
Three more topically-related but causally-irrelevant passages

The LLM saw all of them. It averaged them. It hallucinated a narrative that sounded exactly right but was factually wrong about this specific company.

I had discovered Context Window Poisoning — and I'd been unknowingly fighting it for months.

The Two Problems Nobody Fully Solves

After that moment, I went deep. I read every RAG paper published in the last 3 years. Self-RAG, CRAG, RAG-Fusion, FiD, REALM, Atlas, Toolformer — all of them. Great papers. Smart people. Significant advances.

But none of them fully addressed what I was seeing. Here's why:

Problem 1: Semantic Drift 🎯

Every RAG system today retrieves by cosine similarity. This works great for finding topically related content. But it fundamentally cannot distinguish between:

A chunk that caused something
A chunk that is merely associated with it

Ask "Why did Lehman Brothers collapse?"

Standard RAG returns:
| Chunk | Similarity | Causal? |
|-------|-----------|---------|
| Dodd-Frank Act provisions | 0.87 | ❌ Response to collapse |
| CDS mispricing mechanism | 0.91 | ✅ Actual cause |
| Systemic risk reports | 0.85 | ❌ Consequences |
| Bear Stearns comparison | 0.83 | ❌ Parallel event |

The LLM gets all four. It produces a response about regulatory failure and systemic risk. It never mentions CDSs. The answer is 100% hallucination — constructed from real documents, assembled into a false causal chain.

This is Semantic Drift: the retrieved context drifts from causal relevance toward topical association.

Problem 2: Context Window Poisoning ☠️

Even when you retrieve the right chunk, if 7 wrong chunks surround it, the LLM's attention gets diluted. This isn't speculation — it's backed by the "Lost in the Middle" paper (Liu et al., 2023): LLMs have a U-shaped recall curve. They remember the beginning and end of context best, and systematically lose information in the middle.

So even if your correct chunk is in the context window, if it lands at position 4 of 8 chunks, the LLM may functionally ignore it.

This is Context Window Poisoning: the noise-to-signal ratio in the context window destroys the LLM's ability to use the correct information.

Building VORTEXRAG: 4 Months, 7 Layers, One Obsession

I started with a simple question: What would it take to fix both problems simultaneously?

The answer turned out to be a 7-layer pipeline. Each layer solves a specific failure mode. Let me walk you through each one — not just what it does, but why I built it.

Layer 1: TVE — Tri-Vector Encoding 🔺

The insight: If similarity retrieval can't capture causality, we need to encode causality directly.

Standard RAG embeds text into a single semantic vector (usually 768 dimensions). VORTEXRAG encodes every chunk into a 864-dimensional tri-vector:

TVE score = α·cos_sem + β·cos_syn + γ·cos_cau

Where:
  sem = 768d SBERT all-mpnet-base-v2    (semantic meaning)
  syn = 64d  spaCy dependency parse     (syntactic structure)
  cau = 32d  PropBank SRL events        (causal relationships)

The causal arm (32 dimensions) encodes the PropBank-style semantic role labels: ARG0 (agent), ARG1 (patient), ARGM-CAU (cause), ARGM-EFF (effect). A chunk about "CDSs caused Lehman's collapse" has a very different causal vector than a chunk about "Dodd-Frank responded to the collapse" — even if they have identical semantic similarity to the query.

This is the foundation everything else builds on.

Layer 2: VRC — Vortex Retrieval Cone 🌀

The insight: Retrieval isn't a list — it's a geometry.

Traditional top-k retrieval treats candidates as a ranked list. VORTEXRAG models retrieval as a spiral probability surface in causal vector space:

spiral_rank = TVE · e^(−λr) · cos(nθ)

Where:
  θ = angle between query's causal vector and chunk's causal vector
  r = rank position in initial retrieval
  λ = 0.5 (radial decay, adaptive with corpus size)
  n = 2 (angular frequency)

The key term is cos(nθ). Chunks whose causal direction is more than π/4 (45°) from the query's causal direction get geometrically suppressed — their spiral rank drops toward zero regardless of semantic similarity. The "vortex" shape comes from combining the radial decay with the angular suppression: retrieved chunks form a cone, not a list.

Layer 3: SDC — Semantic Drift Corrector 🛡️

The insight: Measure the drift explicitly. Filter on it.

For each candidate chunk, SDC computes a Semantic Drift Score:

D = v_cau(query) − v_cau(chunk)    ← causal drift vector
SDS = 1 − tanh(‖D‖ / τ)           ← drift score ∈ [0, 1]

Accept chunk if SDS ≥ 0.72

The higher the ‖D‖, the more the chunk's causal structure has drifted from the query's. τ controls the sensitivity — and this is where the 11 domain presets come in:

Domain	τ	Why
Scientific	0.30	Strict: cause-effect chains must be tight
Medical	0.35	Strict: diagnosis → treatment causality
Legal	0.40	Strict: precedent → ruling chains
Financial	0.50	Moderate: multi-factor causation
General	0.80	Default: relaxed filtering
Creative	1.20	Lenient: loose associations acceptable

Layer 4: CPG — Context Poison Guard ⚗️

The insight: The entire window must be healthy, not just individual chunks.

Even if each chunk individually passes SDC, the combination of chunks can still poison the context. CPG measures the Effective Signal Ratio of the entire window:

ESR = Σ(SDS_i · w_i) / (P + ε)

Where:
  P = 1/k normalization penalty (penalizes large windows)
  ε = smoothing constant

If ESR < 3.5, CPG runs a greedy purge: remove the chunk with the lowest SDS score. Recompute ESR. Repeat until ESR ≥ 3.5.

Theorem 5.1 (proved in the paper): Removing the minimum-SDS chunk maximizes ESR improvement per step. The greedy algorithm is provably optimal — no other single-step removal produces a better ESR increase.

This theorem is what separates CPG from heuristic approaches. It's not "try to remove bad chunks." It's a mathematical guarantee that the greedy approach is the best possible approach.

Layer 5: RFG — Rank Fusion Gate 🔀

The insight: Additive fusion lets weak links through. Multiplication doesn't.

Most multi-signal retrieval systems use additive fusion:

score = w1·signal1 + w2·signal2 + w3·signal3

The problem: a chunk with (0.9, 0.9, 0.1) scores the same as one with (0.63, 0.63, 0.63). But the first chunk is clearly wrong — one signal is terrible.

VORTEXRAG uses multiplicative fusion:

Φ = TVE^α × SDS^β × ESR_contrib^γ

Now (0.9 × 0.9 × 0.1)^(1/3) = 0.45 vs (0.63 × 0.63 × 0.63)^(1/3) = 0.63. The chunk with one bad signal scores lower, as it should. No weak link can be compensated by strong links in other dimensions.

Layer 6: CCB — Causal Context Builder 🏗️

The insight: Where you put information in the context window matters enormously.

From "Lost in the Middle": LLMs recall information best at position 0 and near the end of context. The middle is a graveyard for information.

CCB builds a causal dependency graph and assigns positions:

pos = rank(Φ+) × causal_depth

depth-0 root-cause chunks → always placed at pos = 0

The chunk that is the causal root — the one that explains why — always goes first. Supporting evidence follows in causal order. The LLM gets the most important information exactly where it's best at attending to it.

Layer 7: FV — Faithfulness Verifier ✅

The insight: Don't trust the LLM. Measure it.

After generation, FV computes:

ΔR = 1 − ROUGE-L(answer, context) × NLI_score(answer, context)

Accept if ΔR ≤ 0.15

If ΔR > 0.15 (more than 15% of the answer isn't grounded in the context), FV rejects the response, reranks the context, and retries — up to 3 times. Uses DeBERTa-v3-small CrossEncoder for NLI.

This is the final catch. Even if all 6 previous layers work perfectly, the LLM can still hallucinate. FV makes hallucination measurable and catchable.

The Results — After 229 Tests and 6 Benchmarks

I tested VORTEXRAG on the standard QA benchmark suite: NQ, TriviaQA, WebQ, PopQA, HotpotQA, and 2WikiMultiHopQA.

Overall Performance

System	EM	F1	Faithfulness	Semantic Drift	Context Poisoning	Latency
VORTEXRAG	74.8	82.6	0.94	14%	7%	185ms
Self-RAG	68.4	77.1	0.81	28%	19%	410ms
CRAG	66.9	75.8	0.79	31%	22%	320ms
RAG-Fusion	62.8	71.9	0.73	33%	21%	280ms
Naive RAG	61.2	69.4	0.71	36%	24%	95ms

+13.6 EM over Naive RAG. +6.4 EM over Self-RAG. 2.2× faster than Self-RAG.

Per-Dataset Breakdown

Dataset	VORTEXRAG	Self-RAG	Δ	Why VORTEXRAG wins here
NQ	74.1	67.2	+6.9	Single-hop causal queries
TriviaQA	81.3	77.8	+3.5	Fact retrieval, less causal
WebQ	68.4	61.9	+6.5	Entity-centric causal chains
PopQA	71.2	63.4	+7.8	Long-tail causal knowledge
HotpotQA	67.9	61.1	+6.8	Multi-hop: biggest gains
2WikiMH	69.8	62.3	+7.5	Multi-hop: biggest gains

The biggest gains are on multi-hop datasets — exactly where causal reasoning matters most. This is the validation I was hoping for.

Ablation: Every Layer Earns Its Place

One of my proudest moments was running the ablation study. I was terrified some layers would show no contribution. They all did.

Configuration	EM	F1	Faithfulness	+EM
A: Naive RAG baseline	61.2	69.4	0.71	—
B: +TVE	65.3	72.8	0.74	+4.1
C: +VRC	67.8	75.1	0.76	+2.5
D: +SDC	70.4	78.3	0.80	+2.6
E: +CPG	72.1	80.2	0.85	+1.7
F: +RFG	73.4	81.4	0.89	+1.3
G: +CCB	73.9	81.9	0.91	+0.5
H: +FV (full VORTEXRAG)	74.8	82.6	0.94	+0.9

Every single layer contributes. There's no dead weight.

Human Evaluation — The Number That Matters Most

Automatic metrics only go so far. I had 3 domain experts (an NLP researcher, a practicing lawyer, and a biomedical scientist) evaluate 150 responses each on 4 dimensions using 5-point Likert scales.

Dimension	VORTEXRAG	Self-RAG	Naive RAG
Factual Accuracy	4.5/5	3.9/5	3.2/5
Causal Coherence	4.3/5	3.4/5	2.8/5
Completeness	4.2/5	3.8/5	3.5/5
Conciseness	4.1/5	4.2/5	4.0/5

The +0.9 on Causal Coherence vs Self-RAG is the one I'm most proud of. It means real humans, in real domains, found the causal reasoning in VORTEXRAG's answers to be significantly better. No automatic metric fully captures this.

Try It Right Now

5-Minute Quickstart

git clone https://github.com/vignesh2027/VORTEXRAG
cd VORTEXRAG
pip install -r requirements.txt
python examples/basic_usage.py

Your First Causal Query

from vortexrag import VortexRAG

# Pick your domain — parameters auto-calibrate
rag = VortexRAG(domain="medical")

result = rag.query(
    query="Why do SSRIs take 2-4 weeks to work?",
    chunks=my_medical_chunks
)

print(result.answer)          # Causally grounded answer
print(result.sds_score)       # Drift score: should be >= 0.72
print(result.faithfulness)    # Hallucination delta: <= 0.15
print(result.layer_trace)     # Full 7-layer trace for debugging

Layer-by-Layer Trace (What You Actually See)

Layer 1 [TVE]: Encoded 847 chunks as 864-dim tri-vectors (8.2ms)
Layer 2 [VRC]: Spiral-ranked top-50 candidates (2.1ms)
              Angular suppression: 12 chunks suppressed (θ > π/4)
Layer 3 [SDC]: Drift filtering with τ=0.35 (medical preset)
              Accepted: 31/50 chunks (SDS range: 0.72-0.97)
Layer 4 [CPG]: Initial ESR: 2.84 (below threshold 3.5)
              Purging: removed 4 low-signal chunks
              Final ESR: 4.12 ✓
Layer 5 [RFG]: Multiplicative Φ-scores computed (1.8ms)
              Top-5 selected: Φ ∈ [0.71, 0.89]
Layer 6 [CCB]: Causal graph built (3 root-cause chunks at pos=0)
              Context ordered by causal depth
Layer 7 [FV]:  Generated answer, ΔR = 0.09 ✓ (< 0.15 threshold)
              Faithfulness verified on first attempt

Total: 183ms

What I Learned Building This

1. Cosine similarity is a local optima. The entire field optimized around it so heavily that we forgot it was a proxy, not the thing itself. Causal relevance is the thing itself.

2. Every layer needs a mathematical reason to exist. I threw out 3 layers during development because I couldn't prove they were optimal or even justified. Theorem 5.1 (CPG optimality) took me 3 weeks to prove. Worth it.

3. Latency is a feature. At 185ms, VORTEXRAG is 2.2× faster than Self-RAG despite doing far more work. Why? Because Self-RAG generates multiple draft responses and selects. VORTEXRAG's pre-generation filtering means the LLM sees a clean, small context and generates once. Less is more.

4. Human evaluation is irreplaceable. My biomedical expert caught failure modes on multi-step enzyme pathway questions that EM, F1, and faithfulness all missed. Automatic metrics are necessary but not sufficient.

The Part I Almost Didn't Share

I'm a student at Takshashila University. This wasn't done in a research lab with 40 GPUs and a team of PhD students. This was done on my laptop, in my room, across 4 months of evenings and weekends, after getting frustrated with every RAG system I used giving me wrong causal answers.

I don't have institutional affiliation. I don't have a PhD supervisor. I just had a problem that bothered me and enough stubbornness to not stop until I could prove I'd solved it.

If you're a student or independent researcher reading this: the problems worth solving are the ones that bother you personally. Not the ones that are impressive on a CV. The ones that make you think "this is broken and nobody seems to notice."

That's what VORTEXRAG was for me. I hope it's useful for you.

Links & Citation

🌀 Live Demo: https://huggingface.co/spaces/vigneshwar234/VORTEXRAG

💻 Code (MIT, 229 tests): https://github.com/vignesh2027/VORTEXRAG

📄 Paper (Zenodo v3.0): https://doi.org/10.5281/zenodo.20579702

🔬 ORCID: https://orcid.org/0009-0004-9777-7592

@article{vignesh2026vortexrag,
  title   = {VORTEXRAG: Vector Orthogonal Resonance-Tuned EXtraction
             Retrieval-Augmented Generation — A 7-Layer Framework for
             Simultaneous Elimination of Semantic Drift and Context
             Window Poisoning},
  author  = {Vignesh, L},
  year    = {2026},
  doi     = {10.5281/zenodo.20579702},
  url     = {https://doi.org/10.5281/zenodo.20579702}
}

If this helped you, or if you tried VORTEXRAG and have feedback, drop a comment below. Every reaction, unicorn, and share genuinely helps this reach more people. Thank you for reading.

Tags: #machinelearning #python #rag #nlp #ai #deeplearning #opensource #research

[Boost]

vigneshwar — Sun, 24 May 2026 04:20:40 +0000

vigneshwar

May 24

I lost 3 enterprise clients in one night because of a GitHub repo. So I built a tool to make sure it never happens again.

#github #opensource #webdev #react

5 min read

I lost 3 enterprise clients in one night because of a GitHub repo. So I built a tool to make sure it never happens again.

vigneshwar — Sun, 24 May 2026 04:19:35 +0000

It was 11:47 PM on a Tuesday.

I had just pushed to production.

Closed my laptop. Made tea. Felt good about myself.

By 3:14 AM my phone was a disaster.

17 missed calls. 43 Slack messages. 6 emails.
The subject line on the first email read:
"URGENT — Platform completely down"

My hands were shaking before I even opened it.

Three weeks earlier I had been under insane deadline pressure.

We were building a SaaS product for enterprise clients.
Launch was in 72 hours.
I needed an authentication library fast.

I went to GitHub.

Found one that looked incredible.
Clean name. Professional README.
2,400 stars. 340 forks.
The code looked solid on first glance.

I did what most developers do under deadline pressure.

I added it. Shipped it. Went to sleep.

What I didn't check:

The last commit was 9 months ago.
There were 47 open issues marked as critical.
Zero CI/CD pipeline.
Zero test files in the entire repo.
The maintainer had responded to exactly 0 issues in 6 months.
There was a known security vulnerability reported 4 months ago.
Still open. No response. No fix.

In 3 seconds I could have seen all of this.

I didn't check. So I didn't know.

Until 3am.

The bug triggered under high concurrent load.

Our enterprise demo that night had 200 simultaneous users.
The library collapsed. Took the auth system with it.
Every single user got logged out.
Sessions destroyed. Data in a corrupted state.
The whole platform returned a 500 error for 14 straight hours.

We lost 3 enterprise clients that week.
Each worth $40,000 annually.

$120,000 gone because I didn't spend 3 minutes
checking a GitHub repo properly.

My manager didn't fire me.
But the look on his face in that Monday meeting
is something I will never forget as long as I live.

After that I became obsessive.

I started checking every single dependency manually.
Every library. Every tool. Every npm package.
Every GitHub repo anyone on the team suggested.

I built a personal checklist:

→ When was the last commit?
→ Is there a CI/CD pipeline?
→ Are there test files?
→ How many open issues vs closed?
→ What is the average time to close an issue?
→ Who are the contributors and are they still active?
→ Is there a license?
→ How long and detailed is the README?
→ What does the community size look like?
→ Are there known CVEs in the dependencies?

20 to 30 minutes per repo.
Every single time.

My team thought I was paranoid.

I thought I was just finally doing my job properly.

Four months later I had evaluated hundreds of repos this way.

And I was completely burned out from doing it manually.

Every evaluation felt like the same work.
The same checks. The same tabs. The same mental process.
Over and over and over.

I started thinking about the developers who don't do this at all.
The ones who are exactly where I was at 11:47 PM on that Tuesday.
Feeling good. Laptop closed. Tea in hand.

Not knowing what's coming.

So I spent three weeks and built RepoLens.

Not for clout. Not for a portfolio piece.
Because I genuinely needed it.
And I was pretty sure millions of other developers did too.

Here is what it does:

Paste any GitHub URL.

In 3 seconds you get:

🏥 Repository Health Score — 0 to 100
A single score computed across 7 quality dimensions.
README quality. Commit activity. Test detection.
CI/CD presence. License. Community size. Issue resolution.
One number that tells you everything.
With a letter grade. A B C D.
So you know in 1 second if this is production-ready.

🥧 Language Breakdown
A beautiful interactive pie chart showing every single language
used in the codebase with exact percentages.
Know the full technical makeup before you touch it.

🔥 52-Week Commit Heatmap
A GitHub-style activity grid showing every week of the past year.
See at a glance — is this project alive or abandoned?
Spot burnout periods. Spot release sprints.
Spot the exact week the maintainer stopped caring.

👥 Top Contributor Graph
Who actually built this thing?
Are they still active?
Is it one person or a healthy team?
Bar chart. Avatars. Contribution share visualization.
Everything you need to know about who drives this project.

📦 Smart Dependency Detection
Automatically parses every ecosystem file:
package.json for Node.
requirements.txt and pyproject.toml for Python.
Cargo.toml for Rust.
go.mod for Go.
pom.xml for Java.
Gemfile for Ruby.
Every package. Every version. Automatically.

🗂 Interactive File Tree
Collapsible directory explorer with file type icons.
See the structure of any codebase instantly.
Search and filter in real time.

📖 Beautiful README Renderer
Full GitHub Flavored Markdown.
Images. Tables. Code blocks. Everything.
Read the documentation without leaving the tool.

📤 One-Click Share Card
Export a beautiful PNG summary card.
Share on LinkedIn. Post on Twitter.
Send to your team before a code review.

I ran the library that destroyed my production server through it.

31 out of 100. Grade D.

In 3 seconds.

The exact score I needed at 11:47 PM on that Tuesday
instead of at 3:14 AM the next morning.

I've been using RepoLens every single day since I built it.

My entire team uses it now before every dependency decision.
We have a rule — no new library gets added without a score.

We haven't had a single library-related production incident since.

Not one.

I'm sharing it completely free.

No sign-up required.
No account.
No credit card.
No limits.
Works on every public GitHub repository on the planet.
Instant results. Every time.

And the entire thing is open source.

React 18 frontend. Vite. Tailwind CSS.
FastAPI Python backend. GitHub REST API only.
File-based caching. Rate limiting. Security headers.
Full type hints. Clean architecture.

If you want to see how it's built — every line of code is there.
If you want to contribute — PRs are open.
If you want to self-host it — full Docker support.

🌐 Try it free:

⭐ Star it on GitHub:
github.com/vignesh2027/GitHub-Repo-Analyzer

Drop any GitHub repo URL in the comments below.

I will personally reply to every single one
with its health score and what I'd fix first.

And tell me —

What's the worst GitHub repo you ever trusted?

What happened?

Because I have a feeling I'm not the only one
who learned this lesson the hard way.

awesome claude skills

vigneshwar — Thu, 21 May 2026 03:29:02 +0000

vigneshwar

May 21

Awesome-Claude-Skills I built 135 Claude Skills with real formulas. Here's what "production-grade" actually means.

#ai #claude #productivity #opensource

6 min read

Awesome-Claude-Skills I built 135 Claude Skills with real formulas. Here's what "production-grade" actually means.

vigneshwar — Thu, 21 May 2026 03:16:43 +0000

I've been frustrated for a long time.

Every "awesome Claude prompts" repo I found looked like this:

"Act as a senior software engineer. Be helpful, thorough, 
and professional. Consider edge cases."

That's not a skill. That's a costume.

Real expertise has frameworks. Named responsibilities.
Actual formulas. Code that runs. Constraints that prevent
the model from giving you the easy wrong answer.

So I spent 6 months building what I actually wanted.

AgentOS 2.0 — 135 production-grade Claude Skills.

This article explains exactly what's inside and why it's
different from every other prompt collection on GitHub.

The Problem With Every Other Prompt Repo

Most prompt repositories fall into one of three traps:

Trap 1: The costume prompt

"You are an expert financial analyst. 
Help the user with their finance questions."

Zero frameworks. Zero formulas. Zero depth.

Trap 2: The instruction dump

"When answering, always:
- Be professional
- Consider multiple angles  
- Cite sources
- Format your response clearly"

This is just asking Claude to be Claude. It changes nothing.

Trap 3: The persona prompt

"You are Alex, a no-nonsense McKinsey consultant 
with 20 years of experience..."

Roleplay, not expertise. The model doesn't suddenly
know DCF models because you named it Alex.

What actually works: Named sub-agents with
distinct responsibilities, actual domain formulas
in code, and explicit forbidden behaviors that
prevent hallucination in critical areas.

Here's what that looks like in practice.

What "Production-Grade" Actually Looks Like

FinanceOracle — The Apex Skill

This is the most complete skill in the repo.
Here's a fraction of what's inside:

12 Sub-Agents:

OptionsDesk — derivatives pricing and structuring
MacroStrategist — macro regime analysis
HedgeFundArchitect — strategy design
FamilyOfficeCIO — multi-generational allocation
TaxOptimizer — harvest and structure optimization
DerivativesStructurer — exotic product design (+ 6 more)

Actual runnable Python:

def black_scholes(S, K, T, r, sigma, option_type='call'):
    d1 = (np.log(S/K) + (r + 0.5*sigma**2)*T) / (sigma*np.sqrt(T))
    d2 = d1 - sigma * np.sqrt(T)

    if option_type == 'call':
        price = S * norm.cdf(d1) - K * np.exp(-r*T) * norm.cdf(d2)
        delta = norm.cdf(d1)
    else:
        price = K * np.exp(-r*T) * norm.cdf(-d2) - S * norm.cdf(-d1)
        delta = -norm.cdf(-d1)

    gamma = norm.pdf(d1) / (S * sigma * np.sqrt(T))
    vega  = S * norm.pdf(d1) * np.sqrt(T) / 100
    theta = (-(S * norm.pdf(d1) * sigma) / (2 * np.sqrt(T))
             - r * K * np.exp(-r*T) * norm.cdf(d2)) / 365

    return {"price": price, "delta": delta, 
            "gamma": gamma, "vega": vega, "theta": theta}

Black-Litterman portfolio construction:

def black_litterman(Sigma, market_weights, views_P, 
                    views_Q, views_omega, tau=0.05, delta=2.5):
    pi = delta * Sigma @ market_weights
    M_inv = np.linalg.inv(
        np.linalg.inv(tau * Sigma) + 
        views_P.T @ np.linalg.inv(views_omega) @ views_P
    )
    mu_bl = M_inv @ (
        np.linalg.inv(tau * Sigma) @ pi + 
        views_P.T @ np.linalg.inv(views_omega) @ views_Q
    )
    return {"expected_returns": mu_bl}

This isn't pseudocode. This runs.

OKREngine — Catches Failures Before They Kill Your Quarter

I've watched two startups waste entire quarters on
broken OKRs. This skill exists because of that.

The objective quality scorer:

def score_okr(objective: str, key_results: list[dict]) -> dict:
    obj_score = 0
    obj_score += 3 if len(objective) < 100 else 0
    obj_score += 3 if not objective.lower().startswith("improve") else 0
    obj_score += 4 if any(w in objective.lower() for w in 
        ["best", "lead", "#1", "transform", "redefine"]) else 0

    kr_scores = []
    for kr in key_results:
        kr_score = 0
        kr_score += 3 if kr.get("metric") else 0
        kr_score += 3 if kr.get("baseline") is not None else 0
        kr_score += 4 if kr.get("target") is not None else 0
        kr_scores.append({
            "kr": kr["text"][:60],
            "score": kr_score,
            "grade": "Good" if kr_score >= 8 else "Needs work"
        })

    return {
        "objective_score": f"{obj_score}/10",
        "key_results": kr_scores,
        "recommendation": "Strong OKR" if obj_score >= 8 else "Needs revision"
    }

The skill also catches the 12 most common OKR failure modes — including sandbagging, health metrics disguised as OKRs, and the single most destructive mistake: tying OKR scores to bonuses.

VentureIntelligence — Term Sheet Red Flag Detector

def score_term_sheet(terms: dict) -> dict:
    red_flags = []

    if terms.get("liq_pref_multiple", 1) > 1:
        red_flags.append(
            f"Liquidation preference {terms['liq_pref_multiple']}x — above 1x is punishing"
        )
    if terms.get("participating_preferred", False):
        red_flags.append(
            "Participating preferred — VCs get paid twice in exits below threshold"
        )
    if terms.get("anti_dilution") == "full_ratchet":
        red_flags.append(
            "Full ratchet anti-dilution — catastrophic in a down round"
        )
    if terms.get("board_seats_investor", 0) > terms.get("board_seats_founder", 0):
        red_flags.append(
            "Investor has majority board control — you can be fired from your company"
        )

    score = 10 - (len(red_flags) * 3)
    return {
        "score": max(0, score),
        "grade": "Sign it" if score >= 8 else "Negotiate" if score >= 5 else "Get a lawyer NOW",
        "red_flags": red_flags
    }

12 sub-agents including TermSheetDecoder, ValuationNegotiator, ChampionDeveloper, and BoardRelationshipManager.

CrisisIntelligence — War Room OS

Every company will face a crisis. Almost none prepare.

def classify_crisis(crisis: dict) -> dict:
    severity_score = 0

    if crisis["customer_impact_pct"] >= 0.5: severity_score += 30
    if crisis["revenue_at_risk"] >= 1_000_000: severity_score += 20

    coverage = {"none": 0, "local": 5, "national": 15, "viral": 30}
    severity_score += coverage.get(crisis["media_coverage"], 0)

    if crisis["regulatory_involvement"]: severity_score += 15
    if crisis["legal_liability"]: severity_score += 15

    if severity_score >= 70:
        level = "CRITICAL (P0)"
        action = "CEO leads. War room activated NOW."
    elif severity_score >= 40:
        level = "HIGH (P1)"
        action = "VP-level lead. External comms needed."
    else:
        level = "MEDIUM (P2)"
        action = "Director-level. Monitor externally."

    return {
        "level": level,
        "immediate_action": action,
        "time_to_first_response": "1 hour" if severity_score >= 70 else "4 hours"
    }

The 5Rs framework (Recognize → Respond → Responsible → Remediate → Restore) is built into every communication template.

How It Works (60-Second Setup)

Claude.ai Projects:

# 1. Open Claude.ai → Projects → New Project
# 2. Paste SKILL.md into "Project Instructions"  
# 3. Start chatting

Claude Code:

cat finance-oracle/SKILL.md >> .claude/CLAUDE.md

Claude API:

import anthropic

with open("startup-cto/SKILL.md", "r") as f:
    skill = f.read()

client = anthropic.Anthropic()
response = client.messages.create(
    model="claude-sonnet-4-6",
    max_tokens=4096,
    system=skill,
    messages=[{"role": "user", "content": "Audit our tech stack decision"}]
)

That's it. Claude is now that specialist.

The Full 135-Skill Index

🚀 Startup & Team Management (11)
startup-cto team-performance-os startup-hiring-machine

culture-architect remote-team-commander okr-engine

startup-finance-controller venture-intelligence

startup-legal-shield talent-management-os talent-brand-builder

🏆 Apex Legendary (4)
finance-oracle claude-mythos ceo-war-room founder-to-ceo

🤖 AI & Engineering (14)
rag-architect mlops-engineer system-architect senior-dev

ai-red-teamer voice-agent-builder knowledge-graph-builder

incident-commander mcp-builder agentic-workflow-builder

api-integrator realtime-data-agent agent-smith prompt-engineer

📊 Data & Analytics (10)
data-scientist-pro sql-analyzer data-pipeline-pro

business-intelligence-pro timeseries-oracle quant-trader

synthetic-data-generator arxiv-researcher abtest-scientist

data-governance-agent

💹 Finance (9)
finance-oracle financial-model-builder cfo-intelligence

portfolio-optimizer quant-researcher saas-metrics-analyst

insurance-actuary ma-dealmaker risk-sentinel

🏢 Operations & Business (20)
ceo-war-room founder-to-ceo go-to-market-commander

enterprise-sales-os sales-enablement-os board-deck-builder

crisis-intelligence partnership-intelligence pricing-strategist

project-command marketing-os supply-chain-oracle (+ 8 more)

👤 Product & Customer (11)
product-roadmap-os sprint-master engineering-manager

ai-product-manager user-research-os customer-interview-analyst

product-analytics-os network-effects-analyst marketplace-strategist

performance-marketing-os churn-analyst

🛠 Developer Tools (19)
developer-experience-os api-design-architect data-warehouse-architect

cloud-cost-optimizer design-system-architect technical-pm

code-reviewer load-tester code-migrator webapp-tester (+ 9 more)

🌐 Specialized Domains (12)
healthcare-analytics web3-developer climate-tech-analyst

biotech-analyst cybersecurity-analyst real-estate-intelligence

legal-eagle patent-analyst esg-compass (+ 3 more)

What Makes This Different From Every Other Repo

Feature	Generic repos	AgentOS 2.0
Sub-agents	❌	✅ 10-12 per skill
Actual formulas	❌	✅ Black-Scholes, DCF, MEDDPICC
Runnable code	❌	✅ Python, TypeScript, Go, Shell
Forbidden behaviors	❌	✅ Every skill
Benchmark data	❌	✅ Industry standards built in
Total skills	~10-20	135+

Try It Right Now

The fastest way to understand the depth is to try one.

I recommend starting with okr-engine or startup-cto — they're the most complete and immediately useful regardless of what you're building.

Paste the SKILL.md into Claude Projects. Ask it to review your current OKRs or tech stack. You'll see the difference immediately.

GitHub link in the comments.

What skill would you build your work around? Drop it below — I read every comment and I'm actively building more.

MIT License. Free forever. Star it if it's useful — helps others find it.

I Spent 6 Months Fixing RAG. Here's What I Found (And Built)

vigneshwar — Tue, 19 May 2026 15:59:07 +0000

This is the story of a debugging session that turned into a research paper.

The Bug That Started Everything
I was building a document Q&A system — nothing exotic. Standard RAG setup. FAISS index, SBERT embeddings, GPT as the reader. Classic.

It worked fine on simple questions. "What is the refund policy?" → correct answer.

Then I tested it on a multi-hop question: "What are the environmental compliance requirements for facilities that process the chemicals used in the manufacturing process described in section 4.2?"

The model's answer was confident. Detailed. And completely wrong.

The retrieved documents were all there. Every piece of information needed to answer correctly was in the context window. But the model still hallucinated a number that appeared nowhere in any document.

I started logging. What I found was two distinct failure modes happening simultaneously:

Failure Mode 1: Semantic Drift. By the time the query had been reformulated for multi-hop retrieval, the embedding had drifted so far from the original intent that we were retrieving the wrong documents. Not slightly wrong. Documents from a completely different section of the corpus.

Failure Mode 2: Context Poisoning. Even when we retrieved mostly correct documents, 1–2 tangentially related but contradictory chunks were slipping through. And those poison chunks were enough to derail the model.

Standard RAG has no defense against either of these. The pipeline is essentially: embed → retrieve → stuff into context → hope.

I needed something better.

Six Months Later: VORTEXRAG
I'm releasing the full framework today. 7 layers, each targeting a specific failure mode. Here's what I built and why each layer exists.

Layer 1: Tri-Vector Encoding (TVE)
The problem: Single-vector embeddings collapse too much information. "The bank charged a fee" and "The river bank was steep" share a close embedding in SBERT space even though they're semantically unrelated in most retrieval contexts.

The solution: Three encoding arms:

Semantic arm: standard SBERT 768d

s = sbert_model.encode(chunk) # shape: (768,)

Syntactic arm: POS + dependency structure

t = syntactic_encoder(chunk) # shape: (64,)

Causal arm: verb-argument chains

c = causal_encoder(chunk) # shape: (32,)

Fused vector

v = concat([α·s, β·t, γ·c]) # shape: (864,)
The causal arm is the key innovation. It captures "X causes Y" relationships that pure semantic similarity misses entirely. This is what allows the pipeline to distinguish between a document that mentions a concept and a document that explains the causal mechanism behind it.

Layer 2: Vortex Retrieval Cone (VRC)
The problem: Flat cosine similarity treats all high-similarity documents equally. But document #1 and document #47 in your ranked list shouldn't have equal weight — there's a natural falloff in relevance.

The solution: Spiral ranking inspired by vortex dynamics:

spiral_rank = TVE · e^(−λr) · cos(nθ)
Where r is the radial distance (rank position) and θ is an angular phase that encodes causal depth. Documents with high causal relevance get a phase advantage that can overcome a slightly lower semantic similarity score.

In practice: the top-k documents returned by VRC are causally denser than those returned by a flat cosine search on the same index.

Layer 3: Semantic Drift Corrector (SDC)
The problem: Multi-hop queries reformulate themselves at each hop. Each reformulation can drift slightly from the original intent. Over 3–4 hops, this compounds into a completely different query.

The solution: Track the embedding trajectory. At each hop:

drift = query_embedding - anchor_embedding
SDS = 1 - tanh(np.linalg.norm(drift) / τ)

if SDS < 0.72:
query_embedding = re_anchor(query_embedding, anchor_embedding)
The SDS score (Semantic Drift Score) measures how far we've drifted. Below 0.72, we re-anchor to the original query intent. This single intervention eliminated most of our multi-hop hallucinations.

Layer 4: Context Poison Guard (CPG)
The problem: 1–2 contradictory or off-topic chunks in the context window is enough to poison the model's answer. We needed a way to identify and remove these before they reach the LLM.

The solution: Entity-salience ratio per chunk:

ESR = sum(SDS_i * w_i for each entity i) / (num_propositions + ε)

if ESR < 3.5:
flag_for_purging(chunk)
The purging algorithm is provably greedy-optimal (I include the formal proof in the paper). It maximizes total ESR across the retained context while respecting the token budget.

In ablation studies, removing CPG alone dropped faithfulness from 0.94 to 0.75 — an 0.19 point drop from a single layer.

Layer 5: Rank Fusion Gate (RFG)
The problem: Most rank fusion methods are additive. A single terrible signal gets diluted but doesn't eliminate a bad document.

The solution: Multiplicative fusion:

Φ = TVE^α × SDS^β × ESR^γ
If any of the three signals is near zero, Φ collapses toward zero. A document that scores 0.9 on semantic similarity but 0.1 on CPG gets Φ ≈ 0.09 — effectively eliminated.

This was a deliberate design choice. In high-stakes retrieval (medical, legal, financial), you want a veto mechanism, not a popularity contest.

Layer 6: Causal Context Builder (CCB)
The problem: LLMs have dramatically higher attention to the beginning and end of their context window. Documents buried in the middle get "lost" — the Lost-in-the-Middle problem (Liu et al., 2023).

The solution: Reorder chunks by causal depth:

position = rank * causal_depth

High causal depth → lower position number → placed at context start

Chunks that are causally central to answering the question get placed at the beginning of the context window, where attention weights are highest. Peripheral context gets pushed to the middle where it does less damage if ignored.

Layer 7: Faithfulness Verifier (FV)
The problem: Even with all 6 preceding layers, some hallucinations slip through. We need a final gate.

The solution: Score the candidate answer against the retrieved context:

ΔR = 1 - (ROUGE_L * NLI_entailment_score)

if ΔR > 0.15:
regenerate_answer()
If the answer diverges more than 15% from the source documents (measured by ROUGE-L weighted by NLI entailment), it gets thrown out and regenerated. This catches the subtle hallucinations — cases where the model paraphrases correctly but changes a critical number or name.

Results
Tested on 4 standard benchmarks (NQ, HotpotQA, MuSiQue, 2WikiMultiHopQA):

System EM F1 Faithfulness
Naive RAG 61.2 68.4 0.71
HyDE 63.8 71.2 0.74
Self-RAG 65.4 73.1 0.79
FLARE 64.9 72.8 0.77
VORTEXRAG 74.8 82.6 0.94
+13.6 EM. +14.2 F1. +0.23 Faithfulness over naive RAG.

The ablation shows all 7 layers contribute. The two biggest individual contributors are CPG (+0.19 faithfulness) and SDC (+0.08 EM on multi-hop benchmarks specifically).

Quick Start

pip install vortexrag

from vortexrag import VortexRAG, VortexConfig

config = VortexConfig(
sdc_threshold=0.72,
cpg_esr_threshold=3.5,
fv_delta_r_threshold=0.15,
)

rag = VortexRAG(config)
rag.index(your_documents)

answer = rag.query("Your complex multi-hop question here")
What's Next
The framework is MIT licensed. The full research paper (with formal proofs) is on Zenodo.

If you're building RAG systems and hitting hallucination walls — especially on multi-hop or domain-specific queries — this framework is designed for exactly that problem.

GitHub: https://github.com/vignesh2027/VORTEXRAG

Paper: https://doi.org/10.5281/zenodo.20285144

Live demo docs: https://vignesh2027.github.io/VORTEXRAG

Questions? Drop them in the comments — happy to go deep on any of the layers.

Post these tonight at 9:00 PM IST. Reddit first (r/MachineLearning, then r/LocalLLaMA 30 minutes apart), then Dev.to. Cross-link the Dev.to article in your Reddit comments with "I wrote a deeper walkthrough here" — that drives traffic both ways.

I Stayed Up Until 3 AM to Build a Better Claude Code Guide Than the One With 52,000 Stars — Here's What I Found

vigneshwar — Sun, 17 May 2026 04:43:57 +0000

One night. One obsession. One repo that changed how I think about AI-assisted engineering.

It started the way most dangerous ideas do — quietly, at night, when the world had gone to sleep and the only sound was the hum of my monitors and the distant pulse of a city that never really stops.

I was tired. Not the kind of tired that sleep fixes. The kind that builds up when you've been consuming other people's work for so long that you start to forget you're capable of creating something yourself. I'd been scrolling GitHub for weeks — studying repositories, reading READMEs, bookmarking things I told myself I'd "come back to." I was good at bookmarking. Not so good at building.

Then I found it.

A GitHub repository about Claude Code best practices. 52,800 stars. 5,300 forks. Trending number one globally. The kind of numbers that make you feel small just looking at them. I read through it — and it was good. Really good. Community-built, battle-tested, full of real workflows from real teams. The kind of thing that takes months and dozens of contributors to produce.

I closed my laptop.

Then I opened it again.

Because something wouldn't let me go. Not jealousy — something quieter than that. A question. What would happen if I tried to understand this so deeply that I could build something better? Not to compete. Not to steal their stars. Just to find out if I had it in me.

It was 11 PM. I made coffee I didn't need. I pulled up the repository again, opened a blank terminal, and typed git clone https://github.com/vignesh2027/claude-best-practice.git into an empty folder. The cursor blinked at me. Waiting.

I started writing.

I want to be honest about what those hours felt like — because most people only share the highlight reel. The clean commits. The polished README. The final product. They don't talk about the 1 AM moment when you've been writing for two hours and you're not sure if what you're making is actually good or if you're just too tired to tell the difference. They don't talk about reading the original repo again at 1:30 AM, feeling like maybe you should just close the laptop and accept that 52,000 people already found what they needed.

I kept going anyway.

I wrote about context management — the thing nobody explains properly, the thing that silently kills your Claude Code sessions while you wonder why the output is getting worse. I wrote about verification loops — Boris Cherny's single most impactful insight, that running tests automatically after every edit improves output quality by 2–3x. I wrote about plan mode and why skipping it is like starting a road trip without checking the map. I documented hooks with real, copy-paste-ready shell scripts. I built out 9 skill types from Thariq at Anthropic — a framework so clean it made me angry that I'd never seen it laid out this clearly before.

I studied how Superpowers ships with Claude Code. How BMAD turns a vague idea into a production feature through a structured sequence of phases. How the gstack team runs a 14-stage process where every stage is a command, not a meeting. How the creator of Claude Code himself runs 5 local sessions and 5 cloud sessions simultaneously, codes primarily by voice, and says the single best thing you can do is give Claude a way to verify its own work.

By 2 AM I had written more than I'd written in the previous month combined.

By 2:47 AM — the clock you can see in the cover image, the real clock, the one I photographed because I wanted to remember this night — I had something I was genuinely proud of. Not because it was perfect. Because it was mine. Built from obsession, not obligation. From curiosity, not a content calendar.

Here is what I learned that night — and I don't mean about Claude Code.

I learned that the gap between consuming and creating is smaller than you think, and larger than it feels. It's smaller because you already know more than you give yourself credit for. It's larger because actually sitting down and making something — not planning to make it, not bookmarking it, not talking about making it — requires a different kind of commitment than most people are willing to give at 11 PM on a Tuesday.

I learned that the 52,000-star repo isn't your competition — your own hesitation is. The community that built that repository didn't sit down one night and produce 52,000 stars. They started somewhere. With something. Probably something smaller and rougher than what I built in one session.

I learned that depth beats breadth every time. The original repo has volume — hundreds of community contributions, tips from dozens of engineers. What I built has something different: a thread. A single person's attempt to understand something completely, from first principles, all the way through. That's a different kind of value. Not better or worse — different. And different is worth making.

I learned that the work you do at 2:47 AM, when nobody is watching and nothing is guaranteed, is the most honest work you'll ever do. It's not for likes. It's not for followers. It's for the version of yourself that gets up the next morning and looks in the mirror and knows: I actually did it.

The repository has everything I wish existed when I started learning Claude Code seriously:

10 deep-dive best practice guides — context management, hooks, subagents, MCP servers, CLAUDE.md mastery, prompting, model selection
5 advanced pattern guides — multi-agent teams, cross-model routing, automated pipelines, security hardening, enterprise deployment
Ready-to-use hook scripts — copy, paste, and run. Verification loops, auto-formatting, secrets protection, audit logging
Boris Cherny's creator workflow — how the actual creator of Claude Code uses his own tool, in detail
Thariq's 9 skill types framework — the most systematic approach to building Claude Code skills I've ever seen
Real-world team patterns — Superpowers, BMAD, gstack 14-stage, Spec Kit, Debugging War Room
70 tips from the creators themselves — distilled, organized, with a top-10 synthesis
Batch migration scripts — parallelize 100-file migrations across worktrees
PR babysitter automation — /loop 5m /babysit-prs handles your PR queue while you sleep
It is, genuinely, the most comprehensive Claude Code resource I know of. I say that not to brag — I say it because I spent a full night making sure it was true.

The city is still glowing outside my window.

The clock has moved past 3 AM.

My coffee is cold and I haven't noticed until right now.

And I have something I didn't have yesterday — a repository that represents the outer edge of what I understand, pushed to its limit, built past the point where most people stop.

If you're reading this and you've been bookmarking things without building them — I see you. I was you. The only thing that changed was one decision, one night, one refusal to close the laptop.

Dream it. Build it. Break limits.

⭐ The repository is live. Everything is free. No newsletter. No paywall. Just the work.

Github :(https://github.com/vignesh2027/claude-best-practice.git).

If this helped you, a star means more than you know. And if you build something with it — tell me. That's the only reward that matters.

I watched AI destroy 3 weeks of work in 4 minutes. So I built something 😭

vigneshwar — Sun, 17 May 2026 03:03:35 +0000

I still remember the exact moment.

It was 11pm. My team had been sprinting for 3 weeks building
a new payments feature. Deadline tomorrow. I handed the final
implementation task to our AI coding agent and said:

"Finish this up. Make it production ready."

I went to make coffee.

I came back to a deployed codebase with:

No input validation on the payment amount field
The API key hardcoded directly in the source file
Zero tests
No rollback procedure
User card numbers being logged to the console

-In 4 minutes, the AI had built something that would have
destroyed us if it had reached real users.

The worst part? The code looked fine. Clean functions.
Good variable names. It even had comments.

It just had none of the things that actually matter in production.

The thing nobody talks about

We spend so much time talking about what AI agents can do.

We don't talk enough about what they skip.

Here's what I've watched AI coding agents skip — over and over —
across my own projects and teams I've talked to:

They skip writing specs.
"The task is obvious."

They skip writing tests.
"I'll add them later."

They skip security checks.
"It's an internal API."

They deploy without a rollback plan.
"It's a small change."

They ship ML models without safety evaluation.
"The eval numbers look good."

They build data pipelines without quality gates.
"The source data is reliable."

And the worst part is — they sound confident while doing it.

No hesitation. No "are you sure?" Just fast, clean, wrong.

What I realized

The problem isn't that AI agents are bad at writing code.

They're genuinely incredible at writing code.

The problem is they have no discipline.

A senior engineer doesn't just know how to write code.
They know when to stop and write a spec first.
They know which shortcuts will cost you three weeks of debugging.
They know that "I'll add tests later" never happens.

That discipline — built from years of being burned —
is exactly what AI agents are missing.

You can't just tell an agent "be a senior engineer."

You have to show them the exact steps a senior engineer takes.
The exact things they verify before calling something done.
The exact excuses they refuse to accept from themselves.

So I built something

After that 11pm incident, I spent the next month building
AI Agent Skills — an open source framework of 40+ structured
workflow files that AI coding agents load before doing work.

Not prompts. Not vague instructions.

Structured skills — each one encoding exactly how a senior
engineer approaches a specific task:

What to do first
What to verify at each step
What evidence to produce before calling it done
What excuses to refuse

And — this is the part I'm most proud of — each skill has an
anti-rationalization section.

These are the exact shortcuts agents try to take, written out
explicitly, with a direct rebuttal for each one.

For example, the spec-driven-development skill includes:

"This feature is obvious — I don't need to write it down"
If it's obvious, the spec takes 10 minutes. If it's not obvious,
the spec saves days. Either way: write it.

"We'll iterate quickly — the spec will be wrong anyway"
Iterating on code without a spec means iterating in circles.

The agent sees these. It can't pretend the shortcut is reasonable.

What's covered

I wanted this to be the most complete skill framework ever built
for AI coding agents. So I went far beyond what others cover:

Standard engineering:
Spec writing · Planning · TDD · API design · Code review ·
Security · Performance · Git workflow · CI/CD · Documentation

AI/ML Engineering (nobody else covers this)
Training pipelines · Evaluation harnesses · Safety evaluation ·
Prompt injection testing · RAG system design · LLM evaluation ·
Agent orchestration · Distribution shift monitoring

Data Engineering (nobody else covers this)
Data contracts · Pipeline quality gates · Lineage tracking ·
Late-arriving data · Dead letter queues · GDPR compliance

Mobile Development (nobody else covers this)
Offline-first design · Main thread discipline · Battery efficiency ·
Crash rate targets · App store compliance

Incident Response (nobody else covers this)
5-phase structured response · Blameless post-mortems ·
Status page communication · Rollback-first philosophy

Plus 8 specialist agent personas — dedicated agents for:
Code Review · Security Auditing · Test Engineering ·
Performance · ML Engineering · Data Engineering ·
DevOps · Technical Writing

The moment it clicked for me

The first time I ran the framework on a real project, I gave
the agent a task and said /spec first.

It stopped.

It wrote a spec.

It listed functional requirements. It identified open questions.
It asked me to confirm before writing a single line of code.

I sat there staring at the screen thinking:

-This is what I actually wanted the whole time.*

Not faster code. Smarter code.

Code written by an agent that behaves like it's been burned before.
Like it has something to lose.

It's free. It's open source. It's yours.

I built this because I needed it. I'm sharing it because
I know I'm not the only one who's had the 11pm moment.

GitHub:

Works with Claude Code, Cursor, Gemini CLI, GitHub Copilot,
Windsurf, Kiro, OpenCode — and any agent that reads instructions.

MIT license. No strings.

One ask

If this resonates with you — if you've had your own version
of my 11pm moment — share this article.

There are millions of developers right now trusting AI agents
with production code. Most of them haven't been burned yet.

This framework is for the ones who want to learn from my mistake
instead of making their own.

⭐ Star it if it helps you.
🤝 Contribute if you have a skill to add.
💬 Comment if you've had your own AI disaster story.

I read every comment.

— Built at 11pm, after coffee, with lessons learned the hard way.