DEV Community: Haripriya Veluchamy

Vault cant found What a Live Audit Found That Code Review Never Would

Haripriya Veluchamy — Wed, 29 Jul 2026 08:28:13 +0000

Some bugs hide in logic. This one hid in the gap between "the code looks correct" and "the code is actually working" and the only way I found it was by refusing to trust either claim without checking the live system directly.

The setup

Mid-migration between two credential-storage systems: an older secrets vault (call it the legacy vault) used by an original provider integration, and a newer, purpose-built vault for a self-hosted replacement being rolled out gradually. During the transition, both exist side by side. The code that decides which vault to check for a given user's credentials looked, on review, completely correct it classified each connection record as "legacy" or "new" based on a stored field, and looked in the matching vault.

Nothing about the code review raised a flag. Which is exactly the problem with code review as your only line of defense: it tells you the logic is internally consistent, not that the systems it's talking to still exist.

Building the audit, properly this time

Instead of trusting the classification logic in the abstract, I wrote a small audit script with one rule: don't reconstruct what a secret's name should be and assume it's there actually query the live vault and see. This sounds obvious. It's also very easy to skip, because "reconstruct the expected name, spot-check a couple" feels like it should be equivalent to "check them all live," and it really isn't.

The script pulled every connection record from the database, classified each one the same way the app's own code did, and then made a real, live call to whichever vault it should be in.

What it found

Every single legacy-vault lookup failed. Not "secret not found" a connection failure. The vault itself wasn't resolving anymore.

That's a materially different problem than a missing secret. A missing secret means "someone forgot to write this." A vault that doesn't resolve at all means the entire piece of infrastructure the code has been confidently pointing at no longer exists and every single credential-write attempt to it, for every legacy user, had been silently failing, this whole time, for however long it had been broken.

How did nobody notice?

The honest, slightly embarrassing answer: the code that wrote to this vault caught the failure, logged a warning, and moved on. Not a crash. Not an alert. A line in a log nobody was watching, for a code path nobody expected to fail, because "the vault exists" was baked in as an assumption so early in the system's life that it had stopped being something anyone thought to re-check.

This is the actual shape of the bug, more than the specific vault: a try/catch that swallows a failure gracefully is indistinguishable, from the outside, from success until someone goes looking.

The part that actually surprised me

Here's where it got more interesting than "we found a broken thing, went and fixed it." Digging into why the legacy vault mattered at all, it turned out the underlying legacy provider managed those users' actual live sessions entirely on its own side the app's cached copy of their credentials was never actually read from again in practice. The vault had been dead, silently, for a long stretch, and functionally, nothing legacy-side had actually broken because of it.

That's not a "phew, no harm done, nothing to see here" ending though. It's a different, more uncomfortable one: we got lucky that this specific dead dependency happened to be redundant. The audit didn't know that in advance, and neither did I. The only way to find out whether a silently-dead piece of infrastructure actually matters is to go looking you can't reason your way to "probably fine" from the code alone.

The same audit, run against the newer vault the one actually still in active, load-bearing use found a real, smaller, actionable problem: a couple of expected entries genuinely missing. Real incomplete setup, not infrastructure rot. Fixed directly.

What I took away from this

Code review confirms logic. It cannot confirm that the systems the logic depends on still exist. Those are different guarantees, and it's easy to unconsciously treat the first as if it implies the second.
A caught-and-logged failure is a blind spot by construction, not just bad luck the entire point of catching it gracefully is that nothing breaks loudly. That's good for uptime and bad for ever noticing, unless something is actually watching those logs.
"Reconstruct the expected value and spot-check it" is not the same audit as "query the live system for every case." The first checks your mental model. The second checks reality. They can disagree, and when they do, reality wins.
Every so often, a system this old deserves a live audit, not just a code review specifically to surface the assumptions that were true once, got baked in early, and nobody has had a reason to re-question since.

Why Your CI Agent Can't Install pip

Haripriya Veluchamy — Sat, 25 Jul 2026 16:06:39 +0000

If you've ever set up a self-hosted CI/CD agent on a fresh Ubuntu image and hit pip: command not found or No module named pip, your first instinct is probably to check the network. Mine was too. That instinct is wrong, and the real answer took me four attempts to actually land on.

Here's the full story, including the three dead ends, because I think the dead ends are the useful part.

The setup

A self-hosted CI agent (in my case, an Azure DevOps agent, but this applies just as much to a self-hosted GitHub Actions runner or a plain Jenkins box) running on a minimal Ubuntu image. No sudo access on the box that's deliberate, since the agent shouldn't need root just to run a pipeline. I needed to install a CLI tool that only ships via pip.

Reasonable first attempt:

- script: |
    python3 -m pip install --user --quiet some-cli-tool
  displayName: 'Install tool'

Result:

/usr/bin/python3: No module named pip

Fine, I thought pip module is missing, ensurepip should bootstrap it. Every tutorial says so.

Attempt 2: `ensurepip`

- script: |
    python3 -m ensurepip --user --upgrade
    python3 -m pip install --user --quiet some-cli-tool

Result:

/usr/bin/python3: No module named ensurepip

Not just pip missing ensurepip itself, the thing that's supposed to install pip, doesn't exist either. That's the first real clue something structural is going on, not just a missing package.

The actual reason (worth understanding, not just working around)

Debian and Ubuntu deliberately strip pip and ensurepip out of the base python3 package. This isn't a bug or an oversight it's a packaging policy decision. Both live in separate packages (python3-pip, python3-venv) that you're expected to install via apt. On a full desktop or dev-configured server, you've probably had these installed for so long you forgot they're not actually part of core Python.

On a minimal, no-sudo CI image, you don't have apt access at all so you can't just apt install python3-pip your way out of it.

Attempt 3: `venv` (also fails, same root cause)

My next thought: skip pip/ensurepip entirely, use a virtual environment instead, since venv normally bootstraps its own pip on creation.

- script: |
    python3 -m venv /tmp/tool-venv
    /tmp/tool-venv/bin/pip install --quiet some-cli-tool

Result: fails identically. venv's own pip-bootstrapping step depends on you guessed it the same missing ensurepip module. Same wall, different door.

Attempt 4: the actual fix `get-pip.py`

The real answer is a standalone bootstrapping script, maintained by the Python Packaging Authority specifically for situations like this environments where ensurepip isn't available:

- script: |
    curl -sS https://bootstrap.pypa.io/get-pip.py -o /tmp/get-pip.py
    python3 /tmp/get-pip.py --user --quiet
    python3 -m pip install --user --quiet some-cli-tool
    echo "##vso[task.prependpath]$HOME/.local/bin"
  displayName: 'Install tool (get-pip, no sudo)'

get-pip.py doesn't depend on ensurepip at all it's a self-contained bootstrapper that installs pip directly. This worked immediately, no sudo, no apt, no venv complexity.

The one thing to double-check before you assume this is your problem

Before chasing this fix, rule out the boring explanation first: confirm your agent actually has network access to PyPI at all. A quick diagnostic step saved me from solving the wrong problem:

- script: |
    curl -sS -o /dev/null -w "pypi.org: %{http_code}\n" https://pypi.org --max-time 5

If that comes back with anything other than a fast 200, you're dealing with a firewall/egress issue, not this one and no amount of get-pip.py will fix that.

The takeaway

If python3 -m pip and python3 -m ensurepip are both missing on a Debian/Ubuntu box, that's not a broken image it's the distro's actual, intentional packaging policy. apt install python3-pip is the "normal" fix, and it's simply not available to you on a locked-down, no-sudo CI agent. get-pip.py is the one workaround that doesn't need apt, doesn't need ensurepip, and doesn't need root.

Small thing, but it cost me three wrong turns before I found it hopefully this saves you those three.

Why AI Can't Actually Hit Your Word Limit

Haripriya Veluchamy — Wed, 22 Jul 2026 10:20:08 +0000

I Was drafting my blog post, told Claude "keep it to 250 words." Got back something like 340. Asked again. Same story.

Figured there's a real reason behind this, and since I'm neck-deep in AI/ML fundamentals right now, decided to actually understand it instead of just being annoyed by it.

It generates one token at a time, no plan

LLMs don't write like we do think of the whole thing, then fill it in. They generate autoregressively: predict the next token, add it to the context, predict the next one, repeat. There's no "here's my 250-word essay, let me write it out" step happening anywhere. Every single token is picked based on what came before, nothing more.

So when you ask for 250 words, the model isn't holding a target in memory and counting down. It's pattern-matching: "text like this, following an instruction like this, tends to stop around here." That's a guess shaped by training data, not a computation.

Words ≠ tokens

Also it's not even counting words. Internally, everything is tokens: sub-word chunks. "Debugging" might be one token or three depending on the tokenizer. "250 words" as an instruction gets converted into an expectation of roughly how many tokens that maps to, and that mapping is fuzzy. There's no clean 1:1 between what you asked for and what the architecture tracks.

No counter anywhere in the architecture

This is the part that surprised me most: there's no length-tracking variable sitting in the model's forward pass. No if word_count == 250: stop. The only thing steering generation is the probability distribution over the next token, sampled step by step, until it predicts an end-of-sequence token or hits a max-length cutoff set by the system.

Compare that to something like a for loop with a counter deterministic, exact. LLM generation is closer to sampling from a learned distribution, over and over. Approximate by design, not by bug.

The workaround

Since I can't get an exact count out of a single generation, what actually works:

Ask for a natural stopping point instead of a hard number
Generate first, then trim or expand after, treating the count as a second pass, not the first
For real precision, set limits in characters or use iterative regeneration with a check step, rather than trusting the first output

Why this clicked for me

This ties directly into what I've been learning attention, softmax over vocabulary, sampling strategies. Word-count control would need something structurally different: a running counter fed back into the decoding loop, or a constrained decoding step that force-stops at a token budget. Neither is how these mainstream models work by default.

Small annoyance, decent rabbit hole. Worth knowing before you set a hard word limit and assume the model's counting along with you.

Learning AI/ML from the ground up cloud engineer trying to understand the models she uses daily.

Serving ML Artifacts from Amazon S3 Files How I used After the Launch

Haripriya Veluchamy — Mon, 22 Jun 2026 13:37:34 +0000

The Honest Story

Two months ago, everyone was posting about Amazon S3 Files. New feature, big announcement, screenshots everywhere. I scrolled past most of them another AWS launch, another round of "here's what it does" posts.

I never actually understood what it was until it became my problem.

I was in the middle of my ML learning journey, building a semantic search project. Large FAISS indexes, BM25 artifacts all needed to be available at serving time. I was uploading them directly into my deployment. It worked, until it didn't. Hit the size limit. Container cold starts became painful. Every restart meant downloading hundreds of megabytes before the first request could be served.

I had one option left S3.

And somewhere in the back of my head I remembered: "wait, there's that new S3 file system thing everyone was talking about." S3 is already my favorite AWS service. So why not try it?

I tried it. I built something real with it. And now I'm writing the post I wish existed two months ago not "here's what S3 Files is" but "here's what actually happens when you use it."

What I Built

A semantic search engine over AWS documentation 5048 pages, indexed with FAISS + BM25 hybrid retrieval, served via FastAPI. The entire ML artifact stack (indexes, metadata) is served directly from an S3 Files NFS mount on EC2. No cold start downloads. No boto3 in the serving code. Just open().

Stack:

Embeddings: all-MiniLM-L6-v2
Semantic search: FAISS (IndexFlatIP)
Keyword search: BM25Okapi
Serving: FastAPI + Jinja2
Artifact storage: Amazon S3 + S3 Files (NFS)
Compute: EC2 t3.medium (Ubuntu 22.04)

GitHub: Harivelu0/s3-files-ml-serving

The Problem S3 Files Solves

Before S3 Files, serving large ML artifacts from containers looked like this:

# Every container startup
from huggingface_hub import hf_hub_download

faiss_path = hf_hub_download(repo_id="...", filename="faiss.index")  # 7MB
bm25_path  = hf_hub_download(repo_id="...", filename="bm25_index.pkl")  # 17MB
meta_path  = hf_hub_download(repo_id="...", filename="corpus_meta.pkl")  # 2.6MB

Every container restart meant downloading ~27MB before the first request. For larger models this becomes hundreds of MB or GBs. You'd need to manage download logic, handle failures, worry about /tmp size limits, and every container on the same host duplicates the same data.

With S3 Files, your serving code becomes:

import faiss
import pickle

# Just open() no download, no boto3
index = faiss.read_index("/mnt/artifacts/faiss.index")
with open("/mnt/artifacts/bm25_index.pkl", "rb") as f:
    bm25 = pickle.load(f)

The S3 bucket is mounted as an NFS volume on the EC2 host. Your container reads from it like a local file. That's it.

Architecture

AWS Docs (5048 pages)
        ↓
crawl_aws_docs.py (Sitemap + BeautifulSoup)
        ↓
build_index.py (FAISS + BM25 + corpus_meta)
        ↓
S3 Bucket (versioning enabled)
        ↓
S3 File System (NFS layer on top of bucket)
        ↓
Mount Target (NFS endpoint inside VPC)
        ↓
EC2 Ubuntu 22.04
  └── /mnt/artifacts (S3 Files mounted here)
       └── Docker container
            └── FastAPI reads indexes directly
                 └── http://ec2-ip:8000

Setting Up S3 Files What Actually Matters

1. S3 bucket needs versioning enabled

This one catches everyone. S3 Files will refuse to create a file system without it.

aws s3api put-bucket-versioning \
    --bucket your-bucket \
    --versioning-configuration Status=Enabled

2. IAM role trust principal is `elasticfilesystem.amazonaws.com`

Not s3files.amazonaws.com even though the service is called S3 Files, it's built on EFS under the hood.

{
  "Principal": { "Service": "elasticfilesystem.amazonaws.com" },
  "Condition": {
    "StringEquals": { "aws:SourceAccount": "YOUR_ACCOUNT_ID" }
  }
}

3. The IAM role needs EventBridge permissions

S3 Files uses EventBridge internally to monitor bucket changes. Without these, your file system gets stuck in creating forever.

{
  "Action": [
    "events:PutRule", "events:DeleteRule",
    "events:PutTargets", "events:RemoveTargets"
  ],
  "Resource": "*"
}

4. boto3 API uses camelCase

The S3 Files boto3 client uses camelCase unlike most other AWS services.

# Wrong
s3files.create_file_system(Bucket=bucket_arn, RoleArn=role_arn)

# Correct
s3files.create_file_system(bucket=bucket_arn, roleArn=role_arn)

5. Mount needs `amazon-efs-utils`, not plain NFS

Plain mount -t nfs4 will fail. S3 Files requires the amazon-efs-utils package (v3.0+) which handles TLS + IAM auth automatically.

# Build and install amazon-efs-utils
sudo apt-get install -y cmake golang-go rustc cargo
git clone https://github.com/aws/efs-utils
cd efs-utils && ./build-deb.sh
sudo apt-get install -y ./build/amazon-efs-utils*.deb

# Then mount
sudo mount -t s3files fs-0xxxxxxxxx:/ /mnt/artifacts

6. EC2 instance needs `AmazonS3FilesClientFullAccess`

Without this policy on the EC2 role, the mount returns access denied by server.

aws iam attach-role-policy \
    --role-name your-ec2-role \
    --policy-arn arn:aws:iam::aws:policy/AmazonS3FilesClientFullAccess

The Serving Code

FastAPI startup loads everything from the mount. No download logic. No error handling for network failures during download. The mount is always there.

ARTIFACTS_DIR = Path(os.getenv("ARTIFACTS_DIR", "/mnt/artifacts"))

@asynccontextmanager
async def lifespan(app: FastAPI):
    # Reads directly from S3 Files mount
    state["index"] = faiss.read_index(str(ARTIFACTS_DIR / "faiss.index"))

    with open(ARTIFACTS_DIR / "bm25_index.pkl", "rb") as f:
        d = pickle.load(f)
        state["bm25"] = d["bm25"]

    with open(ARTIFACTS_DIR / "corpus_meta.pkl", "rb") as f:
        state["meta"] = pickle.load(f)

    state["model"] = SentenceTransformer("all-MiniLM-L6-v2")
    yield

Container startup: model loads in ~2s. Artifacts available instantly from mount.

Weekly Index Updates The Real S3 Files Advantage

This is where S3 Files goes beyond just solving cold starts.

AWS updates their docs regularly. With traditional artifact serving you'd need to rebuild indexes, push a new container image, and redeploy causing downtime.

With S3 Files:

# update_pipeline.sh runs weekly via EventBridge
python scripts/crawl_aws_docs.py --update   # only changed pages
python precompute/build_index.py            # rebuild indexes to /tmp

# Atomic swap no downtime
mv /tmp/faiss.index.new     /mnt/artifacts/artifacts/faiss.index
mv /tmp/bm25_index.pkl.new  /mnt/artifacts/artifacts/bm25_index.pkl
mv /tmp/corpus_meta.pkl.new /mnt/artifacts/artifacts/corpus_meta.pkl

The serving container picks up the new indexes on the next query. Zero restart. Zero redeploy. The mount sees the updated S3 objects immediately.

This is the killer feature not just cold start elimination, but live index updates without any deployment.

Results

The search UI running at http://98.93.65.241:8000:

Query: "aws sns permission issue" → 10 results in 334ms
Artifacts loaded at startup: instant (no download)
Index update: rebuild + upload to S3 → serving picks up without restart

What I Learned

S3 Files is genuinely useful for ML workloads where:

Artifacts are large (>100MB)
Multiple containers need the same data
Indexes update regularly without downtime
You want to avoid baking artifacts into Docker images

It's not a replacement for EFS if you need pure file system performance. And it's not for every use case if your artifacts never change and containers rarely restart, the complexity isn't worth it.

But if you've ever stared at a container downloading 500MB on every cold start, S3 Files is exactly what you were waiting for.

Resources

How I Built a Self Resizing EC2 for My ML Data

Haripriya Veluchamy — Sun, 07 Jun 2026 11:45:14 +0000

The Pain Point

I'm on an ML learning journey. That means a lot of data. A lot of processing. And a lot of AWS free credits I really can't afford to waste.

Here's what my typical day looked like:

I'd spin up a t3.large to run a data pipeline. Load some datasets, process them, store them. The pipeline would run for a couple of hours sometimes I didn't even know exactly how long it would take. Then I'd go to sleep.

Next morning I'd check CloudWatch and realise the VM had been sitting idle since 3AM. Running. Doing nothing. Burning credits.

That's the problem. You need a big machine for data loading. You don't need it after.

The First Thought Let AWS ML Decide

My first instinct was to use AWS Compute Optimizer. It's a managed ML service that analyses your EC2 usage patterns and recommends the right instance type. Smart, right?

I enabled it. Waited. And waited.

Turns out Compute Optimizer needs at least 30 consecutive hours of usage data before it generates recommendations. For a VM I spin up occasionally for pipeline runs that's not practical.

So I moved on.

The Second Thought Bedrock Agent

Next idea: use Amazon Bedrock as an agent to reason about when and how to resize. Let an LLM decide.

But the more I thought about it, the more it felt like overkill. The decision isn't complex:

"Did the pipeline finish? Yes → resize down. No → don't."

That's not a reasoning problem. That's an automation problem. Using Bedrock here would be AI washing adding complexity without adding value.

So I kept it simple.

The Solution EventDriven VM Resize

What I built: a lightweight agent that listens for your pipeline to complete, then automatically resizes the VM down.

Stack:

emit_event.py runs on your VM, fires when pipeline exits
Amazon EventBridge receives the event
AWS Lambda handles resize logic
Amazon SNS sends email alert (success or failure)

Pipeline finishes
      ↓
emit_event.py → EventBridge
      ↓
Lambda triggered
      ↓
SUCCESS → resize down + email
FAILURE → keep size (debug) + email

No ML. No LLM. Just the right tool for the job.

Architecture

Your VM                        AWS Cloud
──────────────────             ─────────────────────
run_pipeline.sh                EventBridge (custom bus)
  step1.py               →          ↓
  step2.py                     Lambda
  step3.py                       ├── stop EC2
  emit_event.py ──────────→      ├── resize instance type
                                 ├── start EC2
                                 └── SNS email alert

All AWS resources are deployed with a single CloudFormation command. No manual console clicking.

Setup 3 Steps

Step 1 Deploy AWS infrastructure

git clone https://github.com/Harivelu0/vm-resize-agent
cd vm-resize-agent

aws cloudformation deploy \
  --template-file infra/template.yaml \
  --stack-name vm-resize-agent \
  --capabilities CAPABILITY_NAMED_IAM \
  --parameter-overrides \
      AlertEmail=your@email.com \
      TargetInstanceType=t3.medium \
  --region us-east-1

Check your email → confirm the AWS subscription link.

Step 2 Copy agent to your VM

scp -i your-key.pem -r agent/ pipeline/ \
  ec2-user@your-vm-ip:~/vm-resize-agent/

# on VM
pip install boto3
aws configure

Step 3 Add your pipeline steps

Edit pipeline/steps.conf:

python3 /home/user/myproject/fetch_data.py
python3 /home/user/myproject/transform.py
python3 /home/user/myproject/load_db.py

Run:

bash pipeline/run_pipeline.sh

That's it. When pipeline finishes → VM resizes → email arrives.

The Key Design Decision steps.conf

One thing I was particular about: this tool should work for anyone's pipeline, not just mine.

So run_pipeline.sh never changes. Users only edit steps.conf one command per line, one step per line.

# steps.conf
python3 /home/user/fetch_gdelt.py
python3 /home/user/build_forecasts.py
python3 /home/user/calibrate.py

The wrapper reads each line, runs it in order, tracks success/failure, and emits the event at the end. Generic by design.

The Email Alert

On success:

Pipeline: data-loader
Status:   SUCCESS
Steps:    3/3
Duration: 45m 12s
Instance: i-0abc123
Resized:  Yes -> t3.medium

On failure:

Pipeline:  data-loader
Status:    FAILURE
Steps:     2/3
Failed at: step_3
Instance:  i-0abc123
Resized:   No (kept original size for debugging)

Failure case is important the VM intentionally stays large so you can SSH in and debug without losing state.

Difficulties I Faced

1. IP changes after resize

When Lambda stops and restarts the EC2, it gets a new public IP. Learned this the hard way when SSH stopped working. Fix: assign an Elastic IP before running demos.

aws ec2 allocate-address --region us-east-1
aws ec2 associate-address \
  --instance-id i-xxx \
  --allocation-id eipalloc-xxx

2. Docker goes down after resize

EC2 resize = reboot. Any running Docker containers stop. Add this to /etc/rc.local on your VM so services restart automatically:

sudo service docker start
cd /home/ec2-user/myproject && docker-compose up -d

Live Demo

For the demo I used a real weather dataset loaded into Postgres running in Docker on the EC2.

EC2: t3.large (before)
       ↓
Pipeline runs: download → parse → load into Postgres
       ↓
emit_event.py fires automatically when done
       ↓
Lambda: stops EC2 → resizes → starts EC2
       ↓
EC2: t3.medium (after)
       ↓
Email arrives in Gmail

Cost Reality

Resource	Monthly Cost
EventBridge	~$0 (free tier)
Lambda	~$0 (free tier)
SNS email	~$0 (first 1000 free)
Total stack	~$0

The savings depend on your instance. A t3.large idle for 20hrs/day wastes ~$25/month. For larger instances like c5.4xlarge you're looking at $200+ saved per month.

When to Use This

This is for you if:

You run data pipelines on EC2 manually or on a schedule
Your pipeline takes unpredictable time to complete
You don't need the heavy instance after loading is done
You're not ready for EMR or Glue yet (pre-pipeline stage)

This is not for you if:

You're already using EMR Serverless or Glue (they auto-terminate)
Your pipeline runs less than 30 minutes (manual resize is fine)
You need horizontal scaling (use Auto Scaling Groups instead)

Why a VM and Not Glue or EMR?

Honest answer I didn't know enough about my data yet to make that decision.

Glue and EMR are great services. But they come with questions you need to answer upfront:

What's your data format?
What transformations do you need?
What's the volume?
Do you need Spark?

When you're learning, you don't have those answers yet. You just need to load data, see what you're working with, and prove the pipeline works.

A VM lets you do that with zero infrastructure decisions. Just Python scripts. When the pipeline is proven and you understand your data then you migrate to the right managed service.

This is that phase. Before you know which service you need.

This is actually how real teams work too. Nobody starts with EMR on day one of a new data project. You explore first. You prove it works. Then you scale.

Phase 1: Explore         → VM + Python scripts
Phase 2: Prove it works  → VM + vm-resize-agent
Phase 3: Scale           → Glue / EMR / Spark

Repo

Everything is open source. Clone, edit steps.conf, deploy.

GitHub: https://github.com/Harivelu0/vm-resize-agent

What's Next

A few things I want to add:

Auto resize back up before next scheduled run (cron-based)
Slack alerts alongside email
Support for Azure VMs (same pattern, different SDK)

If you have ideas or run into issues open a GitHub issue. Happy to help.

Building in public as part of my ML learning journey. Follow along for more practical AWS patterns.

How I Built Production AI Agent Monitoring with Langfuse

Haripriya Veluchamy — Wed, 13 May 2026 16:32:02 +0000

Multi-agent AI systems fail silently.

A 200 OK response doesn’t mean the AI made good decisions.

That was the biggest thing I realized while building a multi-agent system.

My architecture looked like this:

User Query → Multi Agent Call → Final Response

Everything looked normal from an infrastructure perspective.

APIs were healthy
Latency looked fine
Users were getting responses

But I still couldn’t answer important questions:

Did the Agent route the query to the right specialist?
Did the agent hallucinate information?
Did it ignore specialist outputs?
Did it attribute responses incorrectly?

Traditional monitoring couldn’t help because the system technically wasn’t failing.

The failures were happening at the decision layer.

Full Trace Visibility

I used Langfuse to trace every agent execution.

That includes:

Tool calls
Input/output payloads
Token usage
Latency per step

If an agent touched something, I wanted visibility into it.

No black boxes.

Deterministic Checks

Some validations didn’t need another LLM.

I added rule-based checks for things like:

Did the agent call tools from the correct domain?
Did the agent call tools it wasn’t supposed to?
Was the expected workflow followed?

These checks are binary:

Pass → 1
Fail → 0

Fast and cheap.

Faithfulness Checks

This was mainly for hallucination detection.

I compare the final response with outputs from specialist agents.

If the Final layer introduces claims that weren’t exist in source outputs, it gets flagged.

This helped catch cases where the system sounded confident but wasn’t grounded.

LLM Judges

For things deterministic checks can’t measure, I use Azure OpenAI as judges.

They evaluate:

Routing correctness
Response quality
Attribution accuracy
Conflict handling

This runs for every multi-agent response.

Expensive? Yes.
Useful? Definitely.

100% Traffic Monitoring

I didn’t want sampling.

Every production request goes through the evaluation pipeline.

Because edge cases are usually the exact things sampling misses.

Cost + Latency Tracking

Multi-agent systems get expensive very fast.

I track:

Tokens per agent
Latency per step
Expensive execution paths

This made optimization much easier.

What This Actually Caught

This surfaced issues normal monitoring completely missed.

Wrong attribution

Correct insights were assigned to the wrong specialist.

Ignored outputs

Sometimes Agent completely ignored specialist responses.

Routing mistakes

The call occasionally sent queries to the wrong agent.

None of these showed up in normal monitoring dashboards.

Everything looked healthy.

Stack

Observability: Langfuse
LLM Evaluation: Azure OpenAI
Deterministic Checks: TypeScript

Final Takeaway

For multi-agent systems, uptime monitoring is not enough.

You also need decision monitoring.

Because a successful response can still be completely wrong.

I Built a Self-Updating SEO Brain Inspired by Andrej Karpathy's LLM Wiki

Haripriya Veluchamy — Sun, 10 May 2026 13:39:47 +0000

The Tweet That Changed How I Think About AI + Knowledge

In early April 2026, Andrej Karpathy (OpenAI co-founder, former Tesla AI Director) posted something deceptively simple:

"Something I'm finding very useful recently: using LLMs to build personal knowledge bases for various topics of research interest."

He followed it up with a GitHub gist titled LLM Wiki an "idea file" describing a pattern for building knowledge bases that actually compound over time instead of rediscovering the same information on every query.

I decided to build it for a real production problem.

The Problem: RAG Has No Memory

In My organization, we were running an SEO monitoring pipeline for our landing site. It used Cognee (a knowledge graph framework) backed by Neo4j + ChromaDB on a $35/month VM.

Every day it would:

Pull Google Search Console data
Scrape our pages for SEO issues
Query the knowledge graph
Post a Slack report

It worked. But it had a fundamental flaw the same flaw Karpathy describes in every RAG system.

Every time it ran, it was rediscovering knowledge from scratch. It had no memory of what it found yesterday. It couldn't connect "we deployed this fix on March 13" with "clicks went up 81% on March 18." It couldn't say "this keyword has been declining for 3 weeks here's why." It just answered the current query and forgot everything.

On top of that: the VM broke 3+ times. Neo4j config issues. ChromaDB API path changes. Sidecar containers failing silently. We spent more time fixing the pipeline than reading its output.

Karpathy's Core Insight

His idea is simple but changes everything:

Instead of retrieving from raw documents at query time compile knowledge once, keep it current, and query the compiled result.

The architecture has 3 layers:

raw/        → immutable source data (never edited)
wiki/       → LLM-maintained markdown knowledge base
AGENTS.md   → schema/rules file telling the LLM what to do

The LLM reads new raw data → integrates it into the wiki → cross-references it with existing knowledge → flags contradictions → the wiki gets richer every day.

His exact framing: "Obsidian is the IDE. The LLM is the programmer. The wiki is the codebase."

No vector database. No embeddings. No $35/month VM. Just markdown.

What I Built: LLM Wiki for SEO

I took Karpathy's pattern and applied it specifically to SEO monitoring for vibetrader.com.

The 3 Layers

raw/ : daily immutable snapshots:

raw/gsc/2026-04-17.json      ← Google Search Console (clicks, CTR, positions, queries)
raw/audit/2026-04-17.json    ← page audit (H1, meta, schema, canonical checks)
raw/commits/2026-04-17.json  ← landing-site git log (what code changed)

wiki/ : LLM-maintained knowledge:

wiki/overview.md              ← "story so far" updated daily
wiki/log.md                   ← append-only daily log
wiki/topics/keywords.md       ← keyword clusters + position tracking
wiki/topics/issues.md         ← open SEO issues with severity
wiki/topics/recommendations.md ← history of recs + acted/pending status
wiki/topics/code-changes.md   ← code change impact tracker
wiki/topics/performance.md    ← CAUSAL CHAINS: issue → fix → metric improvement
wiki/topics/competitors.md    ← weekly competitor analysis
wiki/topics/lint-report.md    ← weekly wiki health check

AGENTS.md : the schema file. Tells the LLM exactly how to update each page, what cross-links to write, what format to follow.

The Flow

3:00 PM IST  daily
─────────────────
gsc_pull.py    → pulls GSC data via API → raw/gsc/today.json
audit.py       → fetches vibetrader.com, checks H1/meta/schema → raw/audit/today.json
               → posts Slack audit digest (SEO/GEO/AEO scores)
ingest.py      → reads ALL raw/ + ALL wiki/
               → sends full context to Azure OpenAI
               → LLM updates all wiki pages with cross-links
               → appends to log.md
               → posts enriched Slack report
git commit     → wiki/ + raw/ committed back to main

4:00 PM IST  daily (1 hour after ingest)
─────────────────────────────────────────
fix_agent.py   → reads wiki/issues + recommendations
               → LLM classifies which issues are auto-fixable
               → skill functions run deterministically (not LLM-generated code)
               → raises PR to landing-site dev branch
               → never auto-merges

11:30 AM IST  Saturday
───────────────────────
lint.py        → audits wiki for staleness, contradictions, orphan pages
               → health score /100
               → saves lint-report.md + posts Slack

2:30 PM IST  Sunday
────────────────────
competitor_analysis.py → scrapes competitor sites
                       → LLM analysis → updates competitors.md

The Knowledge Graph in Plain Markdown

The key innovation over standard LLM Wiki I added a performance.md page that tracks causal chains:

### ✅ 2026-03-18 — API response fix
- **Cause:** [[topics/issues]] API returning incomplete pagination data
- **Fix:** API response items + pagination as separate fields → [[topics/code-changes]]
- **Before:** 127 clicks/day
- **After:** 229.7 clicks/day
- **Impact:** +81% clicks
- **Keywords moved:** [[topics/keywords]] — all branded queries improved
- **Confidence:** HIGH — spike confirmed same week as deploy

Every entry answers: what caused this? what moved? how confident?

This is what Cognee was trying to do with Neo4j store causal relationships between entities. We're doing it in plain markdown, with [[wiki-links]] that Obsidian renders as a visual knowledge graph.

Obsidian as the Visualization Layer

Open Obsidian → point it at your wiki/ folder → graph view instantly shows all cross-connections between pages:

performance ↔ keywords ↔ issues ↔ recommendations ↔ code-changes

As the wiki accumulates data daily, the graph gets richer. After 30 days, clicking any node shows a real story: "this keyword dropped because of this issue, which was fixed by this code change, which resulted in this metric improvement."

That's the knowledge graph. No Neo4j required.

Interactive Query Mode

Beyond the daily Slack push, I added a local query script:

# one-shot
python scripts/query.py "what keywords are improving this week?"

# REPL with follow-up context
python scripts/query.py
you> which code change had the most SEO impact?
wiki> Based on code-changes.md:
      March 18 — API response fix
      Before: 127 clicks/day | After: 229.7 clicks/day | Impact: +81%
      This is the highest confirmed impact change in the wiki.

you> why did that happen?   ← follow-up, remembers context
wiki> ...

The LLM answers from the accumulated wiki context not by re-reading 30 days of raw JSON. This is exactly what Karpathy's "query" operation describes.

Weekly Lint Pass

One gap in most LLM Wiki implementations no health check. I added lint.py that runs every Saturday:

Staleness detection : claims in the wiki that contradict newer log entries
Contradiction detection : two pages saying opposite things
Orphan pages : pages with no cross-links pointing to them
Missing concept pages: [[wiki-links]] referenced but never created
Stale recommendations: recs pending > 30 days

Posts a health score /100 to Slack and saves lint-report.md.

The Fix Agent (Skill-Based, Not LLM-Generated Code)

The fix agent reads the wiki, identifies fixable issues, and raises PRs to the landing-site repo. The key design decision: the LLM only classifies issues it never writes code.

Instead, deterministic "skill" functions handle each fix type:

skills/add_jsonld.py          ← checks schema not already present before adding
skills/add_internal_link.py   ← validates page exists in app/ first
skills/fix_meta_tags.py       ← only updates if explicit value given
skills/fix_robots_txt.py      ← safe append-only sitemap fix

The LLM outputs {"skill": "add_jsonld", "page": "app/layout.tsx"}. The skill runs. No hallucinated file paths. No duplicate schemas. No business logic touched.

PRs always target dev branch. Never auto-merge. Human review required.

vs Cognee: Honest Comparison

After running both systems in parallel for a week:

	Cognee	LLM Wiki
Infrastructure	Neo4j + ChromaDB + VM	Plain markdown files
Monthly cost	~$35	~$0 (just Azure OpenAI tokens)
Times it broke	3+	0
Files to maintain	6 JS pipeline files	5 Python scripts
History awareness	Inconsistent	Grounded references exact dates
AEO accuracy	3/10 (missed JSON-LD already added)	7/10 (reflected actual state)
Knowledge compounds	❌	✅

LLM Wiki won on every dimension except one: Cognee had richer graph relationships out of the box. But we replicated that with performance.md and [[wiki-links]].

What Karpathy Got Right

The core insight is correct and the more I use it the more obvious it becomes:

RAG = studying for an exam by re-reading all your textbooks every time you get a question.

LLM Wiki = you already made notes, highlighted the important parts, drew arrows between connected ideas. Now you just read your notes.

The wiki grows daily. The LLM gets smarter about your domain the more data accumulates. By day 30, query.py answers questions with 30 days of grounded context. By day 90, performance.md has a dozen confirmed causal chains real institutional knowledge about what moves your metrics and why.

That's what no RAG system gives you. And you don't need a vector database to get there.

Try It Yourself

The pattern works for any domain where you're accumulating knowledge over time:

SEO monitoring (what I built)
Competitor tracking
Research — papers, articles, building a thesis over weeks
Engineering team wiki — fed by Slack threads, PRs, incident reports
Personal second brain — journal entries, articles, book notes

The only thing you need:

A raw/ folder where data lands
A wiki/ folder for LLM-maintained markdown
An AGENTS.md (or CLAUDE.md) telling the LLM the rules
A cron job that runs ingest daily

Start with one raw data source and one wiki page. Let it run for a week. The compounding effect becomes obvious fast.

I built a 20-20-20 eye reminder because my eyes were dying at my desk

Haripriya Veluchamy — Thu, 16 Apr 2026 08:45:00 +0000

I have severe eye dryness. My doctor told me to follow the 20-20-20 rule every 20 minutes, look at something 20 feet away for 20 seconds. Simple. Except when you're deep in a bug or a feature, 20 minutes evaporates and you've been staring at your monitor for 3 hours straight.

I tried phone reminders. I dismissed them without thinking. I tried sticky notes. I ignored them. I needed something that actually blocked me from working until I did the break.

The key insight: a dismissible reminder is just noise. The popup had to be impossible to close until the 20-second countdown finished.

What it does

Runs silently in the system tray in the background
Every 20 minutes fires a loud beeping alarm (winsound, no external files needed)
Dark blocking popup appears X button is disabled, you cannot close it
20-second countdown runs automatically with a progress bar
The "Continue" button is locked until the countdown finishes, then turns green
Two buttons: Continue (back to work) or Stop (end the session)

The core trick: disabling the close button

The single most important line in the whole app:

win.protocol("WM_DELETE_WINDOW", lambda: None)

By overriding WM_DELETE_WINDOW with a no-op lambda, clicking the X does absolutely nothing. The window stays open. You have to wait out the countdown.

The countdown + locked button

The "Continue" button starts state="disabled" and only becomes clickable when the timer hits zero:

def tick():
    remaining["s"] -= 1
    count_var.set(str(max(remaining["s"], 0)))

    if remaining["s"] <= 0:
        continue_btn.configure(
            state="normal",
            bg="#003322",
            fg="#44ff88",
            cursor="hand2"
        )
    else:
        after_id["id"] = win.after(1000, tick)

The button turns green when it unlocks a satisfying visual reward for actually doing the break.

The alarm

No external audio files needed. winsound is part of the Windows standard library:

for _ in range(6):
    winsound.Beep(1000, 300)
    time.sleep(0.1)
    winsound.Beep(1400, 300)
    time.sleep(0.1)

Two alternating frequencies, six times. Annoying enough that you cannot ignore it. Runs in a daemon thread so it does not block the UI.

The full popup

def show_popup():
    result = {"action": None}
    root = tk.Tk()
    root.withdraw()

    win = tk.Toplevel(root)
    win.title("Eye Break!")
    win.configure(bg="#0a0a0a")
    win.attributes("-topmost", True)
    win.protocol("WM_DELETE_WINDOW", lambda: None)  # <-- the magic line

    # ... countdown, progress bar, locked button ...

    win.after(1000, tick)
    play_alarm()
    root.mainloop()
    return result["action"] == "continue"

The popup is always-on-top (-topmost True) so it cannot be buried under other windows.

Setup: auto-start on Windows boot

Step 1 Install dependencies:

pip install pystray pillow

Step 2 Save eye_reminder.py to a permanent location:

C:\Users\you\EyeReminder\eye_reminder.py

Step 3 Create a startup shortcut. Press Win+R, type shell:startup, and create a shortcut there pointing to:

pythonw C:\Users\you\EyeReminder\eye_reminder.py

Using pythonw instead of python means no terminal window appears on startup.

Step 4 Reboot. The app silently starts in your system tray. Right-click the tray icon to quit anytime.

Testing it

Before running with 20-minute intervals, test with 5 seconds:

TEST_MODE = True
INTERVAL_SECONDS = 5 if TEST_MODE else 20 * 60
BREAK_SECONDS = 5 if TEST_MODE else 20

Flip TEST_MODE = False when you are happy with it.

Dependencies

Package	Why
`tkinter`	Built-in — popup UI
`winsound`	Built-in — alarm beeps
`pystray`	System tray icon
`Pillow`	Required by pystray for the icon image

Core functionality (popup + alarm) works with zero installs. pystray and Pillow are optional the app works without them, just without the tray icon.

Full source

Harivelu0 / eye-remainder

eye-reminder

A lightweight Python app that enforces the 20-20-20 rule for eye health on Windows.

Every 20 minutes, a blocking popup appears with a loud alarm. You cannot dismiss it until a 20-second countdown finishes. No cheating.

Why

The 20-20-20 rule says: every 20 minutes, look at something 20 feet away for 20 seconds. Every reminder app I tried was too easy to dismiss. This one isn't.

Demo

[20 min timer] --> ALARM fires --> Blocking popup appears
                                        |
                                   20-sec countdown
                                        |
                               "Continue" button unlocks
                                        |
                              Click Continue --> back to work
                              Click Stop    --> session ends

Features

Runs silently in the system tray
Loud beeping alarm on every trigger (no audio files needed)
Popup is always-on-top and the X button is disabled
"Continue" button locked until full 20-second countdown completes
System tray icon with right-click Quit option
Built-in test mode (5-sec interval) for quick verification

Requirements

Windows
Python 3.8+
…

View on GitHub

My eyes are noticeably less dry after two weeks of using this. Sometimes the best tool is the one you build in an afternoon because nothing else works the way you need it to.

I Built an AI SEO Monitor That Remembers Everything

Haripriya Veluchamy — Fri, 10 Apr 2026 14:16:39 +0000

Most SEO monitoring tools give you a snapshot: today's clicks, today's issues, today's recommendations. You fix something, come back tomorrow, and the tool has no idea what you did or whether it helped.

I wanted something smarter a system that remembers site's history, correlates code changes with ranking shifts, and gives AI-generated insights that get better every day. Here's what I built and how.

The Problem With Basic SEO Monitoring

A typical monitoring setup looks like this:

Fetch Google Search Console data
Run a Lighthouse audit
Send a Slack message with today's numbers

That's fine. But it can't answer questions like:

"Did that metadata fix I deployed 10 days ago actually improve rankings?"
"This recommendation has been flagged for 15 days why hasn't it been fixed?"
"Clicks dropped this week was it a code change or an algorithm shift?"

To answer those questions, you need memory. That's where Cognee comes in.

What Is Cognee?

Cognee is a knowledge graph SDK. Instead of storing data as flat rows in a database, it extracts entities and relationships and stores them as nodes and edges in a graph (Neo4j) with vector embeddings in a vector database (ChromaDB).

Think of it like this: a normal database stores "clicks = 262 on April 8". A knowledge graph stores "keyword 'vibe trading' ranked at position 1.78 on April 8, which is 12 spots better than March 25, and that improvement happened 3 days after a metadata fix was deployed".

The difference matters when you want AI to reason across weeks of history not just today.

System Architecture

The full pipeline runs daily via GitHub Actions:

PHASE 1 — Parallel data collection
├── Lighthouse audit (performance, SEO scores)
├── Broken links check
├── Meta tags validation
├── Core Web Vitals (via PageSpeed API)
└── Google Search Console (clicks, CTR, position, queries)

PHASE 2 — Main analysis job
├── git-change-detector.js    → scans commits, classifies SEO-relevant changes
├── cognee_ingest.py          → writes today's data to Neo4j + ChromaDB
├── cognee-store-updater.js   → updates 30-day rolling JSON snapshot
├── audit-scraper.js          → fetches live pages, scores SEO/GEO/AEO signals
├── audit-ingest.py           → stores audit scores in the knowledge graph
├── cognee-analyzer.js        → builds enriched AI context, calls Azure OpenAI
├── send-ai-slack.js          → posts daily report to Slack
└── cognee-blob-sync.js       → backs up knowledge graph to Azure Blob Storage

PHASE 3 — Weekly (Sundays)
└── competitor-monitor.js     → fetches competitor pages, scores them, posts comparison

Why Cognee? The Knowledge Graph Advantage

Every day, cognee_ingest.py builds a structured document containing today's GSC metrics, top queries, AI recommendations, and recent git commits. Azure OpenAI reads this and extracts entities keywords, positions, dates, code changes which Cognee writes to Neo4j as connected nodes. graph starts to grow.

After 30 days, the graph contains nodes like:

(Keyword: "platform name") —[RANKED_AT]→ (Position: 1.78, date: April 8)
(CodeChange: "metadata fix") —[HAPPENED_BEFORE]→ (MetricSnapshot: April 5)
(Recommendation: "add FAQ schema") —[FLAGGED_ON]→ (Date: April 1)
(Recommendation: "add FAQ schema") —[FLAGGED_ON]→ (Date: April 2)
... (flagged 12 days in a row)

Now when the AI runs its daily analysis, it doesn't just see today's data. It sees patterns:

Keyword velocity: which keywords improved or dropped more than 5 positions in 14 days
Stuck recommendations: same issue flagged 3+ days in a row, still unactioned
Code change impact: did clicks or position change after a specific deploy?

The Slack report reflects this. Instead of "your CTR is 18.94%", it says:

"Your site has more than doubled daily clicks over the past month (106% growth), driven by a metadata fix on March 26 and header overlap fixes on March 28. Short-term momentum is slowing 7-day clicks are -3% suggesting you need to now expand content around fast-moving branded keywords."

That's a different class of insight.

The Audit Scrape SEO/GEO/AEO Scoring

Beyond GSC data, audit-scraper.js fetches your actual pages daily and scores them across three dimensions:

SEO classic signals: title tag, meta description, H1, canonical, OG tags, schema markup, JS-gated content detection

GEO (Generative Engine Optimization) how well AI search engines like Perplexity or ChatGPT Search can read and cite your content: structured data presence, content density, crawlability

AEO (Answer Engine Optimization) featured snippet and voice search readiness: FAQ schema, article schema, H2 density, word count

Each page gets a score out of 10. The system flags critical issues (JS-gated content = crawlers see a blank page, missing H1 = no primary ranking signal) and sends a separate Slack message with the audit digest.

🔴 SEO Audit — 2026-04-08
Scores: SEO 9/10 | GEO 9/10 | AEO 3/10 | Combined 21/30

Critical Issues:
🚨 missing_h1 on Pricing — Missing primary ranking signal
🚨 js_gated_content on Pricing — Crawlers see blank page

Code Change Impact Tracking

This is the part I'm most proud of. git-change-detector.js scans git commits and classifies them it looks for commit messages mentioning SEO-related terms (metadata, schema, redirect, canonical, performance, etc.) and logs them with their date.

change-impact-tracker.js then cross-references those commits with GSC metrics. For each logged change, it compares the 7-day window before vs after deployment:

✅ Migrate to new partition keys (2026-03-30)
   → Position improved 6.7 spots (17.39 → 10.71)

✅ API pagination fix (2026-03-18)
   → Clicks grew 81% (127 → 229.7/day)

⏳ content deploy (2026-03-13)
   → Monitoring... (not enough post-deploy data yet)

This surfaces directly in the Slack report under "Code Change Tracker". Over time, it tells you which types of changes actually move the needle.

Storage Architecture

Three layers, each with a different purpose:

Layer	What it stores	Why
Neo4j (Azure VM)	Graph nodes + edges — keywords, positions, code changes, relationships	Multi-hop reasoning: "which keyword improved after which deploy?"
ChromaDB (Azure VM)	Vector embeddings of all entities	Semantic search across history
cognee-knowledge.json (Azure Blob)	30-day rolling JSON snapshots	Fast daily reads without querying the graph every run

The JSON file is the workhorse for the daily Slack report. Neo4j and ChromaDB are queried for deeper pattern analysis and become increasingly valuable as history accumulates.

Key Things I Learned

Cognee initializes config at import time. If you set environment variables after import cognee, they're ignored. You have to call cognee.config.set_graph_db_config() directly after import to update the live config object. This cost me several hours.

The mistralai import conflict. Cognee's dependency instructor==1.14.x tries to import Mistral from mistralai at import time regardless of whether you use it. Fix: inject a fake mistralai module into sys.modules before importing Cognee.

JS-gated content is invisible to the audit scraper. If your page renders entirely client-side, the raw HTML fetch returns fewer than 80 words. The scraper flags this as js_gated_content — which is actually useful because it means Google probably can't index it either.

The knowledge graph gets smarter non-linearly. Day 1 the system is just a fancier GSC dashboard. Day 7 you start seeing real code change verdicts. Day 30 the AI recommendations start referencing patterns that span weeks. The value compounds.

Tech Stack

GitHub Actions — pipeline orchestration, daily cron
Node.js — audit scraper, Cognee analyzer, Slack formatting, git change detection
Python — Cognee SDK ingestion (cognee_ingest.py, audit-ingest.py)
Cognee 0.5.3 — knowledge graph SDK
Neo4j Community — graph database
ChromaDB — vector database
Azure OpenAI — GPT-4.1 for analysis, text-embedding-3-large for vectors
Azure Blob Storage — knowledge graph backup/restore
Azure VM (Standard B2s) — hosts Neo4j + ChromaDB via Docker Compose
Google Search Console API — real click/impression/position data

🚀 Beyond RAG: Simulating the Future with MiroFish

Haripriya Veluchamy — Tue, 07 Apr 2026 17:15:58 +0000

Lately, most of us have been working with RAG systems retrieving context, grounding responses, improving accuracy.

But what if instead of just retrieving knowledge, we could simulate outcomes?

I recently came across MiroFish, and decided to test it out.

🧪 What I Tried

I cloned the repo, ran it locally, and fed it a simple scenario:

“What happens when an AI assistant is introduced into a company’s daily workflow?”

Instead of a static answer, it generated a multi-agent simulation over time.

🧠 What Makes It Different

Unlike traditional systems, MiroFish:

Creates a virtual environment
Generates multiple agents (employees, managers, etc.)
Simulates interactions over time
Produces a temporal report (day-by-day evolution)

This means you’re not just asking:

“What will happen?”

You’re observing:

“How things evolve step by step.”

📊 Sample Insights from My Test

From a 14-day simulation, I observed:

📈 Initial boost in productivity
⚖️ Diverging employee satisfaction
🔁 Emerging dependency on AI
🧩 Different behaviors across teams

It felt less like querying an LLM… and more like watching a system evolve.

💡 Where This Can Be Useful

This kind of simulation opens up interesting possibilities:

🏢 Organization & Product

AI adoption strategies
Remote work policy changes
Feature rollout impact

📦 Business Decisions

Pricing experiments
Customer behavior prediction
Growth strategy testing

🌍 Macro Scenarios

Economic shifts
Supply chain disruptions
Policy or geopolitical changes

🔄 RAG vs Simulation (My Take)

Approach	What it does
RAG	Retrieves and explains existing knowledge
Simulation (MiroFish)	Models and predicts possible futures

Both are powerful but they solve very different problems.

⚡ Final Thoughts

We’re slowly moving from:

👉 “Answering questions”
to
👉 “Rehearsing decisions”

MiroFish feels like an early step in that direction.

Still experimenting, but this approach definitely opens up a new way of thinking about AI systems.

If you’ve tried something similar or have ideas for scenarios to test — would love to hear 👇

Everyone Suddenly Said “RAG is Dead”

Haripriya Veluchamy — Sat, 04 Apr 2026 13:41:22 +0000

Lately I keep seeing this everywhere:

“RAG is dead”
“Vector search is outdated”
“Reasoning-based retrieval is the future”

And suddenly… everyone is talking like vector search is useless.

I’m not against the hype. These things happen.

But honestly, this whole idea didn’t just click for me immediately.

Because for me, this problem was already in my head for a long time.

Not because of hype.

Just because of my use case.

What I Was Actually Trying to Figure Out

I read a lot of long tech blogs and architecture posts.

After reading, I always have questions like:

“why did they do this?”
“what’s the tradeoff here?”
“what happens if we change this design?”

So I wanted a system where I can:

paste a document
ask questions
actually get useful answers

At some point I started thinking:

should I just stick with vector RAG?
or should I try something like PageIndex / reasoning-based retrieval?
or even something like Agent-style flow later?

That curiosity is what pushed me to build this.

Not the “RAG is dead” trend.

So I Built a Simple Comparison

Nothing fancy.

Just a small app where:

same document
same question
same model
only retrieval changes

Pipeline 1 — Vector RAG

split document
embed
store in ChromaDB
retrieve top-k
answer

This is what most of us are already doing.

Pipeline 2 — PageIndex

build a tree structure from the document
let the model navigate it
pick relevant sections
answer from that

This felt very different.

Not “searching”.

More like… “reading with guidance”.

What I Noticed

The difference is actually deeper than I expected.

Vector RAG:

find similar chunks

PageIndex:

figure out where the answer should be, then go there

That “where” part is interesting.

One Example

I tested with a Netflix architecture article.

Question:

Why did they use live origin instead of only CDN?

Vector RAG

faster (~7s)
decent answer
but retrieval had some noise

PageIndex

slower (~11s)
answer felt more precise
citations were cleaner

My Honest Take

Vector RAG is not dead.

But…

Blind chunking + embedding + top-k is not enough anymore (at least for some cases).

Where I See the Difference

Vector RAG works well when:

you have multiple documents
you want speed
you just need “good enough” answers

PageIndex works well when:

single long document
structured content
you want cleaner reasoning

What I’m Actually Thinking Now

I don’t think this is:

“one replaces the other”

Feels more like:

both solve different parts of the problem

What I’m more interested in now is:

can I combine them?
use vector search to find documents
then use something like PageIndex inside that?

That feels more practical.

Why I’m Exploring This

For my use case, I’m also thinking about:

can I plug this into an agent flow later?
how does retrieval affect agent decisions?
does better retrieval reduce hallucination in multi-step tasks?

That’s where this is going for me.

Final Thought

Honestly, I didn’t build this to prove anything.

Just to understand.

And one thing became clear very fast:

hype says “X is dead”
reality says “it depends”

If you’re building something similar, I’d really suggest:

Don’t pick a side early.

Test both.

You’ll understand the difference immediately.

Harness Engineering: The Concept I Didn't Know I Needed

Haripriya Veluchamy — Wed, 25 Mar 2026 18:24:15 +0000

Honestly, when I first heard the term Harness Engineering, I thought it was just another buzzword.

I already knew about Prompt Engineering. I had heard about Context Engineering. I thought, okay this is probably just the same thing with a fancier name.

But then I started actually using agentic tools like Cursor and Windsurf in my day-to-day work. And something clicked.

"Wait... this thing is not just answering my question. It's planning, building, testing, fixing — all on its own. How?"

That's when I went deeper. And what I found actually changed how I think about building with AI.

First What Even is a Context Window?

Before we get into Harness Engineering, need to understand one thing.

Every AI model has something called a context window. Think of it like a whiteboard. The model can only see what's written on that whiteboard right now. Once the conversation gets too long, old stuff disappears. And when you start a brand new chat the whiteboard is completely blank.

That's the core problem:

AI has no memory between sessions. Every new session, it starts fresh.

For a simple question answer task, that's fine. But what if the task takes days?

What is Harness Engineering?

Let me show you how this concept evolved:

Prompt Engineering   → How do I ask better questions?
Context Engineering  → How do I manage what's inside one session?
Harness Engineering  → How do I make an agent work across many sessions?

Harness Engineering is not about writing better prompts. It's about designing the system around the model so the agent always knows where it is, what it has done, and what it needs to do next. Even after the context window resets completely.

The Moment I Really Got It

When I was exploring how tools like Cursor work under the hood, I realized something.

When Cursor builds a feature for you:

It scans your codebase
Makes a plan
Implements step by step
Runs tests automatically
Fixes bugs it finds
Continues without you prompting every single move

That is Harness Engineering. The tool is not just "smart." Someone designed a system that makes it stay on track even as context windows reset.

A Real Example: Building an App with an Agent

Let's say you ask an AI agent:

"Build me a complete Food Delivery App."

This is not a one-session task. Here's what happens without any harness:

Session 1:
Agent builds Login page, starts Restaurant list...
Context window fills up. Stops.

Session 2:
Agent starts fresh. No memory.
Builds Login page again. 😵
Duplicate code. Broken app. Confused agent.

Now with Harness Engineering:

Before any coding starts, an Initializer Agent sets up three simple things:

features.json — Every task with a status:

[
  { "task": "Login Page", "status": "pending" },
  { "task": "Restaurant List", "status": "pending" },
  { "task": "Cart System", "status": "pending" }
]

progress.txt — A running log:

Last completed: Nothing yet
Next task: Login Page

setup.sh — A script to spin up the dev server automatically.

Now every new session just does this:

Read progress.txt  → know where to continue
Read features.json → pick the next pending task
Run setup.sh       → environment is ready
Build → Test → Update files → Git commit
Session ends cleanly. Next session picks up exactly here.

The agent has no memory but it doesn't need memory. The system remembers for it.

The 3 Things That Actually Make This Work

1. Legible Environment

Every session should be able to answer three questions just by reading files:

What is the goal?
What is done?
What is next?

Feature lists, progress logs, git history, docs — these are not optional. They are the foundation.

2. Verification Before Moving On

Agents have a habit of saying "Done!" when things are actually broken. I've seen this personally with Claude Code and Cursor.

The fix is giving the agent real tools to test its own work like running the app, checking the UI, catching bugs end to end. Not just saying it worked. Actually proving it.

3. Use Simple, Familiar Tools

This one surprised me the most.

Vercel built a very fancy, specialized agent with custom tools and heavy prompt engineering. It worked but barely. Fragile. Slow.

Then they removed almost all the custom tools and replaced everything with one simple batch command tool.

Result? 3.5x faster. 37% fewer tokens. Success rate went from 80% to 100%.

Why? Because models like Claude have seen billions of lines of code using git, grep, npm. They understand these natively. Custom tools are unfamiliar territory.

Simple tools the model already knows > Fancy tools you built from scratch.

How This Connects to MCP

If you've worked with MCP (Model Context Protocol) before, this connects directly.

In a Harness Engineering setup:

The Host (Claude Desktop, Cursor) is your computer
The MCP Client is like an adapter built into the host, you don't touch it
The MCP Server is what you build your custom tools, your file readers, your test runners

Your MCP Server becomes the hands of your long-running agent. It reads progress files, runs tests, queries databases, and verifies work all between sessions.

You only build the server. The host handles the rest.

Common Mistakes to Avoid

Letting the agent "one-shot" the whole task it will run out of context and leave things half done
Not giving the agent a way to test its own work it will always claim success
Building overly specialized tools simpler is almost always better
No clean state at end of each session the next session will be confused

A Simple Harness Checklist

Before building a long-running agent system, make sure:

[ ] Feature list exists with pass/fail status per task
[ ] Progress file updated at end of every session
[ ] Git commits made with descriptive messages
[ ] Dev environment spins up automatically (setup script)
[ ] Agent has real testing tools not just unit tests
[ ] Generic tools used wherever possible

Final Thoughts

The models today are genuinely capable. The missing piece is almost never the model itself.

It's the system around it.

That's what Harness Engineering is. Not a new model. Not a new prompt trick. Just smart system design that lets an agent stay on track across sessions, verify its own work, and actually finish what it started.

Once I understood this, the way I think about building AI-powered tools completely changed.

If you're building anything agentic even something small think about what happens when the context window resets. Does your agent know how to pick up where it left off?

If yes, you're already doing Harness Engineering. 😊

DEV Community: Haripriya Veluchamy

Vault cant found What a Live Audit Found That Code Review Never Would

The setup

Building the audit, properly this time

What it found

How did nobody notice?

The part that actually surprised me

What I took away from this

Why Your CI Agent Can't Install pip

The setup

Attempt 2: ensurepip

The actual reason (worth understanding, not just working around)

Attempt 3: venv (also fails, same root cause)

Attempt 4: the actual fix get-pip.py

The one thing to double-check before you assume this is your problem

The takeaway

Why AI Can't Actually Hit Your Word Limit

It generates one token at a time, no plan

Words ≠ tokens

No counter anywhere in the architecture

The workaround

Why this clicked for me

Serving ML Artifacts from Amazon S3 Files How I used After the Launch

The Honest Story

What I Built

The Problem S3 Files Solves

Architecture

Setting Up S3 Files What Actually Matters

1. S3 bucket needs versioning enabled

2. IAM role trust principal is elasticfilesystem.amazonaws.com

3. The IAM role needs EventBridge permissions

4. boto3 API uses camelCase

5. Mount needs amazon-efs-utils, not plain NFS

6. EC2 instance needs AmazonS3FilesClientFullAccess

The Serving Code

Weekly Index Updates The Real S3 Files Advantage

Results

What I Learned

Resources

How I Built a Self Resizing EC2 for My ML Data

The Pain Point

The First Thought Let AWS ML Decide

The Second Thought Bedrock Agent

The Solution EventDriven VM Resize

Architecture

Setup 3 Steps

Step 1 Deploy AWS infrastructure

Step 2 Copy agent to your VM

Step 3 Add your pipeline steps

The Key Design Decision steps.conf

The Email Alert

Difficulties I Faced

Live Demo

Cost Reality

When to Use This

Why a VM and Not Glue or EMR?

Repo

What's Next

How I Built Production AI Agent Monitoring with Langfuse

Full Trace Visibility

Deterministic Checks

Faithfulness Checks

LLM Judges

100% Traffic Monitoring

Cost + Latency Tracking

What This Actually Caught

Wrong attribution

Ignored outputs

Routing mistakes

Stack

Final Takeaway

I Built a Self-Updating SEO Brain Inspired by Andrej Karpathy's LLM Wiki

The Tweet That Changed How I Think About AI + Knowledge

The Problem: RAG Has No Memory

Karpathy's Core Insight

What I Built: LLM Wiki for SEO

The 3 Layers

The Flow

The Knowledge Graph in Plain Markdown

Obsidian as the Visualization Layer

Attempt 2: `ensurepip`

Attempt 3: `venv` (also fails, same root cause)

Attempt 4: the actual fix `get-pip.py`

2. IAM role trust principal is `elasticfilesystem.amazonaws.com`

5. Mount needs `amazon-efs-utils`, not plain NFS

6. EC2 instance needs `AmazonS3FilesClientFullAccess`