DEV Community: Haripriya Veluchamy

One Lookup Table Turned a Risky Server Migration Into a One-Line Change

Haripriya Veluchamy — Tue, 18 Aug 2026 16:28:54 +0000

The setup

I was running a system where multiple backend servers each served a set of users, and a routing layer decided which server a given user's requests should go to. Nothing unusual this is a common shape for anything that scales past one machine.

The question that turned out to matter a lot more than I expected: when something in the routing layer refers to "the server," what is it actually storing? A real address? Or a name that gets looked up?

The moment it mattered

At some point I needed to move a group of users off one server and onto another the kind of thing that happens for all sorts of reasons: retiring old hardware, rebalancing load, recovering from an incident.

Before touching anything, I checked how the routing data was actually structured. It turned out every record referenced servers by a stable label not a raw address. The label was just an ID; a separate lookup table mapped each label to wherever that server actually lived right now.

That one fact changed the entire migration from "rewrite a bunch of records for every affected user" into "update one row in the lookup table." The routing records themselves never needed to change. Every user still pointed at the same label the label's meaning just changed.

I verified it worked by checking the routing behavior for an affected user right after the update, before assuming anything.

The pattern, generalized

This is the indirection pattern, and it's worth naming explicitly because it's easy to skip when you're building the first version of something:

Direct reference: Store the real, resolvable thing (an IP address, a file path, a specific resource ID) everywhere it's used.
Indirect reference: Store a stable name/label everywhere it's used, and keep exactly one place that resolves that label to the real thing.

Direct references feel simpler at first there's no lookup step, nothing extra to maintain. But that simplicity is a trap: the real address ends up copied into every place that uses it, and moving the real thing means finding and updating every one of those places.

Indirect references cost you one extra lookup table. In exchange, moving the real thing becomes a single update, in a single place, and everything downstream is unaffected because none of it ever knew the real address to begin with.

Where this shows up beyond servers

The same trade-off appears constantly:

DNS names vs. hardcoded IPs this is the same pattern at internet scale. Change the DNS record, nothing downstream needs to know.
Feature flags referencing a config key vs. a hardcoded value the flag's name stays stable while what it resolves to changes.
Database foreign keys vs. duplicating a denormalized value everywhere the ID stays stable; look up the current details when you need them.
Service discovery vs. hardcoded service addresses exactly the migration scenario above, generalized to any service-to-service call.

In every case, the question is the same: if the "real thing" changes, how many places need to be touched? If the answer is "more than one," you're using a direct reference somewhere it should have been indirect.

The lesson

Before building the first version of anything that will eventually need to move, scale, or be replaced a server, a config value, a downstream dependency decide up front whether callers will reference it directly or through a label that gets resolved. It's a small design decision early on, and it's the single biggest factor in whether a future migration is a five-minute change or a project.

If you're not sure whether you have this in place already, ask the question I asked before touching anything: what is actually stored in the place that decides where a request goes a real address, or a name? If it's a real address, that's worth fixing before you need to move it under pressure.

Multi region deployment with Terraform modules

Haripriya Veluchamy — Thu, 13 Aug 2026 16:59:01 +0000

The problem

I was managing infrastructure across multiple regions a handful of virtual machines, one per region, each running the same service. The Terraform setup had grown the way these things usually do: when I needed a VM in a new region, I copied the folder for an existing region, renamed a few things, and adjusted the values that were different (location, network ranges, naming).

It worked. Until it didn't.

Every region had its own full copy of:

The provider block
The resource definitions (VM, network interface, disk, public IP, etc.)
The output block

Three regions in, I had three nearly-identical folders, each a few hundred lines, differing in maybe ten lines of actual content. Any time I wanted to change something structural add a tag, adjust a disk size, fix a naming convention I had to make that change in every single folder, and hope I didn't miss one or introduce a subtle inconsistency between them.

This is the classic copy-paste infrastructure trap: it feels fast in the moment (just copy the folder!) but every region you add makes the next change more expensive, not less.

The pattern: separate "what's shared" from "what's different"

The fix was to stop treating each region as its own Terraform project and instead treat the infrastructure definition as shared code, with only the differences expressed as data.

Concretely, that meant three layers:

1. A shared module one Terraform module containing the actual resource definitions (VM, networking, disk, etc.), written with input variables for everything that varies between regions: location, name suffix, network ranges, and so on.

2. One shared entry point a single root configuration that calls the module, rather than one root configuration per region.

3. A small config file per region instead of a full copy of the module, each region gets a short file (five or six lines) specifying just its own values: which cloud location to deploy into, what to name things, what network ranges to use.

Adding a new region now means adding one small config file. No new Terraform resource code, no new pipeline definition, nothing to copy and rename.

Why this matters beyond "less typing"

The obvious win is less duplication. The less obvious win is correctness:

A structural fix only needs to happen once. Change the module, every region picks it up. Before, a structural change meant editing N folders and trusting yourself to get all N edits identical.
Drift between regions becomes visible. With copy-pasted folders, two regions can quietly diverge over time as one gets a fix the other doesn't. With a shared module, divergence can only happen in the small config file which is short enough to diff at a glance.
Review gets easier. A pull request adding a new region is now a five-line diff to a new config file, not a few hundred lines of near-duplicate resource code that's hard to review carefully.

What I kept separate on purpose

One thing worth calling out: sharing the code doesn't mean sharing everything. Each region still keeps:

Its own Terraform state file, so a mistake applying to one region can't touch another region's resources.
Its own deployment approval gate in the pipeline, so promoting a change to one region doesn't silently promote it everywhere.

The pattern is "share the how, isolate the where it lands." Sharing a module is safe. Sharing a blast radius is not.

The general lesson

If you find yourself about to copy a folder to stand up a new instance of something a new region, a new environment, a new tenant that's usually the signal to stop and ask: what's actually different here? Usually the answer is "not much" a handful of values. Everything else can be a shared module, parameterized by those values.

Copy-paste feels like the fast path. It's fast for the first copy. It gets slower and riskier every time after that.

The App Needed a Real Desktop, Not Just a Real Windows Machine

Haripriya Veluchamy — Sat, 08 Aug 2026 17:37:06 +0000

The App Needed a Real Desktop, Not Just a Real Windows Machine

Before any of the debugging stories, gotchas, or architecture arguments in this series existed, there was one earlier decision that had to fail first and it's the one that actually explains why everything after it looks the way it does.

The plan that looked right on paper

The system in question wraps a Windows-only, GUI-first desktop application the kind of software that ships as a .exe, expects to be installed via a wizard, and exposes automation only through a first-party SDK that talks to the running application process. No web API, no headless mode, nothing designed for servers at all.

The instinct, reasonably, was: don't fight that. Run it inside a Windows compatibility layer, in a Linux container. Keeps everything in the same containerized, cloud-native tooling as the rest of the stack. Cheaper to host. Fits the mental model everything else in the system already used.

It did not work.

What "did not work" actually looked like

Not a clean, informative failure a slow, grinding one. The automation layer would intermittently fail to connect to the running application with low-level timeout errors, with no consistent trigger. Sometimes it worked for hours. Sometimes it failed within minutes of a restart. The compatibility layer itself would occasionally become unstable in ways that were hard to attribute to any single cause. Every fix felt like it addressed a symptom, and a slightly different symptom would show up a few days later.

Eventually the right call was: stop trying to make this specific approach reliable, and question whether the approach itself was ever going to be reliable rather than continuing to patch a foundation that might be structurally wrong.

The pivot, and the deeper thing it revealed

The fix was to abandon the compatibility-layer approach entirely and run the application on an actual Windows machine, natively. This alone mostly solved it but "mostly" is doing real work in that sentence, because it surfaced a second, more specific requirement that the first failure had been obscuring the whole time.

Running on real Windows wasn't sufficient by itself. The application's automation layer would still fail to connect if the underlying process was running in a headless or service-style context even on genuine Windows, even with no compatibility layer involved at all. It only worked reliably when the process was running inside a real, interactive, logged-in desktop session the same kind of session that exists when an actual person is sitting at the keyboard, not a background service context that happens to also be "on Windows."

This turned out to be the actual root requirement the whole time. The compatibility-layer approach hadn't just been unstable in some vague sense it was fundamentally incapable of providing this, no matter how much it was tuned, because a compatibility layer running headless in a container was never going to look like an interactive desktop session to begin with.

Why this isn't just a trading-software quirk

I think this generalizes further than it looks. A meaningful category of Windows software licensed engineering tools, certain legacy financial or back-office systems, some CAD and design applications was written assuming a human is logged in: window handles exist, GUI event loops are running, certain OS-level session facilities are available that simply don't exist in a headless context. None of that is a bug in the software. It's just an assumption baked in from an era when "run this on a server" wasn't a use case anyone designing it considered.

If you're trying to automate something in this category in the cloud, "get it running on Windows" is necessary but not sufficient. The actual question to ask early ideally before building anything around it is: does this specific piece of software's automation surface require a real interactive session, or does it genuinely work headless? That's usually one focused test, and it's a much cheaper question to answer on day one than to discover, the way this system did, after weeks of chasing intermittent failures in the wrong layer.

Once that requirement was understood clearly, the actual infrastructure need became concrete and solvable: real auto-logon configuration, launching the process specifically as an interactive session rather than a generic service, and importantly verifying which kind of session a running process is actually in as a first-class health check, not just checking whether the process exists at all. "The process is running" and "the process is running somewhere it can actually work" turned out to be two different, both necessary, checks.

What ties this back to everything else in this series

Looking back across the debugging stories, the audit findings, and the architecture arguments this series has covered, I think they're all really one lesson wearing different clothes: the constraint that looks like an inconvenient afterthought is often the actual load-bearing wall. A packaging policy decision that seems like a small CI annoyance. A "designed to fail" test that quietly carries an unrelated assumption about safety. A vault that's been silently unreachable long enough that nobody remembers assuming it worked. A scaling pattern that's right for most systems and simply wrong for this one. And here, at the root of all of it: an assumption that "a real Windows machine" and "a real interactive desktop session" were the same thing, when they were never quite the same thing at all.

None of these were exotic problems. They were all findable early, cheaply, by asking one honest, specific question before building rather than discovering the honest answer later, expensively, after something was already built on top of the wrong assumption.

Autoscaling Doesn't Fit Every Workload Here's How to Tell, and What to Build Instead

Haripriya Veluchamy — Mon, 03 Aug 2026 15:06:37 +0000

"Just put it behind an autoscaling group" is one of those pieces of advice that's right so often it stops getting questioned. For a stateless API or a web server, it's genuinely close to free scaling add a replica, traffic balances across it, remove a replica, nobody notices. It's such a reliable default that it's easy to reach for it everywhere.

It doesn't fit every workload, though, and the difference matters enough that I think it's worth naming precisely not as "autoscaling is bad," but as: here's the specific property a workload needs to have before reactive autoscaling makes sense, and here's what to do instead when it doesn't.

The property that actually matters: interchangeability

A stateless replica is interchangeable. Any instance can serve any request, right now, with zero setup cost specific to that request. That's the entire reason autoscaling groups work as well as they do the thing you're adding is immediately, fully useful the moment it's up, and removing one costs nothing because whatever it was doing, another replica can pick up instantly.

A lot of backends quietly violate this assumption without anyone noticing, because the violation only shows up under real scale pressure, not in normal development. Two common ways this happens:

Session affinity. Some external systems only allow a single active session per account, or per some other resource key. Once a session is established on a specific instance, that instance not just "some instance in the pool" is now the only place that session lives. A second replica can't share it, take it over seamlessly, or load-balance it away.

Slow, non-trivial startup. A stateless container can typically start serving traffic in well under a second. Some backends anything wrapping a heavier external process, a licensed desktop application being automated, a protocol requiring a multi-step handshake before it's usable take real, measurable time (seconds to minutes) before a fresh instance is actually useful. If your scaling trigger is "traffic just spiked, react now," and your instances take two minutes to become ready, you've built a system that's structurally too slow for the exact problem it's meant to solve.

If either of these is true for your backend, a generic reactive autoscaling group isn't a slightly-worse fit it's actively the wrong shape of solution, and no amount of tuning the trigger thresholds fixes that.

Scaling out: the fix is timing, not tooling

The good news: scaling out for this class of system is solvable, just not with second-scale reactivity. Instead of "traffic spiked, add a replica now," the right pattern is a slower, trend-based trigger: watch a capacity metric (percentage of current instances' sessions in use, say), and cross a threshold 70-75% is a common choice before you're actually out of room, giving the new instance its full startup time to become ready ahead of when it's actually needed.

This is the same idea as reactive autoscaling, just running on a slower clock that respects your workload's real startup cost, instead of pretending it doesn't have one.

Scaling in: the part almost everyone skips

Scaling out gets most of the attention because it's the "we're growing" story. Scaling in safely is the harder half, and it's the part a lot of systems never actually build, quietly assuming it'll be fine.

It won't be fine, by default, for exactly the same reason scaling out needed rethinking: an instance in this class of system might have live, in-progress sessions on it. A naive scale-in policy "utilization dropped, terminate an instance" has no way to know that, and will happily kill active work.

This isn't a novel problem, and it's worth knowing the industry already has a real, named answer for it: AWS's Auto Scaling supports lifecycle hooks specifically for this a termination hook can hold an instance in a pending state, giving your own code time to drain in-progress work or migrate it elsewhere, rather than the platform just pulling the plug. There's an even more direct newer option built for exactly this situation: instance lifecycle policies that keep an instance retained rather than force-terminating it if a graceful shutdown doesn't complete cleanly. The pattern to copy, regardless of which cloud you're on: stop routing new work to an instance first, wait for its existing sessions to end naturally, only then actually remove it. That's not exotic engineering it's the same "drain, then remove" idea load balancers have used for connection draining for years, just applied one layer up, at the compute level instead of the request level.

This isn't a niche problem real, large-scale systems already do this

It's worth checking whether an actual production system at scale validates this shape, rather than trusting my own reasoning alone. Video conferencing platforms are a good real-world example a live meeting session is about as textbook session-affine as workloads get. Zoom's own published architecture describes exactly the pattern this post is arguing for: participants get routed to the least-loaded available server for their region, via a dedicated control-plane component that tracks real server load not round-robin, not a generic autoscaling group blindly adding and removing capacity. That's a capacity-aware assignment layer sitting in front of a pool of session-bound servers, which is precisely the shape you land on once you take "sessions can't just move between replicas" seriously.

The actual takeaway

Before reaching for an autoscaling group, ask one honest question: can any instance in this pool serve any unit of work right now, with no setup cost specific to that instance? If yes, standard autoscaling is a great, close-to-free default use it. If no because of session affinity, slow startup, or both you need two separate, deliberately different mechanisms: a slower, trend-based trigger for scaling out, and real drain logic for scaling in. Trying to force a session-affine, slow-starting workload into a fast, reactive, interchangeable-replica model doesn't just work worse it doesn't really work at all, and the failure mode is subtle enough that it's easy to only discover it once you're already depending on it.

Vault cant found What a Live Audit Found That Code Review Never Would

Haripriya Veluchamy — Wed, 29 Jul 2026 08:28:13 +0000

Some bugs hide in logic. This one hid in the gap between "the code looks correct" and "the code is actually working" and the only way I found it was by refusing to trust either claim without checking the live system directly.

The setup

Mid-migration between two credential-storage systems: an older secrets vault (call it the legacy vault) used by an original provider integration, and a newer, purpose-built vault for a self-hosted replacement being rolled out gradually. During the transition, both exist side by side. The code that decides which vault to check for a given user's credentials looked, on review, completely correct it classified each connection record as "legacy" or "new" based on a stored field, and looked in the matching vault.

Nothing about the code review raised a flag. Which is exactly the problem with code review as your only line of defense: it tells you the logic is internally consistent, not that the systems it's talking to still exist.

Building the audit, properly this time

Instead of trusting the classification logic in the abstract, I wrote a small audit script with one rule: don't reconstruct what a secret's name should be and assume it's there actually query the live vault and see. This sounds obvious. It's also very easy to skip, because "reconstruct the expected name, spot-check a couple" feels like it should be equivalent to "check them all live," and it really isn't.

The script pulled every connection record from the database, classified each one the same way the app's own code did, and then made a real, live call to whichever vault it should be in.

What it found

Every single legacy-vault lookup failed. Not "secret not found" a connection failure. The vault itself wasn't resolving anymore.

That's a materially different problem than a missing secret. A missing secret means "someone forgot to write this." A vault that doesn't resolve at all means the entire piece of infrastructure the code has been confidently pointing at no longer exists and every single credential-write attempt to it, for every legacy user, had been silently failing, this whole time, for however long it had been broken.

How did nobody notice?

The honest, slightly embarrassing answer: the code that wrote to this vault caught the failure, logged a warning, and moved on. Not a crash. Not an alert. A line in a log nobody was watching, for a code path nobody expected to fail, because "the vault exists" was baked in as an assumption so early in the system's life that it had stopped being something anyone thought to re-check.

This is the actual shape of the bug, more than the specific vault: a try/catch that swallows a failure gracefully is indistinguishable, from the outside, from success until someone goes looking.

The part that actually surprised me

Here's where it got more interesting than "we found a broken thing, went and fixed it." Digging into why the legacy vault mattered at all, it turned out the underlying legacy provider managed those users' actual live sessions entirely on its own side the app's cached copy of their credentials was never actually read from again in practice. The vault had been dead, silently, for a long stretch, and functionally, nothing legacy-side had actually broken because of it.

That's not a "phew, no harm done, nothing to see here" ending though. It's a different, more uncomfortable one: we got lucky that this specific dead dependency happened to be redundant. The audit didn't know that in advance, and neither did I. The only way to find out whether a silently-dead piece of infrastructure actually matters is to go looking you can't reason your way to "probably fine" from the code alone.

The same audit, run against the newer vault the one actually still in active, load-bearing use found a real, smaller, actionable problem: a couple of expected entries genuinely missing. Real incomplete setup, not infrastructure rot. Fixed directly.

What I took away from this

Code review confirms logic. It cannot confirm that the systems the logic depends on still exist. Those are different guarantees, and it's easy to unconsciously treat the first as if it implies the second.
A caught-and-logged failure is a blind spot by construction, not just bad luck the entire point of catching it gracefully is that nothing breaks loudly. That's good for uptime and bad for ever noticing, unless something is actually watching those logs.
"Reconstruct the expected value and spot-check it" is not the same audit as "query the live system for every case." The first checks your mental model. The second checks reality. They can disagree, and when they do, reality wins.
Every so often, a system this old deserves a live audit, not just a code review specifically to surface the assumptions that were true once, got baked in early, and nobody has had a reason to re-question since.

Why Your CI Agent Can't Install pip

Haripriya Veluchamy — Sat, 25 Jul 2026 16:06:39 +0000

If you've ever set up a self-hosted CI/CD agent on a fresh Ubuntu image and hit pip: command not found or No module named pip, your first instinct is probably to check the network. Mine was too. That instinct is wrong, and the real answer took me four attempts to actually land on.

Here's the full story, including the three dead ends, because I think the dead ends are the useful part.

The setup

A self-hosted CI agent (in my case, an Azure DevOps agent, but this applies just as much to a self-hosted GitHub Actions runner or a plain Jenkins box) running on a minimal Ubuntu image. No sudo access on the box that's deliberate, since the agent shouldn't need root just to run a pipeline. I needed to install a CLI tool that only ships via pip.

Reasonable first attempt:

- script: |
    python3 -m pip install --user --quiet some-cli-tool
  displayName: 'Install tool'

Result:

/usr/bin/python3: No module named pip

Fine, I thought pip module is missing, ensurepip should bootstrap it. Every tutorial says so.

Attempt 2: `ensurepip`

- script: |
    python3 -m ensurepip --user --upgrade
    python3 -m pip install --user --quiet some-cli-tool

Result:

/usr/bin/python3: No module named ensurepip

Not just pip missing ensurepip itself, the thing that's supposed to install pip, doesn't exist either. That's the first real clue something structural is going on, not just a missing package.

The actual reason (worth understanding, not just working around)

Debian and Ubuntu deliberately strip pip and ensurepip out of the base python3 package. This isn't a bug or an oversight it's a packaging policy decision. Both live in separate packages (python3-pip, python3-venv) that you're expected to install via apt. On a full desktop or dev-configured server, you've probably had these installed for so long you forgot they're not actually part of core Python.

On a minimal, no-sudo CI image, you don't have apt access at all so you can't just apt install python3-pip your way out of it.

Attempt 3: `venv` (also fails, same root cause)

My next thought: skip pip/ensurepip entirely, use a virtual environment instead, since venv normally bootstraps its own pip on creation.

- script: |
    python3 -m venv /tmp/tool-venv
    /tmp/tool-venv/bin/pip install --quiet some-cli-tool

Result: fails identically. venv's own pip-bootstrapping step depends on you guessed it the same missing ensurepip module. Same wall, different door.

Attempt 4: the actual fix `get-pip.py`

The real answer is a standalone bootstrapping script, maintained by the Python Packaging Authority specifically for situations like this environments where ensurepip isn't available:

- script: |
    curl -sS https://bootstrap.pypa.io/get-pip.py -o /tmp/get-pip.py
    python3 /tmp/get-pip.py --user --quiet
    python3 -m pip install --user --quiet some-cli-tool
    echo "##vso[task.prependpath]$HOME/.local/bin"
  displayName: 'Install tool (get-pip, no sudo)'

get-pip.py doesn't depend on ensurepip at all it's a self-contained bootstrapper that installs pip directly. This worked immediately, no sudo, no apt, no venv complexity.

The one thing to double-check before you assume this is your problem

Before chasing this fix, rule out the boring explanation first: confirm your agent actually has network access to PyPI at all. A quick diagnostic step saved me from solving the wrong problem:

- script: |
    curl -sS -o /dev/null -w "pypi.org: %{http_code}\n" https://pypi.org --max-time 5

If that comes back with anything other than a fast 200, you're dealing with a firewall/egress issue, not this one and no amount of get-pip.py will fix that.

The takeaway

If python3 -m pip and python3 -m ensurepip are both missing on a Debian/Ubuntu box, that's not a broken image it's the distro's actual, intentional packaging policy. apt install python3-pip is the "normal" fix, and it's simply not available to you on a locked-down, no-sudo CI agent. get-pip.py is the one workaround that doesn't need apt, doesn't need ensurepip, and doesn't need root.

Small thing, but it cost me three wrong turns before I found it hopefully this saves you those three.

Why AI Can't Actually Hit Your Word Limit

Haripriya Veluchamy — Wed, 22 Jul 2026 10:20:08 +0000

I Was drafting my blog post, told Claude "keep it to 250 words." Got back something like 340. Asked again. Same story.

Figured there's a real reason behind this, and since I'm neck-deep in AI/ML fundamentals right now, decided to actually understand it instead of just being annoyed by it.

It generates one token at a time, no plan

LLMs don't write like we do think of the whole thing, then fill it in. They generate autoregressively: predict the next token, add it to the context, predict the next one, repeat. There's no "here's my 250-word essay, let me write it out" step happening anywhere. Every single token is picked based on what came before, nothing more.

So when you ask for 250 words, the model isn't holding a target in memory and counting down. It's pattern-matching: "text like this, following an instruction like this, tends to stop around here." That's a guess shaped by training data, not a computation.

Words ≠ tokens

Also it's not even counting words. Internally, everything is tokens: sub-word chunks. "Debugging" might be one token or three depending on the tokenizer. "250 words" as an instruction gets converted into an expectation of roughly how many tokens that maps to, and that mapping is fuzzy. There's no clean 1:1 between what you asked for and what the architecture tracks.

No counter anywhere in the architecture

This is the part that surprised me most: there's no length-tracking variable sitting in the model's forward pass. No if word_count == 250: stop. The only thing steering generation is the probability distribution over the next token, sampled step by step, until it predicts an end-of-sequence token or hits a max-length cutoff set by the system.

Compare that to something like a for loop with a counter deterministic, exact. LLM generation is closer to sampling from a learned distribution, over and over. Approximate by design, not by bug.

The workaround

Since I can't get an exact count out of a single generation, what actually works:

Ask for a natural stopping point instead of a hard number
Generate first, then trim or expand after, treating the count as a second pass, not the first
For real precision, set limits in characters or use iterative regeneration with a check step, rather than trusting the first output

Why this clicked for me

This ties directly into what I've been learning attention, softmax over vocabulary, sampling strategies. Word-count control would need something structurally different: a running counter fed back into the decoding loop, or a constrained decoding step that force-stops at a token budget. Neither is how these mainstream models work by default.

Small annoyance, decent rabbit hole. Worth knowing before you set a hard word limit and assume the model's counting along with you.

Learning AI/ML from the ground up cloud engineer trying to understand the models she uses daily.

Serving ML Artifacts from Amazon S3 Files How I used After the Launch

Haripriya Veluchamy — Mon, 22 Jun 2026 13:37:34 +0000

The Honest Story

Two months ago, everyone was posting about Amazon S3 Files. New feature, big announcement, screenshots everywhere. I scrolled past most of them another AWS launch, another round of "here's what it does" posts.

I never actually understood what it was until it became my problem.

I was in the middle of my ML learning journey, building a semantic search project. Large FAISS indexes, BM25 artifacts all needed to be available at serving time. I was uploading them directly into my deployment. It worked, until it didn't. Hit the size limit. Container cold starts became painful. Every restart meant downloading hundreds of megabytes before the first request could be served.

I had one option left S3.

And somewhere in the back of my head I remembered: "wait, there's that new S3 file system thing everyone was talking about." S3 is already my favorite AWS service. So why not try it?

I tried it. I built something real with it. And now I'm writing the post I wish existed two months ago not "here's what S3 Files is" but "here's what actually happens when you use it."

What I Built

A semantic search engine over AWS documentation 5048 pages, indexed with FAISS + BM25 hybrid retrieval, served via FastAPI. The entire ML artifact stack (indexes, metadata) is served directly from an S3 Files NFS mount on EC2. No cold start downloads. No boto3 in the serving code. Just open().

Stack:

Embeddings: all-MiniLM-L6-v2
Semantic search: FAISS (IndexFlatIP)
Keyword search: BM25Okapi
Serving: FastAPI + Jinja2
Artifact storage: Amazon S3 + S3 Files (NFS)
Compute: EC2 t3.medium (Ubuntu 22.04)

GitHub: Harivelu0/s3-files-ml-serving

The Problem S3 Files Solves

Before S3 Files, serving large ML artifacts from containers looked like this:

# Every container startup
from huggingface_hub import hf_hub_download

faiss_path = hf_hub_download(repo_id="...", filename="faiss.index")  # 7MB
bm25_path  = hf_hub_download(repo_id="...", filename="bm25_index.pkl")  # 17MB
meta_path  = hf_hub_download(repo_id="...", filename="corpus_meta.pkl")  # 2.6MB

Every container restart meant downloading ~27MB before the first request. For larger models this becomes hundreds of MB or GBs. You'd need to manage download logic, handle failures, worry about /tmp size limits, and every container on the same host duplicates the same data.

With S3 Files, your serving code becomes:

import faiss
import pickle

# Just open() no download, no boto3
index = faiss.read_index("/mnt/artifacts/faiss.index")
with open("/mnt/artifacts/bm25_index.pkl", "rb") as f:
    bm25 = pickle.load(f)

The S3 bucket is mounted as an NFS volume on the EC2 host. Your container reads from it like a local file. That's it.

Architecture

AWS Docs (5048 pages)
        ↓
crawl_aws_docs.py (Sitemap + BeautifulSoup)
        ↓
build_index.py (FAISS + BM25 + corpus_meta)
        ↓
S3 Bucket (versioning enabled)
        ↓
S3 File System (NFS layer on top of bucket)
        ↓
Mount Target (NFS endpoint inside VPC)
        ↓
EC2 Ubuntu 22.04
  └── /mnt/artifacts (S3 Files mounted here)
       └── Docker container
            └── FastAPI reads indexes directly
                 └── http://ec2-ip:8000

Setting Up S3 Files What Actually Matters

1. S3 bucket needs versioning enabled

This one catches everyone. S3 Files will refuse to create a file system without it.

aws s3api put-bucket-versioning \
    --bucket your-bucket \
    --versioning-configuration Status=Enabled

2. IAM role trust principal is `elasticfilesystem.amazonaws.com`

Not s3files.amazonaws.com even though the service is called S3 Files, it's built on EFS under the hood.

{
  "Principal": { "Service": "elasticfilesystem.amazonaws.com" },
  "Condition": {
    "StringEquals": { "aws:SourceAccount": "YOUR_ACCOUNT_ID" }
  }
}

3. The IAM role needs EventBridge permissions

S3 Files uses EventBridge internally to monitor bucket changes. Without these, your file system gets stuck in creating forever.

{
  "Action": [
    "events:PutRule", "events:DeleteRule",
    "events:PutTargets", "events:RemoveTargets"
  ],
  "Resource": "*"
}

4. boto3 API uses camelCase

The S3 Files boto3 client uses camelCase unlike most other AWS services.

# Wrong
s3files.create_file_system(Bucket=bucket_arn, RoleArn=role_arn)

# Correct
s3files.create_file_system(bucket=bucket_arn, roleArn=role_arn)

5. Mount needs `amazon-efs-utils`, not plain NFS

Plain mount -t nfs4 will fail. S3 Files requires the amazon-efs-utils package (v3.0+) which handles TLS + IAM auth automatically.

# Build and install amazon-efs-utils
sudo apt-get install -y cmake golang-go rustc cargo
git clone https://github.com/aws/efs-utils
cd efs-utils && ./build-deb.sh
sudo apt-get install -y ./build/amazon-efs-utils*.deb

# Then mount
sudo mount -t s3files fs-0xxxxxxxxx:/ /mnt/artifacts

6. EC2 instance needs `AmazonS3FilesClientFullAccess`

Without this policy on the EC2 role, the mount returns access denied by server.

aws iam attach-role-policy \
    --role-name your-ec2-role \
    --policy-arn arn:aws:iam::aws:policy/AmazonS3FilesClientFullAccess

The Serving Code

FastAPI startup loads everything from the mount. No download logic. No error handling for network failures during download. The mount is always there.

ARTIFACTS_DIR = Path(os.getenv("ARTIFACTS_DIR", "/mnt/artifacts"))

@asynccontextmanager
async def lifespan(app: FastAPI):
    # Reads directly from S3 Files mount
    state["index"] = faiss.read_index(str(ARTIFACTS_DIR / "faiss.index"))

    with open(ARTIFACTS_DIR / "bm25_index.pkl", "rb") as f:
        d = pickle.load(f)
        state["bm25"] = d["bm25"]

    with open(ARTIFACTS_DIR / "corpus_meta.pkl", "rb") as f:
        state["meta"] = pickle.load(f)

    state["model"] = SentenceTransformer("all-MiniLM-L6-v2")
    yield

Container startup: model loads in ~2s. Artifacts available instantly from mount.

Weekly Index Updates The Real S3 Files Advantage

This is where S3 Files goes beyond just solving cold starts.

AWS updates their docs regularly. With traditional artifact serving you'd need to rebuild indexes, push a new container image, and redeploy causing downtime.

With S3 Files:

# update_pipeline.sh runs weekly via EventBridge
python scripts/crawl_aws_docs.py --update   # only changed pages
python precompute/build_index.py            # rebuild indexes to /tmp

# Atomic swap no downtime
mv /tmp/faiss.index.new     /mnt/artifacts/artifacts/faiss.index
mv /tmp/bm25_index.pkl.new  /mnt/artifacts/artifacts/bm25_index.pkl
mv /tmp/corpus_meta.pkl.new /mnt/artifacts/artifacts/corpus_meta.pkl

The serving container picks up the new indexes on the next query. Zero restart. Zero redeploy. The mount sees the updated S3 objects immediately.

This is the killer feature not just cold start elimination, but live index updates without any deployment.

Results

The search UI running at http://98.93.65.241:8000:

Query: "aws sns permission issue" → 10 results in 334ms
Artifacts loaded at startup: instant (no download)
Index update: rebuild + upload to S3 → serving picks up without restart

What I Learned

S3 Files is genuinely useful for ML workloads where:

Artifacts are large (>100MB)
Multiple containers need the same data
Indexes update regularly without downtime
You want to avoid baking artifacts into Docker images

It's not a replacement for EFS if you need pure file system performance. And it's not for every use case if your artifacts never change and containers rarely restart, the complexity isn't worth it.

But if you've ever stared at a container downloading 500MB on every cold start, S3 Files is exactly what you were waiting for.

Resources

How I Built a Self Resizing EC2 for My ML Data

Haripriya Veluchamy — Sun, 07 Jun 2026 11:45:14 +0000

The Pain Point

I'm on an ML learning journey. That means a lot of data. A lot of processing. And a lot of AWS free credits I really can't afford to waste.

Here's what my typical day looked like:

I'd spin up a t3.large to run a data pipeline. Load some datasets, process them, store them. The pipeline would run for a couple of hours sometimes I didn't even know exactly how long it would take. Then I'd go to sleep.

Next morning I'd check CloudWatch and realise the VM had been sitting idle since 3AM. Running. Doing nothing. Burning credits.

That's the problem. You need a big machine for data loading. You don't need it after.

The First Thought Let AWS ML Decide

My first instinct was to use AWS Compute Optimizer. It's a managed ML service that analyses your EC2 usage patterns and recommends the right instance type. Smart, right?

I enabled it. Waited. And waited.

Turns out Compute Optimizer needs at least 30 consecutive hours of usage data before it generates recommendations. For a VM I spin up occasionally for pipeline runs that's not practical.

So I moved on.

The Second Thought Bedrock Agent

Next idea: use Amazon Bedrock as an agent to reason about when and how to resize. Let an LLM decide.

But the more I thought about it, the more it felt like overkill. The decision isn't complex:

"Did the pipeline finish? Yes → resize down. No → don't."

That's not a reasoning problem. That's an automation problem. Using Bedrock here would be AI washing adding complexity without adding value.

So I kept it simple.

The Solution EventDriven VM Resize

What I built: a lightweight agent that listens for your pipeline to complete, then automatically resizes the VM down.

Stack:

emit_event.py runs on your VM, fires when pipeline exits
Amazon EventBridge receives the event
AWS Lambda handles resize logic
Amazon SNS sends email alert (success or failure)

Pipeline finishes
      ↓
emit_event.py → EventBridge
      ↓
Lambda triggered
      ↓
SUCCESS → resize down + email
FAILURE → keep size (debug) + email

No ML. No LLM. Just the right tool for the job.

Architecture

Your VM                        AWS Cloud
──────────────────             ─────────────────────
run_pipeline.sh                EventBridge (custom bus)
  step1.py               →          ↓
  step2.py                     Lambda
  step3.py                       ├── stop EC2
  emit_event.py ──────────→      ├── resize instance type
                                 ├── start EC2
                                 └── SNS email alert

All AWS resources are deployed with a single CloudFormation command. No manual console clicking.

Setup 3 Steps

Step 1 Deploy AWS infrastructure

git clone https://github.com/Harivelu0/vm-resize-agent
cd vm-resize-agent

aws cloudformation deploy \
  --template-file infra/template.yaml \
  --stack-name vm-resize-agent \
  --capabilities CAPABILITY_NAMED_IAM \
  --parameter-overrides \
      AlertEmail=your@email.com \
      TargetInstanceType=t3.medium \
  --region us-east-1

Check your email → confirm the AWS subscription link.

Step 2 Copy agent to your VM

scp -i your-key.pem -r agent/ pipeline/ \
  ec2-user@your-vm-ip:~/vm-resize-agent/

# on VM
pip install boto3
aws configure

Step 3 Add your pipeline steps

Edit pipeline/steps.conf:

python3 /home/user/myproject/fetch_data.py
python3 /home/user/myproject/transform.py
python3 /home/user/myproject/load_db.py

Run:

bash pipeline/run_pipeline.sh

That's it. When pipeline finishes → VM resizes → email arrives.

The Key Design Decision steps.conf

One thing I was particular about: this tool should work for anyone's pipeline, not just mine.

So run_pipeline.sh never changes. Users only edit steps.conf one command per line, one step per line.

# steps.conf
python3 /home/user/fetch_gdelt.py
python3 /home/user/build_forecasts.py
python3 /home/user/calibrate.py

The wrapper reads each line, runs it in order, tracks success/failure, and emits the event at the end. Generic by design.

The Email Alert

On success:

Pipeline: data-loader
Status:   SUCCESS
Steps:    3/3
Duration: 45m 12s
Instance: i-0abc123
Resized:  Yes -> t3.medium

On failure:

Pipeline:  data-loader
Status:    FAILURE
Steps:     2/3
Failed at: step_3
Instance:  i-0abc123
Resized:   No (kept original size for debugging)

Failure case is important the VM intentionally stays large so you can SSH in and debug without losing state.

Difficulties I Faced

1. IP changes after resize

When Lambda stops and restarts the EC2, it gets a new public IP. Learned this the hard way when SSH stopped working. Fix: assign an Elastic IP before running demos.

aws ec2 allocate-address --region us-east-1
aws ec2 associate-address \
  --instance-id i-xxx \
  --allocation-id eipalloc-xxx

2. Docker goes down after resize

EC2 resize = reboot. Any running Docker containers stop. Add this to /etc/rc.local on your VM so services restart automatically:

sudo service docker start
cd /home/ec2-user/myproject && docker-compose up -d

Live Demo

For the demo I used a real weather dataset loaded into Postgres running in Docker on the EC2.

EC2: t3.large (before)
       ↓
Pipeline runs: download → parse → load into Postgres
       ↓
emit_event.py fires automatically when done
       ↓
Lambda: stops EC2 → resizes → starts EC2
       ↓
EC2: t3.medium (after)
       ↓
Email arrives in Gmail

Cost Reality

Resource	Monthly Cost
EventBridge	~$0 (free tier)
Lambda	~$0 (free tier)
SNS email	~$0 (first 1000 free)
Total stack	~$0

The savings depend on your instance. A t3.large idle for 20hrs/day wastes ~$25/month. For larger instances like c5.4xlarge you're looking at $200+ saved per month.

When to Use This

This is for you if:

You run data pipelines on EC2 manually or on a schedule
Your pipeline takes unpredictable time to complete
You don't need the heavy instance after loading is done
You're not ready for EMR or Glue yet (pre-pipeline stage)

This is not for you if:

You're already using EMR Serverless or Glue (they auto-terminate)
Your pipeline runs less than 30 minutes (manual resize is fine)
You need horizontal scaling (use Auto Scaling Groups instead)

Why a VM and Not Glue or EMR?

Honest answer I didn't know enough about my data yet to make that decision.

Glue and EMR are great services. But they come with questions you need to answer upfront:

What's your data format?
What transformations do you need?
What's the volume?
Do you need Spark?

When you're learning, you don't have those answers yet. You just need to load data, see what you're working with, and prove the pipeline works.

A VM lets you do that with zero infrastructure decisions. Just Python scripts. When the pipeline is proven and you understand your data then you migrate to the right managed service.

This is that phase. Before you know which service you need.

This is actually how real teams work too. Nobody starts with EMR on day one of a new data project. You explore first. You prove it works. Then you scale.

Phase 1: Explore         → VM + Python scripts
Phase 2: Prove it works  → VM + vm-resize-agent
Phase 3: Scale           → Glue / EMR / Spark

Repo

Everything is open source. Clone, edit steps.conf, deploy.

GitHub: https://github.com/Harivelu0/vm-resize-agent

What's Next

A few things I want to add:

Auto resize back up before next scheduled run (cron-based)
Slack alerts alongside email
Support for Azure VMs (same pattern, different SDK)

If you have ideas or run into issues open a GitHub issue. Happy to help.

Building in public as part of my ML learning journey. Follow along for more practical AWS patterns.

How I Built Production AI Agent Monitoring with Langfuse

Haripriya Veluchamy — Wed, 13 May 2026 16:32:02 +0000

Multi-agent AI systems fail silently.

A 200 OK response doesn’t mean the AI made good decisions.

That was the biggest thing I realized while building a multi-agent system.

My architecture looked like this:

User Query → Multi Agent Call → Final Response

Everything looked normal from an infrastructure perspective.

APIs were healthy
Latency looked fine
Users were getting responses

But I still couldn’t answer important questions:

Did the Agent route the query to the right specialist?
Did the agent hallucinate information?
Did it ignore specialist outputs?
Did it attribute responses incorrectly?

Traditional monitoring couldn’t help because the system technically wasn’t failing.

The failures were happening at the decision layer.

Full Trace Visibility

I used Langfuse to trace every agent execution.

That includes:

Tool calls
Input/output payloads
Token usage
Latency per step

If an agent touched something, I wanted visibility into it.

No black boxes.

Deterministic Checks

Some validations didn’t need another LLM.

I added rule-based checks for things like:

Did the agent call tools from the correct domain?
Did the agent call tools it wasn’t supposed to?
Was the expected workflow followed?

These checks are binary:

Pass → 1
Fail → 0

Fast and cheap.

Faithfulness Checks

This was mainly for hallucination detection.

I compare the final response with outputs from specialist agents.

If the Final layer introduces claims that weren’t exist in source outputs, it gets flagged.

This helped catch cases where the system sounded confident but wasn’t grounded.

LLM Judges

For things deterministic checks can’t measure, I use Azure OpenAI as judges.

They evaluate:

Routing correctness
Response quality
Attribution accuracy
Conflict handling

This runs for every multi-agent response.

Expensive? Yes.
Useful? Definitely.

100% Traffic Monitoring

I didn’t want sampling.

Every production request goes through the evaluation pipeline.

Because edge cases are usually the exact things sampling misses.

Cost + Latency Tracking

Multi-agent systems get expensive very fast.

I track:

Tokens per agent
Latency per step
Expensive execution paths

This made optimization much easier.

What This Actually Caught

This surfaced issues normal monitoring completely missed.

Wrong attribution

Correct insights were assigned to the wrong specialist.

Ignored outputs

Sometimes Agent completely ignored specialist responses.

Routing mistakes

The call occasionally sent queries to the wrong agent.

None of these showed up in normal monitoring dashboards.

Everything looked healthy.

Stack

Observability: Langfuse
LLM Evaluation: Azure OpenAI
Deterministic Checks: TypeScript

Final Takeaway

For multi-agent systems, uptime monitoring is not enough.

You also need decision monitoring.

Because a successful response can still be completely wrong.

I Built a Self-Updating SEO Brain Inspired by Andrej Karpathy's LLM Wiki

Haripriya Veluchamy — Sun, 10 May 2026 13:39:47 +0000

The Tweet That Changed How I Think About AI + Knowledge

In early April 2026, Andrej Karpathy (OpenAI co-founder, former Tesla AI Director) posted something deceptively simple:

"Something I'm finding very useful recently: using LLMs to build personal knowledge bases for various topics of research interest."

He followed it up with a GitHub gist titled LLM Wiki an "idea file" describing a pattern for building knowledge bases that actually compound over time instead of rediscovering the same information on every query.

I decided to build it for a real production problem.

The Problem: RAG Has No Memory

In My organization, we were running an SEO monitoring pipeline for our landing site. It used Cognee (a knowledge graph framework) backed by Neo4j + ChromaDB on a $35/month VM.

Every day it would:

Pull Google Search Console data
Scrape our pages for SEO issues
Query the knowledge graph
Post a Slack report

It worked. But it had a fundamental flaw the same flaw Karpathy describes in every RAG system.

Every time it ran, it was rediscovering knowledge from scratch. It had no memory of what it found yesterday. It couldn't connect "we deployed this fix on March 13" with "clicks went up 81% on March 18." It couldn't say "this keyword has been declining for 3 weeks here's why." It just answered the current query and forgot everything.

On top of that: the VM broke 3+ times. Neo4j config issues. ChromaDB API path changes. Sidecar containers failing silently. We spent more time fixing the pipeline than reading its output.

Karpathy's Core Insight

His idea is simple but changes everything:

Instead of retrieving from raw documents at query time compile knowledge once, keep it current, and query the compiled result.

The architecture has 3 layers:

raw/        → immutable source data (never edited)
wiki/       → LLM-maintained markdown knowledge base
AGENTS.md   → schema/rules file telling the LLM what to do

The LLM reads new raw data → integrates it into the wiki → cross-references it with existing knowledge → flags contradictions → the wiki gets richer every day.

His exact framing: "Obsidian is the IDE. The LLM is the programmer. The wiki is the codebase."

No vector database. No embeddings. No $35/month VM. Just markdown.

What I Built: LLM Wiki for SEO

I took Karpathy's pattern and applied it specifically to SEO monitoring for vibetrader.com.

The 3 Layers

raw/ : daily immutable snapshots:

raw/gsc/2026-04-17.json      ← Google Search Console (clicks, CTR, positions, queries)
raw/audit/2026-04-17.json    ← page audit (H1, meta, schema, canonical checks)
raw/commits/2026-04-17.json  ← landing-site git log (what code changed)

wiki/ : LLM-maintained knowledge:

wiki/overview.md              ← "story so far" updated daily
wiki/log.md                   ← append-only daily log
wiki/topics/keywords.md       ← keyword clusters + position tracking
wiki/topics/issues.md         ← open SEO issues with severity
wiki/topics/recommendations.md ← history of recs + acted/pending status
wiki/topics/code-changes.md   ← code change impact tracker
wiki/topics/performance.md    ← CAUSAL CHAINS: issue → fix → metric improvement
wiki/topics/competitors.md    ← weekly competitor analysis
wiki/topics/lint-report.md    ← weekly wiki health check

AGENTS.md : the schema file. Tells the LLM exactly how to update each page, what cross-links to write, what format to follow.

The Flow

3:00 PM IST  daily
─────────────────
gsc_pull.py    → pulls GSC data via API → raw/gsc/today.json
audit.py       → fetches vibetrader.com, checks H1/meta/schema → raw/audit/today.json
               → posts Slack audit digest (SEO/GEO/AEO scores)
ingest.py      → reads ALL raw/ + ALL wiki/
               → sends full context to Azure OpenAI
               → LLM updates all wiki pages with cross-links
               → appends to log.md
               → posts enriched Slack report
git commit     → wiki/ + raw/ committed back to main

4:00 PM IST  daily (1 hour after ingest)
─────────────────────────────────────────
fix_agent.py   → reads wiki/issues + recommendations
               → LLM classifies which issues are auto-fixable
               → skill functions run deterministically (not LLM-generated code)
               → raises PR to landing-site dev branch
               → never auto-merges

11:30 AM IST  Saturday
───────────────────────
lint.py        → audits wiki for staleness, contradictions, orphan pages
               → health score /100
               → saves lint-report.md + posts Slack

2:30 PM IST  Sunday
────────────────────
competitor_analysis.py → scrapes competitor sites
                       → LLM analysis → updates competitors.md

The Knowledge Graph in Plain Markdown

The key innovation over standard LLM Wiki I added a performance.md page that tracks causal chains:

### ✅ 2026-03-18 — API response fix
- **Cause:** [[topics/issues]] API returning incomplete pagination data
- **Fix:** API response items + pagination as separate fields → [[topics/code-changes]]
- **Before:** 127 clicks/day
- **After:** 229.7 clicks/day
- **Impact:** +81% clicks
- **Keywords moved:** [[topics/keywords]] — all branded queries improved
- **Confidence:** HIGH — spike confirmed same week as deploy

Every entry answers: what caused this? what moved? how confident?

This is what Cognee was trying to do with Neo4j store causal relationships between entities. We're doing it in plain markdown, with [[wiki-links]] that Obsidian renders as a visual knowledge graph.

Obsidian as the Visualization Layer

Open Obsidian → point it at your wiki/ folder → graph view instantly shows all cross-connections between pages:

performance ↔ keywords ↔ issues ↔ recommendations ↔ code-changes

As the wiki accumulates data daily, the graph gets richer. After 30 days, clicking any node shows a real story: "this keyword dropped because of this issue, which was fixed by this code change, which resulted in this metric improvement."

That's the knowledge graph. No Neo4j required.

Interactive Query Mode

Beyond the daily Slack push, I added a local query script:

# one-shot
python scripts/query.py "what keywords are improving this week?"

# REPL with follow-up context
python scripts/query.py
you> which code change had the most SEO impact?
wiki> Based on code-changes.md:
      March 18 — API response fix
      Before: 127 clicks/day | After: 229.7 clicks/day | Impact: +81%
      This is the highest confirmed impact change in the wiki.

you> why did that happen?   ← follow-up, remembers context
wiki> ...

The LLM answers from the accumulated wiki context not by re-reading 30 days of raw JSON. This is exactly what Karpathy's "query" operation describes.

Weekly Lint Pass

One gap in most LLM Wiki implementations no health check. I added lint.py that runs every Saturday:

Staleness detection : claims in the wiki that contradict newer log entries
Contradiction detection : two pages saying opposite things
Orphan pages : pages with no cross-links pointing to them
Missing concept pages: [[wiki-links]] referenced but never created
Stale recommendations: recs pending > 30 days

Posts a health score /100 to Slack and saves lint-report.md.

The Fix Agent (Skill-Based, Not LLM-Generated Code)

The fix agent reads the wiki, identifies fixable issues, and raises PRs to the landing-site repo. The key design decision: the LLM only classifies issues it never writes code.

Instead, deterministic "skill" functions handle each fix type:

skills/add_jsonld.py          ← checks schema not already present before adding
skills/add_internal_link.py   ← validates page exists in app/ first
skills/fix_meta_tags.py       ← only updates if explicit value given
skills/fix_robots_txt.py      ← safe append-only sitemap fix

The LLM outputs {"skill": "add_jsonld", "page": "app/layout.tsx"}. The skill runs. No hallucinated file paths. No duplicate schemas. No business logic touched.

PRs always target dev branch. Never auto-merge. Human review required.

vs Cognee: Honest Comparison

After running both systems in parallel for a week:

	Cognee	LLM Wiki
Infrastructure	Neo4j + ChromaDB + VM	Plain markdown files
Monthly cost	~$35	~$0 (just Azure OpenAI tokens)
Times it broke	3+	0
Files to maintain	6 JS pipeline files	5 Python scripts
History awareness	Inconsistent	Grounded references exact dates
AEO accuracy	3/10 (missed JSON-LD already added)	7/10 (reflected actual state)
Knowledge compounds	❌	✅

LLM Wiki won on every dimension except one: Cognee had richer graph relationships out of the box. But we replicated that with performance.md and [[wiki-links]].

What Karpathy Got Right

The core insight is correct and the more I use it the more obvious it becomes:

RAG = studying for an exam by re-reading all your textbooks every time you get a question.

LLM Wiki = you already made notes, highlighted the important parts, drew arrows between connected ideas. Now you just read your notes.

The wiki grows daily. The LLM gets smarter about your domain the more data accumulates. By day 30, query.py answers questions with 30 days of grounded context. By day 90, performance.md has a dozen confirmed causal chains real institutional knowledge about what moves your metrics and why.

That's what no RAG system gives you. And you don't need a vector database to get there.

Try It Yourself

The pattern works for any domain where you're accumulating knowledge over time:

SEO monitoring (what I built)
Competitor tracking
Research — papers, articles, building a thesis over weeks
Engineering team wiki — fed by Slack threads, PRs, incident reports
Personal second brain — journal entries, articles, book notes

The only thing you need:

A raw/ folder where data lands
A wiki/ folder for LLM-maintained markdown
An AGENTS.md (or CLAUDE.md) telling the LLM the rules
A cron job that runs ingest daily

Start with one raw data source and one wiki page. Let it run for a week. The compounding effect becomes obvious fast.

I built a 20-20-20 eye reminder because my eyes were dying at my desk

Haripriya Veluchamy — Thu, 16 Apr 2026 08:45:00 +0000

I have severe eye dryness. My doctor told me to follow the 20-20-20 rule every 20 minutes, look at something 20 feet away for 20 seconds. Simple. Except when you're deep in a bug or a feature, 20 minutes evaporates and you've been staring at your monitor for 3 hours straight.

I tried phone reminders. I dismissed them without thinking. I tried sticky notes. I ignored them. I needed something that actually blocked me from working until I did the break.

The key insight: a dismissible reminder is just noise. The popup had to be impossible to close until the 20-second countdown finished.

What it does

Runs silently in the system tray in the background
Every 20 minutes fires a loud beeping alarm (winsound, no external files needed)
Dark blocking popup appears X button is disabled, you cannot close it
20-second countdown runs automatically with a progress bar
The "Continue" button is locked until the countdown finishes, then turns green
Two buttons: Continue (back to work) or Stop (end the session)

The core trick: disabling the close button

The single most important line in the whole app:

win.protocol("WM_DELETE_WINDOW", lambda: None)

By overriding WM_DELETE_WINDOW with a no-op lambda, clicking the X does absolutely nothing. The window stays open. You have to wait out the countdown.

The countdown + locked button

The "Continue" button starts state="disabled" and only becomes clickable when the timer hits zero:

def tick():
    remaining["s"] -= 1
    count_var.set(str(max(remaining["s"], 0)))

    if remaining["s"] <= 0:
        continue_btn.configure(
            state="normal",
            bg="#003322",
            fg="#44ff88",
            cursor="hand2"
        )
    else:
        after_id["id"] = win.after(1000, tick)

The button turns green when it unlocks a satisfying visual reward for actually doing the break.

The alarm

No external audio files needed. winsound is part of the Windows standard library:

for _ in range(6):
    winsound.Beep(1000, 300)
    time.sleep(0.1)
    winsound.Beep(1400, 300)
    time.sleep(0.1)

Two alternating frequencies, six times. Annoying enough that you cannot ignore it. Runs in a daemon thread so it does not block the UI.

The full popup

def show_popup():
    result = {"action": None}
    root = tk.Tk()
    root.withdraw()

    win = tk.Toplevel(root)
    win.title("Eye Break!")
    win.configure(bg="#0a0a0a")
    win.attributes("-topmost", True)
    win.protocol("WM_DELETE_WINDOW", lambda: None)  # <-- the magic line

    # ... countdown, progress bar, locked button ...

    win.after(1000, tick)
    play_alarm()
    root.mainloop()
    return result["action"] == "continue"

The popup is always-on-top (-topmost True) so it cannot be buried under other windows.

Setup: auto-start on Windows boot

Step 1 Install dependencies:

pip install pystray pillow

Step 2 Save eye_reminder.py to a permanent location:

C:\Users\you\EyeReminder\eye_reminder.py

Step 3 Create a startup shortcut. Press Win+R, type shell:startup, and create a shortcut there pointing to:

pythonw C:\Users\you\EyeReminder\eye_reminder.py

Using pythonw instead of python means no terminal window appears on startup.

Step 4 Reboot. The app silently starts in your system tray. Right-click the tray icon to quit anytime.

Testing it

Before running with 20-minute intervals, test with 5 seconds:

TEST_MODE = True
INTERVAL_SECONDS = 5 if TEST_MODE else 20 * 60
BREAK_SECONDS = 5 if TEST_MODE else 20

Flip TEST_MODE = False when you are happy with it.

Dependencies

Package	Why
`tkinter`	Built-in — popup UI
`winsound`	Built-in — alarm beeps
`pystray`	System tray icon
`Pillow`	Required by pystray for the icon image

Core functionality (popup + alarm) works with zero installs. pystray and Pillow are optional the app works without them, just without the tray icon.

Full source

Harivelu0 / eye-remainder

eye-reminder

A lightweight Python app that enforces the 20-20-20 rule for eye health on Windows.

Every 20 minutes, a blocking popup appears with a loud alarm. You cannot dismiss it until a 20-second countdown finishes. No cheating.

Why

The 20-20-20 rule says: every 20 minutes, look at something 20 feet away for 20 seconds. Every reminder app I tried was too easy to dismiss. This one isn't.

Demo

[20 min timer] --> ALARM fires --> Blocking popup appears
                                        |
                                   20-sec countdown
                                        |
                               "Continue" button unlocks
                                        |
                              Click Continue --> back to work
                              Click Stop    --> session ends

Features

Runs silently in the system tray
Loud beeping alarm on every trigger (no audio files needed)
Popup is always-on-top and the X button is disabled
"Continue" button locked until full 20-second countdown completes
System tray icon with right-click Quit option
Built-in test mode (5-sec interval) for quick verification

Requirements

Windows
Python 3.8+
…

View on GitHub

My eyes are noticeably less dry after two weeks of using this. Sometimes the best tool is the one you build in an afternoon because nothing else works the way you need it to.

DEV Community: Haripriya Veluchamy

One Lookup Table Turned a Risky Server Migration Into a One-Line Change

The setup

The moment it mattered

The pattern, generalized

Where this shows up beyond servers

The lesson

Multi region deployment with Terraform modules

The problem

The pattern: separate "what's shared" from "what's different"

Why this matters beyond "less typing"

What I kept separate on purpose

The general lesson

The App Needed a Real Desktop, Not Just a Real Windows Machine

The plan that looked right on paper

What "did not work" actually looked like

The pivot, and the deeper thing it revealed

Why this isn't just a trading-software quirk

What ties this back to everything else in this series

Autoscaling Doesn't Fit Every Workload Here's How to Tell, and What to Build Instead

The property that actually matters: interchangeability

Scaling out: the fix is timing, not tooling

Scaling in: the part almost everyone skips

This isn't a niche problem real, large-scale systems already do this

The actual takeaway

Vault cant found What a Live Audit Found That Code Review Never Would

The setup

Building the audit, properly this time

What it found

How did nobody notice?

The part that actually surprised me

What I took away from this

Why Your CI Agent Can't Install pip

The setup

Attempt 2: ensurepip

The actual reason (worth understanding, not just working around)

Attempt 3: venv (also fails, same root cause)

Attempt 4: the actual fix get-pip.py

The one thing to double-check before you assume this is your problem

The takeaway

Why AI Can't Actually Hit Your Word Limit

It generates one token at a time, no plan

Words ≠ tokens

No counter anywhere in the architecture

The workaround

Why this clicked for me

Serving ML Artifacts from Amazon S3 Files How I used After the Launch

The Honest Story

What I Built

The Problem S3 Files Solves

Architecture

Setting Up S3 Files What Actually Matters

1. S3 bucket needs versioning enabled

2. IAM role trust principal is elasticfilesystem.amazonaws.com

3. The IAM role needs EventBridge permissions

4. boto3 API uses camelCase

5. Mount needs amazon-efs-utils, not plain NFS

6. EC2 instance needs AmazonS3FilesClientFullAccess

The Serving Code

Weekly Index Updates The Real S3 Files Advantage

Results

What I Learned

Resources

How I Built a Self Resizing EC2 for My ML Data

The Pain Point

The First Thought Let AWS ML Decide

The Second Thought Bedrock Agent

The Solution EventDriven VM Resize

Architecture

Setup 3 Steps

Step 1 Deploy AWS infrastructure

Step 2 Copy agent to your VM

Step 3 Add your pipeline steps

The Key Design Decision steps.conf

The Email Alert

Difficulties I Faced

Live Demo

Cost Reality

When to Use This

Why a VM and Not Glue or EMR?

Attempt 2: `ensurepip`

Attempt 3: `venv` (also fails, same root cause)

Attempt 4: the actual fix `get-pip.py`

2. IAM role trust principal is `elasticfilesystem.amazonaws.com`

5. Mount needs `amazon-efs-utils`, not plain NFS

6. EC2 instance needs `AmazonS3FilesClientFullAccess`