DEV Community: Haripriya Veluchamy

Two Autoscaling Policies, Zero Coordination: Why More Signals Made Scaling Worse

Haripriya Veluchamy — Sun, 20 Sep 2026 07:48:47 +0000

I had a fleet of stateful worker instances behind an Auto Scaling Group On-Demand
floor for guaranteed baseline capacity, Spot instances above that for burst. Standard
setup. Then we added a second scaling signal to fix a real gap in the first one, and
the fleet started flipping size every 30–90 seconds. This is the story of why that
happened, and why it wasn't a tuning problem.

The setup

Each instance in the fleet holds a fixed pool of "slots" a bounded number of
concurrent stateful sessions it can serve. Autoscaling started with the obvious
signal: CPU. A TargetTrackingScaling policy on average CPU across the ASG, scaling
out when the fleet got busy.

It worked, until it didn't. CPU is a fleet-wide average. An individual instance
could be completely full zero free slots, rejecting new sessions while the
fleet's average CPU still looked comfortable, because three other instances were
idle. CPU told you the fleet was fine. The thing that actually mattered was any
single instance out of room was invisible to it.

So we added a second signal: a CloudWatch Alarm watching per-instance free-slot
count, tied to a Step Scaling policy, firing when any instance hit zero free slots
for a sustained window. This is a completely reasonable instinct CPU is a lagging,
indirect proxy for capacity; free-slot count is the real thing.

What went wrong

Now there were two independent scaling authorities, reacting to two different
metrics, with no way to reconcile a disagreement. CPU's policy would look at the
fleet-wide average, see it drop after a burst, and scale in. The pool-alarm's policy
would look at the resulting per-instance squeeze, see zero free slots, and scale
right back out. Neither policy knew the other existed. Neither was wrong on its own
terms. Together, they fought.

Real observed behavior, pulled from describe-scaling-activities: the fleet size
flipping between 5 and 6 instances every 30 to 90 seconds, repeatedly, for minutes
at a stretch.

First attempt: tune the timing. Lengthened the pool alarm's sustained-breach
window from 3 minutes to 10, on the theory the two policies just needed more
separation. Deployed. The real activity log afterward showed it was still
oscillating just on a 5–6 minute cadence instead of 30–90 seconds. The timing
change reduced frequency. It didn't touch the actual disagreement. Two authorities
still existed, still had different opinions, still had no way to agree.

The actual fix: one authority, not two

The right move wasn't tuning either policy. It was removing one of them.

The metric that actually reflects real capacity is occupancy slots in use divided
by slots available, aggregated properly across the fleet, not two separate numbers
each policy interprets differently. CloudWatch Metric Math makes this
computable: occupied_per_instance = pool_max_size - avg(free_slots), targeting a
real utilization band (in our case, 75%).

Two things worth knowing if you're building this yourself, because both cost real
debugging time:

SEARCH() is not supported inside CloudWatch Metric Alarms. Only inside
Dashboards and direct GetMetricData calls. First attempt at a true fleet-wide
aggregate (summing free slots across every instance, dynamically, without hardcoding
instance IDs) used SEARCH inside a plain Alarm's metric query. PutMetricAlarm
rejected it outright not a syntax error, a hard platform limitation. Confirmed
against AWS's own documentation, not assumed.

Target Tracking policies support Metric Math as a separate, genuinely different
feature from Alarms live since December 2022, and not subject to the same
SEARCH restriction in the same way, because Target Tracking evaluates the metric
expression directly rather than routing through the Alarms evaluation engine. This
is the detail that made the fix possible: replace both old policies the CPU
target-tracking policy and the pool-based step-scaling alarm with a single
TargetTrackingScaling policy computing occupancy via Metric Math.

One authority. One metric that actually reflects real capacity. Nothing left to
disagree with.

Verified before applying terraform plan showed exactly the intended diff, one
resource added, three removed, nothing else touched. Applied cleanly on the first
real attempt. Confirmed afterward via describe-policies: exactly one scaling
policy exists. No oscillation since.

The generalizable part

This wasn't really an AWS-specific bug. It's a distributed-systems pattern that
shows up anywhere two control loops react to different observations of the same
underlying system state, without a shared source of truth:

Each loop is locally correct it's doing exactly what its own metric tells it to.
Neither loop is aware the other exists, let alone that it just undid the other's decision.
The oscillation isn't a bug in either policy. It's an emergent property of running two of them.

The fix generalizes too: don't tune the disagreement, find the metric that actually
represents the thing you care about, and have exactly one thing making the decision
based on it. If you're tempted to add a second scaling signal because the first one
has a real, valid blind spot that blind spot is real, but the fix is usually a
better single metric, not a second policy running in parallel.

AWS building blocks used: EC2 Auto Scaling Group (On-Demand + Spot mix), Target
Tracking Scaling Policies, CloudWatch Metric Math, CloudWatch Alarms, Step Scaling
Policies, SNS (for the alert path that first surfaced the oscillation as noisy
paging).

I Spent a Month on Architecture The Migration Took Two Days

Haripriya Veluchamy — Tue, 15 Sep 2026 07:22:06 +0000

I recently moved a production backend service from one cloud provider to another. Solo. Zero downtime. Real users, real traffic, no maintenance window.

The real reason I moved it: auto-scaling and spot instances cheaper, flexible compute that scales up and down with demand never worked cleanly on the old provider. I kept fighting it instead of benefiting from it. On top of that, I kept running into permission issues that had nothing to do with what I was actually trying to do. Both were signals the same thing: time to move to a cleaner setup, on infrastructure we already had access to, where I could design things properly instead of working around old debt.

The actual migration took two days. Getting the architecture right took a month.

I used to think migration was the hard part. It's not. Design is.

Here's what that month actually looked like including the parts I got wrong before I got them right.

The wrong turn I almost shipped

The old system had a setup step: each machine needed to establish its own identity before it could do real work. The old approach was simple configure one machine correctly, capture its state, copy that state onto every new machine. It had worked fine there for a long time.

I assumed the same approach would work on the new provider. It didn't. Figuring out why took three tries.

Try 1: captured the state, copied it to new machines. Every one failed the same way.

I traced it back to how the original setup was done through a remote session, by clicking an icon, instead of launching the software with a flag that tells it to use an isolated local config. Without that flag, the setup silently saved itself somewhere else entirely. The "source" machine was never actually configured right in the first place.

Try 2: fixed that, redid the setup, recaptured, recopied. Same failure, on a different fresh machine, even though I'd verified the source was correct this time.

That ruled out my first theory.

Try 3: tried a literal clone of the working machine, same hardware profile, no changes. Same failure. Ruled out hardware identity too.

I also tried automating the manual setup step with a scripting tool, so no human would ever need to do it. It could find the right window real progress but couldn't actually type into it. A session can exist without a real display attached, and that turns out to matter.

The actual fix: stop copying state entirely. Install fresh, log in through a plain config file, let each machine do its own setup from scratch. No copying, no cloning. It worked immediately, on the very next machine, with zero manual steps.

The lesson: don't copy identity between machines. Let each one earn its own.

The number I didn't expect

Before rolling this out for real, I ran an actual load test real concurrent usage, not a guess.

The new machine had the same CPU as the old one, just more memory. I assumed load would scale roughly the way it had before. It didn't. At light load, CPU sat around 22%. At moderate load, it jumped to 92% nowhere near a straight line.

I lowered the safe capacity limit by about half, based on that real number, instead of keeping the number that had worked on the old setup. More memory didn't matter here the real limit was CPU, and the only way to find that was to actually test it.

The same test also caught a real bug: one failed connection was silently blocking the next unrelated one, because two different failure types looked identical to my code. Only showed up under real concurrent load.

A trade-off I made on purpose

Once more than one machine was handling traffic behind a shared entry point, a returning user could land on a different machine than before. Fixing that properly meant building a much more complex routing layer.

I chose not to build it. The system already had a way to recover gracefully when that happened it just set itself up fresh wherever it landed. Building the complex fix would have solved a problem the system already tolerated fine. I wrote the trade-off down clearly, including exactly when I'd need to revisit it.

Choosing the entry point on purpose

Two reasonable options existed for routing traffic in: a simple passthrough, or a heavier one that handles encryption itself. The service already handled its own encryption. Picking the heavier option would've meant solving a problem I didn't have. I went with the simple passthrough.

I also rolled it out in two small, safe steps instead of one big one: first, add the new entry point alongside the old direct access, so I could prove it worked with zero risk. Only once that was confirmed did I remove the old path. Two small reversible steps beat one big risky one.

Scaling the actual reason I moved

This was the part I was most careful about, since it's what drove the whole move.

The setup could scale a small always-on baseline, with cheaper burst capacity added on top when needed. That part was straightforward.

What I deliberately didn't do yet: build a fully automatic policy that scales on its own based on load. I didn't have real usage data yet, and an automatic policy built on a guess is worse than no policy it either reacts too late or too early, and you won't know which until it's live.

Instead, I scaled manually for the specific period it mattered, watching real alerts. That gave me real numbers under real load. Only after that did building a real automatic policy become a genuine next step instead of a guess.

Designing access properly this time

Since permission friction was part of what pushed this move in the first place, I made sure not to carry the same problem forward. I reviewed every broad permission the new setup had, kept the ones that genuinely couldn't be scoped any narrower, and added explicit restrictions on the sensitive actions that could be. That review caught one real gap before it caused a problem a resource was missing a tag that a new safety rule depended on, which would have accidentally blocked a legitimate action.

The principle: design access narrowly from day one. It's much harder to tighten permissions later than to grant them as they're actually needed.

Automating the pipeline, deliberately

I set up the deployment pipeline to be fully automatic any change that passes its checks deploys on its own, no manual approval step in between.

This was a deliberate choice, not laziness. If every deploy needs someone to manually approve it, that someone becomes a bottleneck and in practice, that someone is me, getting pulled in for routine changes that don't actually need a human gate. Automating it means any developer on the team can ship their own change without needing to loop me in every time. Fewer interruptions, faster shipping, same real safety controls just enforced differently through proper checks and scoped permissions, not a manual click.

Alerts for everything that can fail

Simple rule I followed throughout: if something can fail silently, it needs an alert. Not one flat "something's wrong" notification different problems need different urgency. A minor slowdown shouldn't page the same way a real outage does.

Getting this right early meant I could rely on watching alerts instead of guessing, during the manual-scaling window above and afterward.

Retiring the old system carefully

Once the new setup was proven, I didn't just delete the old one. Before removing anything, I checked what was still actually depending on it and found one thing still quietly running against it that I would have broken. Instead of accepting that disruption, I fixed the actual gap first, confirmed nothing else was still depending on the old system, and only then removed it for good.

The takeaway

None of these decisions were obvious going in. Each one meant testing an assumption instead of trusting it that state would copy over cleanly, that load would scale the way it used to, that an old permission setup was fine to carry forward.

The month wasn't spent writing documents. It was spent finding out which assumptions were wrong before they became real problems. That's what made the two-day execution possible not extra planning for its own sake, but planning that had already been tested against reality.

Curious if others have hit the same wall with auto-scaling or spot instances on a provider that just doesn't handle it cleanly feels like a more common trigger for switching providers than people usually admit.

I Built a Production Grade ML Candidate Ranking System

Haripriya Veluchamy — Wed, 09 Sep 2026 17:19:32 +0000

The Challenge

Three months ago, I participated in the Redrob India Data & AI Challenge. Track 1: given 100,000 candidate profiles and a job description for a Senior ML/AI Engineer role, build an AI system that ranks candidates the way a great recruiter would not by matching keywords, but by actually understanding who fits.

The output needed to be a ranked CSV of the top 100 candidates, with reasoning for each pick.

I'm a cloud and backend engineer, about 2 years into my career. I've never formally studied machine learning. I learned what I needed by building things. This felt like exactly that kind of problem.

Why Keyword Matching Fails

Before jumping into what I built, here's the thing that bothered me about standard systems.

A candidate who writes "built dense retrieval pipeline serving 50M+ queries" and another who writes "implemented embedding-based search system" are saying the same thing. BM25 the industry standard keyword matcher sees two completely different documents because the words don't overlap.

Meanwhile, a verbose mediocre engineer who keyword-stuffs their resume scores higher than a terse brilliant engineer who just ships.

And humans doing this at scale have their own version of the same problem: juggling 20+ signals inconsistently across hundreds of profiles. Notice period, location, GitHub activity, company prestige, career trajectory, skill depth no one can hold all of that in their head fairly.

I wanted to build something that combines signals the way a thoughtful recruiter actually would.

The Architecture: Don't Depend on Any Single Signal

The core insight I started with: no single signal is trustworthy alone.

BM25 misses semantic matches. Embedding similarity misses keyword-heavy roles. Skill matching without trajectory context rewards buzzword collectors. Trajectory scoring without behavioral signals misses the person who's perfect on paper but wants 6 months notice.

So I built a 4-signal system where every signal checks the others.

The Formula

score = (0.20 × BM25 + 0.30 × FAISS + 0.20 × Skill + 0.30 × Trajectory)
      × yoe_factor
      × ml_product_years_factor
      × gate_multiplier
      × behavioral_gate

Let me explain each piece.

Stage 1: Precompute Everything (once, ~1.7 hours)

I split the pipeline into two stages. Stage 1 is the heavy lifting run once, save everything to disk. Stage 2 is pure math runs in under 2 minutes on CPU.

Signal 1: BM25 (weight: 0.20)

Classic keyword search using BM25Okapi. I built a query from the JD specific terms like "vector search", "FAISS", "learning to rank", "production ML", "retrieval" and scored all 100K candidates against it.

BM25 is fast and good at catching explicit keyword matches. It's also easily gamed, which is why it only gets 20% weight.

Signal 2: FAISS Cosine Similarity (weight: 0.30)

I encoded all 100K career text descriptions using BAAI/bge-small-en-v1.5 into 384-dimensional vectors, then searched with the JD vector using FAISS IndexFlatIP.

Why BGE-small over the more common all-MiniLM-L6-v2? On MTEB retrieval benchmarks, BGE-small scores 61.7 vs MiniLM's 56.9. That's an 8.4% improvement and with the same 384 dimensions and similar inference speed. For a retrieval task specifically, it matters.

BGE also uses asymmetric encoding: the JD (query) gets a prefix:

jd_text = "Represent this sentence for searching relevant passages: " + jd_embed_text

Candidate texts are encoded without prefix. This is the correct BGE usage for retrieval and it makes a real difference in practice. Meta's Lead AI Engineer jumped from rank 13 to rank 9 after switching from MiniLM to BGE-small.

Signal 3: Skill Quality (weight: 0.20)

For every skill in the dataset, I computed cosine_similarity(skill_embedding, jd_vector) to get a relevance score. Then for each candidate:

skill_quality = Σ (skill_relevance × proficiency_weight × min(1.0, duration_months / 24))

So "Expert in Vector Search for 36 months" beats "Beginner in Vector Search for 48 months". And only JD-relevant skills count Figma proficiency doesn't help a ranking engineer.

Signal 4: Trajectory (weight: 0.30)

This is the most novel signal and the one I'm most proud of.

trajectory = 0.35 × production_score
           + 0.25 × pre_llm_score
           + 0.15 × still_coding
           + 0.10 × title_progression
           +        prestige_bonus

Production score: I scanned career descriptions for production markers "shipped", "deployed", "serving N users", "A/B test", "latency", "99th percentile". Engineers who actually shipped things use this vocabulary. Those who only researched don't.

Pre-LLM ML depth: Total months spent on scikit-learn, PyTorch, XGBoost, TensorFlow, FAISS, Spark MLlib. JD explicitly says "if you learned ML after the LLM wave, this role isn't for you." I implemented a soft ramp to 48 months instead of a hard cutoff so someone with 3 skills × 18 months each scores properly instead of getting zeroed out.

Still coding: GitHub activity score from the platform signals + coding keywords in current role. Penalizes engineers who've moved entirely into management.

Title progression: Senior/Lead/Principal/Staff titles get a 1.30× multiplier on the title progression component.

Prestige bonus: This is data-driven from the candidates' own profile fields. Instead of a hardcoded list of "good companies", I read industry and company_size directly from the JSONL. Software + 10000+ employees → 0.15 bonus. Fintech/SaaS/AI + mid-size → 0.10 bonus. Only the FAANG tier (Google, Meta, Apple, etc.) gets a hardcoded 0.20 override. This means Razorpay, Zomato, Paytm automatically get classified correctly based on their industry + size no manual list maintenance.

The Gates: Hard Filtering Before Ranking

Some candidates shouldn't be in the top 100 regardless of score. I implemented hard gate multipliers:

Condition	Multiplier
Honeypot (impossible profile data)	× 0.00
Anti-title (Operations Manager, HR, Accountant...)	× 0.05
Consulting-only career (TCS, Infosys, Wipro entire career)	× 0.20
CV/Speech primary domain	× 0.15
Pure research (no production evidence)	× 0.15
Framework-only (only LangChain/LlamaIndex, no real ML)	× 0.20

And a behavioral gate (range 0.50–1.20) that accounts for:

Notice period (≤30 days → +0.15, >120 days → -0.10)
Location (Pune/Noida → +0.20 per JD)
Willingness to relocate (+0.10)
Overseas without relocation intent (-0.30)
GitHub activity signals

Years of Experience Soft Multipliers

The JD says 5–9 years. I implemented:

Under 3.5yr → hard exclude (hard gate)
3.5–5yr → max(0.40, yoe / 5.0) soft penalty
5–9yr → 1.0× (ideal)
Above 9yr → taper: max(0.85, 1.0 - (yoe - 9.0) × 0.02)

And separately: ml_product_years years spent in applied ML roles at product companies (not consulting). JD explicitly requires "4-5 years in applied ML at product companies." This was a direct scoring multiplier: min(1.0, 0.50 + (ml_yrs / 4.0) × 0.50).

The Reasoning Layer

Every candidate gets a written explanation in the CSV. No LLM involved all facts come directly from the raw profile.

Three tiers based on rank:

Ranks 1–10: Full evidence company, ML years at product companies, GitHub activity score, and a recent work description snippet.

5.9yr exp, currently Senior AI Engineer at Apple; top skills: scikit-learn,
TensorFlow, Python; 5.8yr at ML product companies github_activity=97;
Trivandrum, Kerala; immediately available (30d notice). Recent: Built and
shipped a production recommendation system at a marketplace product, going
from offline experimentation to live A/B test in 5 months.

Ranks 11–50: Summary with top skills, location, availability, and the most recent work description where not duplicated.

Ranks 51–100: Gap analysis honest about why they ranked lower, with specific concerns (job-hop rate, limited ML product tenure, semantic mismatch).

I also de-duplicated reasoning synthetic data had identical career descriptions, so I tracked seen descriptions and skipped the "Recent:" snippet for duplicates, preventing identical reasoning entries in the CSV.

Problems I Actually Hit

1. `ml_product_years` was computed but never wired in

I had a whole function that computed years in applied ML at product companies. It was stored in features.pkl. It was never referenced in rank.py. The JD literally says this is a requirement. I caught it during a cross-read of the scoring formula. Fixed by adding it as a multiplier.

2. Prestige was hardcoded

My first version had a list of "good companies" hardcoded. That's not data-driven it's bias. Switched to reading industry + company_size from the JSONL. Razorpay (Fintech + 5000+ employees) now auto-classifies correctly. Infosys (IT Services) auto-penalizes. No manual list maintenance.

3. Pre-LLM 24-month hard cutoff

I had if pre_llm_months < 24: score = 0. A candidate with 3 skills × 18 months each = 54 months total but no single skill over 24 months got zeroed out. Wrong. Switched to min(1.0, deep_months / 48.0) a soft ramp to 48 months.

4. HuggingFace rejecting binary files via git

HF now uses Xet storage for binary files and rejects them via regular git push. I had 154MB FAISS index and 175MB BM25 index. Solution: uploaded them via huggingface_hub Python API to a separate HF Dataset repo (Haripvelu/redrob-artifacts), then used hf_hub_download() in the demo to fetch them on first run.

5. `TODAY` hardcoded

I hardcoded TODAY = date(2026, 6, 13) during development and forgot to change it. It was used for computing "days since last active" in the behavioral gate. Fixed to TODAY = date.today().

6. Reasoning cut mid-sentence

I was truncating reasoning at 200 characters with a word-boundary cut. "Built and shipped a production recommendation system at a marketplace product, going from..." got cut after "going" sometimes. Switched to sentence-boundary detection using rfind('.') within 400 chars.

Results

Top 10: Apple, Salesforce, Mad Street Den, Zomato, Amazon, Razorpay, Ola, Microsoft, Meta, Netflix. All product companies. All India-based. All 5–9yr YoE range. All ML/NLP/search roles.

After switching to BGE-small:

Meta Lead AI Engineer: rank 13 → rank 9 ✅
Zomato Senior ML Engineer: rank 8 → rank 4 ✅
Google Search Engineer: rank 17 → rank 40 ✅ (their actual skills were SAP, YOLO, Kubeflow not a semantic match for NLP/search despite the "Google" halo)

Evaluation:

62/62 custom eval checks passing (format validation, artifact integrity, signal sanity, adversarial probes, ranking quality)
48/48 unit tests passing
0 honeypots in top 100
0 consulting-only in top 100
validate_submission.py (official checker) passes

Runtime: ~1.7 hours precompute (one-time), under 2 minutes to rank 100K candidates on CPU.

What I'd Do Next

Learning to Rank: Replace the hand-tuned weights with LambdaMART trained on historical recruiter feedback. The 0.20/0.30/0.20/0.30 weights were arrived at through experimentation and intuition real feedback data would make them rigorous.

LLM Re-ranker on top 50: Add a small local model (Phi-3 Mini) to re-rank the top 50 after the initial retrieval. Still CPU-friendly, but adds semantic reasoning the embedding similarity can't capture.

Dynamic JD embedding: The JD embed text is currently hardcoded in precompute.py. The pipeline should accept any JD and re-embed at precompute time. That makes it a general-purpose ranking system, not a one-role system.

Three Months Later

I submitted everything. GitHub, HuggingFace Space with live demo, ranked CSV, PDF deck answering all the required questions. I waited.

Still waiting.

I don't know if results were announced privately, if the challenge was extended, or if it quietly ended. I'm not bitter I got something valuable regardless: a real end-to-end ML pipeline I built from scratch under pressure, a working understanding of hybrid retrieval systems, and a few bugs caught that I'd have shipped to production.

If you're building something similar a hiring system, a recommendation engine, a search ranker the core lesson is: don't depend on any single signal. Ensemble them. Gate the obvious failures hard. Be honest in your reasoning. And build the eval suite before you think you need it.

Code

GitHub: github.com/Harivelu0/redrob-ranker

Live demo: huggingface.co/spaces/Haripvelu/redrob-ranker

The Space runs the full BM25 + FAISS pipeline (not a fake demo) the large indexes are fetched from HF Dataset on first click. Takes ~45 seconds to download, then ranks instantly.

I'm Haripriya (hp). I'm a software engineer focused on cloud, backend, and ML infrastructure. I write about building real things.

Why I Stopped Clicking "Create Alert" in the Cloud Console

Haripriya Veluchamy — Sun, 06 Sep 2026 15:12:06 +0000

What I used to do

Every time I set up a new server, I'd go into the cloud console and manually click through the UI to set up alerts "tell me if CPU goes above 80%", "tell me if this server stops responding", that kind of thing.

It worked. For one server.

Where it broke

When I had 5 servers instead of 1, I noticed something: only some of them had alerts. A couple of new servers had none at all.

Why? Because setting up alerts by clicking through the console is a manual step. And manual steps get forgotten. I'd create a new server, get it running, move on to the next thing and just... not go back and click through the alert setup again.

Nobody did anything wrong. It's just what happens when "add monitoring" depends on a human remembering to do it every single time.

What I changed

Instead of clicking through the console, I wrote a small script that creates the alerts for me using code, not clicks.

Now, whenever I create a new server, I run the same script, and it automatically sets up the same alerts every time: heartbeat check (is it alive?), CPU check, error-rate check. Every server gets the exact same coverage, because it's the same script every time not me remembering to click 10 buttons in the right order.

Why this is better

Before (manual clicking):

Easy to forget for a new server
Easy to accidentally set it up slightly differently each time
No record of what alerts exist or why

After (script):

Every new server automatically gets the same alerts
The script itself is proof of what alerts exist I can read the code
If I want to change something (like the CPU threshold), I change it once in the script, and it's correct for every server going forward

The simple lesson

If you're creating something by clicking through a UI more than once, that's usually a sign you should write a script instead. It's not about being fancy it's that scripts don't forget, and clicking does.

This applies to alerts, but it applies to almost anything repetitive in the cloud: creating servers, setting permissions, configuring backups. If you do it more than once by hand, write it down as code once, and just run that instead.

How i did the Multi VM monitoring

Haripriya Veluchamy — Tue, 01 Sep 2026 07:09:08 +0000

I Had 5 Servers Sending Metrics But Couldn't Tell Which One Was Which

What happened

I had one server. It sent metrics (CPU, errors, etc.) to my monitoring dashboard. Fine.

Then I added more servers same service, running on 4-5 machines now.

All the metrics from all the machines showed up in the same dashboard, but with no way to tell which machine sent which metric. If CPU spiked, I couldn't tell if it was server 1 or server 4. If errors went up, same problem.

This is useless when something breaks. "Something is wrong" isn't helpful I need "server 3 is wrong."

Why it happened

Each server was supposed to tag its own metrics with its own name like a label saying "this metric came from server-3."

I had set up a config value meant to do exactly that. But when I checked, that name was never actually reaching the code that sends metrics. The setting existed in a file, but it wasn't making it into the running program the way I assumed it would.

So every server, silently, was sending metrics with no name attached or the same default name. No error. No warning. It just quietly didn't work.

How I fixed it

Instead of hoping a config file setting would "just work" and reach the right place automatically, I made it explicit:

When a server starts up, its name gets set directly, as a hard requirement.
That name gets passed by hand, step by step, into the exact piece of code that sends metrics.
Nothing is assumed to "just be available" it's handed over directly, on purpose, every time.

I checked it worked by looking at the actual metrics in the dashboard afterward and confirming each server's name showed up correctly.

The real lesson

If your app depends on a setting "just working" through some automatic config-loading magic double check it's actually arriving where you think it is. Don't assume. Go look.

And before you scale from 1 server to many: make sure you can tell them apart in your monitoring before you actually need to, during a real problem, at 2am, with no way to know which machine to even look at.

I did Golden Images

Haripriya Veluchamy — Wed, 26 Aug 2026 15:23:15 +0000

Golden Images How I Stopped Manually Logging Into Every New Server

The problem

Every time I spun up a new server for a service, it worked but it wasn't actually ready. There was always one manual step left: log in, run through some interactive setup, get the application into a working state. Only after that could the server actually do its job.

For one server, that's a minor annoyance. For a fleet that's supposed to scale up and down on demand, it's a dealbreaker. You can't call something "automated provisioning" if a human still has to remote in and click through a setup wizard before it's usable.

The fix: capture the setup once, replay it everywhere

The pattern here is usually called a golden image and the idea is simple: instead of repeating a manual setup step on every new machine, do it once, capture the result of that setup, and have every future machine apply that captured state automatically during provisioning.

Concretely, I built a small tool that:

Connects to a machine that's already been through the manual setup and is in a known-good state.
Packages up just the state that setup actually produced not the whole machine, just the specific files/config that resulted from the manual steps.
Uploads that package to storage, versioned.

Then the provisioning script for every new machine downloads that package and applies it automatically as part of boot no human, no remote session, no wizard.

The mistake worth mentioning

My first version of this captured too much. Instead of packaging just the setup-derived state, it grabbed an entire application data folder which included the application's own installed binaries, not just the configuration that setup had produced.

That meant every new machine, when it applied the "golden" package, got its fresh application install silently overwritten with whatever binary version happened to be running on the machine I captured from. New servers ended up running an older version of the software than the one they'd just installed a regression that was confusing to trace, because nothing had "changed" from the provisioning script's point of view.

The fix was narrowing the capture to exactly the two or three subfolders that actually held setup-derived state, and leaving the application's own installation untouched. Smaller package, and it stopped fighting the install step instead of complementing it.

Why this is worth doing before you scale

The manual-setup step is easy to live with when you have one or two servers it's a one-time cost. It stops being easy to live with the moment you want:

Autoscaling a new instance needs to be usable within seconds of being created, with zero human involvement.
Disaster recovery if a machine dies, replacing it shouldn't require someone to remember and redo a manual setup checklist from memory.
Consistency every machine that skips the manual step (or where someone does it slightly differently) is a machine that behaves subtly differently from the rest of the fleet.

A golden image turns "someone has to set this machine up" into "every machine sets itself up, identically, using the same captured state." It's the same principle as infrastructure-as-code, applied to the data a machine needs rather than just its configuration.

The general lesson

If any part of your provisioning process still requires a human to log in and do something interactive even something that "only takes two minutes" that's the step blocking real automation. Capture what that manual step actually produces, package just that, and make applying it part of the automated boot process.

And when you build the capture step: be precise about what you're capturing. It's tempting to grab "the whole folder" for simplicity, but bundling setup-derived state together with things that shouldn't be frozen (like application binaries) turns your automation tool into a tool that silently reintroduces old versions. Capture the smallest thing that actually represents "the setup happened," and nothing more.

One Lookup Table Turned a Risky Server Migration Into a One-Line Change

Haripriya Veluchamy — Tue, 18 Aug 2026 16:28:54 +0000

The setup

I was running a system where multiple backend servers each served a set of users, and a routing layer decided which server a given user's requests should go to. Nothing unusual this is a common shape for anything that scales past one machine.

The question that turned out to matter a lot more than I expected: when something in the routing layer refers to "the server," what is it actually storing? A real address? Or a name that gets looked up?

The moment it mattered

At some point I needed to move a group of users off one server and onto another the kind of thing that happens for all sorts of reasons: retiring old hardware, rebalancing load, recovering from an incident.

Before touching anything, I checked how the routing data was actually structured. It turned out every record referenced servers by a stable label not a raw address. The label was just an ID; a separate lookup table mapped each label to wherever that server actually lived right now.

That one fact changed the entire migration from "rewrite a bunch of records for every affected user" into "update one row in the lookup table." The routing records themselves never needed to change. Every user still pointed at the same label the label's meaning just changed.

I verified it worked by checking the routing behavior for an affected user right after the update, before assuming anything.

The pattern, generalized

This is the indirection pattern, and it's worth naming explicitly because it's easy to skip when you're building the first version of something:

Direct reference: Store the real, resolvable thing (an IP address, a file path, a specific resource ID) everywhere it's used.
Indirect reference: Store a stable name/label everywhere it's used, and keep exactly one place that resolves that label to the real thing.

Direct references feel simpler at first there's no lookup step, nothing extra to maintain. But that simplicity is a trap: the real address ends up copied into every place that uses it, and moving the real thing means finding and updating every one of those places.

Indirect references cost you one extra lookup table. In exchange, moving the real thing becomes a single update, in a single place, and everything downstream is unaffected because none of it ever knew the real address to begin with.

Where this shows up beyond servers

The same trade-off appears constantly:

DNS names vs. hardcoded IPs this is the same pattern at internet scale. Change the DNS record, nothing downstream needs to know.
Feature flags referencing a config key vs. a hardcoded value the flag's name stays stable while what it resolves to changes.
Database foreign keys vs. duplicating a denormalized value everywhere the ID stays stable; look up the current details when you need them.
Service discovery vs. hardcoded service addresses exactly the migration scenario above, generalized to any service-to-service call.

In every case, the question is the same: if the "real thing" changes, how many places need to be touched? If the answer is "more than one," you're using a direct reference somewhere it should have been indirect.

The lesson

Before building the first version of anything that will eventually need to move, scale, or be replaced a server, a config value, a downstream dependency decide up front whether callers will reference it directly or through a label that gets resolved. It's a small design decision early on, and it's the single biggest factor in whether a future migration is a five-minute change or a project.

If you're not sure whether you have this in place already, ask the question I asked before touching anything: what is actually stored in the place that decides where a request goes a real address, or a name? If it's a real address, that's worth fixing before you need to move it under pressure.

Multi region deployment with Terraform modules

Haripriya Veluchamy — Thu, 13 Aug 2026 16:59:01 +0000

The problem

I was managing infrastructure across multiple regions a handful of virtual machines, one per region, each running the same service. The Terraform setup had grown the way these things usually do: when I needed a VM in a new region, I copied the folder for an existing region, renamed a few things, and adjusted the values that were different (location, network ranges, naming).

It worked. Until it didn't.

Every region had its own full copy of:

The provider block
The resource definitions (VM, network interface, disk, public IP, etc.)
The output block

Three regions in, I had three nearly-identical folders, each a few hundred lines, differing in maybe ten lines of actual content. Any time I wanted to change something structural add a tag, adjust a disk size, fix a naming convention I had to make that change in every single folder, and hope I didn't miss one or introduce a subtle inconsistency between them.

This is the classic copy-paste infrastructure trap: it feels fast in the moment (just copy the folder!) but every region you add makes the next change more expensive, not less.

The pattern: separate "what's shared" from "what's different"

The fix was to stop treating each region as its own Terraform project and instead treat the infrastructure definition as shared code, with only the differences expressed as data.

Concretely, that meant three layers:

1. A shared module one Terraform module containing the actual resource definitions (VM, networking, disk, etc.), written with input variables for everything that varies between regions: location, name suffix, network ranges, and so on.

2. One shared entry point a single root configuration that calls the module, rather than one root configuration per region.

3. A small config file per region instead of a full copy of the module, each region gets a short file (five or six lines) specifying just its own values: which cloud location to deploy into, what to name things, what network ranges to use.

Adding a new region now means adding one small config file. No new Terraform resource code, no new pipeline definition, nothing to copy and rename.

Why this matters beyond "less typing"

The obvious win is less duplication. The less obvious win is correctness:

A structural fix only needs to happen once. Change the module, every region picks it up. Before, a structural change meant editing N folders and trusting yourself to get all N edits identical.
Drift between regions becomes visible. With copy-pasted folders, two regions can quietly diverge over time as one gets a fix the other doesn't. With a shared module, divergence can only happen in the small config file which is short enough to diff at a glance.
Review gets easier. A pull request adding a new region is now a five-line diff to a new config file, not a few hundred lines of near-duplicate resource code that's hard to review carefully.

What I kept separate on purpose

One thing worth calling out: sharing the code doesn't mean sharing everything. Each region still keeps:

Its own Terraform state file, so a mistake applying to one region can't touch another region's resources.
Its own deployment approval gate in the pipeline, so promoting a change to one region doesn't silently promote it everywhere.

The pattern is "share the how, isolate the where it lands." Sharing a module is safe. Sharing a blast radius is not.

The general lesson

If you find yourself about to copy a folder to stand up a new instance of something a new region, a new environment, a new tenant that's usually the signal to stop and ask: what's actually different here? Usually the answer is "not much" a handful of values. Everything else can be a shared module, parameterized by those values.

Copy-paste feels like the fast path. It's fast for the first copy. It gets slower and riskier every time after that.

The App Needed a Real Desktop, Not Just a Real Windows Machine

Haripriya Veluchamy — Sat, 08 Aug 2026 17:37:06 +0000

The App Needed a Real Desktop, Not Just a Real Windows Machine

Before any of the debugging stories, gotchas, or architecture arguments in this series existed, there was one earlier decision that had to fail first and it's the one that actually explains why everything after it looks the way it does.

The plan that looked right on paper

The system in question wraps a Windows-only, GUI-first desktop application the kind of software that ships as a .exe, expects to be installed via a wizard, and exposes automation only through a first-party SDK that talks to the running application process. No web API, no headless mode, nothing designed for servers at all.

The instinct, reasonably, was: don't fight that. Run it inside a Windows compatibility layer, in a Linux container. Keeps everything in the same containerized, cloud-native tooling as the rest of the stack. Cheaper to host. Fits the mental model everything else in the system already used.

It did not work.

What "did not work" actually looked like

Not a clean, informative failure a slow, grinding one. The automation layer would intermittently fail to connect to the running application with low-level timeout errors, with no consistent trigger. Sometimes it worked for hours. Sometimes it failed within minutes of a restart. The compatibility layer itself would occasionally become unstable in ways that were hard to attribute to any single cause. Every fix felt like it addressed a symptom, and a slightly different symptom would show up a few days later.

Eventually the right call was: stop trying to make this specific approach reliable, and question whether the approach itself was ever going to be reliable rather than continuing to patch a foundation that might be structurally wrong.

The pivot, and the deeper thing it revealed

The fix was to abandon the compatibility-layer approach entirely and run the application on an actual Windows machine, natively. This alone mostly solved it but "mostly" is doing real work in that sentence, because it surfaced a second, more specific requirement that the first failure had been obscuring the whole time.

Running on real Windows wasn't sufficient by itself. The application's automation layer would still fail to connect if the underlying process was running in a headless or service-style context even on genuine Windows, even with no compatibility layer involved at all. It only worked reliably when the process was running inside a real, interactive, logged-in desktop session the same kind of session that exists when an actual person is sitting at the keyboard, not a background service context that happens to also be "on Windows."

This turned out to be the actual root requirement the whole time. The compatibility-layer approach hadn't just been unstable in some vague sense it was fundamentally incapable of providing this, no matter how much it was tuned, because a compatibility layer running headless in a container was never going to look like an interactive desktop session to begin with.

Why this isn't just a trading-software quirk

I think this generalizes further than it looks. A meaningful category of Windows software licensed engineering tools, certain legacy financial or back-office systems, some CAD and design applications was written assuming a human is logged in: window handles exist, GUI event loops are running, certain OS-level session facilities are available that simply don't exist in a headless context. None of that is a bug in the software. It's just an assumption baked in from an era when "run this on a server" wasn't a use case anyone designing it considered.

If you're trying to automate something in this category in the cloud, "get it running on Windows" is necessary but not sufficient. The actual question to ask early ideally before building anything around it is: does this specific piece of software's automation surface require a real interactive session, or does it genuinely work headless? That's usually one focused test, and it's a much cheaper question to answer on day one than to discover, the way this system did, after weeks of chasing intermittent failures in the wrong layer.

Once that requirement was understood clearly, the actual infrastructure need became concrete and solvable: real auto-logon configuration, launching the process specifically as an interactive session rather than a generic service, and importantly verifying which kind of session a running process is actually in as a first-class health check, not just checking whether the process exists at all. "The process is running" and "the process is running somewhere it can actually work" turned out to be two different, both necessary, checks.

What ties this back to everything else in this series

Looking back across the debugging stories, the audit findings, and the architecture arguments this series has covered, I think they're all really one lesson wearing different clothes: the constraint that looks like an inconvenient afterthought is often the actual load-bearing wall. A packaging policy decision that seems like a small CI annoyance. A "designed to fail" test that quietly carries an unrelated assumption about safety. A vault that's been silently unreachable long enough that nobody remembers assuming it worked. A scaling pattern that's right for most systems and simply wrong for this one. And here, at the root of all of it: an assumption that "a real Windows machine" and "a real interactive desktop session" were the same thing, when they were never quite the same thing at all.

None of these were exotic problems. They were all findable early, cheaply, by asking one honest, specific question before building rather than discovering the honest answer later, expensively, after something was already built on top of the wrong assumption.

Autoscaling Doesn't Fit Every Workload Here's How to Tell, and What to Build Instead

Haripriya Veluchamy — Mon, 03 Aug 2026 15:06:37 +0000

"Just put it behind an autoscaling group" is one of those pieces of advice that's right so often it stops getting questioned. For a stateless API or a web server, it's genuinely close to free scaling add a replica, traffic balances across it, remove a replica, nobody notices. It's such a reliable default that it's easy to reach for it everywhere.

It doesn't fit every workload, though, and the difference matters enough that I think it's worth naming precisely not as "autoscaling is bad," but as: here's the specific property a workload needs to have before reactive autoscaling makes sense, and here's what to do instead when it doesn't.

The property that actually matters: interchangeability

A stateless replica is interchangeable. Any instance can serve any request, right now, with zero setup cost specific to that request. That's the entire reason autoscaling groups work as well as they do the thing you're adding is immediately, fully useful the moment it's up, and removing one costs nothing because whatever it was doing, another replica can pick up instantly.

A lot of backends quietly violate this assumption without anyone noticing, because the violation only shows up under real scale pressure, not in normal development. Two common ways this happens:

Session affinity. Some external systems only allow a single active session per account, or per some other resource key. Once a session is established on a specific instance, that instance not just "some instance in the pool" is now the only place that session lives. A second replica can't share it, take it over seamlessly, or load-balance it away.

Slow, non-trivial startup. A stateless container can typically start serving traffic in well under a second. Some backends anything wrapping a heavier external process, a licensed desktop application being automated, a protocol requiring a multi-step handshake before it's usable take real, measurable time (seconds to minutes) before a fresh instance is actually useful. If your scaling trigger is "traffic just spiked, react now," and your instances take two minutes to become ready, you've built a system that's structurally too slow for the exact problem it's meant to solve.

If either of these is true for your backend, a generic reactive autoscaling group isn't a slightly-worse fit it's actively the wrong shape of solution, and no amount of tuning the trigger thresholds fixes that.

Scaling out: the fix is timing, not tooling

The good news: scaling out for this class of system is solvable, just not with second-scale reactivity. Instead of "traffic spiked, add a replica now," the right pattern is a slower, trend-based trigger: watch a capacity metric (percentage of current instances' sessions in use, say), and cross a threshold 70-75% is a common choice before you're actually out of room, giving the new instance its full startup time to become ready ahead of when it's actually needed.

This is the same idea as reactive autoscaling, just running on a slower clock that respects your workload's real startup cost, instead of pretending it doesn't have one.

Scaling in: the part almost everyone skips

Scaling out gets most of the attention because it's the "we're growing" story. Scaling in safely is the harder half, and it's the part a lot of systems never actually build, quietly assuming it'll be fine.

It won't be fine, by default, for exactly the same reason scaling out needed rethinking: an instance in this class of system might have live, in-progress sessions on it. A naive scale-in policy "utilization dropped, terminate an instance" has no way to know that, and will happily kill active work.

This isn't a novel problem, and it's worth knowing the industry already has a real, named answer for it: AWS's Auto Scaling supports lifecycle hooks specifically for this a termination hook can hold an instance in a pending state, giving your own code time to drain in-progress work or migrate it elsewhere, rather than the platform just pulling the plug. There's an even more direct newer option built for exactly this situation: instance lifecycle policies that keep an instance retained rather than force-terminating it if a graceful shutdown doesn't complete cleanly. The pattern to copy, regardless of which cloud you're on: stop routing new work to an instance first, wait for its existing sessions to end naturally, only then actually remove it. That's not exotic engineering it's the same "drain, then remove" idea load balancers have used for connection draining for years, just applied one layer up, at the compute level instead of the request level.

This isn't a niche problem real, large-scale systems already do this

It's worth checking whether an actual production system at scale validates this shape, rather than trusting my own reasoning alone. Video conferencing platforms are a good real-world example a live meeting session is about as textbook session-affine as workloads get. Zoom's own published architecture describes exactly the pattern this post is arguing for: participants get routed to the least-loaded available server for their region, via a dedicated control-plane component that tracks real server load not round-robin, not a generic autoscaling group blindly adding and removing capacity. That's a capacity-aware assignment layer sitting in front of a pool of session-bound servers, which is precisely the shape you land on once you take "sessions can't just move between replicas" seriously.

The actual takeaway

Before reaching for an autoscaling group, ask one honest question: can any instance in this pool serve any unit of work right now, with no setup cost specific to that instance? If yes, standard autoscaling is a great, close-to-free default use it. If no because of session affinity, slow startup, or both you need two separate, deliberately different mechanisms: a slower, trend-based trigger for scaling out, and real drain logic for scaling in. Trying to force a session-affine, slow-starting workload into a fast, reactive, interchangeable-replica model doesn't just work worse it doesn't really work at all, and the failure mode is subtle enough that it's easy to only discover it once you're already depending on it.

Vault cant found What a Live Audit Found That Code Review Never Would

Haripriya Veluchamy — Wed, 29 Jul 2026 08:28:13 +0000

Some bugs hide in logic. This one hid in the gap between "the code looks correct" and "the code is actually working" and the only way I found it was by refusing to trust either claim without checking the live system directly.

The setup

Mid-migration between two credential-storage systems: an older secrets vault (call it the legacy vault) used by an original provider integration, and a newer, purpose-built vault for a self-hosted replacement being rolled out gradually. During the transition, both exist side by side. The code that decides which vault to check for a given user's credentials looked, on review, completely correct it classified each connection record as "legacy" or "new" based on a stored field, and looked in the matching vault.

Nothing about the code review raised a flag. Which is exactly the problem with code review as your only line of defense: it tells you the logic is internally consistent, not that the systems it's talking to still exist.

Building the audit, properly this time

Instead of trusting the classification logic in the abstract, I wrote a small audit script with one rule: don't reconstruct what a secret's name should be and assume it's there actually query the live vault and see. This sounds obvious. It's also very easy to skip, because "reconstruct the expected name, spot-check a couple" feels like it should be equivalent to "check them all live," and it really isn't.

The script pulled every connection record from the database, classified each one the same way the app's own code did, and then made a real, live call to whichever vault it should be in.

What it found

Every single legacy-vault lookup failed. Not "secret not found" a connection failure. The vault itself wasn't resolving anymore.

That's a materially different problem than a missing secret. A missing secret means "someone forgot to write this." A vault that doesn't resolve at all means the entire piece of infrastructure the code has been confidently pointing at no longer exists and every single credential-write attempt to it, for every legacy user, had been silently failing, this whole time, for however long it had been broken.

How did nobody notice?

The honest, slightly embarrassing answer: the code that wrote to this vault caught the failure, logged a warning, and moved on. Not a crash. Not an alert. A line in a log nobody was watching, for a code path nobody expected to fail, because "the vault exists" was baked in as an assumption so early in the system's life that it had stopped being something anyone thought to re-check.

This is the actual shape of the bug, more than the specific vault: a try/catch that swallows a failure gracefully is indistinguishable, from the outside, from success until someone goes looking.

The part that actually surprised me

Here's where it got more interesting than "we found a broken thing, went and fixed it." Digging into why the legacy vault mattered at all, it turned out the underlying legacy provider managed those users' actual live sessions entirely on its own side the app's cached copy of their credentials was never actually read from again in practice. The vault had been dead, silently, for a long stretch, and functionally, nothing legacy-side had actually broken because of it.

That's not a "phew, no harm done, nothing to see here" ending though. It's a different, more uncomfortable one: we got lucky that this specific dead dependency happened to be redundant. The audit didn't know that in advance, and neither did I. The only way to find out whether a silently-dead piece of infrastructure actually matters is to go looking you can't reason your way to "probably fine" from the code alone.

The same audit, run against the newer vault the one actually still in active, load-bearing use found a real, smaller, actionable problem: a couple of expected entries genuinely missing. Real incomplete setup, not infrastructure rot. Fixed directly.

What I took away from this

Code review confirms logic. It cannot confirm that the systems the logic depends on still exist. Those are different guarantees, and it's easy to unconsciously treat the first as if it implies the second.
A caught-and-logged failure is a blind spot by construction, not just bad luck the entire point of catching it gracefully is that nothing breaks loudly. That's good for uptime and bad for ever noticing, unless something is actually watching those logs.
"Reconstruct the expected value and spot-check it" is not the same audit as "query the live system for every case." The first checks your mental model. The second checks reality. They can disagree, and when they do, reality wins.
Every so often, a system this old deserves a live audit, not just a code review specifically to surface the assumptions that were true once, got baked in early, and nobody has had a reason to re-question since.

Why Your CI Agent Can't Install pip

Haripriya Veluchamy — Sat, 25 Jul 2026 16:06:39 +0000

If you've ever set up a self-hosted CI/CD agent on a fresh Ubuntu image and hit pip: command not found or No module named pip, your first instinct is probably to check the network. Mine was too. That instinct is wrong, and the real answer took me four attempts to actually land on.

Here's the full story, including the three dead ends, because I think the dead ends are the useful part.

The setup

A self-hosted CI agent (in my case, an Azure DevOps agent, but this applies just as much to a self-hosted GitHub Actions runner or a plain Jenkins box) running on a minimal Ubuntu image. No sudo access on the box that's deliberate, since the agent shouldn't need root just to run a pipeline. I needed to install a CLI tool that only ships via pip.

Reasonable first attempt:

- script: |
    python3 -m pip install --user --quiet some-cli-tool
  displayName: 'Install tool'

Result:

/usr/bin/python3: No module named pip

Fine, I thought pip module is missing, ensurepip should bootstrap it. Every tutorial says so.

Attempt 2: `ensurepip`

- script: |
    python3 -m ensurepip --user --upgrade
    python3 -m pip install --user --quiet some-cli-tool

Result:

/usr/bin/python3: No module named ensurepip

Not just pip missing ensurepip itself, the thing that's supposed to install pip, doesn't exist either. That's the first real clue something structural is going on, not just a missing package.

The actual reason (worth understanding, not just working around)

Debian and Ubuntu deliberately strip pip and ensurepip out of the base python3 package. This isn't a bug or an oversight it's a packaging policy decision. Both live in separate packages (python3-pip, python3-venv) that you're expected to install via apt. On a full desktop or dev-configured server, you've probably had these installed for so long you forgot they're not actually part of core Python.

On a minimal, no-sudo CI image, you don't have apt access at all so you can't just apt install python3-pip your way out of it.

Attempt 3: `venv` (also fails, same root cause)

My next thought: skip pip/ensurepip entirely, use a virtual environment instead, since venv normally bootstraps its own pip on creation.

- script: |
    python3 -m venv /tmp/tool-venv
    /tmp/tool-venv/bin/pip install --quiet some-cli-tool

Result: fails identically. venv's own pip-bootstrapping step depends on you guessed it the same missing ensurepip module. Same wall, different door.

Attempt 4: the actual fix `get-pip.py`

The real answer is a standalone bootstrapping script, maintained by the Python Packaging Authority specifically for situations like this environments where ensurepip isn't available:

- script: |
    curl -sS https://bootstrap.pypa.io/get-pip.py -o /tmp/get-pip.py
    python3 /tmp/get-pip.py --user --quiet
    python3 -m pip install --user --quiet some-cli-tool
    echo "##vso[task.prependpath]$HOME/.local/bin"
  displayName: 'Install tool (get-pip, no sudo)'

get-pip.py doesn't depend on ensurepip at all it's a self-contained bootstrapper that installs pip directly. This worked immediately, no sudo, no apt, no venv complexity.

The one thing to double-check before you assume this is your problem

Before chasing this fix, rule out the boring explanation first: confirm your agent actually has network access to PyPI at all. A quick diagnostic step saved me from solving the wrong problem:

- script: |
    curl -sS -o /dev/null -w "pypi.org: %{http_code}\n" https://pypi.org --max-time 5

If that comes back with anything other than a fast 200, you're dealing with a firewall/egress issue, not this one and no amount of get-pip.py will fix that.

The takeaway

If python3 -m pip and python3 -m ensurepip are both missing on a Debian/Ubuntu box, that's not a broken image it's the distro's actual, intentional packaging policy. apt install python3-pip is the "normal" fix, and it's simply not available to you on a locked-down, no-sudo CI agent. get-pip.py is the one workaround that doesn't need apt, doesn't need ensurepip, and doesn't need root.

Small thing, but it cost me three wrong turns before I found it hopefully this saves you those three.

DEV Community: Haripriya Veluchamy

Two Autoscaling Policies, Zero Coordination: Why More Signals Made Scaling Worse

The setup

What went wrong

The actual fix: one authority, not two

The generalizable part

I Spent a Month on Architecture The Migration Took Two Days

The wrong turn I almost shipped

The number I didn't expect

A trade-off I made on purpose

Choosing the entry point on purpose

Scaling the actual reason I moved

Designing access properly this time

Automating the pipeline, deliberately

Alerts for everything that can fail

Retiring the old system carefully

The takeaway

I Built a Production Grade ML Candidate Ranking System

The Challenge

Why Keyword Matching Fails

The Architecture: Don't Depend on Any Single Signal

The Formula

Stage 1: Precompute Everything (once, ~1.7 hours)

Signal 1: BM25 (weight: 0.20)

Signal 2: FAISS Cosine Similarity (weight: 0.30)

Signal 3: Skill Quality (weight: 0.20)

Signal 4: Trajectory (weight: 0.30)

The Gates: Hard Filtering Before Ranking

Years of Experience Soft Multipliers

The Reasoning Layer

Problems I Actually Hit

1. ml_product_years was computed but never wired in

2. Prestige was hardcoded

3. Pre-LLM 24-month hard cutoff

4. HuggingFace rejecting binary files via git

5. TODAY hardcoded

6. Reasoning cut mid-sentence

Results

What I'd Do Next

Three Months Later

Code

Why I Stopped Clicking "Create Alert" in the Cloud Console

What I used to do

Where it broke

What I changed

Why this is better

The simple lesson

How i did the Multi VM monitoring

I Had 5 Servers Sending Metrics But Couldn't Tell Which One Was Which

What happened

Why it happened

How I fixed it

The real lesson

I did Golden Images

Golden Images How I Stopped Manually Logging Into Every New Server

The problem

The fix: capture the setup once, replay it everywhere

The mistake worth mentioning

Why this is worth doing before you scale

The general lesson

One Lookup Table Turned a Risky Server Migration Into a One-Line Change

The setup

The moment it mattered

The pattern, generalized

Where this shows up beyond servers

The lesson

Multi region deployment with Terraform modules

The problem

The pattern: separate "what's shared" from "what's different"

Why this matters beyond "less typing"

What I kept separate on purpose

The general lesson

The App Needed a Real Desktop, Not Just a Real Windows Machine

The plan that looked right on paper

What "did not work" actually looked like

The pivot, and the deeper thing it revealed

Why this isn't just a trading-software quirk

What ties this back to everything else in this series

Autoscaling Doesn't Fit Every Workload Here's How to Tell, and What to Build Instead

The property that actually matters: interchangeability

1. `ml_product_years` was computed but never wired in

5. `TODAY` hardcoded

Attempt 2: `ensurepip`

Attempt 3: `venv` (also fails, same root cause)

Attempt 4: the actual fix `get-pip.py`