DEV Community: Dinesh Wijethunga

The Login That Never Worked: Debugging 5 Layers Deep

Dinesh Wijethunga — Mon, 24 Aug 2026 18:54:04 +0000

Part 5, the finale of a 5-part series on using Claude AI to run, secure, and ship a real production server. The pipeline shipped (Part 4). Then I tried to log in.

The Login That Never Worked — Debugging Five Layers Deep with AI (Part 5)

The pipeline was green. Both releases were live. I opened my SaaS to see it working — and got a 502. Then, after fixing that, a "CORS error." Then a 500. Then an OAuth failure. Then a 403. Each fix revealed the next problem underneath, like peeling an onion that makes you cry five times.

This is the most instructive post in the series, because it shows what AI debugging actually looks like on a real system: not one magic answer, but a disciplined peeling of layers, where each error's real cause is hidden behind a misleading symptom. Here's all five, in order.

Layer 1: The 502 — A Stopped Process

The frontend returned 502 Bad Gateway. nginx proxies the site to a Node process on port 3003, and Claude's first check — pm2 list — showed the visa-saas process was stopped. The deploy's process-reload had left it in a bad state.

The fix wasn't a blind restart (that can hide a crash loop). Claude had me delete the corrupted process entry and start it fresh:

pm2 delete visa-saas
pm2 start ecosystem.config.js
pm2 save   # so it survives reboots

Process online, port 3003 answering. Lesson learned: the deploy's "reload" logic couldn't recover a stopped process — a real pipeline bug we noted for a follow-up fix. A green deploy that leaves the site down is worse than one that fails honestly.

Layer 2: The "CORS Error" That Wasn't CORS

Now the site loaded but login threw what the browser called a CORS error. Here's the trap Claude flagged that saves hours: when a Laravel API throws a 500, the error response often has no CORS headers — so the browser reports "CORS policy" when the real problem is the API crashing. The CORS message was a symptom, not the disease.

And the disease connected back to our Part 3 work. Remember the storage-permissions fix? On this new release, the storage folder was owned by the deploy user, but PHP-FPM runs as the web user — so PHP couldn't write logs or sessions, threw 500s, and the browser painted them as CORS. Claude spotted it because it read the actual error, not the browser's guess. One chown and the 500s vanished.

Layer 3: The OAuth Failure — A Wrong Client ID

"Client authentication failed"
League\OAuth2\Server\Exception\OAuthServerException

Laravel Passport couldn't issue a token. Claude traced it into the database. There were two OAuth clients:

ID	Client	Type
1	Password Grant Client	personal = 0
2	Personal Access Client	personal = 1

But the app's config pointed PASSPORT_PERSONAL_ACCESS_CLIENT_ID=1 — the wrong type. It was trying to issue a personal access token using a password-grant client. Changing the ID to 2 and rebuilding the config cache fixed it. Almost certainly a bug that had existed since setup — because with login broken, nobody had ever successfully authenticated on this deployment. Our work didn't break it; it was the first time anyone got far enough to hit it.

Layer 4: The Permission-Write 500s

Logged in! Then the dashboard threw 500s on loading modules. Same root cause as Layer 2, different file: the app writes a modules_statuses.json at runtime, owned by the wrong user, so the web process couldn't write it. Another targeted chown. This was the moment the deeper pattern became clear: the deploy creates files as one user, the app runs as another — the real fix is a permission-reconciliation step in the pipeline itself, so every future deploy doesn't reintroduce it.

Layer 5: The 403 — A One-Word Typo

The final boss. Every data endpoint returned 403 Forbidden — "You do not have the required permissions." But I was logged in as a super-admin. Claude checked the database: my user had the super-admin role, the role had all 60 permissions, everything linked correctly. The data was perfect. So why 403?

This is where I pointed Claude Code at the actual codebase, and it traced the exact chain. The route guarding /users required a permission named manage-users. But the seeder that creates permissions created users.view — the permission name the route demanded had never been created. A single naming mismatch between the route and the seeder. Every user in the system was locked out of that endpoint, and had been forever.

Route wants:    permission:manage-users   ← doesn't exist
Seeder creates: users.view                ← the real name

The fix: change the route guard to the permission that actually exists. And because AI is good at exactly this kind of grind, I had it grep the entire route file for other mismatches — so we could fix them all in one pass instead of discovering them one 403 at a time.

Why This Story Matters More Than a Clean Success

Five layers. A blank page that was a stopped process. A CORS error that was a permissions problem. An OAuth failure that was a config typo. A 403 that was a naming mismatch. Not one of these symptoms pointed at its own cause. That's what real debugging is — and it's exactly where an AI partner earns its place: reading the actual error instead of the misleading one, tracing a symptom to its true source, and grepping a codebase for every instance of a bug in seconds.

The AI didn't replace my judgment — I decided what to fix by hand, what to route through the pipeline, and when to stop and think. But it turned a bewildering cascade into a solvable sequence. And critically: bugs 3, 4, and 5 were pre-existing — they'd been in that codebase since setup, invisible because nobody could log in to trigger them. We didn't cause them. We were the first to reach them, and the AI helped root-cause each one instead of flailing.

The Whole Journey, Start to Finish

Over one day, with Claude AI as an investigate-and-verify partner, this server went from: an internet-exposed financial API, world-readable secrets across 20 projects, password-based SSH open to the world, no deployment pipeline, and a frontend that had never worked — to: a locked-down box, key-only SSH, a zero-downtime CI/CD pipeline, and a working login on a live dashboard.

Key Takeaways for New Developers

The symptom is almost never the cause. A 502, a CORS error, and a 403 all lied about what was really wrong.
Read the actual error, not the browser's interpretation of it.
AI is exceptional at tracing a symptom to its source and grepping a whole codebase for every instance of a bug.
Keep the human in charge of what to fix and where it belongs (hand-fix vs. pipeline); let the AI do the tracing and the grind.
A working system revealing old bugs isn't a regression — it's progress. You can only find the login bug once login gets far enough to fail.

That's the series. One developer, one AI partner, one very real production server — audited, secured, shipped, and debugged, the honest way, with a paper trail at every step. If you're nervous about letting AI near your infrastructure, I hope this showed you the pattern that makes it not just safe but genuinely powerful: the AI gets the brains, you keep the keys.

Thanks for reading all five parts. If this helped, the whole series is built to be followed step by step on your own server. What would you point an AI agent at first — a security audit, or that one bug you've been avoiding?

Originally published at dineshstack.com — read the full version with code samples and updates there.

How to Build a Zero-Downtime CI/CD Pipeline with an AI Pair

Dinesh Wijethunga — Sat, 22 Aug 2026 07:52:06 +0000

Part 4 of a 5-part series on using Claude AI to run, secure, and ship a real production server. The server is now secure (Parts 1–3). Time to ship code the right way.

Build a Zero-Downtime CI/CD Pipeline with an AI Pair (Part 4)

Up to now I'd been deploying by hand — SSH in, pull, hope. For my main SaaS project I wanted the real thing: push to GitHub, tests run automatically, and if they pass, the server swaps to a fresh release with a symlink so rollback is instant. In this post, Claude plays two roles: code-review partner for my pipeline, and server-prep engineer to get the box ready — using the same investigate-then-execute pattern from the earlier parts.

The Architecture (Capistrano-Style Releases)

The idea is simple and powerful. Instead of overwriting your live app in place, each deploy lands in a timestamped folder, and a symlink points to the current one:

/var/www/visa-saas/
├── releases/
│   ├── 20260709224220/   ← previous release
│   └── 20260712172016/   ← new release
├── shared/               ← .env, storage (survive across deploys)
├── api  → releases/20260712172016   ← symlink, flipped atomically
└── web  → releases-web/...           ← same idea for the frontend

Deploy = build a new release folder, then flip the symlink. Rollback = flip the symlink back. That's the whole magic: rollback becomes one command instead of a panic.

Claude as Code Reviewer: What It Caught in My Pipeline

I had draft GitHub Actions workflows and asked Claude to review them like a senior engineer. It found real problems I'd have shipped:

Issue Claude flagged	Why it mattered
Web deploy did `rm -rf` then move — no rollback	The old release was destroyed at activation. One bad build = no way back.
Prune step would eventually delete the original backup	"Keep 5 newest" quietly wipes your `initial` safety copy after 5 deploys.
Bare `php` on a server with 6 PHP versions	Migrations could run under the wrong PHP than the site serves.
Migrations with no database backup first	A symlink rollback can't undo a schema change — you need a dump.
Third-party Actions pinned to tags, not commit SHAs	A moved tag could inject malicious code into a job that holds your SSH key.

It also caught the classic PHP-plus-symlink gotcha: without the right nginx setting ($realpath_root), OPcache can keep serving old code after you flip the symlink. That's the kind of subtle, experience-earned detail that makes an AI reviewer genuinely valuable — not because it's magic, but because it's read every variation of this mistake.

The Security Angle Nobody Writes Down

One point Claude raised that I hadn't considered: once GitHub Actions can SSH into production, anyone who can push a workflow change can run code on your server. So the deploy key got its own dedicated keypair, restricted in authorized_keys (no-agent-forwarding,no-port-forwarding), and the deploy job was gated behind a protected GitHub environment. Your CI is only a safety gate if nothing can route around it.

Server Prep: Investigate, Then One Script

Before merging, the server needed preparing — release directories, a database-backup folder, the nginx storage path, a narrow passwordless-sudo rule for reloading PHP. True to the pattern, Claude did a read-only readiness report first, and it changed the plan: three things I'd assumed needed fixing were already correct. Had we skipped straight to scripting, we'd have installed a duplicate sudo rule and "fixed" a setting that was already right. The verify-first phase paid for itself again.

Then Claude generated one prep script with its now-familiar signature: timestamped backups, diff checkpoints, per-step verification, and printed (never auto-run) rollback commands. It even excluded Passport's private keys from a bulk permission change so they'd stay locked at 600 — a detail I'd have missed.

Merge, and the First Real Deploy

All checks green, PR merged. Both API and web pipelines fired. Watching the symlinks flip in real time was the payoff:

watch -n 2 'readlink /var/www/visa-saas/api; readlink /var/www/visa-saas/web'
# api  → releases/20260712172016
# web  → releases-web/20260712174334   ← it flipped!

The First Failure (And Why It Didn't Matter)

The web deploy failed on its first attempt — an SSH i/o timeout uploading the release. My first instinct was "the hardening broke CI." But we checked the logs instead of guessing: the deploy key authenticated fine; it was just a transient network blip on one job. A re-run sailed through.

The crucial point: because of the release architecture, the site never went down during that failure. The old release stayed live until the new one was ready to flip. A failed deploy on this design is a non-event, not an outage. That safety net is the entire reason to build it this way.

Key Takeaways

Timestamped releases + a symlink flip turn rollback from a crisis into one command.
An AI code reviewer shines at catching the subtle, experience-earned mistakes — missing backups, wrong PHP, OPcache traps, supply-chain risks.
When CI can reach production, treat the deploy key like the sensitive credential it is.
Investigate-then-script saved us from "fixing" things that were already fine — twice.
A good pipeline fails safely: the live site stays up until the new release is proven.

The pipeline worked. Both releases were live. And then I opened the site to admire it — and the login was broken. Not a little broken. A cascading, five-layers-deep broken that started as a blank page and ended at a one-word typo buried in a route file. That debugging journey — the most instructive part of the whole day — is Part 5.

👉 Coming up in Part 5: "The Login That Never Worked — Debugging Five Layers Deep with AI." What's the worst production bug you've shipped through a green pipeline?

Originally published at dineshstack.com — read the full version with code samples and updates there.

Laravel queue jobs not processing: wrong connection

Dinesh Wijethunga — Fri, 21 Aug 2026 11:44:04 +0000

If your jobs table is growing while the queue worker sits there reporting healthy, the two processes are reading and writing different queues. Laravel lets you set the connection in two places, and the one in the worker's command line silently wins over the one in your environment file.

Ours disagreed for nineteen days. It cost 55,470 orphaned jobs, every transactional email in that window, and — the part I find hardest to defend — nobody noticed, because from every angle the system looked fine.

What we saw

We were auditing a production database before an unrelated migration when a count came back wrong:

SELECT COUNT(*) FROM jobs;
-- 56588

Fifty-six thousand pending jobs. The oldest was nineteen days old. The newest was thirty seconds old and there was a fresh one every thirty seconds, forever.

The queue worker container had been up for weeks. Its logs showed a clean supervisord boot and nothing else — no errors, no warnings, no processed jobs. Every health check passed. The application served traffic normally. Customers were completing orders.

The arithmetic told us where to look before the configuration did. One job class accounted for 55,470 of the rows, and the scheduler dispatched it every thirty seconds:

2 per minute × 60 × 24 = 2,880 per day
2,880 × 19 days ≈ 54,720

That is not a job failing and retrying. That is a job being enqueued perfectly and never once being read, since the day the scheduler entry went live.

The two places a queue connection is set

The environment file said one thing:

QUEUE_CONNECTION=database

The worker's container definition said another, hardcoded months earlier and never revisited:

SUPERVISOR_PHP_COMMAND: "php /var/www/html/artisan queue:work redis
  --queue=high,default,low --sleep=3 --tries=3 --max-time=3600"

That first argument to queue:work is the connection name, and it overrides QUEUE_CONNECTION completely. The behaviour is documented and genuinely useful — it is how you run separate workers against separate backends. It is dangerous only because it is set in a different file, in a different repository concern, from the value it overrides.

So the application dispatched jobs into MySQL. The worker polled Redis, found nothing, slept three seconds, and polled again. It did that several million times without complaint.

Why nothing logged an error

This is the part worth internalising, because it generalises far beyond queues.

Nothing failed. Dispatch wrote a row and returned success — that is what dispatch does. The worker blocked on an empty list and returned success — that is what polling an empty queue does. An error requires some component to attempt something impossible, and neither component ever attempted anything impossible. Each half was working correctly. The system was broken only in the relationship between them, and nothing in the stack is responsible for that relationship.

Every monitor we had was pointed at a component. Container health: passing. Error rate: zero. Database: fine. Not one of them was pointed at the contract.

Why the damage was smaller than it should have been

The dominant job class was an outbox recovery job — a safety net that re-publishes events whose delivery was not confirmed after the transaction committed. So we checked what it would have had to recover:

SELECT status, COUNT(*) FROM outbox_messages GROUP BY status;
-- sent  1668

Every row sent. Zero pending. The primary publish path had a perfect record for the entire window, which is why nineteen days of a dead safety net produced no visible symptom.

We got away with it. Read that sentence as the accusation it is: we did not detect the failure, we were rescued by the fact that the thing it protected never needed protecting. Had the primary path faltered once during those nineteen days, the recovery mechanism would have been sitting in a MySQL table watching it happen.

The rest of the backlog was less lucky. Several hundred registration notifications and booking-status emails were in there. Those never sent, and nobody filed a ticket — which tells you something uncomfortable about how much of that mail anyone was reading.

Clearing it: why we did not just release the backlog

The instinct on finding 56,000 stuck jobs is to point a worker at them and let them drain. We deliberately did not.

Those jobs were nineteen days stale. Releasing them would have delivered hundreds of "welcome, you've registered" emails to people who registered three weeks ago, and status updates for rides that finished long before. Delivering a stale message is a worse outcome than never delivering it, and unlike the silent failure, customers would definitely have noticed that one.

So: group by class, decide per class, back up, then discard.

mysqldump --single-transaction app_prod jobs failed_jobs | gzip > jobs-backup.sql.gz
TRUNCATE TABLE jobs;

The backup makes the decision reversible for the cost of a few megabytes. Take it even when you are confident, because the confidence is about the job classes you identified, not the ones you skimmed past.

Then the fix, which was one line — pointing the application at the connection the worker had been watching all along. Jobs began clearing in two to four milliseconds each.

The same bug is not equally dangerous everywhere

Here is the detail that changes how you should weight this. We fixed the mismatch as part of moving the queue onto Redis, and that move altered the failure's blast radius entirely.

In MySQL, 56,000 orphaned jobs were a large table on a disk with hundreds of gigabytes free. Genuinely harmless — which is precisely why it survived nineteen days.

On Redis, the identical bug consumes memory, and queue entries carry no TTL. They sit there until a worker takes them. On a shared box without swap, unbounded memory growth does not politely degrade; it reaches a limit and something gets killed, and the process the kernel selects is chosen by size rather than by blame — frequently your database rather than the cache that caused it.

Same misconfiguration, same silence, radically different consequence. Moving a queue to a faster substrate also moves it to a less forgiving one. If you are making that migration, fix your queue-depth monitoring first, not afterwards.

You cannot alert on a metric that does not exist

The obvious follow-up is an alert on queue depth. We went to add one and found the gap was one level deeper than expected.

The Redis exporter reports totals — memory, client count, keys per database — but it does not publish the length of any individual list unless you name it explicitly:

REDIS_EXPORTER_CHECK_KEYS: "queues:*"

Until that line existed there was no queue-depth metric in Prometheus at all. A dashboard would have shown a healthy Redis for all nineteen days, because every metric it displayed was genuinely healthy. An absent metric and a good metric look identical on a graph.

The alert we wrote was wrong, and testing caught it

A healthy queue drains in milliseconds, so any depth that survives a long window means nothing is consuming it. That was the rule:

min_over_time(redis_key_size{key=~"queues:.*"}[30m]) > 10

We tested it by planting a synthetic backlog. It went pending, correctly. Then we deleted the key — and it stayed pending.

min_over_time keeps returning samples from its whole window after a key disappears. So any burst that got scraped once would fire this five minutes later and hold it for half an hour: exactly the false-positive noise that teaches a team to ignore alerts. Pairing it with a check for a currently-present sample fixes it, because an emptied Laravel queue deletes its Redis list and the series simply stops:

redis_key_size{key=~"queues:.*"} > 10
  and
min_over_time(redis_key_size{key=~"queues:.*"}[30m]) > 10

Re-tested: pending with a backlog, inactive the moment it drains. An alert is a piece of production code, and an untested one is likelier to erode trust than to protect anything.

What to check on your own system

Two commands, worth running now rather than during an incident. What the application believes:

php artisan tinker --execute="echo config('queue.default');"

And what the worker is actually executing — read the process, not the config file that you believe produced it:

docker compose exec app_queue ps aux | grep queue:work

If those two disagree, you have this bug, and your logs will not tell you. Then confirm something is genuinely draining, rather than that nothing has arrived: a depth of zero and a broken consumer look the same from outside.

The principle

Health checks verify components. This failure lived in the space between two healthy components, where nothing was looking, and the only honest signal available was a number that nobody was collecting.

For any handoff between two processes, monitor the queue between them rather than the processes themselves. Depth over time is the cheapest true statement you can make about a distributed system: it goes up when the producer outruns the consumer, and it stays up when the consumer is gone. Neither of those facts is visible from either end alone.

Originally published at dineshstack.com — read the full version with code samples and updates there.

Locking Down Secrets and SSH with AI (and the Cloud-Init Trap)

Dinesh Wijethunga — Thu, 20 Aug 2026 11:18:04 +0000

Part 3 of a 5-part series on using Claude AI to run, secure, and ship a real production server. Part 2 closed an exposed API. Now we tackle the biggest structural risk the audit found — and lock the front door.

Locking Down Secrets and SSH with AI — and the Trap That Almost Fooled Us (Part 3)

The audit's number-one risk wasn't dramatic, but it was the scariest: across nearly 20 projects, the .env files — the ones holding database passwords and API keys — were world-readable. Any process, any local user, any path-traversal bug in the weakest app could read every other client's secrets. This post is how Claude and I fixed all of them at once, then shut off password-based SSH entirely — and hit a trap that silently tried to undo the whole thing.

Why World-Readable .env Files Are a Slow-Motion Disaster

A .env at mode 644 means "owner can write, everyone can read." On a server with one project, that's sloppy. On a server with 20 unrelated client projects sharing a web user, it means a single vulnerability anywhere gives an attacker every tenant's credentials. The fix is simple — 640, owned by the right user — but doing it across 20 live sites without breaking any of them takes care.

Claude's Approach: One Script, Verify Every Step

I asked Claude to write a single script following the same safety pattern from Part 2: back up, change, verify, and — crucially — check every website still works after each change. The clever part it added on its own: a baseline pass that curls every site before touching anything, so a site that was already broken wouldn't be misreported as something the script broke.

# For each project: capture current perms, fix, verify the live site
for project in "${PROJECTS[@]}"; do
  # backup-aware: record old mode/owner first
  sudo chown www-data:www-data "$project/.env"
  sudo chmod 640 "$project/.env"
  # then curl the site and compare against the pre-change baseline
done

Claude also caught two things a blanket script would have broken:

Next.js apps under pm2 run as a different user than PHP apps — chowning their .env to the web user would lock out the process that reads it. Claude special-cased those.
One .env shared by two apps (a Laravel API and a Next.js frontend) needed split ownership so both could still read it while dropping public access.

That's the difference between "AI runs a chmod loop" and "AI understands the runtime." It ran across 18 projects plus a world-writable storage directory and a trading-bot secret — every one verified against baseline, zero regressions.

The Bonus: Baseline Checks Found Pre-Existing Problems

The baseline pass earned its keep immediately. Before changing a thing, it revealed three sites that were already broken — two APIs returning 500 errors and one domain that wouldn't connect at all. None caused by the script; all surfaced by it. That's a lovely side effect of doing things carefully: you discover problems you didn't know you had.

Then: Locking SSH to Keys Only

Next, the front door. The server still accepted password logins from the entire internet — meaning it was brute-forceable. The plan: install a personal SSH key, prove it works, then disable passwords. Order matters, because a mistake here locks you out of your own server.

# 1. On my Mac: create a key if I didn't have one
ssh-keygen -t ed25519

# 2. Install it on the server
ssh-copy-id deploy_user@SERVER_IP

# 3. PROVE key auth works BEFORE disabling passwords
ssh -o PasswordAuthentication=no deploy_user@SERVER_IP

Only after that login succeeded did we disable passwords in sshd_config. And this is where the trap sprang.

⚠ The Cloud-Init Trap That Almost Fooled Us

I set PasswordAuthentication no in the main SSH config. Clean. Done, right? Claude insisted on one more check — grepping all the config, including the /etc/ssh/sshd_config.d/ drop-in directory. And there it was:

/etc/ssh/sshd_config:65:            PasswordAuthentication no
/etc/ssh/sshd_config.d/50-cloud-init.conf:1: PasswordAuthentication yes   ← !!!

A cloud-init drop-in file was setting PasswordAuthentication yes — and here's the killer detail: SSH reads the "Include" directive near the top of the config, so the drop-in file's setting wins over the main file. Without catching this, my "hardening" would have changed nothing. Password auth would have stayed wide open while I believed it was closed. Claude also caught that root login was still enabled and flagged that too.

We fixed both files, validated with sshd -t (test before reload, like nginx -t), reloaded, and then ran the two-sided proof from my laptop:

ssh deploy_user@SERVER_IP 'echo OK'          # key works
ssh -o PubkeyAuthentication=no deploy_user@SERVER_IP   # password refused
ssh root@SERVER_IP                            # root refused

Result: key works, password bounces, root bounces. Three for three.

The Deeper Lesson: Verify the Baseline, Not Just Your Changes

This was the second time that day a "should already be hardened" assumption turned out false when Claude actually checked. That's the real discipline this whole experience taught me: verify-don't-assume applies to the starting state, not just to the changes you make. The cloud-init file had been silently overriding intent for who knows how long. An AI that grepped everything instead of trusting the obvious file is what caught it.

Key Takeaways

World-readable .env files are the quiet #1 risk on a multi-project box. Fix to 640, but verify each site still runs after.
Always baseline before a bulk change, so you can tell your breakage apart from pre-existing breakage.
Install and prove your SSH key before disabling passwords — keep your current session open as a lifeline.
On Ubuntu, always grep sshd_config.d/ — a drop-in file can silently override your main config.

The server was now genuinely locked down: secrets protected, SSH key-only, root disabled — all verified from outside. With the house secure, it was time to build something: a proper CI/CD pipeline so I could ship code changes with zero downtime and one-command rollbacks. That's Part 4, where Claude becomes a code-review partner and we ship a real deployment pipeline — and hit our first live deploy failure.

👉 Coming up in Part 4: "Building a Zero-Downtime CI/CD Pipeline with an AI Pair." Run sudo grep -r PasswordAuthentication /etc/ssh/ right now — are you sure it says what you think it does?

Originally published at dineshstack.com — read the full version with code samples and updates there.

24 workers configured, 6 ever used, and the number that actually mattered was MySQL's max_connections

Dinesh Wijethunga — Wed, 19 Aug 2026 12:10:06 +0000

PHP-FPM worker count is not a performance dial you turn up. It is the smallest of three separate limits — memory, database connections, and CPU — and on most Laravel deployments the database decides it long before memory does. Raising pm.max_children past that point does not make the application faster; it converts a slow application into a broken one.

This is what we learned taking a ride-hailing API off PHP's development server, and the number we ended up with was smaller than our first instinct by an order of magnitude.

The failure that started it

The API was running under php artisan serve. Not by decision — it had been that way since the project was scaffolded, and nothing had ever pushed hard enough to expose it.

A load test did. At roughly 15 requests per second of mixed traffic the service stopped responding, and here is the part that mattered: a ten-minute traffic spike produced a thirty-five-minute outage. Arrivals stopped and the service stayed down. It recovered only when we restarted the container.

That asymmetry is the whole lesson. A system that degrades gracefully returns when load returns to normal. A system that queues without bound does not — it keeps working through a backlog that no longer has anyone waiting on it.

Why one process is a hard ceiling

artisan serve wraps PHP's built-in development server. It is single-process and handles one request at a time. Request two waits for request one, whatever it is doing.

PHP-FPM's model is different in the way that matters: it maintains a pool of worker processes, and each concurrent request occupies exactly one worker for its entire lifetime. Not its CPU time — its lifetime. A worker blocked for 300ms waiting on a database query is unavailable for those 300ms even though it is consuming almost no CPU.

So your concurrency ceiling is the worker count, and the obvious move is to make the worker count large. That is where people get hurt.

The three limits that decide max_children

Limit 1: memory

Every worker is a real OS process with its own memory. The arithmetic is unforgiving:

max_children ≤ (RAM available to PHP) / (average worker RSS)

Measure the average rather than guessing it. A Laravel worker serving a JSON API commonly sits between 40 MB and 120 MB depending on how much of the framework each route touches. Take the figure under real traffic, not at boot — a freshly forked worker is always smaller than one that has served a hundred requests.

The trap here is that exceeding this limit does not produce a clean error. It produces the OOM killer choosing a victim, and the victim is chosen by memory footprint, not by fault. On a shared box the process that dies is frequently your database, not the PHP pool that caused it.

Limit 2: database connections — the one people miss

This is the limit that actually bound us, and it is invisible until it isn't.

Each worker handling a request generally holds its own database connection. MySQL's default max_connections is 151. If you set pm.max_children = 200 because the box has the RAM for it, then at the exact moment your traffic justifies 200 workers, roughly fifty of them receive a connection error instead of a database handle.

max_children ≤ max_connections − headroom

Headroom is not optional and it is larger than it looks. Reserve connections for the queue worker, the scheduler, any Kafka or event consumer, migrations during a deploy, your monitoring exporter, and a human being with a database client open during an incident. That last one has ended more incidents badly than it should have.

The failure mode is worth dwelling on. Under-sizing workers gives you a slow site. Over-sizing them past the connection cap gives you a site that returns 500s specifically when it is busiest, which is both the worst time and the hardest to reproduce afterwards.

Limit 3: CPU, weighted by what your requests actually do

For CPU-bound work, more workers than cores buys nothing — it adds context switching to the same finite compute. For I/O-bound work, where workers spend most of their lifetime waiting on a database or an upstream HTTP call, worker count can exceed core count substantially, because the waiting overlaps.

Most Laravel API requests are I/O-bound, which is why a modest core count still supports a healthy pool. But the ratio is a property of your routes, not a constant. Measure it before borrowing anyone's rule of thumb, including this one.

What we set, and what we measured

The pool, on an eight-core host shared with the database, cache, message broker and several Node services:

pm = dynamic
pm.max_children = 24
pm.start_servers = 8
pm.min_spare_servers = 6
pm.max_spare_servers = 12
pm.max_requests = 1000
pm.status_path = /fpm-status

Twenty-four, on a box that could hold far more by the memory arithmetic alone. The database cap and the shared-tenancy reality set it, not RAM.

Then we measured. Under a full booking workload — WebSocket connections, dispatch, offer handling, ride completion — the pool peaked at six of twenty-four workers, with a listen queue of zero and max children reached at zero.

Six of twenty-four is not a sign the setting is wrong. It means the ceiling is currently generous, which is exactly what you want a ceiling to be. The number that would tell us to raise it is max children reached, and it has never left zero.

pm.max_requests = 1000 deserves a note: each worker retires after a thousand requests and is replaced. That bounds the damage from any slow leak in your code or an extension, at the cost of an occasional process fork. On a long-lived pool it is close to free insurance.

Three traps that cost us time

opcache looks disabled when you check it from the command line

We ran a quick command-line check and got zeros back:

php -r 'var_dump(opcache_get_status(false));'
# Warning: Trying to access array offset on false

That output is correct and means nothing. The CLI SAPI has opcache.enable_cli off by default, so a command-line probe reports on a completely different configuration from the one serving your web traffic. Check through the FPM binary instead:

php-fpm -i | grep -E 'opcache.enable|opcache.memory'
# opcache.enable => On => On
# opcache.memory_consumption => 256 => 256

Ten minutes disappeared into diagnosing a problem that did not exist. Verify through the same SAPI that serves the traffic, always.

A Docker memory cap can grant more memory than you think

Setting --memory=5g without --memory-swap does not confine a container to 5 GB. Docker grants that much RAM plus the same again in swap. A cap set above the box's available RAM therefore licenses the container to exhaust memory and then thrash, which is slower and harder to diagnose than a clean kill.

# Bounded: the container is OOM-killed alone, the host survives
docker run --memory=3g --memory-swap=3g ...

An uncached route table taxes every request equally

Worth checking before you touch the pool at all. Our route file compiles to roughly 1.5 MB. Without route:cache, that table is rebuilt on every single request — a flat cost of several hundred milliseconds on every endpoint, which no amount of worker tuning removes.

php artisan route:cache

A worker held for 300ms of avoidable work is a worker you do not have. Fixing this is often worth more than doubling the pool, and it is one command.

How to tell whether workers are your problem at all

Enable the status endpoint and read two fields:

curl -s localhost/fpm-status | grep -E 'active processes|listen queue|max children reached'

max children reached climbing — the pool is genuinely the ceiling. Raise it, within the three limits above.
listen queue above zero while max children reached stays at zero — requests are waiting, but not for workers. Look downstream: slow queries, an uncached route table, a blocking upstream call.
Both at zero under load — the web tier is not your bottleneck. Measure elsewhere before changing anything here.

That second case is the common one, and it is where worker tuning becomes cargo cult. If your workers are idle-but-occupied, you do not have a concurrency problem; you have a latency problem wearing a concurrency costume.

The principle

Every capacity fix relocates the bottleneck rather than removing it. Moving off the development server did not make the system fast — it made the next constraint visible, which turned out to be the database sitting behind those workers.

So size the pool from the limits you can measure, set it to something defensible, and then watch the counter that tells you it was wrong. A number you can justify and monitor beats a larger number you picked because the box looked like it could take it.

Originally published at dineshstack.com — read the full version with code samples and updates there.

The Exposed API Claude AI Found in Its First Hour

Dinesh Wijethunga — Tue, 18 Aug 2026 17:07:05 +0000

Part 2 of a 5-part series on using Claude AI to run, secure, and ship a real production server. In Part 1 we connected Claude safely and ran a read-only audit. It found something alarming. Now we fix it.

The Exposed API Claude Found in Its First Hour (Part 2)

The audit from Part 1 ranked its top risks, and number three stopped me cold: a Python FastAPI service — the backend for a trading bot — listening on 0.0.0.0:8100, directly on the public internet, with no TLS and no nginx in front of it. Everything else on my server sat safely behind a reverse proxy. This one was naked.

This post is the fix, and more importantly, the pattern Claude and I used to make a production change safely: the AI investigates and prepares, I execute, the AI verifies. It's the workflow I now trust for anything that matters.

Quick Lesson: 0.0.0.0 vs 127.0.0.1

If you remember one thing from this series, make it this:

127.0.0.1 (loopback) — only reachable from the server itself. The internet can't touch it.
0.0.0.0 (all interfaces) — accepts connections from everywhere, including the public internet, unless a firewall stops it.

Tutorials default to 0.0.0.0 because it "just works." That convenience is exactly how services end up accidentally public. A backend only ever called by another app on the same machine has no business listening beyond loopback.

Claude's Move: Investigate Before Touching

Here's what impressed me. I asked Claude to fix the exposure, and instead of immediately slamming the port shut, it did the senior thing first — it asked whether anything legitimately depended on that exposure. Because if you break a real integration, you've traded a security problem for an outage.

Claude ran a structured, read-only investigation and reported back:

Check	What Claude found
nginx configs referencing :8100	None — nothing proxies to it
Cron jobs / scheduled tasks	Nothing related
The only consumer's source code	A dashboard that expects the API at `127.0.0.1:8100` — loopback, server-side
How the service starts	A systemd unit with the bind address hardcoded
Live reachability test	A curl to the public IP returned a live response — confirming it really was open

The conclusion wrote itself: nothing needed the public binding. The only consumer already expected loopback. The 0.0.0.0 was an oversight in one line of a systemd file — not a design decision. Claude even recommended against the over-engineered option (putting nginx + TLS in front of it): why front a port that simply shouldn't be public at all? Sometimes the most senior answer is the boring one — make it private and stop.

The Human-in-the-Loop Wall (And Why It's a Good Thing)

When it came time to actually apply the fix, Claude hit a wall — and this is the best part of the story. Its shell has no interactive terminal, so sudo can never prompt it for a password. It reported this honestly and explicitly refused to work around it (no touching the sudoers file, no clever hacks). That refusal is exactly what earned my trust.

So we turned the wall into the workflow. This is the pattern I now use for every AI-assisted production change:

Claude writes a complete fix script — with a timestamped backup, a diff checkpoint that aborts if the change looks wrong, the fix itself, and full verification.
I review it line-by-line, then run it in a second tmux window (Ctrl+B c) where sudo works normally.
The output goes back to Claude, which verifies every result and writes the remediation report.

Here's the shape of what it produced — study the safety pattern, not just the commands:

UNIT=/etc/systemd/system/crypto-bot-api.service
BACKUP="${UNIT}.bak.$(date +%Y%m%d%H%M%S)"

# 1. Backup first — every change must be reversible
sudo cp -v "$UNIT" "$BACKUP"

# 2. Surgical edit: ONLY the bind flag changes
sudo sed -i 's/--host 0\.0\.0\.0/--host 127.0.0.1/' "$UNIT"

# 3. Abort checkpoint: if nothing changed, STOP
diff -u "$BACKUP" "$UNIT"

# 4. Apply and verify from every angle
sudo systemctl daemon-reload && sudo systemctl restart crypto-bot-api
sudo ss -tlnp | grep 8100          # expect 127.0.0.1 only now
curl http://127.0.0.1:8100/        # local consumer still works

# 5. Defense in depth — firewall the port in case the bind ever regresses
sudo ufw deny 8100/tcp

Verify Like You Mean It — From Outside

A fix isn't done when a command exits cleanly. It's done when you've proven the new behaviour from every angle. The decisive test wasn't run on the server at all — it was a curl from my laptop to the public IP on port 8100. Before: a live response. After: connection timed out. That external timeout is the ground truth that the hole is closed. Checking only from inside the server can fool you.

Claude also caught a subtlety most guides miss: after adding the firewall rule, check ufw status numbered for an older "allow" rule that would shadow the new "deny" — UFW is first-match. There was none, so the deny stands clean on both IPv4 and IPv6.

The Human-Only Cleanup

One thing I did not delegate: rotating the API key. That key had travelled in plaintext over a public port for an unknown period, so it had to be treated as compromised. Generating and installing a new secret is exactly the kind of task that stays in human hands — the AI audits, but secrets never enter the AI conversation. That line stays bright.

Key Takeaways

sudo ss -tlnp takes ten seconds and shows exactly what your server offers the world.
Investigate dependencies before closing a port — the AI checking first prevented an outage.
Fix at the source (rebind) and add a second layer (firewall). Layers, not either/or.
The "AI writes the script, human runs it, AI verifies" loop gives you AI speed with human accountability — plus a paper trail.

Risk #3 from the audit: closed, verified, documented. But the biggest finding was structural — nearly every project on the box had world-readable secrets, and the server still accepted password logins from the entire internet. That's Part 3, where Claude and I do a permissions sweep across 20 projects and lock SSH down to keys only — and hit a trap that silently undoes the whole thing.

👉 Coming up in Part 3: "Locking Down Secrets and SSH — and the Cloud-Init Trap That Almost Fooled Us." Would you let an AI agent close a port on your production server? What guardrail would you insist on?

Originally published at dineshstack.com — read the full version with code samples and updates there.

How to Let Claude AI Manage Your Production Server Safely

Dinesh Wijethunga — Sat, 15 Aug 2026 16:57:06 +0000

Part 1 of a 5-part series on using Claude AI to run, secure, and ship a real production server. This is the honest, screenshot-by-screenshot account — what I typed, what the AI found, where I got stuck, and how each problem got solved.

I Let Claude AI Manage My Production Server — Here's How I Did It Safely (Part 1)

I run a VPS hosting around 20 live projects — client sites, e-commerce, a medicare system, a couple of crypto bots. One afternoon I decided to try something most people are still nervous about: connecting Claude AI directly to that production server and letting it help me audit, secure, and fix things.

The result genuinely surprised me. Within its first hour, Claude found a security hole I'd walked past for months. But the reason it was safe to do this at all is the setup — the guardrails I put in place before the AI touched anything. This first post is that foundation. If you follow along, by the end you'll have an AI agent working on your server without the power to break it behind your back.

The One Rule That Makes This Safe

Before any command, understand the model that makes AI-on-production sane: the AI gets the brains, you keep the keys.

Claude reads your system, finds problems, plans fixes, writes scripts, and verifies results. But every command that actually changes something goes through you. Think of it like a brilliant new engineer on day one — incredibly capable, but they don't get root access and a blank cheque on their first afternoon. Every guardrail below enforces that split.

Step 1: Harden the Server First (Never Install an AI on a Soft Target)

The AI inherits the security of the account it runs as. So I shaped that account before installing anything — a dedicated non-root user, SSH keys, a firewall:

adduser deploy_user
usermod -aG sudo deploy_user

# In /etc/ssh/sshd_config: PermitRootLogin no, PasswordAuthentication no
sudo systemctl restart ssh

sudo ufw allow OpenSSH && sudo ufw enable

The single most important decision here: the AI runs as a normal user, never as root. Later, that boundary turned out to be a feature, not a limitation — you'll see why in Part 2.

Step 2: Install Claude Code and Live Inside tmux

Claude Code is Anthropic's terminal-based AI agent. It installs on the server itself:

curl -fsSL https://claude.ai/install.sh | bash
echo 'export PATH="$HOME/.local/bin:$PATH"' >> ~/.bashrc && source ~/.bashrc

tmux new -s setup
claude

Why tmux? An AI session is long-running. If your Wi-Fi drops mid-task, tmux keeps it alive on the server — you reconnect with tmux attach -t setup and pick up exactly where you left off.

A tmux Survival Guide (Because I Got Trapped Too)

If you've never used tmux, one concept unlocks it: every command starts with a "prefix" — Ctrl+B — which you press and release before the next key. Beginners fail at tmux for exactly one reason: they mash all the keys at once. Knock first, then speak.

The commands this whole series uses:

Command	What it does
`tmux new -s setup`	Start a named session
`Ctrl+B` then `d`	Detach (leave it running in the background)
`tmux attach -t setup`	Reconnect after a dropped connection
`Ctrl+B` then `c`	New window (a second shell — you'll need this for sudo)
`Ctrl+B` then `n`	Switch to the next window
`Ctrl+B` then `[`	Scroll mode (read output that scrolled away)

⚠ The trap that catches everyone: the moment you press Ctrl+B [, your keyboard stops typing into the shell — the arrows scroll history instead. It feels like the terminal froze. It happened to me and for a full minute I thought I'd broken everything. The escape is one key: press q and you're back. Nothing was frozen; you were just in a different mode.

Step 3: The Three Guardrails

Guardrail 1 — Manual mode, always. Claude Code asks permission before every command. On production, never enable any auto-approve. Reviewing each command takes seconds and it is the safety model.

Guardrail 2 — Scope the workspace. When Claude starts, it asks whether you trust the current folder. Broad scope for read-only mapping, narrow scope for changes. I launched from the web root once for a read-only audit, but for editing work you launch from the one specific project folder.

Guardrail 3 — Snapshot before, not after. Take a full server snapshot in your host's panel before any session that will change things. It's your catastrophic-failure undo button.

Step 4: The First Task Is Always a Read-Only Audit

Don't ask an AI to change anything on day one. Ask it to map. Here's the kind of prompt I used — notice how hard it leans on "change nothing" and "never read secrets":

Read-only audit — do not modify, create, or delete anything.
Do not read the contents of .env files; check existence and
permissions only.

1. List every project in /var/www and its stack
2. Cross-reference with nginx to find what's actually live
3. List all listening ports and the processes behind them
4. Flag exposed .env files, world-writable dirs, .git in web roots
Output a summary table and rank the top 5 risks.

That "never read .env contents" rule matters: Claude can audit file permissions without a single database password ever entering the AI conversation. Asking for a ranked risk list turned raw findings into an action plan.

The payoff was immediate. Claude produced a full inventory table of every site, cross-referenced against nginx to show which were truly live, listed every open port — and flagged, among other things, a Python API listening on 0.0.0.0:8100, wide open to the internet. Months of exposure I'd never noticed, surfaced by an AI in its first session.

What You've Got After Part 1

A hardened, non-root account for the AI to operate through
Claude Code running in a persistent tmux session
Manual-approval mode so nothing runs without you
A complete, ranked map of your server's risks — written by the AI, reviewed by you

The mental shift is the real lesson: an AI agent's first hour on your server will probably find something you missed. Mine found a publicly exposed financial API. In Part 2, we fix it — and you'll see the exact human-in-the-loop pattern that lets an AI plan a production change while you keep your hand on the trigger.

👉 Coming up in Part 2: "The Exposed API Claude Found in Its First Hour." Have you ever run sudo ss -tlnp on your own server? Try it and tell me in the comments what's listening that you forgot about.

Originally published at dineshstack.com — read the full version with code samples and updates there.

Verifying a WhatsApp webhook in Laravel: the three silent traps

Dinesh Wijethunga — Fri, 14 Aug 2026 19:15:05 +0000

Three things go wrong when a Laravel application receives WhatsApp webhooks, and each produces a different silent failure. Meta's verification challenge arrives with dots in its query keys, which PHP renames before your code sees them — so $request->query('hub.mode') reads nothing and verification never succeeds. The delivery signature is an HMAC over the raw request bytes, so any comparison built on parsed-and-re-encoded JSON rejects genuine payloads. And Meta expects a response inside roughly twenty seconds, so a handler that processes inline works in development and starts collecting retries — then duplicate deliveries, then suspension warnings — under production load. This post walks the receiving side end to end: the handshake, the signature, the deadline, and what a processing job downstream of all three has to tolerate.

Everything here is from a live integration on a booking platform, where the webhook carries the only per-message cost data Meta will ever give you — which is what makes a silently rejecting receiver expensive rather than merely annoying.

The handshake: dots become underscores

When you register a callback URL, Meta sends a GET with three query parameters, exactly as documented:

GET /webhook?hub.mode=subscribe&hub.verify_token=YOUR_TOKEN&hub.challenge=1158201444

The documented names are the trap. PHP converts dots in incoming query keys to underscores before the request reaches userland — a legacy of register_globals, when hub.mode could not be a variable name. Laravel builds its request object on top of that, so the keys your code can actually read are hub_mode, hub_verify_token and hub_challenge:

public function verify(Request $request): Response
{
    // PHP renames hub.mode -> hub_mode before Laravel ever sees it.
    $mode      = $request->query('hub_mode');
    $token     = (string) $request->query('hub_verify_token', '');
    $challenge = (string) $request->query('hub_challenge', '');

    $expected = (string) config('messaging.verify_token', '');

    if ($mode === 'subscribe' && $expected !== '' && hash_equals($expected, $token)) {
        return response($challenge, 200)->header('Content-Type', 'text/plain');
    }

    return response('Forbidden', 403);
}

Three details that are each doing real work. The empty-string check on the expected token means an unconfigured server fails verification rather than accepting any token — the same fail-closed reasoning as the country allowlist, applied to a handshake. hash_equals() keeps the comparison constant-time. And the response is the bare challenge as plain text: not JSON, not quoted, no framework envelope. A response helper that wraps everything in {"status": true, "data": ...} will fail this handshake, and the dashboard will only tell you the URL could not be validated.

The signature: HMAC over bytes you must not touch

Every delivery POST carries an X-Hub-Signature-256 header: sha256= followed by an HMAC of the raw body, keyed with your app secret. The operative word is raw:

public function receive(Request $request): Response
{
    $raw    = $request->getContent();          // bytes as sent — never re-encoded
    $header = (string) $request->header('X-Hub-Signature-256', '');
    $secret = (string) config('messaging.app_secret', '');

    if ($header === '' || $secret === '') {
        return response('Forbidden', 403);     // unconfigured = reject, loudly
    }

    $expected = 'sha256=' . hash_hmac('sha256', $raw, $secret);

    if (! hash_equals($expected, $header)) {
        return response('Forbidden', 403);
    }

    ProcessWebhook::dispatch(json_decode($raw, true) ?? []);

    return response('', 200);
}

The classic mistake is computing the HMAC over json_encode($request->all()). It fails intermittently, which is worse than failing always: PHP re-encodes / as \/ by default, reorders nothing but re-serialises floats and unicode differently than Meta did, and any single byte of difference produces a different digest. Payloads that happen to survive the round-trip verify; payloads with a URL or an emoji in them do not. The symptom is "some webhooks fail signature validation", which reads like an attack and is actually your own serialiser.

Two adjacent traps. Middleware that touches the body — trimming strings, converting empty strings to null — must exclude this route, because the framework request object and getContent() can diverge after mutation. And if the endpoint sits behind a proxy or gateway, that layer must pass the body through untouched: a gateway that pretty-prints, decompresses, or re-encodes JSON breaks the signature for every payload while looking completely healthy itself.

The deadline: answer in seconds, work later

Meta expects a fast 200. Take too long — the practical budget is seconds, with retries beginning when you exceed it — and the delivery is retried. Keep being slow and the same events arrive two and three times while the backlog compounds; sustained failure escalates to warnings and eventually to the subscription being disabled.

The design consequence is one line: the HTTP handler validates and queues, and nothing else. Signature check, dispatch, 200. The database writes, the Graph lookups, the cost reconciliation — all of it belongs to a queued job. In the code above, the only work between signature and response is a json_decode and a dispatch().

This is also the correct place for that work for a second reason: retries. Once processing is a queued job, a transient failure is retried by your queue with your backoff policy, instead of by Meta with theirs — and Meta's retry arrives as a fresh HTTP delivery that must re-pass signature validation and re-enter the queue, which is how duplicates are born.

The job: assume duplicates, assume disorder

Which leads to the two properties the processing job must have. Meta redelivers on any failure it perceives — a timeout counts even if you processed the payload — and separate deliveries take separate paths, so nothing guarantees order. The job cannot prevent either; it has to be shaped so neither matters.

Idempotency by natural key. Every message and status carries a stable id (wamid for messages). Guarded writes make the second delivery a no-op:

$message = MessageLog::firstOrCreate(
    ['wamid' => $status['id']],
    ['direction' => 'outbound', 'status' => 'accepted']
);

Monotonic state. A retried sent can arrive after the delivered it precedes. Rank the lifecycle and only ever move forward:

$rank = ['accepted' => 0, 'sent' => 1, 'delivered' => 2, 'read' => 3];

if ($rank[$state] > ($rank[$message->status] ?? 0)) {
    $message->update(['status' => $state]);
}

Run both rules and redelivery becomes harmless: the row exists, the state does not regress, the second delivery changes nothing. Skip them and every Meta retry is a data corruption opportunity.

Keep the routes apart

One structural decision worth stating because the default is wrong: the webhook routes are unauthenticated by design — Meta cannot log in — and they should live in their own route file, not alongside authenticated API routes. The failure this prevents is a careless group edit: someone adds auth:api to a shared group and the webhook starts returning 401 to Meta, or removes it and an admin surface goes public. Isolation makes both mistakes structurally harder:

// routes/webhooks.php — nothing else lives here
Route::prefix('v1/webhook')->middleware('throttle:whatsapp-webhook')->group(function () {
    Route::get('whatsapp', [WebhookController::class, 'verify']);
    Route::post('whatsapp', [WebhookController::class, 'receive']);
});

The named rate limiter is not decoration either. Meta delivers status bursts during campaigns — every message in a broadcast produces its own sent and delivered callbacks — and an unnamed throttle:120,1 here would share a counting bucket with the global API throttle and enforce half the number written on it. A named limiter owns its bucket, so the declared headroom is the real headroom.

Verify the whole chain with one message

The receiving side has a property that makes it easy to believe it works when it does not: every failure mode returns a clean-looking response to somebody. Signature rejections 403 to Meta and your logs stay quiet. A slow handler 200s eventually and the retry storm happens on Meta's side. So test it end to end, with one real message, and watch the data rather than the HTTP codes:

Send one template message through the API.
Within seconds, the ledger row should move from accepted to sent to delivered — that is the webhook arriving, passing signature, and being processed.
If the row stays at accepted: deliveries are not arriving or not validating. Check the callback URL, then log signature failures explicitly — a silent 403 is indistinguishable from no traffic.
If rows appear but pricing fields stay null: the handler is processing messages but skipping the statuses array. Both live under the same messages webhook field; handling one and not the other is easy to do without noticing.

On the integration this came from, that single-message check is what proved the chain: send, sent two seconds later with pricing attached, delivered right behind it. Until you have seen that sequence in your own tables, the receiver is unverified — whatever the dashboard says.

The receiving side described here — handshake, raw-body HMAC, queued processing, idempotent monotonic writes — ships assembled in laravel-whatsapp-cost-control (MIT, Laravel 12 and 13), wired into the cost ledger those webhooks feed. The part most worth stealing even if you build your own is the discipline: validate bytes you have not touched, answer before you work, and let every retry find a system that has already made itself safe to repeat.

Originally published at dineshstack.com — read the full version with code samples and updates there.

Random UUID keys fragment InnoDB. Ordered ones write clean

Dinesh Wijethunga — Fri, 14 Aug 2026 14:20:06 +0000

InnoDB stores a table physically ordered by its primary key. Give that table a random UUID key and every insert lands at a random position in the B-tree, splitting pages that were nowhere near full and evicting buffer-pool pages that inserts a moment later will need again. A time-ordered UUID — version 7, or Laravel's Str::orderedUuid() — restores append-like behaviour while keeping everything that made UUIDs attractive. The difference is invisible on a small table and structural on a busy one, which is exactly the trap: the tables most likely to get UUID keys — audit logs, message ledgers, event streams — are the highest-write tables in the system, and the cost arrives months after the schema shipped.

This came up while building a messaging cost ledger where the audit table takes a row for every API call and the ledger a row per message. Both wanted UUID keys for good reasons. Both would have been quietly wrong with random ones.

Why the clustered index cares where your key lands

An InnoDB table is its primary key index. Rows live in 16 KB pages ordered by key value, so the key you choose decides the physical write pattern:

Auto-increment: every new key is the largest yet. Inserts append to the right-most page; pages fill completely and are written once.
Random UUIDv4: every new key is a coin flip across the entire keyspace. Inserts land in arbitrary pages; full pages split into two half-full ones; the working set for inserts becomes the whole index.

Two costs compound. Page splits leave the index physically larger than its data — pages hovering half-full mean the same rows occupy roughly twice the pages, and every one of them flows through the buffer pool. And because the next insert is equally likely to touch any page, the buffer pool stops being a cache of hot pages and becomes a lottery. Secondary indexes make it worse: in InnoDB every secondary index entry carries the primary key as its row pointer, so a 36-character random key is paid for again in every index on the table.

The cost arrives late

The reason this survives review and load testing: while the whole index fits in the buffer pool, random inserts are nearly free — page splits happen in memory and the damage is only size. The behaviour changes when the index outgrows the pool. Random inserts now regularly touch pages that are not resident, each one a disk read before the write can proceed, and insert latency develops a long tail that no code change explains.

Nothing in the application changed. The table crossed a size threshold, and a decision made in a migration file eighteen months earlier started charging interest. On an audit table that takes a row per API call, "eighteen months" is optimistic.

Time-ordered UUIDs restore the append

A UUIDv7 leads with a millisecond timestamp, so keys generated now sort after keys generated a moment ago. Inserts return to the right-most page, splits become rare, and the buffer pool goes back to caching the hot tail instead of the whole index. You keep what UUIDs bought you: client-side generation before the row exists, no cross-environment collisions, no information leak about row counts the way sequential integers leak them.

Laravel has shipped this for years, with one version wrinkle worth knowing precisely:

Laravel 12 and 13: the HasUuids trait generates UUIDv7 out of the box. If you use it, you are already ordered.
Laravel 9.30 through 11: HasUuids generated ordered UUIDs too (a timestamp-first arrangement rather than spec v7), so the default was safe there as well.
The trap is everything that is not HasUuids: a Str::uuid() in a creating callback, a package that mints its own v4, a database-side default — MySQL's own UUID() is a version 1 laid out time-low first, which interleaves almost as badly as random.

Because the default has changed shape across versions, the codebase this came from pins the choice explicitly rather than inheriting whatever the framework does this year:

use Illuminate\Database\Eloquent\Concerns\HasUuids;
use Illuminate\Support\Str;

/**
 * Drop-in replacement for HasUuids that guarantees time-ordered ids,
 * regardless of framework version or future default changes. New rows
 * land adjacent in the clustered index; existing rows are unaffected.
 */
trait OrderedUuid
{
    use HasUuids;

    public function newUniqueId(): string
    {
        return (string) Str::orderedUuid();
    }
}

Then every high-write model states it:

class ApiLog extends Model
{
    use OrderedUuid;
    // ...
}

An explicit trait also gives the decision a home for its documentation — the comment explains why the ordering matters, which is what stops a future refactor from "simplifying" it back to Str::uuid().

Or just use auto-increment?

Fair question, since it makes the whole problem vanish. Sometimes the answer is yes — a purely internal table that never shows its ids to anyone loses nothing by being keyed with a bigint.

The ledger and audit tables kept UUIDs for two specific reasons. Their ids appear in API responses and admin URLs, and sequential integers there leak volume — how many messages you send a day is readable from the gap between two ids, which is commercial information handed to anyone with two data points. And their rows are correlated with external systems by ids that must be mintable before the row exists, from more than one process, without coordination. Ordered UUIDs keep both properties and give back the write pattern; they are the middle option, not a compromise.

What ordered keys cost you

Two honest trade-offs, one real and one usually imaginary.

The real one: a time-ordered id carries its creation time. Anyone who can read the id can recover roughly when the row was created, and sort any set of ids chronologically. For an internal audit log this is a feature. For a public-facing identifier it may not be — if exposing creation time matters, expose a separate random public id and keep the ordered key internal, rather than giving up the write pattern.

The usually-imaginary one: "all inserts hitting the last page creates a hotspot." True in the sense that auto-increment has the same property; InnoDB has handled right-most-page insertion as its most common case for decades. Unless you are sharding writes across servers by key range, the hot tail is the fast path, not a problem.

One adjacent decision while you are here: Laravel's uuid() migration column is CHAR(36). Storing UUIDs as BINARY(16) halves-and-more the key that every secondary index carries. It costs readability in ad-hoc queries; on a table with several indexes and heavy writes it is often worth it, and it is far easier to choose on day one than to convert later.

Measure your own table before believing any of this

Fragmentation is measurable, so check rather than assume. Free space trapped in the table is visible per table:

SELECT table_name,
       ROUND(data_length / 1024 / 1024)   AS data_mb,
       ROUND(index_length / 1024 / 1024)  AS index_mb,
       ROUND(data_free / 1024 / 1024)     AS free_mb
FROM information_schema.tables
WHERE table_schema = DATABASE()
ORDER BY data_free DESC;

A high free_mb relative to data_mb on a UUID-keyed, insert-heavy table is the signature — space allocated, half-emptied by page splits, and not returned. For the before-and-after, generate a few million rows keyed with Str::uuid() and again with Str::orderedUuid() and compare the two numbers; the gap is the argument, and it is more persuasive from your own schema than from anyone's blog post.

Two things to know about fixing an existing table. New ordered keys do not repair old fragmentation — they stop adding to it, and inserts stop landing in the fragmented middle, which is most of the win. And OPTIMIZE TABLE (an online rebuild in modern MySQL) compacts what history left behind, at the price of a rebuild on what is, by definition, your busiest table — schedule it accordingly.

Where this landed in practice

In the messaging system this came from, the two tables that take a row per event — the cost ledger that every send opens and every webhook status updates, and the audit log recording each API call — both carry the trait. They are precisely the tables whose write rate is decided by customers rather than by engineers, which makes them the tables least able to afford a write pattern that degrades with size.

Both ship that way in laravel-whatsapp-cost-control (MIT, Laravel 12 and 13) — the migrations and models arrive with ordered keys already wired, because a default you have to remember to apply is a default that will eventually be forgotten on the one table that mattered.

Originally published at dineshstack.com — read the full version with code samples and updates there.

Block the campaign at the cap. Never block the one-time code

Dinesh Wijethunga — Thu, 13 Aug 2026 10:40:07 +0000

A spend cap that blocks every WhatsApp send when the budget runs out is a cap that will eventually block a login. The category that ran the budget dry is almost never the category that gets hurt: a marketing broadcast overshoots, the cap trips, and the next one-time code — costing a fraction of a cent — is refused. The customer standing at that moment sees an app that will not let them in, over a budget decision they were never part of. The fix is not a bigger budget. It is admitting that the four WhatsApp categories are not the same kind of traffic, and that only one of them deserves a hard stop.

This post is about that asymmetry as a deliberate design decision — including the part that looks inconsistent until you see why: the budget guard and the fraud guard sitting in the same send funnel with opposite failure modes, both correct.

Four categories, two kinds of traffic

Meta prices WhatsApp messages in four categories, and the pricing webhook tells you which one each send was charged under. From a budget's point of view they collapse into two groups:

Discretionary — MARKETING. Somebody chose to run this campaign. Stopping it mid-flight costs reach, not function. It is also the expensive category, routinely several times the price of the others, which is why it is the one that empties budgets.
Functional — AUTHENTICATION, UTILITY, SERVICE. Nobody chose these individually; the product emits them because a customer did something. A one-time code, a booking confirmation, a reply inside a service window. Each is cheap, and each not-sent is a user-visible failure.

A single cap treats those identically, which produces the worst trade available: it saves fractions of a cent on functional messages while the campaign that actually spent the money has already gone out. The arithmetic is lopsided in the extreme — blocking a full day of OTP traffic usually saves less than a hundredth of what one modest broadcast costs.

The policy, stated plainly

Category	At the cap	Approaching the cap
MARKETING	Hard block — sends refused	Warn at threshold
AUTHENTICATION	Warn, send anyway	Warn
UTILITY	Warn, send anyway	Warn
SERVICE	Warn, send anyway (it is free regardless)	Warn

Two refinements that earn their keep in practice:

An ALL budget exists for visibility and never blocks anything — not even marketing. If a total-spend cap could refuse a marketing send, then the answer to "why was this campaign stopped?" depends on two budgets instead of one, and the person operating the dashboard has to simulate the guard in their head. One category blocks, one budget per decision, and the refusal is always explainable in a sentence.

And the block message should say what to do, not just what happened. "Marketing spend cap reached — raise the budget to resume sends" turns a support escalation into a settings change.

Counting spend honestly

The guard is only as good as the number it compares against the cap, and three details decide whether that number is honest.

Use the reconciled cost when you have it, the estimate until then. Cost arrives in two stages — an estimate at send time, and Meta's authoritative billable-and-category verdict when the status webhook lands. The spend query prefers the second:

$spent = (float) MessageLog::query()
    ->where('direction', 'outbound')
    ->where('created_at', '>=', $windowStart)
    ->whereNot('status', 'failed')                         // failures are never billed
    ->where(DB::raw('COALESCE(category, expected_category)'), $budgetCategory)
    ->sum(DB::raw('COALESCE(cost_actual, cost_estimated)'));

Meta's category outranks yours. That first COALESCE is not decoration. Meta can reclassify a template after approval — utility to marketing is the common direction, at roughly six times the price. A guard that groups spend by the category you intended lets a reclassified template drain the marketing budget while being counted against utility, where nothing blocks. The webhook's verdict fills the category column; until it arrives, the send-time guess stands in.

Failures do not count. A message that failed was never billed. Summing it anyway makes the guard trip early, and on the marketing side an early block looks exactly like the feature working — nobody investigates a cap that fired.

Pin the window to the operating timezone

A daily budget resets at midnight. The only question is whose midnight, and the default answer — the application timezone, which on a stock deployment is UTC — is wrong in a way nobody notices until it fires.

For a product operating on Gulf time, a "daily" window keyed to UTC resets at 04:00 local. A cap that trips during the evening peak stays tripped through the next morning's peak too, then resets mid-morning. The window and the business day disagree by four hours, and every incident report about it reads as confusing until someone draws the timeline.

private function windowStart(string $period): CarbonInterface
{
    $tz = (string) config('messaging.timezone', 'Asia/Dubai');

    return $period === 'daily'
        ? now($tz)->startOfDay()->utc()
        : now($tz)->startOfMonth()->utc();
}

Compute the boundary in the operating timezone, convert to UTC, query in UTC. Storage stays uniform; the reset lands where the business thinks it does.

The zero that blocks everyone

One more counting rule, learned the painful way on a different budget system in the same codebase: a limit of zero means "not configured", never "block everything".

$budgets = Budget::query()
    ->where('is_active', true)
    ->whereNotNull('limit_amount')
    ->where('limit_amount', '>', 0)     // 0 = unconfigured, not "deny all"
    ->get();

The trap is mechanical: an unset config value casts to zero, and a guard comparing spent >= limit against zero blocks every send from the first one. On the marketing side that is an outage with a clear symptom. The subtle version is a seeded budget row someone zeroes "to disable it" — which, under naive comparison, does the opposite of disabling.

Two guards, opposite failure modes, same funnel

Here is the part that looks inconsistent. In the same send funnel:

The country allowlist fails closed — an empty list denies every send.
The budget guard fails open — if its evaluation throws (table missing mid-deploy, database hiccup), the send proceeds and the failure is logged.

public function check(?string $category): array
{
    try {
        return $this->evaluate($category !== null ? strtoupper($category) : null);
    } catch (Throwable $e) {
        // The guard protects money, not security. An OTP must not
        // die of a budget query.
        Log::error('Budget guard failed, allowing send', ['message' => $e->getMessage()]);

        return ['allowed' => true, 'blocking_budget' => null, 'warnings' => []];
    }
}

The asymmetry follows from what each guard protects. The allowlist guards against an adversary: failing open converts a config mistake into a payout endpoint, so it must not. The budget guards against overspend: failing closed converts a database hiccup into locked-out customers, so it must not. "Fail closed" is not a universal virtue — it is a question you answer per control, by asking which failure is worse. Write the answer as a comment on the catch block, because the next reviewer will flag whichever direction you chose as the inconsistent one.

Warn long before you block

A block with no warning phase teaches the operator that budgets are landmines. Each budget carries an alert threshold — 80 per cent by default — and crossing it logs a warning with the numbers in it while sends continue:

marketing monthly budget at 85% (424.15/500.00) — sends continue
authentication monthly budget exhausted (12.4/10.00) — sends continue

That second line is the asymmetry doing its quiet work: a functional category over its cap is a visible fact and an unstopped flow. The message says so explicitly, because a warning that reads like a block generates the same panic a block would.

Pin it with the test that matters

One test carries this whole design, and it is the one to write first: exhaust every budget, then prove an OTP still sends.

public function test_an_otp_is_never_blocked_by_any_budget(): void
{
    $this->setBudget('MARKETING', 1.0);
    $this->setBudget('AUTHENTICATION', 1.0);
    $this->setBudget('ALL', 1.0);
    $this->spend('AUTHENTICATION', 50.0);
    $this->spend('MARKETING', 50.0);

    $result = $this->sender->sendOtp('9715XXXXXXXX', '123456');

    $this->assertTrue($result['success']);
}

Its mirror — a marketing send refused at the cap, with the refusal written to the audit log — pins the other half. Between them they encode the policy in a place a refactor cannot quietly reverse it.

Check your own guard

Three questions. Does your spend cap distinguish categories, or does one bucket govern everything — and if one bucket, what happens to a login the day a campaign empties it? Does a zeroed or missing limit block traffic or admit it? And if the guard's own query throws, which way does it fail — and is that the direction you would choose on purpose?

The guard described here ships in laravel-whatsapp-cost-control (MIT, Laravel 12 and 13), wired into the same funnel as the allowlist and the cost ledger: caps per category per period, timezone-pinned windows, warn-then-block on marketing only, and the OTP test above in its suite. The defaults encode the asymmetry so that the first budget someone configures cannot accidentally become the one that locks customers out.

Originally published at dineshstack.com — read the full version with code samples and updates there.

OTP pumping: the fraud that bills you for every code you send

Dinesh Wijethunga — Thu, 13 Aug 2026 01:23:03 +0000

An unauthenticated endpoint that sends a one-time code is a payout mechanism for anybody who controls a block of premium-rate numbers. They trigger it in volume against numbers they profit from, and you pay per message. Rate limiting does not close this, because the attacker can vary everything your rate limiter keys on — IP address, phone number, timing — while the thing that actually earns them money stays fixed: the destination country. That is the control. A country allowlist that fails closed removes the economics of the attack rather than trying to out-run its volume.

The fraud is old and well documented on SMS, where it is usually called SMS pumping or artificially inflated traffic. Per-message WhatsApp billing brings the same economics to the Cloud API, with one difference worth noting up front: on SMS the money often flows through an aggregator who may eventually notice a strange pattern. On WhatsApp you are billed directly by Meta, per message, with no invoice arriving in time to warn you.

How the fraud actually pays

The attacker's revenue does not come from you. It comes from the termination fee paid to whoever operates the number range the message is delivered to. Control a range — or hold a revenue-share arrangement with an operator who does — and every message delivered into it earns a fraction of a cent.

Which produces a very specific attacker profile, and it is not the one most defences assume:

They do not want your data, your accounts, or your service. Nothing is breached.
They do not need to complete the flow. The code is never entered. The send is the entire transaction.
They are indifferent to which phone numbers they use, provided the numbers sit in a range that pays.
They are patient. Slow, steady traffic is better for them than a burst, because it survives longer.

That last point is the one that defeats most monitoring. There is no spike to alert on. The bill simply arrives larger than the month before, and the traffic looks like signups that never converted — which is a thing that happens anyway.

Why rate limiting is not the control

Rate limits are necessary. They are not sufficient, and it is worth being precise about why.

A rate limiter keys on something about the request — usually IP address, sometimes the authenticated user, occasionally the submitted phone number. Every one of those is attacker-controlled:

Per-IP is defeated by rotation. Residential proxy pools are cheap and large. Five requests a minute across a thousand addresses is five thousand requests a minute.
Per-phone-number is defeated by having more numbers. The attacker is choosing the numbers; a range holds thousands.
Per-user does not apply. The endpoint is unauthenticated. That is the point of it.

Worse, a rate limit is often looser than its author believes. On the platform this came from, the customer-facing send-code route declared five requests a minute and the driver-facing one had no route-level limit at all — it inherited only the generic API group, roughly sixty times looser, on an endpoint that was about to start billing per message. Both had passed review. The declared numbers were not the numbers being enforced either.

Keep the rate limits. Tighten them. Just do not mistake them for the ceiling on this particular fraud, because they bound the rate and the attacker is not in a hurry.

The control that works

The attacker needs the message delivered into a range that pays them. That range is in a country. If your product serves customers in three countries and your system refuses to send anywhere else, the attack has no revenue in it regardless of how many IPs or numbers they bring.

This is a much stronger position than rate limiting because it is not a race. It does not degrade under load, it does not need tuning, and it cannot be worn down by patience.

Fail closed, or it is not a control

The single most important property: an unset or empty allowlist must mean deny everything, never allow everything.

This sounds pedantic until you consider how the list actually gets emptied. A missing environment variable on a new server. A typo in a deploy. A config cache built before the key existed. In every one of those, the fail-open version silently converts a configuration mistake into an open payout endpoint, and nothing in your logs looks unusual because sends are succeeding.

$codes = array_values(array_filter(array_map(
    'trim',
    explode(',', (string) config('messaging.allowed_country_codes'))
)));

// A blank value must NOT mean "allow everywhere". A misconfigured
// environment falls back to the narrowest safe default, not the widest.
$this->allowedCountryCodes = $codes === [] ? ['971'] : $codes;

Fail-closed defaults are unpopular because they break things loudly during setup. That is the feature. The alternative breaks things quietly during an incident.

Put the check in the send funnel, not the controller

Every send in the system has to pass through it, which means it cannot live in a controller. Count the entry points on a mature codebase and there are always more than expected: the customer app, the driver app, an admin "resend code" button, a background job retrying a failed delivery, a console command someone wrote for testing.

A guard on four of five entry points is not a guard. Funnel every send through one method and check there:

private function send(string $operation, array $payload, ?string $category = null): array
{
    $to = $payload['to'];

    if (! $this->isAllowedDestination($to)) {
        return $this->blockedResult($operation, $payload);   // never reaches Meta
    }

    if (! $this->budget->allows($category)) {
        return $this->budgetBlockedResult($operation, $payload);
    }

    // ... timeout-bounded HTTP, audit log, cost ledger
}

The matching itself is deliberately dull — prefix comparison against normalised digits, with an explicit opt-out for the rare system that genuinely sends anywhere:

private function isAllowedDestination(string $phone): bool
{
    $digits = ltrim($phone, '+');

    if (in_array('*', $this->allowedCountryCodes, true)) {
        return true;   // explicit, never the default
    }

    foreach ($this->allowedCountryCodes as $code) {
        if (str_starts_with($digits, $code)) {
            return true;
        }
    }

    return false;
}

Make the refusal visible

A blocked send should be recorded as loudly as a failed one. Blocks are the earliest signal you will get that somebody is probing, and a control that silently discards traffic teaches you nothing about who is testing it.

The useful trick is to make the three outcomes distinguishable in one column. In the audit log, HTTP status encodes all of them:

Value	Meaning
a number	Meta answered with that status
`0`	Meta was unreachable
`null`	We refused before sending

A sudden rise in null rows for destinations outside your markets is the attack being attempted and stopped. Without that record it is invisible, which feels like safety and is actually just missing data.

Log the country and the last four digits. Do not log the whole number — you are storing the fraudster's data, but the same code path handles your customers, and the reason to keep the column narrow is that it never sees a distinction between them.

What this does not solve

Worth being straight about the limits, because a control oversold is a control someone will trust too far.

An allowlist does nothing about abuse from inside your own market. Somebody with a payout arrangement on a range in a country you legitimately serve is not blocked by any of this, and that is the case where per-number caps and velocity monitoring earn their place.

It does not help if your markets are genuinely global. A system that must send anywhere has to fall back on the weaker controls, and should expect to spend more on monitoring as a result.

And it is not a substitute for a spend cap. The allowlist bounds where money can go; it says nothing about how much. Those are separate questions and they want separate answers — a budget that hard-blocks a runaway campaign while never blocking a login is the other half, and it gets its own post.

Check your own endpoints

Three questions, in order of how much they will tell you.

First: can your send-code endpoint deliver to a country you do not sell in? Try it against a number outside your markets in a non-production environment. If the message goes, you have no allowlist.

Second: what happens with the setting removed entirely? Blank the config value and try again. A send that still succeeds means the implementation fails open, which is the failure mode that actually bites — the list is rarely wrong on purpose, it is empty by accident.

Third: how many entry points reach your sender? Grep for it and compare against where the guard lives:

grep -rn "sendTemplate|sendOtp|sendMessage" app/ Modules/ --include="*.php" | grep -v "Tests|/Messaging/"

Every result is a path that must pass the check. If the guard is in a controller and that list is longer than one, it is already being bypassed.

The funnel described here — allowlist, then budget, then a timeout-bounded call, then the audit log and cost ledger — is packaged as laravel-whatsapp-cost-control, MIT licensed, for Laravel 12 and 13. The allowlist ships fail-closed: it will refuse to send anywhere until you configure the countries you actually serve, which is a deliberately annoying five minutes.

Originally published at dineshstack.com — read the full version with code samples and updates there.

My CV Got Me UAE Tech Jobs Through Indeed for Three Years. Then It Stopped.

Dinesh Wijethunga — Thu, 13 Aug 2026 01:00:07 +0000

TL;DR: My CV brought in multiple UAE offers between 2022 and the end of 2024, every one of them through Indeed and none through LinkedIn. Then it stopped. It did not stop because it got worse or because the market collapsed. It stopped because I changed — seniority, specialism, the kind of role I was going for — and the document did not. And LinkedIn never worked at all, for a separate reason it took me years to see: I had a profile the whole time, but I was never findable.

This is the first of three posts. This one is what happened and why. The second is the mechanic underneath it — why job boards and LinkedIn are not two versions of the same thing. The third is the part nobody selling CV templates will tell you.

What actually happened

I moved from Sri Lanka to a tech job in the UAE. I am now a Tech Lead in Abu Dhabi with more than ten years of experience.

The window this post is about — 2022 to late 2024 — is entirely after that move. I was living and working in the UAE for all of it. That matters, because it removes the easiest explanation before we start.

Across those three years the same CV produced multiple UAE offers. Every single one came through Indeed. Not one came through LinkedIn, despite my having a LinkedIn profile the entire time.

I have friends here who get roles through LinkedIn and genuinely cannot explain how. Ask them what they did and the answer is some version of "a recruiter messaged me." That is not modesty. It is the whole point, and it took me a long time to understand why.

After December 2024 the same CV produced nothing.

Three reasons, and the first one is the least flattering

1. I stopped applying

I got a job in December 2024. My application volume collapsed. Some meaningful portion of "my CV stopped working" is simply that I stopped sending it.

I am putting this first because it is the one I would most like to skip. Every post in this genre blames the market, and blaming the market is comfortable — it makes the failure external and the solution purchasable. Before I tell you anything about market conditions, I want to be clear that a chunk of my own data is just reduced input. If you are drawing conclusions from your own search, run this check first. Fewer replies from a tenth as many applications is not a signal about your CV.

2. I crossed the seniority line

At mid-level, job boards work. The roles are posted, the volume is high, and the process is designed to filter a large inbound pile.

Lead and senior roles are mostly not filled that way. They go through networks and recruiter outreach. By 2025 I was applying for roles that largely are not advertised on the boards I was searching — and the ones that are tend to have already been filled through other routes by the time they appear.

The channel did not break. It aged out of my career stage.

3. The market tightened — but less than it feels, and not evenly

It is harder, and there is real data on it. LinkedIn research reported by Gulf News found that 65% of UAE professionals say finding a role has become harder over the past twelve months, while 72% plan to look for a new job anyway.

But look at the reason they gave. 63% named an overcrowded candidate pool as the biggest obstacle — not a shortage of roles. That is a different problem with a different fix. If jobs had vanished, nothing about your CV would matter. If you are one of far more applicants for the same jobs, then standing out is the entire game, and a document that makes you look like everyone else is an active liability rather than a neutral one.

Demand in tech also did not fall so much as move. PwC's 2026 Global AI Jobs Barometer puts UAE job postings requiring AI skills at 1.0% in 2021 and 3.2% in 2025 — roughly 4,600 adverts rising to 12,200, moving the UAE from 21st to 13th globally in four years. Those roles pay a premium of up to 92% in financial services and around 50% in technology, media and telecoms.

Read that last number again, because it is the whole argument for specialising. The market is not paying more for people who can do everything. It is paying up to 92% more for people who can demonstrably do one thing that is currently scarce.

My CV led with "full stack" and listed more than twenty technologies. In 2022 that read as range. By 2025 it reads as someone who has not decided what they are — in a market where the premium goes to people who have.

The part I did not see for two years

Those three causes look independent. They are not. Look at what changed between 2022 and now:

	2022	Late 2024 onward
Seniority	Mid-level	Tech Lead, 10+ years
Roles I was going for	Developer	Lead and senior
Positioning	Generalist	Production AI and LLM work
City	Dubai	Abu Dhabi
What my CV said	~8 years, full stack, 20+ technologies	~8 years, full stack, 20+ technologies

Read the last row again. Everything above it changed. That row did not.

That is the actual failure, and it is not a writing problem. The CV was a snapshot of the person who last needed one. Almost nobody updates a CV except while job hunting, which means everyone's CV describes the last version of them that was looking — and the gap gets wider the longer the job goes well.

Mine understated me in three separate ways at once. It said developer where the answer was Tech Lead. It said eight years where the answer was more than ten. And it said generalist where the honest answer had become a specialism in production AI systems — the kind of work I now write about in detail, like the bilingual Arabic and English voice assistant we ran at roughly 40ms latency.

The city row is a small one, but it is the same failure in a different artifact. I moved from Dubai to Abu Dhabi when I took the December 2024 job. Recruiter search filters by city. If a profile still says the city you left, you are absent from searches for the city you are actually in — and, like the CV, nothing tells you.

Why that last one matters more than it looks

There is a mechanic under all of this that took me far too long to work out, and it explains my friends.

Indeed and LinkedIn are not two places to find the same jobs. They run in opposite directions.

	Indeed	LinkedIn
Direction	You apply outward	Recruiters search inward
What you control	How many applications you send	Whether you are findable at all
How it fails	You get rejected	You are never seen

Recruiters do not browse LinkedIn the way you do. They use LinkedIn Recruiter, which is a search tool with dozens of filters running against a very large index of profiles. You either surface in the result set or you functionally do not exist for that search. There is no rejection, because there was never an application.

That is why my friends cannot explain what they did. They are not doing anything. They are being found. You cannot apply your way into an inbound channel, and no amount of application volume substitutes for being in the result set.

Now put my own three years against that. I was in the UAE the entire time. Both channels were open to me, the whole way through. Indeed produced offers for three years. LinkedIn produced nothing, ever — not fewer results, none.

Same person, same city, same experience, same week. One channel worked and the other never did once. That is not a market story and it is not bad luck. I was doing the work for one channel and none of the work for the other. On Indeed I was sending applications, which is the entire job on Indeed. On LinkedIn I had a profile and assumed that was participation. It is not. A profile is not a fishing line in the water; it is a page that either matches a recruiter's search or does not.

Mine did not. My headline said "full stack developer" — one of the most crowded search terms in the market, where I was competing with thousands of profiles saying exactly the same thing, and offering a recruiter no reason to pick mine out. I had never turned on the recruiter-facing "open to work" setting. My skills list was an afterthought. Every one of those is a filter I was failing without ever seeing a result.

On the outbound channel, effort looks like applications. On the inbound channel, effort looks like being findable. Doing a lot of the first has never once produced the second.

So there were two separate failures running at the same time, and I had been reading them as one. The CV went stale, which cost me the outbound channel as I moved up into roles that boards do not carry. And the profile was never findable, which meant the inbound channel — the one that actually serves senior roles — had never been switched on at all.

One more filter worth naming, because it does not apply to me but will apply to a lot of people reading this: those search filters include location. A recruiter hiring in Dubai searches Dubai. If you are still in Colombo or Chennai or Karachi, you are excluded before a word of your headline is read — which is why applying outward through job boards is the realistic route until you have arrived. I will cover the one setting that partly gets around it in the next post.

What this means if you are in the middle of it

Before you rewrite anything, work out which channel you are actually in, because the advice inverts.

If you are still outside the UAE, outbound is your game, and that is fine — it is the channel that got me here too. Volume matters. Your CV is doing the heavy lifting because it is the only artifact in the process. State your visa position explicitly rather than leaving a recruiter to assume the expensive answer, and say that you can relocate. Most LinkedIn optimisation will not reach you yet, for the location reason above, with one exception I will cover next.

If you are already here and applying, as I was for three years: you are running one channel out of two. That can work for a long time, exactly as it worked for me — right up until the roles you want stop being posted on it. Then it stops, and because there is no rejection to read, it feels like the market turned rather than like a channel you never opened.

If you are already here and senior, volume is not your problem — visibility is. The CV matters less than you think, because it is what you send after being found, and being found is a different skill with different levers.

And for everyone: open your CV and check the date on the claims, not the formatting. Does it say your current title? Your current years? The thing you are actually good at now, or the thing you were good at when you last needed a job? A stale CV fails quietly. There is no bounce, no rejection email, no signal at all — which is exactly why it can keep failing for a year without you noticing.

Check yours before you rewrite it

I built a free checker for this, because I could not find one that understood this market. It scores your CV against what Gulf recruiters actually filter on — visa status, notice period, formatting that survives an applicant tracking system, and whether you carry the keywords for the role you are targeting.

It runs entirely in your browser. Your CV is not uploaded anywhere, there is no signup, and the score and every fix are free.

Check your CV against the UAE market

One honest limit, the same one I put on the tool itself: this tells you whether your CV survives the filter. It cannot tell you whether the right roles are open. If nothing posted matches your experience, a perfect CV will not create one.

Next in this series: why you cannot apply your way into LinkedIn, what actually decides whether a recruiter's search returns you, and the one findability lever that works even before you have arrived.