DEV Community: Odinaka Robert Nnamani

Debugging a distributed mystery

Odinaka Robert Nnamani — Tue, 27 Oct 2026 12:02:04 +0000

In a monolith, a bug lives in one place. You open the logs, you find it.

Microservices change the problem. A single user action can cross 5+ services — and the failure can sit in any one of them, or in the gap between two of them.

Take a signup:

Auth service → User service → Notification service → Email provider

Say it fails halfway. The user exists, but the welcome never arrives. Or the account never gets created, but the notification already went out. Where do you look?

Without centralised logging and tracing, you open four dashboards, four log files, four different formats — and you guess. Each team can honestly say "it worked on my side," because each side only saw its own hop.

The fix is not more logs. It is one thread through all of them:

One request ID, carried across every hop, from the first request to the last.
Centralised logs, searchable by that ID — so one search shows the whole journey, not one service's slice of it.
Tracing that shows the order and timing of each hop — so you can see where the chain broke, not just that it broke.

With that, debugging stops being a mystery. You follow the request the way the user experienced it: Auth, then User, then Notification, then the provider — and the hop where the thread goes quiet is where you look.

The rule I keep: no service starts work without the request ID. A log line you cannot tie back to a request is a dead end, and a system full of dead ends is a system you cannot debug.

Microservices do not make bugs harder to write. They make evidence harder to follow. Centralise the evidence, and the mystery goes away.

Next in this series: 7 backend lessons I learned the hard way. I write about backend systems that look small until they aren't — follow along if that's your world; the longer pieces live on my Substack.

All DB access lives in one place

Odinaka Robert Nnamani — Mon, 26 Oct 2026 12:40:54 +0000

One mistake I see in backend codebases:

ORM queries scattered everywhere.

Controllers query the DB. Services query the DB. Helpers query the DB. The little cron job queries the DB. Change one column and you're grep-ing the whole repo, praying you found every query that touches it.

Soon your database logic is impossible to track — and nobody can tell you where a value actually gets read or written.

A better pattern: the repository layer.

All DB access lives in one place. Controllers ask the repository; they never touch the ORM. Schema knowledge — joins, filters, transaction boundaries — lives in one module per entity. The rest of the codebase just calls methods with names that say what they mean: findActiveUsers(), not db.select().where(...).

Three things this buys you:

Cleaner code — business logic stops drowning in query syntax.
Safer schema changes — rename a column in one file, not seventeen.
A clean test seam — the boundary you need to mock is obvious.

The rule I keep: no raw ORM calls outside the repository. No exceptions for "it's just a small helper." Break the rule once and it's no longer a rule.

All DB access lives in one place. That's the whole point.

Next in this series: debugging a distributed mystery. I write about backend systems that look small until they aren't — follow along if that's your world; the longer pieces live on my Substack.

Code evolves quickly, databases remember everything

Odinaka Robert Nnamani — Sun, 25 Oct 2026 12:01:19 +0000

A common backend mistake: treating database schemas like temporary code.

Code deploys in minutes. You ship it, watch it, roll it back if it's wrong. A schema doesn't work like that. The moment production data exists, the schema becomes the memory of the system — and memory doesn't roll back.

Every shortcut compounds:

The column you added as nullable "for now" still has three meanings two years later
The status value you retired still sits in a million old rows
The constraint you skipped in the app gets skipped by the second service too

So schema design has to consider what code design can postpone: relationships, constraints, future queries, data integrity. Put the constraint in the database, not just the validation layer — applications change, get rewritten, multiply. The database is the last writer standing, and the only one that sees every row.

And migrations on live tables are production events, not chores: expand, backfill, dual-write, contract — in that order, over days, not in one deploy.

Code evolves quickly. Databases remember everything. Design like it.

Next in this series: all DB access lives in one place. I write about backend systems that look small until they aren't — follow along if that's your world; the longer pieces live on my Substack.

One action, multiple reactions

Odinaka Robert Nnamani — Sat, 24 Oct 2026 12:01:43 +0000

Building event-driven systems changed how I think about software.

Services shouldn't call each other constantly. They should react to events.

Example:

User signs up → Auth emits an event
→ Notification service sends the welcome email
→ Analytics service records the signup
→ User service continues onboarding

One action. Multiple reactions.

The biggest shift is mental: you stop designing request chains and start designing system reactions. The signup service doesn't need to know who cares that a user signed up — it publishes the fact, and the system reacts.

The honest cost: debugging gets harder. A request chain fails in one place, with a stack trace. An event fans out, and the failure can hide inside any of the reactions — so observability stops being optional, and patterns like the transactional outbox (write the event in the same transaction as the state change, so the two can never disagree) stop being nice-to-haves.

I run an email platform built this way: a send is one action, and delivery, bounces, complaints and suppression are all reactions to provider events. It's the only shape I've found that scales without the services strangling each other.

Next in this series: code evolves quickly, databases remember everything. I write about backend systems that look small until they aren't — follow along if that's your world; the longer pieces live on my Substack.

Auth systems aren't features, they're mini platforms

Odinaka Robert Nnamani — Fri, 23 Oct 2026 12:02:24 +0000

Authentication systems are more complex than most apps expect.

Signup alone involves:

Email verification
OTP storage
Redis for temporary tokens
Event emission for notifications
Audit logging
User service onboarding

And that's just creating an account.

Auth systems aren't features. They're mini platforms.

Login is the smallest part of one. Underneath it sits identity proofing, session lifecycle, token storage, rate limiting on credential attempts, and an audit trail you can actually reconstruct an incident from — user agent, IP, every credential attempt, logged in a shape fraud detection can use later.

Most teams scope auth as a ticket. Then they rebuild it three times: once when the first fraud wave hits, once when a customer asks for the audit log, and once when sessions have to work across services.

I write about the backend systems that look small until they aren't — email infra, retries, identity. Follow along if that's your world; the longer pieces live on my Substack (link in bio).

One tenant's bad list is spending your SES reputation

Odinaka Robert Nnamani — Thu, 22 Oct 2026 12:00:08 +0000

One tenant's bad list is spending your SES reputation

If you run a multi-tenant SaaS on Amazon SES, you probably have one SES account, one or two verified identities, and every tenant's email flowing through them.

That means you also have exactly one sender reputation. And every tenant you onboard gets to spend it.

Here's the scenario that should worry you. A tenant imports a purchased list — or an old one, scraped from a CRM export nobody cleaned. They send a campaign. Hard bounces spike. A few recipients mark it as spam. SES watches two numbers across your whole account: bounce rate and complaint rate. Drift toward a 5% bounce rate or a 0.1% complaint rate and your account goes under review. Push further and SES pauses your sending.

Not that tenant's sending. Your sending. Every tenant. The good ones — the ones with clean lists and double opt-ins — wake up to the same dead deliverability as the one who caused it. Their password resets land in spam. Their invoices never arrive. And your dashboard, if it only tracks account-level numbers, tells you something is wrong long after all your tenants already know.

This isn't a monitoring problem you can alert your way out of. It's an architecture problem. Reputation on SES is pooled by default, so isolation has to be something you build — in three separate places.

1. Give each tenant its own sending identity

Reputation signals attach to identities: domains, MAIL FROM domains, IP pools. If all your tenants send from one domain on the shared IP pool, there is nothing for those signals to segment by.

What I build instead:

A sending subdomain per tenant — mail.tenant.yourdomain.com, or the tenant's own domain once they're serious — with its own SPF/DKIM alignment. A tenant who destroys their subdomain's standing has hurt their subdomain, not your root domain and not the tenant next door.
A configuration set per tenant. This is the unglamorous piece everything else hangs off. Configuration sets are how you tag a tenant's sends so every bounce, complaint, and delivery event comes back labelled with which tenant produced it. One shared configuration set means unattributed events, and unattributed events mean you're back to guessing.
Dedicated IP pools only for tenants who earn them. SES lets you assign configuration sets to dedicated IP pools. A tenant with steady, real volume can graduate to their own pool, fully out of the shared blast radius. Tenants without consistent volume stay on shared IPs — a cold dedicated IP with sporadic sends is its own deliverability problem.

2. Keep a per-tenant ledger, not an account-level blob

SES hands you an account-level suppression list and a stream of events. Most teams wire the events to an SNS topic, glance at a dashboard, and move on.

Split the stream by tenant (that's what the per-tenant configuration sets are for) and persist it as a ledger: every send, bounce, complaint, and suppression, recorded per tenant, append-only. Two things fall out of this:

Suppression becomes per-tenant too. An address that hard-bounced for Tenant A gets suppressed for Tenant A. On a shared account-level list, one tenant's typo'd import can silently block addresses your other tenants legitimately need to reach.
You get a reputation balance per tenant. Bounce rate, complaint rate, suppression growth — per tenant, over time. Not "the account looks bad this week," but "this tenant has been decaying for six days." That's the difference between an incident and a conversation.

3. Gate sending on per-tenant outcomes

Isolation without enforcement is just better observability of your own outage. The point of measuring per tenant is that you can act per tenant.

The pattern I use — and the one I keep coming back to when I talk to other builders about this — is an outcome gate per tenant:

Thresholds per tenant. When a tenant's bounce or complaint rate crosses your line, the gate pauses that tenant's sending automatically. The account stays healthy, the other tenants never notice, and the paused tenant gets a specific reason instead of your whole platform getting an SES review.
A ramp for new tenants. Nobody sends at full volume on day one. New tenants start on a paced schedule with a daily ceiling that rises as their outcome history proves out. A bad list then burns a small, contained allowance — not your account.
List hygiene before the first campaign, not after. The gate is the last line of defence. Checking an import for the obvious rot before it's allowed to send is the first one.

I sketched this gate design out with another builder in a GitHub thread recently — he'd been running deliverability as one global dashboard and treating every incident as platform-wide. He folded the per-tenant gate into his product's roadmap the same week and told me it filled a real gap in his design. It usually does. Most multi-tenant stacks built on SES have exactly this hole; they just haven't had the tenant yet who finds it.

What to check in your stack today

You don't need a rebuild to find out how exposed you are. Four questions:

If one tenant spiked your bounce rate tonight, could you name them from your own data tomorrow morning — or only that "the account" had a bad day?
Do your bounce and complaint events arrive tagged by tenant, or in one undifferentiated stream?
Is suppression enforced per tenant, or is one tenant's bad import blocking addresses for everyone?
What, mechanically, stops a tenant from sending their worst list at full volume right now? If the answer is "we'd notice," you don't have a control — you have a hope.

If you can't answer those cleanly, your tenants are sharing more than an SES account. They're sharing a reputation, a suppression list, and an outage waiting for its trigger.

I'm building these three pieces — per-tenant identities, per-tenant ledgers, per-tenant outcome gates — into rootmail, an email infrastructure layer on SES, and the beta is open if you want to see the approach in practice: https://rootmail.io/beta

Your microservices aren’t decoupled

Odinaka Robert Nnamani — Wed, 21 Oct 2026 12:42:43 +0000

Splitting a monolith into services gives you the vocabulary for free: independent, decoupled, autonomous. The architecture has to earn those words — and most don’t.

There’s a fast way to tell whether yours did. Don’t look at the deployment diagram. Look at where the contracts live.

The setup almost everyone starts with

In a gRPC system, the contracts are the .proto files: the message shapes and service definitions every service agrees to speak. They have to live somewhere, and somewhere usually means "inside whichever service was built first."

So the orders service owns the protos. The users service needs them, so it imports them straight from the orders repo. Payments needs the same messages, and rather than set up the import, someone copy-pastes the files and moves on.

orders-service/          ← .proto files live here
├── users-service        imports them from orders
└── payments-service     copy-pasted them

It works. It ships. And it quietly wires every service to every other service.

The hidden coupling

Coordinated deployments. The moment a contract lives inside one service, changing it stops being one team’s change. Rename a field, add a required field, and every service importing that proto has to move with you. Your "independent" services now deploy in lockstep — the exact thing microservices were supposed to end.

Breaking change chaos. A breaking change in the proto doesn’t break one build. It breaks every consumer, including the ones you forgot existed. The blast radius of a contract change becomes the whole system.

Version mismatch bugs. The copy-paste in payments drifts. It always drifts. Now two services disagree about the contract while both believe they’re current — and the mismatch doesn’t show up in CI, because each service compiles fine against its own copy. It shows up in production, as garbled payloads and errors nobody can reproduce locally.

Tight inter-service dependency. Underneath all of it: services are depending on each other’s code. Not on an interface — on an implementation, owned by another team, on another team’s release schedule.

This is not independence. It’s a monolith that pays network latency for its coupling.

The fix: contracts are a product, not a folder

Contracts get their own repo.

users-service    ──┐
orders-service   ──┼──►  contracts repo (versioned, published)
payments-service ──┘

Version it. The contracts repo releases like a library: v1.4.2, with breaking changes forcing a new major version instead of ambushing consumers.

Publish it. Each service pulls the contracts as a versioned package — the same way it pulls any other dependency.

Consume it. Every service pins the version it was built against and upgrades deliberately, on its own schedule, after its own tests pass. A team that needs the new field upgrades this sprint; a team that doesn’t stays on the old version until it’s ready.

And the hard rule that makes it real: no service imports another service’s code. Not the protos, not a shared "utils" package that quietly becomes a coupling channel. Services meet at the contract, nowhere else.

Why it works

API-first architecture. The contract becomes the designed artifact — reviewed, versioned, and intentional — instead of a byproduct of whichever service happened to be written first.
Independent deployments. A contract change ships as a new version, not a coordinated release. Teams deploy when they’re ready.
Controlled breaking changes. Breaking changes still happen; now they’re explicit, versioned, and adopted consumer by consumer instead of detonating everywhere at once.
Strong service boundaries. When the only shared thing is a published contract, the boundary between services is real — enforced by the dependency graph, not by convention and good intentions.

The principle

Services depend on abstractions. Not implementations.

It’s an old rule, and gRPC doesn’t exempt you from it — it just changes what the abstraction is. In a gRPC system, your .proto files are the abstraction. Everything else follows from where you put them.

Run the audit

Three questions, answerable in five minutes:

Does one service "own" your proto files?
Do services import code from each other?
Are your contracts versioned independently of any single service?

Three yeses to the wrong questions, and the diagnosis writes itself: the services are coupled at the contract layer, and no amount of separate repos, pipelines, or deploy targets changes that.

Decoupling was never a property of how many services you run. It’s a property of what they’re allowed to depend on.

If your contracts aren’t independent, your services aren’t either.

I build email infrastructure (rootmail, in beta) and write about backend systems every week. More here: https://robertpaschal.substack.com

My site is done. It's not a portfolio — it's a menu.

Odinaka Robert Nnamani — Wed, 21 Oct 2026 09:16:15 +0000

I finished building heypascal.me this week, and I made a deliberate choice: it's not a portfolio. It's a menu.

Eight ways to work with me, from a $100 retry-safety audit to a full MVP backend: backend APIs, AWS & deployment, document workflows, retry-safe systems, architecture reviews, MVP backends, technical writing, API docs.

Everything is scoped and priced up front. I've been on the buying side of the "let's hop on a discovery call" maze enough times to know nobody enjoys it — so I killed it. You see the offering, the scope, the price. Then you decide.

The retry-safety audit is the one I'm most proud of, honestly. Idempotency keys, duplicate-delivery handling, outbox patterns — the unglamorous stuff that decides whether your API survives its first traffic spike. Most teams don't know they need it until something double-charges a customer.

If you've got something to build: https://heypascal.me

Every service sends emails — stop that

Odinaka Robert Nnamani — Tue, 20 Oct 2026 12:39:04 +0000

A mistake I see in many microservice systems:

Every service sends emails.

Auth sends emails. Billing sends emails. User service sends emails.

Soon you have 5 different email implementations, 5 retry strategies, 5 ways to get rate-limited.

The fix: one notification service. Every service emits events — the notification service owns delivery, retries, rate limits, and user preferences.

Notifications are deceptively complex. Once you add retries, rate limits, preferences, and multiple channels, it deserves its own service.

Your SES failover will never fire — and that's the point

Odinaka Robert Nnamani — Mon, 19 Oct 2026 12:00:00 +0000

A practitioner running 50k emails/day on SES told me his Postmark failover only triggers on SES errors. Sounds reasonable. Except a suppressed address never errors — SES accepts it, returns 200, and quietly drops it. The backup never fires.

This is the trap: failovers are built around API errors, but deliverability dies in the silent middle — suppressions, throttled sends, greylisting. Your fallback logic watches the wrong signal.

What actually works: treat the suppression list as a first-class delivery signal. Monitor bounces and complaints as events, not just error codes. If your failover can't see a suppressed address, it's not a failover — it's a decoration.

I gave an AI my job hunt. Here's the system that kept it honest.

Odinaka Robert Nnamani — Sat, 17 Oct 2026 06:54:51 +0000

Yesterday I applied to 12 engineering jobs. I didn't open a single application form.

My AI chief of staff did it — research, verification, form-filling, submission, audit trail. I woke up to 12 submitted applications, 8 dead listings caught before I wasted a click, and 3 forms flagged as broken instead of faked.

This isn't a "10x your job hunt with ChatGPT" post. It's about the verification layer that makes delegating to an AI actually work — because an AI that applies to jobs wrong is worse than no AI at all.

The architecture

The system has four stages, and the important thing is what happens between them:

Research — find candidate roles across job boards, company career pages, LinkedIn. Rank by hire probability, not salary.
Live verification — open every listing and confirm it's real: no expired banner, an actual Apply control, remote terms that include Nigeria. Yesterday: 8 of 20 listings were dead on arrival. Stale aggregators are the norm, not the exception.
Application — fill forms with verified facts only. Resume, GitHub, portfolio attached. Every answer grounded in something checkable.
Audit — every submission logged in a tracker with the confirmation (or the honest lack of one), timestamped, with caveats noted.

The honesty constraint

Here's the part that surprised me. Asked about production LLM-API experience, the AI wrote: "No production LLM-API experience." Asked about shipped autonomous agents: "None." Asked about production Go: "No production Go."

It disclosed what I haven't done instead of inventing it. That's not a limitation of the system — it IS the system. A fabricated application that gets you an interview you can't survive is negative value. An honest application to the right role is the whole game.

The rule I gave it: only those you record are true. Every claim in every application had to trace back to a verified fact — my resume, my GitHub, my site. No invented years of experience, no inflated metrics, no phantom credentials.

The failure modes (all real, all yesterday)

Broken submit handlers. Three application forms (all on the same company's careers site) filled completely, passed validation, then silently refused to submit — the Send button dispatched and nothing happened. The system stopped, flagged them for manual action, and logged them as Drafting. It did not fake a confirmation.
Dead listings. 8 of 20 from aggregators were expired, closed, or region-locked. Caught at verification, never shown to me as options.
The honesty tax. Disclosing "no production LLM experience" on an AI Engineer application probably tanks that application. Good. I'd rather lose honestly than win fraudulently and get exposed in a technical screen.

What I'd do differently

The verification stage needs to run before research finishes ranking — I burned research effort on roles that died at verification. And the broken-form problem needs a human-in-the-loop escape hatch earlier: three fully-filled forms sitting in Drafting is a queue, not a result.

The uncomfortable takeaway

The bottleneck was never my ability. It was my bandwidth. Twenty applications in a day is a data-entry problem, and data entry is exactly what machines are for — as long as the verification layer is ruthless.

The engineers who thrive with AI won't be the ones who prompt hardest. They'll be the ones who build verification into the loop.

I'm Odinaka Nnamani — backend engineer (TypeScript, Node.js, PostgreSQL, AWS) and founder of Rootmail, email infrastructure that tells you what happened to every email. Open to remote backend/infrastructure roles: heypascal.me

I launched heypascal.me — here's what I built, what broke, and what I'd do differently

Odinaka Robert Nnamani — Fri, 16 Oct 2026 22:02:09 +0000

heypascal.me is live. Let me tell you the real story.

What I built: A personal site — portfolio, blog, and services page in one. Static HTML/CSS/JS on S3 + CloudFront, GitHub Actions deploys, Lucide icons.

What broke: I restarted three times. v1 was a portfolio template mindset — felt fake. v2 was blog-only — felt incomplete. The breakthrough was realizing the site had to be all three things I am: engineer, founder, writer.

Technical opinions baked in:

Content over chrome: no hero carousel, no stock photos
"Contact for quote" is a dark pattern: 8 services, all with starting prices ($80–$800)
Performance as a feature: fast, low-data friendly
Zero frameworks: sometimes hand-rolled is the right call

What I'd do differently: Start with the positioning question, not the tech. I burned weeks on pixel decisions for a site whose purpose I hadn't nailed.

What's next: The "Work with me" section is live with real offerings. If you're a founder who needs backend work, AWS setup, or technical writing — that's what it's for.

heypascal.me — would love your teardown.