Engineering Resilience: Two Lessons from Building Under Pressure

Afeh — Sat, 13 Jun 2026 09:10:02 +0000

A reflection on performance optimization at scale and building reliability mechanisms; two tasks that defined my internship.

Every engineering internship has its share of "aha" moments; those late-night debugging sessions where a breakthrough finally clicks, or the PR that takes seven commits to get right. As I wrap up my time as an Intern with HNG, I want to write about two tasks that stuck with me. Not because they were the hardest, but because they taught me something real about building systems that have to work.

One was individual, optimizing a demographic intelligence API to handle millions of records with sub-second query times. The other was a team effort; building reliability mechanisms into an AI-powered interview platform so that when things break (and they will), the system degrades gracefully instead of falling apart.

Lets walk through both.

Part I: Working on Insighta (Individual Task — Stage 4B)

What it was

Insighta IQ is a demographic intelligence API; think "find me Nigerian females aged 20 to 45." I nicknamed it Stereo API(Stereo short for stereotyping of course). Users query a PostgreSQL database of millions of demographic profiles through a FastAPI backend, via both CLI and web clients.

Stage 4 asked us to make it perform under serious assumptions:

Data growth: from millions toward tens of millions of profiles
Traffic: hundreds to low thousands of queries per minute, multiple teams using it daily
Performance targets: P50 latency under 500ms, P95 under 2 seconds

The task had three parts: query optimization, query normalization, and large-scale CSV ingestion. The CSV ingestion piece was the one that kept me up at night.

The problem

Users needed to upload CSV files with up to 500,000 rows of profile data. These weren't trivial constraints:

You cannot insert rows one by one (500k individual inserts = death by a thousand round trips)
You cannot load the entire file into memory (500k rows of profile data can easily exceed available RAM)
Uploads must not block ongoing query traffic (the system can't go dark during ingestion)
The system must handle concurrent uploads
A single bad row must never fail the entire upload

On top of that, the database is hosted remotely, so every query — including every INSERT, incurs network latency. And we were already under read pressure from the query workload.

How I approached it

I broke the problem into layers:

Streaming, not loading: Read the file in 256KB chunks, never hold the whole thing in memory
Batch processing, not row-by-row: Validate rows, accumulate 10,000 valid ones, then bulk-insert
Dedicated thread pool: Run the CPU+DB-bound CSV work off the event loop so the API stays responsive
Off-loading the uploading to a background task: It is better to just return a success response to let the user know that the uploading is working in the background rather than waiting endlessly for the upload to get done.
Partial failure tolerance: Each batch commits independently; failures don't roll back valid work
Comprehensive skip reporting: Return a structured summary of exactly what went wrong and why

Here's what the ingestion pipeline looked like:

File upload → Chunked read (256KB) → Decode (UTF-8 → Latin-1 fallback)
  → CSV DictReader (streaming) → Validate row → Batch (10k rows)
    → Deduplicate in-batch → Single DB query for existing names
      → Bulk INSERT (ON CONFLICT DO NOTHING) → Report summary

What broke and how I fixed it

The first approach was naive. I read the entire CSV into memory, parsed every row, and then tried to insert everything at once. For a 100KB test file, it worked beautifully. For the 500,000-row file? Memory exploded.

Fix: I switched to streaming with csv.DictReader over a StringIO buffer. But I still needed to validate rows against each other (duplicate names within the same upload) and against the database. That meant holding state across batches of 10,000 rows.

The second problem was intra-batch deduplication. Without it, two rows with the same name in the same upload would both pass validation, only to conflict during insert. The fix was a global_seen: Set[str]; a set of all names already inserted in this upload session, passed between batch flushes.

The third problem was edge cases in CSV data I never anticipated:

Rows with extra commas producing wrong column counts
UTF-16 encoded files that failed UTF-8 decoding
Age values like "twenty-five" instead of "25"
Country codes in varying cases ("ng", "NG", "Ng")

Each edge case got its own validator function. The _parse_row function became a gauntlet of pure functions — no database calls, just deterministic validation. If a row failed any check, it was counted and skipped, but processing continued.

What I took away

Batch everything. Validate early. Never trust user input.

The batch size of 10,000 wasn't arbitrary. Below 5,000, the overhead of DB round trips dominated. Above 20,000, memory pressure started climbing without significant throughput gains. 10,000 was the sweet spot.

The pattern that emerged; stream, validate, batch, insert, report — is something I now see everywhere: ETL pipelines, message queues, log processors. It's a universal pattern for handling lots of data with limited resources.

The caching and query optimization work (indexes on gender, age, country_id, composite indexes, connection pooling, result caching with 5-minute TTL) brought the query side in line with our P50/P95 targets. But the CSV ingestion piece — that's what I remember most vividly, because it forced me to think about resource constraints, failure modes, and graceful degradation all at once.

Why I picked this

The CSV ingestion task wasn't just about writing code. It was about engineering for constraints: memory, concurrency, consistency, partial failure. Every decision (batch size, decoding fallback strategy, ON CONFLICT DO NOTHING vs pre-query) was a trade-off. I had to justify each one. That process of articulating why a decision is right, not just that it works is what I think separates engineering from coding.

Part II: Building Reliability Into an AI Interview Platform (Team Task)

What it was

MeetMind is an AI-powered interview platform. Candidates join live audio interviews with an AI interviewer, and the system records transcripts, generates assessments, sends emails, and provides a chat interface for recruiters to query interview data. I really had a lot of fun building this with my teammates (though I nearly lost my mind a couple of times).

The catch? It depends on several third-party APIs:

Gemini (Google) for generating interview questions, assessments, and extracting resume information
Resend for transactional email delivery (welcome emails, password resets, interview invites)
LiveKit for real-time audio/video sessions

Any of these could fail at any moment from network blips, rate limits, service outages, or transient errors. Before this task, a failed API call meant a 500 error and a frustrated user.

My PR to address that issue added two reliability mechanisms:

Retry with exponential backoff — a centralized retry_async utility that wraps any async function with automatic retries, logging, and backoff
Transcript fallback — when individual transcript turns are missing in the database, fall back to the session's stored transcript JSON

The problem

Two concrete failure scenarios:

Scenario A: A candidate finishes an interview, and the system fires off a background task to generate an assessment summary via Gemini. Halfway through, Gemini returns a 429 (rate limit). The assessment fails. The summary is stuck in "generating" status. The recruiter sees a blank report.

Scenario B: Real-time interview transcript turns are stored one by one as the AI interviewer and candidate speak. If the persistence pipeline drops a few turns — or fails entirely — the transcript is incomplete. The recruiter can't review the interview. The AI can't generate an assessment.

Both scenarios had the same root cause: no mechanism for transient failure recovery.

How I approached it

For the retry mechanism, I wanted something that:

Works with any async function (generic, reusable)
Uses exponential backoff (don't hammer the failing service)
Logs warnings on intermediate failures, errors on exhaustion
Accepts configurable retry count, delay, backoff factor, and exception types
Integrates with existing code without massive refactoring

The signature was clean:

async def retry_async(
    func: Callable[..., T],
    *args,
    max_retries: int = 3,
    initial_delay: float = 2.0,
    backoff_factor: float = 2.0,
    exceptions: tuple[type[BaseException], ...] = (Exception,),
    task_name: str = "Task",
    **kwargs,
) -> T:

For the transcript fallback, the pattern was: try the primary data source first, and if unavailable, reconstruct from the session's transcript_json field. The fallback had to be transparent to callers — get_chat_history, get_transcript, and get_transcript_export all work the same way regardless of which data source backs them.

What broke and how I fixed it

The first bug was an off-by-one in the backoff calculation. I had delay *= backoff_factor happening before the sleep, so the first retry was 2x the initial delay instead of the initial delay itself. It felt trivial, but it meant retries took longer than necessary — 2s, 4s, 8s instead of 1s, 2s, 4s.

Fix: Moved the delay *= backoff_factor to after the sleep.

The second issue was exception type granularity. Initially, the retry caught all Exception subclasses. But some exceptions shouldn't be retried — like ValueError from bad user input, or KeyError from a missing dictionary key. A retry won't fix a programming error.

Fix: Made the exceptions tuple a parameter so callers can specify which exceptions are retryable. For Gemini calls, we retry (Exception,) since the API can fail for many transient reasons. For email delivery, we retry the Resend-specific exception.

The third issue was the transcript fallback format mismatch. The session's transcript_json stored timestamps as Unix seconds, but the transcript endpoints expected them formatted as "HH:MM:SS" elapsed time. The fallback function needed to reproduce the same relative timestamp calculation that the primary path used.

Fix: Created _format_elapsed_timestamp as a shared utility and used it in both the primary and fallback paths. The fallback also had to generate deterministic UUIDs for each fallback turn (using uuid.uuid5 with a namespace) since there were no database IDs available.

What I took away

Graceful degradation > perfect failure. The goal isn't to never fail — it's to fail in a way that doesn't cascade into user-facing errors. The retry mechanism handles transient failures silently. The fallback mechanism ensures that even if the real-time pipeline drops data, users can still review interview transcripts. The user never needs to know something went wrong.

I also learned that reliability is visible in the logs. After the retry mechanism was deployed, we stopped seeing "Assessment generation failed" errors in Sentry. Instead, we saw "Attempt 1/3 failed for Generate assessment... Retrying in 2.00s..." — which is an entirely different class of log. It means the system handled the failure rather than succumbing to it.

Why I picked this

This task taught me that reliability isn't a feature you add at the end — it's a design philosophy that affects how you structure every external dependency call. The retry_async utility now wraps every Gemini generation, every email send, every document embedding. It's invisible infrastructure. But it's the difference between a system that occasionally returns 500s and one that absorbs transient failures and moves on. It is important to build Fault-tolerant systems.

And the transcript fallback taught me something subtler: data can come from multiple sources, and that's okay. The real-time turns are the ideal data source. But the session JSON is always there as a safety net. Engineering for multiple data paths — with clear fallback semantics — makes the system more robust without adding complexity to the API surface.

Bringing It Together

Looking back, both tasks taught me the same lesson from different angles:

Design for failure, optimize for reality.

The CSV ingestion system assumes every row could be bad and handles it gracefully without stopping. The retry mechanism assumes every API call could fail and recovers transparently. The transcript fallback assumes the primary data path could be incomplete and provides an alternative.

Systems that work under pressure aren't the ones that never break. They're the ones that break gracefully, recover quickly, and leave useful evidence behind when they can't.

That's what I'll carry forward from this internship.

Building a Dynamic Profile API with FastAPI: My HNG Stage 0 Experience

Afeh — Mon, 20 Oct 2025 13:16:54 +0000

During Stage 0 of the Backend Wizards program, I built a simple yet insightful REST API endpoint called /me using FastAPI. The goal was to return my personal profile information, fetch a random cat fact from an external API, include a dynamic UTC timestamp, and format everything neatly in JSON. While it looked straightforward, it pushed me to understand key backend concepts like API integration, error handling, and deployment. I structured the project using Python, FastAPI, and the requests library, managed environment variables with python-dotenv, and deployed it seamlessly on Railway. I learned to anticipate failures from external APIs by adding try-except blocks, format timestamps in ISO 8601, and configure environment variables properly for cloud deployment. Seeing my live endpoint— https://introproj-production.up.railway.app/me
—work flawlessly was satisfying. More than just completing a task, this experience taught me how much discipline and attention to detail backend development truly demands, from clean code structure to reliable production deployment.

Doing Hard Things

Afeh — Tue, 02 Jul 2024 09:28:46 +0000

"What doesn't kill you makes you stronger" - A strong man(probably).

On the 9th of March, 2024, I woke up around 6 am. It was a Saturday so it was unusual for me to be up that early. I said my prayers, picked up my phone and opened WhatsApp. Scrolling through unread messages, a post in my school's Google Developer Student Club group caught my eye. "Django Developer Needed. DM." Interesting I thought. After contemplating for some minutes and going through some moments of self-doubt, I opened the group and messaged the person who posted it.

There was a catch. He said he was in need of a Django Developer with workable knowledge of Bootstrap. Bruhhh I thought. I was a decent Django Developer, but front-end stuff like Bootstrap and CSS? Not my thing. I had a little chat with God on it and I felt cool about it. Taking a deep breath, I told the guy I'd take the gig. We discussed my charge and deadline- I had 24 hours to come up with a working Application. I said "No p", trying to sound confident(even though this was going to be my first paid gig, ever).

There was no power in my hostel, so I packed my Laptop, my charger, Rubik's Cube(for when my brain stops cooperating) and notepad into my bag, and had my bath, drank tea, and headed to the co-working space on campus. While walking, I brainstormed about how the structure of the App would be- team management features, am I using Class Based Views or Function Based Views?, stuff like that. Half-excitement, half-terror tightened my chest. Had I bitten more than I can chew??

I got to the co-working space, paid the fee, took a seat. I booted my Laptop, opened VSCode and just stared. Instead of the usual thrill of building something new, a wave of imposter syndrome washed over me. Omoo can I actually do this thing? I closed my eyes, said a quick prayer and started working. I had a deadline to beat.

The next 15 hours were a blur of coding, troubleshooting, and frantic calls to friends for help and moral support. The only breaks I took were to solve my Cube and eat. I had never worked that long in my short coding life before and the exhilarating feeling I had when I made my final push to GitHub might just be similar to what Neil Armstrong felt when he took that first step on the moon. One small step for Afebu and a giant leap for his coding career lol

"I can do all things..." Paul (f.k.a Saul) from Tarsus

Last month, I saw a post on X about HNG Internship 11. I had heard stories about how difficult, grueling and rigorous the whole process is. But then I remembered that 24-hour coding marathon. "Well, well, well," I thought, a smile creeping across my face. "Hello hard things, I guess we meet again."

If you are interested in doing hard things too click here to register for the HNG Internship.

I am Afebu Victor Balogun. This is my story, and I am sticking to it.

DEV Community: Afeh

Engineering Resilience: Two Lessons from Building Under Pressure

Part I: Working on Insighta (Individual Task — Stage 4B)

What it was

The problem

How I approached it

What broke and how I fixed it

What I took away

Why I picked this

Part II: Building Reliability Into an AI Interview Platform (Team Task)

What it was

The problem

How I approached it

What broke and how I fixed it

What I took away

Why I picked this

Bringing It Together

Building a Dynamic Profile API with FastAPI: My HNG Stage 0 Experience

Doing Hard Things

"What doesn't kill you makes you stronger" - A strong man(probably).