Over the past two years, I've been a laravel developer. I've been so stuck with this framework that I felt like I didn't know much about actual programming. So, a few months ago, I decided to take a bold step. I started learning Javascript from scratch, and in the process, I heard about HNG. I felt like I needed the challenge, so I joined.
The HNG Internship is a remote, intensive program that throws you into real world engineering work from day one. No holding of hands, no toy projects. You're given tasks, a deadline, and a Slack channel full of people figuring things out alongside you. It's a sink or swim structure, and that's the point. The pressure is designed to surface what you're actually made of as an engineer, and not what you know, but how you think when things break.
I want to write about two tasks from my time there. One I did individually. One I did as part of a team. Both took longer than they should have. Both broke in ways I didn't predict. And both left me better at this job than I was before I started them.
First Task: Retry Engine (Individual Task)
What it was
This task was a Research and Development Task. I built a background job processing system with automatic retries, exponential backoff, jitter, and a dead-letter queue for jobs that exhausted all their retry attempts. It was a prototype, not a production system work, but it needed to work correctly under real conditions, not just in the happy path.
The stack was Node.js with better-sqlite3 for persistence. No ORM, just completely raw SQL.
The problem it was solving
Distributed systems fail all the time. There could be network errors, third-party APIs could go down. A naive system that just fires a job and forgets about it is a problem. If something goes wrong, that work is silently lost. A retry engine gives it resilience. When a job fails, the system waits, tries again with increasing delays, and only gives up after a configurable number of attempts. Failed jobs that exhaust their retries land in a dead-letter queue so nothing disappears without a trace.
How I approached it
The core process was straightforward to design. First, poll the database every 500ms for jobs in a pending state, pick one up, execute it, mark it completed or bump its retry count on failure. Exponential backoff with jitter meant each retry waited longer than the last, with some randomness added to avoid thundering herd problems when many jobs fail at once.
I built the schema first, a jobs table with columns for status, attempt count, next retry time, error logs, and a dead_letter table for terminal failures. Then the processor. Then the retry logic. It felt clean. Until it wasn't :).
What broke
The race condition. About halfway through testing, I noticed the same job was sometimes being attempted twice simultaneously. Two instances of the polling loop were picking up the same row at roughly the same time, both reading its status as pending, both deciding to process it.
The obvious fix was to check status before processing, but it didn't work, because the check and the update weren't atomic. There was a gap between "read this job as pending" and "mark it as in-progress" where another instance could read the same row.
I went down a rabbit hole of flags and locks before I landed on the actual solution: SQLite's BEGIN IMMEDIATE transaction combined with a synchronous row update. The trick was to make the status check and the status update a single atomic operation. One SQL statement that read and wrote in the same transaction, failing silently if the row had already been claimed. Because better-sqlite3 is synchronous by design, this actually worked. The synchronous nature of the library, which I saw as a limitation in the beginning, turned out to be the exact property that made the locking reliable. So, no more duplicate attempts.
What I took away
Two things stuck with me.
The first is that concurrency bugs are the most humbling bugs to debug. They don't reproduce reliably, the symptoms point you in the wrong direction, and the fix often requires rethinking a design assumption you made early and didn't question. I had assumed stateless polling was safe. But it isn't, by default.
The second is about tools. I reached for better-sqlite3 because it was simple, and I almost switched to a more "serious" version when the race condition appeared, as if the problem was the database's fault, but it wasn't. Understanding the tool you're using well enough to use its properties intentionally is different from just knowing its API. The synchronous execution model in better-sqlite3 wasn't a quirk to work around, rather it was the solution.
Why I picked this one
Because the race condition is the kind of bug that doesn't care how clean your code is. You can write perfectly readable, well-structured code and still produce a system that corrupts its own state under concurrent load. That was an uncomfortable thing to learn, and an important one.
Second Task: The Candidate Assessment Feature at Skillbridge (Team Task)
What it was
SkillBridge is a talent-pipeline application built to connect employers with candidates. As part of a team, we built the candidate assessment feature, a multi-step evaluation pipeline covering personal assessment, skills assessment, and advanced evaluation stages. The goal was a production-grade flow: handling concurrent submissions, partial failures, incomplete inputs, and edge cases that real users would inevitably hit.
My contribution was primarily backend: the full personal assessment flow, step-by-step validation, graceful failure handling, integration testing, edge case handling, and API contract definition with the frontend team.
The problem it was solving
Candidate assessments in most hiring tools are either too rigid (submit everything at once or lose your progress) or too loose (no validation, inconsistent states). SkillBridge needed something in between. They needed a pipeline that remembered where you were, validated at each step, handled partial saves without corrupting the overall state, and gave the frontend a predictable contract to build against.
How I approached it
I designed the flow stage-by-stage, treating each step as an independent unit with its own validation rules and its own error responses. The personal assessment stage validated required fields, returned structured errors for invalid inputs, and saved partial state so a candidate could return without losing what they'd already completed.
The API contract got documented early and shared with the frontend team before I'd finished building it. The idea was to unblock them and reduce integration surprises. That part worked, but the surprises came from somewhere else.
What broke
Three things broke, really. I'll list them in increasing order of how much they cost us.
The first was the case convention mismatch. The frontend expected snake_case responses, like first_name, assessment_status, created_at. The backend was sending camelCase, like firstName, assessmentStatus, createdAt. This is exactly the kind of thing that should be caught in contract review before a single line of code is written. It wasn't, and we only found it during integration. The fix was straightforward, a response serializer that transformed outgoing data, but the time lost debugging why the frontend was receiving undefined for every field was frustrating and avoidable.
The second was the last minute spec change. Midway through the build, the PMs decided that assessment questions, which had originally been managed by the frontend, should instead be generated and served by the backend. This wasn't a small adjustment. It meant redesigning the data model, adding a questions table, seeding it, building an endpoint the frontend hadn't planned for, and updating a contract that both teams were already building against. The lesson wasn't that specs change, because they always do. It was that a spec change mid-build is much more expensive than one caught in design review.
The third was testing. I came into this task underprepared for writing tests, particularly e2e tests. My early test suites failed often, sometimes because my assumptions about state were wrong, sometimes because I wasn't isolating test data properly, and sometimes because I was testing implementation details instead of behaviour. The first few sessions of watching a test suite go entirely red were demoralising. I almost cried. But fixing failing tests teaches you the code more deeply than writing passing tests ever could. By the end, I had a working grasp of what a good e2e test actually checks, why test isolation matters, and what "testing the right thing" actually means in practice.
What I took away
Team tasks surface a different set of problems than solo work. In a solo task, your assumptions only have to be consistent with themselves. In a team task, your assumptions have to be consistent with everyone else's, and that requires communication infrastructure, not just good intentions.
A shared, written API contract reviewed by both teams before either team starts building would have caught the case convention issue immediately. A stronger change management habit, requiring a documented impact assessment before any mid-job spec change would have reduced the disruption from the question generation pivot.
And on testing, the value of tests isn't in the green checkmarks. It's in what the failures tell you. Every red test I had to fix was the codebase trying to explain something to me. Learning to listen to that was one of the more valuable things this internship gave me.
Why I picked this one
I picked this because it's the task where I most clearly learned the difference between building something and shipping something. Building is technical. Shipping is coordination, communication, contracts, and resilience to change. I got better at both during this task, but the coordination lessons were the ones I didn't expect to need.
Honestly, these two tasks are not the most impressive things I could write about. They're not the biggest system or the most sophisticated algorithm. But they're the ones where I left a worse engineer than I arrived and came out better, which is probably the only metric that actually counts.
I believe I learnt more in these two months than I have learnt in a really long time. And for that, I'm super grateful. This internship brought out the sudden zeal to learn more about not just how systems work, but why they work the way they do.
The HNG Internship doesn't protect you from hard problems, and that's what makes it beautiful.
Top comments (0)