DEV Community: Sahan

I Grew an Engineering Blog from 0 to 463,000 Pageviews - Here's What Worked

Sahan — Tue, 04 Aug 2026 01:43:00 +0000

I published the first posts on sahansera.dev in December 2019.

There was no launch campaign, existing audience, or reliable stream of visitors waiting for them. Like most new personal sites, the blog started at zero. I wrote about problems I had encountered, shared the posts where I could, and hoped somebody searching for the same answers would eventually find them.

By 1 August 2026, the blog had accumulated 463,362 recorded pageviews.

Along the way, it has reached readers in at least 184 countries and territories. Here is where readers came from last month:

Earlier geography data is not included, so the lifetime reach may be broader.

That number is personally meaningful, but it is not the most interesting part of the story. The useful part is what happened underneath it: a small collection of practical engineering articles generated most of the traffic, some posts kept helping people for years, and many things I assumed would matter barely moved the numbers at all.

This is what I learned.

The short version

Specific, practical articles generated most of the traffic.
Evergreen posts kept growing for years after publication.
The blog continued reaching readers during a long publishing break.
Traffic revealed demand, but it did not tell me whether readers returned.
Comparing posts fairly requires consistent time windows, not raw lifetime totals.

From zero to 463,362 pageviews

I calculated the lifetime total using two consecutive analytics exports with no overlapping dates.

A few posts did most of the work

The distribution of traffic surprised me more than the total. Just five articles account for roughly 46% of all the pageviews the blog has recorded since launch.

The leading articles are remarkably consistent in what they offer:

Article	Lifetime pageviews
Simple In-Memory Caching in .NET with `IMemoryCache`	86,975
Understanding WebSockets with ASP.NET	43,272
Distributed Caching in ASP.NET Core with Redis	31,911
Having Fun with Microsoft IoC Container for .NET Core	27,812
Understanding the .NET Generic Host Model	23,040

Other durable performers cover securing the Hangfire dashboard, Kubernetes commands and arguments, and gRPC across Go and .NET.

These are not broad opinion pieces. Each one helps a developer understand a specific concept or complete a specific task.

The lesson is not simply that these technologies are popular. It is that clear intent compounds. A post answering a concrete question can remain useful every day for years, even when I am not actively promoting it.

💡 The pattern: Useful, specific posts can compound for years.

Evergreen technical writing compounds slowly

My highest-traffic article is an introduction to in-memory caching that I published in January 2020. Years later, it is still the largest entry point to the site.

That changed how I think about the return on writing.

A social post has a short distribution window. It may reach many people immediately and then disappear. A useful technical article behaves differently. It may receive very little attention on its first day, but it can be discovered repeatedly through search, links, code repositories, and recommendations.

The early results can feel underwhelming because the compounding is almost invisible. The article needs to be indexed. It needs to answer the query well enough for people to stay. Other pages need to link to it. Search engines need time to understand whether it is useful.

None of my successful posts felt like a breakthrough when I pressed publish. Their value accumulated quietly.

The blog kept working while life took priority

The growth was not driven by a perfectly consistent publishing schedule.

In 2023, I became a first-time dad. My family also moved into a new home, and I changed jobs. I wrote about that season in my plans for sahansera.dev in 2024, acknowledging that blogging had taken a back seat while I focused on being the best dad I could be.

The publication dates tell the story plainly. I published two posts in 2023, none in 2024, and returned with seven posts in 2025. I had hoped to resume a regular schedule sooner, but life had a different rhythm.

I am glad I did not treat the pause as a reason to abandon the blog. While I was not publishing, the existing articles continued answering questions, appearing in search results, and bringing new readers to the site. The work I had already done kept compounding when I did not have the time or energy to add more.

That makes the lifetime total more meaningful to me. It did not come from operating a content machine or forcing myself to publish through every season of life. It came from building a useful body of work, letting it breathe, and returning when I had something worthwhile to share.

💡 The pause: The blog kept growing even when publishing had to wait.

When I started writing again in 2025, I explored streaming APIs, HTTP internals, Python environments, and home-lab Kubernetes. The break had not erased the audience. It gave me a chance to return with different experiences and better questions.

Practical specificity beats broad ambition

The best-performing titles make a small promise:

Configure in-memory caching.
Build a gRPC server or client.
Run Kafka locally for testing.
Secure a Hangfire dashboard.
Understand how WebSockets work.

The posts are narrow enough for the reader to know why they should click, but substantial enough to teach the surrounding concepts.

This balance matters. A title such as "Everything You Need to Know About Distributed Systems" sounds ambitious but does not reveal which problem it solves. "Building a gRPC Server in Go" is less grand and much more useful to the person who needs exactly that.

My better articles tend to combine three layers:

A direct answer to a practical problem.
An explanation of what is happening underneath.
A working implementation readers can adapt.

That combination has become the clearest description of what I want this blog to be.

One successful article should become a cluster

For a long time, I treated each post as an isolated piece of work. The analytics show why that leaves value on the table.

The audience for an in-memory caching tutorial is likely to care about distributed caching, Redis, invalidation, cache stampedes, testing, and production failure modes. Someone building a gRPC server may next need a client, authentication, retries, deadlines, streaming, observability, or Kubernetes deployment guidance.

💡 The opportunity: A successful article is evidence that a useful topic cluster exists.

The gRPC series already demonstrates this. Its introduction, Go server and client, .NET server and client, and deployment posts reinforce one another. The individual posts can satisfy focused searches while the series gives interested readers a natural route through the broader subject.

I want to apply the same model to three areas:

.NET caching and application reliability
Kafka and event-driven system failure modes
Kubernetes operations and production troubleshooting

This does not mean publishing minor variations of the same article. Each post still needs a distinct problem and search intent. The connection between them should help a reader progress from a basic implementation to the difficult production questions.

Traffic is not the same as an audience

The historical reports showed that most visitors left after reading a single page. That sounds alarming until the context is considered.

Many visitors arrive from search, find a code sample or explanation, solve their immediate problem, and leave. For a reference-style engineering article, that can be a successful visit rather than a rejection.

At the same time, the data exposes a real weakness: I made it easy to consume one answer but did not always make the next useful step obvious.

Chronological previous-and-next links are not enough. A reader on a caching article probably does not want the post I happened to publish immediately afterward. They want the most relevant continuation of the problem they are already solving.

The improvements I am making are straightforward:

Add contextual links to related articles within the explanation.
Show a clear next step at the end of high-traffic posts.
Organise related material into visible series and topic hubs.
Keep the email and RSS subscription options easy to find.
Link runnable examples to maintained repositories.

The goal is not to trap somebody on the site. It is to make the site more useful when they want to go deeper.

💡 The real goal: Traffic becomes an audience only when readers have a reason and a way to return.

I measured traffic but not outcomes

Another uncomfortable lesson is that I collected a lot of traffic data without defining what success should mean beyond pageviews.

My current analytics setup contains no configured conversion events. I can see that people read an article, but I cannot reliably answer questions such as:

Did the article lead someone to another useful post?
Did they subscribe by email or RSS?
Did they visit the example repository?
Did they copy a code sample?
Which landing pages create returning readers?

These questions matter more now than the raw total.

Pageviews helped me understand which subjects have demand. The next stage is to measure whether the blog creates a relationship with the reader. I plan to treat a confirmed email subscription as the primary conversion, then track supporting actions such as RSS clicks, repository visits, code copying, deep scrolling, and movement between related articles.

Not every personal blog needs a conversion funnel. But if I want to improve something, I need to be explicit about what "better" means.

Analytics data needs maintenance too

Just over 35% of the views in the recent export have the page title (not set). The file also contains fragmented title variants and at least one suspicious spam-like title.

That is a useful reminder that analytics is not automatically a source of truth just because it contains precise-looking numbers.

Collection can break. Titles can change. A migration can alter definitions. Filters can create misleading exports. Bots and spam can contaminate reports. Privacy settings can change how users and sessions are identified.

I will use page paths as the canonical dimension for content reporting, investigate why page titles are missing, filter known noise, and verify that a single page view is recorded for each navigation. I also want Google Search Console beside Analytics so I can see queries, impressions, rankings, and click-through rates - not only the visits that already happened.

Measurement is part of maintaining the site, not something completed by pasting in a tracking ID once.

New posts need a fair evaluation window

Several of my newer articles have far fewer lifetime views than posts published in 2020 or 2021. That does not necessarily mean they failed.

An article published last month should not be compared directly with one that has accumulated search traffic for six years. Lifetime totals reward age.

For new work, I am moving toward a smaller set of time-normalised measures:

Search impressions and clicks during the first 28 and 90 days
Views per 30 days since publication
Engagement and code-copy actions
Movement to another related article
Email or RSS subscription actions
Whether traffic continues growing after the initial promotion window

This should make it easier to distinguish a promising article that needs time from one whose topic, title, or distribution genuinely missed the mark.

What 463,000 pageviews means to me

Four hundred and sixty-three thousand is not a huge number on the scale of the internet. It is huge compared with the zero visitors I had in December 2019.

More importantly, it represents individual moments when somebody had a problem and something I wrote may have helped them move forward.

The experience has changed my view of successful technical writing:

You do not need a large initial audience.
Useful, specific posts can compound for years.
A handful of durable articles may matter more than a constant publishing schedule.
Updating and connecting existing work can be more valuable than always starting from zero.
Honest measurement is more useful than the largest possible headline.

If you are starting an engineering blog with no visitors, that is normal. Write down the problem you just solved. Explain enough of the underlying system that the solution remains useful. Include the details you wish had been available when you were searching.

Then publish it and give it time.

That is more or less how this blog went from zero to 463,000 pageviews - one specific problem at a time.

If you write technical articles, what has surprised you most about the posts that keep finding readers? I would love to compare notes in the comments.

How We Replaced a Critical Data Path Without a Flag Day

Sahan — Mon, 03 Aug 2026 12:55:00 +0000

Replacing an API call is easy. Replacing the source of truth behind an automated decision is not.

I was reminded of this while migrating a critical workflow from a legacy event feed to a canonical state API. Both systems appeared to answer the same question: should the workflow act on this record now? But they had different schemas, different update timings, and, more importantly, slightly different models of the same lifecycle.

This was not a path where we could deploy the new code on Friday and watch the error rate. A false negative could leave work undone. A false positive could trigger an irreversible action from stale data. The HTTP request succeeding told us almost nothing about whether the new path was making the right decision.

So we did not treat it as a normal code replacement. We treated it as a controlled transfer of authority.

The migration used a query-only mode, shadow comparisons, discrepancy alerts, and a deliberately boring cutover. None of those techniques are particularly novel. What mattered was how we combined them, what we chose to compare, and how we decided the old path was finally safe to delete.

This post walks through that process.

The dangerous assumption: same data, newer API

The legacy path consumed a source built specifically around one class of lifecycle event. The replacement exposed a broader canonical record containing current state, relevant dates, and other attributes used by several workflows.

On paper, the migration looked like this:

That diagram hides the risky part. The two sources did not merely encode the same fact using different field names.

A purpose-built event feed tends to answer an event-shaped question: which transitions were recorded? A canonical API tends to answer a state-shaped question: what is true about this entity now?

Those questions overlap, but they are not identical.

An entity can have an old end-state event and later return to an active state. A record can contain several dates, each valid in its own business context. One source may update immediately while another catches up later. Missing data can mean “not applicable,” “not received yet,” or “something is broken.”

If we had translated fields one-for-one and switched traffic, the code would have looked correct while preserving none of those semantics.

The first useful decision was therefore to stop calling this an API migration. We were migrating a business decision from one model of the world to another.

Start by writing down the invariants

Before building the new path, we wrote down what the workflow must continue to guarantee.

The important invariants were roughly:

An active entity must never be processed because of a stale historical event.
A legitimate lifecycle transition must not be missed because one optional field is absent.
The effective date must have the same business meaning before and after migration.
Ambiguous or contradictory data must fail safely and become visible.
Reprocessing the same record must not produce duplicate downstream actions.

This step sounds obvious, but it changed the review entirely. Instead of asking whether the new client correctly parsed a payload, we could ask whether it preserved the rules the system existed to enforce.

It also exposed a subtle problem with “the old system is the source of truth.” If the legacy path had known defects, perfect agreement would reproduce them. The old output was a baseline, not an oracle.

That meant every mismatch needed investigation, but it did not mean the new path was automatically wrong.

Phase one: make the new path incapable of causing damage

The first version of the new integration was deliberately incomplete. It could query the canonical state API, normalize the response, and calculate what action it would take. It could not perform that action.

I think of this as query-only mode :

decision = evaluate_canonical_record(record)

if query_only:
    return decision

execute(decision)

The real implementation had more safeguards than this, but the boundary was just as explicit. Read and decide on one side; mutate on the other.

Query-only mode gave us a few useful properties:

We could run the code with realistic production data.
We could inspect decisions without creating tickets, sending notifications, or changing access.
We could debug authentication, pagination, missing fields, and schema assumptions separately from cutover risk.
We had a reusable operational tool for investigating individual records.

This last point was unexpectedly valuable. Migration controls are often treated as temporary scaffolding, but a safe read-only execution mode is also a good diagnostic interface. It lets an engineer ask, “What would the system do with this input right now?” without having to reproduce the entire workflow locally.

There was one rule we kept firm: query-only could not mean “mostly read-only.” If a code path still emitted an event or called a downstream service before checking the flag, the control was cosmetic. The no-side-effect guarantee had to sit at the boundary where side effects began.

Phase two: one writer, two decision-makers

Once the new path could evaluate real records safely, we ran it alongside the legacy implementation.

Only the legacy path was allowed to perform actions. The new path observed the same logical input and produced a candidate decision. We then compared the two.

This is usually called a shadow migration or dark launch. The new code sees production-shaped traffic but does not own production effects.

The obvious implementation compares two Boolean values:

legacy_should_process == new_should_process

That is useful, but not enough. When the values differ, a Boolean tells you that the migration is unsafe and nothing about why.

We made the decision explain itself:

Decision(
    should_process=False,
    effective_date=None,
    reason="currently_active",
    source_record_id="...",
)

Now the shadow result could compare:

Whether each path would act.
Which effective date it would use.
Why it reached that conclusion.
Which source record contributed to the decision.
Whether either path considered the input incomplete or ambiguous.

This is the part of the migration I would reuse everywhere: compare normalized decisions, not raw responses.

Raw payload comparison is noisy. One API may use null where another omits a field. Dates may use different time zones. Identifiers may refer to different resources. A hundred harmless representation differences can hide the one semantic difference that matters.

The normalized decision is the contract your users actually experience.

A mismatch is a finding, not just an error

As soon as the comparison ran against real data, discrepancies appeared. That was the point.

It is tempting to turn every mismatch into a page. I would avoid that. Early shadow traffic can be noisy, and training people to ignore an alert stream is a poor way to launch a critical system.

Instead, we recorded every mismatch with enough context to investigate it and grouped them into a small taxonomy:

Expected timing differences — one source had updated before the other.
Representation differences — the sources agreed, but normalization was incomplete.
Semantic differences — both records were valid, but the business interpretation differed.
Data-quality problems — records were missing, stale, or internally contradictory.
Implementation defects — our new code had selected the wrong field or applied the rule incorrectly.

This classification matters because each category leads to a different response.

A timing difference may need a grace period or a later recheck. A representation problem belongs in the adapter. A semantic difference needs a product or domain decision. Bad source data needs a defensive rule and an escalation path. An implementation bug needs code and a regression test.

Without a taxonomy, teams tend to “fix the diff” until the graphs become green. That can accidentally teach the new system to mimic legacy behaviour without understanding it.

The goal was not zero differences at any cost. It was zero unexplained differences.

The edge cases were the migration

The happy path agreed quickly. That did not make the migration nearly complete.

Two edge cases forced us to refine the new model.

The first involved a historical end-state event for an entity whose current state had since changed. If we looked only for the existence of that event, we could produce a false positive. The canonical record gave us another signal: the entity’s current status. We added that verification before allowing the workflow to proceed.

The second involved choosing the effective date. The new source exposed more than one plausible date, and the most conveniently named field was not necessarily the date the downstream process expected. We had to trace the business meaning through the old path and deliberately select the corresponding value.

Neither bug was difficult to fix once understood. The hard part was creating a migration that allowed us to see them before they became actions.

That is why I do not judge shadow migrations by the amount of traffic replayed. A million ordinary records can give more confidence than they deserve. One reactivation, one delayed update, or one contradictory date can tell you much more about whether the new model is correct.

Coverage should be measured across business scenarios , not only request counts.

For this kind of workflow I want an explicit scenario set:

Normal lifecycle transition.
Future-dated transition.
Reactivation or status reversal.
Missing optional attributes.
Conflicting dates.
Duplicate input.
Source timeout or partial response.
A record that changes while being processed.

Some scenarios will occur naturally during the shadow period. Rare but dangerous ones should be exercised with fixtures or controlled replay rather than waiting for production to provide them.

Deciding when to cut over

“The dashboards look fine” is not a cutover criterion.

Before transferring authority to the new path, we wanted evidence in several dimensions:

Correctness: no unexplained decision or effective-date mismatches.
Scenario coverage: important lifecycle transitions had been observed or tested.
Stability: the comparison stayed clean across a meaningful observation window, not just one quiet day.
Failure behaviour: timeouts, missing records, and contradictory data failed safely.
Observability: we could tell whether the new path queried, decided, skipped, failed, or acted.
Rollback: restoring the old authority was understood and quick.

There is no universal percentage or number of days that makes a migration safe. The right window depends on the frequency and consequence of the events you are trying to observe.

If an important edge case happens once a month, a clean afternoon tells you nothing about it. If the cost of a false positive is high, the threshold should reflect that asymmetry.

The cutover itself was intentionally uneventful. We changed which path was authoritative while retaining the ability to compare and roll back. We did not bundle unrelated cleanup into the same release. Boring is a feature when transferring control of a critical workflow.

Do not leave the old path “just in case”

After the new path had operated successfully, we removed the legacy implementation and the shadow comparison.

This can feel premature. Keeping the old path around appears to preserve a fallback. In reality, a dormant fallback decays quickly:

Its credentials and dependencies still need maintenance.
Engineers must continue reasoning about two implementations.
Future changes may update one path but not the other.
Someone can accidentally reactivate code that has not been exercised in months.
The temporary feature flag becomes permanent architecture.

A rollback path is valuable during migration. A second production system with no clear retirement date is technical debt.

We treated deletion as a planned migration phase rather than optional cleanup:

The order matters. Deleting before the soak period removes your quickest recovery option. Never deleting leaves you paying for the migration forever.

The comparison infrastructure should also be removed or deliberately repurposed. Shadow code often doubles reads, emits high-cardinality logs, and contains branching that the main workflow no longer needs. Once its question has been answered, it should not quietly become part of the permanent request path.

What I would carry into the next migration

The practical pattern is straightforward:

Define the invariants before translating fields.
Separate reads and decisions from side effects.
Give the new path a genuine query-only mode.
Keep exactly one writer while both paths evaluate.
Compare normalized decisions and their reasons.
Investigate and classify every meaningful mismatch.
Measure coverage across business scenarios, not only traffic volume.
Set evidence-based cutover and rollback criteria.
Keep the cutover small.
Delete the legacy path after a defined soak period.

The broader lesson is that migrations like this are not primarily data plumbing exercises. They are exercises in transferring trust.

The old path has years of accumulated behaviour, including behaviour nobody documented because the code made it seem obvious. The new source may be cleaner and more canonical, but that does not make your interpretation of it correct. Confidence comes from forcing both systems to make their decisions in the open, then explaining every place they disagree.

That takes longer than changing an endpoint. It is still much cheaper than discovering after cutover that a green deployment was making the wrong decision perfectly.

Thanks for reading ✌️

When "no healthy upstream" isn't about the upstream you think

Sahan — Thu, 23 Jul 2026 00:00:00 +0000

no healthy upstream is the kind of error that makes you expect wreckage.

Then you open the dashboards and find… almost nothing. CPU is low. No pods have crashed. The last deployment was hours ago. By the time you refresh the page, the service has recovered by itself.

That was the scene a few weeks ago when I started chasing an intermittent failure in a search backend. The eventual fix was only a few lines. The interesting part was getting there.

We already had a confident root-cause analysis (RCA), complete with a tidy explanation and a one-line remedy. It was also pointing at the wrong subsystem. This post is about the gap between a plausible story and the evidence, and about a common failure mode in which a mostly healthy fleet slowly removes itself from service.

The shape of the failure

The setup was ordinary: a search backend behind a load balancer, with a fixed pool of worker processes on each instance.

Every so often, with no obvious schedule, a burst of requests failed. The browser showed a bare no healthy upstream, and a minute or two later everything worked again.

One clue appeared every time. Backend p99 latency climbed to almost exactly the load balancer timeout, stayed flat, and then dropped back to normal:

That shape matters. Organic slowdowns tend to be uneven. This was a cliff, a flat top, and a recovery. Requests were not gradually becoming slower; they were running into a deadline and being cut off.

The theory I inherited was CPU throttling. A heavy periodic job supposedly pegged the pod’s CPU, the scheduler throttled it, request handling starved, and the load balancer eventually evicted the instance.

It was coherent. Better still, it came with a one-line fix: raise the CPU limit. That is an attractive combination during an incident. But a root-cause theory makes predictions, and these predictions did not survive contact with the evidence.

Treating the RCA as a hypothesis, not a conclusion

Instead of treating the existing RCA as a conclusion, I treated it as a hypothesis: if CPU throttling caused the incidents, what else should I be able to observe?

Three checks came back wrong.

There was no CPU-bound work in the serving path. The compute-heavy batch indexers ran on separate machines and reached the datastore over the network. They never ran inside the pods serving requests. A process outside the pod’s cgroup cannot cause that pod to be CPU-throttled. The graphs agreed: container throttling counters stayed flat during every event.

The timing did not fit a scheduled trigger. The batch jobs ran on a coarse schedule and produced one sustained utilisation bump. The incidents arrived at arbitrary minutes and happened far more often than the jobs ran. If a timer were responsible, the failures should have followed the timer. They did not.

The failures ignored instance and version boundaries. The same symptom appeared across different pods and deployment SHAs. A regression usually follows a version. A shared external event usually hits the fleet together. These failures did neither. The pattern looked more like each instance was doing something to itself, triggered by its own traffic.

To keep the CPU theory alive, I would have had to explain away the topology, the timing, and the distribution of failures. At that point the theory was creating more questions than it answered, so I dropped it and went back to the logs.

One line in the logs, and why it changes the model

The event window contained one recurring error:

ConnectionTimeout: Connection timed out

The stack ended in the datastore client, blocked on a socket read that never returned.

That one line flipped the model.

A CPU-throttled worker is ready to run but cannot get enough scheduler time. A worker blocked on I/O is off-CPU, waiting on the network while still occupying its worker slot. From the outside, both look like “latency went up, then requests timed out.” Underneath, they are opposites.

Adding CPU to an I/O stall does not unblock the socket. At best, it gives you more workers to park behind the same slow dependency. This is why edge symptoms are a dangerous thing to tune against: resource saturation and dependency blocking can produce the same fever while needing completely different treatment.

Why one slow dependency saturates the whole instance

The mechanism came down to two ordinary client settings that were dangerous in combination:

No timeout on individual datastore calls. One call could occupy a worker long after the user had given up.
Retries on connection timeouts. After waiting too long once, the worker would wait again, with backoff in between.

Nothing exotic. That is partly what makes this failure mode easy to miss.

Little’s Law shows why it is fatal to a fixed worker pool. Concurrency is L = λW: the arrival rate (λ) multiplied by the average time each request spends in the system (W).

Suppose the service receives 200 requests per second and normally responds in 40 ms:

L = 200 req/s × 0.04 s = 8 concurrent requests

Eight concurrent requests are easy for the pool to absorb. But if the datastore slows down and requests wait for tens of seconds, W increases by three orders of magnitude. Retries stretch it further. The required concurrency quickly exceeds the number of workers available.

The pool then looks something like this:

This creates head-of-line blocking. Workers parked on datastore calls cannot serve the fast requests behind them, so latency rises for everything, not only for requests that reached the slow dependency.

Health checks are caught in the same queue. They time out, the load balancer removes the instance from rotation, and the remaining instances receive more traffic. Once enough instances fail their health checks, the load balancer has nowhere to send the next request. That is when the user sees no healthy upstream.

There was one more twist. The client’s total retry time could exceed the load balancer’s deadline:

The load balancer returned an error while the worker continued retrying a result nobody could receive. Every millisecond after the outer deadline was wasted work, and the wasted work held a scarce worker slot.

This is a deadline-propagation failure. Inner operations must finish inside the outer request deadline. Better still, pass the outer deadline through the call chain so every layer knows when its result has become useless.

Why it recovered by itself

The recovery was initially reassuring. In hindsight, it was the worrying part.

The datastore blip triggered retries. Those retries added load to the datastore while it was already struggling, which caused more timeouts and therefore more retries:

That is the start of a metastable failure : a brief trigger knocks the system out of its healthy state, then a feedback loop keeps it unhealthy after the original trigger has passed.

We got lucky. The datastore blips were short enough that traffic fell below the tipping point before the retry loop became self-sustaining. A slightly longer blip could have kept the loop alive until we restarted the fleet or shed enough traffic to escape it.

Self-recovery was not proof of resilience. It was a warning shot.

The fix, and why we kept it small

The instinct during an availability incident is to add headroom: raise CPU limits, increase the worker pool, add replicas. That can help with genuine capacity problems. Here, it would only give the retry loop more workers to occupy.

The useful fix was to put a hard bound on the cost of one request.

First, add an explicit timeout to every downstream call. Choose it from the dependency’s healthy latency distribution rather than picking a pleasing round number. It should sit comfortably above healthy p99.9, but well below the load balancer’s deadline. A call that can wait forever is a worker you can lose forever.

Second, bound retries as a fraction of normal traffic, not as an unconditional count on every request. A rule such as “retry three times” allows every client to multiply traffic precisely when the dependency is least able to handle it. A token-bucket retry budget keeps the added load bounded; the Google SRE guidance uses 10% as an example. Add jitter too, or clients can wake up and retry in synchronised waves.

Third, be especially reluctant to retry connection timeouts. During saturation, a timeout often means the dependency is already over its limit. Another immediate attempt is unlikely to help. Retries make the most sense for independent, transient failures and only for idempotent operations. Search requests were idempotent, at least, so duplicate side effects were not another problem waiting for us.

The decision for each failed call becomes straightforward:

Circuit breakers, load shedding, and bulkheads could all strengthen this design:

A circuit breaker stops callers repeatedly rediscovering the same outage.
Load shedding rejects excess work early enough to keep the service responsive.
A bulkhead gives the dependency its own bounded concurrency pool, preventing it from occupying every worker.

All three add code, tuning, and operational state. We did not have evidence that we needed them yet. Once a small timeout and a bounded retry policy removed the amplifier, adding more machinery would have solved a hypothetical problem rather than the incident in front of us.

“Won’t failing fast just move the errors to the caller?”

This was the immediate pushback: if the backend gives up sooner, won’t users simply see more errors?

Only if we compare failing fast with a world in which every request eventually succeeds. That was not the world we had.

With the worker pool full of hung calls, every request eventually failed. The entire search feature became a no healthy upstream page, including requests that never needed the slow dependency in the first place.

Failing fast keeps the instances responsive and in rotation. It turns one correlated, fleet-wide outage into a smaller number of independent failures: the requests that actually hit the bad path. Those are failures a caller can absorb with a cached result, an empty state, or a retry button.

A nonessential component should not be able to take down the whole page. Give it its own deadline and a graceful fallback, and a slow dependency degrades one part of the experience instead of replacing the entire document with an error page.

Verifying the bound, not asserting it

I did not want to ship a fix that worked only on a whiteboard, so the validation was intentionally mechanical.

With a healthy dependency , normal traffic should remain normal: the same latency distribution and no new errors. If the timeout clips healthy p99.9 requests, it is too aggressive and will create the very failures it is meant to prevent.

With an unreachable dependency , requests should fail near the configured bound and before the outer load balancer deadline. More importantly, worker occupancy and in-flight request counts should remain flat instead of climbing.

That flat line under induced failure is the real acceptance test. It shows that the amplifier is gone.

The saturating case needs its own load test. The happy path will never prove that a service behaves well when every downstream call is stuck.

What I actually took from it

The technical lesson is easy to audit for: an unbounded downstream call plus eager retries is a latent outage with a feedback loop attached. It can sit quietly for months, green on every dashboard, until a dependency has a bad thirty seconds. Then it turns that blip into a fleet-wide event.

It is worth searching your own services for downstream calls without deadlines and retries without a shared budget. Those two settings deserve to be reviewed together, because together they can change the shape of a failure.

The lesson that stayed with me, though, was about diagnosis.

The CPU theory was clean. It was mechanistic. It came with a satisfying one-line fix. And it survived three pieces of contradictory evidence because we had started treating it as an answer instead of a claim.

A useful root cause makes predictions you can check. When the topology, timing, and distribution of failures all disagree with the story, elegance stops counting. In this case, one dull line in a log file told us more than the tidy explanation we had already grown attached to.

That was the expensive part of the incident: not the eventual three-line fix, but learning to let the evidence ruin a good story.

Thanks for reading ✌️

References

The Acknowledgment Gap - How Event-Driven Systems Lose Messages Without Errors

Sahan — Thu, 16 Jul 2026 01:05:00 +0000

This is a submission for DEV's Summer Bug Smash: Smash Stories powered by Sentry.

If you have built event-driven systems for any length of time, you have probably internalized the mantra of at-least-once delivery: keep retrying until the work is done, and design everything downstream to be idempotent. It’s good advice. But there’s a subtle failure mode that hides right underneath it - one where your system faithfully reports success, commits its progress, and quietly drops work on the floor. No exception, no alert, no dead letter. Just a message that was supposed to do something, and didn’t.

I ran into this recently while debugging why a small percentage of events were mysteriously never being processed. Everything looked healthy. The producer got a 2xx. The consumer committed its offset. The dashboards were green. And yet the work never happened. This post is about that gap - the space between “the system accepted my request” and “the system actually did the work” - and why it’s one of the more dangerous places for a distributed system to lose data.

Motivation

Most write-ups about message-processing reliability focus on the well-known hazards: duplicate delivery, out-of-order messages, poison pills, consumer lag. Those are real, and there’s plenty written about them. What I found much less discussed is the case where the acknowledgment itself is a lie - where a component reports success for an operation that has only been accepted, not completed, and a second component treats that acknowledgment as permission to throw the original message away.

This is a design smell that shows up across all sorts of stacks: a queue consumer that calls an async API, a workflow engine that enqueues a job, a service that hands off to a background worker. Any time you have a handoff across an asynchronous boundary , you have the potential for this gap. So I want to walk through the anatomy of the bug from first principles, then talk about how to close it properly.

Background: two kinds of “yes”

Before we get to the bug, let’s be precise about acknowledgments, because the whole problem lives in some sloppy vocabulary.

When a system replies to your request, it can mean one of two very different things:

“I have accepted your request.” - I’ve durably recorded your intent, and I promise to try to do the work. Think HTTP 202 Accepted. The work hasn’t happened yet.
“I have completed your request.” - The work is done, and its effects are durable. Think 200 OK with a result body.

These are worlds apart, and conflating them is the root of a lot of pain. The trouble is that many APIs return the same status code for “accepted” whether or not the work will eventually succeed. A 202 (or a 204 No Content, which is even more ambiguous) tells you the request was received. It tells you nothing about whether the work will run.

Now layer on the consumer side. A huge number of event-driven systems are built on brokers that use offset-based consumer groups - Apache Kafka being the canonical example. If you want a primer, I wrote an introduction to Apache Kafka a while back. The mental model is simple:

Messages in a partition have monotonically increasing offsets.
Your consumer reads a message, does some work, and then commits (or “marks”) the offset to say “I’m done with everything up to here.”
If the consumer crashes before committing, the broker redelivers from the last committed offset. That’s what gives you at-least-once semantics.

💡 The offset commit is a promise about the past. When you commit offset N, you are telling the broker “every message up to and including N has been fully handled, and you never need to give them to me again.” If that statement isn’t actually true, you have manufactured data loss with your own hands.

Hold onto those two ideas - the ambiguous “yes” and the offset-as-promise - because the bug is what happens when they collide.

Anatomy of the pipeline

Let me describe a deliberately generic pipeline. Strip away the specific technologies and almost every async system looks like this:

The consumer’s job is to read an event and translate it into an action by calling some downstream control plane - an async job API, a workflow trigger, a task queue. The control plane accepts the request and, at some later point, a worker actually executes it.

The consumer’s loop, in pseudocode, looked roughly like this:

for {
    msg := broker.Read()

    err := dispatchJob(msg) // calls the async control plane
    if err != nil {
        retryWithBackoff(dispatchJob, msg)
    }

    // Always commit so we never get stuck reprocessing a bad message.
    broker.Commit(msg)
}

At a glance this looks reasonable, even defensive. There’s a retry with backoff. There’s a comment explaining that we always commit to avoid getting wedged on a poison message. Someone clearly thought about failure here.

And that is exactly what makes the bug so insidious.

The bug: the acknowledgment gap

Here’s the sequence that loses data.

The consumer reads a message and calls dispatchJob.
The control plane returns 204 No Content - “request accepted, a job has been created.” From the consumer’s point of view, this is success. err is nil.
Milliseconds later, and entirely outside the consumer’s view , the control plane rejects the job before it executes. Maybe a concurrency quota was exceeded. Maybe an admission controller said no. Maybe the queue was full. The job transitions straight to a terminal “rejected” state without a single line of business logic ever running.
Back in the consumer, dispatchJob returned nil, so the retry loop never fires - there was nothing to retry, as far as it knows.
The consumer commits the offset.

The offset moves forward. The broker will never redeliver that message. The job never ran. And nobody was told.

This is the acknowledgment gap. The failure happened in the window between “accepted” and “executed” , and our success signal was wired to the wrong end of that window. We treated “a job was created” as if it meant “a job will run,” and those are not the same statement.

💡 The most dangerous bugs aren’t the ones that throw. They’re the ones that return nil. An exception is a gift - it’s the system telling you something is wrong. Silent, structurally-invisible loss gives you nothing to catch.

What made it worse is that the “always commit” decision - added defensively to avoid an infinite reprocessing loop - turned a recoverable failure into an unrecoverable one. The one safety mechanism that could have saved us (letting the broker redeliver) was disabled precisely when we needed it.

Why the usual instincts don’t save you

When engineers first see this, they reach for familiar fixes. Most of them don’t actually close the gap:

“Just check the status code.” We did. It was 204. The status code describes the acceptance, not the outcome. The information we needed didn’t exist yet at the moment we got the response.
“Add a retry.” There was one. It only triggers on a failed dispatch, not a failed execution. You can’t retry something you don’t know failed.
“Make it idempotent.” Idempotency is necessary but not sufficient here. Idempotency protects you from doing the work twice; it does nothing to protect you from doing it zero times.
“Use exactly-once semantics.” Setting aside the long debate about whether exactly-once is even a coherent goal across independent systems - the transactional guarantees of your broker do not extend into a third-party control plane you’re calling over HTTP. The moment you cross that boundary, you’re back to coordinating two independent systems with no shared transaction.

The real issue is architectural: we committed our durable progress based on a signal that didn’t actually confirm the work was durable. No amount of tuning the individual pieces fixes that. You have to move the acknowledgment.

Fixing it: verify before you ack

The core principle is a single sentence:

💡 Never acknowledge a message until you have confirmed the work it represents has actually started (or completed) - not merely been accepted.

Everything else is mechanics. Let’s walk through them, because the mechanics are where the interesting distributed-systems problems hide.

1. Close the loop: confirm execution, don’t assume it

Instead of trusting the 204, the consumer now verifies that the dispatched job reached a real running (or terminal-success) state before committing. In practice that means polling the control plane’s read API after dispatch:

started, err := dispatchAndVerify(msg)
if started {
    broker.Commit(msg) // safe: the work is genuinely underway
} else {
    // do NOT commit - let redelivery give us another shot
    alert(msg, err)
}

dispatchAndVerify dispatches, then polls: did a job actually enter a non-rejected state? If it sees the tell-tale “rejected before executing” terminal state, or it can’t find the job at all within a bounded window, it treats that as a failure - which is the thing our original code could never see.

This is really just applying read-after-write thinking to a control plane. Don’t trust the write acknowledgment; go read the state back and confirm reality matches your intent.

2. The correlation problem

Here’s a genuinely tricky sub-problem that this exposes, and it’s a great example of why distributed systems are hard: the dispatch API often doesn’t tell you the ID of the thing it just created. You fire a request, you get back 204 No Content - literally no content - and now you need to find “the job I just created” among all the jobs.

You’re left correlating on secondary signals:

A creation timestamp window (“a job created after time T”), which is racy under concurrency - two near-simultaneous dispatches can be ambiguous.
A business key embedded into the job’s metadata at creation time, if the API lets you set something like a name or a label.

The robust fix is to make the work self-identifying : stamp a correlation key you already own (the entity ID, a request UUID) into the job at dispatch time, so that when you read the state back you can match on it exactly rather than guessing by time. If your control plane supports naming or tagging the work, use it. This is the async equivalent of propagating a trace ID, and it pays for itself the first time you have to debug a race.

💡 Any time you hand work across an async boundary, ask: “When this comes back, how will I know it’s mine?” If the answer is “by timestamp,” you have a race waiting to happen.

3. Bounded redelivery, or: don’t trade loss for a hot loop

The moment you say “don’t commit on failure so the broker redelivers,” someone will rightly point out the opposite failure mode: what if the work keeps failing? Now you’ve built an infinite reprocessing loop, and you’re hammering a control plane that’s already unhappy. This is the eternal tension:

Commit too eagerly → you lose messages (the original bug).
Never commit on failure → you can wedge the consumer forever.

The answer is bounded retries with escalation. Track how many times a given message has been through the wringer - keyed by its stable identity (partition + offset, or a business key) - and:

Retry with exponential backoff while attempts remain, so a transient quota exhaustion gets a chance to clear.
Once you’ve exhausted the budget, stop, escalate loudly, and then commit so a single doomed message can’t block the whole partition behind it.

That final commit is not “giving up silently” - it’s a deliberate, observable decision to route the message to a human (or a dead-letter queue) instead of blocking the stream. The difference between this and the original bug is everything: the original dropped work with zero signal; this drops it only after N visible, alarmed attempts.

4. Separate transient failures from terminal ones

A subtle but important refinement: when you poll to verify, a failure to read the state is not the same as the work being rejected. If your verification call itself hits a network blip or a 503, and you treat that as “the job failed,” you’ll re-dispatch and potentially create duplicate work - trading a lost-message bug for a double-processing bug.

So the verification loop needs to distinguish:

“I confirmed the job was rejected” → terminal, re-dispatch is warranted.
“I couldn’t reach the control plane to check” → transient, just retry the read, don’t re-dispatch.

Only definitive answers should drive irreversible decisions. Everything else is a retryable read. This is the same discipline as not making state transitions on ambiguous signals - you wait until you actually know.

5. Make the invisible visible

The reason this bug survived in production is that it was structurally unobservable. So the last piece is observability, and it’s not optional:

Emit a metric/event whenever a dispatched unit of work fails to start.
Alert when the retry budget is exhausted and a message is dropped.
Log the correlation key, the attempt count, and a link to the rejected work.

If your system is going to make a hard decision like “I’m dropping this after five tries,” that decision must be the loudest thing in the room, not a silent commit. A good rule of thumb: every place your code can decide to discard work should be capable of paging a human. You may choose not to page - but the capability being there forces you to consciously design the failure path instead of falling into one.

Stepping back: the general principle

If you zoom out from the specific mechanics, this whole class of bug reduces to a few reusable principles that are worth carrying into any distributed system you build:

Distinguish “accepted” from “completed” - always. Treat them as different events with different names, different metrics, and different handling. Never let one masquerade as the other.
Anchor your durable acknowledgment to the durable outcome. Commit your offset (or delete your message, or mark your row done) based on confirmation of the effect you care about, not on a transport-level receipt.
A nil error is a claim, not a fact. Verify claims that cross trust boundaries, especially async ones. Read-after-write is cheap insurance.
Bound every retry, and escalate at the boundary. Unbounded retries and silent drops are two sides of the same coin; the cure for both is a visible, finite budget with a loud exit.
Only act irreversibly on unambiguous signals. Transient “I don’t know” should never trigger a decision that assumes “no.”

None of these are novel on their own. What’s interesting is how a single missing distinction - accepted vs. executed - cascades into silent data loss when it meets an offset commit. It’s a good reminder that in distributed systems, the bugs rarely live inside a component. They live in the seams between components , where two reasonable local decisions add up to one unreasonable global one.

Tradeoffs

Nothing here is free, and I’d be doing you a disservice to pretend otherwise.

What you gain

No more silent loss - the failure mode that’s hardest to detect and most damaging to trust.
A verifiable, observable processing pipeline where “done” actually means done.

What it costs

Extra latency and load. Verifying execution means additional reads against the control plane per message. Poll intervals and budgets need tuning.
More moving parts. Correlation keys, attempt tracking, and escalation paths are code you now own and test.
You must embrace at-least-once for real. Verify-and-redeliver will occasionally produce duplicates (e.g., if a job actually started but your confirmation read failed). Idempotency downstream stops being optional - but that was always true; this just makes it honest.

For low-value, high-volume telemetry you might happily accept silent loss and skip all of this. For work where every single message must result in an action, the cost is obviously worth it. As always, the right answer depends on what the data is worth.

Conclusion

The most memorable bugs are the ones that teach you to distrust a word you’d been using carelessly. For me, that word was “success.” A 2xx is not success. A committed offset is not success. Success is the effect you actually wanted, confirmed to be durable. Everything else is just a system being polite.

If you take one thing away: go look at your event-driven pipelines and ask where you commit progress based on an acknowledgment rather than a confirmation. If those two things are wired together, you probably have an acknowledgment gap hiding in there too - quietly green on every dashboard, right up until someone asks where their data went.

Thanks for reading ✌️

References

Introducing gh-weekly-updates - Automate Your Weekly GitHub Impact Summaries

Sahan — Sat, 21 Mar 2026 22:22:00 +0000

If you are anything like me, you’ve probably spent a Friday afternoon trying to remember everything you did that week. Maybe it’s for a standup, a 1:1 with your manager, or just to keep track of your own progress. You end up clicking through PRs, issues, and Slack threads, trying to piece together a coherent story. It’s tedious, and honestly, it’s time you could spend doing actual work.

That’s why I built gh-weekly-updates - a CLI tool that automatically collects your GitHub activity and generates a structured weekly summary using AI.

💡 GitHub repo : github.com/sahansera/gh-weekly-updates. It’s open source and available on PyPI!

The Problem

As engineers, we’re constantly shipping code, reviewing PRs, filing issues, and jumping into discussions. But when it comes time to reflect on the week, all that context is scattered across repos. I wanted something that could:

Pull all my GitHub activity into one place
Summarise it in a way that highlights what actually matters
Run on a schedule so I don’t have to think about it

I couldn’t find anything that did exactly this, so I built it.

What It Does

gh-weekly-updates connects to the GitHub API, collects your activity for a given period, and sends it to an AI model (via GitHub Models) to produce a structured Markdown summary.

Here’s what it picks up:

Pull requests you authored (with merge status, additions/deletions, changed files)
Pull requests you reviewed
Issues you created
Issue comments you left
Discussions you started or participated in

The output is grouped by project or theme and structured into sections like Wins, Challenges, and What’s Next. You can also customise the prompt to match whatever format your team uses.

Getting Started

It’s a Python CLI tool, so you can install it with pip:

pip install gh-weekly-updates

Then just run it:

# If you're already logged in with the GitHub CLI
gh auth login
gh-weekly-updates

That’s it. It will auto-discover repos you contributed to in the past week and generate a summary.

You can also point it at specific repos and date ranges:

gh-weekly-updates --since 2026-02-09 --until 2026-02-16 --repos my-org/my-repo

Configuration

For more control, you can create a config.yaml:

org: my-org

repos:
  - my-org/api-service
  - my-org/web-app

model: openai/gpt-4.1

# Automatically push the summary to a repo
push_repo: my-user/my-weekly-updates

# Customise the AI prompt
prompt_file: my-prompt.txt

The config supports everything from repo lists to custom prompts. You can even swap out the AI model if you have a preference.

Running on a Schedule with GitHub Actions

This is where it gets really useful. You can set up a GitHub Actions workflow to run it every Monday morning and push the summary to a repo automatically:

name: Weekly Summary

on:
  schedule:
    - cron: '0 9 * * 1' # Every Monday at 9am UTC
  workflow_dispatch:

jobs:
  summarise:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - uses: actions/setup-python@v5
        with:
          python-version: '3.12'

      - run: pip install gh-weekly-updates

      - name: Generate summary
        env:
          GITHUB_TOKEN: ${{ secrets.GH_PAT }} # must be named GITHUB_TOKEN
        run: gh-weekly-updates --config config.yaml

Now every Monday, you get a fresh summary committed to your repo. No manual effort required.

Custom Prompts

The default prompt produces a summary with Wins, Challenges, and What’s Next sections. But you can tailor it to your needs. For example, if your team does impact-style updates, you might want sections like Strategic Influence or Next Steps.

Just create a text file with your prompt and reference it in your config:

prompt_file: my-custom-prompt.txt

The prompt receives all your raw activity data as context, so you can shape the output however you like.

Why Open Source?

I initially built this for myself to automate my own weekly updates at work. But I figured other engineers probably have the same problem, so I cleaned it up and open-sourced it. The tool is intentionally simple - it does one thing and tries to do it well.

If you find it useful, give it a ⭐ on GitHub. And if you have ideas for improvements, PRs and issues are always welcome!

What’s Next

A few things I’m thinking about for future releases:

More activity sources : Picking up commit messages, release notes, and code review comments
Multiple output formats : Slack messages, email digests, Notion pages
Team summaries : Aggregate activity across a whole team, not just one person

If any of these sound interesting to you, feel free to open an issue or start a discussion on the repo.

Deploying GitHub Self-Hosted Runners on Your Home Kubernetes Cluster with ARC

Sahan — Tue, 05 Aug 2025 12:28:00 +0000

If you followed my last post, Building a Home Lab Kubernetes Cluster with Old Hardware and k3s, you now have a proper x86 Kubernetes cluster humming away on your old laptops. So, what’s next? Time to put that cluster to work—let’s run GitHub Actions jobs on your own hardware!

Why? Because if you have got decent hardware - self-hosting your runners might be faster, gives you full control (no GitHub minutes limit!), and lets you run bigger jobs (CI, builds, ML, you name it) on your home infra. And with Actions Runner Controller - ARC, managing runners at scale on Kubernetes is surprisingly easy.

Here’s how to set it all up.

What is ARC and Why Should You Care?

ARC (Actions Runner Controller) is an open-source Kubernetes operator from GitHub. It spins up and manages GitHub Actions runners as Kubernetes pods—no more manually registering runners, no more pets, just cattle. Runners auto-scale up and down as jobs arrive. It’s perfect for CI/CD, especially on clusters you own.

Here’s a high-level view of how it works under the hood

Prerequisites

Working Kubernetes cluster (see previous post)
kubectl and helm installed on your machine
A GitHub Personal Access Token (PAT) with repo and admin:org scopes

1️⃣ Pre-Setup: Quick Checks

Make sure you have what you need:

which helm
kubectl version --client
helm list -A

If those commands work, you’re good to go.

2️⃣ Install ARC Controller

Let’s install the ARC controller into your control plane namespace:

NAMESPACE="actions-runner-controller"
helm install arc \
  --namespace "${NAMESPACE}" \
  --create-namespace \
  oci://ghcr.io/actions/actions-runner-controller-charts/gha-runner-scale-set-controller

This deploys the controller which manages your runners.

3️⃣ Deploy a Runner Scale Set

Time to create the runners that will actually do the work. Replace the example GitHub URL and PAT with your details:

INSTALLATION_NAME="arc-runner-set"
NAMESPACE="arc-runners"
GITHUB_CONFIG_URL="https://github.com/youruser/yourrepo"
GITHUB_PAT="ghp_123456..."

helm install "${INSTALLATION_NAME}" \
  --namespace "${NAMESPACE}" \
  --create-namespace \
  --set githubConfigUrl="${GITHUB_CONFIG_URL}" \
  --set githubConfigSecret.github_token="${GITHUB_PAT}" \
  oci://ghcr.io/actions/actions-runner-controller-charts/gha-runner-scale-set

GITHUB_CONFIG_URL: The repo or org you want to run jobs for.
GITHUB_PAT: Your Personal Access Token.

4️⃣ Check That It’s Working

Verify the controller and runner pods are up:

# Controller
kubectl get pods -n actions-runner-controller

# Runners
kubectl get pods -n arc-runners

# Runner set status
kubectl get AutoscalingRunnerSet -A
kubectl describe AutoscalingRunnerSet arc-runner-set -n arc-runners

You should see your runners show up as pods. If you trigger a workflow in your GitHub repo, you’ll see a pod spin up, do the job, and then shut down—magic.

5️⃣ Testing It Out

Here’s the fun part. Create a simple GitHub Actions workflow in your repo to test the runners:

name: Test ARC Runners
on: [push]

jobs:
  build:
    runs-on: arc-runners # This tells GitHub to use your self-hosted runners. Use the NAMESPACE name you defined in step 3.
    steps:
      - name: Checkout code
        uses: actions/checkout@v2

      - name: Run a script
        run: echo "Hello from ARC Runner!"
      - name: List files
        run: ls -la

Here’s ci.yaml from my githubstats repo if you need a working example: ci.yaml

Make a commit to trigger the workflow. You should see the runner pod spin up, execute the job, and then terminate.

Here's the workflow running on my ARC self-hosted runner

6️⃣ Monitoring (Bonus: Grafana)

Want to geek out and monitor your runners? If you’ve set up Prometheus/Grafana (see my upcoming post if not!), you can:

Check pod CPU/memory usage
Track how many runners are running
See logs for each pod

Handy queries for Grafana dashboards:

# Number of ARC runner pods
kube_pod_status_phase{namespace="arc-runners", phase="Running"}

# Pod CPU usage
rate(container_cpu_usage_seconds_total{namespace="arc-runners"}[5m])

Here's mine:

Here's what the workflow looks like in action

6️⃣ Useful Commands for Day-to-Day Ops

# Watch runner scaling in real-time
kubectl get AutoscalingRunnerSet -n arc-runners -w

# See events and troubleshoot
kubectl get events -n arc-runners --sort-by=.metadata.creationTimestamp

# Pod logs (for a specific runner)
kubectl logs -n arc-runners <pod-name>

Troubleshooting Tips

Runner not connecting? Double-check your PAT and network access.
Pods stuck or crash-looping? Check logs for clues and make sure your cluster has enough resources.
Can’t see runners in GitHub? Make sure the config URL matches your repo/org and the PAT has correct scopes.

That’s It! You’re Running GitHub Actions on Your Own Cluster

You now have GitHub Actions jobs running at home on your cluster, scaling up and down automatically. No more slow or limited runners. Your home lab just levelled up—CI/CD, builds, ML, you name it.

Stay tuned for my next post where I’ll show you how to get beautiful observability dashboards and set up alerting for your home cluster.

Happy automating!

Questions? Want to show off your setup? Ping me on X (Twitter) or drop a comment below!

Building a Home Lab Kubernetes Cluster with Old Hardware and k3s

Sahan — Fri, 01 Aug 2025 10:30:00 +0000

If you've read my previous post about building a Raspberry Pi k3s cluster, you know I'm a huge fan of home labs. There's something uniquely satisfying about getting distributed systems running on a bunch of hardware you already own. This time, though, I wanted something a bit more powerful-a cluster that could handle not just learning and tinkering, but also heavier dev, CI/CD, and even ML workloads.

And as it turns out, there's a ton you can do with a handful of old laptops, a simple switch, and Ubuntu Server. If you're thinking about upgrading your home cluster or want to avoid vendor lock-in, this one's for you.

Why Move From Raspberry Pi to x86?

Don't get me wrong-RPi clusters are awesome for learning, hacking, and even some light home automation. But if you've ever tried running real dev pipelines, CI/CD, or any data-heavy ML stuff, you'll hit those limits fast. Plus, WiFi can get a bit flaky when you're trying to keep nodes connected under load.

I had a few spare laptops sitting around, and it made perfect sense to give them a second life and push them to their limits. Bonus: with Ethernet and a decent switch, you get much more reliable connectivity.

Network Setup: Keep It Simple (and Reliable)

For this build, I kept things straightforward: all nodes are wired to a simple network switch. This gives much better performance and reliability compared to WiFi, but honestly, you could still pull this off over wireless if needed.

My network switch - how it all started 🛜

A quick tip:

Reserve static IPs for each node using your router's DHCP reservations.
This makes node management, SSH, and Kubernetes networking so much easier.
Check your DHCP table and make sure every machine has a unique, predictable IP address.

Getting Ubuntu Server Set Up on Each Node

Here's a high level diagram of what my setup looks like:

Let's talk about prepping each machine. My old laptops had a new lease on life running Ubuntu Server. I recommend Ubuntu Server LTS for its simplicity and compatibility. If you're using VMs, just make sure to set the NIC to bridged mode so each VM acts as a full member of your home LAN.

Here's my go-to checklist for each node:

Update the system and install SSH:

   sudo apt update && sudo apt upgrade -y
   sudo apt install openssh-server -y
   sudo systemctl enable ssh
   sudo systemctl start ssh

Set a memorable hostname:

   sudo hostnamectl set-hostname master   # or worker-1, etc.

(Optional) Update /etc/hosts for local resolution:

   sudo nano /etc/hosts

Change any lines like 127.0.1.1 old-hostname to your new hostname.

Find your node's IP address:

   ip a

Use this IP for your DHCP reservation so it always stays the same.

Test SSH from another machine:

   ssh <username>@<node-ip>

Installing k3s: Lightweight, Powerful, Familiar

This process feels pretty magical the first time you see it all come together. I stuck with k3s-lightweight and perfect for home or edge clusters.

On the Control Plane Node ("master")

Install k3s:

   curl -sfL https://get.k3s.io | sh -

Check that k3s is running:

   sudo systemctl status k3s

Get your cluster join token:

   sudo cat /var/lib/rancher/k3s/server/node-token

Copy this somewhere safe-you'll need it for your worker nodes.

Get the node's LAN IP:

   ip a

Let's say your master node IP is 192.168.0.200.

Check your cluster status:

   kubectl get nodes

If you ever want to reset/reinstall k3s, just run:

sudo /usr/local/bin/k3s-uninstall.sh

On Each Worker Node ("worker-1", "worker-2", etc.)

Join the cluster:

   curl -sfL https://get.k3s.io | K3S_URL=https://192.168.0.200:6443 K3S_TOKEN=<your-token-here> INSTALL_K3S_EXEC="--node-ip=<worker-ip>" sh -

Replace 192.168.0.200 with your control plane's IP.

Replace <your-token-here> with the token from your master node.

Replace <worker-ip> with this worker's IP, e.g., 192.168.0.201.

That's it! The node will auto-register with your cluster. No manual kubeconfig needed.
Check the cluster again from the master:

   kubectl get nodes

You should see both master and your worker(s) as Ready!

Troubleshooting and Gotchas

As with any home lab project, there are always a few snags-here's what I ran into and how to fix it:

Node not joining?
- Double-check your token (no spaces, copy the whole thing).
- Make sure you can ping the control plane from your worker node.
- Firewalls can get in the way. Ensure port 6443 is open between nodes.
- If you get hostname conflicts, just change the hostname and restart the k3s agent (sudo systemctl restart k3s-agent).
Cluster state looks weird after hostname change?
- You might see both the old and new hostnames in kubectl get nodes. Just delete the old one:
```
kubectl delete node <old-node-name>
```
IP address keeps changing?
- Make a DHCP reservation for each node's MAC address in your router.
WiFi unreliable?
- Ethernet is always the best bet for clusters, especially for heavy workloads.

What's Next?

With the basics in place, you're ready to:

Run and test real-world apps and dev environments
Build your own CI/CD pipelines
Experiment with ML workloads on dedicated nodes

This way, you get all the power of a real Kubernetes lab, but full control and no monthly surprises from a cloud provider.

Lessons Learned

RPi clusters are great for edge computing and learning, but once you need real muscle, x86 hardware makes a huge difference.
WiFi is convenient, but Ethernet is still king for stable clusters-especially if you're doing anything performance-sensitive.
Old laptops/desktops are a goldmine for home lab builds. Don't let them collect dust!

Final Thoughts

If you have old machines and a bit of curiosity, you can build a cluster that's surprisingly capable-no Raspberry Pis required.

And if you've already done it with RPis, this is the perfect upgrade path.

Keep an eye out for my next post where I'll deep-dive into adding observability and tooling!

Happy clustering! 🫡

How Does the Python Virtual Environment Work?

Sahan — Mon, 28 Jul 2025 00:17:00 +0000

When you start working with Python, one of the first recommendations you’ll hear is to use a “virtual environment.” But what exactly is a Python virtual environment, and how does it work under the hood?

The Problem: Dependency Hell

Python projects often rely on third-party libraries. If you install packages globally, different projects can end up fighting over package versions. This is called “dependency hell.” For example, Project A might require requests==2.25, while Project B needs requests==2.31. Installing both globally can cause conflicts and break your projects.

The Solution: Virtual Environments

A virtual environment is an isolated workspace for your Python project. It lets you install packages locally, so each project can have its own dependencies, regardless of what’s installed elsewhere on your system.

How Does It Work?

When you create a virtual environment (using python -m venv myenv or virtualenv myenv), Python does the following:

Creates a Dedicated Directory Structure

A bin/ or Scripts/ directory with a Python executable and activation scripts
A lib/ directory with a copy of the Python standard library
A pyvenv.cfg config file for metadata

myenv/
├── bin/                # (Note: Scripts\ on Windows)
│   ├── activate        # Shell script to activate the environment (Unix)
│   ├── activate.bat    # Batch script (Windows CMD)
│   ├── Activate.ps1    # PowerShell script (Windows PowerShell)
│   ├── pip             # Environment-specific pip
│   └── python          # Environment-specific Python interpreter
├── lib/
│   └── pythonX.Y/
│       └── site-packages/  # Installed packages go here
├── pyvenv.cfg          # Configuration file with environment metadata

Configures a Standalone Python Interpreter:

The environment includes its own Python executable (or a symlink to it), ensuring that all commands run from within the environment use the correct interpreter.

On most systems, this is a symlink or copy of the base Python binary
This interpreter respects only the packages installed within the environment

which python
/path/to/myenv/bin/python

Sets Up Local Package Management:

Each environment gets its own site-packages directory:

When you run pip install, packages go here instead of the global location
This isolation prevents version conflicts and makes dependency management predictable

Creates Activation Scripts:

Activation scripts help you enter the environment by:

Modifying your $PATH so that python and pip point to the virtual environment
Optionally updating your shell prompt (e.g., showing (myenv))
Ensuring commands are scoped to the environment

These scripts are OS-specific:

Unix/macOS: source myenv/bin/activate
Windows CMD: myenv\Scripts\activate.bat
PowerShell: myenv\Scripts\Activate.ps1

Includes a Configuration File:

The pyvenv.cfg file records metadata about the environment, including the Python version and the location of the base interpreter.

This file stores:

The Python version used
The path to the base interpreter
Whether system site packages are accessible (default: no)

This metadata is used when running the environment to preserve consistent behavior.

Command Resolution

So what happens when you are in a venv as opposed to running python command globally? The diagram below illustrates how Python and pip commands are resolved with and without a virtual environment. When no virtual environment is active, your system’s PATH directs these commands to the globally installed Python interpreter and packages.

However, once a virtual environment is activated, the PATH is modified to point to the environment’s own executables. This ensures that all Python commands and package installations stay isolated within the virtual environment, avoiding conflicts with system-wide installations.

Why Is This Powerful?

Isolation: Each project gets its own dependencies and versions.
Reproducibility: You can lock dependencies with a requirements.txt or pyproject.toml file, making it easy for others (or yourself in the future) to recreate the environment.
No Admin Rights Needed: You don’t need system-wide permissions to install packages.

Advanced Use Cases

Multiple Python Versions: Use virtual environments to test your code against different Python versions.
Custom Activation Scripts: Modify the activation script to set environment variables specific to your project.
Integration with CI/CD: Virtual environments are essential for setting up isolated builds in CI/CD pipelines.

Under the Hood: What’s Really Happening?

The virtual environment is just a directory with a specific structure.
No containers or VMs are involved-just clever manipulation of paths and environment variables.
Deleting the virtual environment directory removes all installed packages for that project.

Debugging Tips

If activation doesn’t work, check your shell configuration.
Use python -m site to inspect the site-packages directory.
Verify the pyvenv.cfg file for any misconfigurations.

Alternative Tools: venv is standard for Python 3.3+, but tools like virtualenv, conda, or pipenv exist for advanced use cases or older Python versions.

Conclusion

Python virtual environments are a foundational tool for modern Python development. They solve the problem of dependency conflicts, make projects more portable, and keep your system clean. Whether you’re building a quick script or a large application, understanding how virtual environments work will save you countless headaches down the road.

References

Deep Dive - How Chunked Transfer Encoding Works

Sahan — Fri, 04 Apr 2025 05:51:00 +0000

Chunked transfer encoding is a key HTTP/1.1 feature that allows servers to stream data incrementally without knowing the total size of the response upfront. It’s particularly useful in streaming APIs, live updates, and large or dynamically-generated responses.

In this post, we’ll practically explore how chunked transfer encoding works using the backend we developed in my previous blog post on Streaming APIs with FastAPI and Next.js — Part 2.

🤔 What is Chunked Transfer Encoding?

Chunked transfer encoding modifies HTTP responses into a series of chunks , each prefixed with its size in bytes. It allows servers to start sending response data immediately, without having to calculate the full content length beforehand.

When Transfer-Encoding: chunked is present, the client receives data incrementally and knows the response has ended when a zero-length chunk appears.

💻 Hands-on Example

Let’s use the FastAPI backend we built.

make start-backend

Let’s hit the /stream endpoint with curl to see how chunked transfer encoding works in practice.

curl -i --raw http://localhost:8000/stream

-i: Include headers in output.
--raw: Disable curl’s automatic decoding, revealing raw chunked encoding.

Expected output:

curl -i --raw localhost:8000/stream
HTTP/1.1 200 OK
date: Mon, 31 Mar 2025 09:51:47 GMT
server: uvicorn
content-type: text/plain; charset=utf-8
Transfer-Encoding: chunked

1f
Waiting for new log entries...

1f
Waiting for new log entries...

1f
Waiting for new log entries...

1f
Waiting for new log entries...

1f
Waiting for new log entries...

30
Simulated log entry at Mon Mar 31 20:21:53 2025

0

Here’s a diagram to help visualize the chunked transfer encoding:

Here’s what’s happening:

Each chunk starts with its length in hexadecimal (1f = 31 bytes).
The data follows the length, and the next chunk starts after a newline.
The chunk with 30 represents a simulated log entry (30 = 48 bytes).
The response ends with a zero-length chunk (0).

Note: This aligns with the techniques demonstrated in my previous blog series Streaming APIs with FastAPI and Next.js (Part 1) and Part 2.

🛠️ Step-by-Step Breakdown of Chunking

We’ll be using the index.py. Here’s exactly what’s happening under the hood:

1. Generator (yield):

Every time yield is executed, the Starlette framework (used internally by FastAPI) receives a new piece of data to stream to the client.
Each yielded data segment corresponds directly to one HTTP chunk.

For example, the yielded line:

yield "Waiting for new log entries...\n"

is packaged into one HTTP chunk.

2. Starlette’s StreamingResponse Handling :

The StreamingResponse from Starlette wraps the async generator.
Starlette doesn’t wait until the generator finishes (which might be infinite). Instead, it immediately pushes each yielded chunk to the underlying ASGI server, typically Uvicorn.

3. Uvicorn’s Chunk Formatting :

Uvicorn (the ASGI server you’re using) receives the yielded chunk from Starlette and formats it according to the HTTP/1.1 chunked transfer encoding specification :

Each chunk is transmitted as follows:

<chunk-size in hexadecimal>\r\n
<chunk-data>\r\n

Here’s how one of your actual data chunks might look:

1f\r\n
Waiting for new log entries...\n\r\n

1f = 31 bytes, the exact length of "Waiting for new log entries...\n"

4. Continuous Chunk Transmission :

Uvicorn immediately sends each formatted chunk down the TCP connection.
Your client (like curl) receives each chunk as soon as it’s sent, which allows incremental processing.

5. Ending the Stream :

If your generator ever completes (or if the server shuts down the connection), Uvicorn sends a special zero-length chunk (0\r\n\r\n) to indicate that transmission has ended.

Example final chunk:

0\r\n
\r\n

🙋 What about HTTP/2 and HTTP/3?

The short answer: HTTP/2+ does not use chunked encoding at all. In fact, the HTTP/2 specification explicitly forbids the use of the Transfer-Encoding: chunked header; if a client incorrectly tries to send it, it’s considered a protocol error.

Instead, HTTP/2 uses a more efficient binary framing layer that allows multiplexing multiple streams over a single connection. This means that chunked transfer encoding is not necessary in HTTP/2 and HTTP/3, as the protocol itself handles streaming more efficiently.

Key Takeaways

Through these practical examples, you’ve seen firsthand how chunked transfer encoding enables incremental streaming of data:

Responses are sent as a series of chunks, each with a defined size.
The end of data transmission is indicated by a zero-length chunk.
Tools like curl, Python frameworks like FastAPI, and browser developer tools help visualize and debug chunked encoding.

Understanding this helps you build better streaming APIs and debug complex HTTP interactions effectively.

Happy Streaming! 🚀

Upgrading sahansera.dev to Gatsby 5

Sahan — Wed, 02 Apr 2025 08:10:00 +0000

The time is now 1:45 AM and I’m finally done upgrading my blog to Gatsby V5. It’s been one heck of a ride, but it was worth it.

This blog post is a reflection of my experience upgrading my blog to Gatsby V5. I’ll cover the challenges I faced, the solutions I found, and the lessons learned along the way.

The Upgrade Journey

After a two year long hiatus from blogging, I decided to dust off my old blog and give it a much-needed facelift. When I tried to run gatsby develop, I was greeted with a slew of warnings and errors. It was clear that my blog was long overdue for an upgrade.

I took a crack at it a few months ago, but I quickly got overwhelmed by the number of breaking changes and decided to put it on hold. This time it’s different because I was determined to fully utilize LLMs for my advantage.

Sneak peak of my last 3 commits 🤣

Being a weekend, which is filled with family time and chores, I decided to timebox it just for a few hours. I thought, “How hard can it be? Just a few package updates and some code tweaks, right?” Little did I know that this would turn into a full-blown adventure.

Dependency Overhauls

One of the first tasks was to ensure all my plugins were updated to their latest non-breaking minor versions. I used npm-check-updates to help with this.

ncu --interactive --format group

Example of ncu CLI tool in action

💡 Tip: Always stay on a separate branch, use commits for each change you make, while doing updates.

There were bunch of warnings, but nothing major. Next, I upgraded React and ReactDOM to v18. This was a bit tricky because I had to ensure that all my dependencies were compatible with React 18. I also had to update my Babel configuration to support the new JSX transform.

npm install react@^18.0.0 react-dom@^18.0.0

After that, I bumped up the Gatsby version to ^5.0.0 and ran npm install. This is where the fun really began 😅

Plugin Upgrades

Many of my plugins were still on versions designed for Gatsby V4. I had to update or replace:

gatsby-plugin-catch-links, gatsby-plugin-feed, gatsby-plugin-google-gtag, gatsby-plugin-layout, gatsby-plugin-manifest, gatsby-plugin-offline, gatsby-plugin-react-helmet, gatsby-plugin-sharp, and gatsby-transformer-sharp etc..

Each of these required an upgrade to versions like ^5.14.0 or ^6.14.0 to resolve the peer dependency conflicts with Gatsby V5.
Image Handling:

I migrated from the deprecated gatsby-image to the modern gatsby-plugin-image. This required updating my GraphQL queries from using fluid and fixed fragments to using the gatsbyImageData field. I updated imports, replaced <Img> with <GatsbyImage>, and used getImage() to extract image data.

Much of this is covered in the official migration guide.

Then I ran into the infamous TypeComposer error.

Cannot create as TypeComposer the following value:
  GraphQLScalarType({ name: "Date", description: "A date string, such as 2007-12-03, compliant with the
 ISO 8601 standard for representation of dates and times using the Gregorian calendar.",
specifiedByURL: undefined, serialize: [function String], parseValue: [function String], parseLiteral:
[function parseLiteral], extensions: { }, astNode: undefined, extensionASTNodes: [] }).

💡 The tl;dr here is there were multiple versions of graphql being used by other dependencies. I had to ensure that all my dependencies were using the same version of graphql. I did this by adding a resolutions field in my package.json:

{
  "resolutions": {
    "graphql": "^16.6.0"
  }
}

Then I ran rm -rf node_modules package-lock.json && npm install again to ensure all dependencies were using the same version of graphql. TypeComposer error was gone, but I still had a few warnings.

Next, I transformed my old GraphQL queries to the new format using the Gatsby codemod.

npx gatsby-codemods@latest sort-and-aggr-graphql .

This worked like a charm and fixed most of the query related issues. Everything else, I just asked ChatGPT to help me with. I had to tweak a few queries here and there, but nothing major.

Styling Challenges

I also relied heavily on styled-jsx for component-scoped CSS. However, Gatsby’s official plugin, gatsby-plugin-styled-jsx, only supports styled-jsx v3—and I needed styled-jsx v5 for React 18 compatibility. After some consideration, I decided to remove the plugin entirely and instead configured Babel to handle styled-jsx directly.

This is where ChatGPT had most of problems. It went through a diamond dependency resolution problem and got stuck in a loop. I had to manually intervene and guide it through the process.

Enabling Nested CSS with styled-jsx

Even after removing the old plugin, I ran into issues when my <style jsx> blocks used nested CSS rules. By default, styled-jsx doesn’t support nesting. I resolved this by integrating styled-jsx-plugin-postcss along with the PostCSS Nested plugin:

Installed the Packages:
Created a Configuration File:

I added a styled-jsx.config.js at the project root:
Updated Babel Config:

In my .babelrc, I ensured the styled-jsx plugin was configured to use the PostCSS plugin:

This allowed my nested CSS in components like the header, footer, and blog items to compile correctly. I would say this the part where LLM helped me the most! 💪

PostCSS Configuration Adjustments

During the upgrade, I also encountered warnings about duplicate autoprefixer instances. My initial postcss.config.js was using postcss-cssnext, which is now deprecated. After some experiments and research, I updated the config to use postcss-preset-env—a modern alternative that handles vendor prefixes efficiently.

Lessons Learned

Upgrade in Small Steps:

Tackling dependency conflicts one by one and verifying functionality helps isolate problems.
Read Plugin Changelogs:

Understanding what’s changed in plugin APIs (like the migration from fluid/fixed to gatsbyImageData) is crucial.
Be Ready to Reconfigure:

Sometimes it’s necessary to remove old plugins (like gatsby-plugin-styled-jsx) and configure Babel or PostCSS directly to maintain compatibility with the latest React and Gatsby versions.
Test Thoroughly:

Both in development mode and via production builds (gatsby build and gatsby serve), to ensure SSR and dynamic behaviors work as expected.

Final Thoughts

Upgrading to Gatsby V5 has not only improved performance and introduced new features, but it also forced me to re-examine and modernize the entire toolchain—from image handling and styling to PostCSS configurations.

If you’re planning a similar upgrade, take your time, tackle one dependency at a time, and don’t hesitate to experiment with configurations until everything clicks.

Prepare to do some A/B testing by having a develop branch and a production branch. This way, you can compare the performance and functionality of your old setup with the new one.

Happy coding, and enjoy your faster, modernized Gatsby blog! 💜

References

Gatsby V5 Migration Guide

Feel free to leave a comment if you have any questions or if you’d like to share your upgrade experiences!

Streaming APIs with FastAPI and Next.js — Part 2

Sahan — Mon, 31 Mar 2025 08:10:00 +0000

In Part 1, we explored how to stream data into a React component using modern browser APIs. Now it’s time to build the other half: the FastAPI backend that makes it all work.

In this post, we’ll walk through setting up a streaming endpoint using FastAPI and discuss how chunked transfer encoding works on the server side.

💡 Code Repository : The complete code is available on GitHub. You can clone it and run it locally to follow along.

🛠️ Building the Streaming Backend with FastAPI

FastAPI makes it really easy to return a streaming response using its StreamingResponse class from starlette.responses.

Let’s build a /stream endpoint in index.py that simulates real-time data like server logs or chat messages.

🚀 Simulating a Real-Time Log Stream

Here’s a minimal FastAPI app with a streaming endpoint:

# backend/index.py
from typing import Any, Generator

from fastapi import FastAPI
from fastapi.responses import StreamingResponse
from fastapi.middleware.cors import CORSMiddleware
import time
import os
import uvicorn # Import uvicorn for running the server
import threading

app = FastAPI()

# Add CORS middleware to allow cross-origin requests
# ...

LOG_FILE_PATH = "logs/server.log"

def log_stream(log_file_path: str) -> Generator[str, None, None]:
    try:
        with open(log_file_path, "r") as log_file:
            # Move to the end of the file
            _ = log_file.seek(0, os.SEEK_END)
            while True:
                line = log_file.readline()
                if line:
                    yield line
                else:
                    yield "Waiting for new log entries...\n" # Heartbeat message
                    time.sleep(1) # Wait for new lines to be written
    except FileNotFoundError:
        yield "Log file not found.\n"
    except Exception as e:
        yield f"Error reading log file: {str(e)}\n"

def simulate_log_generation():
    """Simulate log entries being written to the log file."""
    while True:
        with open(LOG_FILE_PATH, "a") as log_file:
            log_file.write(f"Simulated log entry at {time.ctime()}\n")
        time.sleep(5) # Write a new log entry every 5 seconds

@app.on_event("startup")
def start_log_simulation():
    """Start the log simulation in a background thread."""
    threading.Thread(target=simulate_log_generation, daemon=True).start()

@app.get("/stream")
def stream():
    return StreamingResponse(log_stream(LOG_FILE_PATH), media_type="text/plain")

🧠 How It Works

Let’s break it down:

1. Synchronous Generator

def log_stream(log_file_path: str) -> Generator[str, None, None]:
    try:
        with open(log_file_path, "r") as log_file:
            _ = log_file.seek(0, os.SEEK_END)
            while True:
                line = log_file.readline()
                if line:
                    yield line
                else:
                    yield "Waiting for new log entries...\n"
                    time.sleep(1)
    except FileNotFoundError:
        yield "Log file not found.\n"
    except Exception as e:
        yield f"Error reading log file: {str(e)}\n"

We define a synchronous generator function that yields data one chunk at a time. Each yield becomes a chunk sent to the client. The time.sleep(1) simulates delay between events (like logs being written).

2. StreamingResponse

StreamingResponse(log_stream(LOG_FILE_PATH), media_type="text/plain")

StreamingResponse tells FastAPI to send the data as it becomes available, rather than waiting for the entire response to be generated. The media_type is optional but helps inform the browser how to handle the content.

⚙️ Running the Server

To run the server:

make start-backend

Then hit:

http://localhost:8000/stream

You should see logs appear one line at a time in the terminal or browser, depending on how you call the endpoint.

💡 Tips for Production

✅ Keep the Stream Alive

In a real app, your data stream might be longer-running. Our example already implements this with a heartbeat:

if line:
    yield line
else:
    yield "Waiting for new log entries...\n" # Heartbeat message
    time.sleep(1) # Wait for new lines to be written

This ensures the client knows the connection is still alive even when there’s no new data.

🧹 Handle Disconnects Gracefully

If the client disconnects, your generator should stop yielding. Starlette handles this internally, but you can wrap the stream in a try/except block to catch cancellations if needed.

🔐 Secure Your Stream

Add authentication if you’re streaming sensitive data.
Rate-limit the endpoint to avoid abuse.

🧪 Testing with Curl

You can test the streaming endpoint with:

curl http://localhost:8000/stream

You should see each message appear line by line, every second.

🔗 Hooking it up with the Frontend

Now that our backend is streaming correctly, the frontend from Part 1 will handle it smoothly:

const response = await fetch("http://localhost:8000/stream");
const reader = response.body.getReader();
// ...

As each yield in the backend emits data, your React UI will update in near real-time. ✨

🧠 Recap

In this post, we built a real-time streaming API using FastAPI with just a few lines of code. Here’s what we did:

Built a synchronous generator to simulate streaming logs
Used StreamingResponse to stream text over HTTP
Connected it to our frontend from Part 1

Together, the backend and frontend make a simple but powerful full-stack streaming system.

🧱 Next Steps

🔌 Add dynamic data (e.g. logs from a file or DB)
📦 Stream structured data (like JSON Lines)
📈 Use this setup for real-time dashboards or log viewers

🛠️ Useful Links

If you enjoyed this series, feel free to star the repo ⭐ or share it with a friend. Got ideas or feedback? Hit me up on Twitter or drop an issue in the GitHub repo.

Happy streaming! 🚀

Streaming APIs with FastAPI and Next.js — Part 1

Sahan — Sun, 30 Mar 2025 09:00:00 +0000

Streaming data in the browser is one of those things that feels magical the first time you see it: data appears live — no need to wait for the full response to load. In this two-part series, we’ll walk through building a small full-stack app that uses FastAPI to stream data and a Next.js frontend to consume and render it in real-time.

There are many ways to stream data in the browser, from WebSockets to Server-Sent Events (SSE). But in this post, we’ll focus on a lesser-known method: chunked transfer encoding. This technique allows the server to send data in small, manageable chunks, which the browser can process as they arrive.

This post focuses on the frontend bit. In Part 2, we’ll dive into building the FastAPI backend.

🔧 The Setup

Let’s say you have a streaming API running locally at:

http://localhost:8000/stream

This endpoint sends back a stream of text data — think server logs, chat messages, or real-time updates. The goal is to connect to this endpoint from a React component and display data as it arrives.

Here’s the React component (stream.tsx) that handles this:

import { useEffect, useState } from "react";

export default function StreamPage() {
  const [dataChunks, setDataChunks] = useState<string[]>([]);
  const [isLoading, setIsLoading] = useState(true);
  const [error, setError] = useState<string | null>(null);

  useEffect(() => {
    async function fetchStream() {
      try {
        const response = await fetch("http://localhost:8000/stream");

        if (!response.ok) {
          throw new Error(`HTTP error! Status: ${response.status}`);
        }

        if (!response.body) {
          throw new Error("Response body is null");
        }

        const reader = response.body.getReader();
        const decoder = new TextDecoder("utf-8");

        while (true) {
          const { value, done } = await reader.read();
          if (done) break;

          const chunk = decoder.decode(value, { stream: true });
          setDataChunks((prev) => [...prev, chunk]);
        }

        setIsLoading(false);
      } catch (err) {
        setError(err instanceof Error ? err.message : String(err));
        setIsLoading(false);
      }
    }

    fetchStream();
  }, []);

  return (
    <div>
      <h1>Streaming Data</h1>
      {error && <p>Error: {error}</p>}
      {isLoading && dataChunks.length === 0 ? (
        <p>Loading stream data...</p>
      ) : (
        <pre>{dataChunks.join("")}</pre>
      )}
    </div>
  );
}

🧠 What’s Really Happening?

Let’s break this down and understand the key concepts behind streaming in the browser:

1. Fetching the Stream

const response = await fetch("http://localhost:8000/stream");

Unlike regular fetch() calls that wait for the entire response before handing it over, this gives you access to the ReadableStream — letting you process data chunk-by-chunk.

2. Getting a Reader

const reader = response.body.getReader();

This gives us a ReadableStreamDefaultReader, which lets us manually pull chunks of data from the response. This is part of the Streams API, now supported in all major browsers.

3. Reading Chunks

const { value, done } = await reader.read();

value: a Uint8Array representing a chunk of binary data.
done: true when the stream is finished.

We loop until done becomes true.

4. Decoding the Text

const decoder = new TextDecoder("utf-8");
const chunk = decoder.decode(value, { stream: true });

Streaming responses may split characters across chunks, especially for multi-byte encodings like UTF-8. TextDecoder handles this for us, making sure our text is correctly reconstructed.

5. Updating the UI

setDataChunks((prev) => [...prev, chunk]);

We append each new chunk to our state array. React re-renders the component with every update, giving us that real-time feel.

6. One-Time Effect

useEffect(() => { ... }, []);

The fetch logic runs once when the component mounts. Perfect for one-time side effects like opening a connection.

✅ Recap

With just a few lines of code, we’ve created a streaming experience in the browser using modern Web APIs. The key things were:

fetch() with a streaming response
ReadableStream + TextDecoder
Updating React state to progressively display data

In Part 2, we’ll build the FastAPI backend to power this stream — including how to set up a streaming route and control flush timing for chunked responses.

💡 Gotchas to Watch Out For

CORS : Make sure your FastAPI server has CORS enabled if frontend and backend are on different ports/domains.
Buffering : Some servers (or even browsers like Safari) buffer streaming responses — so test in Chrome/Edge for best results.
Cleanup : If you’re adding WebSocket or long-running fetches, remember to cancel or clean them up in useEffect cleanup.