DEV Community: Sahan

Introducing gh-weekly-updates - Automate Your Weekly GitHub Impact Summaries

Sahan — Sat, 21 Mar 2026 22:22:00 +0000

If you are anything like me, you’ve probably spent a Friday afternoon trying to remember everything you did that week. Maybe it’s for a standup, a 1:1 with your manager, or just to keep track of your own progress. You end up clicking through PRs, issues, and Slack threads, trying to piece together a coherent story. It’s tedious, and honestly, it’s time you could spend doing actual work.

That’s why I built gh-weekly-updates - a CLI tool that automatically collects your GitHub activity and generates a structured weekly summary using AI.

💡 GitHub repo : github.com/sahansera/gh-weekly-updates. It’s open source and available on PyPI!

The Problem

As engineers, we’re constantly shipping code, reviewing PRs, filing issues, and jumping into discussions. But when it comes time to reflect on the week, all that context is scattered across repos. I wanted something that could:

Pull all my GitHub activity into one place
Summarise it in a way that highlights what actually matters
Run on a schedule so I don’t have to think about it

I couldn’t find anything that did exactly this, so I built it.

What It Does

gh-weekly-updates connects to the GitHub API, collects your activity for a given period, and sends it to an AI model (via GitHub Models) to produce a structured Markdown summary.

Here’s what it picks up:

Pull requests you authored (with merge status, additions/deletions, changed files)
Pull requests you reviewed
Issues you created
Issue comments you left
Discussions you started or participated in

The output is grouped by project or theme and structured into sections like Wins, Challenges, and What’s Next. You can also customise the prompt to match whatever format your team uses.

Getting Started

It’s a Python CLI tool, so you can install it with pip:

pip install gh-weekly-updates

Then just run it:

# If you're already logged in with the GitHub CLI
gh auth login
gh-weekly-updates

That’s it. It will auto-discover repos you contributed to in the past week and generate a summary.

You can also point it at specific repos and date ranges:

gh-weekly-updates --since 2026-02-09 --until 2026-02-16 --repos my-org/my-repo

Configuration

For more control, you can create a config.yaml:

org: my-org

repos:
  - my-org/api-service
  - my-org/web-app

model: openai/gpt-4.1

# Automatically push the summary to a repo
push_repo: my-user/my-weekly-updates

# Customise the AI prompt
prompt_file: my-prompt.txt

The config supports everything from repo lists to custom prompts. You can even swap out the AI model if you have a preference.

Running on a Schedule with GitHub Actions

This is where it gets really useful. You can set up a GitHub Actions workflow to run it every Monday morning and push the summary to a repo automatically:

name: Weekly Summary

on:
  schedule:
    - cron: '0 9 * * 1' # Every Monday at 9am UTC
  workflow_dispatch:

jobs:
  summarise:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - uses: actions/setup-python@v5
        with:
          python-version: '3.12'

      - run: pip install gh-weekly-updates

      - name: Generate summary
        env:
          GITHUB_TOKEN: ${{ secrets.GH_PAT }} # must be named GITHUB_TOKEN
        run: gh-weekly-updates --config config.yaml

Now every Monday, you get a fresh summary committed to your repo. No manual effort required.

Custom Prompts

The default prompt produces a summary with Wins, Challenges, and What’s Next sections. But you can tailor it to your needs. For example, if your team does impact-style updates, you might want sections like Strategic Influence or Next Steps.

Just create a text file with your prompt and reference it in your config:

prompt_file: my-custom-prompt.txt

The prompt receives all your raw activity data as context, so you can shape the output however you like.

Why Open Source?

I initially built this for myself to automate my own weekly updates at work. But I figured other engineers probably have the same problem, so I cleaned it up and open-sourced it. The tool is intentionally simple - it does one thing and tries to do it well.

If you find it useful, give it a ⭐ on GitHub. And if you have ideas for improvements, PRs and issues are always welcome!

What’s Next

A few things I’m thinking about for future releases:

More activity sources : Picking up commit messages, release notes, and code review comments
Multiple output formats : Slack messages, email digests, Notion pages
Team summaries : Aggregate activity across a whole team, not just one person

If any of these sound interesting to you, feel free to open an issue or start a discussion on the repo.

Deploying GitHub Self-Hosted Runners on Your Home Kubernetes Cluster with ARC

Sahan — Tue, 05 Aug 2025 12:28:00 +0000

If you followed my last post, Building a Home Lab Kubernetes Cluster with Old Hardware and k3s, you now have a proper x86 Kubernetes cluster humming away on your old laptops. So, what’s next? Time to put that cluster to work—let’s run GitHub Actions jobs on your own hardware!

Why? Because if you have got decent hardware - self-hosting your runners might be faster, gives you full control (no GitHub minutes limit!), and lets you run bigger jobs (CI, builds, ML, you name it) on your home infra. And with Actions Runner Controller - ARC, managing runners at scale on Kubernetes is surprisingly easy.

Here’s how to set it all up.

What is ARC and Why Should You Care?

ARC (Actions Runner Controller) is an open-source Kubernetes operator from GitHub. It spins up and manages GitHub Actions runners as Kubernetes pods—no more manually registering runners, no more pets, just cattle. Runners auto-scale up and down as jobs arrive. It’s perfect for CI/CD, especially on clusters you own.

Here’s a high-level view of how it works under the hood

Prerequisites

Working Kubernetes cluster (see previous post)
kubectl and helm installed on your machine
A GitHub Personal Access Token (PAT) with repo and admin:org scopes

1️⃣ Pre-Setup: Quick Checks

Make sure you have what you need:

which helm
kubectl version --client
helm list -A

If those commands work, you’re good to go.

2️⃣ Install ARC Controller

Let’s install the ARC controller into your control plane namespace:

NAMESPACE="actions-runner-controller"
helm install arc \
  --namespace "${NAMESPACE}" \
  --create-namespace \
  oci://ghcr.io/actions/actions-runner-controller-charts/gha-runner-scale-set-controller

This deploys the controller which manages your runners.

3️⃣ Deploy a Runner Scale Set

Time to create the runners that will actually do the work. Replace the example GitHub URL and PAT with your details:

INSTALLATION_NAME="arc-runner-set"
NAMESPACE="arc-runners"
GITHUB_CONFIG_URL="https://github.com/youruser/yourrepo"
GITHUB_PAT="ghp_123456..."

helm install "${INSTALLATION_NAME}" \
  --namespace "${NAMESPACE}" \
  --create-namespace \
  --set githubConfigUrl="${GITHUB_CONFIG_URL}" \
  --set githubConfigSecret.github_token="${GITHUB_PAT}" \
  oci://ghcr.io/actions/actions-runner-controller-charts/gha-runner-scale-set

GITHUB_CONFIG_URL: The repo or org you want to run jobs for.
GITHUB_PAT: Your Personal Access Token.

4️⃣ Check That It’s Working

Verify the controller and runner pods are up:

# Controller
kubectl get pods -n actions-runner-controller

# Runners
kubectl get pods -n arc-runners

# Runner set status
kubectl get AutoscalingRunnerSet -A
kubectl describe AutoscalingRunnerSet arc-runner-set -n arc-runners

You should see your runners show up as pods. If you trigger a workflow in your GitHub repo, you’ll see a pod spin up, do the job, and then shut down—magic.

5️⃣ Testing It Out

Here’s the fun part. Create a simple GitHub Actions workflow in your repo to test the runners:

name: Test ARC Runners
on: [push]

jobs:
  build:
    runs-on: arc-runners # This tells GitHub to use your self-hosted runners. Use the NAMESPACE name you defined in step 3.
    steps:
      - name: Checkout code
        uses: actions/checkout@v2

      - name: Run a script
        run: echo "Hello from ARC Runner!"
      - name: List files
        run: ls -la

Here’s ci.yaml from my githubstats repo if you need a working example: ci.yaml

Make a commit to trigger the workflow. You should see the runner pod spin up, execute the job, and then terminate.

Here's the workflow running on my ARC self-hosted runner

6️⃣ Monitoring (Bonus: Grafana)

Want to geek out and monitor your runners? If you’ve set up Prometheus/Grafana (see my upcoming post if not!), you can:

Check pod CPU/memory usage
Track how many runners are running
See logs for each pod

Handy queries for Grafana dashboards:

# Number of ARC runner pods
kube_pod_status_phase{namespace="arc-runners", phase="Running"}

# Pod CPU usage
rate(container_cpu_usage_seconds_total{namespace="arc-runners"}[5m])

Here's mine:

Here's what the workflow looks like in action

6️⃣ Useful Commands for Day-to-Day Ops

# Watch runner scaling in real-time
kubectl get AutoscalingRunnerSet -n arc-runners -w

# See events and troubleshoot
kubectl get events -n arc-runners --sort-by=.metadata.creationTimestamp

# Pod logs (for a specific runner)
kubectl logs -n arc-runners <pod-name>

Troubleshooting Tips

Runner not connecting? Double-check your PAT and network access.
Pods stuck or crash-looping? Check logs for clues and make sure your cluster has enough resources.
Can’t see runners in GitHub? Make sure the config URL matches your repo/org and the PAT has correct scopes.

That’s It! You’re Running GitHub Actions on Your Own Cluster

You now have GitHub Actions jobs running at home on your cluster, scaling up and down automatically. No more slow or limited runners. Your home lab just levelled up—CI/CD, builds, ML, you name it.

Stay tuned for my next post where I’ll show you how to get beautiful observability dashboards and set up alerting for your home cluster.

Happy automating!

Questions? Want to show off your setup? Ping me on X (Twitter) or drop a comment below!

Building a Home Lab Kubernetes Cluster with Old Hardware and k3s

Sahan — Fri, 01 Aug 2025 10:30:00 +0000

If you've read my previous post about building a Raspberry Pi k3s cluster, you know I'm a huge fan of home labs. There's something uniquely satisfying about getting distributed systems running on a bunch of hardware you already own. This time, though, I wanted something a bit more powerful-a cluster that could handle not just learning and tinkering, but also heavier dev, CI/CD, and even ML workloads.

And as it turns out, there's a ton you can do with a handful of old laptops, a simple switch, and Ubuntu Server. If you're thinking about upgrading your home cluster or want to avoid vendor lock-in, this one's for you.

Why Move From Raspberry Pi to x86?

Don't get me wrong-RPi clusters are awesome for learning, hacking, and even some light home automation. But if you've ever tried running real dev pipelines, CI/CD, or any data-heavy ML stuff, you'll hit those limits fast. Plus, WiFi can get a bit flaky when you're trying to keep nodes connected under load.

I had a few spare laptops sitting around, and it made perfect sense to give them a second life and push them to their limits. Bonus: with Ethernet and a decent switch, you get much more reliable connectivity.

Network Setup: Keep It Simple (and Reliable)

For this build, I kept things straightforward: all nodes are wired to a simple network switch. This gives much better performance and reliability compared to WiFi, but honestly, you could still pull this off over wireless if needed.

My network switch - how it all started 🛜

A quick tip:

Reserve static IPs for each node using your router's DHCP reservations.
This makes node management, SSH, and Kubernetes networking so much easier.
Check your DHCP table and make sure every machine has a unique, predictable IP address.

Getting Ubuntu Server Set Up on Each Node

Here's a high level diagram of what my setup looks like:

Let's talk about prepping each machine. My old laptops had a new lease on life running Ubuntu Server. I recommend Ubuntu Server LTS for its simplicity and compatibility. If you're using VMs, just make sure to set the NIC to bridged mode so each VM acts as a full member of your home LAN.

Here's my go-to checklist for each node:

Update the system and install SSH:

   sudo apt update && sudo apt upgrade -y
   sudo apt install openssh-server -y
   sudo systemctl enable ssh
   sudo systemctl start ssh

Set a memorable hostname:

   sudo hostnamectl set-hostname master   # or worker-1, etc.

(Optional) Update /etc/hosts for local resolution:

   sudo nano /etc/hosts

Change any lines like 127.0.1.1 old-hostname to your new hostname.

Find your node's IP address:

   ip a

Use this IP for your DHCP reservation so it always stays the same.

Test SSH from another machine:

   ssh <username>@<node-ip>

Installing k3s: Lightweight, Powerful, Familiar

This process feels pretty magical the first time you see it all come together. I stuck with k3s-lightweight and perfect for home or edge clusters.

On the Control Plane Node ("master")

Install k3s:

   curl -sfL https://get.k3s.io | sh -

Check that k3s is running:

   sudo systemctl status k3s

Get your cluster join token:

   sudo cat /var/lib/rancher/k3s/server/node-token

Copy this somewhere safe-you'll need it for your worker nodes.

Get the node's LAN IP:

   ip a

Let's say your master node IP is 192.168.0.200.

Check your cluster status:

   kubectl get nodes

If you ever want to reset/reinstall k3s, just run:

sudo /usr/local/bin/k3s-uninstall.sh

On Each Worker Node ("worker-1", "worker-2", etc.)

Join the cluster:

   curl -sfL https://get.k3s.io | K3S_URL=https://192.168.0.200:6443 K3S_TOKEN=<your-token-here> INSTALL_K3S_EXEC="--node-ip=<worker-ip>" sh -

Replace 192.168.0.200 with your control plane's IP.

Replace <your-token-here> with the token from your master node.

Replace <worker-ip> with this worker's IP, e.g., 192.168.0.201.

That's it! The node will auto-register with your cluster. No manual kubeconfig needed.
Check the cluster again from the master:

   kubectl get nodes

You should see both master and your worker(s) as Ready!

Troubleshooting and Gotchas

As with any home lab project, there are always a few snags-here's what I ran into and how to fix it:

Node not joining?
- Double-check your token (no spaces, copy the whole thing).
- Make sure you can ping the control plane from your worker node.
- Firewalls can get in the way. Ensure port 6443 is open between nodes.
- If you get hostname conflicts, just change the hostname and restart the k3s agent (sudo systemctl restart k3s-agent).
Cluster state looks weird after hostname change?
- You might see both the old and new hostnames in kubectl get nodes. Just delete the old one:
```
kubectl delete node <old-node-name>
```
IP address keeps changing?
- Make a DHCP reservation for each node's MAC address in your router.
WiFi unreliable?
- Ethernet is always the best bet for clusters, especially for heavy workloads.

What's Next?

With the basics in place, you're ready to:

Run and test real-world apps and dev environments
Build your own CI/CD pipelines
Experiment with ML workloads on dedicated nodes

This way, you get all the power of a real Kubernetes lab, but full control and no monthly surprises from a cloud provider.

Lessons Learned

RPi clusters are great for edge computing and learning, but once you need real muscle, x86 hardware makes a huge difference.
WiFi is convenient, but Ethernet is still king for stable clusters-especially if you're doing anything performance-sensitive.
Old laptops/desktops are a goldmine for home lab builds. Don't let them collect dust!

Final Thoughts

If you have old machines and a bit of curiosity, you can build a cluster that's surprisingly capable-no Raspberry Pis required.

And if you've already done it with RPis, this is the perfect upgrade path.

Keep an eye out for my next post where I'll deep-dive into adding observability and tooling!

Happy clustering! 🫡

How Does the Python Virtual Environment Work?

Sahan — Mon, 28 Jul 2025 00:17:00 +0000

When you start working with Python, one of the first recommendations you’ll hear is to use a “virtual environment.” But what exactly is a Python virtual environment, and how does it work under the hood?

The Problem: Dependency Hell

Python projects often rely on third-party libraries. If you install packages globally, different projects can end up fighting over package versions. This is called “dependency hell.” For example, Project A might require requests==2.25, while Project B needs requests==2.31. Installing both globally can cause conflicts and break your projects.

The Solution: Virtual Environments

A virtual environment is an isolated workspace for your Python project. It lets you install packages locally, so each project can have its own dependencies, regardless of what’s installed elsewhere on your system.

How Does It Work?

When you create a virtual environment (using python -m venv myenv or virtualenv myenv), Python does the following:

Creates a Dedicated Directory Structure

A bin/ or Scripts/ directory with a Python executable and activation scripts
A lib/ directory with a copy of the Python standard library
A pyvenv.cfg config file for metadata

myenv/
├── bin/                # (Note: Scripts\ on Windows)
│   ├── activate        # Shell script to activate the environment (Unix)
│   ├── activate.bat    # Batch script (Windows CMD)
│   ├── Activate.ps1    # PowerShell script (Windows PowerShell)
│   ├── pip             # Environment-specific pip
│   └── python          # Environment-specific Python interpreter
├── lib/
│   └── pythonX.Y/
│       └── site-packages/  # Installed packages go here
├── pyvenv.cfg          # Configuration file with environment metadata

Configures a Standalone Python Interpreter:

The environment includes its own Python executable (or a symlink to it), ensuring that all commands run from within the environment use the correct interpreter.

On most systems, this is a symlink or copy of the base Python binary
This interpreter respects only the packages installed within the environment

which python
/path/to/myenv/bin/python

Sets Up Local Package Management:

Each environment gets its own site-packages directory:

When you run pip install, packages go here instead of the global location
This isolation prevents version conflicts and makes dependency management predictable

Creates Activation Scripts:

Activation scripts help you enter the environment by:

Modifying your $PATH so that python and pip point to the virtual environment
Optionally updating your shell prompt (e.g., showing (myenv))
Ensuring commands are scoped to the environment

These scripts are OS-specific:

Unix/macOS: source myenv/bin/activate
Windows CMD: myenv\Scripts\activate.bat
PowerShell: myenv\Scripts\Activate.ps1

Includes a Configuration File:

The pyvenv.cfg file records metadata about the environment, including the Python version and the location of the base interpreter.

This file stores:

The Python version used
The path to the base interpreter
Whether system site packages are accessible (default: no)

This metadata is used when running the environment to preserve consistent behavior.

Command Resolution

So what happens when you are in a venv as opposed to running python command globally? The diagram below illustrates how Python and pip commands are resolved with and without a virtual environment. When no virtual environment is active, your system’s PATH directs these commands to the globally installed Python interpreter and packages.

However, once a virtual environment is activated, the PATH is modified to point to the environment’s own executables. This ensures that all Python commands and package installations stay isolated within the virtual environment, avoiding conflicts with system-wide installations.

Why Is This Powerful?

Isolation: Each project gets its own dependencies and versions.
Reproducibility: You can lock dependencies with a requirements.txt or pyproject.toml file, making it easy for others (or yourself in the future) to recreate the environment.
No Admin Rights Needed: You don’t need system-wide permissions to install packages.

Advanced Use Cases

Multiple Python Versions: Use virtual environments to test your code against different Python versions.
Custom Activation Scripts: Modify the activation script to set environment variables specific to your project.
Integration with CI/CD: Virtual environments are essential for setting up isolated builds in CI/CD pipelines.

Under the Hood: What’s Really Happening?

The virtual environment is just a directory with a specific structure.
No containers or VMs are involved-just clever manipulation of paths and environment variables.
Deleting the virtual environment directory removes all installed packages for that project.

Debugging Tips

If activation doesn’t work, check your shell configuration.
Use python -m site to inspect the site-packages directory.
Verify the pyvenv.cfg file for any misconfigurations.

Alternative Tools: venv is standard for Python 3.3+, but tools like virtualenv, conda, or pipenv exist for advanced use cases or older Python versions.

Conclusion

Python virtual environments are a foundational tool for modern Python development. They solve the problem of dependency conflicts, make projects more portable, and keep your system clean. Whether you’re building a quick script or a large application, understanding how virtual environments work will save you countless headaches down the road.

References

Deep Dive - How Chunked Transfer Encoding Works

Sahan — Fri, 04 Apr 2025 05:51:00 +0000

Chunked transfer encoding is a key HTTP/1.1 feature that allows servers to stream data incrementally without knowing the total size of the response upfront. It’s particularly useful in streaming APIs, live updates, and large or dynamically-generated responses.

In this post, we’ll practically explore how chunked transfer encoding works using the backend we developed in my previous blog post on Streaming APIs with FastAPI and Next.js — Part 2.

🤔 What is Chunked Transfer Encoding?

Chunked transfer encoding modifies HTTP responses into a series of chunks , each prefixed with its size in bytes. It allows servers to start sending response data immediately, without having to calculate the full content length beforehand.

When Transfer-Encoding: chunked is present, the client receives data incrementally and knows the response has ended when a zero-length chunk appears.

💻 Hands-on Example

Let’s use the FastAPI backend we built.

make start-backend

Let’s hit the /stream endpoint with curl to see how chunked transfer encoding works in practice.

curl -i --raw http://localhost:8000/stream

-i: Include headers in output.
--raw: Disable curl’s automatic decoding, revealing raw chunked encoding.

Expected output:

curl -i --raw localhost:8000/stream
HTTP/1.1 200 OK
date: Mon, 31 Mar 2025 09:51:47 GMT
server: uvicorn
content-type: text/plain; charset=utf-8
Transfer-Encoding: chunked

1f
Waiting for new log entries...

1f
Waiting for new log entries...

1f
Waiting for new log entries...

1f
Waiting for new log entries...

1f
Waiting for new log entries...

30
Simulated log entry at Mon Mar 31 20:21:53 2025

0

Here’s a diagram to help visualize the chunked transfer encoding:

Here’s what’s happening:

Each chunk starts with its length in hexadecimal (1f = 31 bytes).
The data follows the length, and the next chunk starts after a newline.
The chunk with 30 represents a simulated log entry (30 = 48 bytes).
The response ends with a zero-length chunk (0).

Note: This aligns with the techniques demonstrated in my previous blog series Streaming APIs with FastAPI and Next.js (Part 1) and Part 2.

🛠️ Step-by-Step Breakdown of Chunking

We’ll be using the index.py. Here’s exactly what’s happening under the hood:

1. Generator (yield):

Every time yield is executed, the Starlette framework (used internally by FastAPI) receives a new piece of data to stream to the client.
Each yielded data segment corresponds directly to one HTTP chunk.

For example, the yielded line:

yield "Waiting for new log entries...\n"

is packaged into one HTTP chunk.

2. Starlette’s StreamingResponse Handling :

The StreamingResponse from Starlette wraps the async generator.
Starlette doesn’t wait until the generator finishes (which might be infinite). Instead, it immediately pushes each yielded chunk to the underlying ASGI server, typically Uvicorn.

3. Uvicorn’s Chunk Formatting :

Uvicorn (the ASGI server you’re using) receives the yielded chunk from Starlette and formats it according to the HTTP/1.1 chunked transfer encoding specification :

Each chunk is transmitted as follows:

<chunk-size in hexadecimal>\r\n
<chunk-data>\r\n

Here’s how one of your actual data chunks might look:

1f\r\n
Waiting for new log entries...\n\r\n

1f = 31 bytes, the exact length of "Waiting for new log entries...\n"

4. Continuous Chunk Transmission :

Uvicorn immediately sends each formatted chunk down the TCP connection.
Your client (like curl) receives each chunk as soon as it’s sent, which allows incremental processing.

5. Ending the Stream :

If your generator ever completes (or if the server shuts down the connection), Uvicorn sends a special zero-length chunk (0\r\n\r\n) to indicate that transmission has ended.

Example final chunk:

0\r\n
\r\n

🙋 What about HTTP/2 and HTTP/3?

The short answer: HTTP/2+ does not use chunked encoding at all. In fact, the HTTP/2 specification explicitly forbids the use of the Transfer-Encoding: chunked header; if a client incorrectly tries to send it, it’s considered a protocol error.

Instead, HTTP/2 uses a more efficient binary framing layer that allows multiplexing multiple streams over a single connection. This means that chunked transfer encoding is not necessary in HTTP/2 and HTTP/3, as the protocol itself handles streaming more efficiently.

Key Takeaways

Through these practical examples, you’ve seen firsthand how chunked transfer encoding enables incremental streaming of data:

Responses are sent as a series of chunks, each with a defined size.
The end of data transmission is indicated by a zero-length chunk.
Tools like curl, Python frameworks like FastAPI, and browser developer tools help visualize and debug chunked encoding.

Understanding this helps you build better streaming APIs and debug complex HTTP interactions effectively.

Happy Streaming! 🚀

Upgrading sahansera.dev to Gatsby 5

Sahan — Wed, 02 Apr 2025 08:10:00 +0000

The time is now 1:45 AM and I’m finally done upgrading my blog to Gatsby V5. It’s been one heck of a ride, but it was worth it.

This blog post is a reflection of my experience upgrading my blog to Gatsby V5. I’ll cover the challenges I faced, the solutions I found, and the lessons learned along the way.

The Upgrade Journey

After a two year long hiatus from blogging, I decided to dust off my old blog and give it a much-needed facelift. When I tried to run gatsby develop, I was greeted with a slew of warnings and errors. It was clear that my blog was long overdue for an upgrade.

I took a crack at it a few months ago, but I quickly got overwhelmed by the number of breaking changes and decided to put it on hold. This time it’s different because I was determined to fully utilize LLMs for my advantage.

Sneak peak of my last 3 commits 🤣

Being a weekend, which is filled with family time and chores, I decided to timebox it just for a few hours. I thought, “How hard can it be? Just a few package updates and some code tweaks, right?” Little did I know that this would turn into a full-blown adventure.

Dependency Overhauls

One of the first tasks was to ensure all my plugins were updated to their latest non-breaking minor versions. I used npm-check-updates to help with this.

ncu --interactive --format group

Example of ncu CLI tool in action

💡 Tip: Always stay on a separate branch, use commits for each change you make, while doing updates.

There were bunch of warnings, but nothing major. Next, I upgraded React and ReactDOM to v18. This was a bit tricky because I had to ensure that all my dependencies were compatible with React 18. I also had to update my Babel configuration to support the new JSX transform.

npm install react@^18.0.0 react-dom@^18.0.0

After that, I bumped up the Gatsby version to ^5.0.0 and ran npm install. This is where the fun really began 😅

Plugin Upgrades

Many of my plugins were still on versions designed for Gatsby V4. I had to update or replace:

gatsby-plugin-catch-links, gatsby-plugin-feed, gatsby-plugin-google-gtag, gatsby-plugin-layout, gatsby-plugin-manifest, gatsby-plugin-offline, gatsby-plugin-react-helmet, gatsby-plugin-sharp, and gatsby-transformer-sharp etc..

Each of these required an upgrade to versions like ^5.14.0 or ^6.14.0 to resolve the peer dependency conflicts with Gatsby V5.
Image Handling:

I migrated from the deprecated gatsby-image to the modern gatsby-plugin-image. This required updating my GraphQL queries from using fluid and fixed fragments to using the gatsbyImageData field. I updated imports, replaced <Img> with <GatsbyImage>, and used getImage() to extract image data.

Much of this is covered in the official migration guide.

Then I ran into the infamous TypeComposer error.

Cannot create as TypeComposer the following value:
  GraphQLScalarType({ name: "Date", description: "A date string, such as 2007-12-03, compliant with the
 ISO 8601 standard for representation of dates and times using the Gregorian calendar.",
specifiedByURL: undefined, serialize: [function String], parseValue: [function String], parseLiteral:
[function parseLiteral], extensions: { }, astNode: undefined, extensionASTNodes: [] }).

💡 The tl;dr here is there were multiple versions of graphql being used by other dependencies. I had to ensure that all my dependencies were using the same version of graphql. I did this by adding a resolutions field in my package.json:

{
  "resolutions": {
    "graphql": "^16.6.0"
  }
}

Then I ran rm -rf node_modules package-lock.json && npm install again to ensure all dependencies were using the same version of graphql. TypeComposer error was gone, but I still had a few warnings.

Next, I transformed my old GraphQL queries to the new format using the Gatsby codemod.

npx gatsby-codemods@latest sort-and-aggr-graphql .

This worked like a charm and fixed most of the query related issues. Everything else, I just asked ChatGPT to help me with. I had to tweak a few queries here and there, but nothing major.

Styling Challenges

I also relied heavily on styled-jsx for component-scoped CSS. However, Gatsby’s official plugin, gatsby-plugin-styled-jsx, only supports styled-jsx v3—and I needed styled-jsx v5 for React 18 compatibility. After some consideration, I decided to remove the plugin entirely and instead configured Babel to handle styled-jsx directly.

This is where ChatGPT had most of problems. It went through a diamond dependency resolution problem and got stuck in a loop. I had to manually intervene and guide it through the process.

Enabling Nested CSS with styled-jsx

Even after removing the old plugin, I ran into issues when my <style jsx> blocks used nested CSS rules. By default, styled-jsx doesn’t support nesting. I resolved this by integrating styled-jsx-plugin-postcss along with the PostCSS Nested plugin:

Installed the Packages:
Created a Configuration File:

I added a styled-jsx.config.js at the project root:
Updated Babel Config:

In my .babelrc, I ensured the styled-jsx plugin was configured to use the PostCSS plugin:

This allowed my nested CSS in components like the header, footer, and blog items to compile correctly. I would say this the part where LLM helped me the most! 💪

PostCSS Configuration Adjustments

During the upgrade, I also encountered warnings about duplicate autoprefixer instances. My initial postcss.config.js was using postcss-cssnext, which is now deprecated. After some experiments and research, I updated the config to use postcss-preset-env—a modern alternative that handles vendor prefixes efficiently.

Lessons Learned

Upgrade in Small Steps:

Tackling dependency conflicts one by one and verifying functionality helps isolate problems.
Read Plugin Changelogs:

Understanding what’s changed in plugin APIs (like the migration from fluid/fixed to gatsbyImageData) is crucial.
Be Ready to Reconfigure:

Sometimes it’s necessary to remove old plugins (like gatsby-plugin-styled-jsx) and configure Babel or PostCSS directly to maintain compatibility with the latest React and Gatsby versions.
Test Thoroughly:

Both in development mode and via production builds (gatsby build and gatsby serve), to ensure SSR and dynamic behaviors work as expected.

Final Thoughts

Upgrading to Gatsby V5 has not only improved performance and introduced new features, but it also forced me to re-examine and modernize the entire toolchain—from image handling and styling to PostCSS configurations.

If you’re planning a similar upgrade, take your time, tackle one dependency at a time, and don’t hesitate to experiment with configurations until everything clicks.

Prepare to do some A/B testing by having a develop branch and a production branch. This way, you can compare the performance and functionality of your old setup with the new one.

Happy coding, and enjoy your faster, modernized Gatsby blog! 💜

References

Gatsby V5 Migration Guide

Feel free to leave a comment if you have any questions or if you’d like to share your upgrade experiences!

Streaming APIs with FastAPI and Next.js — Part 2

Sahan — Mon, 31 Mar 2025 08:10:00 +0000

In Part 1, we explored how to stream data into a React component using modern browser APIs. Now it’s time to build the other half: the FastAPI backend that makes it all work.

In this post, we’ll walk through setting up a streaming endpoint using FastAPI and discuss how chunked transfer encoding works on the server side.

💡 Code Repository : The complete code is available on GitHub. You can clone it and run it locally to follow along.

🛠️ Building the Streaming Backend with FastAPI

FastAPI makes it really easy to return a streaming response using its StreamingResponse class from starlette.responses.

Let’s build a /stream endpoint in index.py that simulates real-time data like server logs or chat messages.

🚀 Simulating a Real-Time Log Stream

Here’s a minimal FastAPI app with a streaming endpoint:

# backend/index.py
from typing import Any, Generator

from fastapi import FastAPI
from fastapi.responses import StreamingResponse
from fastapi.middleware.cors import CORSMiddleware
import time
import os
import uvicorn # Import uvicorn for running the server
import threading

app = FastAPI()

# Add CORS middleware to allow cross-origin requests
# ...

LOG_FILE_PATH = "logs/server.log"

def log_stream(log_file_path: str) -> Generator[str, None, None]:
    try:
        with open(log_file_path, "r") as log_file:
            # Move to the end of the file
            _ = log_file.seek(0, os.SEEK_END)
            while True:
                line = log_file.readline()
                if line:
                    yield line
                else:
                    yield "Waiting for new log entries...\n" # Heartbeat message
                    time.sleep(1) # Wait for new lines to be written
    except FileNotFoundError:
        yield "Log file not found.\n"
    except Exception as e:
        yield f"Error reading log file: {str(e)}\n"

def simulate_log_generation():
    """Simulate log entries being written to the log file."""
    while True:
        with open(LOG_FILE_PATH, "a") as log_file:
            log_file.write(f"Simulated log entry at {time.ctime()}\n")
        time.sleep(5) # Write a new log entry every 5 seconds

@app.on_event("startup")
def start_log_simulation():
    """Start the log simulation in a background thread."""
    threading.Thread(target=simulate_log_generation, daemon=True).start()

@app.get("/stream")
def stream():
    return StreamingResponse(log_stream(LOG_FILE_PATH), media_type="text/plain")

🧠 How It Works

Let’s break it down:

1. Synchronous Generator

def log_stream(log_file_path: str) -> Generator[str, None, None]:
    try:
        with open(log_file_path, "r") as log_file:
            _ = log_file.seek(0, os.SEEK_END)
            while True:
                line = log_file.readline()
                if line:
                    yield line
                else:
                    yield "Waiting for new log entries...\n"
                    time.sleep(1)
    except FileNotFoundError:
        yield "Log file not found.\n"
    except Exception as e:
        yield f"Error reading log file: {str(e)}\n"

We define a synchronous generator function that yields data one chunk at a time. Each yield becomes a chunk sent to the client. The time.sleep(1) simulates delay between events (like logs being written).

2. StreamingResponse

StreamingResponse(log_stream(LOG_FILE_PATH), media_type="text/plain")

StreamingResponse tells FastAPI to send the data as it becomes available, rather than waiting for the entire response to be generated. The media_type is optional but helps inform the browser how to handle the content.

⚙️ Running the Server

To run the server:

make start-backend

Then hit:

http://localhost:8000/stream

You should see logs appear one line at a time in the terminal or browser, depending on how you call the endpoint.

💡 Tips for Production

✅ Keep the Stream Alive

In a real app, your data stream might be longer-running. Our example already implements this with a heartbeat:

if line:
    yield line
else:
    yield "Waiting for new log entries...\n" # Heartbeat message
    time.sleep(1) # Wait for new lines to be written

This ensures the client knows the connection is still alive even when there’s no new data.

🧹 Handle Disconnects Gracefully

If the client disconnects, your generator should stop yielding. Starlette handles this internally, but you can wrap the stream in a try/except block to catch cancellations if needed.

🔐 Secure Your Stream

Add authentication if you’re streaming sensitive data.
Rate-limit the endpoint to avoid abuse.

🧪 Testing with Curl

You can test the streaming endpoint with:

curl http://localhost:8000/stream

You should see each message appear line by line, every second.

🔗 Hooking it up with the Frontend

Now that our backend is streaming correctly, the frontend from Part 1 will handle it smoothly:

const response = await fetch("http://localhost:8000/stream");
const reader = response.body.getReader();
// ...

As each yield in the backend emits data, your React UI will update in near real-time. ✨

🧠 Recap

In this post, we built a real-time streaming API using FastAPI with just a few lines of code. Here’s what we did:

Built a synchronous generator to simulate streaming logs
Used StreamingResponse to stream text over HTTP
Connected it to our frontend from Part 1

Together, the backend and frontend make a simple but powerful full-stack streaming system.

🧱 Next Steps

🔌 Add dynamic data (e.g. logs from a file or DB)
📦 Stream structured data (like JSON Lines)
📈 Use this setup for real-time dashboards or log viewers

🛠️ Useful Links

If you enjoyed this series, feel free to star the repo ⭐ or share it with a friend. Got ideas or feedback? Hit me up on Twitter or drop an issue in the GitHub repo.

Happy streaming! 🚀

Streaming APIs with FastAPI and Next.js — Part 1

Sahan — Sun, 30 Mar 2025 09:00:00 +0000

Streaming data in the browser is one of those things that feels magical the first time you see it: data appears live — no need to wait for the full response to load. In this two-part series, we’ll walk through building a small full-stack app that uses FastAPI to stream data and a Next.js frontend to consume and render it in real-time.

There are many ways to stream data in the browser, from WebSockets to Server-Sent Events (SSE). But in this post, we’ll focus on a lesser-known method: chunked transfer encoding. This technique allows the server to send data in small, manageable chunks, which the browser can process as they arrive.

This post focuses on the frontend bit. In Part 2, we’ll dive into building the FastAPI backend.

🔧 The Setup

Let’s say you have a streaming API running locally at:

http://localhost:8000/stream

This endpoint sends back a stream of text data — think server logs, chat messages, or real-time updates. The goal is to connect to this endpoint from a React component and display data as it arrives.

Here’s the React component (stream.tsx) that handles this:

import { useEffect, useState } from "react";

export default function StreamPage() {
  const [dataChunks, setDataChunks] = useState<string[]>([]);
  const [isLoading, setIsLoading] = useState(true);
  const [error, setError] = useState<string | null>(null);

  useEffect(() => {
    async function fetchStream() {
      try {
        const response = await fetch("http://localhost:8000/stream");

        if (!response.ok) {
          throw new Error(`HTTP error! Status: ${response.status}`);
        }

        if (!response.body) {
          throw new Error("Response body is null");
        }

        const reader = response.body.getReader();
        const decoder = new TextDecoder("utf-8");

        while (true) {
          const { value, done } = await reader.read();
          if (done) break;

          const chunk = decoder.decode(value, { stream: true });
          setDataChunks((prev) => [...prev, chunk]);
        }

        setIsLoading(false);
      } catch (err) {
        setError(err instanceof Error ? err.message : String(err));
        setIsLoading(false);
      }
    }

    fetchStream();
  }, []);

  return (
    <div>
      <h1>Streaming Data</h1>
      {error && <p>Error: {error}</p>}
      {isLoading && dataChunks.length === 0 ? (
        <p>Loading stream data...</p>
      ) : (
        <pre>{dataChunks.join("")}</pre>
      )}
    </div>
  );
}

🧠 What’s Really Happening?

Let’s break this down and understand the key concepts behind streaming in the browser:

1. Fetching the Stream

const response = await fetch("http://localhost:8000/stream");

Unlike regular fetch() calls that wait for the entire response before handing it over, this gives you access to the ReadableStream — letting you process data chunk-by-chunk.

2. Getting a Reader

const reader = response.body.getReader();

This gives us a ReadableStreamDefaultReader, which lets us manually pull chunks of data from the response. This is part of the Streams API, now supported in all major browsers.

3. Reading Chunks

const { value, done } = await reader.read();

value: a Uint8Array representing a chunk of binary data.
done: true when the stream is finished.

We loop until done becomes true.

4. Decoding the Text

const decoder = new TextDecoder("utf-8");
const chunk = decoder.decode(value, { stream: true });

Streaming responses may split characters across chunks, especially for multi-byte encodings like UTF-8. TextDecoder handles this for us, making sure our text is correctly reconstructed.

5. Updating the UI

setDataChunks((prev) => [...prev, chunk]);

We append each new chunk to our state array. React re-renders the component with every update, giving us that real-time feel.

6. One-Time Effect

useEffect(() => { ... }, []);

The fetch logic runs once when the component mounts. Perfect for one-time side effects like opening a connection.

✅ Recap

With just a few lines of code, we’ve created a streaming experience in the browser using modern Web APIs. The key things were:

fetch() with a streaming response
ReadableStream + TextDecoder
Updating React state to progressively display data

In Part 2, we’ll build the FastAPI backend to power this stream — including how to set up a streaming route and control flush timing for chunked responses.

💡 Gotchas to Watch Out For

CORS : Make sure your FastAPI server has CORS enabled if frontend and backend are on different ports/domains.
Buffering : Some servers (or even browsers like Safari) buffer streaming responses — so test in Chrome/Edge for best results.
Cleanup : If you’re adding WebSocket or long-running fetches, remember to cancel or clean them up in useEffect cleanup.

📚 References

Basics of Apache Kafka - An Overview

Sahan — Tue, 03 Jan 2023 16:01:00 +0000

A few months ago I wrote an article on setting up a local Apache Kafka instance. In this article, I will briefly introduce Apache Kafka and some of its core concepts. We will look at the these concepts from a high level and understand how they are put together.

What is Kafka?

Apache Kafka is an open-source distributed event streaming platform for building real-time data pipelines and streaming applications. It is designed to handle high volumes of data and enable real-time processing of data streams.

Kafka is typically used in scenarios where data needs to be processed in real-time, such as in real-time analytics, event-driven architectures, and other streaming applications. It can be integrated with various systems, such as databases, storage systems, and stream processing frameworks, to support a wide range of use cases.

Event Streaming vs Messages Queues

I have seen people using Kafka interchangeably with message queues. However, they are two things. Kafka is a distributed event streaming system whereas message queuing systems like RabbitMQ, Amazon SQS, and Azure Queue Storage.

I suppose confusion must have stemmed mainly from the fact that the feature sets of these systems are getting more or less the same and the line between them is getting blurred. However, it is advisable to understand trade-offs among these systems and pick one that suits you best without overcomplicating the overall solution.

Why Kafka?

There are several reasons why Apache Kafka is a popular choice for building real-time data pipelines and streaming applications:

Scalability: Kafka is designed to handle high volumes of data and to scale horizontally, making it suitable for handling large amounts of data.
Real-time processing: Kafka enables real-time processing of data streams, allowing applications to react to new data as soon as it arrives.
Fault-tolerant: Kafka is fault-tolerant and highly available, meaning it can continue operating even if some of its components fail.
Flexibility: Kafka can be integrated with many systems and frameworks, making it a versatile platform for building streaming applications.
High performance: Kafka has been designed to provide high throughput and low latency, making it suitable for applications that require fast processing of data streams.

In a nutshell, imagine you have the following tightly coupled spaghetti of systems (could be deployed internally within your organization or a public-facing app)

For instance, you could imagine these services to be separate microservices or even monoliths that need to defer their workloads to some other systems. As we can see services 1 - 5 talk to services A - E. This becomes extremely difficult to maintain and will cause massive tech debt.

Following is a refactored communication flow utilizing Kafka. Your line of business applications, or microservices can be added or removed as you go. It also provides a consistent API for both produces and consumers. But, could this be a single point of failure? What if Kafka goes down? or can it go down? You’ll find the answers in the upcoming sections, why that’s not the case.

Putting everything together

Below is a diagram that I’ve created to show you a typical request flow in Kafka. The numbers correspond to each subheading that I will describe in much detail below. In each section, I will zoom in on a particular concept to provide any relevant context.

Before we jump in, I need to mention that Kafka exposes a client API that could be used by users to write and read messages. You could use your programming language of choice to interact with the cluster using these API calls.

1. Producers

Producers write data (or events) to topics, and consumers read data from topics. Producers could be anything like an IoT sensor, activity tracking system, a frontend server sending metrics etc. You can specify a key/id in your message in order to make sure that the messages always end up in the same partition. More on these will be covered in Events, Topics and Partitions sections.

Other than that, it’s worth noting that the Producers can choose 3 methods to send messages (fire and forget, Synchronous send & Asynchronous send) and also they can wait for acknowledgements (none, any or all) as well.

2. Events

An event in Apache Kafka is a record of an occurrence or state change in a system. Sometimes it’s also called a record or a message as well. This could be a user interaction, a change in data, or a notification from another system. Events are stored in Kafka topics and are typically published by producers and consumed by consumers. Events in Kafka are immutable , such that they cannot be changed once they have been published.

Data formats

Kafka sees messages are just byte arrays. However, we humans need a way to serialize and deserialize this data. Therefore, you could use a schema for it such as JSON, Protocol Buffers, or Apache Avro (used in Hadoop).

What’s important to understand is that you use a consistent data format in your applications.

Here is an example of a Kafka event with a key and value that are both strings:

Key: "user_id"
Value: "12345"
Timestamp: "20 Dec 2022"
Headers: {}

3. Topics and Partitions

In Kafka, a topic is a way to separate out different messages coming in from different systems. Each topic is partitioned, meaning that the data in a topic is split into multiple partitions distributed across the Kafka cluster. This allows for redundancy and scalability within Kafka.

If a given topic is partitioned across multiple nodes (i.e. servers), how does Kafka know which message goes to which partition also while preserving the order? This is where the key of a message comes in handy. Kafka uses the key of a message and runs it through a hash function and mod (%) the resulting number so that it knows where to write that message.

What if a message doesn’t have a key (key being NULL)? Well, in that case, Kafka defaults to a Round-Robin approach to write the messages to the partitions.

💡 Messages with the same key go to the same partition thus guaranteeing the order.

It’s also worth mentioning that a partition is not strictly tied to a single server. They can also scale out horizontally depending on your requirements.

Brokers

if you start working on Kafka, you’d come across the term brokers. A broker is just a computer, server, container, or cloud instance that’s running a Kafka broker process. So what do they do? They are responsible for managing the partitions that we talked about in the previous section. They also handle any read/write requests.

Remember I mentioned that a partition can be replicated across multiple nodes? Brokers manage that replication too. But there has to be some kind of way to orchestrate this replication, right? In Kafka, they follow a leader-follower approach to tackle that. Generally speaking, reading and writing happens at the leader level. Then the leader and followers work among themselves to get the replication done.

4. Consumers & Consumer Groups

The “consumer” is a complex concept in Kafka. Consumers are applications that subscribe to a set of Kafka topics and read from topic partitions. Consumers track their offsets as they progress through topic partitions and periodically commit offsets to Kafka, which ensures that they can resume from the last processed offset in the event of a restart. These offsets are incremental and the brokers will know which messages to send when the consumer requests more data.

As shown above, same consumer can also listen to mutiple topics as well.

💡 What sets Kafka apart from traditional Message Queues is that reading a message does not destroy it.

Why is Kafka fast?

Apache Kafka uses a combination of in-memory storage and disk storage to achieve high scalability. Kafka stores data in memory in a log structure, which allows for fast access and processing.

However, as the amount of data in a Kafka cluster grows, it can eventually exceed the available memory. In this case, Kafka will automatically flush data to disk to free up memory. This allows Kafka to scale horizontally by adding more nodes to the cluster, which can provide additional memory and disk storage capacity. By combining in-memory and disk storage, Kafka can achieve high scalability while still providing fast access and processing of data.

How do I build a Kafka cluster?

If your organization has a need to deploy your own cluster at a data center you can do that. Kafka’s deployment targets range from bare metal servers to cloud providers. If you are rolling out your own, beware that it’s not easy to set it up it will take time.

Locally (Development)

Using Docker is the easiest I’ve found. Follow this guide to understand how you can do that.
Or you could YOLO and install the binaries yourself 😀

Cloud (Production)

For your production needs, your best bet would be to use

Different cloud providers might charge you differently. It’s always wise to consider the pricing options before locking yourself into a specific cloud provider.

Conclusion

In this article we looked at Apache Kafka is an open-source distributed event streaming platform for building real-time data pipelines and streaming applications. We went over the main concepts that should be enough to get you started. I’m planning to write an article series on implementing producers and consumers in .NET, Go & Python. So stay tuned! Until next time 👋

References

Serverless Go with Azure Functions and GitHub Actions

Sahan — Sun, 11 Sep 2022 14:55:00 +0000

In this article, we are going to learn how to host a serverless Go web service with Azure Functions leveraging custom handlers in Azure functions. We will also automate the deployments by using GitHub Actions 🚀 You can even apply the same concepts to other languages Rust as long as you can self-contained binary.

Walkthrough video

If you prefer to watch a video, I have created a walkthrough video on my YouTube channel 😊

Introduction

Azure Functions is Microsoft’s offering for serverless computing that enables you to run code on-demand without having to explicitly provision or manage infrastructure. Azure Functions is a great way to run your code in response to events, such as HTTP requests, timers, or messages from Azure services.

As of today, they support multiple runtimes such as .NET, Java, JavaScript, Python, TypeScript. But what if we want to write our app in Go? Well, now you can do that too - by using Custom Handlers.

Before we move on, so what really is a custom handler and how does it work? Custom handlers let your Function app to accept events (eg. HTTP requests) from the Global host (aka Function host - that powers your Function apps) - as long as your chosen language supports HTTP primitives.

Here’s a great overview from Microsoft on how this is achieved.

Source: Microsoft Docs

So, in our case, we are going to wrap a Go binary as a Function app and deploy it to Azure. Sounds good? Let’s jump right into it.

Prerequisites

Make sure you have the following setup locally.

Go SDK
Azure Functions Core Tools
Prefererrably, Azure Functions VS Code extension

The plan

Clone the repo or create one
Create the Azure function resources
Test out the app
Deploy to Azure

1. Clone the repo or create one

Our code is going to be pretty simple. All it does is, whenever we make a request, it will return a random quote on programming.

💡 You can clone the repo I have created from here.

package main

// Removed for brevity

var quotes = []string{
    "Talk is cheap. Show me the code.",
    "First, solve the problem. Then, write the code.",
    "Experience is the name everyone gives to their mistakes.",
    "Any fool can write code that a computer can understand. Good programmers write code that humans can understand.",
}

func quotesHandler(w http.ResponseWriter, r *http.Request) {
    // Get a random quote
    message := quotes[rand.Intn(len(quotes))]

    // Write the response
    fmt.Fprint(w, message)
}

func main() {
    listenAddr := ":8080"
    if val, ok := os.LookupEnv("FUNCTIONS_CUSTOMHANDLER_PORT"); ok {
        listenAddr = ":" + val
    }
    http.HandleFunc("/api/GetQuotes", quotesHandler)
    log.Printf("About to listen on %s. Go to https://127.0.0.1%s/", listenAddr, listenAddr)
    log.Fatal(http.ListenAndServe(listenAddr, nil))
}

Explanation

We setting port 8080 as the default port for our app. But if we are running this in Azure Functions, we will be using a different port. So, we are checking if the FUNCTIONS_CUSTOMHANDLER_PORT environment variable is set. If it is, we will use that port instead.
We are registering a handler for the /api/GetQuotes endpoint. This is the endpoint that we will be using to make requests to our app.
quotesHandler is a simple function that returns a random quote from the quotes array.

2. Create the Azure function resources

You can create your own Azure function from portal.azure.com or using ARM templates. Below are the configuration I chose.

Publish: Code
Runtime Stack: Custom Handler
Operating System: Linux
Plan type: Consumption (Serverless)

3. Test out the app

If you have cloned the project you should see a folder structure similar to what’s shown below.

At the root of the project folder let’s run the following commands.

go build handler.go # To build a binary
func start # Start the Function app service

You should see the output like so:

Let’s run through each file now.

GetQuotes/function.json: This file defines what happens when a request comes in and what should go out from the function. These are known as bindings. Our function is triggered by HTTP requests and we will return a response
handler.go: This is our Go web service where our main logic lives in. We listen on port 8080 and expose an HTTP endpoint called /api/GetQuotes
host.json: Take note under customHandler.description.defaultExecutablePath is set to handler which says where to find the compiled binary of our Go app and enableForwardingHttpRequest where we tell the function host to forward our traffic

4. Deploy to Azure with GitHub Actions

Now that we have everything ready to go let’s deploy this to Azure! 🚀 To get this done, we are going to use the Azure Functions Action from the GH Actions Marketplace.

From a high-level this is what we need to do in order to deploy this.

Authenticate with Azure
Build the project
Deploy

Authenticate with Azure

Since our app is written in “Go” which is not really supported out of the box, we won’t be able to use the Publish Profile method for this. So we are going to focus on using an Azure Service Principal for RBAC.

💡 Remember to follow the steps according to this guide to create an SP.

Once you have created the service principal, we need to add that as a secret in our repo so that it can be used for authenticating with Azure Resource Manager during the deployment step. Head over to your repo → Settings → Secrets → Actions
Create a secret a called AZURE_RBAC_CREDENTIALS and update its content with what you got when you created the service principal.

Now the that the credentials are in place we can refer to that from the GitHub Action workflow file.

Build the project

This step is pretty straightforward as we are using the Go SDK to build the project with GOOS=linux GOARCH=amd64 config.

This is what the final workflow file should look like:

deploy.yml

name: CI/CD

on:
  push:
    branches: ["main"]
  pull_request:
    branches: ["main"]

env:
  AZURE_FUNCTIONAPP_NAME: azgofuncapp # set this to your application's name

jobs:

  build:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v3

    - name: 'Login via Azure CLI'
      uses: azure/login@v1
      with:
        creds: ${{ secrets.AZURE_RBAC_CREDENTIALS }}

    - name: 'Set up Go'
      uses: actions/setup-go@v3
      with:
        go-version: 1.18

    - name: Build
      run: GOOS=linux GOARCH=amd64 go build handler.go

    - name: 'Deploy to Azure'
      uses: Azure/functions-action@v1
      with:
        app-name: ${{ env.AZURE_FUNCTIONAPP_NAME }}

Once everything is done, head over to the following URL and refresh a couple of times to see our programming quotes! 😀

https://<your_app_URL>.azurewebsites.net/api/GetQuotes

Here’s the an example.

Conclusion

Well, that’s it folks! Today we built a small Go web service, wrapped it in an Azure Function and deployed it to Azure by using GitHub Actions!

Troubleshooting

I ran into couple of issues when creating this project.

Are you getting the Value cannot be null. (Parameter 'provider') error?

I was able to resolve it by following the exact config as below.

{
  "version": "2.0",
  "logging": {...},
  "customHandler": {
    "description": {
      "defaultExecutablePath": "handler",
      "workingDirectory": "",
      "arguments": []
    },
    "enableForwardingHttpRequest": true
  }
}

Are you getting the Azure Functions Runtime is unreachable error?

For me, this went away when I did the first deployment. If it still doesn’t go away, check out this link for more info.

References

Deploying a .NET gRPC Server on Azure App Service

Sahan — Mon, 05 Sep 2022 03:59:00 +0000

In this article we are going to be deploying a gRPC web service written in .NET to an Azure App Service. We will then interact with it and also automate deployment process of the Azure resources and the build.

Introduction

About a few months ago I wrote a blog series on creating a simple gRPC server and a client locally. This time we are going to take the next steps by deploying it to the cloud.

If you missed the series you can get yourself up to speed with the below links 🚀

Walkthrough video

If you like to watch a video walkthrough instead of this article, you can follow along on my YouTube channel too 😊

The Plan

Clone the above repo.
We will create an App Service instance on Azure and deploy it (automated).
Getting ready for deployment.
Deploying the BookshopServer code.
Testing the web service.

1. Clone the repo (or build one)

Since we already have an app in handy we are going to use the same thing and make this work. More specifically we are going to use the exact code we looked at in the Building a gRPC Server in .NET blog post.

💡 The code can be found under this repo.

We are going to be using the BookshopServer as our web service and BookshopClient as the client to interact with it.

Once you have it cloned, you can set the branch to app-service with,

git checkout app-service

2. Create an App Service instance in Azure

You can head over to portal.azure.com and create the App Service instance manually or you could use the ARM template down below.

💡 To save you some time and trouble, I’d stay away from the F1 tier - it was unreliable for me and had a hard time deploying my code. I went ahead with B1 (and you get 30 days free for Linux). It costs you a little bit, but heaps more reliable for Dev/Testing work.

ARM template for easy deployment

If you haven’t got az cli head over here to get it set up. Once installed, do a az login and select the account you want to use for deployments.

I have created a shell script to automate all the steps. This is the usage of the script.

cd BookshopServer/Infrastructure/
./deploy.sh net6-grpc australiasoutheast template.json parameters.json

The parameters to pass in are as follows:

Resource group name (eg: net6-grpc)
Region you want to create the resources in (eg: australiasoutheast)
Name of the ARM template file (eg: template.json)
Name of the parameters file (eg: parameters.json)

If you are unsure, you can see what’s what just by typing ./deploy as well. Once deployed, you should be able to see something like below.

App Service configuration

The ARM template takes care of most of the configuration for us. However there’s one more thing that we need to enabled, called HTTP 2.0 Proxy

This is what your configuration should look like:

3. Getting ready for deployment

Before we do any deployment there are a couple of things we need to change in our code. These are all done for you in my repo. But for the curious,

BookshopServer/Program.cs

Add/Update the configuration to tell Kestrel to listen on all IPs on port 8080 and also on port 8585 but with HTTP 2.0 only.

// ...
builder.WebHost.ConfigureKestrel(options =>
{
  options.ListenAnyIP(8080);
  options.ListenAnyIP(8585, listenOptions => 
  { 
      listenOptions.Protocols = Microsoft.AspNetCore.Server.Kestrel.Core.HttpProtocols.Http2; 
  }); 
});

// ...

Now that we have that config we no longer need the configuration specified in appSettings.json. Let’s remove the following lines like so.

BookshopServer/appSettings.json

...
// Delete the following lines
"Kestrel": {
    "EndpointDefaults": {
      "Protocols": "Http2"
    }
  }
...

4. Deploying the BookshopServer code

Let’s go ahead and deploy our code! 🚀 There are many ways to do this. If you are setting it up for testing you can simply deploy from your local Git repo. To keep the scope of this blog post small I’m not going to be covering other continous deployment methods. Let me know if you like me to cover them too!

You can follow these steps from the official docs to quickly deploy from locally. There are some configurations that you need to do.

Go to you app service instance > click Deployment Center
Select Local Git from the Source dropdown
Click Local Git/FTPS Credentials
Set a username and a password under User Scope
Hit Save
(Important) Go back to the app service Overview
Copy the URL under Git Clone URL
Go to your local repo
Do a git remote add azure <Git URL you copied before>
Change deployment branch with git push azure <local branch name>:master

This might take a couple of minutes for the initial deployment. After the last step you should be able to see a log like so:

Although it my seem a little daunting at first, this is only a one-off config. For any code changes you make, you just need to commit your changes to your local branch and then do a git push like we saw in step 10 above.

5. Testing the web service

Now to the interesting part. Let’s see how we can interact with our web service 🤔

For some reason I couldn’t get grpcurl to work, thankfully, we already have the BookshopClient project - so let’s use that!

When you open up the project, make sure to update the URL to point to your App service - and that’s all you need to do!

BookshopClient/Program.cs

using var channel = GrpcChannel.ForAddress("https://your_app_service.azurewebsites.net/");

(make sure you add the / at the end of the URL)

and finally let’s run the project.

dotnet run --project BookshopClient

That should return back the list of books as shown below.

That’s it! now you know how to deploy a gRPC .NET web service in Azure 🎉

Conclusion

In this article we looked at how we can leverage Azure App Service to deploy a gRPC server and interact with it. Until next time 👋

References

Setting up a local Apache Kafka instance for testing

Sahan — Thu, 01 Sep 2022 00:00:00 +0000

I recently started digging into Apache Kafka for a project I’m working on at work. The more I started looking into it I realized I need to get myself a local Kafka cluster to play around with. This article walks you through that process and hopefully save your some time 😄.

If you don’t know what Kafka is, here’s a great video from Confluent.

💡 You can follow along by cloning the repo I have created over here.

Installing the binaries

There’s a quickstart guide over at the official docs. To be honest, I didn’t want to install anything locally in order to keep the Kafka instance fairly separate my dev environment, hence went with the Docker way which you will find below.

The Docker way

This is by far the easiest I have found and saves you ton of time. Once you have cloned the repo just spin it up!

docker-compose up -d

If you want to know what’s going on in here let’s take a look at the docker-compose.yaml file.

docker-compose.yaml

---
version: '3'
services:
  zookeeper:
    image: confluentinc/cp-zookeeper:7.0.1
    container_name: zookeeper
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000

  broker:
    image: confluentinc/cp-kafka:7.0.1
    container_name: broker
    ports:
    # To learn about configuring Kafka for access across networks see
    # https://www.confluent.io/blog/kafka-client-cannot-connect-to-broker-on-aws-on-docker-etc/
      - "9092:9092"
    depends_on:
      - zookeeper
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: 'zookeeper:2181'
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_INTERNAL:PLAINTEXT
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://localhost:9092,PLAINTEXT_INTERNAL://broker:29092
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1

We spin up a Zookeeper instance at port 2181 internally within the Docker network (i.e. not accessible from the host)
Next we spin up our broker which is the Kafka instance which has a dependency on Zookeeper. This is exposed at port 9092 and accessible from the host.

Everything else is set to default. If you want to learn what each environment variable does, head over here. You can add multiple brokers (Kafka instances) if you want in the same docker-compose file.

This approach is quite beneficial if you want to stand up an instance for testing on the go or even in a CI environment.

Interacting with the cluster

Before we publish anything, let’s spin up a consumer so that we know our setup is working. To do that, run the following command from the CLI.

docker exec --interactive --tty broker \
kafka-console-consumer --bootstrap-server localhost:9092 \
                       --topic example-topic \
                       --from-beginning

Now let’s create some messages and see them in action!

docker exec --interactive --tty broker \
kafka-console-producer --bootstrap-server localhost:9092 \
                       --topic example-topic

This is what it will look like from the terminal. To your left is the producer and consumer to your right hand side.

A GUI perhaps?

Not a fan of the CLI? There’s many tools out there. I wanted to find a nice, lightweight, OSS solution for this. I decided to go ahead with Tools for Apache Kafka for VS Code.

Once you have that going you could interact with it from the side bar. There will be a new icon with Kafka’s logo on it!

Here’s what it looks like when it finds your instance.

producer.kafka

PRODUCER keyed-message
topic: example-topic
key: mykeyq
record content

###

PRODUCER non-keyed-json-message
topic: example-topic
{
    "type": "my_test_event-{{random.number}}"
}

consumer.kafka

CONSUMER consumer-group-id
topic: example-topic
partitions: 0
from: 1

You can invoke the producers and consumers right within VS Code, which is pretty easy and cool!

Conclusion

Welll that’s it! Standing up a docker container certainly did save me some time just for testing purposes. Hope you enjoyed this post and see you next time 👋

DEV Community: Sahan

Introducing gh-weekly-updates - Automate Your Weekly GitHub Impact Summaries

The Problem

What It Does

Getting Started

Configuration

Running on a Schedule with GitHub Actions

Custom Prompts

Why Open Source?

What’s Next

Links

Deploying GitHub Self-Hosted Runners on Your Home Kubernetes Cluster with ARC

What is ARC and Why Should You Care?

Prerequisites

1️⃣ Pre-Setup: Quick Checks

2️⃣ Install ARC Controller

3️⃣ Deploy a Runner Scale Set

4️⃣ Check That It’s Working

5️⃣ Testing It Out

6️⃣ Monitoring (Bonus: Grafana)

6️⃣ Useful Commands for Day-to-Day Ops

Troubleshooting Tips

That’s It! You’re Running GitHub Actions on Your Own Cluster

Building a Home Lab Kubernetes Cluster with Old Hardware and k3s

Why Move From Raspberry Pi to x86?

Network Setup: Keep It Simple (and Reliable)

Getting Ubuntu Server Set Up on Each Node

Installing k3s: Lightweight, Powerful, Familiar

On the Control Plane Node ("master")

On Each Worker Node ("worker-1", "worker-2", etc.)

Troubleshooting and Gotchas

What's Next?

Lessons Learned

Final Thoughts

How Does the Python Virtual Environment Work?

The Problem: Dependency Hell

The Solution: Virtual Environments

How Does It Work?

Command Resolution

Why Is This Powerful?

Advanced Use Cases

Under the Hood: What’s Really Happening?

Debugging Tips

Conclusion

References

Deep Dive - How Chunked Transfer Encoding Works

🤔 What is Chunked Transfer Encoding?

💻 Hands-on Example

🛠️ Step-by-Step Breakdown of Chunking

🙋 What about HTTP/2 and HTTP/3?

Key Takeaways

Further Reading

Upgrading sahansera.dev to Gatsby 5

The Upgrade Journey

Dependency Overhauls

Plugin Upgrades

Styling Challenges

Enabling Nested CSS with styled-jsx

PostCSS Configuration Adjustments

Lessons Learned

Final Thoughts

References

Streaming APIs with FastAPI and Next.js — Part 2

🛠️ Building the Streaming Backend with FastAPI

🚀 Simulating a Real-Time Log Stream

🧠 How It Works

1. Synchronous Generator

2. StreamingResponse

⚙️ Running the Server

💡 Tips for Production

✅ Keep the Stream Alive

🧹 Handle Disconnects Gracefully

🔐 Secure Your Stream

🧪 Testing with Curl

🔗 Hooking it up with the Frontend

🧠 Recap

🧱 Next Steps

🛠️ Useful Links

Streaming APIs with FastAPI and Next.js — Part 1

🔧 The Setup