DEV Community: Jaison D Souza

What Happens When You Build a Database for One Job Only?

Jaison D Souza — Sat, 14 Nov 2026 11:23:26 +0000

In the modern world, anything that moves generates this exact type of data: where something is(space) and when it was there(time).
It could be a runner running around the city, a drone trying to deliver a product, or you wandering through a maze not knowing which direction you are running.

When you need to store this data, the conventional choice is to reach for a general-purpose relational database like PostgreSQL or an embedded store like SQLite. And for many applications, that is a perfectly reasonable choice. General-purpose databases can certainly handle high ingestion rates when tuned properly.

However, general-purpose databases are designed to support a broad range of workloads: mutable tables, random-access updates, secondary indexes, and relational joins.

potDB isn't built on the premise that general-purpose databases are inherently unsuitable. Instead, it explores an alternative: what if your workload is extraordinarily predictable, and you design a lightweight storage engine optimised specifically for that narrower use case?

So, let me introduce you to potDB.

Well, before that, the initial idea for potDB wasn't this. I had an old project lying around, which was a clone of SQLite written in C. Initially, I was thinking of adding some features like crash recovery and transactions to it to make it something more than just a clone. Then I thought, since I'm planning to work on this anyway, why not build my own custom database? So, after snooping around the internet, I decided to build a simple database called potDB. And no, "pot" isn't the pot you might be thinking of, it stands for Position Over Time.

Chapter 1: The Requirements

Let's start with the what we want from this database i.e, requirements.

Functional Requirements:

Strict Mapping: Strictly map timestamps to spatial coordinates: timestamp -> (x, y) or (x, y, z).
Chronological Ingestion: Data always arrives in real-time order. No out-of-order writes or editing past points.
Smart Flushing: Flush data when the buffer is full OR when telemetry time progresses past a configured window , supported by cooperative polling so slow streams don't get trapped in RAM.
Range Queries: Quickly filter points within a specific time window and spatial bounding box without reading the entire file.
Data Integrity: Checksum every chunk to catch corrupted data or torn writes immediately.
Configurable Durability: Let the user choose between extreme speed or maximum safety.
Crash Recovery: If power cuts unexpectedly, already-saved data must never be corrupted, and the database should recover cleanly on restart.

Non-Functional Requirements:

Small Memory Footprint: Must use a lean, predictable memory footprint so it can comfortably fit onto resource-constrained edge devices.
Fast Writes: Ingestion must be fast without being bogged down by complex tree rebalancing.
Zero Third-Party Dependencies: Written in clean C relying solely on standard POSIX system interfaces.
Portable: The database files must follow explicit byte-level serialisation so a file written on an ARM board is readable on an x86 laptop.
Truly Embedded: Lives directly inside the client application as a library with no background database server to manage.

These requirements come from the primary use cases I had in mind when envisioning potDB. For example, the database should easily fit onto a smartwatch to map the route the user is running, or run on low-power, resource-constrained edge devices.

Choosing this niche means recognising that while a general-purpose database can serve most use cases, potDB is specifically designed around having a small memory footprint and working smoothly on edge hardware. Again, this is not built around the idea that it must solve every problem for every use case.

To put its scale into perspective:
We are talking about an engine with a modest, predictable memory footprint that compiles to a compact library, yet can comfortably ingest high volumes of coordinates on platforms ranging from embedded devices to laptops.

Chapter 2: Designing the Database

When designing a database, the first question that comes to mind is: which data structure should we use to store the data?

There are many data structures to consider when it comes to time-series databases, such as Log-Structured Append-Only files, R-Trees, and LSM-Trees. Choosing the right data structure comes down to handling the time axis and the space axis efficiently.

For the time axis, the primary architectural choices are either a Chunked Append-Only Log (grouping sequential samples into immutable contiguous blocks) or a Log-Structured Merge-tree / LSM variant (such as InfluxDB's TSM or Cassandra's SSTables).

LSM-trees and complex merge architectures are very useful when handling out-of-order data and frequent overwrites. But since our timestamps are monotonically increasing in chronological order, out-of-order writes are practically non-existent. Therefore, we will go with a Chunked Append-Only Array.

The Chunked Append-Only Log

The log is the foundation of this database.
potDB organises all data as a strictly sequential, binary append-only stream.
It is append-only because this database has no updates. It doesn't make sense to modify a position that has already happened in the past!

Each data point is represented as a key-value pair:

Key: Timestamp
Value: Spatial coordinates $(x, y)$ or $(x, y, z)$

potDB supports both 2D and 3D coordinates, configured during database initialisation.

This takes care of inserting data. But the next question is: should we just dump everything into one giant flat file and call it a day?
The reading side is still a challenge. If we dump everything into a single massive file, how will we read or query it efficiently?

At the very start of the database file sits a 16-byte File Header, written during database creation. It records the database magic number (0x42445450, which spells 'PTDB' in little-endian), the version, the spatial dimension (2D or 3D), and the time unit.

Following the file header, coordinates are grouped into neat, bite-sized packages called chunks (for example, 500 points at a time).

Every chunk consists of two parts: a Chunk Header (44 bytes for 2D, 52 bytes for 3D) and the Payload (the actual coordinates).

Why do we need each of these fields?

The Chunk Header tells us everything we need to know about what's inside:

Magic Number (2 Bytes): At the very start of every chunk, we store the constant 0x4348(translates for HC in little Endian). If these bytes do not match, the chunk is treated as corrupted.
Count (2 Bytes): Tells us exactly how many coordinates are inside this chunk.
Time Window (16 Bytes): Records the exact starting and ending time of all points in this chunk (start_time and end_time) for fast range pruning.
Bounding Box (MBR) (16 Bytes for 2D / 24 Bytes for 3D): Tracks the lowest and highest coordinates ($X, Y$ for 2D; $X, Y, Z$ for 3D) to draw a tight bounding box around all points in the chunk.
Payload Size (4 Bytes): Tells us the exact number of bytes the coordinate data takes up on disk. If the time window or bounding box don't match what you're looking for, the database can instantly skip over the entire chunk on disk in one leap, without wasting time reading it into RAM.
Checksum (CRC-32) (4 Bytes): Before saving the chunk, we compute an IEEE 802.3 CRC-32 checksum covering both the chunk header metadata (excluding the checksum field itself) and the coordinate payload. If the power cuts while writing or a sector on your storage goes bad, the seal won't match. potDB catches this instantly and safely ignores the bad chunk instead of feeding you garbage data.
The Payload: The contiguous array of binary coordinate points.

Chapter 3: Storage and Persistence

Based on everything we've discussed up to this point, you probably have a clear idea of what and how we are storing data.
But where are we storing it?

Initially, it goes into RAM. But RAM is volatile, anything in RAM vanishes the moment the system shuts down or loses power.
What non-volatile memory comes to our rescue? The disk (or ROM/Flash).

So, problem solved, right? Not quite.
Even though persisting everything immediately seems like the safest approach, writing every single coordinate straight to disk is terribly inefficient due to disk I/O overhead.

Yet, we still need to persist the data safely.
Like every database predecessor before us, we balance both RAM and disk. Remember the chunks we discussed earlier? Leveraging the concept of chunks, we keep an in-memory buffer matching the chunk size. We accumulate incoming coordinates in user-space RAM until the buffer is full, and only then do we flush the entire chunk sequentially. Using write(), we send the chunk to the operating system, which handles writing it to permanent storage.

Chapter 4: Crash Recovery and Durability

What happens if, mid-flight, a drone runs out of battery and crashes? If the chunk buffer is large and the system shuts down before flushing, you lose the trajectory of the drone. I know if a drone crashes, the trajectory might be the least of your worries, but as developers, we should definitely care about protecting that data.

This brings us to crash recovery and data durability.

Traditionally, relational databases like PostgreSQL handle durability using a Write-Ahead Log (WAL). But here, there are no complex transactions or in-place table overwrites requiring a WAL. In fact, the entire database is an append-only log, so why add another log on top of it? We could take a primitive approach and achieve ultra-durability by forcing every single coordinate directly to disk with fsync(), like I did in my early SQLite clone. But it's called primitive for a reason: it kills performance.

potDB simplifies crash recovery through its append-only chunk architecture. Since existing chunks are never modified, recovery primarily involves identifying and validating the last complete chunk.

In addition to capacity-based flushing, potDB checks timestamp progression during point insertion to flush chunks when timestamp - start_time >= max_time_window_ms. Since potDB runs as an embedded library without background threads, slow or idle streams that stop sending points can be explicitly flushed by calling ptdb_poll with the current wall-clock timestamp.

A crash can happen at three distinct stages:

The Three Crash Scenarios

Scenario 1: Crash in Memory (Before Flush)
- Un-flushed points sitting in user-space RAM are lost on a sudden power cut or crash, but already-persisted chunks remain completely intact. Because potDB never overwrites existing data, there are no corrupted B-tree pointers or broken node links.
- You can bound this loss window (e.g., set max_time_window_ms = 1000 for at most 1 second of data in RAM). However, two nuances are important:
  - Event Time vs. Wall-Clock: During active streaming, potDB checks the time difference between the incoming point and the first point in the chunk to decide when to flush. This is based on incoming data, not a background timer.
  - Cooperative Polling: If incoming data stops, a partially filled chunk will not be flushed automatically. Since potDB has no background timer, the host application must periodically call ptdb_poll to check whether the buffer needs to be flushed.
Scenario 2: Crash Mid-Write (Torn Chunks)
- If power cuts halfway through writing a chunk, potDB detects it immediately on reboot:
  1. Header check: Checks header size and magic number.
  2. Payload check: Ensures bytes_read == payload_size.
  3. CRC-32 checksum: Verifies that both the chunk header metadata and payload bytes haven't suffered bit-rot or torn sectors.
- Reader behavior differs by access path: sequential chunk traversal returns explicit error codes, whereas range queries halt cleanly at the last verified chunk.

3. Scenario 3: Crash in Cache (The Durability Dial)

Once write() sends data to the OS page cache, it may not yet be stored permanently. potDB provides three durability modes:

PTDB_SYNC_ALWAYS: Calls fsync() after every chunk flush. A process crash or power failure should not lose successfully synced chunks. The current chunk may be lost if the crash occurs before syncing. (~96.7K pts/sec)
PTDB_SYNC_PERIODIC: Calls fsync() every N flushes. A process crash preserves data already written to the OS, but a power failure can lose all chunks written since the last successful fsync(). (~813K pts/sec at N=10)
PTDB_SYNC_NONE: Relies on OS background writeback. A process crash generally preserves data already written to the page cache, but a power failure can lose any data not yet written to permanent storage. (~4.37M pts/sec)

Benchmarks use 500-point chunks. Actual power-loss protection depends on the OS, filesystem, and storage device.

Crash Recovery: Truncation over Traditional WAL

Traditional relational databases maintain a Write-Ahead Log (WAL) to undo partial writes or replay committed transactions across in-place table pages. Because potDB never overwrites historical chunks in place, it does not need a traditional WAL.

When a crash or system shutdown occurs mid-write, partial trailing bytes may remain on disk. potDB recovers from incomplete writes through file truncation:

If a write error occurs during ptdb_flush(), the engine immediately rolls back the partial chunk using ftruncate().
On database restart, potDB scans sequential chunks from the beginning, verifies header integrity and CRC-32 checksums, and invokes ftruncate() to roll back any torn trailing bytes to the end of the last complete, valid chunk.

Chapter 5: Querying for You

How do you query points inside a spatial bounding box across a specific window of time without scanning gigabytes of data?

Because every chunk in potDB is sealed with a header containing:

The Time Window: [start_time .. end_time]
The Spatial Bounding Box (MBR): [min_x .. max_x, min_y .. max_y] (plus [min_z .. max_z] in 3D)

potDB performs Two-Level Spatiotemporal Chunk Skipping:

Level 1 (Header Pruning): potDB sequentially scans chunk headers (reading 44 bytes for 2D or 52 bytes for 3D). If the chunk's time window or spatial MBR does not intersect your query bounds, it calls lseek() to jump straight over the entire payload without reading coordinates from disk.
Level 2 (Point Filtering): Only if both the time window and bounding box overlap does potDB read the payload into memory, verify its CRC-32 checksum, and stream matching points to your callback.

Chapter 6: Reliability and Concurrency Guarantees

Even though potDB is a specialized, lightweight time-series storage engine without general SQL transactions (BEGIN/COMMIT), it provides specific guarantees around persistence:

Atomicity (Chunk-Level Ingestion): potDB does not implement multi-statement ACID transactions. Instead, chunks serve as the atomic unit of persistence. Partial or torn chunk writes caused by mid-write crashes are rejected by CRC-32 integrity validation on recovery and rolled back via ftruncate(), ensuring incomplete chunks are never visible to readers.
Consistency: Coordinate dimension and time units are locked in the file header at initialisation. Timestamp monotonicity is enforced at runtime during point insertion.
Isolation: Designed for single-writer access without built-in concurrency control or multi-process locking. Range queries read completed chunks up to the last verified boundary, though concurrent uncoordinated writes without external synchronisation can cause transient read errors.
Durability: Configurable per application requirements via different durability modes.

Chapter 7: Benchmarks and Performance

All benchmarks were conducted on a specific hardware configuration. The results are empirical measurements, not universal performance guarantees.

Benchmark Environment

Component	Specification
CPU	AMD Ryzen 7 7840HS
RAM	16 GB DDR5
Storage	1 TB PCIe Gen 4 NVMe SSD
OS	CachyOS Linux, Kernel 7.2.10
Compiler	GCC 16.2.1 (`-std=c99 -Wall -Wextra -D_GNU_SOURCE`)
Dataset	200K–1M synthetic 2D points
Benchmark	Native C (`bench/bench_ptdb.c`)

Synchronization Scope: The ingestion benchmark measures the wall-clock time to insert points, serialize chunk headers, issue write() calls to the OS page cache, and execute fsync() barriers according to the chosen durability mode until ptdb_close() flushes remaining buffers.

1. Architectural Properties

These properties are determined by potDB's binary format and implementation.

Metric	2D	3D	Notes
Logical point data	16 bytes	20 bytes	`timestamp (8B) + coordinates (4B each)`
On-disk stride per point	16 bytes	24 bytes	3D aligned to 8-byte boundary (4B padding)
Chunk header	44 bytes	52 bytes	Packed binary struct (`#pragma pack(1)`)
Buffer (500 points)	8,000 bytes	12,000 bytes	`500 * point_stride` in user-space RAM
Total chunk on disk (500 points)	8,044 bytes	12,052 bytes	`Header + (500 * point_stride)`

Note: A 3D point contains 20 bytes of actual data: an 8-byte timestamp and three 4-byte floats. However, the 64-bit system adds 4 bytes of padding, making the structure 24 bytes. potDB writes all 24 bytes to disk, allowing it to copy the structure directly without extra serialisation.

Storage density: Binary storage uses approximately 50–75% less space than the example JSON and CSV representations.
Memory: The insertion path performs no dynamic allocations after initialisation.
Syscall batching: A 1,000-point chunk requires two write() calls (header and payload), instead of 1,000 individual writes.

2. Ingestion Throughput

Measured ingestion throughput across different chunk capacities and durability modes, using 200,000 2D points:

Chunk Capacity	SYNC_ALWAYS	SYNC_PERIODIC (N=10)	SYNC_NONE
100	19.7K	194.8K	2.97M
500	96.7K	813K	4.37M
1,000	188.1K	1.39M	4.61M
5,000	774.5K	3.25M	5.49M
10,000	1.25M	4.25M	5.29M
50,000	3.05M	4.75M	5.40M

All values are points per second.

Larger chunks amortise synchronisation overhead across more points.

SYNC_ALWAYS: Synchronises after every chunk, providing the strongest synchronisation frequency.
SYNC_PERIODIC: Synchronises every 10 flushes, balancing throughput and durability.
SYNC_NONE: Delegates synchronisation to the operating system's background writeback.

3. Sequential Scan Performance

Tested using 500,000 points stored in 1,000-point chunks.

Metric	Result
Scan time	29.1 ms
Throughput	17.2M points/sec
Data bandwidth	263.32 MB/sec
CRC-32 validation	All 500 chunks verified

The scan was performed against the operating system cache, with CRC-32 verification enabled.

4. In-Memory Accessor Performance

Measured byte-level deserialisation from an already-loaded memory buffer:

Accessor	Time per Point
`ptdb_get_point_2d`	10.7–11.0 ns
`ptdb_get_point_3d`	13.3–15.3 ns

These measurements represent in-memory decoding, not disk access latency.

Benchmark Takeaway

The results demonstrate how chunk capacity and durability configuration influence potDB's performance. Larger chunks significantly improve ingestion throughput, particularly when synchronisation is enabled, while sequential scanning and in-memory decoding remain efficient.

Chapter 8: Limitations and Trade-offs

A specialised engine is defined as much by what it refuses to do as by what it implements. potDB makes deliberate engineering trade-offs:

No Out-of-Order Ingestion: All points must arrive with non-decreasing timestamps. Ingestion of late-arriving or historical out-of-order data is rejected. Applications requiring historical backfilling should consider LSM-based architectures.
No In-Place Point Updates or Deletions: Historical coordinates are immutable. You cannot modify or delete an individual point recorded ten minutes ago.
No Generalised Relational Schema: potDB stores only timestamps and spatial coordinates. It does not support arbitrary columns, dynamic metadata attributes, strings, or relational joins.
Single-Writer Concurrency: The engine does not provide multi-process writer coordination or file locking. Ingestion must be handled by a single dedicated writer process or thread.
Sequential Header Scanning: Without the planned sparse index, range queries must read every chunk header sequentially, making query performance proportional to the total number of chunks.

These limitations are deliberate design decisions tailored for predictable, embedded spatial telemetry.

Chapter 9: Future Improvements & Planned Features

potDB is already fast and functional, but building a database is a never-ending rabbit hole. Here are the features and improvements I'm planning to add down the road:

Binary Search on Time (Sparse Index):
Right now, time queries check chunk headers one by one. Since time is strictly sorted, building a tiny in-memory index of chunk start times will allow us to use binary search, jumping directly to the requested time window without sequentially scanning preceding headers.
Spatial Grid Index:
For bounding box queries across long periods, we still have to check every chunk's MBR. Adding a simple coarse spatial grid (dividing space into large tiles) or a space-filling curve (like Z-order) will let us look up only the chunks touching that specific geographic area.
Delta Compression (Gorilla-style):
Coordinates and timestamps from moving objects often change gradually between consecutive samples. Using delta compression (storing differences between consecutive points instead of full raw values) could noticeably reduce on-disk file size with modest CPU overhead.
Circular Ring-Buffer & Auto-Retention (TTL):
On small devices with limited flash storage (like smartwatches or micro-controllers), disk space is precious. Adding an automatic retention mode to prune chunks older than $N$ days, or rolling over a fixed-size ring-buffer will prevent disks from filling up.
First-Party Language Packages:
Packaging potDB into standard package managers (pip install potdb for Python data science, npm install potdb for Node/Next.js, and an official Go module) so developers can embed it into their favourite stack without manual C linking.

Conclusion

Designing systems is a fundamental part of software engineering, especially in the era of AI-assisted coding. Taking the time to design things from scratch helps you understand how systems truly work under the hood.

I learned an enormous amount while building potDB. And I am not a seasoned database architect who knows everything. These are my thoughts, experiments, and architectural decisions. If you have any suggestions, feedback, or corrections, please feel free to drop a comment.

I hope you learned something new today!

Check out the introductory site and interactive live demo: https://potdb.vercel.app/

Codebase: https://github.com/jsndz/potdb

How HTML works? In Simple words.

Jaison D Souza — Thu, 19 Feb 2026 18:30:52 +0000

HTML is a markup language responsible for the structure of a web page.
The skeleton of a web page is HTML.

Instead of talking about various tags and how to use them, let us understand how HTML works, how it renders, and how the browser displays it.

When a client asks a server for a web page (this happens when you load a link), the server sends an HTML text file.
It is a plain text file, nothing else.

<html>
  <head>
    <title> My Web Page </title>
  </head>
  <body>
    <p> Hello World </p>
  </body>
</html>

The browser cannot directly render this raw text in a meaningful way.
So it must convert the text into internal structures that it understands.

I will briefly explain the steps involved.

Tokenization and State Machine

Step 1: Tokenization

The browser reads the HTML character by character and converts it into tokens.
Tokens follow a set of structured rules defined by the HTML specification.

The HTML code shown earlier would be converted into tokens like:

StartTag: html
StartTag: head
StartTag: title
Text: My Web Page
EndTag: title
EndTag: head
StartTag: body
StartTag: p
Text: Hello World
EndTag: p
EndTag: body
EndTag: html

Why is this conversion needed, and how does it happen?

This process is implemented using a state machine.

The diagram shows multiple states with transitions based on characters such as <, >, /, and text content. These transitions help the parser identify opening tags, closing tags, and text nodes.

The state machine follows strict rules to convert the input stream into tokens.
Although it may look complicated, it is essentially a series of conditional transitions (similar to if-else logic).

Why do we need tokens?
Tokens make it easier to convert the HTML into a DOM because each token clearly represents what kind of element it is.

DOM, CSSOM, and Render Tree

Step 2: Tokens → DOM

The next step is building the DOM (Document Object Model).

The tokens are processed and converted into a tree structure called the DOM.
Each node in the tree represents an element, text node, or attribute from the HTML document.

A hierarchical structure starting with the html root node, branching into head (containing title → My Web Page) and body (containing p → Hello World).

Step 3: DOM + CSSOM → Render Tree

When CSS is added, the browser parses it into another structure called the CSSOM (CSS Object Model).

DOM contains structure and content
CSSOM contains styling rules

The browser combines the DOM and CSSOM to build the Render Tree.

The Render Tree contains only visible elements and the styles that apply to them (for example, elements with display: none are excluded).

The Render Tree defines:

what needs to be displayed
how it should be styled

Note on JavaScript:
When JavaScript modifies the DOM, for example:

document.body.innerHTML += "<div>Hello</div>"

the browser may trigger layout recalculation (reflow) and repainting.
However, the entire tree is not always re-rendered—only the affected parts are updated when possible.

Layout and Paint

Step 4: Layout Calculation (Reflow)

In this step, the browser calculates the exact position and size of each node in the Render Tree.

For every node, the browser computes:

width
height
position

This process is recursive and can be expensive, especially when changes affect large portions of the page.

Layout calculation produces precise instructions required to generate the final visual output.

Step 5: Paint

During the paint phase, the browser converts layout information into actual pixels.

This includes painting:

text
colors
borders
backgrounds
shadows

In simple terms:

Layout → Paint → Pixels on the screen

Optional next step (for completeness)

Modern browsers often perform an additional step called Compositing, where painted layers are combined and optimized (often using the GPU) before being displayed on the screen.

That's all. Thank you reading till the end. You might have a basic idea about how HTML and small part of browser works. These were my learning based on studying the working of HTML. If any mistakes are there fell free to correct me.

Ditch JSON, Go Binary: A Simple Message Protocol

Jaison D Souza — Sun, 24 Aug 2025 10:17:49 +0000

Binary Message Protocol — that's what I am calling this method. Sounds cool right? Well maybe not.

Fancy name, but it’s really just a simple method of sending data by message framing.

Problem

In real-time applications, speed and efficiency are critical. Sending large JSON payloads over WebSockets can add unnecessary overhead, making communication slower and more resource-intensive.

At the same time, the server needs a clear way to identify what kind of message it's receiving. For example, whether it's a request to join a session, an update, or some other action.

When I was building a collaboration platform with TypeScript and Golang, I ran into this exact problem: JSON was too heavy for real-time communication, and I needed a faster, lighter protocol that still allowed the server to understand the type of each message.

JSON Examples

Joining:

{
  "type": "join",
  "payload": { "token": "...", "doc_id": "..." }
}

Updating a document:

json { "type": "update", "payload": { "doc_id": "abc123", "delta": [1, 0, 2, 3] } } // The delta could be very large

Looks fine — but that's a lot of extra characters being sent on every message.

Solution: Binary Message Protocol

A simple method where:

First byte = message type
Remaining bytes = payload

That’s it.

So instead of sending the verbose JSON above, we just send something like:

[2, 21, 1212, 1, 21, 2, 4, 243, 544, 4, 5, 67, 7]

Here:

2 = message type (join)
The rest = encoded payload

Message Type Mapping

Here’s an example mapping:

Byte	Type	Description
1	update	Document update (deltas)
2	join	Join a document session
3	leave	Leave a document session
4	ping	Keep-alive / heartbeat
5	auth	Authentication / validation

Encoding & Decoding

This isn’t just theory — here’s how you could implement it.

Encoding in Go

go func (c *Client) HandleNotification(msgType uint, message string) { msgBytes := []byte(message) payload := make([]byte, 1+len(msgBytes)) payload[0] = byte(msgType) copy(payload[1:], msgBytes) c.Send <- payload }

Decoding in TypeScript

ts function parseMessage(message: Uint8Array): { type: number; payload: Uint8Array; } { const type = message[0]; const payload = message.slice(1); return { type, payload }; }

Sending as Binary over WebSocket

In Go, you just tell the WebSocket library that you're writing binary messages:

go func (c *Client) WritePump() { for msg := range c.Send { // websocket.BinaryMessage ensures data is sent as binary if err := c.Conn.WriteMessage(websocket.BinaryMessage, msg); err != nil { log.Println("write error:", err) return } } }

Why Not JSON? Why Not Protobuf?

JSON → Easy to use but heavy.
Protobuf → Lightweight but complex.
Binary Message Protocol → Minimal, custom, and just enough for real-time collaboration.

Closing Thoughts

This might not be advanced, and many senior engineers may find it straightforward, but I just wanted to share what I learned. Hopefully, you found it useful too.

Project

🔗 Kairo on GitHub

Build a GitHub App in 5 Minutes with Probot

Jaison D Souza — Tue, 29 Jul 2025 15:28:13 +0000

Build a GitHub App in 5 Minutes with Probot

In this article I will show you how to build your own GitHub app in 5 minutes. You can put this in your portfolio, show a friend or just do it for fun. Without any further ado, let’s begin.

Prerequisites

ANY code editor.
Node.js installed.
Little bit of programming knowledge. Just a little.
Oh, and a GitHub account.

Step 1: Setup the project

Open a code editor. I will use VS Code. Then open a terminal.

Run this command:

npx create-probot-app app_name

Enter the name, description, and your name. Then it will ask you to choose a template.

If you’re using JavaScript, select basic-js; for TypeScript, choose basic-ts — which is what I'll use in this example.

There are also some built-in templates like checks, deploy, and git-data, which you can explore if you'd like.

Step 2: Initialize your Smee client

Visit https://smee.io/new to generate a channel and copy the URL.

Open terminal and run the following command:

bash npm install --global smee-client

Then open .env.example file and rename it to .env.

Paste the URL from Smee in front of WEBHOOK_PROXY_URL.

You might be wondering what to paste in APP_ID?
We will do that in the next step.

Step 3: Initialize GitHub App

Go to: https://github.com/settings/apps/new

Fill out the name, description, and homepage URL of your app. If you don’t have a homepage URL, just give your GitHub page.

Webhook URL: Use the same WEBHOOK_PROXY_URL from the previous step.
Webhook Secret: Use whatever you set in your .env file.
Permissions: Choose permissions required for your app. If unsure, use checks, contents, issues, push, and pull requests. You can change them later.

Click on Create GitHub App. You will be redirected.

Copy the App ID and paste it into your .env.

Scroll down to the Private Keys section.

Click on Generate a Private Key.

A .pem file will be downloaded. Copy the file into your project folder.

Rename it to github_key.pem.

Remember to add github_key.pem and .env files to .gitignore.

Paste this into the .env file:

env PRIVATE_KEY_PATH="./github_key.pem"

Then in terminal, open your app folder and run:

bash cd app_name npm start

Final Step: Add a Simple Feature

You have successfully created your GitHub App. You can now modify the code to respond to events like push, pull_request, or issues.

Here’s an example that comments "thank you for raising the issue!" on every new issue:

`ts
import { Probot } from "probot";

export default (app: Probot) => {
app.on("issues.opened", async (context) => {
const issueComment = context.issue({
body: "thank you for raising the issue!",
});
await context.octokit.issues.createComment(issueComment);
});
};
`

That’s how you build your own GitHub App with Probot — quick and easy!

Check It Out

I used this same method to build Codexa — a GitHub Copilot-like tool for reviewing pull requests.
Feel free to check out the code and see how it works.

Code: github.com/jsndz/blogs/tree/master/tutorial
Wanna connect? Reach out on LinkedIn
Also, check out more of my projects on GitHub