DEV Community: Siddhesh Khandagale

A must read if you are passionate about cryptography or security.

Siddhesh Khandagale — Thu, 06 Feb 2025 12:15:16 +0000

Splitting Secrets Like a Spy: Shamir’s Secret Sharing in Go

Siddhesh Khandagale ・ Feb 6

#go #cryptography #shamirsecretsharing #security

Splitting Secrets Like a Spy: Shamir’s Secret Sharing in Go

Siddhesh Khandagale — Thu, 06 Feb 2025 12:12:40 +0000

Introduction

Ever watched a movie where a nuclear launch code is split among multiple people, ensuring no single person has full control? Well, that’s Shamir’s Secret Sharing (SSS) in action! 🕵️‍♂️✨

A while back, I built a CLI app using Shamir’s Secret Sharing in Go, allowing secure file sharing. The flow was simple: upload a file, define the number of shares and threshold, and assign them to specific people through the app. To reconstruct the file, shareholders had to collect enough shares and input them into the CLI. If the threshold was met with correct shares, the encrypted file was recovered. Pretty cool, right? 🔥

You can read more about this CLI app here 👉 link

Now, let’s break down how this powerful cryptographic method works and how you can implement it in Go! 🚀

What is Shamir’s Secret Sharing?

Shamir’s Secret Sharing (SSS) is a cryptographic method where a secret is split into multiple shares. Only a certain number of shares (threshold) are needed to reconstruct the original secret. Think of it like splitting a treasure map into different pieces — nobody can find the treasure unless they have enough parts! 🏴‍☠️

Example:

You split your private key into 5 shares.
You set a threshold of 3 (minimum required to reconstruct it).
Even if 2 shares are lost, the key is still recoverable!

Implementing Shamir’s Secret Sharing in Go

We’ll use github.com/hashicorp/vault/shamir, a robust Go package that makes splitting and combining secrets effortless.

Step 1: Installing the Package

go get github.com/hashicorp/vault/shamir

Step 2: Splitting the Secret

package main

import (
 "fmt"
 "github.com/hashicorp/vault/shamir"
)

func main() {
 secret := []byte("super-secret-key") // Your top-secret data
 nShares := 5  // Total shares to generate
 threshold := 3 // Minimum shares required to reconstruct

 shares, err := shamir.Split(secret, nShares, threshold)
 if err != nil {
  panic(err)
 }

 fmt.Println("Generated Shares:")
 for i, share := range shares {
  fmt.Printf("Share %d: %x\n", i+1, share)
 }
}

✅ How It Works:

Takes a secret (e.g., private key, password, etc.).
Splits it into nShares (total number of shares).
Requires a threshold (minimum shares needed to reconstruct the secret).
Prints out each generated share.

Step 3: Reconstructing the Secret

package main

import (
 "fmt"
 "github.com/hashicorp/vault/shamir"
)

func main() {
 // Use any 3 shares from the previous output
 shares := [][]byte{
  {0x01, 0xA2, 0x45, 0x56, 0x78},
  {0x02, 0xB3, 0x67, 0x89, 0x90},
  {0x03, 0xC4, 0x89, 0xAB, 0xBC},
 }

 secret, err := shamir.Combine(shares)
 if err != nil {
  panic(err)
 }

 fmt.Println("Reconstructed Secret:", string(secret))
}

✅ How It Works:

Uses at least 3 shares to reconstruct the original secret.
If fewer than 3 shares are used, the reconstruction fails (that’s the security!).

Why is This Useful? 🤔

🔹 Securely store private keys — Avoid a single point of failure.
🔹 Distribute credentials safely — Team-based authentication.
🔹 Decentralized access control — Ensures secrets can’t be accessed by a single person.
🔹 Disaster recovery — Lost a part of your key? No problem!

Conclusion

We just built a spy-level secret-sharing system in Go! 🎩🚀 Whether you’re handling encryption keys, sensitive files, or secure authentication, Shamir’s Secret Sharing is an incredible way to safeguard data against loss or theft.

Would you use this in real life? Or build something even crazier? Let me know! 🔥

Building a Self-Destructing CLI Tool in Go 🚀💣

Siddhesh Khandagale — Tue, 04 Feb 2025 06:37:37 +0000

Ever wondered what it feels like to build a program that deletes itself after execution? Sounds crazy, right? Well, today, we are going to do just that, build a Self-Destructing CLI Tool in Go!

This is a fun experiment, but it also introduces some cool file handling and process management techniques in Go. Let’s get started!

How It Works
The idea is simple:

The Go program executes normally.
Before exiting, it deletes its own binary file.
Once destroyed, the program is gone forever (unless you recompile it). 🤯

Writing the Self-Destructing Go Code

1. Initialize a Go Project
First, create a new project directory and initialize a Go module:

mkdir self_destruct && cd self_destruct
go mod init self_destruct

Create a file named self_destruct.go and add the following code:

package main

import (
 "fmt"
 "os"
 "os/exec"
)

func main() {
 fmt.Println("💀 Running self-destructing program...")

 // Get the current executable path
 exePath, err := os.Executable()
 if err != nil {
  fmt.Println("Failed to get executable path:", err)
  return
 }

 // Print the file path before deletion
 fmt.Println("Deleting:", exePath)

 // Create a command to delete the file
 if runtime.GOOS == "windows" {
     cmd = exec.Command("cmd", "/C", "del", exePath)
 } else {
     cmd = exec.Command("sh", "-c", fmt.Sprintf("rm -f %s", exePath))
 }
 cmd.Start() // Start but don't wait to prevent blocking
}

Let’s break down what this code does:

os.Executable() returns the absolute path of the currently running executable.
fmt.Println("Deleting:", exePath): Prints the file path before deletion so the user can see what’s happening.
exec.Command("cmd", "/C", "del", exePath) creates a command to delete the executable file.
.Start() runs the command asynchronously to ensure the program doesn't hang while deleting itself.

Running the Program

Compile the Program : go build .
Run the Program : ./self_destruct

The binary file deletes itself after execution! Try running it again you’ll see it’s gone.

While this was a fun experiment, there are some real-world applications:

Security & Privacy — Temporary scripts that auto-delete after execution.
Cleanup Mechanisms — Self-cleaning build artifacts.
One-Time Setup Scripts — Ensure scripts don’t run twice.

This was a crazy but entertaining experiment with Go’s file handling and process execution.

Would you ever use this in real life? Or have any other wacky Go ideas? Let me know! 🚀🔥

A Guide on Building a Concurrent Key-Value store using Golang

Siddhesh Khandagale — Sat, 01 Feb 2025 06:49:09 +0000

How to Build a Concurrent Key-Value Store in Go

Siddhesh Khandagale ・ Jan 31

#go #database #keyvaluestore #tutorial

How to Build a Concurrent Key-Value Store in Go

Siddhesh Khandagale — Fri, 31 Jan 2025 15:23:36 +0000

Key-Value stores are one of the simplest yet most effective forms of data storage used in modern applications.

In this tutorial, we will walk through building NanoKV, a minimalist in-memory key-value store written in Go, using the Fiber web framework.

By the end of this tutorial, you’ll understand:

How key-value stores work under the hood
Implementing basic CRUD operations in Go
Using concurrency and TTL (Time-To-Live) for ephemeral data storage
Running the application in a Docker container

Prerequisites

Before we dive in, ensure you have the following:

Go installed (Download here)
Basic understanding of Go programming
Docker installed (optional, for containerization)

Getting Started

Let’s start by setting up our project structure.

1. Initialize a Go Project

First, create a new project directory and initialize a Go module:

mkdir NanoKV && cd NanoKV
go mod init NanoKV

2. Install Dependencies

We’ll use Fiber for building the API. Install it using:

go get github.com/gofiber/fiber/v2

3. Project Structure

Your project should have the following structure:

NanoKV/
│── kvstore/
│   └── kvstore.go
│── main.go
│── go.mod
│── go.sum
│── Dockerfile

Implementing the Key-Value Store

1. Defining the Store Structure

Create a new file kvstore/kvstore.go and define the data structure:

package kvstore

import (
    "sync"
    "time"
)

type Data struct {
    value      string
    expiration time.Time
}

type KeyValueStore struct {
    mu   sync.RWMutex
    data map[string]Data
}

func NewKeyValueStore() *KeyValueStore {
    return &KeyValueStore{
        data: make(map[string]Data),
    }
}

we use a map to store the actual key-value pairs. A map in Go is a built-in data structure that allows us to associate keys with values efficiently. However, since our key-value store is meant to be accessed concurrently (multiple goroutines may try to read or write to it at the same time), we need a mechanism to ensure safe concurrent access.

This is where sync.RWMutex comes in. It is a read-write mutual exclusion lock that allows multiple goroutines to read data simultaneously but ensures that only one goroutine can write at a time. This helps prevent race conditions and maintains data consistency.

sync.RWMutex has two primary operations:
RLock() and RUnlock() are used for read operations, allowing multiple readers to access the data simultaneously.
Lock() and Unlock() are used for write operations, ensuring that only one goroutine can modify the data at any given time.

By wrapping our key-value store operations (such as Set, Get, and Delete) with these locks, we guarantee thread safety while maintaining high performance for concurrent reads.

2. Implementing CRUD Operations with TTL Support

Next, we implement methods to set, get, and delete key-value pairs.

func (kv *KeyValueStore) Set(key, val string, ttl time.Duration) {
    kv.mu.Lock()
    defer kv.mu.Unlock()

    kv.data[key] = Data{
        value:      val,
        expiration: time.Now().Add(ttl),
    }
}

func (kv *KeyValueStore) Get(key string) (string, bool) {
    kv.mu.Lock()
    defer kv.mu.Unlock()
    val, ok := kv.data[key]

    // Check for Item Expiry
    if val.expiration.IsZero() || time.Now().Before(val.expiration) {
        return val.value, ok
    }

    // Delete if Expired
    delete(kv.data, key)
    return "", false
}

func (kv *KeyValueStore) Delete(key string) bool {
    kv.mu.Lock()
    defer kv.mu.Unlock()

    _, ok := kv.data[key]
    if !ok {
        return false
    }
    delete(kv.data, key)
    return true
}

Set Method: Stores a key-value pair with an expiration time. If the TTL (time-to-live) is set, it calculates the expiration timestamp. The function locks the store before writing and unlocks after completion.
Get Method: Retrieves a value from the store. If the key exists and has not expired, it returns the value. If expired, it removes the key and returns false.
Delete Method: Removes a key-value pair from the store. If the key exists, it deletes it and returns true; otherwise, it returns false.
Concurrency Handling: A read-write mutex (sync.RWMutex) ensures thread safety by preventing race conditions when multiple goroutines access the store.

Exposing an HTTP API with Fiber

To make our key-value store accessible via HTTP, we set up a simple REST API using Fiber.

1. Setting Up the HTTP Server

Create a main.go file and add the following:

package main

import (
    "time"
    "NanoKV/kvstore"
    "github.com/gofiber/fiber/v2"
)

func main() {
    app := fiber.New()
    kv := kvstore.NewKeyValueStore()

    app.Get("/", func(c *fiber.Ctx) error {
        return c.SendString("This is a Simple Key-Value store like Redis in Go.")
    })

    app.Get("/get/:key", func(c *fiber.Ctx) error {
        key := c.Params("key")
        value, ok := kv.Get(key)
        if !ok {
            return c.SendString("The Key " + key + " doesn't exist")
        }
        return c.SendString("The Key " + key + " has Value " + value)
    })

    app.Post("/set/:key/:value", func(c *fiber.Ctx) error {
        key := c.Params("key")
        value := c.Params("value")
        kv.Set(key, value, 10*time.Minute)
        return c.SendString("Key " + key + " Value " + value)
    })

    app.Delete("/delete/:key", func(c *fiber.Ctx) error {
        key := c.Params("key")
        ok := kv.Delete(key)
        if !ok {
            return c.SendString("The Key " + key + " doesn't exist")
        }
        return c.SendString("Successfully Deleted!!")
    })

    app.Listen(":3000")
}

GET /get/:key: Fetches the value of a given key if it exists; otherwise, returns a message indicating the key does not exist.
POST /set/:key/:value: Stores a key-value pair with a default TTL of 10 minutes.
DELETE /delete/:key: Deletes a key from the store if it exists.

Running the Server

Run the application:

go run main.go

Test it using curl:

curl -X POST http://localhost:3000/set/foo/bar
curl -X GET http://localhost:3000/get/foo
curl -X DELETE http://localhost:3000/delete/foo

Containerizing the Application with Docker

We can containerize our Go application using the following Dockerfile:

FROM golang:alpine

WORKDIR /app

COPY go.mod ./
COPY go.sum ./
RUN go mod download

COPY . ./

RUN go build -o NanoKV .

EXPOSE 3000

CMD ["/app/NanoKV"]

Building and Running the Docker Container

docker build -t nanokv .
docker run -p 3000:3000 nanokv

You can get the complete code here.

Conclusion

We have successfully built a lightweight in-memory key-value store in Go, complete with:

- Basic CRUD operations
- Concurrency handling
- TTL-based expiration
- An HTTP API using Fiber
- Containerization with Docker

You can further enhance this by adding features like persistent storage, distributed setup, or enhanced performance optimizations.

That’s it for this Tutorial. If this blog was helpful to you do share it and to get more information about Golang concepts, projects, etc. and to stay updated on the Tutorials do follow Siddhesh on Twitter and GitHub.

Until then Keep Learning, Keep Building 🚀🚀

Here's the new Step-by-Step tutorial for building a URL Shortner from scratch using Go, Redis and Docker.

Siddhesh Khandagale — Wed, 29 Jan 2025 16:20:23 +0000

URL Shortening Service using Go

Siddhesh Khandagale ・ Jan 29

#go #api #backend #tutorial

URL Shortening Service using Go

Siddhesh Khandagale — Wed, 29 Jan 2025 16:15:58 +0000

Ever wondered how services like Bitly turn long, ugly URLs into short, shareable links?

In this tutorial, I’ll walk you through building a URL Shortener from scratch using Go, Redis and Docker. If you’re a budding developer, this project is a fantastic way to level up your skills and learn practical web development.

What You’ll Learn

Setting up a Go project with modules.
Building RESTful APIs in Go.
Using Redis for quick key-value storage.
Organizing Go code for readability and scalability.

Prerequisites / Installation:

Docker
Docker Desktop ( Install Docker Desktop on your system)

Step 1: Initialize Your Project

Start by creating a directory for your project and initializing a Go module:

mkdir url-shortener
cd url-shortener
go mod init github.com/<username>/url-shortener

This will create a go.mod file to manage dependencies.

Step 2: Set Up the Project Structure

To keep things clean and modular, create the following folder structure:

url-shortener/
├── handlers/         # API logic for handling requests
│   └── handlers.go
├── models/           # Data models
│   └── url.go
├── router/           # Routing setup
│   └── router.go
├── storage/          # Redis interactions
│   └── redis-store.go
├── main.go           # Application entry point
├── Dockerfile        # Docker configuration
├── docker-compose.yml
└── go.mod            # Go module file

Step 3: Install Dependencies

We’ll use the following Go packages:

github.com/go-redis/redis/v8 for Redis interactions.
github.com/gorilla/mux for routing.

Install them using:

go get github.com/go-redis/redis/v8
go get github.com/gorilla/mux

Step 4: Define Data Models

Create models/url.go to define the data structures for request and response:

package models

type ShortenRequest struct {
 URL string `json:"url"` // URL to be shortened
}

type ShortenResponse struct {
 ShortURL string `json:"short_url"` // Generated short URL
}

The ShortenRequest struct represents the input, and ShortenResponse defines the output format.

Step 5: Configure Redis Storage

In storage/redis-store.go, create a RedisStore struct to handle Redis operations:

package storage

import (
 "context"
 "fmt"
 "hash/fnv"
 "net/url"
 "strings"

 "github.com/go-redis/redis/v8"
)

// Context for Redis operations
var ctx = context.Background()

type RedisStore struct {
 Client *redis.Client
 // client *redis.Client
}

// NewRedisStore initializes a new Redis client and returns a RedisStore
func NewRedisStore() *RedisStore {
 rdb := redis.NewClient(&redis.Options{
  Addr: "localhost:6379",
  DB:   0,
 })

 return &RedisStore{Client: rdb}
}

func NewTestRedisStore() *RedisStore {
 rdb := redis.NewClient(&redis.Options{
  Addr: "localhost:6379",
  DB:   1,
 })

 return &RedisStore{Client: rdb}
}

// SaveURL stores the original URL and its shortened version in Redis
func (s *RedisStore) SaveURL(originalURL string) (string, error) {
 // Check if the URL already exists
 shortURL, err := s.Client.Get(ctx, originalURL).Result()
 if err == redis.Nil {
  // URL not found, generate a new short URL
  shortURL = s.generateShortURL(originalURL)
  err = s.Client.Set(ctx, originalURL, shortURL, 0).Err()
  if err != nil {
   return "", err
  }

  // Store the original URL and the short URL in Redis
  err = s.Client.Set(ctx, shortURL, originalURL, 0).Err()
  if err != nil {
   return "", err
  }

  // Increment the domain count in Redis
  domain, err := s.getDomain(originalURL)
  if err != nil {
   return "", err
  }

  err = s.Client.Incr(ctx, fmt.Sprintf("domain:%s", domain)).Err()
  if err != nil {
   return "", err
  }

 } else if err != nil {
  return "", err
 }

 return shortURL, nil
}

// GetOriginalURL retrieves the original URL from Redis using the short URL
func (s *RedisStore) GetOriginalURL(shortURL string) (string, error) {
 originalURL, err := s.Client.Get(ctx, shortURL).Result()
 if err == redis.Nil {
  return "", fmt.Errorf("URL not found")
 } else if err != nil {
  return "", err
 }

 return originalURL, nil
}

// GetDomainCounts retrieves the counts of shortened URLs per domain from Redis
func (s *RedisStore) GetDomainCounts() (map[string]int, error) {
 keys, err := s.Client.Keys(ctx, "domain:*").Result()
 if err != nil {
  return nil, err
 }

 domainCounts := make(map[string]int)

 for _, key := range keys {
  count, err := s.Client.Get(ctx, key).Int()
  if err != nil {
   return nil, err
  }

  domain := strings.TrimPrefix(key, "domain:")
  domainCounts[domain] = count
 }

 return domainCounts, nil
}

// generateShortURL creates a shortened URL string using a hash function
func (s *RedisStore) generateShortURL(originalURL string) string {
 h := fnv.New32a()
 h.Write([]byte(originalURL))
 return fmt.Sprintf("%x", h.Sum32())
}

// getDomain extracts the domain name from a URL
func (s *RedisStore) getDomain(originalURL string) (string, error) {
 parsedURL, err := url.Parse(originalURL)
 if err != nil {
  return "", err
 }

 return strings.TrimPrefix(parsedURL.Host, "www."), nil
}

func (s *RedisStore) FlushTestDB() {
 s.Client.FlushDB(ctx)
}

SaveURL: checks if a URL exists and generates a short URL if it doesn’t.
GetOriginalURL: Retrieves the original URL from the short URL.
GetDomainCounts: Fetches statistics for the top domains.
generateShortURL: hashes the original URL using FNV-32. (fnv.New32a() generates a 32-bit hash for a given URL.)
getDomain: Extracts the domain name from a URL.

Why FNV-1a?

It’s a fast, non-cryptographic hash function.
Generates a short, deterministic hash for a given input.
Perfect for URL shortening since we don’t need cryptographic security.
Reduces collisions compared to basic modulo-based hashing.

Step 6: Build Handlers

Create handlers/handlers.go to implement API logic:

package handlers

import (
 "encoding/json"
 "net/http"
 "sort"

 "github.com/Siddheshk02/url-shortener/models"
 "github.com/Siddheshk02/url-shortener/storage"
 "github.com/gorilla/mux"
)

// Initialize the store as a RedisStore
var store = storage.NewRedisStore()

func ShortenURL(w http.ResponseWriter, r *http.Request) {
 // Decode the request body into a ShortenRequest struct
 var req models.ShortenRequest
 if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
  http.Error(w, err.Error(), http.StatusBadRequest)
  return
 }

 // Save the URL using the store
 shortURL, err := store.SaveURL(req.URL)
 if err != nil {
  http.Error(w, err.Error(), http.StatusInternalServerError)
 }

 // Respond with the shortened URL
 res := models.ShortenResponse{ShortURL: shortURL}
 json.NewEncoder(w).Encode(res)
}

func RedirectURL(w http.ResponseWriter, r *http.Request) {
 // Get the short URL from the request variables
 vars := mux.Vars(r)
 shortURL := vars["shortURL"]

 // Get the original URL from the store
 originalURL, err := store.GetOriginalURL(shortURL)
 if err != nil {
  http.Error(w, err.Error(), http.StatusNotFound)
  return
 }

 // Redirect to the original URL
 http.Redirect(w, r, originalURL, http.StatusMovedPermanently)
}

func GetTopDomains(w http.ResponseWriter, r *http.Request) {
 // Get the domain counts from the store
 domainCounts, err := store.GetDomainCounts()
 if err != nil {
  http.Error(w, err.Error(), http.StatusInternalServerError)
  return
 }

 // Sort the domains by count
 type kv struct {
  Key   string
  Value int
 }

 var sortedDomains []kv
 for k, v := range domainCounts {
  sortedDomains = append(sortedDomains, kv{k, v})
 }

 sort.Slice(sortedDomains, func(i, j int) bool {
  return sortedDomains[i].Value > sortedDomains[j].Value
 })

 // Prepare the top 3 domains to return
 topDomains := make(map[string]int)
 for i, domain := range sortedDomains {
  if i >= 3 {
   break
  }
  topDomains[domain.Key] = domain.Value
 }

 // Respond with the top 3 domains
 json.NewEncoder(w).Encode(topDomains)
}

ShortenURL: Handles URL shortening requests.
RedirectURL: Redirects users to the original URL.
GetTopDomains: Returns the most frequently shortened domains.

Step 7: Set Up Routes

In router/router.go, define all the API routes:

package router

import (
 "github.com/Siddheshk02/url-shortener/handlers"
 "github.com/gorilla/mux"
)

func SetupRouter() *mux.Router {
 // Initialize a new router
 r := mux.NewRouter()

 // Define route for getting the top domains
 r.HandleFunc("/metrics", handlers.GetTopDomains).Methods("GET")

 // Define route for shortening URLs
 r.HandleFunc("/shorten", handlers.ShortenURL).Methods("POST")

 // Define route for redirecting to the original URL
 r.HandleFunc("/{shortURL}", handlers.RedirectURL).Methods("GET")

 return r
}

/shorten (POST): Shortens a given URL.
/{shortURL} (GET): Redirects to the original URL using the short URL.
/metrics (GET): Returns the top 3 most-used domains.

Step 8: Main Application Entry Point

In main.go, initialize the router and start the HTTP server:

package main

import (
 "log"
 "net/http"

 "github.com/Siddheshk02/url-shortener/router"
 "github.com/gorilla/mux"
)

func main() {
 // Setup the router from the router package
 r := router.SetupRouter()

 // Log all registered routes
 r.Walk(func(route *mux.Route, router *mux.Router, ancestors []*mux.Route) error {
  path, err := route.GetPathTemplate()
  if err != nil {
   return err
  }
  log.Println("Registered route:", path)
  return nil
 })

 log.Println("Starting server on 8080")
 log.Fatal(http.ListenAndServe(":8080", r))
}

router.SetupRouter: Initializes all routes.
Route Logging: Lists all registered routes for debugging.
HTTP Server: Runs the app on port 8080.

Step 9: Dockerize the Application

Create a Dockerfile to containerize the app:

# Start with the official Golang image
FROM golang:1.22-alpine

# Set the working directory inside the container
WORKDIR /app

# Copy the Go modules files first and download dependencies
COPY go.mod go.sum ./
RUN go mod download

# Copy the rest of the application code
COPY . .

# Build the application
RUN go build -o url-shortener

# Expose the port on which the app will run
EXPOSE 8080

# Command to run the executable
CMD ["./url-shortener"]

The Dockerfile builds the Go application into a Docker image and sets it up to run on port 8080.

Step 10: Create a docker-compose.yml File

Use docker-compose.yml to define both the app and the Redis service:

version: '3'

services:
  url-shortener:
    image: siddheshk02/url-shortener:latest
    ports:
      - "8080:8080"
    depends_on:
      - redis
    environment:
      - REDIS_HOST=redis
    networks:
      - app-network

  redis:
    image: redis
    ports:
      - "6379:6379"
    networks:
      - app-network

networks:
  app-network:
    driver: bridge

url-shortener Service: Runs the Go app and depends on Redis.
redis Service: Provides the Redis database for storage.
network: Ensures communication between the two services.

Step 11: Run the Application

Build and run the application using Docker Compose:

docker-compose up --build

Output:

The Go app runs on http://localhost:8080.
Redis is available on port 6379.

Step 12: Test the API Endpoints

Use Postman, cURL, or a browser to test the service:

Shorten a URL:

curl -X POST -H "Content-Type: application/json" \
-d '{"url": "https://www.example.com"}' \
http://localhost:8080/shorten

Response:

{
    "short_url": "5c285a56"
}

Redirect a Shortened URL: Visit http://localhost:8080/5c285a56 in your browser.
It redirects to https://www.example.com.
Get Top Domains:

curl http://localhost:8080/metrics

Response :

{
    "example.com": 5,
    "github.com": 3,
    "google.com": 2
}

Conclusion

Congratulations! 🎉 You’ve built a fully functional URL Shortening Service in Go. This project covered:

URL Shortening: Generating short URLs.
Redirection: Navigating users from short to original URLs.
Domain Tracking: Analyzing the most-used domains.
Dockerization: Deploying the app in a containerized environment.

You can get the complete code repository here.

Next Steps
Here are some ways to enhance the project:

Custom Short URLs: Allow users to create custom short links.
Analytics: Track the number of visits per short URL.
Expiration: Add expiration times for short URLs.

That’s it for this Tutorial. If this blog was helpful to you do share it with other and to get more information about Golang concepts, projects, etc. and to stay updated on the Tutorials do follow Siddhesh on Twitter and GitHub.

Until then Keep Learning, Keep Building 🚀🚀

Here's the 2nd Tutorial for the Scalable Go API Series 🚀

Siddhesh Khandagale — Wed, 22 Jan 2025 16:11:23 +0000

Taking Pagination to the Next Level: Sorting and Filtering in Go APIs

Siddhesh Khandagale ・ Jan 22

#go #api #pagination #tutorial

Taking Pagination to the Next Level: Sorting and Filtering in Go APIs

Siddhesh Khandagale — Wed, 22 Jan 2025 15:58:36 +0000

In the last post, we tackled Pagination breaking down large API responses into manageable chunks. But if you've ever wanted to let users control how data is sorted or filter specific results, then you're ready for the next step: sorting and filtering.
Let's dive in and make our APIs even more powerful by adding these features.

If you haven't gone through the previous tutorial, here's the link.

Why Sorting and Filtering Matter

Pagination alone isn't always enough. Imagine a user searching for the newest items or only those created in a specific timeframe. Sorting and filtering let users:

- Sort: Choose the order of results (e.g. newest to oldest)
- Filter: Narrow down results to what they need (e.g. items created today)

By combining pagination with sorting and filtering, we create a more user-friendly API.

Extending Our Database Query

We'll build on the items table from the previous blog. To recap, here's the scheme:

CREATE TABLE items (
    id SERIAL PRIMARY KEY,
    name TEXT NOT NULL,
    created_at TIMESTAMP DEFAULT NOW()
);

Adding Sorting

Sorting is all about ordering the results. We'll allow users to sort by the name or created_at columns in ascending or descending order.

Step 1: Accept Sort Parameters

Users will pass two query parameters:

sort : The column to sort by ( name or created_at ).
order : The sort direction ( asc or desc ).

Example URL:

/items?page=1&limit=10&sort=created_at&order=desc

Step 2: Adjust the SQL Query

We'll dynamically modify the SQL query to include sorting:

// Extract sort and order query parameters
sort := r.URL.Query().Get("sort")
order := r.URL.Query().Get("order")

// Validate the sort column
validSortColumns := map[string]bool{"name": true, "created_at": true}
if !validSortColumns[sort] {
    sort = "created_at" // Default sort column
}

// Validate the sort order
if order != "asc" && order != "desc" {
    order = "asc" // Default sort order
}

// Modify the SQL query
query := fmt.Sprintf("SELECT id, name, created_at FROM items ORDER BY %s %s LIMIT $1 OFFSET $2", sort, order)
rows, err := db.Query(query, limit, offset)
if err != nil {
    http.Error(w, "Failed to fetch items", http.StatusInternalServerError)
    return
}

Adding Filtering

Filtering lets users refine their search. For example, we can filter items by a date range or items containing a specific keyword in their name.

Step 1: Accept Filter Parameters

We'll support two filters:

name : Search for items containing a specific substring.
created_after : Fetch items created after a specific date.

Example URL :

/items?page=1&limit=10&name=Item&created_after=2025-01-10 20:38:57.832777

Step 2: Adjust the SQL Query

We'll add WHERE conditions to handle these filters:

// Extract filter query parameters
nameFilter := r.URL.Query().Get("name")
createdAfter := r.URL.Query().Get("created_after")

// Build the WHERE clause dynamically
whereClauses := []string{}
args := []interface{}{}
argIndex := 1

if nameFilter != "" {
    whereClauses = append(whereClauses, "name ILIKE $1")
    args = append(args, "%"+nameFilter+"%")
    argIndex++
}

if createdAfter != "" {
    whereClauses = append(whereClauses, fmt.Sprintf("created_at > $%d", argIndex))
    args = append(args, createdAfter)
    argIndex++
}

args = append(args, limit, offset)

// Combine WHERE clauses
whereSQL := ""
if len(whereClauses) > 0 {
    whereSQL = "WHERE " + strings.Join(whereClauses, " AND ")
}

// Final query
query := fmt.Sprintf("SELECT id, name, created_at FROM items %s ORDER BY %s %s LIMIT $%d OFFSET $%d", whereSQL, sort, order, argIndex, argIndex+1)
rows, err := db.Query(query, args...)
if err != nil {
    http.Error(w, "Failed to fetch items", http.StatusInternalServerError)
    return
}

Testing the Enhanced API

Sorting by name in ascending order:

curl "http://localhost:8080/items?page=1&limit=5&sort=name&order=asc"

Filtering items created after a specific date:

curl "http://localhost:8080/items?page=1&limit=5&created_after=2025-01-10 20:38:57.832777"

Combining filters and sorting:

curl "http://localhost:8080/items?page=1&limit=5&name=Item&sort=created_at&order=desc"

Common Mistake to Avoid

Not validating user input: Allowing arbitrary columns or invalid sort orders can expose your database to SQL injection. Always validate inputs.

Conclusion / Next Steps

You can find the complete code repository for this tutorial here

With pagination, sorting, and filtering in place, your API is now more user-friendly and flexible. For even more advanced functionality, consider adding:

Cursor-based pagination for large datasets.
Faceted filtering for complex searches.

Stay tuned for the next post where we’ll explore these advanced techniques!

To get more information about Golang concepts, projects, etc. and to stay updated on the Tutorials do follow Siddhesh on Twitter and GitHub.

Until then Keep Learning, Keep Building 🚀🚀

Pagination using Go

Siddhesh Khandagale — Fri, 10 Jan 2025 15:49:06 +0000

How to Paginate API Responses in Go

Siddhesh Khandagale ・ Jan 10

#go #api #pagination #backend

How to Paginate API Responses in Go

Siddhesh Khandagale — Fri, 10 Jan 2025 15:46:49 +0000

What Is Pagination, and Why Do We Need It?

Imagine your application has a database with thousands of records. Sending all these records to users in a single API response:

Slows down your application.
Consumes excessive bandwidth.
Overwhelms users with too much data at once.

Pagination solves this problem by splitting the data into smaller pages. Users get only subset of the data at a time, making APIs faster and applications smoother.

Consider a giant bookshelf with hundreds of books. Instead of searching through the entire shelf, wouldn’t it be easier if the shelf were divided into sections like "Page 1", "Page 2", and so on? That’s exactly what pagination does!

Setting Up the Database

To demonstrate pagination, we’ll use a simple items table in a PostgreSQL database. Here's the schema:

CREATE TABLE items (
    id SERIAL PRIMARY KEY,
    name TEXT NOT NULL,
    created_at TIMESTAMP DEFAULT NOW()
);

Now insert some dummy data:

INSERT INTO items (name) 
VALUES 
('Item 1'), ('Item 2'), ('Item 3'), ..., ('Item 100');

Setting Up a Go API with Pagination

Let’s create an API endpoint /items that accepts two query parameters:

page: The page number (default: 1).
limit: The number of records per page (default: 10).

Here’s the full implementation:

package main

import (
    "database/sql"
    "fmt"
    "log"
    "net/http"
    "strconv"

    _ "github.com/lib/pq"
)

func main() {
    // Connect to the database
    db, err := sql.Open("postgres", "user=youruser password=yourpass dbname=yourdb sslmode=disable")
    if err != nil {
        log.Fatal(err)
    }
    defer db.Close()

    http.HandleFunc("/items", func(w http.ResponseWriter, r *http.Request) {
        // Extract 'page' and 'limit' query parameters
        page, err := strconv.Atoi(r.URL.Query().Get("page"))
        if err != nil || page < 1 {
            page = 1 // Default to page 1
        }
        limit, err := strconv.Atoi(r.URL.Query().Get("limit"))
        if err != nil || limit < 1 {
            limit = 10 // Default to 10 items per page
        }

        // Calculate the OFFSET
        offset := (page - 1) * limit

        // Query the database
        rows, err := db.Query("SELECT id, name, created_at FROM items LIMIT $1 OFFSET $2", limit, offset)
        if err != nil {
            http.Error(w, "Failed to fetch items", http.StatusInternalServerError)
            return
        }
        defer rows.Close()

        // Process the rows
        items := []map[string]interface{}{}
        for rows.Next() {
            var id int
            var name string
            var createdAt string
            if err := rows.Scan(&id, &name, &createdAt); err != nil {
                http.Error(w, "Failed to scan items", http.StatusInternalServerError)
                return
            }
            items = append(items, map[string]interface{}{
                "id":         id,
                "name":       name,
                "created_at": createdAt,
            })
        }

        // Respond with JSON
        w.Header().Set("Content-Type", "application/json")
        fmt.Fprintf(w, `{"page":%d,"limit":%d,"items":%v}`, page, limit, items)
    })

    log.Println("Server is running on http://localhost:8080")
    log.Fatal(http.ListenAndServe(":8080", nil))
}

Understanding the Logic :

Pagination Parameters

Page: Determines which set of records to fetch.
Limit: Specifies the number of records per page.

Offset Calculation
The offset determines how many records to skip:
offset = (page - 1) * limit

For example:

Page 1 with limit=5 → offset = 0 (skip 0 records).
Page 2 with limit=5 → offset = 5 (skip the first 5 records).

SQL Query

We use LIMIT and OFFSET in SQL to fetch the desired records:
SELECT id, name, created_at FROM items ORDER BY id LIMIT 5 OFFSET 5;

Testing Your API

Test your API using tools like Postman, cURL, or directly in a browser:

Fetch the first page with 10 items:

curl "http://localhost:8080/items?page=1&limit=10"

Fetch the second page with 20 items:

curl "http://localhost:8080/items?page=2&limit=20"

API Response

Here’s an example response for /items?page=2&limit=2:

{
    "page": 2,
    "limit": 2,
    "items": [map[created_at: 2025-01-10T20: 38: 57.832777Z id: 3 name:Item 3
        ] map[created_at: 2025-01-10T20: 38: 57.832777Z id: 4 name:Item 4
        ]
    ]
}

Common Doubts and Pitfalls

1. Why not fetch all records and slice them in Go?
Because it’s inefficient. Imagine loading a million records into memory—your API will slow down and possibly crash.

2. What happens if the page or limit parameters are missing?
Always set defaults (e.g., page=1, limit=10) to ensure your API doesn’t break.

3. Can we optimize this further?
Yes! Use indexes on frequently queried columns (like id or created_at) for faster lookups.

Conclusion

With just a few lines of code and smart database querying, you’ve turned an overwhelming API response into something lightweight and user-friendly.

Want to take it up a notch? Try adding total pages, next/previous links, or even cursor-based pagination for large-scale applications.

To get more information about Golang concepts, projects, etc. and to stay updated on the Tutorials do follow Siddhesh on Twitter and GitHub.

Until then Keep Learning, Keep Building 🚀🚀

Discover 5 must-know Go libraries to enhance your backend development!

Siddhesh Khandagale — Fri, 03 Jan 2025 13:13:34 +0000

Top 5 Go Libraries Every Backend Developer Should Know

Siddhesh Khandagale ・ Jan 3

#go #backenddevelopment #programming #learning