DEV Community: Daniel Puig Gerarde

Understanding eBPF: A Game-Changer for Linux Kernel Extensions

Daniel Puig Gerarde — Mon, 24 Feb 2025 17:28:38 +0000

eBPF enables safe and efficient kernel extensions without altering the kernel’s source code.
It supports diverse applications in observability, security, and networking.
Its programmability and compatibility make it indispensable in today’s Linux ecosystem.

What is eBPF?

eBPF is an advanced evolution of the original Berkeley Packet Filter (BPF), which was initially developed to filter network packets. While BPF had a narrow focus, eBPF expands its scope dramatically. It enables users to execute custom programs within the Linux kernel without altering the kernel's source code. These programs can hook into various kernel events—such as network packets, system calls, or tracepoints—allowing for diverse tasks like monitoring, filtering, or even modifying system operations.

Why is eBPF Important?

1. Safety and Security

Unlike traditional kernel modules, which could crash the system if flawed, eBPF programs are sandboxed and subjected to rigorous verification by the kernel before execution. This ensures they cannot access unauthorized memory or destabilize the system, providing a robust safety net for developers.

2. Performance

By operating directly within the kernel, eBPF programs achieve low-latency event processing. This is critical for high-performance scenarios like networking or real-time system monitoring, where every microsecond counts.

3. Flexibility and Versatility

eBPF is a Swiss Army knife for kernel extensions. It supports a broad spectrum of use cases:
Observability: Tools like BCC (BPF Compiler Collection) and bpftrace harness eBPF to trace system calls, monitor network traffic, or profile CPU usage with minimal overhead.
Security: eBPF can enforce policies or detect anomalies in real time.
Networking: It powers advanced solutions like load balancing and packet filtering.

4. Programmability

Developers can write eBPF programs in familiar languages like C, which are then compiled into eBPF bytecode and executed by the kernel’s eBPF virtual machine. This programmability empowers users to craft bespoke kernel behaviors tailored to specific needs.

5. Widespread Adoption

eBPF is now a standard feature in modern Linux kernels and is seamlessly supported by major distributions. This broad compatibility ensures that eBPF programs can be deployed across diverse environments without friction.

Real-World Applications

eBPF’s versatility shines through in its practical applications across multiple domains:
Observability: Tools like Pixie and Sysdig leverage eBPF to deliver deep insights into system and application behavior without requiring manual instrumentation. This is invaluable in complex microservices architectures.
Security: eBPF enables real-time monitoring of system calls to detect suspicious activities or enforce network access controls, enhancing system defenses.
Networking: Projects like Cilium use eBPF to provide high-performance networking and security for containerized environments, handling tasks like load balancing, service discovery, and network policy enforcement.

Building a Semantic Search Engine with OpenAI, Go, and PostgreSQL (pgvector)

Daniel Puig Gerarde — Tue, 14 Jan 2025 22:49:31 +0000

In recent years, vector embeddings have become the foundation of modern Natural Language Processing (NLP) and semantic search. Rather than searching by keyword, vector databases allow us to compare “meanings” of text based on numerical representations (embeddings).This example demonstrates how to leverage OpenAI Embeddings, Go, and PostgreSQL with the pgvector extension to create a semantic search engine

What Are Embeddings?

An embedding is a numerical representation of text (or other data) as a vector in a high-dimensional space. If two pieces of text are semantically similar, their vectors will be close in that space. By storing embeddings in a database like PostgreSQL (with the pgvector extension), we can perform similarity searches quickly and accurately.

Why PostgreSQL with pgvector?

pgvector is a popular extension that adds vector data types to PostgreSQL. It enables you to:

Store embeddings as vector columns
Perform approximate or exact nearest neighbor searches
Use standard SQL to run queries

Overview of Our Application

Call OpenAI’s Embedding API to convert input text into vector embeddings.
Store those embeddings in PostgreSQL using the pgvector extension.
Query embeddings to find the most semantically similar entries in our database.

Prerequisites

Go installed (1.19+ recommended).
PostgreSQL installed and running locally or hosted.
pgvector extension installed in PostgreSQL. (See pgvector’s GitHub page for installation instructions.)
An OpenAI API key with access to embeddings.

The Makefile with the tasks related to postgres/pgvector and Docker for local testing.

pgvector:
    @docker run -d \
        --name pgvector \
        -e POSTGRES_USER=admin \
        -e POSTGRES_PASSWORD=admin \
        -e POSTGRES_DB=vectordb \
        -v pgvector_data:/var/lib/postgresql/data \
        -p 5432:5432 \
        pgvector/pgvector:pg17
psql:
    @psql -h localhost -U admin -d vectordb

Make sure you have pgvector installed. Then, in your PostgreSQL database:

CREATE EXTENSION IF NOT EXISTS vector;

The full code

package main

import (
    "context"
    "fmt"
    "log"
    "os"
    "strings"

    "github.com/jackc/pgx/v5/pgxpool"
    "github.com/joho/godotenv"
    "github.com/sashabaranov/go-openai"
)

func floats32ToString(floats []float32) string {
    strVals := make([]string, len(floats))
    for i, val := range floats {
        // Format each float into a string
        strVals[i] = fmt.Sprintf("%f", val)
    }

    // Join them with comma + space
    joined := strings.Join(strVals, ", ")

    // pgvector requires bracketed notation for vector input, e.g. [0.1, 0.2, 0.3]
    return "[" + joined + "]"
}

func main() {
    // Load environment variables
    err := godotenv.Load()
    if err != nil {
        log.Fatal("Error loading .env file")
    }

    // Create the connection pool
    dbpool, err := pgxpool.New(context.Background(), os.Getenv("DATABASE_URL"))
    if err != nil {
        fmt.Fprintf(os.Stderr, "Unable to create connection pool: %v\n", err)
        os.Exit(1)
    }
    defer dbpool.Close()

    // 1. Ensure pgvector extension is enabled
    _, err = dbpool.Exec(context.Background(), "CREATE EXTENSION IF NOT EXISTS vector;")
    if err != nil {
        log.Fatalf("Failed to create extension: %v\n", err)
        os.Exit(1)
    }

    // 2. Create table (if not existing)
    createTableSQL := `
    CREATE TABLE IF NOT EXISTS documents (
        id SERIAL PRIMARY KEY,
        content TEXT,
        embedding vector(1536)
    );
    `
    _, err = dbpool.Exec(context.Background(), createTableSQL)
    if err != nil {
        log.Fatalf("Failed to create table: %v\n", err)
    }

    // 3. Create index (if not existing)
    createIndexSQL := `
    CREATE INDEX IF NOT EXISTS documents_embedding_idx
    ON documents USING ivfflat (embedding vector_l2_ops) WITH (lists = 100);
    `
    _, err = dbpool.Exec(context.Background(), createIndexSQL)
    if err != nil {
        log.Fatalf("Failed to create index: %v\n", err)
    }

    // 4. Initialize OpenAI client
    apiKey := os.Getenv("OPENAI_API_KEY")
    if apiKey == "" {
        log.Fatal("OPENAI_API_KEY is not set")
    }
    openaiClient := openai.NewClient(apiKey)

    // 5. Insert sample documents
    docs := []string{
        "PostgreSQL is an advanced open-source relational database.",
        "OpenAI provides GPT-based models to generate text embeddings.",
        "pgvector allows storing embeddings in a Postgres database.",
    }

    for _, doc := range docs {
        err = insertDocument(context.Background(), dbpool, openaiClient, doc)
        if err != nil {
            log.Printf("Failed to insert document '%s': %v\n", doc, err)
        }
    }

    // 6. Query for similarity
    queryText := "How to store embeddings in Postgres?"
    similarDocs, err := searchSimilarDocuments(context.Background(), dbpool, openaiClient, queryText, 5)
    if err != nil {
        log.Fatalf("Search failed: %v\n", err)
    }

    fmt.Println("=== Most Similar Documents ===")
    for _, doc := range similarDocs {
        fmt.Printf("- %s\n", doc)
    }
}

// insertDocument generates an embedding for `content` using the OpenAI API
// and inserts it into the documents table.
func insertDocument(ctx context.Context, dbpool *pgxpool.Pool, client *openai.Client, content string) error {
    // 1) Get embedding from OpenAI
    embedResp, err := client.CreateEmbeddings(ctx, openai.EmbeddingRequest{
        Model: openai.AdaEmbeddingV2, // "text-embedding-ada-002"
        Input: []string{content},
    })
    if err != nil {
        return fmt.Errorf("CreateEmbeddings API call failed: %w", err)
    }

    // 2) Convert embedding to bracketed string for pgvector
    embedding := embedResp.Data[0].Embedding // []float32
    embeddingStr := floats32ToString(embedding)

    // 3) Insert into PostgreSQL
    insertSQL := `
        INSERT INTO documents (content, embedding)
        VALUES ($1, $2::vector)
    `
    _, err = dbpool.Exec(ctx, insertSQL, content, embeddingStr)
    if err != nil {
        return fmt.Errorf("failed to insert document: %w", err)
    }

    return nil
}

// searchSimilarDocuments takes a user query, gets the embedding, and returns
// the top-k similar documents based on vector similarity.
func searchSimilarDocuments(ctx context.Context, pool *pgxpool.Pool, client *openai.Client, query string, k int) ([]string, error) {
    // 1) Get the embedding for the user’s query via OpenAI
    embedResp, err := client.CreateEmbeddings(ctx, openai.EmbeddingRequest{
        Model: openai.AdaEmbeddingV2, // "text-embedding-ada-002"
        Input: []string{query},
    })
    if err != nil {
        return nil, fmt.Errorf("CreateEmbeddings API call failed: %w", err)
    }

    // 2) Convert the OpenAI embedding to the bracketed string format for pgvector
    queryEmbedding := embedResp.Data[0].Embedding // []float32
    queryEmbeddingStr := floats32ToString(queryEmbedding)
    // e.g. "[0.123456, 0.789012, ...]"

    // 3) Build the SELECT statement that orders by vector similarity
    selectSQL := fmt.Sprintf(`
        SELECT content
        FROM documents
        ORDER BY embedding <-> '%s'::vector
        LIMIT %d;
    `, queryEmbeddingStr, k)

    // 4) Run the query
    rows, err := pool.Query(ctx, selectSQL)
    if err != nil {
        return nil, fmt.Errorf("failed to query documents: %w", err)
    }
    defer rows.Close()

    // 5) Read the matching documents
    var contents []string
    for rows.Next() {
        var content string
        if err := rows.Scan(&content); err != nil {
            return nil, fmt.Errorf("failed to scan row: %w", err)
        }
        contents = append(contents, content)
    }
    if err = rows.Err(); err != nil {
        return nil, fmt.Errorf("row iteration error: %w", err)
    }

    return contents, nil
}

Conclusion

OpenAI embeddings, Go, and pgvector in PostgreSQL offer a straightforward solution for building semantic search applications. By representing text as vectors and leveraging the power of database indexing, we move from traditional keyword-based search to searching by context and meaning.

Setting Up an Ubuntu Dev Environment with Multipass and VS Code Remote-SSH

Daniel Puig Gerarde — Mon, 28 Oct 2024 02:19:28 +0000

Why Develop in Ubuntu While Keeping Your Desktop OS?

If you’re working on projects in embedded development, robotics, or cybersecurity, you know that Ubuntu is often the go-to operating system for many open-source tools and libraries. However, many of us prefer to stick with our native desktop operating system (macOS, Windows) for everyday use. Setting up a dedicated Ubuntu development environment that doesn't disrupt your daily workflow can be easily achieved with Multipass, a tool for running lightweight Ubuntu VMs on your computer. Here’s how you can set up a productive, versatile Ubuntu environment for development while still using your primary OS.

What You'll Need

Multipass: A tool for managing Ubuntu VMs.
Visual Studio Code (VS Code): A powerful, extensible code editor.
VS Code Remote - SSH Extension: To connect seamlessly from your host OS to your Ubuntu VM.
Basic Understanding of the Command Line: To install tools and manage your VM.

Step 1: Install Multipass

Download and Install Multipass
Multipass makes creating and managing Ubuntu VMs extremely simple. Download it from the official website and follow the installation instructions for your OS.
Launch Your Ubuntu VM
Once installed, open your terminal (Command Prompt on Windows, Terminal on macOS or Linux), and use this command to create an Ubuntu instance:

$ multipass launch --name ubuntu-dev --mem 4G --disk 20G

Another way is to launch the VM with the default values and then you can edit the settings

$ multipass launch --name ubuntu-dev

$ multipass stop ubuntu-dev

$ multipass set local.ubuntu-dev.memory=4G

$ multipass set local.ubuntu-dev.disk=20G

$ multipass start ubuntu-dev

Verify and Access Your VM To confirm that your instance is running, use:

$ multipass list

Once it's listed as "Running," you can connect to it:

$ multipass shell ubuntu-dev

# Inside ubuntu-dev 
$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 22.04.5 LTS
Release:        22.04
Codename:       jammy

Step 2: Set Up VS Code with Remote-SSH

One of the most effective ways to code and debug on your Ubuntu VM without leaving your native OS is by using VS Code with the Remote - SSH extension.

Install Visual Studio Code
If you don’t have it yet, download VS Code for your OS.
Install the Remote - SSH Extension
Open VS Code, go to the Extensions view, and install Remote - SSH. Or using the command in a terminal:

$ code --install-extension ms-vscode-remote.vscode-remote-extensionpack

Configure SSH to Connect to Your VM Add your SSH key to the VM How to Use ssh-keygen to Generate a New SSH Key?

$ multipass exec ubuntu-dev -- bash -c "echo `cat ~/.ssh/<key>.pub` >> ~/.ssh/authorized_keys"

Also retrieve the key from GitHub:

multipass exec ubuntu-dev -- bash -c “curl https://github.com/<username>.keys | tee -a ~/.ssh/authorized_keys”

Connect to Your VM in VS Code In VS Code, open the Command Palette (Ctrl+Shift+P or Cmd+Shift+P on macOS) and select Remote-SSH: Connect to Host. Choose ubuntu-dev from the list. Or using the command in a terminal:

$ code --folder-uri vscode-remote://ssh-remote+ubuntu@<vm-ip>/home/ubuntu

You are now remotely editing and debugging code on your Ubuntu VM through VS Code!

Mount Local Folder A good idea is to mount a local folder

$ multipass mount /some/local/path ubuntu-dev:/some/remote/path

Step 3: Set Up Your Development Tools in Ubuntu

Update and Install Essentials First, make sure your system is up-to-date:

$ sudo apt update && sudo apt upgrade -y

Then, install any essentials you might need, such as Git and build-essential:

$ sudo apt install -y git build-essential

Install Specific Tools for Embedded Development, Robotics, or Cybersecurity For embedded development, robotics, or cybersecurity, install domain-specific tools:

# Example packages for embedded dev, robotics, and cybersecurity
$ sudo apt install -y gcc-arm-none-eabi gdb openocd ros-noetic-desktop wireshark nmap

Step 4: Set Up for Embedded Development, Robotics, and Cybersecurity

Embedded Development: Install gcc, gdb, and openocd to compile, debug, and program microcontrollers. These tools make Ubuntu ideal for ARM and RISC-V development workflows.

Robotics: Installing ROS (Robot Operating System) on Ubuntu is straightforward and fully supported. ROS Noetic, the latest version, offers a rich ecosystem for simulation, control, and robot software development.

$ sudo apt install ros-noetic-desktop-full

Cybersecurity: Ubuntu provides an ideal platform for cybersecurity research, with support for tools like Wireshark, Nmap, Metasploit, and others. Installing them is a simple command away:

$ sudo apt install wireshark nmap metasploit-framework

Step 5: Maintenance and Snapshot Management

Multipass makes it easy to take snapshots, which allow you to create backups of your development environment. This is especially useful if you want to test potentially disruptive software.

To take a snapshot:

$ multipass snapshot ubuntu-dev

To restore a snapshot:

$ multipass restore ubuntu-dev

These commands make it easy to experiment with new tools without risking your setup.

Why This Setup Works

By isolating your development environment in Ubuntu, you gain access to the rich Linux ecosystem, which includes tools specifically tailored to robotics, embedded systems, and cybersecurity, while maintaining your main OS for everyday tasks. This approach offers flexibility, reduces clutter on your host OS, and provides an ideal testing ground for domain-specific projects.

Whether you’re building an IoT solution, exploring autonomous vehicle software, or working on network security, this setup offers a powerful and versatile development workflow.

Container Runtime Interfaces (CRI)

Daniel Puig Gerarde — Mon, 07 Oct 2024 21:50:45 +0000

The rise of containerization has revolutionized how we develop, ship, and run applications. At the heart of this revolution lies the container runtime, a critical component that brings containers to life.

"The CRI is a plugin interface which enables the kubelet to use a wide variety of container runtimes, without having a need to recompile the cluster components.

You need a working container runtime on each Node in your cluster, so that the kubelet can launch Pods and their containers.

The Container Runtime Interface (CRI) is the main protocol for the communication between the kubelet and Container Runtime.

The Kubernetes Container Runtime Interface (CRI) defines the main gRPC protocol for the communication between the node components kubelet and container runtime." https://kubernetes.io/docs/concepts/architecture/cri/

Understanding Container Runtimes

Key Objectives of CRI:

Standardization: Provide a consistent interface for all container runtimes.
Extensibility: Allow easy integration of new and innovative container runtimes.
Decoupling: Separate Kubernetes internals from container runtime specifics.

By abstracting the container runtime details, CRI allows Kubernetes to support multiple runtimes without changing its core codebase.

Introducing containerd

containerd is an industry-standard container runtime that emphasizes simplicity, robustness, and portability. Originally developed by Docker, it was donated to the Cloud Native Computing Foundation (CNCF) and has since become a graduated project.

Key Features:

Daemon for managing containers: Provides an API for managing the entire container lifecycle.
Image management: Handles pulling and pushing images to and from container registries.
Snapshotter support: Efficient management of container filesystems.
Support for multiple runtimes: Through the use of runC and other OCI-compliant runtimes.

containerd is designed to be embedded into a larger system, making it an excellent choice for Kubernetes and other orchestrators.

containerd and CRI: How They Work Together

To integrate with Kubernetes via CRI, containerd uses a plugin https://github.com/containerd/cri. However, since containerd version 1.1, the CRI plugin has been built directly into containerd, eliminating the need for additional installations.

Workflow:

Kubelet Interaction: The kubelet communicates with containerd using the CRI API.
Image Management: containerd pulls images from registries as requested.
Container Lifecycle: containerd handles the creation, execution, and termination of containers.
Runtime Interface: containerd uses runC (by default) to run containers according to the OCI specification.

By conforming to the CRI, containerd ensures seamless operation within Kubernetes clusters.

Benefits of Using containerd

1. Simplicity and Focus

containerd is purpose-built to manage containers without the additional features that come with full container platforms like Docker. This focus leads to:

Reduced Overhead: Fewer moving parts mean less complexity.
Enhanced Stability: A smaller codebase can lead to fewer bugs and vulnerabilities.

2. Performance

containerd is optimized for performance, providing:

Faster Startup Times: Quicker container launch and teardown.
Efficient Resource Usage: Minimal footprint on system resources.

3. Compliance and Standardization

OCI Compliance: Supports the Open Container Initiative specifications.
CNCF Project: Backed by a vibrant open-source community and industry leaders.

4. Extensibility

Plugin Architecture: Easy to extend functionality.
Support for Multiple Runtimes: Can integrate with alternatives like Kata Containers for hardware virtualization.

Comparing containerd with Other Runtimes

containerd vs. Docker

Docker: A complete platform for building, shipping, and running containers.
containerd: Focuses solely on the container runtime aspect.

While Docker includes containerd as its runtime component, using containerd directly with Kubernetes removes the extra layers, leading to improved performance and simplicity.

containerd vs. CRI-O

CRI-O: An OCI-compliant runtime designed explicitly for Kubernetes.
containerd: A general-purpose runtime that also supports CRI.

Both are excellent choices, but containerd's broader adoption and backing by major cloud providers make it a more popular option.

Getting Started with containerd

Installation

Most modern Linux distributions include containerd in their repositories. For example, on Ubuntu:

sudo apt-get update
sudo apt-get install -y containerd

Create the configuration file

sudo mkdir -p /etc/containerd
sudo containerd config default | sudo tee /etc/containerd/config.toml

Set the cgroup driver for containerd to systemd which is required for the kubelet.

sudo sed -i 's/            SystemdCgroup = false/            SystemdCgroup = true/' /etc/containerd/config.toml
sudo systemctl restart containerd

Check the status

sudo systemctl status containerd.service

Conclusion

containerd has solidified its place as a leading container runtime, offering a robust, efficient, and straightforward solution for running containers in production environments. Its seamless integration with Kubernetes via the CRI makes it an excellent choice for modern cloud-native applications.

By understanding the role of the CRI and how containerd leverages it, you can make informed decisions about your container orchestration strategy and optimize your Kubernetes clusters for performance and reliability.

Google Cloud Run: A serverless platform for containerized applications

Daniel Puig Gerarde — Sun, 08 Oct 2023 03:33:42 +0000

Google has always been at the forefront of cloud innovation, and with its introduction of Cloud Run, it has once again demonstrated its commitment to providing top-tier cloud solutions for developers. In this blog post, we'll delve deep into the wonders of Google Cloud Run: what it is, its benefits, and how to get started.

Google Cloud Run is a fully managed compute platform that enables you to run containerized applications without provisioning or managing servers. Cloud Run scales your application automatically based on demand, so you can focus on building great applications without worrying about the infrastructure.

There are many benefits to using Cloud Run, including:

Serverless: Cloud Run takes care of all the infrastructure management for you, so you can focus on your code.
Scalability: Cloud Run scales your application automatically based on demand, so you can handle unexpected spikes in traffic without any downtime.
Portability: Cloud Run uses standard containers, so you can easily port your applications to and from Cloud Run.
Pay as You Go: With Cloud Run, you only pay for the compute time you use. If your application isn't running, you aren't charged. This not only makes it cost-effective but also ensures that you're not overspending on idle resources.
Developer Friendly: Cloud Run supports multiple programming languages, and with its integration with Cloud Build, developers can automatically build and deploy their applications directly from their source repositories.

Use cases for Cloud Run

Cloud Run is a good choice for a wide range of applications, including:

Web applications: Cloud Run is a great way to run web applications, such as blogs, e-commerce sites, and content management systems.
Microservices: Cloud Run is a good choice for running microservices architectures, where each service is independently scalable and deployable.
Event-driven applications: Cloud Run can be used to build event-driven applications, such as real-time data processing and notification systems.
Machine learning applications: Cloud Run can be used to run machine learning applications, such as image recognition and natural language processing models.

Click here for more information, how to install and other steps.

Example

Tools

Operating Systems: Kubuntu
IDE: VSCode Google Cloud Code
Programming Language: Go
Container Runtime: Docker

We will create a microservice with some endpoints called patient-svc, which will be part of a Hospital Management System

Register new patient
Update patient details
Retrieve patient details

Ctrl + P then type > Cloud Run: New Application.
Select the option Cloud Run application.
In this case select Go: Cloud Run.
Select the directory to save the project with the name patient-svc.
In the Cloud Code status bar, click the active project name.

main.go

package main

import (
    "encoding/json"
    "log"
    "net/http"
    "sync"
)

type Patient struct {
    ID      string `json:"id"`
    Name    string `json:"name"`
    Address string `json:"address"`
}

var patients = make(map[string]Patient)
var mu sync.Mutex

func RegisterPatient(w http.ResponseWriter, r *http.Request) {
    var p Patient
    if err := json.NewDecoder(r.Body).Decode(&p); err != nil {
        http.Error(w, err.Error(), http.StatusBadRequest)
        return
    }

    mu.Lock()
    patients[p.ID] = p
    mu.Unlock()

    w.WriteHeader(http.StatusCreated)
    json.NewEncoder(w).Encode(p)
}

func UpdatePatientDetails(w http.ResponseWriter, r *http.Request) {
    var p Patient
    if err := json.NewDecoder(r.Body).Decode(&p); err != nil {
        http.Error(w, err.Error(), http.StatusBadRequest)
        return
    }

    mu.Lock()
    if _, exists := patients[p.ID]; exists {
        patients[p.ID] = p
    } else {
        http.Error(w, "Patient not found", http.StatusNotFound)
        mu.Unlock()
        return
    }
    mu.Unlock()

    w.WriteHeader(http.StatusOK)
    json.NewEncoder(w).Encode(p)
}

func RetrievePatientDetails(w http.ResponseWriter, r *http.Request) {
    id := r.URL.Query().Get("id")
    if id == "" {
        http.Error(w, "ID is required", http.StatusBadRequest)
        return
    }

    mu.Lock()
    p, exists := patients[id]
    mu.Unlock()
    if !exists {
        http.Error(w, "Patient not found", http.StatusNotFound)
        return
    }

    json.NewEncoder(w).Encode(p)
}

func main() {
    http.HandleFunc("/register", RegisterPatient)
    http.HandleFunc("/update", UpdatePatientDetails)
    http.HandleFunc("/retrieve", RetrievePatientDetails)

    port := "8080"
    log.Printf("Server started on :%s", port)
    log.Fatal(http.ListenAndServe(":"+port, nil))
}

main_test.go

func TestRegisterPatient(t *testing.T) {
    // Create a new request with a JSON payload
    reqBody := []byte(`{"id": "123", "name": "John Doe", "age": 35}`)
    req, err := http.NewRequest("POST", "/register", bytes.NewBuffer(reqBody))
    if err != nil {
        t.Fatal(err)
    }

    // Set the request header content type to JSON
    req.Header.Set("Content-Type", "application/json")

    // Create a response recorder to record the response
    rr := httptest.NewRecorder()

    // Call the RegisterPatient function with the response recorder and request
    handler := http.HandlerFunc(RegisterPatient)
    handler.ServeHTTP(rr, req)

    // Check the status code is 201 Created
    if status := rr.Code; status != http.StatusCreated {
        t.Errorf("handler returned wrong status code: got %v want %v", status, http.StatusCreated)
    }

    // Check the response body is what we expect
    expected := `{"id":"123","name":"John Doe","age":35}`
    if rr.Body.String() != expected {
        t.Errorf("handler returned unexpected body: got %v want %v", rr.Body.String(), expected)
    }
}

func TestUpdatePatientDetails(t *testing.T) {
    // Create a new request with a JSON payload
    reqBody := []byte(`{"id": "123", "name": "John Doe", "age": 36}`)
    req, err := http.NewRequest("PUT", "/update", bytes.NewBuffer(reqBody))
    if err != nil {
        t.Fatal(err)
    }

    // Set the request header content type to JSON
    req.Header.Set("Content-Type", "application/json")

    // Create a response recorder to record the response
    rr := httptest.NewRecorder()

    // Call the UpdatePatientDetails function with the response recorder and request
    handler := http.HandlerFunc(UpdatePatientDetails)
    handler.ServeHTTP(rr, req)

    // Check the status code is 200 OK
    if status := rr.Code; status != http.StatusOK {
        t.Errorf("handler returned wrong status code: got %v want %v", status, http.StatusOK)
    }

    // Check the response body is what we expect
    expected := `{"id":"123","name":"John Doe","age":36}`
    if rr.Body.String() != expected {
        t.Errorf("handler returned unexpected body: got %v want %v", rr.Body.String(), expected)
    }
}

Debug

In the Cloud Code status bar, click the active project name.
Select Debug on Cloud Run Emulator. If it is the first time a form will appear. This is how I have it configured.
Debug View from the VSCode Code Editor

Test the endpoints

$ curl -X POST -H "Content-Type: application/json" -d '{"id": "123", "name": "John Doe", "age": 35}' http://localhost:8080/register
{"id":"123","name":"John Doe","address":""}

$ curl -X PUT -H "Content-Type: application/json" -d '{"id": "123", "name": "John Doe", "age": 36}' http://localhost:8080/update
{"id":"123","name":"John Doe","address":""}

$ curl -X GET http://localhost:8080/retrieve?id=123
{"id":"123","name":"John Doe","address":""}

Deploy

Refer to this guide for specific details https://cloud.google.com/run/docs/deploying

I present some screenshots of the details of my deployment using the Google Cloud Console.

https://patient-svc-3v56nzrvtq-uc.a.run.app

curl -X POST -H "Content-Type: application/json" -d '{"id": "123", "name": "John Doe", "age": 35}' https://patient-svc-3v56nzrvtq-uc.a.run.app/register 
{"id":"123","name":"John Doe","address":""}

Google Cloud Run provides developers with a robust, scalable, and cost-effective platform to deploy containerized applications without the hassle of server management. As organizations continue to look for solutions that allow them to move fast and adapt to changing needs, platforms like Cloud Run will only become more pivotal. So, whether you're a startup or a well-established business, it might just be the right time to explore what Google Cloud Run can offer you!

Concurrency in Go using Goroutines and Channels.

Daniel Puig Gerarde — Sat, 09 Sep 2023 20:59:41 +0000

"Go serves both siblings, DevOps and SRE, from its fast build times and lean syntax to its security and reliability support. Go’s concurrency and networking features also make it ideal for tools that manage cloud deployment—readily supporting automation while scaling for speed and code maintainability as development infrastructure grows over time." Development Operations & Site Reliability Engineering

Introduction

Concurrency is an important aspect of modern software development that allows multiple tasks to be executed independently, enhancing the overall performance and responsiveness of an application. In Go, a statically-typed compiled language designed by Google, concurrency is one of the built-in features that developers absolutely love. One of the key components that make this possible is the Goroutine.

What is a Goroutine?

A Goroutine is a lightweight thread managed by the Go runtime. It allows you to run functions concurrently with other functions. Goroutines are one of the key elements that allow Go programs to easily implement parallel and concurrent processing. Unlike traditional threads, Goroutines are cheaper to create, and their stack sizes grow and shrink dynamically, making them more efficient.

package main

import (
    "fmt"
    "time"
)

func printMessage(msg string) {
    for i := 0; i < 10; i++ {
        fmt.Println(msg)
        time.Sleep(time.Millisecond * 100)
    }
}

func main() {
    go printMessage("Black")
    printMessage("Yellow")
}

Output

Yellow
Black
Black
Yellow
Yellow
Black
Black
Yellow
Yellow
Black
Black
Yellow
Yellow
Black
Black
Yellow
Yellow
Black
Black
Yellow

In this example, the printMessage function is called twice: once as a Goroutine and once as a normal function call. Both will execute concurrently.

Synchronization Using Channels

While Goroutines make it easy to implement concurrency, they also present challenges, especially when it comes to coordinating tasks or sharing data. Go provides a mechanism called 'channels' for safely communicating between Goroutines.

Here's an example that uses a channel to synchronize two Goroutines:

package main

import "fmt"

func printMessage(msg string, ch chan string) {
    fmt.Println(msg)
    ch <- "Done"
}

func main() {
    ch := make(chan string)

    go printMessage("Hello", ch)
    go printMessage("World", ch)

    fmt.Println(<-ch)
    fmt.Println(<-ch)

}

Output

World
Done
Hello
Done

In this example, each Goroutine sends a "Done" message on the channel ch after completing its task. The main function waits to receive two "Done" messages before terminating.

Advantages of Using Goroutines

Resource Efficiency: Goroutines are incredibly lightweight, requiring only a few kilobytes of stack memory.
Ease of Use: With the simple go keyword, you can convert most functions into Goroutines.
Built-in Support: The Go runtime natively supports Goroutines, eliminating the need for third-party libraries for task scheduling or context switching.

Goroutines vs Threads

Stack Size: Threads typically have a fixed stack size, usually around 1-2MB, whereas Goroutines start with a much smaller stack that can dynamically grow and shrink.
Creation Cost: Goroutines are much cheaper to create in terms of memory and CPU time compared to threads.
Scheduling: Goroutines are cooperatively scheduled by the Go runtime, which simplifies the design compared to preemptively scheduled threads.

Examples:

Using `sync.WaitGroup` for Synchronization

Instead of using channels for synchronization, you can use sync.WaitGroup. A WaitGroup waits for a collection of Goroutines to finish executing.

package main

import (
    "fmt"
    "sync"
    "time"
)

func worker(id int, wg *sync.WaitGroup) {
    fmt.Printf("Worker %d starting\n", id)
    time.Sleep(time.Second)
    fmt.Printf("Worker %d done\n", id)
    wg.Done()
}

func main() {
    var wg sync.WaitGroup

    for i := 1; i <= 5; i++ {
        wg.Add(1)
        go worker(i, &wg)
    }

    wg.Wait()
}

Parallel Summation using Goroutines

In this example, we split an array into two halves and sum them concurrently.

package main

import "fmt"

func sum(a []int, c chan int) {
    sum := 0
    for _, v := range a {
        sum += v
    }
    c <- sum
}

func main() {
    data := []int{1, 2, 3, 4, 5, 6, 7, 8, 9, 10}
    c := make(chan int)

    go sum(data[:len(data)/2], c)
    go sum(data[len(data)/2:], c)

    x, y := <-c, <-c

    fmt.Printf("Sum1: %d, Sum2: %d, Total Sum: %d\n", x, y, x+y)
}

Using select with Channels

The select statement allows you to wait for multiple channel operations, similar to how switch works for value types.

package main

import (
    "fmt"
    "time"
)

func server1(ch chan string) {
    time.Sleep(time.Second * 2)
    ch <- "Response from server 1"
}

func server2(ch chan string) {
    time.Sleep(time.Second * 1)
    ch <- "Response from server 2"
}

func main() {
    output1 := make(chan string)
    output2 := make(chan string)

    go server1(output1)
    go server2(output2)

    select {
    case msg1 := <-output1:
        fmt.Println(msg1)
    case msg2 := <-output2:
        fmt.Println(msg2)
    }
}

In this example, two "servers" send a message on their respective channels. We use select to wait for the first response and print it.

Timer and Ticker

Timers and tickers are other important features in Go that can be used for scheduling.

package main

import (
    "fmt"
    "time"
)

func main() {
    timer := time.NewTimer(time.Second * 2)
    ticker := time.NewTicker(time.Second)

    go func() {
        for t := range ticker.C {
            fmt.Println("Tick at", t)
        }
    }()

    <-timer.C
    ticker.Stop()
    fmt.Println("Timer expired")
}

In this example, the ticker ticks every second, and the timer expires after two seconds. When the timer expires, the ticker is stopped.

These examples demonstrate different facets of Go's concurrency model, helping you understand how versatile and useful Goroutines can be.

Goroutines are a powerful feature in Go for implementing concurrent tasks. They are simple to use, efficient, and well-integrated into the language and its standard library. Channels further enhance their usability by providing a safe way to share data between concurrently running Goroutines. As you dive deeper into Go, you'll find Goroutines and channels to be indispensable tools in your development toolkit.

A Comprehensive Guide to Building Kubernetes Operators with Kubebuilder

Daniel Puig Gerarde — Mon, 28 Aug 2023 16:47:09 +0000

Kubernetes Operators are a powerful way to automate the management of complex applications on Kubernetes. In this blog post, we will provide a hands-on guide for Kubernetes developers who want to learn how to create and use Operators. We will cover the basics of Operators, including how to define custom resources, create controllers, and manage reconciliation loops. We will also provide example of Operator for MySQL.

Prerequisites
go version v1.20.0+
docker version 17.03+.
kubectl version v1.11.3+.
Access to a Kubernetes v1.11.3+ cluster.

Install Kubebuilder

$ curl -L -o kubebuilder "https://go.kubebuilder.io/dl/latest/$(go env GOOS)/$(go env GOARCH)"
$ chmod +x kubebuilder
$ mv kubebuilder /usr/local/bin/

$ kubebuilder version 
Version: main.version{KubeBuilderVersion:"3.11.1", KubernetesVendor:"1.27.1", GitCommit:"1dc8ed95f7cc55fef3151f749d3d541bec3423c9", BuildDate:"2023-07-03T13:10:56Z", GoOs:"linux", GoArch:"amd64"}

Init/bootstrap the project

$ mkdir -p ~/ops/mysql-operator && cd /mysql-operator
$ kubebuilder init --domain dpuigerarde.com --repo github.com/dpuig/mysql-operator

The kubebuilder init --domain command is used to initialize a new Kubernetes Operator project. The domain flag specifies the Kubernetes group for the project's custom resources. The default value for the domain flag is my.domain.

Create an API

$ kubebuilder create api --group apps --version v1alpha1 --kind MySQLCluster

The kubebuilder create api --group command is used to create a new API (custom resource definition) in a Kubernetes Operator project. The group flag specifies the Kubernetes group for the API. The default value for the group flag is the project's domain name.

If you press y for Create Resource [y/n] and for Create Controller [y/n] then this will create the files

api
└── v1alpha1
    ├── groupversion_info.go
    ├── mysqlcluster_types.go
    └── zz_generated.deepcopy.go

where the API is defined

Also the files

internal
└── controller
    ├── mysqlcluster_controller.go
    └── suite_test.go

where the reconciliation business logic is implemented for this Kind(CRD).

Custom Resource Definition (CRD)

The MySQLClusterSpec CRD defines the schema for the MySQLCluster resource. It should include the following fields:

deploymentName: The name of the mysql db.
replicas: The number of mysql pods.
version: The version of mysql to use.
password: The default admin password.

In the generated project, look for api/v1alpha1/mysqlcluster_types.go

Edit the MySQLClusterSpec and MySQLClusterStatus structs:

type MySQLClusterSpec struct {
    // INSERT ADDITIONAL SPEC FIELDS - desired state of cluster
    // Important: Run "make" to regenerate code after modifying this file

    // Foo is an example field of MySQLCluster. Edit mysqlcluster_types.go to remove/update
    // Foo string `json:"foo,omitempty"`

    // +kubebuilder:validation:Required
    // +kubebuilder:validation:Format:=string

    // the name of the deployment
    DeploymentName string `json:"deploymentName"`

    // +kubebuilder:validation:Required
    // +kubebuilder:validation:Minimum=0

    // the number of replicas
    Replicas *int32 `json:"replicas"`

    Version string `json:"version"`

    // +kubebuilder:validation:Required
    // +kubebuilder:validation:Format:=string
    Password string `json:"password"`
}

type MySQLClusterStatus struct {
    // INSERT ADDITIONAL STATUS FIELD - define observed state of cluster
    // Important: Run "make" to regenerate code after modifying this file
    // this is equal deployment.status.availableReplicas
    // +optional
    AvailableReplicas int32 `json:"availableReplicas"`
}

Then run:

$ make manifests

Implement the Controller Logic

Edit the generated controller file, which is located at internal/controller/mysqlcluster_controller.go

Soon we will dedicate a Blog post to focus on the details of the API types and especially the logic in the controllers. For this case and in general terms, this controller is in charge of deploying a Deployment that will launch a the mysql db.

package controller

import (
    "context"

    "k8s.io/apimachinery/pkg/runtime"
    "k8s.io/client-go/tools/record"
    ctrl "sigs.k8s.io/controller-runtime"
    "sigs.k8s.io/controller-runtime/pkg/client"
    "sigs.k8s.io/controller-runtime/pkg/log"

    "github.com/go-logr/logr"

    appsv1 "k8s.io/api/apps/v1"
    corev1 "k8s.io/api/core/v1"
    metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"

    samplecontrollerv1alpha1 "github.com/dpuig/mysql-operator/api/v1alpha1"
)

var (
    deploymentOwnerKey = ".metadata.controller"
    apiGVStr           = samplecontrollerv1alpha1.GroupVersion.String()
)

// MySQLClusterReconciler reconciles a MySQLCluster object
type MySQLClusterReconciler struct {
    client.Client
    Log      logr.Logger
    Scheme   *runtime.Scheme
    Recorder record.EventRecorder
}

//+kubebuilder:rbac:groups=apps.dpuigerarde.com,resources=mysqlclusters,verbs=get;list;watch;create;update;patch;delete
//+kubebuilder:rbac:groups=apps.dpuigerarde.com,resources=mysqlclusters/status,verbs=get;update;patch
//+kubebuilder:rbac:groups=apps.dpuigerarde.com,resources=mysqlclusters/finalizers,verbs=update

// Reconcile is part of the main kubernetes reconciliation loop which aims to
// move the current state of the cluster closer to the desired state.
// TODO(user): Modify the Reconcile function to compare the state specified by
// the MySQLCluster object against the actual cluster state, and then
// perform operations to make the cluster state reflect the state specified by
// the user.
//
// For more details, check Reconcile and its Result here:
// - https://pkg.go.dev/sigs.k8s.io/controller-runtime@v0.15.0/pkg/reconcile
func (r *MySQLClusterReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
    _ = log.FromContext(ctx)

    log := r.Log.WithValues("mysqlCluster", req.NamespacedName)

    var mysqlCluster samplecontrollerv1alpha1.MySQLCluster
    log.Info("fetching MySQLCluster Resource")
    if err := r.Get(ctx, req.NamespacedName, &mysqlCluster); err != nil {
        log.Error(err, "unable to fetch MySQLCluster")
        return ctrl.Result{}, client.IgnoreNotFound(err)
    }

    if err := r.cleanupOwnedResources(ctx, log, &mysqlCluster); err != nil {
        log.Error(err, "failed to clean up old Deployment resources for this Foo")
        return ctrl.Result{}, err
    }

    // get deploymentName from mysqlCluster.Spec
    deploymentName := mysqlCluster.Spec.DeploymentName

    // define deployment template using deploymentName
    deploy := &appsv1.Deployment{
        ObjectMeta: metav1.ObjectMeta{
            Name:      deploymentName,
            Namespace: req.Namespace,
        },
    }

    // Create or Update deployment object
    if _, err := ctrl.CreateOrUpdate(ctx, r.Client, deploy, func() error {
        replicas := int32(1)
        if mysqlCluster.Spec.Replicas != nil {
            replicas = *mysqlCluster.Spec.Replicas
        }
        deploy.Spec.Replicas = &replicas

        labels := map[string]string{
            "app":        "mysql",
            "controller": req.Name,
        }

        // set labels to spec.selector for our deployment
        if deploy.Spec.Selector == nil {
            deploy.Spec.Selector = &metav1.LabelSelector{MatchLabels: labels}
        }

        // set labels to template.objectMeta for our deployment
        if deploy.Spec.Template.ObjectMeta.Labels == nil {
            deploy.Spec.Template.ObjectMeta.Labels = labels
        }

        // set a container for our deployment
        containers := []corev1.Container{
            {
                Name:  "db",
                Image: "mysql:" + mysqlCluster.Spec.Version,
                Env: []corev1.EnvVar{
                    {
                        Name:  "MYSQL_ROOT_PASSWORD",
                        Value: mysqlCluster.Spec.Password,
                    },
                },
                Command: []string{"mysqld", "--user=root"},
                Args:    []string{"--default-authentication-plugin=mysql_native_password"},
                Ports: []corev1.ContainerPort{
                    {
                        Name:          "mysql",
                        ContainerPort: 3306,
                    },
                },
                VolumeMounts: []corev1.VolumeMount{
                    {
                        Name:      "mysql-persistent-storage",
                        MountPath: "/var/lib/mysql",
                    },
                },
                SecurityContext: &corev1.SecurityContext{
                    RunAsUser:  func() *int64 { i := int64(1001); return &i }(),
                    RunAsGroup: func() *int64 { i := int64(1001); return &i }(),
                },
            },
        }

        // set containers to template.spec.containers for our deployment
        if deploy.Spec.Template.Spec.Containers == nil {
            deploy.Spec.Template.Spec.Containers = containers
        }

        deploy.Spec.Strategy.Type = "Recreate"
        deploy.Spec.Template.Spec.Volumes = []corev1.Volume{
            {
                Name: "mysql-persistent-storage",
                VolumeSource: corev1.VolumeSource{
                    PersistentVolumeClaim: &corev1.PersistentVolumeClaimVolumeSource{
                        ClaimName: "mysql-pv-claim",
                    },
                },
            },
        }

        deploy.Spec.Template.Spec.SecurityContext = &corev1.PodSecurityContext{
            FSGroup: func() *int64 { i := int64(1001); return &i }(),
        }

        // set the owner so that garbage collection can kicks in
        if err := ctrl.SetControllerReference(&mysqlCluster, deploy, r.Scheme); err != nil {
            log.Error(err, "unable to set ownerReference from mysqlCluster to Deployment")
            return err
        }

        return nil
    }); err != nil {

        // error handling of ctrl.CreateOrUpdate
        log.Error(err, "unable to ensure deployment is correct")
        return ctrl.Result{}, err

    }

    // get deployment object from in-memory-cache
    var deployment appsv1.Deployment
    var deploymentNamespacedName = client.ObjectKey{Namespace: req.Namespace, Name: mysqlCluster.Spec.DeploymentName}
    if err := r.Get(ctx, deploymentNamespacedName, &deployment); err != nil {
        log.Error(err, "unable to fetch Deployment")
        return ctrl.Result{}, client.IgnoreNotFound(err)
    }

    // set mysqlCluster.status.AvailableReplicas from deployment
    availableReplicas := deployment.Status.AvailableReplicas
    if availableReplicas == mysqlCluster.Status.AvailableReplicas {
        return ctrl.Result{}, nil
    }
    mysqlCluster.Status.AvailableReplicas = availableReplicas

    // update mysqlCluster.status
    if err := r.Status().Update(ctx, &mysqlCluster); err != nil {
        log.Error(err, "unable to update mysqlCluster status")
        return ctrl.Result{}, err
    }

    // create event for updated mysqlCluster.status
    r.Recorder.Eventf(&mysqlCluster, corev1.EventTypeNormal, "Updated", "Update mysqlCluster.status.AvailableReplicas: %d", mysqlCluster.Status.AvailableReplicas)

    return ctrl.Result{}, nil
}

// SetupWithManager sets up the controller with the Manager.
func (r *MySQLClusterReconciler) SetupWithManager(mgr ctrl.Manager) error {
    ctx := context.Background()
    // add deploymentOwnerKey index to deployment object which MySQLCluster resource owns
    if err := mgr.GetFieldIndexer().IndexField(ctx, &appsv1.Deployment{}, deploymentOwnerKey, func(rawObj client.Object) []string {
        // grab the deployment object, extract the owner...
        deployment := rawObj.(*appsv1.Deployment)
        owner := metav1.GetControllerOf(deployment)
        if owner == nil {
            return nil
        }
        // ...make sure it's a MySQLCluster...
        if owner.APIVersion != apiGVStr || owner.Kind != "MySQLCluster" {
            return nil
        }

        // ...and if so, return it
        return []string{owner.Name}
    }); err != nil {
        return err
    }

    // define to watch targets...Foo resource and owned Deployment
    return ctrl.NewControllerManagedBy(mgr).
        For(&samplecontrollerv1alpha1.MySQLCluster{}).
        Owns(&appsv1.Deployment{}).
        Complete(r)
}

// cleanupOwnedResources will delete any existing Deployment resources that
// were created for the given mysqlCluster that no longer match the
// mysqlCluster.spec.deploymentName field.
func (r *MySQLClusterReconciler) cleanupOwnedResources(ctx context.Context, log logr.Logger, mysqlCluster *samplecontrollerv1alpha1.MySQLCluster) error {
    log.Info("finding existing Deployments for Foo resource")

    // List all deployment resources owned by this mysqlCluster
    var deployments appsv1.DeploymentList
    if err := r.List(ctx, &deployments, client.InNamespace(mysqlCluster.Namespace), client.MatchingFields(map[string]string{deploymentOwnerKey: mysqlCluster.Name})); err != nil {
        return err
    }

    // Delete deployment if the deployment name doesn't match foo.spec.deploymentName
    for _, deployment := range deployments.Items {
        if deployment.Name == mysqlCluster.Spec.DeploymentName {
            // If this deployment's name matches the one on the Foo resource
            // then do not delete it.
            continue
        }

        // Delete old deployment object which doesn't match foo.spec.deploymentName
        if err := r.Delete(ctx, &deployment); err != nil {
            log.Error(err, "failed to delete Deployment resource")
            return err
        }

        log.Info("delete deployment resource: " + deployment.Name)
        r.Recorder.Eventf(mysqlCluster, corev1.EventTypeNormal, "Deleted", "Deleted deployment %q", deployment.Name)
    }

    return nil
}

Project Structure

.
├── api
│   └── v1alpha1
│       ├── groupversion_info.go
│       ├── mysqlcluster_types.go
│       └── zz_generated.deepcopy.go
├── bin
│   ├── controller-gen
│   └── kustomize
├── cmd
│   └── main.go
├── config
│   ├── crd
│   │   ├── bases
│   │   │   └── apps.dpuigerarde.com_mysqlclusters.yaml
│   │   ├── kustomization.yaml
│   │   ├── kustomizeconfig.yaml
│   │   └── patches
│   │       ├── cainjection_in_mysqlclusters.yaml
│   │       └── webhook_in_mysqlclusters.yaml
│   ├── default
│   │   ├── kustomization.yaml
│   │   ├── manager_auth_proxy_patch.yaml
│   │   └── manager_config_patch.yaml
│   ├── manager
│   │   ├── kustomization.yaml
│   │   └── manager.yaml
│   ├── prometheus
│   │   ├── kustomization.yaml
│   │   └── monitor.yaml
│   ├── rbac
│   │   ├── auth_proxy_client_clusterrole.yaml
│   │   ├── auth_proxy_role_binding.yaml
│   │   ├── auth_proxy_role.yaml
│   │   ├── auth_proxy_service.yaml
│   │   ├── kustomization.yaml
│   │   ├── leader_election_role_binding.yaml
│   │   ├── leader_election_role.yaml
│   │   ├── mysqlcluster_editor_role.yaml
│   │   ├── mysqlcluster_viewer_role.yaml
│   │   ├── role_binding.yaml
│   │   ├── role.yaml
│   │   └── service_account.yaml
│   └── samples
│       ├── apps_v1alpha1_mysqlcluster.yaml
│       └── kustomization.yaml
├── Dockerfile
├── go.mod
├── go.sum
├── hack
│   └── boilerplate.go.txt
├── internal
│   └── controller
│       ├── mysqlcluster_controller.go
│       └── suite_test.go
├── Makefile
├── mysql-pv.yaml
├── PROJECT
└── README.md

Run Operator Locally (For Development)

For development purposes, you may wish to run your operator locally against a remote cluster. This allows you to iterate more quickly during the development process.

Set the kubeconfig context:

$ export KUBECONFIG=<path-to-your-kubeconfig-file>

Install the CRDs into the cluster:

$ make install

$ kubectl get crds 

NAME                                 CREATED AT
mysqlclusters.apps.dpuigerarde.com   2023-08-28T02:22:43Z

For the purpose of this example. We will create some extra resources, PersistentVolume and PersistentVolumeClaim that will serve as a complement, the file mysql-pv.yaml

apiVersion: v1
kind: PersistentVolume
metadata:
  name: mysql-pv-volume
  labels:
    type: local
spec:
  storageClassName: manual
  capacity:
    storage: 20Gi
  accessModes:
    - ReadWriteOnce
  hostPath:
    path: "/mnt/data"
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: mysql-pv-claim
spec:
  storageClassName: manual
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 20Gi

Apply

$ kubectl apply -f mysql-pv.yaml

$ kubectl get pv,pvc
NAME                               CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                    STORAGECLASS   REASON   AGE
persistentvolume/mysql-pv-volume   20Gi       RWO            Retain           Bound    default/mysql-pv-claim   manual                  103m

NAME                                   STATUS   VOLUME            CAPACITY   ACCESS MODES   STORAGECLASS   AGE
persistentvolumeclaim/mysql-pv-claim   Bound    mysql-pv-volume   20Gi       RWO            manual         103m

Run your controller (this will run in the foreground, so switch to a new terminal if you want to leave it running):

$ make run

Deploy Custom Resources

Make sure to update config/samples/apps_v1alpha1_mysqlcluster.yaml with the actual specification you'd like to use for your MySQLCluster resource.

apiVersion: apps.dpuigerarde.com/v1alpha1
kind: MySQLCluster
metadata:
  labels:
    app.kubernetes.io/name: mysqlcluster
    app.kubernetes.io/instance: mysqlcluster-sample
    app.kubernetes.io/part-of: mysql-operator
    app.kubernetes.io/managed-by: kustomize
    app.kubernetes.io/created-by: mysql-operator
  name: mysqlcluster-sample
spec:
  # TODO(user): Add fields here
  deploymentName: mysqlcluster-sample-deploy
  replicas: 1
  version: "5.6"
  password: example

$ kubectl apply -f apply -f config/samples/apps_v1alpha1_mysqlcluster.yaml

mysqlcluster.apps.dpuigerarde.com/mysqlcluster-sample created

$ kubectl get mysqlclusters
NAME                  AGE
mysqlcluster-sample   10m

At this point, your operator should detect the custom resource and execute the reconcile loop, creating the mysql db as specified.

However, this is a Blog Post that contains problems , the example has problems, I hope I can count on your help to solve this problem. I promise to update soon with the solution to the problem

$ kubectl get deploy

NAME                         READY   UP-TO-DATE   AVAILABLE   AGE
mysqlcluster-sample-deploy   0/1     1            0           11m

$ kubectl get pods   

NAME                                         READY   STATUS             RESTARTS      AGE
mysqlcluster-sample-deploy-79c78b6c5-62jh5   0/1     CrashLoopBackOff   7 (42s ago)   11m

$ kubectl logs mysqlcluster-sample-deploy-79c78b6c5-62jh5 

2023-08-28 16:26:24 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
2023-08-28 16:26:24 0 [Warning] Can't create test file /var/lib/mysql/mysqlcluster-sample-deploy-79c78b6c5-62jh5.lower-test
2023-08-28 16:26:24 0 [Note] mysqld (mysqld 5.6.51) starting as process 1 ...
2023-08-28 16:26:24 1 [Warning] Can't create test file /var/lib/mysql/mysqlcluster-sample-deploy-79c78b6c5-62jh5.lower-test
2023-08-28 16:26:24 1 [Warning] Can't create test file /var/lib/mysql/mysqlcluster-sample-deploy-79c78b6c5-62jh5.lower-test
2023-08-28 16:26:24 1 [Warning] One can only use the --user switch if running as root

2023-08-28 16:26:24 1 [Note] Plugin 'FEDERATED' is disabled.
mysqld: Table 'mysql.plugin' doesn't exist
2023-08-28 16:26:24 1 [ERROR] Can't open the mysql.plugin table. Please run mysql_upgrade to create it.
2023-08-28 16:26:24 1 [Note] InnoDB: Using atomics to ref count buffer pool pages
2023-08-28 16:26:24 1 [Note] InnoDB: The InnoDB memory heap is disabled
2023-08-28 16:26:24 1 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2023-08-28 16:26:24 1 [Note] InnoDB: Memory barrier is not used
2023-08-28 16:26:24 1 [Note] InnoDB: Compressed tables use zlib 1.2.11
2023-08-28 16:26:24 1 [Note] InnoDB: Using Linux native AIO
2023-08-28 16:26:24 1 [Note] InnoDB: Not using CPU crc32 instructions
2023-08-28 16:26:24 1 [Note] InnoDB: Initializing buffer pool, size = 128.0M
2023-08-28 16:26:24 1 [Note] InnoDB: Completed initialization of buffer pool
2023-08-28 16:26:24 1 [ERROR] InnoDB: ./ibdata1 can't be opened in read-write mode
2023-08-28 16:26:24 1 [ERROR] InnoDB: The system tablespace must be writable!
2023-08-28 16:26:24 1 [ERROR] Plugin 'InnoDB' init function returned error.
2023-08-28 16:26:24 1 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
2023-08-28 16:26:24 1 [ERROR] Unknown/unsupported storage engine: InnoDB
2023-08-28 16:26:24 1 [ERROR] Aborting

2023-08-28 16:26:24 1 [Note] Binlog end
2023-08-28 16:26:24 1 [Note] Shutting down plugin 'partition'
2023-08-28 16:26:24 1 [Note] Shutting down plugin 'PERFORMANCE_SCHEMA'
2023-08-28 16:26:24 1 [Note] Shutting down plugin 'INNODB_SYS_DATAFILES'
2023-08-28 16:26:24 1 [Note] Shutting down plugin 'INNODB_SYS_TABLESPACES'
2023-08-28 16:26:24 1 [Note] Shutting down plugin 'INNODB_SYS_FOREIGN_COLS'
2023-08-28 16:26:24 1 [Note] Shutting down plugin 'INNODB_SYS_FOREIGN'
2023-08-28 16:26:24 1 [Note] Shutting down plugin 'INNODB_SYS_FIELDS'
2023-08-28 16:26:24 1 [Note] Shutting down plugin 'INNODB_SYS_COLUMNS'
2023-08-28 16:26:24 1 [Note] Shutting down plugin 'INNODB_SYS_INDEXES'
2023-08-28 16:26:24 1 [Note] Shutting down plugin 'INNODB_SYS_TABLESTATS'
2023-08-28 16:26:24 1 [Note] Shutting down plugin 'INNODB_SYS_TABLES'
2023-08-28 16:26:24 1 [Note] Shutting down plugin 'INNODB_FT_INDEX_TABLE'
2023-08-28 16:26:24 1 [Note] Shutting down plugin 'INNODB_FT_INDEX_CACHE'
2023-08-28 16:26:24 1 [Note] Shutting down plugin 'INNODB_FT_CONFIG'
2023-08-28 16:26:24 1 [Note] Shutting down plugin 'INNODB_FT_BEING_DELETED'
2023-08-28 16:26:24 1 [Note] Shutting down plugin 'INNODB_FT_DELETED'
2023-08-28 16:26:24 1 [Note] Shutting down plugin 'INNODB_FT_DEFAULT_STOPWORD'
2023-08-28 16:26:24 1 [Note] Shutting down plugin 'INNODB_METRICS'
2023-08-28 16:26:24 1 [Note] Shutting down plugin 'INNODB_BUFFER_POOL_STATS'
2023-08-28 16:26:24 1 [Note] Shutting down plugin 'INNODB_BUFFER_PAGE_LRU'
2023-08-28 16:26:24 1 [Note] Shutting down plugin 'INNODB_BUFFER_PAGE'
2023-08-28 16:26:24 1 [Note] Shutting down plugin 'INNODB_CMP_PER_INDEX_RESET'
2023-08-28 16:26:24 1 [Note] Shutting down plugin 'INNODB_CMP_PER_INDEX'
2023-08-28 16:26:24 1 [Note] Shutting down plugin 'INNODB_CMPMEM_RESET'
2023-08-28 16:26:24 1 [Note] Shutting down plugin 'INNODB_CMPMEM'
2023-08-28 16:26:24 1 [Note] Shutting down plugin 'INNODB_CMP_RESET'
2023-08-28 16:26:24 1 [Note] Shutting down plugin 'INNODB_CMP'
2023-08-28 16:26:24 1 [Note] Shutting down plugin 'INNODB_LOCK_WAITS'
2023-08-28 16:26:24 1 [Note] Shutting down plugin 'INNODB_LOCKS'
2023-08-28 16:26:24 1 [Note] Shutting down plugin 'INNODB_TRX'
2023-08-28 16:26:24 1 [Note] Shutting down plugin 'BLACKHOLE'
2023-08-28 16:26:24 1 [Note] Shutting down plugin 'ARCHIVE'
2023-08-28 16:26:24 1 [Note] Shutting down plugin 'MRG_MYISAM'
2023-08-28 16:26:24 1 [Note] Shutting down plugin 'MyISAM'
2023-08-28 16:26:24 1 [Note] Shutting down plugin 'MEMORY'
2023-08-28 16:26:24 1 [Note] Shutting down plugin 'CSV'
2023-08-28 16:26:24 1 [Note] Shutting down plugin 'sha256_password'
2023-08-28 16:26:24 1 [Note] Shutting down plugin 'mysql_old_password'

Helm Chart for an etcd Cluster

Daniel Puig Gerarde — Sun, 06 Aug 2023 03:43:52 +0000

Deploying applications on Kubernetes can often be a complex process. Helm charts simplify this by creating a package of pre-configured Kubernetes resources. In this post, we're going to create a Helm chart to deploy a production-ready etcd cluster.

What is a Helm Chart?

A Helm chart is a collection of files that describe a related set of Kubernetes resources. A single chart might be used to deploy something simple, like a memcached pod, or something complex, like a full web application stack with HTTP servers, databases, caches, and so on.

Prerequisites

A Kubernetes Cluster ready
kubectl
Helm3

Installation

First, download Helm's installation script:

$ curl https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3 > get_helm.sh

Next, give the script executable permissions:

$ chmod 700 get_helm.sh

Finally, run the installation script:

$ ./get_helm.sh

Setting Up the Helm Chart

Creating a Helm chart involves creating a specific directory structure with several different files. Here's a simple outline of our etcd Helm chart:

etcd/
├── Chart.yaml
├── templates/
│   ├── _helpers.tpl
│   ├── service.yaml
│   ├── statefulset.yaml
│   └── pvc.yaml
└── values.yaml

Understanding the Chart Files

Chart.yaml: This is where we define the chart's metadata.
Here's an example:

apiVersion: v3
name: etcd
version: 0.1.0
description: A Helm chart for deploying an etcd cluster in Kubernetes

templates/: This directory contains the template files that generate Kubernetes manifest files when the chart is installed.

_helpers.tpl: This is a place to put template helpers that you can reuse throughout your chart.

{{/* Generate the name */}}
{{- define "etcd.fullname" -}}
{{- .Chart.Name }}-{{ .Release.Name }}
{{- end -}}

service.yaml: This file defines a Service, which can be used to access your etcd cluster from other applications in your Kubernetes cluster.

apiVersion: v1
kind: Service
metadata:
  name: {{ include "etcd.fullname" . }}
spec:
  clusterIP: None
  ports:
    - port: 2379
      name: client
    - port: 2380
      name: peer
  selector:
    app: etcd

statefulset.yaml: This file defines a StatefulSet, which is used to manage the etcd Pods.

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: {{ include "etcd.fullname" . }}
spec:
  serviceName: "{{ include "etcd.fullname" . }}"
  replicas: {{ .Values.replicaCount }}
  selector:
    matchLabels:
      app: etcd
  template:
    metadata:
      labels:
        app: etcd
    spec:
      containers:
      - name: etcd
        image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"
        ports:
        - containerPort: 2379
          name: client
        - containerPort: 2380
          name: peer
        volumeMounts:
        - name: data
          mountPath: /var/run/etcd
  volumeClaimTemplates:
  - metadata:
      name: data
    spec:
      accessModes: [ "ReadWriteOnce" ]
      storageClassName: {{ .Values.storageClass }}
      resources:
        requests:
          storage: {{ .Values.storageSize }}

pvc.yaml: Persistent Volume Claim (PVC) for storage of etcd data.

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: data
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: {{ .Values.storageClass }}
  resources:
    requests:
      storage: {{ .Values.storageSize }}

values.yaml: This is where you define the default values for your chart. Here's a simple example:

replicaCount: 3
image:
  repository: quay.io/coreos/etcd
  tag: v3.5.0
storageClass: standard
storageSize: 1Gi

Run your etcd Helm chart

Make sure you are in the proper kubectl context and create a namespace named etcd.

$ kubectl create namespace etcd

Navigate to the directory containing the Chart files

$ helm install etcd-cluster . --namespace etcd

Output:

NAME: etcd-cluster
LAST DEPLOYED: Sat Aug  5 23:15:25 2023
NAMESPACE: etcd
STATUS: deployed
REVISION: 1
TEST SUITE: None

Get the resources

$ kubectl get all -n etcd

Output:

NAME                      READY   STATUS    RESTARTS   AGE
pod/etcd-etcd-cluster-0   1/1     Running   0          104s
pod/etcd-etcd-cluster-1   1/1     Running   0          96s
pod/etcd-etcd-cluster-2   1/1     Running   0          91s

NAME                        TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)             AGE
service/etcd-etcd-cluster   ClusterIP   None         <none>        2379/TCP,2380/TCP   104s

NAME                                 READY   AGE
statefulset.apps/etcd-etcd-cluster   3/3     104s

Conclusion

This Helm chart sets up an etcd cluster on a Kubernetes environment, using best practices such as StatefulSets for pod identity and PVCs for data persistence.

This setup, however, is quite basic and might need to be adjusted based on your specific use case or production environment. Always remember to test your Helm charts thoroughly in a staging environment before deploying them in production.

Introduction to Kubernetes StatefulSet

Daniel Puig Gerarde — Fri, 28 Jul 2023 02:15:55 +0000

StatefulSet is one of the workload APIs in Kubernetes that manages the deployment and scaling of a set of Pods. It maintains a sticky identity for each of their pods and guarantees that these identities are unique across the cluster. This makes it suitable for applications that require stable, unique network identifiers, stable persistent storage, and ordered deployment and scaling.

One such application is etcd, a reliable key-value store used in clusters environments for distributed configuration management and service discovery.

Kubernetes StatefulSet in Brief

StatefulSet maintains a sticky, unique identity for each pod, making them a perfect solution for maintaining the state or identity of an application. StatefulSets are crucial when running distributed computing systems or systems that maintain their own client connections, like databases.

StatefulSet's key properties are:

Stable, unique network identifiers: Each pod from a StatefulSet gets a hostname that is derived from the name of the StatefulSet and the ordinal of the Pod.
Stable, persistent storage: Pods in a StatefulSet can have stable persistent storage volumes, ensuring that data persists across pod rescheduling.
Ordered, graceful deployment and scaling: Pods are created sequentially and in reverse order during scaling.

Limitations

etcd in Brief

etcd is a distributed, reliable key-value store that is simple, secure, and fast. It is primarily used as the primary datastore of Kubernetes to store all its cluster state, including the state of nodes, pods, configs, secrets, accounts, roles, bindings, and others.

Key features of etcd include:

Simple: etcd has a simple binary that you can run locally or in a docker container. Its APIs are also simple to understand and use, making it ideal for use with almost any programming language.
Secure: etcd supports automatic TLS with optional client cert authentication.
Fast: A typical etcd cluster can do 10,000 writes per second.
Reliable: With its distributed nature, etcd gracefully handles network partitions and machine failures, even in the leader node.

The following are some of the main use cases of etcd:

Store configuration data: You can use etcd to store configurations for your applications and continuously watch for any changes.
Service discovery: etcd allows systems to announce themselves to others, which can find them using etcd's watch functionality.
Distributed coordination: etcd provides primitives like distributed locks which can be used to build distributed coordination systems.

etcd is written in Go and uses the Raft consensus algorithm to manage a highly-available replicated log.

Deploying etcd as a Kubernetes Cluster Service

Now let's go through the deployment process for an etcd cluster.

etcd StatefulSet Configuration

Create a StatefulSet configuration file named etcd-statefulset.yaml.

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: etcd
spec:
  serviceName: "etcd"
  replicas: 3
  selector:
    matchLabels:
      app: etcd
  template:
    metadata:
      labels:
        app: etcd
    spec:
      containers:
      - name: etcd
        image: quay.io/coreos/etcd:v3.4.15
        ports:
        - containerPort: 2379
          name: client
        - containerPort: 2380
          name: peer
        volumeMounts:
        - name: data
          mountPath: /var/run/etcd
        env:
        - name: ETCD_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        - name: ETCD_INITIAL_CLUSTER
          value: etcd-0=http://etcd-0.etcd:2380,etcd-1=http://etcd-1.etcd:2380,etcd-2=http://etcd-2.etcd:2380
        - name: ETCD_INITIAL_CLUSTER_STATE
          value: new
        - name: ETCD_DATA_DIR
          value: /var/run/etcd/default.etcd
  volumeClaimTemplates:
  - metadata:
      name: data
    spec:
      accessModes: [ "ReadWriteOnce" ]
      resources:
        requests:
          storage: 1Gi

This configuration declares a StatefulSet named etcd that runs 3 replicas of the etcd server.

Deploy the StatefulSet

Apply the StatefulSet configuration using the kubectl apply command.

kubectl apply -f etcd-statefulset.yaml

Verifying the etcd Cluster

Use the following command to check if the etcd pods are running:

kubectl get pods -l "app=etcd"

Output:

NAME     READY   STATUS    RESTARTS      AGE
etcd-0   1/1     Running   1 (26s ago)   57s
etcd-1   1/1     Running   1 (25s ago)   56s
etcd-2   1/1     Running   1 (23s ago)   54s

Accessing the etcd Cluster

For accessing the etcd service, you can expose it as a NodePort or LoadBalancer service. Here is an example of how to expose it as a NodePort service:

apiVersion: v1
kind: Service
metadata:
  name: etcd-client
spec:
  type: NodePort
  ports:
    - port: 2379
      targetPort: 2379
      nodePort: 32379
  selector:
    app: etcd

Save the above service configuration as etcd-service.yaml and then use kubectl apply command to deploy it.

kubectl apply -f etcd-service.yaml

Use the following command to check the service:

kubectl get services

Output:

NAME          TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)          AGE
etcd-client   NodePort    10.96.144.250   <none>        2379:32379/TCP   17s

You should now be able to interact with your etcd cluster from any machine that can reach your Kubernetes nodes.

With Kubernetes StatefulSet, you can easily deploy stateful applications like etcd. It provides stable network identity, storage, and ordered, graceful deployment and scaling, which are vital in the real-world applications.

Kubernetes Configuration with Kustomize.

Daniel Puig Gerarde — Fri, 28 Jul 2023 01:25:29 +0000

Kubernetes is undeniably one of the most popular container orchestration tools available today, due to its ability to manage complex containerized applications seamlessly. Despite its powerful feature set, managing Kubernetes configurations can be a daunting task, particularly when dealing with environments that have minor differences. This is where Kustomize comes into play. Kustomize, a standalone tool now native to Kubernetes via kubectl, makes it easy to handle and organize Kubernetes configurations.

What is Kustomize?

Kustomize is a configuration management tool designed specifically for Kubernetes. It was developed to solve the problem of customizing application configurations without the need to edit them directly. It works by allowing you to create a base configuration and then apply overlays to customize it for different environments.

With Kustomize, you can define a "base" configuration with common configuration settings, and then define "overlays" for each of your environments, e.g., development, staging, and production, that modify those base configurations as necessary.

The Benefits of Kustomize

1. Declarative Configuration Management

Kustomize offers a fully declarative approach to configuration management. This means you declare the final state that you want, and Kustomize makes it happen. This model reduces the risk of human error, simplifies application management, and increases the predictability and repeatability of your deployments.

2. Decoupling Configuration

Kustomize helps decouple configuration from the application, which allows you to manage and scale your applications more effectively. This is achieved through the use of "base" and "overlay" configurations, providing a structured and clear separation between common and environment-specific configurations.

3. Reduced Configuration Duplication

By allowing the definition of base configurations, Kustomize helps to reduce duplication. You define common configurations once and apply overlays for specific environments, significantly decreasing the amount of copied and pasted configuration code.

4. Secret and ConfigMap Generation

Kustomize can create Kubernetes Secret and ConfigMap resources from files or literals without requiring you to write the configuration YAML directly. This feature is particularly useful for managing sensitive data, such as API keys or passwords, and reduces the risk of accidentally exposing such data.

5. Template-Free

Unlike Helm, another popular Kubernetes package manager, Kustomize doesn’t rely on templates. Instead, it uses standard Kubernetes YAML files that can be easily validated and processed. This reduces complexity and makes it easier for developers to understand and work with the configurations.

6. Native Integration with kubectl

Kustomize is integrated natively into kubectl from Kubernetes v1.14 onwards. This allows you to use Kustomize’s features directly with kubectl apply -k, making it easier to work with without the need to install additional software.

A Simple Example

Below is a basic example of using Kustomize to handle multi-environment configuration.

First, create a base configuration. Let's say it's a simple deployment of a Nginx server:

Base Configuration

# base/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
spec:
  selector:
    matchLabels:
      app: nginx
  replicas: 1
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.14.2
        ports:
        - containerPort: 80

# base/kustomization.yaml
resources:
- deployment.yaml

Development Environment

In the development environment, you want to increase the number of replicas to 3:

# overlays/development/kustomization.yaml
bases:
- ../../base
patchesStrategicMerge:
- patch.yaml

# overlays/development/patch.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
spec:
  replicas: 3

Staging Environment

In the Q&A environment, you want to keep the number of replicas at 1, but use a different image version:

# overlays/staging/kustomization.yaml
bases:
- ../../base
patchesStrategicMerge:
- patch.yaml

# overlays/staging/patch.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
spec:
  template:
    spec:
      containers:
      - name: nginx
        image: nginx:1.16.1

Production Environment

In the production environment, you want to increase the number of replicas to 5 and use the latest stable Nginx image:

# overlays/production/kustomization.yaml
bases:
- ../../base
patchesStrategicMerge:
- patch.yaml

# overlays/production/patch.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
spec:
  replicas: 5
  template:
    spec:
      containers:
      - name: nginx
        image: nginx:1.18.0

Now, you can build and apply these configurations with kustomize and kubectl. Here's how you do it for each environment:

~/the-solution
.
├── base
│   ├── deployment.yaml
│   └── kustomization.yaml
└── overlays
    ├── development
    │   ├── kustomization.yaml
    │   └── patch.yaml
    ├── production
    │   ├── kustomization.yaml
    │   └── patch.yaml
    └── staging
        ├── kustomization.yaml
        └── patch.yaml

# Apply the development configuration
kustomize build overlays/development | kubectl apply -f -

# Apply the Q&A configuration
kustomize build overlays/staging | kubectl apply -f -

# Apply the production configuration
kustomize build overlays/production | kubectl apply -f -

With this kind of setup, Kustomize lets you manage your different environments in an organized and efficient manner, keeping your base configuration clean and clear and minimizing duplication.

Git: The Cornerstone of Efficient Software Development

Daniel Puig Gerarde — Sun, 23 Jul 2023 20:37:29 +0000

Introduction

In the world of software development, change is a constant. As teams collaborate on complex projects, managing the changes in the codebase becomes an overwhelming task. It's here that the power of version control systems comes into play, with Git being the most recognized tool in the industry. Git has truly revolutionized the way developers work, turning collaboration from a chaotic puzzle into a well-orchestrated ballet.

What is Git?

Git is a distributed version control system, designed to handle everything from small to large projects with speed and efficiency. It was created by Linus Torvalds, the creator of the Linux operating system kernel.

Unlike traditional version control systems that rely on a central server to store all versions of a project's files, Git gives every developer their own local copy of the entire project. This paradigm shift offers tremendous speed, data integrity, and support for distributed, non-linear workflows.

Here are some of the benefits of using Git:

Version control: Git allows you to track every change that is made to your code, so you can easily revert to a previous version if something goes wrong. This is essential for ensuring the stability of your software.
Collaboration: Git makes it easy for developers to collaborate on projects. Each developer can have their own local copy of the code, and they can make changes without affecting anyone else's work. This makes it easy to get feedback and iterate on code quickly.
Deployment: Git can be used to automate the deployment of software. This means that you can deploy new versions of your software with confidence, knowing that you can always roll back to a previous version if necessary.

Importance of Git in the Industry

Efficient Collaboration

Git's primary benefit is that it allows software developers to simultaneously work on different aspects of the same project, without interfering with each other. In a team setting, this collaborative power is essential. Git provides various mechanisms (like branching and merging) to isolate the changes until they are ready to be shared or integrated with the rest of the team's codebase.

Maintaining Project History

With Git, every change to a project's codebase is tracked. Developers can view who made changes, what changes were made, and why those changes were made. This detailed history aids debugging, helps in understanding how the codebase evolved, and makes onboarding new team members more straightforward.

Facilitating DevOps & GitOps

Git plays an instrumental role in DevOps and GitOps. In DevOps, Git provides a common platform for developers and operations teams to collaborate. It helps in implementing Continuous Integration/Continuous Deployment (CI/CD) workflows, where changes are integrated, tested, and deployed frequently.

In GitOps, Git is used as a single source of truth. Infrastructure code is stored and versioned in Git, and changes to infrastructure are made by modifying and merging this code. This brings the same version control principles used in software development to infrastructure management.

Code Review and Quality Assurance

With Git, it's easy to perform code reviews. Before merging changes, other team members can review the code, providing an extra layer of quality assurance. Code reviews also promote knowledge sharing within the team.

Real-World Git Use-Cases

Git's broad adoption in the industry is a testament to its utility. It's used by large tech companies like Google, Facebook, and Microsoft, as well as by small startups. Open source projects on platforms like GitHub and GitLab rely on Git for collaborative development.

Moreover, Git is not limited to software development. Many other fields, such as Data Science, also leverage Git for versioning and collaboration.

Conclusion

In a landscape that demands collaboration, efficiency, and quality assurance, Git is indispensable. Its robust, distributed architecture facilitates a seamless software development process, powering both small-scale projects and enterprise-level systems. As we move towards more collaborative, DevOps-centric models, Git's importance is set to grow even further.

The beauty of Git lies in its simplicity and flexibility. With a little effort to learn its commands and understand its workflows, you can supercharge your software development practices. As the industry continues to evolve, there's no doubt that Git will continue to be a foundational tool in the developer's toolkit.

Here are some additional resources that you may find helpful:

Git documentation: https://git-scm.com/docs/git
Git tutorial: https://try.github.io/
GitOps: https://www.gitops.tech/

Ansible Playbook - Install and Configure Nginx

Daniel Puig Gerarde — Sat, 22 Jul 2023 02:07:53 +0000

In today's post, we are going to discuss Ansible, a powerful IT automation tool that can configure systems, deploy software, and orchestrate more advanced IT tasks. We will create a simple Ansible playbook that installs and configures Nginx to host static websites.

Prerequisites
To follow this guide, you need:

An Ansible control node: This is the machine where Ansible is installed and from where all tasks and playbooks will be executed. You can install Ansible on a machine running Linux, macOS, or Unix.
At least one managed node: This is the host you are managing with Ansible. In our case, this is the server on which we will install Nginx. (In the case of today we will use Ubuntu 22.04 LTS)
SSH access from the control node to the managed node(s). For security, it's recommended to use key-based SSH authentication.
In this example we will use the access of the user called ubuntu and an Ansible inventory file. The Ansible inventory file is a crucial part of Ansible configurations. It is a text file (usually named 'hosts' by convention, but can be named anything) that defines the hosts and groups of hosts upon which commands, modules, tasks, and playbooks will operate. More details

Steps

Create the Nginx Configuration File

On your control node, create a new file called static-site-config:

server {
    listen 80;
    server_name your_domain.com;

    location / {
        root /var/www/html;
        index index.html;
    }
}

Create the Ansible Playbook

Playbooks are the language by which Ansible orchestrates, configures, administers, or deploys systems. They are written in YAML.

On your control node, create a new file called nginx-setup.yml and add the following content:

---
- name: Install and configure nginx for static website hosting
  hosts: webservers
  become: yes

  tasks:
    - name: Install nginx
      apt:
        name: nginx
        state: latest
        update_cache: yes

    - name: Add user to group root
      user:
        name: "{{ user }}"
        groups: root
        state: present

    - name: Remove default nginx configuration
      file:
        path: /etc/nginx/sites-enabled/default
        state: absent

    - name: Add nginx configuration
      copy:
        src: <control-node-path-to>/static-site-config
        dest: /etc/nginx/sites-available/
        owner: "{{ user }}"
        group: root
        mode: '0644'

    - name: Enable website configuration
      file:
        src: /etc/nginx/sites-available/static-site-config
        dest: /etc/nginx/sites-enabled/static-site-config
        state: link

    - name: Ensure nginx is running
      systemd:
        name: nginx
        state: started
        enabled: yes
  vars:
    user: "ubuntu"

Replace your_domain.com with your actual domain. This configuration will serve files from the /var/www/html directory.

Run the Playbook

After you've defined your inventory and created your playbook and Nginx configuration file, you can run the playbook with the following command:

ANSIBLE_HOST_KEY_CHECKING=False ansible-playbook -i hosts.ini nginx-setup.yml -u ubuntu

ANSIBLE_HOST_KEY_CHECKING

Host key checking is a security feature of SSH, which is also used by Ansible, as Ansible primarily communicates with the machines it manages via SSH.

The first time you connect to a new host via SSH, you'll see a prompt similar to this:

The authenticity of host 'hostname (192.168.0.1)' can't be established.
ECDSA key fingerprint is SHA256:3w1zq...Do you want to continue connecting (yes/no)?

If you answer "yes", the host's key is added to the known_hosts file in your .ssh directory. Any subsequent SSH connections to this host will not prompt you again.

This mechanism is designed to prevent man-in-the-middle attacks. If a host's key changes (which could indicate that you're connecting to a different machine impersonating the host you intended to connect to), SSH will warn you and refuse to connect.

By default, Ansible also uses host key checking for the same reason. But in some situations (e.g., when you frequently recreate virtual machines with different keys), it can be inconvenient.

If you want to disable host key checking in Ansible, you can do so by setting the host_key_checking option to False in the Ansible configuration file, or by setting the ANSIBLE_HOST_KEY_CHECKING=False environment variable.

Note that while disabling host key checking can make Ansible easier to use in certain scenarios, it also bypasses an important security feature of SSH, and should only be done if you understand the risks involved.

Explain the Playbook

This playbook will do the following:

It targets hosts in the group "webservers". You need to replace this with the actual group or hosts you want to target in your Ansible inventory file.

become: yes tells Ansible to use sudo to execute the commands.

It installs the latest version of Nginx using the apt module.

It removes the default Nginx configuration file.

It copies a new Nginx configuration file from the control node to the managed node. You need to replace <control-node-path-to>/static-site-config with the path to your Nginx configuration file on the control node.

It creates a symbolic link from the sites-available directory to the sites-enabled directory, enabling our new configuration.

It makes sure that the Nginx service is running and will start on boot.

And there you have it. You have just created and run an Ansible playbook to install and configure Nginx for static website hosting. You can now access your static website at your domain.

DEV Community: Daniel Puig Gerarde

Understanding eBPF: A Game-Changer for Linux Kernel Extensions

What is eBPF?

Why is eBPF Important?

1. Safety and Security

2. Performance

3. Flexibility and Versatility

4. Programmability

5. Widespread Adoption

Real-World Applications

Building a Semantic Search Engine with OpenAI, Go, and PostgreSQL (pgvector)

What Are Embeddings?

Why PostgreSQL with pgvector?

Overview of Our Application

Prerequisites

Conclusion

Setting Up an Ubuntu Dev Environment with Multipass and VS Code Remote-SSH

Why Develop in Ubuntu While Keeping Your Desktop OS?

What You'll Need

Step 1: Install Multipass

Step 2: Set Up VS Code with Remote-SSH

Step 3: Set Up Your Development Tools in Ubuntu

Step 4: Set Up for Embedded Development, Robotics, and Cybersecurity

Step 5: Maintenance and Snapshot Management

Why This Setup Works

Container Runtime Interfaces (CRI)

Understanding Container Runtimes

Key Objectives of CRI:

Introducing containerd

Key Features:

containerd and CRI: How They Work Together

Workflow:

Benefits of Using containerd

1. Simplicity and Focus

2. Performance

3. Compliance and Standardization

4. Extensibility

Comparing containerd with Other Runtimes

containerd vs. Docker

containerd vs. CRI-O

Getting Started with containerd

Installation

Conclusion

Google Cloud Run: A serverless platform for containerized applications

Use cases for Cloud Run

Example

Tools

Debug

Deploy

Concurrency in Go using Goroutines and Channels.

Introduction

What is a Goroutine?

Synchronization Using Channels

Advantages of Using Goroutines

Goroutines vs Threads

Examples:

Using sync.WaitGroup for Synchronization

Parallel Summation using Goroutines

Using select with Channels

Timer and Ticker

A Comprehensive Guide to Building Kubernetes Operators with Kubebuilder

Install Kubebuilder

Init/bootstrap the project

Create an API

Custom Resource Definition (CRD)

Implement the Controller Logic

Run Operator Locally (For Development)

Deploy Custom Resources

Helm Chart for an etcd Cluster

What is a Helm Chart?

Prerequisites

Setting Up the Helm Chart

Understanding the Chart Files

Run your etcd Helm chart

Conclusion

Introduction to Kubernetes StatefulSet

Kubernetes StatefulSet in Brief

etcd in Brief

Deploying etcd as a Kubernetes Cluster Service

Kubernetes Configuration with Kustomize.

Using `sync.WaitGroup` for Synchronization