DEV Community: Christian Seki

PostgreSQL Performance Tuning: The Power of work_mem

Christian Seki — Sun, 01 Dec 2024 23:15:34 +0000

Briefing

Years ago, I was tasked with solving a performance issue in a critical system for the company I worked at. It was a tough challenge, sleepless nights and even more hair loss, The backend used PostgreSQL, and after a lot of effort and digging, the solution turned out to be as simple as one line:

ALTER USER foo SET work_mem='32MB';

Now, to be honest, this might or might not solve your performance issue right away. It depends heavily on your query patterns and the workload of your system. However, if you're a backend developer, I hope this post adds another tool to your arsenal for tackling problems, especially with PostgreSQL 😄

In this post, we’ll create a scenario to simulate performance degradation and explore some tools to investigate the problem, like EXPLAIN, k6 for load testing, and even a dive into PostgreSQL’s source code. I’ll also share some articles to help you solve related issues.

➡️ github repository with the complete implementation

Case Study

Let’s create a simple system to analyze soccer player performance. For now, the only business rule is to answer this question:

Who are the top N players involved in scoring the most?

The following SQL creates our data model and populates it:

CREATE TABLE players (
    player_id SERIAL PRIMARY KEY,
    nationality TEXT,
    age INT,
    position TEXT
);

CREATE TABLE matches (
    match_id SERIAL PRIMARY KEY,
    match_date DATE,
    home_team TEXT,
    away_team TEXT
);

CREATE TABLE player_stats (
    player_stat_id SERIAL PRIMARY KEY,
    player_id INT REFERENCES players(player_id),
    match_id INT REFERENCES matches(match_id),
    goals INT,
    assists INT,
    minutes_played INT
);

-- Populate players with a range of nationalities, ages, and positions
INSERT INTO players (nationality, age, position)
SELECT
    ('Country' || (1 + random()*100)::int),  -- 100 different nationalities
    (18 + random()*20)::int,                 -- Ages between 18 and 38
    (ARRAY['Forward', 'Midfielder', 'Defender', 'Goalkeeper'])[ceil(random()*4)::int]
FROM generate_series(1, 10000);

The script to initialize and populate the database is available in the github repository.

Yes, we could design out database to improve system performance, but the main goal here is to explore unoptimized scenarios. Trust me, you'll likely encounter systems like this, where either poor initial design choices or unexpected growth require significant effort to improve performance.

Debugging the problem

To simulate the issue related to the work_mem configuration, let’s create a query to answer this question: Who are the top 2000 players contributing the most to goals?

SELECT p.player_id, SUM(ps.goals + ps.assists) AS total_score
FROM player_stats ps
JOIN players p ON ps.player_id = p.player_id
GROUP BY p.player_id
ORDER BY total_score DESC
LIMIT 2000;

Alright, but how can we identify bottlenecks in this query? Like other DBMSs, PostgreSQL supports the EXPLAIN command, which helps us understand each step executed by the query planner (optimized or not).

We can analyze details such as:

What kind of scan was used? Index scan, Index Only scan, Seq Scan, etc.
Which index was used, and under what conditions?
If sorting is involved, what type of algorithm was used? Does it rely entirely on memory, or does it require disk usage?
The usage of shared buffers.
Execution time estimation.

You can learn more about the PostgreSQL planner/optimizer here:

Talk is cheap

Talk is cheap, so let’s dive into a practical example. First, we’ll reduce the work_mem to its smallest possible value, which is 64kB, as defined in the source code:

    /*
     * workMem is forced to be at least 64KB, the current minimum valid value
     * for the work_mem GUC.  This is a defense against parallel sort callers
     * that divide out memory among many workers in a way that leaves each
     * with very little memory.
     */
    state->allowedMem = Max(workMem, 64) * (int64) 1024;

Next, let’s analyze the output of the EXPLAIN command:

BEGIN; -- 1. Initialize a transaction.

SET LOCAL work_mem = '64kB'; -- 2. Change work_mem at transaction level, another running transactions at the same session will have the default value(4MB).

SHOW work_mem; -- 3. Check the modified work_mem value.

EXPLAIN (ANALYZE, COSTS, VERBOSE, BUFFERS) -- 4. Run explain with options that help us to analyses and indetifies bottlenecks. 
SELECT 
    p.player_id, 
    SUM(ps.goals + ps.assists) AS total_score 
FROM 
    player_stats ps
INNER JOIN 
    players p ON p.player_id = ps.player_id
GROUP BY 
    p.player_id
ORDER BY 
    total_score DESC
LIMIT 2000;
--

QUERY PLAN                                                                                                                                                          |
--------------------------------------------------------------------------------------------------------------------------------------------------------------------+
Limit  (cost=18978.96..18983.96 rows=2000 width=12) (actual time=81.589..81.840 rows=2000 loops=1)                                                                  |
  Output: p.player_id, (sum((ps.goals + ps.assists)))                                                                                                               |
  Buffers: shared hit=667, temp read=860 written=977                                                                                                                |
  ->  Sort  (cost=18978.96..19003.96 rows=10000 width=12) (actual time=81.587..81.724 rows=2000 loops=1)                                                            |
        Output: p.player_id, (sum((ps.goals + ps.assists)))                                                                                                         |
        Sort Key: (sum((ps.goals + ps.assists))) DESC                                                                                                               |
        Sort Method: external merge  Disk: 280kB                                                                                                                    |
        Buffers: shared hit=667, temp read=860 written=977                                                                                                          |
        ->  GroupAggregate  (cost=15076.66..17971.58 rows=10000 width=12) (actual time=40.293..79.264 rows=9998 loops=1)                                            |
              Output: p.player_id, sum((ps.goals + ps.assists))                                                                                                     |
              Group Key: p.player_id                                                                                                                                |
              Buffers: shared hit=667, temp read=816 written=900                                                                                                    |
              ->  Merge Join  (cost=15076.66..17121.58 rows=100000 width=12) (actual time=40.281..71.313 rows=100000 loops=1)                                       |
                    Output: p.player_id, ps.goals, ps.assists                                                                                                       |
                    Merge Cond: (p.player_id = ps.player_id)                                                                                                        |
                    Buffers: shared hit=667, temp read=816 written=900                                                                                              |
                    ->  Index Only Scan using players_pkey on public.players p  (cost=0.29..270.29 rows=10000 width=4) (actual time=0.025..1.014 rows=10000 loops=1)|
                          Output: p.player_id                                                                                                                       |
                          Heap Fetches: 0                                                                                                                           |
                          Buffers: shared hit=30                                                                                                                    |
                    ->  Materialize  (cost=15076.32..15576.32 rows=100000 width=12) (actual time=40.250..57.942 rows=100000 loops=1)                                |
                          Output: ps.goals, ps.assists, ps.player_id                                                                                                |
                          Buffers: shared hit=637, temp read=816 written=900                                                                                        |
                          ->  Sort  (cost=15076.32..15326.32 rows=100000 width=12) (actual time=40.247..49.339 rows=100000 loops=1)                                 |
                                Output: ps.goals, ps.assists, ps.player_id                                                                                          |
                                Sort Key: ps.player_id                                                                                                              |
                                Sort Method: external merge  Disk: 2208kB                                                                                           |
                                Buffers: shared hit=637, temp read=816 written=900                                                                                  |
                                ->  Seq Scan on public.player_stats ps  (cost=0.00..1637.00 rows=100000 width=12) (actual time=0.011..8.378 rows=100000 loops=1)    |
                                      Output: ps.goals, ps.assists, ps.player_id                                                                                    |
                                      Buffers: shared hit=637                                                                                                       |
Planning:                                                                                                                                                           |
  Buffers: shared hit=6                                                                                                                                             |
Planning Time: 0.309 ms                                                                                                                                             |
Execution Time: 82.718 ms                                                                                                                                    |

COMMIT; -- 5. You can also execute a ROLLBACK, in case you want to analyze queries like INSERT, UPDATE and DELETE.

We can see that the execution time was 82.718ms, and the Sort Algorithm used was external merge. This algorithm relies on disk instead of memory because the data exceeded the 64kB work_mem limit.

For your information, the tuplesort.c module flags when the Sort algorithm will use disk by setting the state to SORTEDONTAPE at this line. Disk interactions is handled by the logtape.c module.

If you're a visual person (like me), there are tools that can help you understand the EXPLAIN output, such as https://explain.dalibo.com/. Below is an example showing a node with the Sort step, including details like Sort Method: external merge and Sort Space Used: 2.2MB:

The "Stats" section is especially useful for analyzing more complex queries, as it provides execution time details for each query node. In our example, it highlights a suspiciously high execution time—nearly 42ms—in one of the Sort nodes:

You can visualize and analyze this query plan here: https://explain.dalibo.com/plan/2gd0a8c8fab6a532#stats

As the EXPLAIN output shows, one of the main reasons for the performance problem is the Sort node using disk. A side effect of this issue, especially in systems with high workloads, is spikes in Write I/O metrics (I hope you’re monitoring these; if not, good luck when you need them!). And yes, even read-only queries can cause write spikes, as the Sort algorithm writes data to temporary files.

Solution

When we execute the same query with work_mem=4MB (the default in PostgreSQL), the execution time decreases by over 50%.

EXPLAIN (ANALYZE, COSTS, VERBOSE, BUFFERS) 
SELECT 
    p.player_id, 
    SUM(ps.goals + ps.assists) AS total_score 
FROM 
    player_stats ps
INNER JOIN 
    players p ON p.player_id = ps.player_id
GROUP BY 
    p.player_id
ORDER BY 
    total_score DESC
LIMIT 2000;
--
QUERY PLAN                                                                                                                                          |
----------------------------------------------------------------------------------------------------------------------------------------------------+
Limit  (cost=3646.90..3651.90 rows=2000 width=12) (actual time=41.672..41.871 rows=2000 loops=1)                                                    |
  Output: p.player_id, (sum((ps.goals + ps.assists)))                                                                                               |
  Buffers: shared hit=711                                                                                                                           |
  ->  Sort  (cost=3646.90..3671.90 rows=10000 width=12) (actual time=41.670..41.758 rows=2000 loops=1)                                              |
        Output: p.player_id, (sum((ps.goals + ps.assists)))                                                                                         |
        Sort Key: (sum((ps.goals + ps.assists))) DESC                                                                                               |
        Sort Method: top-N heapsort  Memory: 227kB                                                                                                  |
        Buffers: shared hit=711                                                                                                                     |
        ->  HashAggregate  (cost=2948.61..3048.61 rows=10000 width=12) (actual time=38.760..40.073 rows=9998 loops=1)                               |
              Output: p.player_id, sum((ps.goals + ps.assists))                                                                                     |
              Group Key: p.player_id                                                                                                                |
              Batches: 1  Memory Usage: 1169kB                                                                                                      |
              Buffers: shared hit=711                                                                                                               |
              ->  Hash Join  (cost=299.00..2198.61 rows=100000 width=12) (actual time=2.322..24.273 rows=100000 loops=1)                            |
                    Output: p.player_id, ps.goals, ps.assists                                                                                       |
                    Inner Unique: true                                                                                                              |
                    Hash Cond: (ps.player_id = p.player_id)                                                                                         |
                    Buffers: shared hit=711                                                                                                         |
                    ->  Seq Scan on public.player_stats ps  (cost=0.00..1637.00 rows=100000 width=12) (actual time=0.008..4.831 rows=100000 loops=1)|
                          Output: ps.player_stat_id, ps.player_id, ps.match_id, ps.goals, ps.assists, ps.minutes_played                             |
                          Buffers: shared hit=637                                                                                                   |
                    ->  Hash  (cost=174.00..174.00 rows=10000 width=4) (actual time=2.298..2.299 rows=10000 loops=1)                                |
                          Output: p.player_id                                                                                                       |
                          Buckets: 16384  Batches: 1  Memory Usage: 480kB                                                                           |
                          Buffers: shared hit=74                                                                                                    |
                          ->  Seq Scan on public.players p  (cost=0.00..174.00 rows=10000 width=4) (actual time=0.004..0.944 rows=10000 loops=1)    |
                                Output: p.player_id                                                                                                 |
                                Buffers: shared hit=74                                                                                              |
Planning:                                                                                                                                           |
  Buffers: shared hit=6                                                                                                                             |
Planning Time: 0.236 ms                                                                                                                             |
Execution Time: 41.998 ms                                                                                                                           |

For a visual analysis, check this link: https://explain.dalibo.com/plan/b094ec2f1cfg44f6#

In this EXPLAIN output, one of the Sort nodes now uses an in-memory algorithm, heapsort. For context, the planner opts for heapsort only when it’s cheaper to execute than quicksort. You can dive deeper into the decision-making process in the PostgreSQL source code.

Additionally, the second Sort node, which previously accounted for almost 40ms of execution time, disappears entirely from the execution plan. This change occurs because the planner now selects a HashJoin instead of a MergeJoin, as the hash operation fits in memory, consuming approximately 480kB.

For more details about join algorithms, check out these articles:

Impact on the API

The default work_mem isn’t always sufficient to handle your system’s workload. You can adjust this value at the user level using:

ALTER USER foo SET work_mem='32MB';

Note: If you’re using a connection pool or a connection pooler, it’s important to recycle old sessions for the new configuration to take effect.

You can also control this configuration at the database transaction level. Let’s run a simple API to understand and measure the impact of work_mem changes using load testing with k6:

k6-test.js

    import http from 'k6/http';
    import { check } from 'k6';

    const BASE_URL = __ENV.BASE_URL || 'http://localhost:8080';
    const ENDPOINT = __ENV.ENDPOINT || '/low-work-mem';

    export const options = {
    stages: [
        { duration: '15s', target: 10 }, // ramp up to 10 users
        { duration: '15s', target: 10 },
        { duration: '15s', target: 0 },  // ramp down to 0 users
    ],
    };

    export default function () {
    const res = http.get(`${BASE_URL}${ENDPOINT}`);
    check(res, { 'status is 200': (r) => r.status === 200 });
    }

The API was implemented in Go and exposes two endpoints that execute the query with different work_mem configurations:

main.go

    package main

    import (
        "context"
        "fmt"
        "log"
        "net/http"
        "os"

        "github.com/jackc/pgx/v5/pgxpool"
    )

    func main() {
        ctx := context.Background()

        dbURL := os.Getenv("POSTGRES_URL")
        if dbURL == "" {
            dbURL = "postgres://local:local@localhost:5432/local?pool_max_conns=100&pool_min_conns=10"
        }

        pgx, err := pgxpool.New(ctx, dbURL)
        if err != nil {
            log.Fatalf("Unable to connect to database: %s", err)
        }
        defer pgx.Close()

        top_players_query := `
            SELECT p.player_id, SUM(ps.goals + ps.assists) AS total_score
            FROM player_stats ps
            JOIN players p ON ps.player_id = p.player_id
            GROUP BY p.player_id
            ORDER BY total_score DESC
            LIMIT 2000;
        `

        http.HandleFunc("/optimized-work-mem", func(w http.ResponseWriter, r *http.Request) {
            _, err := pgx.Exec(ctx, top_players_query)
            if err != nil {
                http.Error(w, fmt.Sprintf("Query error: %v", err), http.StatusInternalServerError)
                return
            }

            log.Println("Successfully executed query with work_mem=4MB")

            w.WriteHeader(http.StatusOK)
        })

        http.HandleFunc("/low-work-mem", func(w http.ResponseWriter, r *http.Request) {
            query := fmt.Sprintf(`
            BEGIN;      
            SET LOCAL work_mem = '64kB';
            %s
            COMMIT;
            `, top_players_query)

            _, err := pgx.Exec(ctx, query)
            if err != nil {
                http.Error(w, fmt.Sprintf("Query error: %v", err), http.StatusInternalServerError)
                return
            }

            log.Println("Successfully executed query with work_mem=64kB")

            w.WriteHeader(http.StatusOK)
        })

        log.Println("Starting server on port 8082")
        log.Fatal(http.ListenAndServe("0.0.0.0:8082", nil))
    }

Below is the docker-compose file containing all the dependencies needed to run the load test:

docker-compose.yaml

    version: "3.8"

    services:
    postgres:
        image: postgres:17
        environment:
        POSTGRES_PASSWORD: local
        POSTGRES_USER: local
        POSTGRES_DB: local
        healthcheck:
        test: ["CMD-SHELL", "pg_isready"]
        interval: 10s
        timeout: 5s
        retries: 5
        volumes:
        - ./init_data.sql:/docker-entrypoint-initdb.d/init_data.sql # Initialize the database
        - ./data:/var/lib/postgresql/data  # Mounts the local "data" directory to the container's data directory

        ports:
        - "5432:5432"

    api:
        build:
        context: .
        dockerfile: Dockerfile.api
        environment:
        POSTGRES_URL: postgres://local:local@postgres:5432/local?pool_max_conns=100&pool_min_conns=10
        ports:
        - "8082:8082"
        depends_on:
        - postgres

    k6:
        image: grafana/k6
        entrypoint: [ "k6", "run", "/scripts/k6-test.js" ]
        environment:
        BASE_URL: http://api:8082
        ENDPOINT: /low-work-mem
        volumes:
        - ./k6-test.js:/scripts/k6-test.js
        depends_on:
        - api

    volumes:
    postgres:

We can set the ENDPOINT environment variable to define the scenario to test: /low-work-mem or /optimized-work-mem. Run the test using: docker compose up --abort-on-container-exit. For this example, I used Docker version 20.10.22.

Test ENDPOINT: /low-work-mem - work_mem=64kB

    ============ 64kB work_mem k6 output =============
    | 
    |      ✓ status is 200
    | 
    |      checks.........................: 100.00% 2846 out of 2846
    |      data_received..................: 214 kB  4.7 kB/s
    |      data_sent......................: 245 kB  5.4 kB/s
    |      http_req_blocked...............: avg=8.78µs   min=2.13µs  med=5.25µs   max=2.62ms   p(90)=7.19µs   p(95)=8.1µs   
    |      http_req_connecting............: avg=1.04µs   min=0s      med=0s       max=429.38µs p(90)=0s       p(95)=0s      
    |      http_req_duration..............: avg=108ms    min=61.55ms med=113.45ms max=198.55ms p(90)=138.63ms p(95)=143.01ms
    |        { expected_response:true }...: avg=108ms    min=61.55ms med=113.45ms max=198.55ms p(90)=138.63ms p(95)=143.01ms
    |      http_req_failed................: 0.00%   0 out of 2846
    |      http_req_receiving.............: avg=84.49µs  min=18.43µs med=73.4µs   max=1.08ms   p(90)=109.13µs p(95)=149.98µs
    |      http_req_sending...............: avg=22.68µs  min=6.49µs  med=17.67µs  max=2.37ms   p(90)=25.87µs  p(95)=30.11µs 
    |      http_req_tls_handshaking.......: avg=0s       min=0s      med=0s       max=0s       p(90)=0s       p(95)=0s      
    |      http_req_waiting...............: avg=107.89ms min=61.45ms med=113.35ms max=198.4ms  p(90)=138.52ms p(95)=142.91ms
    |      http_reqs......................: 2846    63.204112/s
    |      iteration_duration.............: avg=108.2ms  min=61.71ms med=113.83ms max=198.77ms p(90)=138.93ms p(95)=143.17ms
    |      iterations.....................: 2846    63.204112/s
    |      vus............................: 1       min=1            max=10
    |      vus_max........................: 10      min=10           max=10
    | 
    | 
    | running (0m45.0s), 00/10 VUs, 2846 complete and 0 interrupted iterations
    | default ✓ [ 100% ] 00/10 VUs  45s

Test ENDPOINT: /optimized-work-mem - work_mem=4MB

    ============ 4MB work_mem k6 output =============
    |      ✓ status is 200
    | 
    |      checks.........................: 100.00% 4275 out of 4275
    |      data_received..................: 321 kB  7.1 kB/s
    |      data_sent......................: 393 kB  8.7 kB/s
    |      http_req_blocked...............: avg=7.18µs  min=1.71µs  med=5.35µs  max=551.7µs  p(90)=7.45µs   p(95)=8.68µs  
    |      http_req_connecting............: avg=630ns   min=0s      med=0s      max=448.7µs  p(90)=0s       p(95)=0s      
    |      http_req_duration..............: avg=71.77ms min=29.99ms med=76.67ms max=168.83ms p(90)=95.3ms   p(95)=100.53ms
    |        { expected_response:true }...: avg=71.77ms min=29.99ms med=76.67ms max=168.83ms p(90)=95.3ms   p(95)=100.53ms
    |      http_req_failed................: 0.00%   0 out of 4275
    |      http_req_receiving.............: avg=90.41µs min=13.88µs med=77.02µs max=3.68ms   p(90)=115.28µs p(95)=159.52µs
    |      http_req_sending...............: avg=21.4µs  min=6.39µs  med=18.21µs max=612.19µs p(90)=27.02µs  p(95)=32.85µs 
    |      http_req_tls_handshaking.......: avg=0s      min=0s      med=0s      max=0s       p(90)=0s       p(95)=0s      
    |      http_req_waiting...............: avg=71.66ms min=29.9ms  med=76.55ms max=168.71ms p(90)=95.18ms  p(95)=100.4ms 
    |      http_reqs......................: 4275    94.931194/s
    |      iteration_duration.............: avg=71.99ms min=30.17ms med=76.9ms  max=169.05ms p(90)=95.5ms   p(95)=100.74ms
    |      iterations.....................: 4275    94.931194/s
    |      vus............................: 1       min=1            max=10
    |      vus_max........................: 10      min=10           max=10
    | 
    | 
    | running (0m45.0s), 00/10 VUs, 4275 complete and 0 interrupted iterations
    | default ✓ [ 100% ] 00/10 VUs  45s

The results demonstrate that the endpoint with a higher work_mem outperformed the one with a lower configuration. The p90 latency dropped by over 43ms, and throughput improved significantly under the test workload.

If percentile metrics are new to you, I recommend studying and understanding them. These metrics are incredibly helpful for guiding performance analyses. Here are some resources to get you started:

Conclusion

After dreaming about the problem, waking up multiple times to try new solutions, and finally discovering that work_mem could help, the next challenge is figuring out the right value for this configuration. 😬

The default value of 4MB for work_mem, like many other PostgreSQL settings, is conservative. This allows PostgreSQL to run on smaller machines with limited computational power. However, we must be cautious not to crash the PostgreSQL instance with out-of-memory errors. A single query, if complex enough, can consume multiple times the memory specified by work_mem, depending on the number of operations like Sorts, Merge Joins, Hash Joins (influenced by hash_mem_multiplier), and more. As noted in the official documentation:

it is necessary to keep this fact in mind when choosing the value. Sort operations are used for ORDER BY, DISTINCT, and merge joins. Hash tables are used in hash joins, hash-based aggregation, memoize nodes and hash-based processing of IN subqueries.

Unfortunately, there’s no magic formula for setting work_mem. It depends on your system’s available memory, workload, and query patterns. The TimescaleDB Team has a tool to autotune and the topic is widely discussed. Here are some excellent resources to guide you:

But at the end of the day, IMHO, the answer is: TEST. TEST TODAY. TEST TOMORROW. ~~TEST FOREVER~~. Keep testing until you find an acceptable value for your use case that enhances query performance without blowing up your database. 😄

Handling Large JSON Files in Go: Naive vs. Chunked Approaches

Christian Seki — Thu, 18 Jul 2024 13:07:57 +0000

Handling large files in memory can be a real headache, especially in environments with limited memory. In this post, we'll explore two different techniques for processing large JSON files in Go. We'll compare the naive approach, which reads the entire file into memory, with a more efficient buffered processing method. We'll use the pprof tool to profile and analyze the memory usage of each approach to see how they impact performance.

Github Repository

The Naive Approach

The naive approach involves reading the entire JSON file into memory and unmarshalling it into Go structures. While this method is straightforward, it can lead to high memory consumption, especially with really large files.

package parser

import (
    "encoding/json"
    "log"
    "os"

    "github.com/iamseki/dev-to/apps/processing-large-json-golang/internal"
)

func NaiveParseFile(filename string) {
    file, err := os.Open(filename)
    if err != nil {
        log.Fatalf("error on open file: %s, error: %v", filename, err)
    }
    defer file.Close()

    users := []internal.User{}
    decoder := json.NewDecoder(file)
    if err := decoder.Decode(&users); err != nil {
        log.Fatalf("error on decode users from json file: %v", err)
    }

    // DO STUFF
}

Full code here

The Efficient Chunked Processing Approach

On the other hand, the chunked processing approach reads the JSON file in a buffered way. This method helps keep memory usage low, making it perfect for large files.

package parser

import (
    "bufio"
    "encoding/json"
    "log"
    "os"

    "github.com/iamseki/dev-to/apps/processing-large-json-golang/internal"
)

func OptimizedParseFile(filename string) {
    file, err := os.Open(filename)
    if err != nil {
        log.Fatalln("error opening file: ", err)
    }
    defer file.Close()

    reader := bufio.NewReader(file)
    decoder := json.NewDecoder(reader)

    token, err := decoder.Token()
    if err != nil {
        log.Fatalln("Error reading opening token: ", err)
    }

    if delim, ok := token.(json.Delim); !ok || delim != '[' {
        log.Fatalln("Expeceted start of JSON array")
    }

    for decoder.More() {
        user := &internal.User{}
        err := decoder.Decode(user)
        if err != nil {
            log.Fatalln("Error decoding JSON user: ", err)
        }

        // DO STUFF
    }

    // Read the closing bracket of the JSON array
    token, err = decoder.Token()
    if err != nil {
        log.Fatalln("Error reading closing token:", err)
    }

    // Check if the closing token is the end of the array
    if delim, ok := token.(json.Delim); !ok || delim != ']' {
        log.Fatalln("Expected end of JSON array")
    }

}

Full code here

Setting up Memory Profiling with pprof

To get a better understanding of the memory usage of both approaches, we'll use the pprof profiling tool. Here's how to set it up in your Go application:

package main

import (
    "flag"
    "log"
    "os"
    "runtime"
    "runtime/pprof"

    "github.com/iamseki/dev-to/apps/processing-large-json-golang/internal/parser"
)

func main() {
    var filename string
    flag.StringVar(&filename, "filename", "defaultfile-1mb.json", "Filename to parse")
    flag.Parse()

    // profiling CPU
    cpu_prof, err := os.Create("cpu-naive.prof")
    if err != nil {
        log.Fatalf("error create cpu.prof: %v", err)
    }
    pprof.StartCPUProfile(cpu_prof)
    defer pprof.StopCPUProfile()

    parser.NaiveParseFile(filename)

    // profiling MEM
    mem_prof, err := os.Create("mem-naive.prof")
    if err != nil {
        log.Fatalf("error create mem.prof: %v", err)
    }
    defer mem_prof.Close()

    runtime.GC() // get up-to-date statistics
    if err := pprof.WriteHeapProfile(mem_prof); err != nil {
        log.Fatal("could not write memory profile: ", err)
    }

Full code here

Analysing Profiling Data

We have two commands to run both approaches. Each command will also collect profiling data:

For the naive approach: yarn nx process-naive processing-large-json-golang --filename=largefile-20240718-083247-100mb.json
For the optimized approach: yarn nx process-optimized processing-large-json-golang --filename=largefile-20240718-083247-100mb.json

Note: Use the filename generated by the script usin: yarn nx generate-file processing-large-json-golang --size=100 for the --filename argument. You can also use the default 1MB file by not passing the argument at all.

Profiling Analysis

To analyze the profiling data, use the Golang tool pprof with the webserver command. More information about the pprof web interface can be found here.

To analyze the memory profile of the optimized version: yarn nx analyze-mem-optimized processing-large-json-golang

To analyze the memory profile of the naive version: yarn nx analyze-mem-naive processing-large-json-golang

Conclusion

Our analysis shows that the buffered processing approach significantly reduces memory usage compared to the naive method. Future work could explore additional optimizations and alternative methods for efficient file processing. Hopeffuly, this is useful for you guys! 😄

iamseki / dev-to

Implementations of dev.to blog posts

Dealing with Race Conditions: A Practical Example

Christian Seki — Sat, 13 Jul 2024 23:24:11 +0000

In your career, you'll encounter Schrödinger's cat problems, situations that sometimes work and sometimes don't. Race conditions are one of these challenges (yes, just one!).

Throughout this blog post, I'll present a real-world example, demonstrate how to reproduce the problem and discuss strategies for handling race conditions using serializable transaction isolation and advisory locks with PostgreSQL.

Inspired by "Designing Data-Intensive Applications," Chapter 7 - Transactions "Weak Isolation Levels"

Github Repository with Practical Example

The application

This application manages on-call shifts for doctors at a hospital. To focus on the race condition problem, let's simplify our scenario. Our app resolves around this single table:



CREATE TABLE shifts (
    id SERIAL PRIMARY KEY,
    doctor_name TEXT NOT NULL,
    shift_id INTEGER NOT NULL,
    on_call BOOLEAN NOT NULL DEFAULT FALSE
);

We have a critical business rule:

Each shift must always have at least one doctor on call.

As you may have guessed, implementing a naive API can lead to race condition scenarios. Consider this hypothetical situation:

Jack and John are both on-call at the hospital during the same shift. At nearly the same time, they decide to request leave. One succeeds, but the other relies on outdated information about how many doctors are on shift. As a result, both end up leaving their shift, breaking the business rule and leaving a specific shift with no doctors on call:



John --BEGIN------doctors on call: 2-------leave on call-----COMMIT--------> (t)
          \                 \                      \             \
           \                 \                      \             \ 
Database ------------------------------------------------------------------> (t)
               /               /                      /              /
              /               /                      /              /
Jack ------BEGIN------doctors on call: 2-----leave on call----COMMIT-------> (t)

Reproducing the problem

The application is a simple API implemented in Golang. Checkout the GitHub repository for instructions on how to run and execute the script to reproduce this race condition scenario. In summary, you'll need to:

Start the server: yarn nx serve hospital-shifts
Run the k6 test to reproduce the race condition scenario: yarn nx test hospital-shifts

The test attempts to call off two doctors simultaneously, hitting endpoints with different approaches: shiftId=1 uses advisory lock, shiftId=2 uses serializable transaction isolation, and shiftId=3 is a naive implementation without concurrency control.

The k6 results will output custom metrics to indicate which shiftId violated the business rule:



     ✓ at least one doctor on call for shiftId=1
     ✓ at least one doctor on call for shiftId=2
     ✗ at least one doctor on call for shiftId=3
      ↳  36% — ✓ 123 / ✗ 217

You'll need tools such as Yarn, Go, K6 and Docker, or you can use DevBox for an easier setup of repository dependencies.

Addressing the Race Condition

The problem occurs when our application makes a decision based on stale data. This can happen if two transactions run almost simultaneously and both try to call off doctors for their shift. One transaction succeeds as expected, but the other, relying on outdated information, also succeeds incorrectly. How can we prevent this undesired behavior? There are a few ways to achieve this, and I'll explore two options backed by PostgreSQL, though similar solutions can be found in other database management systems.

Serializable Transaction Isolation

Serializable Snapshot Isolation automatically detects and prevents anomalies such as the write skew demonstrated by our application.

I won't dive deep into the theory behind transaction isolation, but it is a common topic in many popular database management systems. You can find good materials by searching for snapshot isolation, like this one from the PostgreSQL official documentation on transaction isolation. Additionally, here is the paper that proposed this solution years ago. Talk is cheap, so let's see the code:

First, start the transaction and set the isolation level to Serializable:



    // Init transaction with serializable isolation level
    tx, err := db.BeginTxx(c.Request().Context(),    &sql.TxOptions{
        Isolation: sql.LevelSerializable,
    })

Then, proceed to execute operations. In our case its execute this function:



CREATE OR REPLACE FUNCTION update_on_call_status_with_serializable_isolation(shift_id_to_update INT, doctor_name_to_update TEXT, on_call_to_update BOOLEAN)
RETURNS VOID AS $$
DECLARE
    on_call_count INT;
BEGIN
    -- Check the current number of doctors on call for this shift
    SELECT COUNT(*) INTO on_call_count FROM shifts s WHERE s.shift_id = shift_id_to_update AND s.on_call = TRUE;

    IF on_call_to_update = FALSE AND on_call_count = 1 THEN
        RAISE EXCEPTION '[SerializableIsolation] Cannot set on_call to FALSE. At least one doctor must be on call for this shiftId: %', shift_id_to_update;
    ELSE
        UPDATE shifts s
        SET on_call = on_call_to_update
        WHERE s.shift_id = shift_id_to_update AND s.doctor_name = doctor_name_to_update;
    END IF;

END;
$$ LANGUAGE plpgsql;

Whenever inconsistent scenarios occur due to concurrent execution, the serializable isolation level will allow one transaction to succeed and will automatically rollback the others with this message, so you can safely retry:



ERROR:  could not serialize access due to read/write dependencies among transactions

You can find the complete example in the function updateWithSerializableIsolation.

Advisory Lock

Another way to ensure our business rules are enforced is by explicitly locking the resource for a specific shift. We can achieve this using an Advisory Lock at the transaction level. This type of lock is fully controlled by the application. You can find more information about it here.

It's crucial to note that locks can be applied at both the session and transaction levels. You can explore the various functions available here. In our case, we'll use pg_try_advisory_xact_lock(key bigint) → boolean, which automatically releases the lock after a commit or rollback:



BEGIN;

-- Attempt to acquire advisory lock and handle failure with EXCEPTION
    IF NOT pg_try_advisory_xact_lock(shift_id_to_update) THEN
        RAISE EXCEPTION '[AdvisoryLock] Could not acquire advisory lock for shift_id: %', shift_id_to_update;
    END IF;

-- Perform necessary operations

-- Commit will automatically release the lock
COMMIT;

Here is the complete function used in our application:



-- Function to Manage On Call Status with Advisory Locks, automatic release when the trx commits
CREATE OR REPLACE FUNCTION update_on_call_status_with_advisory_lock(shift_id_to_update INT, doctor_name_to_update TEXT, on_call_to_update BOOLEAN)
RETURNS VOID AS $$
DECLARE
    on_call_count INT;
BEGIN
    -- Attempt to acquire advisory lock and handle failure with NOTICE
    IF NOT pg_try_advisory_xact_lock(shift_id_to_update) THEN
        RAISE EXCEPTION '[AdvisoryLock] Could not acquire advisory lock for shift_id: %', shift_id_to_update;
    END IF;

    -- Check the current number of doctors on call for this shift
    SELECT COUNT(*) INTO on_call_count FROM shifts s WHERE s.shift_id = shift_id_to_update AND s.on_call = TRUE;

    IF on_call_to_update = FALSE AND on_call_count = 1 THEN
        RAISE EXCEPTION '[AdvisoryLock] Cannot set on_call to FALSE. At least one doctor must be on call for this shiftId: %', shift_id_to_update;
    ELSE
        UPDATE shifts s
        SET on_call = on_call_to_update
        WHERE s.shift_id = shift_id_to_update AND s.doctor_name = doctor_name_to_update;
    END IF;
END;
$$ LANGUAGE plpgsql;

You can find the complete example in the function updateWithAdvisoryLock.

Conclusion

Dealing with race conditions, like the write skew scenario we talked about, can be pretty tricky. There's a ton of research and different ways to solve these problems, so definitely check out some papers and articles if you're curious.

These issues can pop up in real-life situations, like when multiple people try to book the same seat at an event or buy the same spot in a theater. They tend to appear randomly and can be hard to figure out, especially if it's your first time dealing with them.

When you run into race conditions, it's important to look into what solution works best for your specific situation. I might do a benchmark in the future to compare different approaches and give you more insights.

I hope this post has been helpful. Remember, there are tools out there to help with these problems, and you're not alone in facing them!

iamseki / dev-to

Implementations of dev.to blog posts

Caching with Nginx

Christian Seki — Mon, 27 Dec 2021 22:42:15 +0000

Cache is your friend 😄

A lot of performance issues can be improved using cache, there is a lot of tools out here but for this brief tutorial I'll choose the well-known Nginx.

Setup 💻

You can clone the branch nginx-cache or follow this tutorial (I'll only cover the nginx stuff)

iamseki / dev-to

Implementations of dev.to blog posts

App

I'll use an app from previous post of mine, which calculates the nth fibonacci result given a number. The app it self is very simple and you can see it very small piece of code from the (repository)[https://github.com/iamseki/dev-to] that I mentioned before.

Nginx

nginx/nginx.conf

events {
  worker_connections 4096;
}

http {
  proxy_cache_path /tmp/nginx/cache levels=1:2 keys_zone=fibo_cache:10m inactive=60m use_temp_path=off;

  server {
    listen 3000;

    location /fibonacci {
      proxy_cache fibo_cache;
      proxy_cache_use_stale error timeout http_500 http_502 http_503 http_504;

      add_header X-Proxy-Cache $upstream_cache_status;

      proxy_pass http://api:8080/fibonacci;
    }

    location /hc {
      proxy_pass http://api:8080/hc;
    }
  }
}

nginx/dockerfile

FROM nginx
COPY nginx.conf /etc/nginx/nginx.conf

Docker-compose

docker-compose.yml

version: "3"

services:
  nginx:
    build: nginx
    ports:
      - "3000:3000"
    volumes:
      - "./cache:/tmp/nginx/cache"
    container_name: "nginx"
  api:
    build: api
    ports:
      - "8080:8080"
    container_name: "api"

Proving cache is our friend

Using k6 we can simulate a bunch of requests that go through Nginx and others that don't, comparing them.
The test will reach 100 requests in the first 30 seconds(about stages in k6) that will randomly retrieve the nth sequence with 7 digits between (9999991 ~ 9999999) to force the usage of API resources and Nginx caching.

k6-test.js

import http from 'k6/http';
import { sleep } from 'k6';

const SLEEP_DURATION = 0.1;

export let options = {
    stages: [
        { duration: "30s", target: 100 },
        { duration: "30s", target: 0 },
    ],
    thresholds: {
        http_req_duration: ['p(99)<100'] // 99% request must complete below 100ms
    }
}

const NGINX_LOCAL_URL = "http://localhost:3000"
const API_LOCAL_URL = "http://localhost:8080"

const BASE_URL = __ENV.API_URL === "NGINX" ? NGINX_LOCAL_URL : API_LOCAL_URL 
const HEADERS = { "Content-Type": "application/json" }

const randomInteger = (min, max) => Math.floor(Math.random() * (max - min + 1)) + min;

export default () => {
    http.get(`${BASE_URL}/fibonacci/${randomInteger(9999991,9999999)}`);
    sleep(SLEEP_DURATION);
}

Running 📋

docker-compose up -d
Testing with requests passing through Nginx: k6 run k6-test.js -e API_URL='NGINX'
Testing requesting directly the API: k6 run k6-test.js

What if API were down ? Try it out, docker stop api and run the tests above again.

Nginx will be able to answer some requests (maybe all, due to our scenario), cool right?

Conclusion 📝

Results from Nginx:

Results from API:

As we can see, caching improves considerably the response time. Sometimes the back-end will be able to handle requests even if the API is down, when I test this feature for the first time I thought it was amazing!

This is a simple example but I hope that could be helpful for people who's thinking to cache some API routes.I know this is not applicable to every scenario, but maybe can open a world of possibilities to solve performance issues without change the API it self.

REAL-TIME bitcoin price in javascript

Christian Seki — Sun, 01 Aug 2021 14:14:57 +0000

Bitcoin is a decentralized digital currency and it's currently trading above $41000 (at the time I write this article).Due to it's volatility it would be quite a boring task to keep an eye on it's price. Lucky us, as we're all programmers so let's automate this!😎

What are we going to code ?

The bridge to cryptocurrencies pricing

There's a lot of exchange in the market but I choose to use Binance to tracking bitcoin price for some reasons:

They expose a websocket server, so we can track cryptocurrencies price real time like.
All we need is to connect to wss://stream.binance.com:9443 WITHOUT AN API KEY !

btw, you can sign up to Binance and start trading anytime.

Initializing the project

If you don't have nodejs installed follow the official website instructions.

Let's start the project and install two tiny dependencies:

npm init -y
npm i ws prompts

ws it's a client/server websocket implementation, will be used as a client to connect to binance's websocket server.
prompts it's a cli tool, will be used to brings flexibility to our code allowing the user to choose what cryptocurrency to keep it's price tracked.

Coding

const prompts = require('prompts');
const WebSocket = require('ws');

const run = async () => {
  const { symbol } = await prompts({
    type: 'text',
    name: 'symbol',
    message: 'What symbol do you wanna track ?'
  });

  const ws = new WebSocket(`wss://stream.binance.com:9443/ws/${symbol.toLowerCase()}@kline_1m`);

  ws.on('message', async (data) => {
    const incomingData = JSON.parse(data.toString());
    if (incomingData.k) {
      const isClosed = incomingData.k.x;
      const symbolPrice = Number(incomingData.k.c);
      console.log(`${symbol.toUpperCase()} : ${symbolPrice} -- closed = ${isClosed}`);
    }
  });
}

run();

That's it ! Isn't simple? I know that it's a lot of things going on under the hood but let's ignoring for now.

The JSON payload received from the socket it's described in binance websocket official documentation.

Conclusion

From that piece of code you can build an even more elegant bot, maybe do some action when getting the cryptocurrency price, triggering your cellphone to waking you up to buy some cryptocurrency, the sky is the limit!
To be honest I've never make money with that but I just wanted to show you guys an interesting and simple real time application.

Load Test with Javascript

Christian Seki — Sun, 04 Apr 2021 20:40:32 +0000

Javascript is everywhere!

Maybe you're not surprised to know that JS enables us to build nearly any sort of application for any platform.

But what I never heard before is the useability to write Javascript Code to execute load and performance testing, and even better collect metrics from these tests to understand the application behavior.

Tell me more...

The tool that has given us this kind of power is 🔧K6🔧.

But what is this exactly? From k6 official site:

k6 is a developer-centric, free, and open-source load testing tool built for making performance testing a productive and enjoyable experience.

k6 is written in Go but exposes a Javascript API allowing us to write javascript code to interact with.

Installation 💾

There's a lot of ways to get the k6 binary, go ahead into their official guide and get one of them.
You can also follow this guide to run your first load test even in a local environment.

Nodejs vs Golang

Let's use k6 to compare the performance from a simple API written in Golang and the other one in Nodejs.

The Node app uses the well known expressjs framework ➡️ app.js
The Go app uses the iris framework ➡️ main.go

Just to Make it quite clear, the App will consume considerable CPU resources being a disadvantage to Nodejs API 😬

Both apps will calculate the nth value of Fibonacci sequence passed as path param, e.g apiUrl/fibonacci/7 must return:



{
  "fibonacci": 34   
}

There's a health check endpoint either: apiUrl/hc I encourage you to call this endpoint manually during the second minute of the load test.

Test Scenario 📋

We're testing just an API endpoint in isolation to see how the endpoint performance trends over time. The scenario is as follow:

In the first minute of the test the system will ramp up until achieving 100 virtual users.
Each virtual user will make an HTTP Request to the endpoint: apiUrl/fibonacci/9999999 every 100ms.
Holding the step 2 for two minutes.
The last minute will ramp down virtual users to 0.

k6 will take 4 minutes to run the test described above.
The test have a simple goal, declared in options object: http_req_duration: ['p(95)<1000'] which means, 95% of the requests made must take less than 1 second.

k6-test.js



import http from 'k6/http';

import { sleep } from 'k6';

const SLEEP_DURATION = 0.1;

export let options = {

    stages: [

        { duration: "1m", target: 100 },

        { duration: "2m", target: 100 },

        { duration: "1m", target: 0 }

    ],

    thresholds: {

        http_req_duration: ['p(95)<1000'] // 99% request must complete below 1s

    }

}

const BASE_URL = __ENV.API_BASE === "GOLANG" ? "http://localhost:8080" : "http://localhost:9090" 

const HEADERS = { "Content-Type": "application/json" }

export default () => {

    http.get(</span><span class="p">${</span><span class="nx">BASE_URL</span><span class="p">}</span><span class="s2">/fibonacci/9999999);

    sleep(SLEEP_DURATION);

}

Running the load test with k6 💪

1.Clone the 📜example code📜 and execute docker-compose up. Both apps will expose these endpoints:

App	Endpoints	Port
Golang	`/hc` `/fibonacci/n`	8080
Nodejs	`/hc` `/fibonacci/n`	9090

2.You can run both tests in parallel opening two terminals or one at time, it's up to you.

To run the Golang load test in the root project folder, execute: k6 run ./k6-test.js -e API_BASE=GOLANG
And to run the Nodejs load test: k6 run ./k6-test.js

I know that it's a really simple test but I like to keep that phrase in mind:

Simple testing is better than no testing

Show me the winner

k6 output for Nodejs app

k6 output for Golang app

You can call the health check endpoint of both apps during the load test to check the response time.

1.The HTTP Request to Nodejs takes on average 15s and the Golang app 237ms.
2.Due to this HTTP request duration, in the same amount of time Golang handles 52103 more requests than Nodejs.

Golang won, but how ? 🤔

Analyzing containers resource usage statistics during the tests, you can see the Golang container uses more than one CPU to handle the requests while Nodejs uses just one CPU.

docker stats

Another important point is that Golang uses a separate goroutine per HTTP request handling the requests in a concurrency way.

Conclusion

In case that you have executed both tests will figure out that Go surpasses NodeJS in this scenario. But why?
Both apps have one main task which is to calculate the nth sequence of a Fibonacci number, and depending on the nth number the task can consuming considerable CPU resources, and Nodejs is not designed for that.
Therefore using k6, you'll be able to catch performance regression and problems earlier. You can also write stress tests to check whether the application gradually scale up its infrastructure, allowing you to build resilient systems and robust applications.

The Select keyword in Golang

Christian Seki — Thu, 01 Apr 2021 16:51:56 +0000

📑 1. Theory
💻 2. Code
📝 3. Conclusion

The select statement

The select statement in Go looks like a switch
statement but for channels

Each case inside a select holds a channel receive or send.
Select waits until one case is ready and then runs it and its associated case statement.
It chooses one at random if multiple channels are ready.
It will block until one of the channels is ready.
I think the select statement is one of the most important part of the Go concurrency model.

Please let me know if you notice any mistakes or I forgot something important 😅

Show me the code

Let's code a classical program that's generate a Multiplication Table.
Take a look at the main function:

type message struct {
    text   string
    number int
}

func main() {
    msg := make(chan message)
    exit := make(chan bool)

    go readInput(msg, exit)
    for {
        select {
        case m := <-msg:
            validateInputAndGetNumber(&m)
            printMultiplicationTable(m)
            fmt.Print("˲Give me a number or exit: ")
        case <-exit:
            fmt.Println("Exiting program...")
            os.Exit(1)
        }
    }
}

message struct is the type of channel that I use to communicate between goroutines. There are just two goroutines in this program, the execution of main function and the execution of readInput function and yes, the main function is executed as a goroutine by go runtime.
Instantiate two channels: msg and exit I hope the variable name is self-explanatory😁 .
go readInput(msg, exit) starts the goroutine to read the user input stdin data sending to the msg channel.
for { select { ... } } it starts an infinite loop with the for keyword. This way we can keep listening to both channels with select forever.

Therefore the goroutine executed by go readInput(msg, exit) is sending data through the msg and exit channels, and the goroutine executed by main() is reading these channels and then when one of the channels is ready, that operation will proceed, and its corresponding statements.

The running code seems like this:

iamseki / dev-to

Implementations of dev.to blog posts

Conclusion

My goal with this simple program is to demonstrate how we can use the select statement to handle incoming data from channels in a fancy way. I hope it could be helpful, especially for new Gophers 😄

Running a Kubernetes local cluster on Windows

Christian Seki — Sun, 28 Mar 2021 23:44:11 +0000

Nowadays we can use WSL Windows Subsystem for Linux technology to run Kubernetes on Windows almost seamlessly! So let's do this 😎

📋 Installation
❌ Known issues
🎡 Playing with kubernetes
✏️ Conclusion

⚠️ First of all check requirements for running WSL 2 ⚠️

▫️ For x64 systems: Version 1903 or higher, with Build 18362 or higher.
▫️ For ARM64 systems: Version 2004 or higher, with Build 19041 or higher.

Configuration Stuff

To install and enable WSL here's the official guide made by Microsoft: https://docs.microsoft.com/en-us/windows/wsl/install-win10
- In this step, I choose the Ubuntu 20.04 distro.
Install Docker Desktop on Windows: https://www.docker.com/products/docker-desktop
Configuring Docker Desktop to use WSL
- In Docker Desktop Settings
  - ▶️ General
  - Then select:
  - ▶️ Resources
  - ➡️ WSL Integration
  - Then select:
  - Apply and Restart!
To check the previous step, open your wsl distro and execute docker version and kubectl version. Both binaries have been installed in your distro.
Finally, just enable Kubernetes in Docker Desktop settings:
This may take a while so be patient.

Known issues

Be patient but don't wait forever. So if your kubernetes installation process stuck in "Kubernetes is starting.." go to the troubleshooting section, Get Suport and waiting for DIAGNOSTICS.
The output log from Docker Desktop, in my case:

cannot get lease for master node: an error on the server (\"\") has prevented the request from succeeding (get leases.coordination.k8s.io docker-desktop)

You can try any of these solutions from stackoverflow:

https://stackoverflow.com/questions/55361963/kubernetes-is-starting-forever-error-on-windows-10

But I had to reset Docker Desktop to default to solve this issue 😥.

Playing with Kubernetes

You can use kubectl commands in wsl distro or directly from Windows, in this case you have to install the binary

kubectl config get-contexts the expected output is all clusters configured in your kubeconfig file but we're interested at:

kubectl get pods -n kube-system to see related kubernetes system
PODs.
kubectl cluster-info to get cluster information.
kubectl get nodes -o wide to get information about the single node running in our cluster.

Deploying a hello world app

kubectl run hello --image=tutum/hello-world --port=80
kubectl get pods to see the hello POD up and running
kubectl port-forward hello 8080:80 to forward local port 8080 to POD port 80 and make it accessible through http://localhost:8080/
kubectl logs hello to see the stdout from hello POD

Conclusion

I know we can't test all kubernetes power with a single node cluster but even with just a single node there's a lot of features and scenarios that we can test and I promise to write a more interesting example than a basic hello world someday.
I really had a great experience using WSL2 and if you are just like me that have to use Windows on the laptop for whatever reason I encourage you to try it!

Build a super minimalistic Docker Image to run your Golang App

Christian Seki — Wed, 24 Mar 2021 01:50:06 +0000

Why should you care about Docker Images size ?

Basically, to speeds up building, deploying and also cut costs with storage and network egress if you're using some cloud provider.
We can achieve minimal Docker images size using base images thats focus on minimalism such Alpine Linux and others strategies like Multi Stage build which is the approach that I use combining with SCRATCH base image.

Practical Example

Let's play with Docker, even if you don't have it installed in your machine, use the 🐋 online lab it's awesome and gives a real Docker experience through a web browser.

When you sign in into the online lab you can clone the demonstration project and guess what ? There's git binary ready to use in the running instance, so:



git clone -b super-minimalistic-docker-image https://github.com/iamseki/dev-to.git

A brief of Multi Stage build

The main idea is divide Dockerfile into multiple stages passing to following stage just the necessary components to run the image properly. Cool, but how ?
The oficial explanation FROM Docker webpage:

Using multiple FROM statements in your Dockerfile. Each FROM instruction can use a different base, and each of them begins a new stage of the build.

Containerizing a Golang Web App

For this example I continue my previous post example exposing a http server to handle a simple GET request.
You can take a look in the full source code in github repo. The Dockerfile was written as follow:



FROM golang:alpine as builder

WORKDIR /app 

COPY . .

RUN CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -ldflags="-w -s" .

FROM scratch

WORKDIR /app

COPY --from=builder /app/dev-to /usr/bin/

ENTRYPOINT ["dev-to"]

Building and checking the image size:

if you didn't do yet cd dev-to
docker build -t dev-to .
docker images | grep dev-to -B 1

Expected output:

Voila ! The generated Docker Image has 4.55MB and it's a runnable web server -> docker run --rm -p 8080:8080 -d dev-to

Make HTTP GET request in the only route exposed by the server doing:



curl localhost:8080/events

The response should be a JSON of fake events.

What if we remove Multi Stage build from Dockerfile ?

Comment some lines or just rewrite the Dockerfile, for example:



FROM golang:alpine as builder

WORKDIR /app 

COPY . .

RUN CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -ldflags="-w -s" .

RUN mv dev-to /usr/bin/

ENTRYPOINT ["dev-to"]

And we got an image of size 324MB.

Conclusion

This kind of approach is great for static languages like Golang but even with these languages that compiles a raw binary to be executed by the OS it's always a good idea to be careful when writing Dockerfiles to optimizing the CI/CD proccess and cut some costs.
The scratch base Image contains nothing so it's lightweight but you can't debug container from inside, to do so you can download binaries with RUN command but I encorage you to use the busybox as base Image, is also lightweight but non-empty so it's possible to debug containers from inside with exec commands such as docker exec -ti container-name sh.

How I mock unit tests in Golang

Christian Seki — Wed, 17 Mar 2021 23:46:25 +0000

First of all I just want you guys to know that I'm not a golang expert, I started to use golang as a hobbyist a few months ago but now every web backend project I use it.

System

Note the arrow direction what means that Domain layer have no Idea about App layer so I promise I'll try to keep this in the code.

That's all ! Lets start as simple as possible to focus on tests it self and maybe further I can add more layers into this system and update this diagram but thats it for now.

Domain Layer

event.go

package domain

import "time"

type Event struct {
    Title           string    
    Date            time.Time 
    Place           string        
    Category        string    
    KeyWords        []string  
}

type EventSaver interface {
    Save(Event) error
}

Event is the struct that represents an event in the system
EventSaver is the interface that will be implemented by someone who wants to save an event somehow, often called as an use case of the system.

Use Cases Layer

For simplicity, I made just one use case named AddEvent Memory that try to save the event and pass the error to who called it.
event.go

package usecases

import (
    "github.com/iamseki/dev-to/domain"
)

type AddInMemoryRepository interface {
    Add(domain.Event) error
}

type AddEventInMemory struct {
    repository AddInMemoryRepository
}

func (usecase *AddEventInMemory) Save(e domain.Event) error {
    err := usecase.repository.Add(e)
    return err
}

func NewAddEventInMemory(r AddInMemoryRepository) *AddEventInMemory {
    return &AddEventInMemory{repository: r}
}

Testing

With the testing built in package we can easily start to write unit tests in go, so lets do it !

Lets focus on this peace of code:

type addEventFakeRepository struct {
    MockAddFn func(domain.Event) error
}

func (fake *addEventFakeRepository) Add(e domain.Event) error {
    return fake.MockAddFn(e)
}

func newAddEventFakeRepository() *addEventFakeRepository {
    return &addEventFakeRepository{
        MockAddFn: func(e domain.Event) error { return nil },
    }
}

addEventFakeRepository struct implements the AddInMemoryRepository interface due to implementation of Add method so we can inject it into add event use case
MockAddFn it's the function used to mock the implementation of AddInMemoryRepository
newAddEventFakeRepository creates an instance of AddEventFakeRepository then mocking the case of success returning error as a nil.

However if the case of success its not desirable, we can set the mock function as follow:

func TestAddEventInMemoryCustom(t *testing.T) {
    r := newAddEventFakeRepository()
    r.MockAddFn = func(e domain.Event) error { 
        // do something diferent here !
    }
    sut := usecases.NewAddEventInMemory(r)
}

Source code of all test in usecases package:
event_test.go

package usecases_test

import (
    "testing"

    "github.com/iamseki/dev-to/domain"
    "github.com/iamseki/dev-to/usecases"
)

type addEventFakeRepository struct {
    MockAddFn func(domain.Event) error
}

func (fake *addEventFakeRepository) Add(e domain.Event) error {
    return fake.MockAddFn(e)
}

func newAddEventFakeRepository() *addEventFakeRepository {
    return &addEventFakeRepository{
        MockAddFn: func(e domain.Event) error { return nil },
    }
}

func TestAddEventInMemorySucceed(t *testing.T) {
    r := newAddEventFakeRepository()
    sut := usecases.NewAddEventInMemory(r)

    err := sut.Save(domain.Event{})
    if err != nil {
        t.Error("Expect error to be nil but got:", err)
    }
}

Considerations

I hope this can be useful for someone, I really enjoy to mock my tests in the way I describe here but I'm not pretty shure about how idiomatic it is so I open to suggestions !

📜 source code