DEV Community: Walid LAGGOUNE

The Hidden Truth About TCP Reliability: Why Your Data Might Never Arrive

Walid LAGGOUNE — Mon, 04 Aug 2025 19:51:17 +0000

TCP (Transmission Control Protocol) is often described as the "reliable" network protocol. The documentation promises ordered delivery, automatic retransmission of lost packets, and error detection. Yet countless developers have encountered a puzzling phenomenon: data that appears to be sent successfully simply vanishes into the digital ether, never reaching its destination.

This isn't a bug in TCP it's a fundamental misunderstanding of what TCP reliability actually guarantees. The confusion has led to countless hours of debugging, production incidents, and heated discussions on forums. Today, we'll dive deep into this networking mystery and explore why your "reliable" TCP connections might be silently dropping data.

The False Promise of write() Success

Consider this seemingly straightforward scenario: you want to send 1 million bytes from one program to another. Your client code looks something like this:

sock = socket(AF_INET, SOCK_STREAM, 0);
connect(sock, &remote, sizeof(remote));
int bytes_sent = write(sock, buffer, 1000000);
printf("Sent %d bytes\n", bytes_sent); // Prints: "Sent 1000000 bytes"
close(sock);

The write() call returns successfully, indicating all 1 million bytes were "sent." Mission accomplished, right? Wrong.

Here's what actually happens when you call write() on a TCP socket:

Kernel acceptance: The kernel accepts your data and buffers it
Eventual transmission: The kernel will attempt to transmit this data "when it feels like it"
Network traversal: Data packets travel through multiple network adapters and queues
Remote acknowledgment: The remote kernel acknowledges receipt (not the application!)
Application processing: The receiving application must actually read the data

The crucial insight: A successful write() only means the kernel accepted your data nothing more.

The close() Catastrophe

The real trouble begins when you call close() immediately after write(). Here's what the TCP specification (RFC 1122, section 4.2.2.13) says can happen:

"A host MAY implement a 'half duplex' TCP close sequence, so that an application that has called CLOSE cannot continue to read data from the connection. If such a host issues a CLOSE call while received data is still pending in TCP, or if new data is received after CLOSE is called, its TCP SHOULD send a RST to show that data was lost."

In plain English: if there's any unread data on the socket when you call close(), the kernel might immediately terminate the connection with a reset (RST) packet, discarding any data that was still in transit.

The SO_LINGER Red Herring

Most developers discovering this issue quickly stumble upon the SO_LINGER socket option, which seems tailor made for this problem:

"When enabled, a close() or shutdown() will not return until all queued messages for the socket have been successfully sent or the linger timeout has been reached."

Setting SO_LINGER feels like the obvious solution:

struct linger ling = {1, 30}; // Enable with 30 seconds timeout
setsockopt(sock, SOL_SOCKET, SO_LINGER, &ling, sizeof(ling));

Unfortunately, SO_LINGER alone doesn't solve the problem. Even with lingering enabled, pending readable data can still trigger an immediate RST, causing data loss.

Understanding the Root Cause

The fundamental issue is that close() doesn't communicate your actual intent to the kernel. You want to say: "Close this connection after all my data has been successfully delivered." But close() actually says: "I'm done with this socket, tear it down now."

This semantic mismatch leads to three common failure scenarios:

Scenario 1: Unread Data Triggers RST

If the remote end sent any data that you haven't read (even a simple "OK" response), calling close() can trigger an immediate connection reset.

Scenario 2: Buffered Data Discarded

Data still queued in kernel buffers gets discarded when the connection terminates abruptly.

Scenario 3: In Flight Packets Lost

Packets that were transmitted but not yet acknowledged get lost when the connection resets.

The Right Way: shutdown() and Graceful Closure

The solution involves using shutdown() instead of immediately calling close(). Here's the correct pattern:

// Send all your data
write(sock, buffer, data_size);

// Signal that you're done sending
shutdown(sock, SHUT_WR);

// Wait for the remote end to close their side
char dummy_buffer[1024];
while (read(sock, dummy_buffer, sizeof(dummy_buffer)) > 0) {
    // Discard any remaining data
}

// Now it's safe to close
close(sock);

This approach works because:

shutdown(SHUT_WR) sends a FIN packet, signaling you're done sending data
The kernel continues transmitting any buffered data
The remote end eventually closes its side of the connection
read() returns 0 when the remote end closes, confirming graceful shutdown
Only then do you call close()

Linux Specific Solution: SIOCOUTQ

On Linux systems, you can use the SIOCOUTQ ioctl to monitor unacknowledged data:

#include <sys/ioctl.h>
#include <linux/sockios.h>

// Check how much data is still unacknowledged
int unacked_bytes;
ioctl(sock, SIOCOUTQ, &unacked_bytes);

// Wait until all data is acknowledged
while (unacked_bytes > 0) {
    usleep(1000); // Wait 1ms
    ioctl(sock, SIOCOUTQ, &unacked_bytes);
}

// Now it's relatively safe to close
close(sock);

This technique provides stronger guarantees than the shutdown() method because it waits for actual acknowledgment from the remote TCP stack, not just connection closure.

The Gold Standard: Application Level Acknowledgments

The most robust solution is implementing application-level acknowledgments in your protocol:

// Client side
send_data_with_checksum(sock, buffer, size);
wait_for_acknowledgment(sock);

// Server side
bytes_received = receive_data_with_checksum(sock, buffer);
send_acknowledgment(sock, bytes_received, checksum);

This approach guarantees that:

All data was received by the application (not just the kernel)
Data integrity is verified through checksums
The sender knows definitively whether transmission succeeded

Common Misconceptions Debunked

Myth 1: "TCP is always reliable"

Reality: TCP provides reliable transmission, not reliable delivery. There's a crucial difference.

Myth 2: "Successful write() means data was delivered"

Reality: It only means the kernel accepted your data for eventual transmission.

Myth 3: "SO_LINGER solves all problems"

Reality: SO_LINGER helps but doesn't address pending readable data issues.

Myth 4: "Non-blocking sockets fix this"

Reality: Non-blocking I/O doesn't change the fundamental close() semantics.

Real-World Implications

This issue affects many types of applications:

Web servers: HTTP responses might be truncated if connections close prematurely
Database clients: Transaction commits might fail silently
File transfer protocols: Partial uploads/downloads without proper error detection
Real-time systems: Critical control messages might be lost
Microservices: API responses could be incomplete

Best Practices for Reliable Data Delivery

Design your protocol with length information: Include message sizes so receivers know when they have complete data.
Implement application level acknowledgments: Don't rely solely on TCP's transport level guarantees.
Use graceful connection shutdown: Always use the shutdown() → read() → close() pattern when possible.
Monitor connection state: Use platform specific tools like SIOCOUTQ to verify data transmission.
Handle partial reads: Network I/O can be partial; always loop until you've read the expected amount.
Add timeouts: Don't wait forever for acknowledgments or connection closure.
Test failure scenarios: Simulate network interruptions, process crashes, and high load conditions.

Code Example: Robust TCP Client

Here's a complete example implementing these best practices:

#include <sys/socket.h>
#include <sys/ioctl.h>
#include <linux/sockios.h>
#include <unistd.h>
#include <errno.h>

int reliable_send(int sock, const void* data, size_t size) {
    const char* ptr = (const char*)data;
    size_t remaining = size;

    // Send all data
    while (remaining > 0) {
        ssize_t sent = write(sock, ptr, remaining);
        if (sent < 0) {
            if (errno == EINTR) continue;
            return -1; // Error
        }
        ptr += sent;
        remaining -= sent;
    }

    // Signal end of transmission
    if (shutdown(sock, SHUT_WR) < 0) {
        return -1;
    }

    // Wait for all data to be acknowledged (Linux-specific)
    int unacked;
    do {
        if (ioctl(sock, SIOCOUTQ, &unacked) < 0) {
            return -1;
        }
        if (unacked > 0) {
            usleep(1000); // Wait 1ms
        }
    } while (unacked > 0);

    // Wait for remote closure
    char dummy[1024];
    while (read(sock, dummy, sizeof(dummy)) > 0) {
        // Discard remaining data
    }

    return 0; // Success
}

Conclusion

The myth of TCP reliability has misled developers for decades. While TCP does provide crucial guarantees about packet ordering and error detection, it doesn't guarantee that calling write() followed by close() will deliver your data.

Understanding this distinction is crucial for building robust networked applications. By implementing proper connection shutdown procedures, monitoring transmission state, and adding application level acknowledgments, you can achieve true end to end reliability.

The next time you're designing a network protocol or debugging mysterious data loss issues, remember: TCP's reliability guarantees end at the network layer. Everything above that including ensuring your application actually receives the data is up to you.

Don't let the "reliable" in "Reliable Transmission Protocol" fool you. True reliability requires careful protocol design, proper error handling, and a deep understanding of what TCP actually promises to deliver.

This article was inspired by the excellent technical analysis at The Ultimate SO_LINGER page by Bert Hubert. The fundamental insights remain as relevant today as they were in 2009.

Understanding Go Concurrency Pipelines

Walid LAGGOUNE — Tue, 08 Jul 2025 08:58:41 +0000

When I first started learning Go, the idea of concurrency sounded exciting but honestly, a bit mysterious too. Goroutines, channels, fan-out, fan-in... all the buzzwords were floating around, but I didn’t know how to piece them together in a meaningful way.

So I decided to build a small project to help things click. The goal was simple: generate random numbers, filter out the prime ones, and do it as fast as possible by using all the CPU cores. Along the way, I ended up using most of the core concurrency concepts that Go offers. Let me walk you through what I built and what I learned.

The Big Idea

Here’s what the program does:

Continuously generates random integers
Uses multiple workers to check if each number is a prime
Combines the output of those workers into one stream
Stops after printing 10 prime numbers

Along the way, we’ll use channels, goroutines, and cancellation signals and we’ll see how concurrency in Go just fits together naturally.

The Full Code

Here's the complete code.

package main

import (
    "fmt"
    "math/rand"
    "runtime"
    "sync"
)

// Generator: produces values using a function until canceled
func generator[T any, K any](done <-chan K, fn func() T) <-chan T {
    stream := make(chan T)
    go func() {
        defer close(stream)
        for {
            select {
            case <-done:
                return
            case stream <- fn():
            }
        }
    }()
    return stream
}

// Take: limits how many values we read from a stream
func take[T any, K any](done <-chan K, stream <-chan T, n int) <-chan T {
    taken := make(chan T)
    go func() {
        defer close(taken)
        for i := 0; i < n; i++ {
            select {
            case <-done:
                return
            case taken <- <-stream:
            }
        }
    }()
    return taken
}

// Prime finder: filters primes from a stream of integers
func primeFinder(done <-chan int, randIntStream <-chan int) <-chan int {
    isPrime := func(n int) bool {
        for i := n - 1; i > 1; i-- {
            if n%i == 0 {
                return false
            }
        }
        return true
    }

    primes := make(chan int)
    go func() {
        defer close(primes)
        for {
            select {
            case <-done:
                return
            case n := <-randIntStream:
                if isPrime(n) {
                    primes <- n
                }
            }
        }
    }()
    return primes
}

// Mux (fan-in): merges multiple channels into one
func mux[T any](done <-chan int, channels ...<-chan T) <-chan T {
    var wg sync.WaitGroup
    out := make(chan T)

    transfer := func(c <-chan T) {
        defer wg.Done()
        for v := range c {
            select {
            case <-done:
                return
            case out <- v:
            }
        }
    }

    for _, c := range channels {
        wg.Add(1)
        go transfer(c)
    }

    go func() {
        wg.Wait()
        close(out)
    }()

    return out
}

func main() {
    done := make(chan int)
    defer close(done)

    // Generate random integers
    randInt := func() int {
        return rand.Intn(10000000)
    }
    generatorStream := generator(done, randInt)

    // Launch one prime finder per CPU core
    cpuCount := runtime.NumCPU()
    primeFinderChannels := make([]<-chan int, cpuCount)
    for i := 0; i < cpuCount; i++ {
        primeFinderChannels[i] = primeFinder(done, generatorStream)
    }

    // Merge all prime finders into one output
    for prime := range take(done, mux(done, primeFinderChannels...), 10) {
        fmt.Println(prime)
    }
}

Now Let’s Break It Down

1. Generator

The generator function produces values continuously by calling a function (fn()) and sending the result on a channel.

Why the done channel? It allows us to cancel the generator from the outside, which is important to avoid goroutines running forever.

generatorStream := generator(done, randInt)

This gives us a stream of random numbers.

2. Take

The take function just reads the first n values from a stream, and then stops. It's like saying, “I only want 10 results.”

take(done, stream, 10)

This is how we make sure the program ends after getting 10 primes.

3. Prime Checker

This part listens to random numbers and filters the ones that are prime.

func isPrime(n int) bool {
    // Very simple primality check
}

It's not the most optimized algorithm, but it’s perfect for this demo, especially because it’s intentionally slow, so we can see the benefit of running it in parallel.

4. Fan-Out: Run Multiple Prime Checkers

Instead of having just one goroutine checking for primes, we spin up one per CPU core. This way, each core can work independently and in parallel.

cpuCount := runtime.NumCPU()
for i := 0; i < cpuCount; i++ {
    primeFinderChannels[i] = primeFinder(done, generatorStream)
}

Now we have several independent workers filtering primes at the same time.

5. Fan-In: Merging Results

Once we have multiple channels from the prime checkers, we want to combine them into one stream. That’s what mux does it merges multiple input channels into a single output.

mux(done, primeFinderChannels...)

This gives us one unified stream of prime numbers from all workers.

6. Final Output

Now we just take the first 10 primes from the merged stream and print them.

for prime := range take(done, mux(...), 10) {
    fmt.Println(prime)
}

Once we’ve printed 10 primes, we close the done channel, and everything shuts down gracefully.
What I Learned

This small project helped me understand some important Go concurrency patterns:

Generators are a clean way to build infinite or cancellable streams.
Fan-out helps you distribute slow tasks across multiple goroutines.
Fan-in lets you merge results from those goroutines back into one place.
Channels and select statements give you a ton of control without needing locks or semaphores.
And most importantly: clean shutdown matters. The done channel ensures no goroutine is left behind.

Final Thoughts

Even a small project like this can teach you how to structure concurrent programs in an elegant and idiomatic way.

Understanding Weak Pointers in Go

Walid LAGGOUNE — Tue, 22 Apr 2025 17:07:22 +0000

Hi, my name is Walid, a backend developer who’s currently learning Go and sharing my journey by writing about it along the way.

A weak pointer is a reference to an object that does not prevent the garbage collector (GC) from reclaiming the object. If there are no strong references to the object, the GC can collect it, and any weak pointers to it will automatically become nil

Why Use Weak Pointers?

Common use cases include:

Caching: Store objects without forcing them to remain in memory if they’re no longer used elsewhere.
Observer Patterns: Hold references to observers without preventing their garbage collection.
Canonicalization: Ensure only one instance of an object exists without preventing its collection when unused
Dependency Graphs: Prevent cycles in structures like trees or graphs.

The weak Package in Go 1.24

Go 1.24 introduces the weak package, providing a straightforward API for creating and using weak pointers.

import "weak"

type MyStruct struct {
    Data string
}

func main() {
    obj := &MyStruct{Data: "example"}
    wp := weak.Make(obj) // Create a weak pointer
    val := wp.Value()    // Retrieve the strong pointer or nil
    if val != nil {
        fmt.Println(val.Data)
    } else {
        fmt.Println("Object has been garbage collected")
    }
}

In this example, weak.Make(obj) creates a weak pointer to obj. Calling wp.Value() returns the strong pointer if the object is still alive or nil if it has been collected.

Practical Example: Implementing a Cache

One practical application of weak pointers is in building caches that don't prevent their entries from being garbage collected.

import (
    "sync"
    "weak"
)

type Data struct {
    Value string
}

var cache sync.Map // map[string]weak.Pointer[*Data]

func GetData(key string) *Data {
    if wp, ok := cache.Load(key); ok {
        if data := wp.(weak.Pointer[*Data]).Value(); data != nil {
            return data
        }
    }
    // Load data from source
    data := &Data{Value: "fresh data"}
    cache.Store(key, weak.Make(data))
    return data
}

In this cache implementation, entries are stored as weak pointers. If no other strong references to the data exist, the GC can collect it, and subsequent calls to GetData will reload the data

Testing Weak Pointers

import (
    "fmt"
    "runtime"
    "weak"
)

type MyStruct struct {
    Data string
}

func main() {
    obj := &MyStruct{Data: "test"}
    wp := weak.Make(obj)
    obj = nil // Remove strong reference
    runtime.GC()
    if wp.Value() == nil {
        fmt.Println("Object has been garbage collected")
    } else {
        fmt.Println("Object is still alive")
    }
}

Understanding Struct Alignment and Padding in Go

Walid LAGGOUNE — Fri, 04 Apr 2025 08:10:43 +0000

Hi, my name is Walid, a backend developer who’s currently learning Go and sharing my journey by writing about it along the way.
Resource :

The Go Programming Language book by Alan A. A. Donovan & Brian W. Kernighan
Matt Holiday go course
https://www.catb.org/esr/structure-packing/

When writing high-performance Go programs especially those that involve multithreading, networking, or low-level system access understanding how Go places struct fields in memory can dramatically improve both speed and memory usage.

This article breaks down concepts like alignment, padding, and cache-line usage, and shows how to spot and fix hidden inefficiencies. The article is written with clarity in mind while providing insights useful for both beginners and more advanced developers.

What Is Alignment?

Each data type in Go prefers to be stored at a memory address that's a multiple of its size:

uint8 → aligned to 1 byte
uint16 → aligned to 2 bytes
uint32 → aligned to 4 bytes
uint64 → aligned to 8 bytes
float64, int64 → aligned to 8 bytes

Why? Because CPUs are much faster at reading aligned memory. Misalignment can cause performance penalties and even errors on some architectures.

Padding Table by Data Type

Type	Size (bytes)	Alignment	Padding After (If Needed)
`uint8`	1	1	Up to 7 bytes
`uint16`	2	2	Up to 6 bytes
`uint32`	4	4	Up to 4 bytes
`uint64`	8	8	0

What Is Padding and Why Does It Exist?

Padding is extra memory added by the Go compiler between struct fields to maintain proper alignment.

// Misaligned layout
type Bad struct {
    A uint32 // offset 0
    B uint64 // offset 4 (misaligned)
}

To fix this, the compiler adds 4 bytes of padding after A:

// Aligned layout
type Good struct {
    B uint64 // offset 0
    A uint32 // offset 8
}

Memory Layout Schema

Bad:   | A (4) | pad (4) | B (8) | => 16 bytes
Good:  | B (8) | A (4) + pad (4)   | => 16 bytes

Field Ordering Matters: Save Megabytes

Ordering fields by descending size can significantly reduce padding and save memory, especially when creating many struct instances.

Example

type PoorLayout struct {
    A uint8   // 1 byte
    B uint64  // 8 bytes (misaligned)
}

This layout forces 7 bytes of padding → ~7MB wasted for 1 million instances.

Better layout:

type OptimizedLayout struct {
    B uint64
    A uint8
    _ [7]byte // explicit padding if needed
}

Optimizing layout this way can save memory and reduce CPU cache pressure.

Struct Size and Alignment on 32-bit vs 64-bit Systems

System architecture affects alignment:

32-bit systems: Align to 4 bytes
64-bit systems: Align to 8 bytes

type Info struct {
    Flags    uint32
    RegUse   uint64
    RegSet   uint64
    RegIndex uint64
}

32-bit: 4 + 8 + 8 + 8 = 28 bytes
64-bit: 4 + 4 (padding) + 8 + 8 + 8 = 32 bytes

Go may also add trailing padding to ensure the struct size is a multiple of the largest field's alignment, especially for array usage.

The Empty `struct{}` Trick (and Its Gotchas)

While struct{} consumes no storage, you can still take its address. This forces Go (since 1.5) to treat it as if it were 1 byte when at the end of a struct.

type WithEmpty struct {
    A uint32
    Z struct{}
}

This layout adds 3 bytes of padding to align the full struct to a 4-byte boundary on 32-bit systems.

Schema

A (4) + Z (1) + pad (3) = 8 bytes

Move zero-sized fields like struct{} to the top of the struct to avoid unnecessary end padding.

Cache Line Considerations

Modern CPUs load memory in 64-byte cache lines. If two goroutines update fields in the same cache line, they interfere with each other this is called false sharing.

type Shared struct {
    A int64
    B int64
}

If A and B are written concurrently, they may sit in the same cache line, slowing things down due to cache invalidations.

Optimizing with Padding

type Padded struct {
    A int64
    _ [56]byte // to isolate B on another cache line
    B int64
}

This ensures each field is on a separate cache line, improving performance in concurrent contexts.

Go’s Size Classes and Allocation Efficiency

Go groups small allocations into size classes to optimize memory usage. Example:

A 40-byte struct → rounded up to a 48-byte size class
Reduce it to 32 bytes → fits in a smaller 32-byte class

Benefit

Reducing struct size from 296 bytes to 288 bytes moves it from a 320-byte class to 288 → saving 32 bytes per allocation.

In large systems, this adds up to megabytes in savings.

Summary: Practical Struct Design Guidelines

Order fields by size: Largest to smallest to reduce padding
Separate concurrent fields: Avoid false sharing using cache line awareness
Track alignment requirements: Especially for cross-architecture compatibility
Move zero-sized types to the top: Avoid trailing padding
Use benchmarks: Profile before and after changes
Keep structs compact: Memory adds up fast at scale

By understanding Go's struct layout rules and CPU memory behavior, you can write code that's both fast and memory efficient especially in high-scale and concurrent environments.

Integrating C libraries into Go (Example with libbz2)

Walid LAGGOUNE — Tue, 01 Apr 2025 06:22:54 +0000

Hi, my name is Walid, a backend developer who’s currently learning Go and sharing my journey by writing about it along the way.
Resource :

The Go Programming Language book by Alan A. A. Donovan & Brian W. Kernighan
Matt Holiday go course

In this tutorial, we will demonstrate how to integrate Go with the libbzip2 compression library using cgo. We will keep the C code in a separate file and call it from Go to compress a file.

Prerequisites

Before we begin, ensure you have the following installed:

libbzip2 Development Library:
- On Debian-based systems:
```
sudo apt install libbz2-dev
```

Project Structure

To keep things organized, we will separate our Go and C code into different files. Create the following directory structure:

project-directory/
├── main.go
├── bz2compress.c
└── bz2compress.h

Writing the C Code

1. Create `bz2compress.h`

This header file declares the functions we will implement in bz2compress.c.

#ifndef BZ2COMPRESS_H
#define BZ2COMPRESS_H

#include <bzlib.h>

int bz2compress(bz_stream *strm, int action, char *in, unsigned int *inlen, char *out, unsigned int *outlen);
bz_stream* bz2alloc(void);
void bz2free(bz_stream* strm);

#endif // BZ2COMPRESS_H

2. Create `bz2compress.c`

This file implements the functions declared in the header.

#include "bz2compress.h"
#include <stdlib.h>

int bz2compress(bz_stream *strm, int action, char *in, unsigned int *inlen, char *out, unsigned int *outlen) {
    strm->next_in = in;
    strm->avail_in = *inlen;
    strm->next_out = out;
    strm->avail_out = *outlen;
    int ret = BZ2_bzCompress(strm, action);
    *inlen -= strm->avail_in;
    *outlen -= strm->avail_out;
    strm->next_in = strm->next_out = NULL;
    return ret;
}

bz_stream* bz2alloc(void) {
    return (bz_stream*)calloc(1, sizeof(bz_stream));
}

void bz2free(bz_stream* strm) {
    free(strm);
}

Writing the Go Code

Now, let's write the Go code to use these C functions.

3. Create `main.go`

package main

/*
#cgo LDFLAGS: -lbz2
#include "bz2compress.h"
*/
import "C"
import (
    "fmt"
    "io"
    "os"
    "unsafe"
)

func main() {
    inputFile := "input.txt"
    outputFile := "output.bz2"

    inFile, err := os.Open(inputFile)
    if err != nil {
        fmt.Fprintf(os.Stderr, "Error opening input file: %v\n", err)
        os.Exit(1)
    }
    defer inFile.Close()

    outFile, err := os.Create(outputFile)
    if err != nil {
        fmt.Fprintf(os.Stderr, "Error creating output file: %v\n", err)
        os.Exit(1)
    }
    defer outFile.Close()

    const bufferSize = 64 * 1024
    inBuffer := make([]byte, bufferSize)
    outBuffer := make([]byte, bufferSize)

    strm := C.bz2alloc()
    defer C.bz2free(strm)

    if ret := C.BZ2_bzCompressInit(strm, 9, 0, 0); ret != C.BZ_OK {
        fmt.Fprintf(os.Stderr, "BZ2_bzCompressInit failed: %d\n", ret)
        os.Exit(1)
    }
    defer C.BZ2_bzCompressEnd(strm)

    for {
        inLen, err := inFile.Read(inBuffer)
        if err != nil && err != io.EOF {
            fmt.Fprintf(os.Stderr, "Error reading input file: %v\n", err)
            os.Exit(1)
        }

        if inLen == 0 {
            break
        }

        inLenC := C.uint(inLen)
        outLenC := C.uint(bufferSize)

        ret := C.bz2compress(strm, C.BZ_RUN,
            (*C.char)(unsafe.Pointer(&inBuffer[0])), &inLenC,
            (*C.char)(unsafe.Pointer(&outBuffer[0])), &outLenC)

        if ret != C.BZ_RUN_OK {
            fmt.Fprintf(os.Stderr, "BZ2_bzCompress failed: %d\n", ret)
            os.Exit(1)
        }

        if _, err := outFile.Write(outBuffer[:outLenC]); err != nil {
            fmt.Fprintf(os.Stderr, "Error writing to output file: %v\n", err)
            os.Exit(1)
        }
    }

    fmt.Println("Compression completed successfully.")
}

Building and Running the Application

Initialize the Go Module

   go mod init project-directory

Build the Project

   go build

Run the Application

   ./project-directory

Ensure input.txt exists before running the program. The compressed output will be saved as output.bz2.

Conclusion

By following this tutorial, you have successfully integrated C code with Go using cgo. Keeping the C code separate improves maintainability and allows better modularity. This approach can be extended to use other C libraries in Go applications efficiently.

If you have any questions or suggestions, feel free to leave a comment below!

Understanding Cyclomatic Complexity in Go: A Comprehensive Guide

Walid LAGGOUNE — Sun, 30 Mar 2025 01:56:49 +0000

Hi, my name is Walid, a backend developer who’s currently learning Go and sharing my journey by writing about it along the way.
Resource :

The Go Programming Language book by Alan A. A. Donovan & Brian W. Kernighan
Matt Holiday go course

In software development, writing clean, maintainable, and efficient code is paramount. One critical metric that aids in achieving this goal is cyclomatic complexity. This metric helps developers assess the complexity of their code by quantifying the number of independent paths through a program's source code. In this article, we'll delve into the concept of cyclomatic complexity, its significance in Go programming, methods to measure it, and strategies to reduce it for better code quality.

What is Cyclomatic Complexity?

Cyclomatic complexity is a software metric introduced by Thomas J. McCabe in 1976. It measures the number of linearly independent paths through a program's source code, providing an indication of the code's complexity. A higher cyclomatic complexity value suggests a more complex program, which may be harder to understand, test, and maintain.

The cyclomatic complexity of a program can be calculated using the formula:

M = E - N + 2P

Where:

( M ) = Cyclomatic complexity
( E ) = Number of edges (transitions between nodes) in the control flow graph
( N ) = Number of nodes (statements or decision points) in the control flow graph
( P ) = Number of connected components (typically 1 for a single program)

In simpler terms, cyclomatic complexity can also be determined by counting the number of decision points (such as if, for, switch, etc.) in a function and adding 1.

Significance of Cyclomatic Complexity in Go

In Go, as in other programming languages, cyclomatic complexity serves as an indicator of code quality. Functions with high cyclomatic complexity are often more prone to errors, harder to test, and challenging to maintain. By monitoring and managing this metric, Go developers can:

Enhance Code Maintainability: Simpler functions are easier to understand and modify.
Improve Testability: Lower complexity reduces the number of test cases needed for thorough coverage.
Reduce Error Rates: Less complex code is less likely to contain bugs.

Measuring Cyclomatic Complexity in Go

To measure cyclomatic complexity in Go projects, tools like gocyclo are commonly used. gocyclo calculates the cyclomatic complexities of functions in Go source code, helping identify functions that may need refactoring.

Installing gocyclo

To install gocyclo, run:

go install github.com/fzipp/gocyclo/cmd/gocyclo@latest

Using gocyclo

Navigate to your project's directory and execute:

gocyclo -over 10 ./...

This command lists all functions with a cyclomatic complexity exceeding 10, indicating potential candidates for refactoring.

Strategies to Reduce Cyclomatic Complexity in Go

Reducing cyclomatic complexity leads to more readable and maintainable code. Here are some effective strategies:

1. Refactor Large Functions into Smaller Ones

Breaking down complex functions into smaller, focused functions enhances clarity and reusability.

Before Refactoring:

func ProcessData(data []int) {
    for _, v := range data {
        if v > 0 {
            // Process positive values
        } else {
            // Process non-positive values
        }
    }
}

After Refactoring:

func ProcessData(data []int) {
    for _, v := range data {
        processValue(v)
    }
}

func processValue(v int) {
    if v > 0 {
        // Process positive values
    } else {
        // Process non-positive values
    }
}

This approach simplifies the ProcessData function and delegates specific tasks to processValue, reducing complexity.

2. Use Early Returns to Simplify Logic

Employing early returns can eliminate nested if statements, making the code more straightforward.

Before:

func ValidateInput(input string) bool {
    if len(input) > 0 {
        if isValidFormat(input) {
            return true
        }
    }
    return false
}

After:

func ValidateInput(input string) bool {
    if len(input) == 0 {
        return false
    }
    return isValidFormat(input)
}

This refactoring reduces nesting and clarifies the function's intent.

3. Replace Complex Conditionals with Polymorphism

When dealing with multiple conditions that determine behavior, consider using interfaces and polymorphism to simplify the code.

Before:

func GetDiscount(userType string) float64 {
    if userType == "regular" {
        return 0.1
    } else if userType == "premium" {
        return 0.2
    } else if userType == "vip" {
        return 0.3
    }
    return 0.0
}

After:

type User interface {
    Discount() float64
}

type RegularUser struct{}
func (r RegularUser) Discount() float64 { return 0.1 }

type PremiumUser struct{}
func (p PremiumUser) Discount() float64 { return 0.2 }

type VIPUser struct{}
func (v VIPUser) Discount() float64 { return 0.3 }

func GetDiscount(user User) float64 {
    return user.Discount()
}

This design leverages polymorphism to eliminate complex conditionals, enhancing code maintainability.

Conclusion

Managing cyclomatic complexity is crucial for writing maintainable and efficient Go code. By understanding and measuring this metric, developers can identify complex functions and apply strategies to simplify them. Utilizing tools like gocyclo aids in monitoring

Writing Software That Works with the Machine, Not Against It in GO

Walid LAGGOUNE — Fri, 28 Mar 2025 19:36:24 +0000

Hi, my name is Walid, a backend developer who’s currently learning Go and sharing my journey by writing about it along the way.
Resource :

The Go Programming Language book by Alan A. A. Donovan & Brian W. Kernighan
Matt Holiday go course

In high-performance computing, mechanical sympathy refers to writing software that aligns with the way hardware operates, ensuring that the code leverages CPU architecture efficiently rather than working against it. Go, while being a high-level language, allows developers to write performance-sensitive code when necessary.

This article dives deep into key performance principles, covering CPU architecture, memory hierarchies, data structures, false sharing, allocation strategies, and execution patterns—all with a focus on writing Go code that maximizes hardware efficiency.

Understanding CPU Cores and Memory Hierarchy

Modern CPUs are built with multiple cores, each with its own cache hierarchy:

L1 Cache (Fastest, ~1ns latency) – Closest to the CPU core but small in size (32KB–64KB per core).
L2 Cache (Fast, ~4-10ns latency) – Larger (256KB–1MB per core), shared between threads within the same core.
L3 Cache (Slower, ~10-30ns latency) – Shared across multiple cores, much larger (4MB–128MB).
RAM (Much slower, ~100ns latency) – Main system memory, much larger but orders of magnitude slower than caches.

When a program accesses data, the CPU first looks for it in L1, then L2, then L3, before going to RAM. Cache misses (when the requested data isn’t in L1/L2/L3) cause significant slowdowns.

GOAL: Structure data for locality of reference so that frequently accessed data stays in cache as long as possible.

Dynamic Dispatch: Why Too Many Short Method Calls Are Expensive

In Go, calling methods through an interface involves dynamic dispatch, which introduces an extra layer of indirection:

The vtable (virtual method table) lookup adds an overhead.
Indirect calls prevent the CPU from prefetching the next instruction, leading to branch mispredictions.
The function call may jump to a memory location outside the CPU cache, causing cache misses.

Example: Indirect vs. Direct Method Calls

type Shape interface {
    Area() float64
}

type Circle struct {
    radius float64
}

func (c Circle) Area() float64 {
    return 3.14 * c.radius * c.radius
}

func main() {
    c := Circle{radius: 10}
    var s Shape = c  // Dynamic dispatch (interface method call)

    _ = s.Area() // Indirect call via interface (slower)
    _ = c.Area() // Direct call (faster)
}

💡 Optimizing for Mechanical Sympathy:

Avoid interfaces in hot loops if possible.
Use concrete types when performance matters.
Inlining small functions can reduce function call overhead.

Struct Layout: Contiguous Fields vs. Pointers

A struct with contiguous fields performs better than a struct containing multiple pointers due to cache locality.

Bad Example: Struct with Pointers (Fragmented in Memory)

type Node struct {
    data int
    next *Node  // Pointer to another memory location
}

Each Node might be scattered in memory, causing cache misses when traversing a linked list.

Good Example: Struct with Contiguous Fields (Better Cache Usage)

type Employee struct {
    id    int
    age   int
    score int
}

Since fields are stored together in memory, accessing them stays within the CPU cache, leading to fewer cache misses and faster execution.

Best Practices:

Store data contiguously in memory.
Minimize pointers in frequently accessed structs.
Use arrays or slices over linked lists when possible.

Why Slices Beat Linked Lists

A slice (backed by an array) is better than a pointer-based linked list because of cache efficiency:

Slices store elements sequentially in memory → better cache utilization.
Linked lists scatter nodes across memory → cache misses on every node access.

Example: Iterating Over a Slice vs. Linked List

numbers := []int{1, 2, 3, 4, 5} // Contiguous in memory (cache-friendly)

for _, n := range numbers {
    fmt.Println(n)
}

Takeaway: Use slices (or arrays) when performance matters, avoid linked lists unless there's a strong reason (e.g., frequent insertions/removals).

False Sharing: When Cores Fight Over a Cache Line

False sharing occurs when multiple CPU cores modify different variables that happen to share the same cache line, causing unnecessary cache invalidations.

Example: False Sharing Issue

var data [2]int64 // Both integers may share the same cache line

func worker(i int, wg *sync.WaitGroup) {
    for j := 0; j < 1000000; j++ {
        data[i]++  // Two goroutines modifying different array indices
    }
    wg.Done()
}

Fix: Use padding or align variables on separate cache lines to prevent contention.

type PaddedData struct {
    val int64
    _   [56]byte // Padding to avoid false sharing (64-byte cache line)
}

Optimizing Memory: Reducing Allocations and Embedded Pointers

1. Reduce Unnecessary Allocations

Avoid creating short-lived objects frequently—use sync.Pool when possible.
Prefer value types over heap-allocated types for small objects.

2. Reduce Embedded Pointers in Objects

Every pointer access incurs an extra memory lookup, which can slow things down.
If a struct is small, pass it by value instead of by pointer.

Bad (Heap Allocation, Extra Indirection):

type User struct {
    name *string // Pointer requires an extra memory fetch
}

Better (Stored Inline, More Efficient):

type User struct {
    name string // Directly stored, no pointer overhead
}

Heap Considerations: You Want a Larger Heap

If your program frequently allocates memory, ensure that the heap is large enough to reduce GC (Garbage Collection) overhead.

Use GOGC=off in extreme cases (e.g., low-latency systems).
Avoid excessive short-lived allocations—use object pools instead.

Execution Efficiency: Do Less, Do Often, Do It Faster

1️Do Less → Optimize algorithms to reduce redundant computations.

2️Do Often → Keep workloads balanced across CPU cores (e.g., use worker pools).

3️Do It Faster → Minimize memory latency, leverage CPU caches, and use efficient data structures.

Final Thoughts: Writing Go Code with Mechanical Sympathy

By understanding the CPU, memory hierarchy, and performance trade-offs, Go developers can write code that runs efficiently with the machine rather than against it.

Key Takeaways:

Keep data contiguous → fewer cache misses.
Avoid excessive dynamic dispatch → prefer direct calls.
Reduce pointer indirection → avoid unnecessary heap allocations.
Prevent false sharing → align data correctly.
Use slices over linked lists → leverage cache-friendly structures.
Optimize heap allocations → use sync.Pool and minimize GC pressure.

By applying these principles, your Go applications will not only run faster but will also make optimal use of modern CPU architectures—embodying true mechanical sympathy.

Understanding Memory Synchronization in Go: Why Even Read Operations Require Mutexes

Walid LAGGOUNE — Wed, 26 Mar 2025 01:16:22 +0000

Hi, my name is Walid, a backend developer who’s currently learning Go and sharing my journey by writing about it along the way.
Resource :

The Go Programming Language book by Alan A. A. Donovan & Brian W. Kernighan
Matt Holiday go course

In concurrent programming with Go, developers often focus on synchronizing write operations to prevent race conditions. However, it's equally critical to synchronize read operations. This article delves into the nuances of memory synchronization, illustrating why even simple read operations, such as reading a Balance() function, necessitate the use of mutexes.

The Issue: Beyond Mere Execution Order

It's a common misconception that read operations are inherently safe in concurrent environments:

"Since Balance() is just reading, it should always see the latest value, right?"
"As long as goroutines don’t interrupt each other, everything should be fine."

However, concurrency challenges extend beyond the interleaving of execution sequences. Modern CPUs and compilers may reorder operations to optimize performance, leading to unexpected behaviors.

Example: Out-of-Order Execution

Consider the following scenario with two goroutines:

var x, y int

go func() {
    x = 1  // A1
    fmt.Print("y:", y, " ") // A2
}()

go func() {
    y = 1  // B1
    fmt.Print("x:", x, " ") // B2
}()

Intuitively, one might expect the outputs to be among:

y:0 x:1
x:0 y:1
x:1 y:1
y:1 x:1

Surprisingly, the program might also produce:

x:0 y:0
y:0 x:0

Why does this happen?

CPU Caches and Instruction Reordering

Modern processors employ various techniques to enhance performance:

Local Caches: Each CPU core has its own cache, storing copies of frequently accessed memory locations.
Write Buffers: Writes to memory can be buffered, meaning they might not be immediately visible to other cores.
Instruction Reordering: To optimize execution, the CPU may reorder instructions, provided the reordering doesn't affect the outcome within a single thread.

Without proper synchronization, one goroutine may not observe the latest writes made by another due to these optimizations.

Illustrative Scenario:

Goroutine A executes x = 1. This write is stored in CPU 1's cache but isn't immediately visible to other CPUs.
Goroutine B executes y = 1. Similarly, this write resides in CPU 2's cache.
When Goroutine A reads y, it accesses the main memory or its cache, which still holds y = 0, since CPU 1 hasn't seen CPU 2's update.
Conversely, Goroutine B reads x and sees x = 0 for the same reasons.

This lack of visibility results in both goroutines printing stale values.

Ensuring Proper Synchronization

To address these issues, synchronization primitives like mutexes (sync.Mutex) enforce memory visibility and operation ordering.

Using sync.Mutex for Memory Synchronization:

var mu sync.Mutex
var x, y int

func safeA() {
    mu.Lock()
    x = 1
    fmt.Print("y:", y, " ")
    mu.Unlock()
}

func safeB() {
    mu.Lock()
    y = 1
    fmt.Print("x:", x, " ")
    mu.Unlock()
}

By locking the mutex before accessing shared variables and unlocking it afterward, we ensure:

Exclusive Access: Only one goroutine can execute the critical section at a time.
Memory Visibility: Changes made by one goroutine become visible to others, as the mutex operations synchronize the caches.

This approach guarantees that read operations observe the most recent writes, preventing anomalies like reading stale data.

Best Practices for Synchronizing Shared Data

Single Goroutine Access: If a variable is accessed by only one goroutine, synchronization isn't necessary.
Multiple Readers: When multiple goroutines only read a variable, consider using sync.RWMutex to allow concurrent reads while still protecting against concurrent writes.
Mixed Reads and Writes: If goroutines perform both reads and writes on shared data, use sync.Mutex to ensure mutual exclusion.

Key Takeaway: Concurrency challenges aren't solely about the timing of operations but also about memory visibility. Without proper synchronization, goroutines may read outdated or inconsistent data due to CPU caching and instruction reordering. Employing synchronization primitives like mutexes is essential to maintain data integrity in concurrent Go programs.

Serial Confinement in go

Walid LAGGOUNE — Wed, 26 Mar 2025 00:43:37 +0000

Hi, my name is Walid, a backend developer who’s currently learning Go and sharing my journey by writing about it along the way.
Resources :

The Go Programming Language book by Alan A. A. Donovan & Brian W. Kernighan
Matt Holiday go course

In concurrent programming, managing access to shared data is crucial to prevent race conditions and ensure the correctness of a program. One effective strategy to achieve this is serial confinement, a pattern that restricts data access to a single thread or goroutine at any given time. This article delves into the concept of serial confinement, its implementation, benefits, and practical applications, particularly in the Go programming language.

Understanding Serial Confinement

Serial confinement is a concurrency pattern where an object or data structure is exclusively owned and manipulated by a single thread or goroutine at any point in time. This exclusivity ensures that no other concurrent process can access or modify the data simultaneously, thereby eliminating the need for explicit synchronization mechanisms like mutexes. The core idea is to transfer the ownership of the data from one thread to another in a controlled and sequential manner.

In the context of Go, serial confinement can be effectively achieved using channels. By passing data through channels, ownership is transferred between goroutines, ensuring that only one goroutine has access to the data at any given moment. This approach aligns with Go's philosophy of sharing memory by communicating, rather than communicating by sharing memory.

Implementing Serial Confinement in Go

To implement serial confinement in Go, follow these steps:

Initialize a Channel: Create a channel to facilitate the transfer of data between goroutines.
Pass Data Through the Channel: Send the data from one goroutine to another via the channel, ensuring that only the receiving goroutine has access to the data after the transfer.
Avoid Access After Transfer: The sending goroutine should refrain from accessing the data after it has been sent through the channel, maintaining the integrity of the confinement.

Here's a concrete example illustrating serial confinement in Go:

package main

import (
    "fmt"
    "sync"
)

// Task represents a unit of work
type Task struct {
    ID int
}

// worker processes tasks received through the tasks channel
func worker(tasks <-chan Task, wg *sync.WaitGroup) {
    defer wg.Done()
    for task := range tasks {
        fmt.Printf("Processing task with ID: %d\n", task.ID)
        // Perform task processing here
    }
}

func main() {
    const numWorkers = 3
    tasks := make(chan Task, 10)
    var wg sync.WaitGroup

    // Start worker goroutines
    for i := 0; i < numWorkers; i++ {
        wg.Add(1)
        go worker(tasks, &wg)
    }

    // Send tasks to workers
    for i := 0; i < 5; i++ {
        tasks <- Task{ID: i}
    }
    close(tasks)

    // Wait for all workers to complete
    wg.Wait()
}

In this example:

A buffered channel tasks is created to hold Task instances.
Multiple worker goroutines are launched, each receiving tasks from the tasks channel.
Each task is processed by only one worker, ensuring that the Task instance is confined to a single goroutine during its processing.
The sync.WaitGroup ensures that the main function waits for all worker goroutines to complete before exiting.

By adhering to this pattern, each Task instance is accessed by only one goroutine at a time, maintaining thread safety without explicit locks.

Another example :

package main

import (
    "fmt"
    "sync"
)

// Cake represents a cake in the production line
type Cake struct {
    state string
}

// baker bakes cakes and sends them to the cooked channel
func baker(cooked chan<- *Cake, wg *sync.WaitGroup) {
    defer wg.Done()
    for i := 0; i < 5; i++ { // Producing 5 cakes
        cake := &Cake{state: "cooked"}
        fmt.Println("Baked:", cake.state)
        cooked <- cake
    }
    close(cooked)
}

// icer receives cooked cakes, applies icing, and sends them to the iced channel
func icer(iced chan<- *Cake, cooked <-chan *Cake, wg *sync.WaitGroup) {
    defer wg.Done()
    for cake := range cooked {
        cake.state = "iced"
        fmt.Println("Iced:", cake.state)
        iced <- cake
    }
    close(iced)
}

// packager receives iced cakes, packages them, and sends them to the packaged channel
func packager(packaged chan<- *Cake, iced <-chan *Cake, wg *sync.WaitGroup) {
    defer wg.Done()
    for cake := range iced {
        cake.state = "packaged"
        fmt.Println("Packaged:", cake.state)
        packaged <- cake
    }
}

func main() {
    cooked := make(chan *Cake)
    iced := make(chan *Cake)
    packaged := make(chan *Cake)

    var wg sync.WaitGroup

    wg.Add(3)

    go baker(cooked, &wg)
    go icer(iced, cooked, &wg)
    go packager(packaged, iced, &wg)

    go func() {
        wg.Wait()
        close(packaged)
    }()

    for cake := range packaged {
        fmt.Println("Final product:", cake.state)
    }
}

Explanation:

Cake Struct: Defines the Cake type with a state field to represent its current status in the production line.
Baker Function: Simulates the baking process by creating cakes, setting their state to "cooked", and sending them to the cooked channel.
Icer Function: Receives cooked cakes from the cooked channel, changes their state to "iced", and forwards them to the iced channel.
Packager Function: Takes iced cakes from the iced channel, updates their state to "packaged", and sends them to the packaged channel.
Main Function: Sets up the channels and goroutines for each stage, initiates the process, and collects the final products from the packaged channel

Benefits of Serial Confinement

Implementing serial confinement offers several advantages:

Simplified Synchronization: By ensuring exclusive access to data, the need for complex synchronization primitives is reduced or eliminated.
Enhanced Performance: Avoiding locks can lead to more efficient execution, as the overhead associated with locking mechanisms is bypassed.
Improved Maintainability: Code that follows the serial confinement pattern is often easier to understand and maintain, as the flow of data ownership is clear and predictable.

Practical Applications

Serial confinement is particularly useful in scenarios such as:

Pipeline Processing: In data processing pipelines, each stage can own and process the data before passing it to the next stage, ensuring that each piece of data is handled by only one goroutine at a time.
Task Queues: Distributing tasks among worker goroutines where each task is processed independently by a single worker, as demonstrated in the example above.
Resource Management: Managing resources that are not safe for concurrent access by confining their usage to a single goroutine, thereby preventing race conditions and ensuring safe operations.

Conclusion

Serial confinement is a powerful concurrency pattern that leverages the principle of exclusive data ownership to ensure thread safety and simplify concurrent programming. By structuring your Go programs to transfer data ownership between goroutines using channels, you can achieve safe and efficient concurrent operations without the complexities of traditional synchronization mechanisms. Embracing this pattern can lead to more robust and maintainable concurrent applications.

Concurrency gotchas in go

Walid LAGGOUNE — Mon, 24 Mar 2025 19:44:22 +0000

Hi, my name is Walid, a backend developer who’s currently learning Go and sharing my journey by writing about it along the way.
Resource :

The Go Programming Language book by Alan A. A. Donovan & Brian W. Kernighan
Matt Holiday go course Concurrency is a cornerstone of Go's design, enabling developers to build efficient, high-performance applications. However, leveraging concurrency effectively requires a deep understanding of potential pitfalls. This article delves into common concurrency challenges in Go and offers strategies to address them.

1. Race Conditions

Race conditions occur when multiple goroutines access and modify shared data concurrently without proper synchronization, leading to unpredictable behavior. For example, incrementing a shared counter without synchronization can result in incorrect values.

Solution: Utilize synchronization primitives like sync.Mutex to ensure exclusive access to shared resources.

var (
    counter int
    mu      sync.Mutex
)

func increment() {
    mu.Lock()
    counter++
    mu.Unlock()
}

Additionally, Go's race detector is a powerful tool for identifying race conditions early in the development process. Activate it using the -race flag:

go run -race main.go
go test -race ./...

2. Goroutine Leaks

Goroutine leaks occur when goroutines remain blocked indefinitely, consuming resources and potentially leading to memory exhaustion. This often happens when goroutines are waiting on channels that are never closed or when exit conditions are not properly managed.

Solution: Ensure that all goroutines have a clear exit strategy. Use context cancellation to signal goroutines to terminate gracefully.

ctx, cancel := context.WithCancel(context.Background())
defer cancel()

go func() {
    select {
    case <-ctx.Done():
        // Cleanup and exit
    }
}()

Regularly monitoring and profiling your application can help detect unexpected goroutine growth.

3. Deadlocks

Deadlocks occur when two or more goroutines are waiting indefinitely for each other to release resources, causing the program to halt. For instance, if two goroutines each hold a lock and attempt to acquire the other's lock, a deadlock ensues.

Solution: Adopt a consistent locking order to prevent circular dependencies. Alternatively, consider using channels for synchronization, adhering to Go's philosophy of communicating by sharing memory.

ch := make(chan struct{})

go func() {
    // Perform operations
    ch <- struct{}{}
}()

<-ch // Wait for goroutine to finish

This approach reduces the risk of deadlocks by avoiding explicit locks.

4. Starvation and Livelocks

Starvation occurs when a goroutine is perpetually denied the resources it needs to proceed, often due to higher-priority goroutines monopolizing those resources. Livelocks are similar, where goroutines continuously change state in response to each other without making progress.

Solution: Design your concurrency control mechanisms to ensure fair access to resources. Avoid excessive locking durations and consider using buffered channels to manage IO resource availability.

sem := make(chan struct{}, N) // N is the maximum number of concurrent goroutines

for i := 0; i < totalTasks; i++ {
    sem <- struct{}{} // works like token ring , get a token and release it after
    go func(task int) {
        defer func() { <-sem }()
        // Perform task
    }(i)
}

This pattern, known as a semaphore, limits the number of concurrent goroutines, preventing resource starvation.

A counting semaphore is a type of lock that allows you to limit the number of processes that can concurrently access a resource to some fixed number. You can think of the lock that we just created as being a counting semaphore with a limit of 1.

5. Improper Use of Channels

Channels are powerful tools for communication between goroutines, but misuse can lead to issues such as deadlocks, data races, and goroutine leaks. Common mistakes include:

Unbuffered Channels: Requiring both sender and receiver to be ready can cause deadlocks if not managed carefully.
Buffered Channels: While they can decouple sender and receiver, improper sizing or assumptions about buffer capacity can hide race conditions or lead to unexpected behavior.
Channel Closure: Closing a channel from multiple goroutines or sending on a closed channel can cause panics.

Solution: Adhere to best practices when working with channels:

Clearly define channel ownership and responsibility for closing.
Use unbuffered channels for synchronization and buffered channels for batching or managing bursts of data.
Always check for the closed state when ranging over channels.

ch := make(chan int)
go func() {
    defer close(ch)
    for i := 0; i < 5; i++ {
        ch <- i
    }
}()

for val := range ch {
    fmt.Println(val)
}

This pattern ensures that the receiver terminates gracefully when the channel is closed.

6. Shared Data Structures

Modifying shared data structures like maps without proper synchronization can lead to data corruption or panics. In Go, maps are not safe for concurrent use by default.

Solution: Use synchronization mechanisms such as sync.Mutex or sync.RWMutex to protect access to shared data structures.

var (
    m  = make(map[string]int)
    mu sync.Mutex
)

func set(key string, value int) {
    mu.Lock()
    m[key] = value
    mu.Unlock()
}

func get(key string) int {
    mu.Lock()
    defer mu.Unlock()
    return m[key]
}

Alternatively, consider using sync.Map, which is designed for concurrent use cases.

In Go, managing concurrent access to shared resources can be achieved using synchronization primitives like mutexes and atomic operations. Each has distinct characteristics and use cases, and understanding their differences is crucial for writing efficient concurrent programs.

Atomic Operations

The sync/atomic package provides low-level atomic memory primitives that allow for lock-free synchronization on single variables. These operations are generally faster than mutexes because they translate directly to single CPU instructions, ensuring atomicity without the overhead of locking mechanisms. However, atomic operations are limited to specific types, such as integers and pointers, and are best suited for simple read-modify-write scenarios.

Mutexes

A sync.Mutex is a mutual exclusion lock that provides a way to serialize access to shared resources. Unlike atomic operations, mutexes can protect complex data structures and multiple variables, offering more flexibility. However, they introduce additional overhead due to locking and unlocking operations, which can impact performance, especially in high-contention scenarios.

Choosing Between Atomics and Mutexes

Use Atomics When:
- Performing simple operations on a single variable.
- Minimizing synchronization overhead is critical.
Use Mutexes When:
- Protecting complex data structures or multiple variables.
- Requiring more sophisticated synchronization patterns.

It's important to note that while atomics offer performance benefits, they require a solid understanding of memory models and can be error-prone if not used carefully. Mutexes, being more straightforward, are often preferred for their clarity and ease of use in complex scenarios.

In summary, both atomics and mutexes are essential tools in Go's concurrency model. Selecting the appropriate synchronization mechanism depends on the specific requirements of your application and the complexity of the data being managed.

7. Overusing Goroutines

While goroutines are lightweight, spawning an excessive number can lead to high memory consumption and increased scheduling overhead. For example, creating a new goroutine for each incoming request without limitation can overwhelm the system.

Solution: Implement worker pools or use bounded concurrency patterns to control the number of active goroutines.

sem := make(chan struct{}, maxGoroutines)
var wg sync.WaitGroup

for _, task := range tasks {
    sem <- struct{}{} // Acquire a token before launching a goroutine
    wg.Add(1)
    go func(t Task) {
        defer wg.Done()
        defer func() { <-sem }() // Release the token after the goroutine completes

        // Perform the task
        t.Execute()
    }(task)
}

wg.Wait() // Wait for all goroutines to complete

Concurrency and File Hashing in Go: Lessons and Best Practices

Walid LAGGOUNE — Sun, 23 Mar 2025 16:22:29 +0000

Hi, my name is Walid, a backend developer who’s currently learning Go and sharing my journey by writing about it along the way.
Resource :

-The Go Programming Language book by Alan A. A. Donovan & Brian W. Kern ighan

Matt Holiday go course ## Introduction

Concurrency is one of the standout features of Go, allowing developers to write highly efficient, parallelized programs with minimal complexity. In this article, we explore key lessons from a practical Go program that scans directories, hashes files using MD5, and identifies duplicate files. We'll discuss Go’s concurrency model, channels, synchronization, and best practices for managing goroutines efficiently. At the end, we provide a full example demonstrating these concepts in action.

Go’s Concurrency Model

Go’s concurrency model revolves around goroutines and channels. A goroutine is a lightweight thread managed by the Go runtime, and channels facilitate safe communication between goroutines.

Channels as Messaging and Synchronization Tools

Channels in Go serve two primary purposes:

Messaging Tool: They allow goroutines to communicate by passing values safely between them.
Synchronization Tool: Channels can be used to control execution flow and coordinate goroutines, ensuring proper sequencing and avoiding race conditions.

Unidirectional Channels

By default, channels allow bidirectional communication, but Go also supports unidirectional channels, which can only send or receive data:

func sendData(ch chan<- int) { // Only send
    ch <- 42
}

func receiveData(ch <-chan int) { // Only receive
    fmt.Println(<-ch)
}

Unidirectional channels help enforce safe communication patterns and make the code easier to reason about.

Deadlocks in Go

A deadlock occurs when all goroutines are waiting on a channel operation that will never happen, effectively causing the program to freeze. For example:

ch := make(chan int)
ch <- 1 // Deadlock: no goroutine is receiving

Go detects deadlocks at runtime and will panic if no goroutines are executing.

Unbuffered and Buffered Channels

Unbuffered Channels (Rendezvous Model)

An unbuffered channel requires a sender and a receiver to be ready at the same time, making it a synchronization mechanism. This ensures data is passed immediately without any intermediate storage.

ch := make(chan int) // Unbuffered

go func() {
    ch <- 42 // This will block until a receiver is ready
}()
fmt.Println(<-ch) // Receives immediately

Buffered Channels

A buffered channel allows sending multiple values without requiring an immediate receiver. This can improve performance but also introduce subtle race conditions if not managed carefully.

ch := make(chan int, 3) // Buffered with capacity 3
ch <- 1
ch <- 2
ch <- 3
// Sending a 4th value without a receiver will block

Buffered channels can hide race conditions by delaying the need for a receiver. If the buffer is large enough, senders may not block immediately, leading to unpredictable behavior in concurrent programs.

Counting Semaphore with Buffered Channels

A counting semaphore is a concurrency control pattern used to limit the number of simultaneous operations. A buffered channel can act as a semaphore to control resource usage efficiently.

sem := make(chan struct{}, 5) // Limit to 5 concurrent operations
for i := 0; i < 10; i++ {
    sem <- struct{}{} // Acquire a slot
    go func(i int) {
        defer func() { <-sem }() // Release slot after completion
        fmt.Println("Processing", i)
    }(i)
}

Managing Goroutines When Performing I/O

When performing I/O operations such as file reads/writes or network requests, we must limit the number of goroutines to prevent overwhelming system resources. This is achieved using semaphores or worker pools.

Example:

workers := 4 * runtime.GOMAXPROCS(0) // Limit based on CPU cores
limits := make(chan bool, workers)

for _, file := range fileList {
    limits <- true // Acquire slot
    go func(file string) {
        defer func() { <-limits }() // Release slot after completion
        processFile(file)
    }(file)
}

This prevents excessive goroutines from causing resource exhaustion.

Full Example: File Hashing with Concurrency in Go

Below is the complete example demonstrating efficient file processing using concurrency:

package main

import (
    "crypto/md5"
    "fmt"
    "io"
    "log"
    "os"
    "path/filepath"
    "runtime"
    "sync"
)

type pair struct {
    hash string
    path string
}

type fileList []string
type results map[string]fileList

func hashFile(path string) pair {
    file, err := os.Open(path)
    if err != nil {
        log.Fatal(err)
    }
    defer file.Close()

    hash := md5.New()
    if _, err := io.Copy(hash, file); err != nil {
        log.Fatal(err)
    }

    return pair{fmt.Sprintf("%x", hash.Sum(nil)), path}
}

func processFile(path string, pairs chan<- pair, wg *sync.WaitGroup, limits chan bool) {
    defer wg.Done()
    limits <- true
    defer func() { <-limits }()
    pairs <- hashFile(path)
}

func collectHashes(pairs <-chan pair, result chan<- results) {
    hashes := make(results)
    for p := range pairs {
        hashes[p.hash] = append(hashes[p.hash], p.path)
    }
    result <- hashes
}

func searchTree(dir string, pairs chan<- pair, wg *sync.WaitGroup, limits chan bool) error {
    defer wg.Done()
    visit := func(p string, fi os.FileInfo, err error) error {
        if err != nil && err != os.ErrNotExist {
            return err
        }
        if fi.Mode().IsDir() && p != dir {
            wg.Add(1)
            go searchTree(p, pairs, wg, limits)
            return filepath.SkipDir
        }
        if fi.Mode().IsRegular() && fi.Size() > 0 {
            wg.Add(1)
            go processFile(p, pairs, wg, limits)
        }
        return nil
    }
    limits <- true
    defer func() { <-limits }()
    return filepath.Walk(dir, visit)
}

func run(dir string) results {
    workers := 4 * runtime.GOMAXPROCS(0)
    limits := make(chan bool, workers)
    pairs := make(chan pair)
    result := make(chan results)
    wg := new(sync.WaitGroup)
    go collectHashes(pairs, result)
    wg.Add(1)
    err := searchTree(dir, pairs, wg, limits)
    if err != nil {
        log.Fatal(err)
    }
    wg.Wait()
    close(pairs)
    return <-result
}

func main() {
    if len(os.Args) < 2 {
        log.Fatal("Missing parameter, provide dir name!")
    }
    if hashes := run(os.Args[1]); hashes != nil {
        for hash, files := range hashes {
            if len(files) > 1 {
                fmt.Println(hash[len(hash)-7:], len(files))
                for _, file := range files {
                    fmt.Println("  ", file)
                }
            }
        }
    }
}

Conclusion

Go’s concurrency model, when used properly, allows highly efficient parallel execution. Understanding channels, deadlocks, synchronization, and goroutine management is key to writing performant and safe Go programs.

Composite Types in Go: Arrays, Slices, Maps, Structs, JSON, and Templates

Walid LAGGOUNE — Sat, 22 Mar 2025 20:57:48 +0000

Hi, my name is Walid, a backend developer who’s currently learning Go and sharing my journey by writing about it along the way.
Resource :

The Go Programming Language book by Alan A. A. Donovan & Brian W. Kernighan
Matt Holiday go course

Composite Types in Go: Arrays, Slices, Maps, Structs, JSON, and Templates

Go (Golang) provides several powerful composite types, including arrays, slices, maps, and structs, which allow efficient data handling and organization. In this article, we will explore these types in depth, along with JSON handling and HTML templates, culminating in a practical example that fetches GitHub issues and displays them using an HTML template.

Arrays in Go

An array is a fixed-size sequence of elements of the same type.

var arr [5]int // An array of 5 integers
arr[0] = 10 // Assigning a value
fmt.Println(arr) // Output: [10 0 0 0 0]

Arrays have fixed sizes, making them inflexible for dynamic data handling.

Iterating Over an Array

for i, v := range arr {
    fmt.Println("Index:", i, "Value:", v)
}

Slices in Go

Slices are dynamically sized, flexible views of arrays.

s := []int{1, 2, 3, 4, 5} // Slice initialization
fmt.Println(s[1:4]) // Output: [2 3 4]

Slices have length and capacity, and can be extended efficiently.

The `append` Function

The append function dynamically resizes slices.

s = append(s, 6, 7, 8)
fmt.Println(s) // Output: [1 2 3 4 5 6 7 8]

Maps in Go

Maps provide a key-value data structure.

m := make(map[string]int)
m["apple"] = 3
m["banana"] = 5
fmt.Println(m["apple"]) // Output: 3

Checking Key Existence

value, exists := m["orange"]
if exists {
    fmt.Println("Value:", value)
} else {
    fmt.Println("Key not found")
}

Structs in Go

A struct is a composite type that groups multiple fields together.

type Person struct {
    Name string
    Age  int
}
p := Person{Name: "Alice", Age: 30}
fmt.Println(p)

Struct Literals

p := Person{"Bob", 25} // Shorthand initialization

Comparing Structs

Structs can be compared if all fields are comparable.

type Point struct {
    X, Y int
}
p1 := Point{1, 2}
p2 := Point{1, 2}
fmt.Println(p1 == p2) // Output: true

Struct Embedding and Anonymous Fields

Go supports embedding structs for composition.

type Address struct {
    City, Country string
}
type Employee struct {
    Name    string
    Address // Embedded struct (anonymous field)
}
e := Employee{Name: "John", Address: Address{"New York", "USA"}}
fmt.Println(e.City) // Accessing embedded field directly

JSON Handling in Go

The encoding/json package provides JSON serialization and deserialization.

import (
    "encoding/json"
    "fmt"
)
type User struct {
    Name  string `json:"name"`
    Email string `json:"email"`
}
user := User{"Alice", "alice@example.com"}
jsonData, _ := json.Marshal(user)
fmt.Println(string(jsonData)) // Output: {"name":"Alice","email":"alice@example.com"}

Text and HTML Templates in Go

Go’s html/template package provides safe HTML rendering.

import (
    "html/template"
    "os"
)
tmpl := template.Must(template.New("test").Parse("<h1>Hello, {{.}}</h1>"))
tmpl.Execute(os.Stdout, "World")

Fetching GitHub Issues and Displaying in an HTML Template

This example fetches issues from a GitHub repository and displays them in an HTML template.

package main

import (
    "encoding/json"
    "fmt"
    "html/template"
    "net/http"
)

type Issue struct {
    Title string `json:"title"`
    State string `json:"state"`
}

type Issues struct {
    Items []Issue `json:"items"`
}

func fetchGitHubIssues(repo string) ([]Issue, error) {
    url := fmt.Sprintf("https://api.github.com/search/issues?q=repo:%s", repo)
    resp, err := http.Get(url)
    if err != nil {
        return nil, err
    }
    defer resp.Body.Close()

    var issues Issues
    if err := json.NewDecoder(resp.Body).Decode(&issues); err != nil {
        return nil, err
    }
    return issues.Items, nil
}

func handler(w http.ResponseWriter, r *http.Request) {
    repo := "golang/go"
    issues, err := fetchGitHubIssues(repo)
    if err != nil {
        http.Error(w, "Failed to load issues", http.StatusInternalServerError)
        return
    }

    tmpl := template.Must(template.New("issues").Parse(`
    <html>
    <head><title>GitHub Issues</title></head>
    <body>
        <h1>Issues for {{.Repo}}</h1>
        <ul>
            {{range .Issues}}
                <li>{{.Title}} - {{.State}}</li>
            {{end}}
        </ul>
    </body>
    </html>`))

    data := struct {
        Repo   string
        Issues []Issue
    }{repo, issues}

    tmpl.Execute(w, data)
}

func main() {
    http.HandleFunc("/", handler)
    fmt.Println("Server started at http://localhost:8080")
    http.ListenAndServe(":8080", nil)
}

Explanation:

Fetches GitHub issues using http.Get.
Decodes JSON into a Go struct.
Uses html/template to render issues dynamically.
Runs a web server to serve the data.

Conclusion

Go’s composite types (arrays, slices, maps, and structs) allow efficient data organization. With JSON and templates, Go excels in building web applications. The example provided demonstrates Go’s simplicity and power for fetching and rendering external data dynamically.

DEV Community: Walid LAGGOUNE

The Hidden Truth About TCP Reliability: Why Your Data Might Never Arrive

The False Promise of write() Success

The close() Catastrophe

The SO_LINGER Red Herring

Understanding the Root Cause

Scenario 1: Unread Data Triggers RST

Scenario 2: Buffered Data Discarded

Scenario 3: In Flight Packets Lost

The Right Way: shutdown() and Graceful Closure

Linux Specific Solution: SIOCOUTQ

The Gold Standard: Application Level Acknowledgments

Common Misconceptions Debunked

Myth 1: "TCP is always reliable"

Myth 2: "Successful write() means data was delivered"

Myth 3: "SO_LINGER solves all problems"

Myth 4: "Non-blocking sockets fix this"

Real-World Implications

Best Practices for Reliable Data Delivery

Code Example: Robust TCP Client

Conclusion

Understanding Go Concurrency Pipelines

The Big Idea

The Full Code

1. Generator

2. Take

3. Prime Checker

4. Fan-Out: Run Multiple Prime Checkers

5. Fan-In: Merging Results

6. Final Output

Final Thoughts

Understanding Weak Pointers in Go

Why Use Weak Pointers?

The weak Package in Go 1.24

Practical Example: Implementing a Cache

Testing Weak Pointers

Understanding Struct Alignment and Padding in Go

What Is Alignment?

Padding Table by Data Type

What Is Padding and Why Does It Exist?

Memory Layout Schema

Field Ordering Matters: Save Megabytes

Example

Struct Size and Alignment on 32-bit vs 64-bit Systems

The Empty struct{} Trick (and Its Gotchas)

Schema

Cache Line Considerations

Optimizing with Padding

Go’s Size Classes and Allocation Efficiency

Benefit

Summary: Practical Struct Design Guidelines

Integrating C libraries into Go (Example with libbz2)

Prerequisites

Project Structure

Writing the C Code

1. Create bz2compress.h

2. Create bz2compress.c

Writing the Go Code

3. Create main.go

Building and Running the Application

Conclusion

Understanding Cyclomatic Complexity in Go: A Comprehensive Guide

What is Cyclomatic Complexity?

Significance of Cyclomatic Complexity in Go

Measuring Cyclomatic Complexity in Go

Installing gocyclo

Using gocyclo

Strategies to Reduce Cyclomatic Complexity in Go

1. Refactor Large Functions into Smaller Ones

2. Use Early Returns to Simplify Logic

3. Replace Complex Conditionals with Polymorphism

Conclusion

Writing Software That Works with the Machine, Not Against It in GO

Understanding CPU Cores and Memory Hierarchy

Dynamic Dispatch: Why Too Many Short Method Calls Are Expensive

Example: Indirect vs. Direct Method Calls

Struct Layout: Contiguous Fields vs. Pointers

Bad Example: Struct with Pointers (Fragmented in Memory)

Good Example: Struct with Contiguous Fields (Better Cache Usage)

Why Slices Beat Linked Lists

The Empty `struct{}` Trick (and Its Gotchas)

1. Create `bz2compress.h`

2. Create `bz2compress.c`

3. Create `main.go`

The `append` Function