DEV Community: Kostas Kalafatis

Using Semaphores for Producer-Consumer Problems

Kostas Kalafatis — Mon, 25 Nov 2024 14:52:52 +0000

Introduction

The Producer-Consumer problem is a classic set of scenarios in concurrent programming, first described by Edsger W. Dijkstra in 1965. The problem shows the need of synchronizing several threads or processes sharing a common resource.

In this problem, there are two kinds of entities that interact with a common shared resource. Producers, who create data, and Consumers that process these data. The difficulty occurs when producers and consumers operate at different rates, potentially leading to circumstances in which producers overrun the buffer (filling it) or consumers exhaust it (emptying it).

To overcome this, synchronization techniques are required to ensure that the shared resource is used safely and orderly. For example, when the buffer is filled, producers must wait for consumers to remove elements from the buffer, allowing more space for newer items. Consumers must wait when the buffer is empty since there are no data points to process. If we let the producers and consumers run without supervision, the threads may encounter race conditions, inconsistent states, or even deadlocks, all of which compromising the reliability of our system.

This challenge replicates real-world situations including work scheduling, logging systems, or data streaming pipelines rather than only a theoretical exercise. In multithreaded systems, the producer-consumer pattern is fundamental since it captures the complexity of organizing independent entities with a dependence.

But where is my `ConcurrentQueue<T>`?

This post focuses on using lock to build a producer-consumer system, instead of using higher-level tools like ConcurrentQueue<T>. While ConcurrentQueue<T> makes it easy to create thread-safe systems, and is something you should use 99.9999% of the time in your production code, it hides how synchronization actually works. By using lock, you can see how to protect shared resources and prevent threads from interacting with each other.

Understanding the Problem

The Producer-Consumer problem can be boiled down to the following definition. We have a finite-size buffer and two classes of threads, producers and consumers, that put items into the buffer (producers) and take items out of the buffer (consumers). Let's break it down:

What is a Shared Buffer?

A shared buffer is a common data structure, more often than not a queue, that acts as the intermediary between producers and consumers, that allows us to decouple them. Producers add items (or data) in the shared buffer, while consumers remove and process these items. The shared buffer has a fixed size meaning that it does not have infinite space, meaning that only a fixed number of items can exist at any point inside the buffer. Thus a buffer can become full or empty which can cause issues:

When the Buffer is Full:

A full buffer means no more items can be added until space is freed. If a producer tries to add an item to a full buffer, several things can go wrong. The buffer might throw an exception, disrupting the system and potentially causing a crash. In extreme cases, the producer might overwrite memory it doesn’t own, leading to unpredictable and catastrophic behavior. Alternatively, the buffer might silently drop the item, resulting in data loss and leaving consumers in an inconsistent state. Finally, the producer could block indefinitely while waiting for space, and if consumers are also stalled, this can lead to a deadlock, where neither producers nor consumers can make progress.
When the Buffer is Empty:

An empty buffer means there are no items available for consumers to process. If a consumer tries to fetch data from an empty buffer, it might throw an exception or return invalid data, potentially crashing the thread or propagating errors. Without proper safeguards, the consumer could also process uninitialized or corrupted data, leading to incorrect behavior. Worse, the consumer might block indefinitely while waiting for items that will never arrive, resulting in a deadlock, as both producers and consumers remain stuck and unable to continue.

What is a Consumer?

A Consumer is a key component in a producer-consumer system, responsible for retrieving items from a shared buffer and perform operations on them. A consumer's role is to process the data or tasks provided by the producer, usually finishing the workflow by providing results, storing information, or initiating downstream actions. Consumers in a logging system, for example, write log entries to a file or database; in a data processing pipeline, they analyze or transform incoming data; and in a task queue, they perform tasks such as image rendering or transaction processing.

A consumer relies only on the shared buffer for input. If the buffer goes empty, the consumer must either wait for more items to arrive or gracefully deal with the lack of data. The way a consumer deals with this is determined by how it is designed. Blocking consumers will pause execution until the items are available, but non-blocking consumers may skip processing or retry later. A badly designed consumer may even crash or consume erroneous data if it is not correctly synced with the buffer.

What is a Producer?

A Producer is an essential component in a producer-consumer system, responsible for generating data or tasks and placing them into a shared buffer for consumers to process. Producers act as the starting point of the workflow, supplying the system with the necessary input to keep it running. In a logging system, a producer may generate log messages; in a data processing pipeline, a producer may read sensor data or retrieve information from an external source. The primary responsibility of a producer is to guarantee that relevant data or tasks are regularly made available to consumers without overwhelming the system.

A producer relies only on the shared buffer for input. It must also handle situations in which the buffer is full. Depending on the design, the producer may block until space becomes available, drop the item, or activate backpressure mechanisms to slow its generation rate. The choice is based on the system's requirements and tolerance for delays or data loss. For example, a real-time system may favor speed over dependability and drop data, but a critical system may stall the producer in order to ensure that every item is processed.

What is a Semaphore?

A Semaphore is a synchronization primitive used in computer science to control access to a common resource by multiple threads and avoid critical section problems in a concurrent system. It's like a manager that orchestrates which threads can enter a section of code that shouldn't be accessed by multiple threads at the same time. A semaphore uses a counter to track how many threads or processes can access the resource simultaneously. The counter begins with a specified initial value, representing the initial number of "permits" for accessing the resource. When a thread wants to use a resource, it decrements the counter, effectively "acquiring" a permit. If the counter is greater than zero, the thread can proceed. However, if the counter reaches zero, the thread blocks and waits until another thread releases a permit by incrementing the counter.

To understand how it works in practice, consider a semaphore initialized with a count of 3, representing three available slots for accessing a shared resource.

When processes interact with the semaphore, the count is decremented or incremented based on whether they are acquiring or releasing a resource. For instance:

Initial Semaphore Count: 3

Process A --wait()-> Decrease Count: 2
Process B --wait()-> Decrease Count: 1
Process C --wait()-> Decrease Count: 0 (No more available slots)

Process A --signal()-> Increase Count: 1 (Resource released)

Let’s say three processes—Process A, Process B, and Process C—attempt to access the resource sequentially. When Process A calls wait(), it decrements the semaphore’s count from 3 to 2 and proceeds to access the resource. Similarly, Process B calls wait(), further decreasing the count to 1, and gains access. When Process C calls wait(), it reduces the count to 0, taking the last available slot. At this point, the semaphore’s counter indicates that no slots are left, and any subsequent process calling wait() will block until a slot becomes available.

Once Process A completes its work, it calls signal() to release its slot, increasing the semaphore’s count from 0 to 1. This signals that a resource is now free, allowing another waiting process to proceed. The flow of these operations ensures that only up to three processes can access the resource simultaneously, while additional processes are queued and served in turn as resources are freed.

In a producer-consumer system, semaphores play a critical role in managing the flow of data. For instance, a semaphore can track the number of items in the buffer, blocking consumers if the buffer is empty, or track the available slots, blocking producers if the buffer is full. This coordination ensures that producers and consumers interact with the shared resource in an orderly and efficient manner, preventing race conditions or data corruption.

What is a Mutex?

A mutex (short for mutual exclusion) is a synchronization tool that allows only one thread or process to access a shared resource at a time. Unlike a semaphore, which can manage multiple simultaneous accesses, a mutex ensures exclusive access by locking the resource when one thread acquires it. Other threads attempting to acquire the mutex will block until the resource is released. This behavior prevents race conditions and ensures thread safety when accessing critical sections of code or shared data.

To understand how a mutex works, imagine a scenario where multiple threads compete to access a shared resource, such as a log file or a database connection. When a thread acquires the mutex, it effectively locks the resource, preventing other threads from interfering. Once the thread completes its task, it releases the mutex, unlocking the resource for the next waiting thread.

Initial Mutex State: Unlocked

Thread A --acquire()-> Mutex is Locked
Thread B --acquire()-> Mutex is Locked, wait
Thread C --acquire()-> Mutex is Locked, wait

Thread A --release()-> Mutex is Unlocked

This flow ensures that only one thread can access the shared resource at any given time. Unlike a semaphore, which tracks permits with a counter, a mutex is binary—it is either locked or unlocked. Additionally, mutexes are typically owned by the thread that locks them, meaning only the owning thread can release the mutex. This ownership guarantees that other threads cannot accidentally unlock a mutex they did not acquire, ensuring safe and predictable behavior.

Using Semaphores and Locks

To solve the producer-consumer problem, we can use two semaphores and a lock. Each plays a distinct role in ensuring safe and efficient synchronization.

First we are going to use two semaphores for tracking the items in the buffer. The ConsumerSemaphore tracks the number of available items in the buffer. Consumers wait on this semaphore when the buffer is empty. The ProducerSemaphore tracks the number of available slots in the buffer for producers. Producers wait on this semaphore when the buffer is full.

The BufferLock is used to ensure mutual exclusion when we are modifying our shared buffer. Both producers and consumers must acquire the lock before accessing or modifying the buffer.

The solution works by coordinating the actions of producers and consumers using semaphores and locks. Semaphores ensure that threads only proceed when specific conditions are met. For example, a producer is blocked from adding items if the buffer is full, as the space semaphore’s count will be zero. Similarly, a consumer cannot remove an item from an empty buffer, as the item semaphore’s count will prevent them from proceeding. These semaphores act as gatekeepers, ensuring producers and consumers interact with the buffer only when it is in a valid state.

The lock ensures mutual exclusion when modifying the shared buffer. Without it, multiple threads could access the buffer simultaneously, leading to race conditions, data corruption, or other undefined behavior. By requiring each thread to acquire the lock before modifying the buffer, we ensure that only one thread at a time can perform operations like adding or removing items. This protects the integrity of the buffer and prevents conflicts between producers and consumers.

Finally, the combination of semaphores and locks maintains a balanced workflow between producers and consumers. The semaphores dynamically manage the flow of threads, allowing producers to proceed only when there is space in the buffer and consumers to act only when there are items to process. This synchronization mechanism prevents deadlocks and ensures that both producers and consumers can operate efficiently without wasting resources.

Now, lets see the Produce method, that is used by producers to add data to the buffer:

PROCEDURE Produce();
BEGIN
    Wait(ProducerSemaphore);            # Wait for space in the buffer.
    Lock(BufferLock);                   # Lock the buffer for safe access.

    Add(item);                          # Add an item in the buffer.

    Unlock(BufferLock);                 # Unlock the buffer.
    Signal(ConsumerSemaphore);          # Signal that an item is available.
END;

Here's how it works:

Wait for Space:
- Wait(ProducerSemaphore) ensures that there is available space in the buffer before proceeding. If the buffer is full, the producer will block until space becomes available.
Lock the Buffer:
- Lock(BufferLock) acquires a lock on the shared buffer to prevent simultaneous access by other threads (producers or consumers). This ensures mutual exclusion.
Add an Item:
- Add(item) safely places a new item into the buffer while the lock is held.
Unlock the Buffer:
- Unlock(BufferLock) releases the lock, allowing other threads to access the buffer.
Signal Availability:
- Signal(ConsumerSemaphore) increments the semaphore, notifying consumers that a new item is now available in the buffer.

The Consume method, is used by consumers to get data from the buffer:

PROCEDURE Consume();
BEGIN
    WAIT(ConsumerSemaphore);            # Wait for items in the buffer.
    Lock(BufferLock);                   # Lock the buffer for safe access.

    Remove(item);                       # Remove an item from the buffer.

    Unlock(BufferLock);                 # Unlock the buffer.
    Signal(ProducerSemaphore);          # Signal that a slot is available
END;

Here's how it works:

Wait for Items:
- WAIT(ConsumerSemaphore) ensures that there is at least one item in the buffer before proceeding. If the buffer is empty, the consumer will block until an item becomes available.
Lock the Buffer:
- Lock(BufferLock) acquires a lock on the shared buffer to ensure mutual exclusion, preventing other threads (producers or consumers) from accessing the buffer simultaneously.
Remove an Item:
- Remove(item) safely retrieves an item from the buffer while holding the lock. This operation ensures that no other thread modifies the buffer during the removal.
Unlock the Buffer:
- Unlock(BufferLock) releases the lock, allowing other threads to access the buffer.
Signal Slot Availability:
- Signal(ProducerSemaphore) increments the semaphore, notifying producers that a slot in the buffer is now free and available for new items.

With that out of the way, lets move to the actual C# implementation.

Implementation

Now that we've discussed the concept of solving the producer-consumer problem using semaphores and locks, let's dive into the implementation. Our goal is to create a system where multiple producers can generate items and add them into a shared buffer, while multiple consumers can retrieve and process those items, all in a thread-safe manner.

The Shared Buffer

In this implementation, we are going to use a queue as the shared buffer to hold items produced by the producers until they are consumed. The queue allows us to demonstrate the concepts of synchronization and coordination in a straightforward and easily understandable manner. Specifically, the queue serves as a simple first-in, first-out (FIFO) data structure, making it intuitive to see how producers add items and consumers retrieve them.

However, it’s important to note that in production environments, you should almost always use a concurrent data structure, such as ConcurrentQueue in .NET. These data structures are optimized for multi-threaded scenarios and handle thread safety internally, making them more efficient and less error-prone. While our implementation adds locks and semaphores to make a standard queue thread-safe, using a ConcurrentQueue would eliminate the need for such explicit synchronization logic. This approach ensures better performance and maintainability in real-world applications.

For the purpose of this demonstration, we are intentionally using a manually synchronized queue to showcase how semaphores and locks work together to solve the producer-consumer problem. This allows you to understand the mechanics behind concurrency before relying on higher-level abstractions in production code.

The shared resource is designed to manage a queue in a thread-safe way, allowing producers to add items and consumers to retrieve them without conflicts. To achieve this, we use a combination of a queue, semaphores, and a lock.

public class SharedResource
{
    /// <summary>  
    /// The underlying queue that holds the data.  
    /// Access is synchronized with the <see cref="_lock"/> object.  
    /// </summary>  
    private readonly Queue<int> _queue;  

    /// <summary>  
    /// A Semaphore that tracks the number of items currently available in the queue.  
    /// Ensures consumers block when the queue is empty.  
    /// </summary>  
    private readonly SemaphoreSlim _emptyQueueSemaphore;  

    /// <summary>  
    /// A Semaphore that tracks the number of slots currently available in the queue.  
    /// Ensures producers block when the queue is full.  
    /// </summary>  
    private readonly SemaphoreSlim _fullQueueSemaphore;  

    /// <summary>  
    /// Lock object to ensure thread-safe access to the <see cref="_queue"/>.  
    /// This prevents race conditions during enqueue and dequeue operations.  
    /// </summary>  
    private readonly object _lock = new object();  

    /// <summary>  
    /// The capacity of our queue.  
    /// </summary>  
    private readonly int _maxQueueSize;}

The _queue variable is the actual queue where items are stored. It acts as the shared buffer between producers and consumers. Since multiple threads can access this queue at the same time, we need to protect it using a lock to avoid issues like race conditions.

The _emptyQueueSemaphore keeps track of how many items are currently in the queue. If the queue is empty, consumers will block on this semaphore until a producer adds an item. It ensures consumers don’t try to take something that isn’t there.

The _fullQueueSemaphore manages the available space in the queue. If the queue is full, producers will block on this semaphore until a consumer removes an item, making space available. It ensures the queue never overflows.

The _lock is a simple mechanism to ensure only one thread accesses the queue at a time. This is critical when adding or removing items to avoid data corruption or unexpected behavior.

Finally, the _maxQueueSize is the maximum number of items the queue can hold. It defines the size of the buffer and ensures that the system respects the queue’s limits.

public class Shared Resource
{
    // Variable declaration...

    public SharedResource(int capacity)  
    {  
        if (capacity <= 0)  
        {  
            throw new ArgumentOutOfRangeException(nameof(capacity), "Capacity must be greater than zero.");  
        }  
        _maxQueueSize = capacity;  
        _queue = new Queue<int>(capacity);  
        _emptyQueueSemaphore = new SemaphoreSlim(0);
        _fullQueueSemaphore = new SemaphoreSlim(capacity);
    }
}

The constructor sets everything up to make the queue functional and safe for concurrent use.

The constructor ensures the capacity is greater than zero. If someone tries to create a queue with zero or negative capacity, it throws an error. After all, a queue with no capacity wouldn’t make sense, and the _queue is created with the given capacity. This prepares the queue to handle items up to the specified limit.

Also the two semaphores are initialized. The _emptyQueueSemaphore starts with a count of 0, meaning that the queue is initially empty, so consumers will wait until producers add items. The _fullQueueSemaphore starts with a count equal to the queue's capacity, indicating that the queue has all its slots available for producers to use.

With the shared resource in place, we can now define the methods for producing and consuming items. These methods handle the core logic of adding and removing items from the queue, ensuring thread safety and synchronization through semaphores and locks.

Both methods are implemented as asynchronous to prevent blocking the execution of other tasks. Without this, producers or consumers waiting for semaphores to release (e.g., when the queue is full or empty) would block the entire thread, reducing the overall responsiveness and scalability of the system. By using asynchronous methods, threads can perform other work while waiting, making the application more efficient, especially in scenarios involving multiple producers and consumers.

The `ProduceAsync` method

The ProduceAsync method is responsible for generating items and adding them to the queue, waiting if necessary for space to become available.

public async Task ProduceAsync(int value, CancellationToken cancellationToken = default)  
{  
    await _fullQueueSemaphore.WaitAsync(cancellationToken); // Wait for space.  
    lock (_lock) // Enter critical section.  
    {  
        _queue.Enqueue(value);  
        Console.WriteLine($"Produced: {value}");  
    } // Exit critical section.  
    _emptyQueueSemaphore.Release(); // Signal that there are items available in the queue.  
}

The ProduceAsync method is the core logic for adding items to the shared queue in a thread-safe and synchronized manner. It ensures that producers only add items when there is space available and that access to the queue is properly coordinated with other threads.

The method starts by awaiting _fullQueueSemaphore.WaitAsync. This semaphore tracks the available slots in the queue. If the queue is full, the producer will wait until a consumer removes an item, freeing up space. The asynchronous nature of WaitAsync ensures that the thread isn’t blocked while waiting, allowing other operations to proceed.

A lock on _lock is used to enter the critical section where the producer interacts with the queue. This guarantees that only one thread can modify the queue at a time, preventing race conditions. Inside the critical section, the item (value) is added to the queue using _queue.Enqueue(value).

Once the item is added, the lock is released, allowing other threads (producers or consumers) to access the queue.

Finally, the producer calls _emptyQueueSemaphore.Release to signal that an item is now available in the queue. This semaphore notifies waiting consumers, allowing them to proceed with consuming the item.

The `ConsumeAsync` method

The ConsumeAsync method handles retrieving items from the shared queue in a thread-safe manner. It ensures that consumers only access the queue when there are items available and coordinates access using semaphores and locks.

public async Task<int> ConsumeAsync(CancellationToken cancellationToken = default)  
{  
    await _emptyQueueSemaphore.WaitAsync(cancellationToken);  
    var value = 0;  
    lock (_lock)  
    {  
        value = _queue.Dequeue();  
        Console.WriteLine($"Consumed: {value}");  
    }  
    _fullQueueSemaphore.Release();  

    return value;  
}

The method starts by awaiting _emptyQueueSemaphore.WaitAsync. This semaphore tracks the number of items currently available in the queue. If the queue is empty, the consumer will block here, waiting for a producer to add an item. By using the asynchronous version (WaitAsync), the thread is not blocked, allowing other tasks to execute while the consumer waits.

A lock on _lock ensures mutual exclusion while interacting with the queue. Once inside the critical section, the consumer safely retrieves an item using _queue.Dequeue(). This ensures no other threads (producers or consumers) can modify the queue while the current thread is working. After retrieving the item, the lock is released, allowing other threads to access the queue.

The consumer calls _fullQueueSemaphore.Release, signaling that a slot in the queue is now available for producers to use. This step ensures producers don’t remain blocked when the queue has room for new items.

Finally, the method returns the value retrieved from the queue, allowing the caller to process the consumed item.

Finally, we have two helper methods AvailableSpaces and AvailableElements, that return the current count of the semaphores:

public int AvailableSpaces => _fullQueueSemaphore.CurrentCount;
public int AvailableElements => _emptyQueueSemaphore.CurrentCount;

The Consumer Class

The Consumer class represents an entity that retrieves and processes items from the shared resource in a thread-safe and synchronized manner. Its role is to repeatedly consume items as they become available, ensuring no race conditions occur during the interaction with the shared resource.

private readonly SharedResource _resource;  

public Consumer(SharedResource queue)  
{  
    _resource = queue;   
}

The constructor takes a single parameter of the SharedResource type. This is the shared buffer (queue) that the consumer will interact with.

The ConsumeAsync method performs the actual consumption of items in an infinite loop, which is gracefully terminated using a CancellationToken.

public async Task ConsumeAsync(CancellationToken cancellationToken)  
{  
    while (!cancellationToken.IsCancellationRequested)  
    {   
        if (_resource.AvailableElements == 0)  
        {  
            Console.WriteLine($"Consumer: No available elements, waiting for semaphore access");  
        }  

        var item = await _resource.ConsumeAsync(cancellationToken);  
        Console.WriteLine($"Consumer: Processed {item}");  
    }  
}

Before attempting to consume an item, the method checks the shared resource’s AvailableElements property. If the queue is empty, it logs a message indicating that the consumer is waiting for semaphore access.

The consumer calls the shared resource’s ConsumeAsync method to dequeue an item. This method handles all synchronization (waiting for the semaphore, locking the queue, and signaling availability to producers). Once an item is successfully retrieved, the consumer logs the action with its identifier (Consumer) and the item value. This provides clear visibility into the actions of consumers.

The loop continues until the CancellationToken signals a request to stop. This allows the system to gracefully terminate consumers when needed.

The Producer Class

The Producer class represents an entity that generates and adds items to the shared resource. Its role is to repeatedly produce items while ensuring thread-safe interactions with the shared buffer. Like the Consumer class, it operates continuously and can be gracefully terminated using a CancellationToken.

public class Producer  
{  
    private readonly SharedResource _resource;  

    public Producer(SharedResource resource)  
    {  
        _resource = resource;  
    }  

    public async Task ProduceAsync(CancellationToken token)  
    {  
        var item = 0;  
        while (!token.IsCancellationRequested)  
        {    
            if (_resource.AvailableSpaces == 0)  
            {  
                Console.WriteLine($"Producer: No available spaces, waiting for semaphore access");  
            }  

            Console.WriteLine($"Producer: Adding {item}");  
            await _resource.ProduceAsync(item++, token);  
        }  
    }
}

The ProduceAsync method contains the logic for generating and adding items to the shared resource. The method uses a local variable item to track the current value being produced. This value increments with each iteration of the loop, ensuring each item added to the queue is unique.

Before attempting to produce an item, the method checks the shared resource’s AvailableSpaces property to determine if the queue has room. If the queue is full, it logs a message indicating that the producer is waiting for space. The method logs the item being added, providing a clear view of the producer's activity in the system.

The producer calls the shared resource’s ProduceAsync method, passing the item and the cancellation token. This method handles synchronization (waiting for the semaphore, locking the queue, and signaling availability to consumers).

The loop continues producing items until the CancellationToken signals a request to stop. This ensures the producer can be gracefully terminated when the system needs to shut down or adjust workload.

Demonstrating the Producer-Consumer System

To see the producer-consumer system in action, we need a main method to tie everything together. This method will:

Initialize the Shared Resource: Set up the queue with a fixed capacity.
Create Producers and Consumers: Instantiate multiple producers and consumers to simulate real-world behavior.
Run Asynchronous Tasks: Start the producers and consumers on separate tasks so they can work concurrently.
Graceful Shutdown: Use a CancellationTokenSource to allow stopping the system by user input.

public static async Task Main(string[] args)
{
    const int bufferSize = 5; // Define the size of the queue
    const int producerCount = 2; // Number of producers
    const int consumerCount = 3; // Number of consumers

    // Create the shared resource
    var sharedResource = new SharedResource(bufferSize);

    // Create a cancellation token for stopping the system
    var cts = new CancellationTokenSource();

    // Create and start producer tasks
    var producers = Enumerable.Range(0, producerCount)
        .Select(_ => new Producer(sharedResource))
        .Select(producer => Task.Run(() => producer.ProduceAsync(cts.Token)))
        .ToArray();

    // Create and start consumer tasks
    var consumers = Enumerable.Range(0, consumerCount)
        .Select(i => new Consumer(sharedResource, i))
        .Select(consumer => Task.Run(() => consumer.ConsumeAsync(cts.Token)))
        .ToArray();

    Console.WriteLine("Press any key to stop the system...");
    Console.ReadKey();
    cts.Cancel(); // Signal cancellation to stop all tasks

    // Wait for all tasks to complete
    await Task.WhenAll(producers.Concat(consumers));
    Console.WriteLine("System shut down gracefully.");
}

Enhancements

The producer-consumer implementation we’ve discussed is robust and demonstrates key synchronization principles. However, there are a few enhancements worth noting to make the example more realistic and to clarify its behavior under certain conditions.

Adding Random Delays to Simulate Real-Life Behavior

In real-world scenarios, producers and consumers rarely operate at consistent, predictable rates. For example:

A producer might experience delays due to external factors like data acquisition time or resource availability.
A consumer might take longer to process certain items depending on their complexity.

To simulate this variability, we can introduce random delays into the producer and consumer loops. This creates a more dynamic system and allows us to observe how the semaphores and locks handle non-deterministic behavior.

Here's how we can modify the methods:

Producer

var random = new Random();
while (!token.IsCancellationRequested)
{
    if (_resource.AvailableSpaces == 0)
    {
        Console.WriteLine($"Producer: No available spaces, waiting for semaphore access");
    }

    Console.WriteLine($"Producer: Adding {item}");
    await _resource.ProduceAsync(item++, token);

    await Task.Delay(random.Next(100, 1000)); // Random delay between 100ms and 1000ms
}

Consumer

var random = new Random();
while (!cancellationToken.IsCancellationRequested)
{   
    if (_resource.AvailableElements == 0)
    {
        Console.WriteLine($"Consumer: No available elements, waiting for semaphore access");
    }

    var item = await _resource.ConsumeAsync(cancellationToken);
    Console.WriteLine($"Consumer: Processed {item}");

    await Task.Delay(random.Next(500, 1500)); // Random delay between 500ms and 1500ms
}

By introducing these delays, the producer-consumer system more closely resembles real-world conditions where production and consumption rates fluctuate. It also showcases how the synchronization mechanisms handle unpredictable workloads.

The Illusion of First-Come, First-Served

While the implementation might seem like it operates on a First-Come, First-Served (FCFS) basis, this is not strictly true. The order in which threads are scheduled and granted access to semaphores is determined by the operating system’s thread scheduler, which prioritizes threads based on several factors, such as:

Thread priority.
CPU time availability.
The system’s overall load.

This means that even if Producer A tries to enqueue an item before Producer B, the OS might schedule Producer B’s thread first, allowing it to acquire the semaphore and the lock before Producer A. Similarly, consumers might not process items in the exact order they arrive, depending on how the OS schedules their threads.

If two consumers are waiting for items, you might observe that the one which started waiting last gets to consume first, even though the queue maintains the correct order of items. This is because semaphore access is not inherently fair—it does not guarantee that threads are served in the order they began waiting.

Understanding this nuance is important when designing systems where strict fairness or predictable thread execution order is critical. If FCFS behavior is a strict requirement, additional mechanisms, such as a per-thread semaphore or queueing requests, would need to be implemented.

If strict First-Come, First-Served (FCFS) behavior is critical for your use case, additional mechanisms such as per-thread semaphores or request queues can be implemented. If you’re interested in exploring solutions to enforce true FCFS order in a producer-consumer system, let me know in the comments—I’d be happy to dive into it in a future post!

Conclusion

The producer-consumer problem is a cornerstone of concurrent programming, demonstrating how multiple threads can safely interact with shared resources. In this post, we explored how to solve this problem using semaphores and locks, breaking down the core concepts into an intuitive, step-by-step approach. We built a shared resource with a thread-safe queue, implemented producers and consumers that operate asynchronously, and demonstrated how semaphores and locks work together to ensure synchronization and prevent race conditions.

Beyond the implementation, we delved into enhancements that bring the system closer to real-world scenarios, such as adding random delays to simulate varying workloads. We also discussed the nuances of thread scheduling and why, despite appearances, our solution isn’t strictly First-Come, First-Served (FCFS)—a limitation determined by the operating system’s thread scheduler.

This journey through the producer-consumer problem has provided both practical and conceptual insights. Whether you’re implementing this pattern in a real-world application or simply looking to deepen your understanding of concurrency, this post should serve as a solid foundation.

If you have questions, suggestions, or would like to see specific aspects—such as enforcing true FCFS order—covered in future posts, let me know in the comments.

Configuration of Services

Kostas Kalafatis — Mon, 26 Aug 2024 06:00:00 +0000

In cloud-native applications, we typically use environment variables to manage configuration settings. These variables offer a flexible way to adjust settings without modifying the source code. They are stored on Linux-based systems and can be accessed by applications to control their behavior.

For example, suppose your application relies on a LOG_LEVEL environment variable to determine the level of detail in its logs. If you change this variable from INFO to DEBUG and restart the application, you'll start seeing more detailed logs, which can help you diagnose issues more effectively.

This approach is also handy for deploying the same application across different environments, such as staging, testing, and production. You can simply adjust the environment variables for each environment to meet its specific needs.

Similarly, when using Docker Compose to manage containers, you configure services by setting environment variables for each container. This keeps your configurations consistent and easy to manage across different environments.

There are three methods of defining environment variables in Docker Compose arranged from the most stable, with the least frequent changes, to the most dynamic, with the most frequent changes:

Using the Compose file
Using shell environment variables
Using the environment file

If the environment variables you need for your containers don’t change often, it's a good idea to keep them in your docker-compose.yaml file. For sensitive information like passwords, it’s safer to set these variables directly in your shell environment before running the docker-compose command. However, if you have a lot of variables or if they differ between your testing, staging, and production environments, it's more manageable to put them in .env files and then reference those in your docker-compose.yaml.

In the services section of your docker-compose.yaml file, you can specify environment variables for each service. For instance, you can set environment variables like LOG_LEVEL and METRICS_PORT for the server service like this:

services:
    server:
        image: nginx-custom
        environment:
            - LOG_LEVEL=debug
            - METRICS_PORT=8445

If the environment variables are not defined directly in the docker-compose.yaml file, Docker Compose can still retrieve their values from the shell environment. For example, if you want the HOSTNAME environment variable for the server service to be set from the shell, you can ensure it's available by exporting it in your shell before running Docker Compose. Here's how it might look:

services:
    server:
        image: nginx-custom
        environment:
            - HOSTNAME

And exporting the HOSTNAME variable to the shell:

export HOSTNAME=myhost
docker-compose up

If the shell running the docker-compose command doesn't have a value set for the HOSTNAME environment variable, the container will start with HOSTNAME as an empty environment variable.

You can also store environment variables in .env files and configure them in your docker-compose.yaml files. For example, you might have a database.env file that looks like this:

DB_HOST=localhost
DB_PORT=5432
DB_USER=myuser
DB_PASSWORD=mypassword

In the docker-compose.yaml file, you can specify the .env file for each service using the env_file field. Here's how you would configure it for a service:

services:
    my-service:
        image: my-image
        env_file:
            - ./database.env

When Docker Compose sets up the server service, it will apply all the environment variables specified in the database.env file to the container. In the following example, you'll get hands-on experience configuring an application using all three methods for setting environment variables in Docker Compose.

Configuring Services with Docker Compose

In this example, you'll set up a Docker Compose application using various methods for configuring environment variables. You’ll start by defining two environment variables in a file named print.env. Next, you'll add one environment variable directly in the docker-compose.yaml file and set another one from the Terminal on the fly. By the end, you’ll see how these four environment variables from different sources are combined and used in your container.

Create a folder named configuration-server and navigate into it using the cd command

mkdir configuration-server
cd configuration-server

Create an .env file with the name display.env and the following content, using your favorite code editor. I am going to use VSCode:

VAR_FROM_ENV_FILE_1=HELLO
VAR_FROM_ENV_FILE_2=WORLD

Create a docker-compose.yaml file similar to the following:

services:
    display:
        image: busybox
        command: sh -c 'sleep 5 && env'
        env_file:
            - display.env
        environment:
            - ENV_FROM_COMPOSE_FILE=HELLO
            - ENV_FROM_SHELL

The docker-compose.yaml file defines a print service using the busybox image. It runs a command that sleeps for 5 seconds and then prints the environment variables. Environment variables are configured in three ways: from the print.env file, which contains key-value pairs; directly in the Compose file with ENV_FROM_COMPOSE_FILE=HELLO; and via shell environment variables with ENV_FROM_SHELL, which will be set at runtime if available.

Export the ENV_FROM_SHELL variable to the shell with the following command:

export ENV_FROM_SHELL=WORLD

Use the docker-compose up command to start the app. The output should be similar to the following:

docker-compose up
error during connect: this error may indicate that the docker daemon is not running: Get "http://%2F%2F.%2Fpipe%2Fdocker_engine/v1.24/containers/json?all=1&filters=%7B%22label%22%3A%7B%22com.docker.compose.config-hash%22%3Atrue%2C%22com.docker.compose.project%3Dconfiguration-server%22%3Atrue%7D%7D": open //./pipe/docker_engine: The system cannot find the file specified.

After this moment of embarrassment, you will remember to start the actual Docker daemon, and the output should be the following:

display Pulling
 ec562eabd705 Already exists
 display Pulled
 Network configuration-server_default  Creating
 Network configuration-server_default  Created
 Container configuration-server-display-1  Creating
 Container configuration-server-display-1  Created
Attaching to configuration-server-display-1
configuration-server-display-1  | HOSTNAME=c791b58e8a00
configuration-server-display-1  | SHLVL=1
configuration-server-display-1  | HOME=/root
configuration-server-display-1  | VAR_FROM_ENV_FILE_1=HELLO
configuration-server-display-1  | PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
configuration-server-display-1  | VAR_FROM_ENV_FILE_2=WORLD
configuration-server-display-1  | ENV_FROM_COMPOSE_FILE=HELLO
configuration-server-display-1  | ENV_FROM_SHELL=WORLD
configuration-server-display-1  | PWD=/
configuration-server-display-1 exited with code 0

The output comes from the display container defined in the docker-compose.yaml file, which runs the env command to display its environment variables. The results show two variables from the display.env file: VAR_FROM_ENV_FILE_1 with the value HELLO, and VAR_FROM_ENV_FILE_2 with the value WORLD. Additionally, the ENV_FROM_COMPOSE_FILE variable defined directly in the docker-compose.yaml file has the value HELLO. Finally, the ENV_FROM_SHELL variable, set in the shell before running Docker Compose, appears with the value WORLD.

In this example, you set up a Docker Compose application using various methods for configuration, such as Docker Compose files, environment definition files, and shell-exported values. This flexibility allows you to deploy the same application across different platforms.

Since Docker Compose handles multi-container setups, it's essential to define how these containers depend on each other. The next section will cover how to manage these interdependencies within Docker Compose applications.

Service Dependency

Docker Compose is used to manage multi-container applications, which are defined in docker-compose.yaml files. Even though the containers operate as separate microservices, it's common to have them depend on each other.

For example, in a two-tier application with a PostgreSQL database and a Java backend, the backend needs the PostgreSQL service to be running before it can connect and function correctly. Docker Compose allows you to control the startup and shutdown order of these services, ensuring that dependencies are handled properly.

services:
  init:
    image: busybox
  pre:
    image: busybox
    depends_on:
      - init
  main:
    image: busybox
    depends_on:
      - pre

In the above example, the main container depends on the pre container, and the pre container depends on the init container. Docker Compose will start the containers in the order of init, pre and main. Also, the containers will be stopped in reverse order: main, pre and then init.

In the following example, the order of containers will be used to fill a file and serve it with a web server.

Service Dependency with Docker Compose

In this exercise, you'll configure a Docker Compose application with four containers. The first three containers will run in sequence to generate a static file, which will then be served by the fourth container. This setup demonstrates how to manage and coordinate dependencies between services in Docker Compose.

Create a folder named service-dependency-example and navigate into it using the cd command

mkdir service-dependency-example
cd service-dependency-example

Create your docker-compose.yaml similar to the following

# This is the main configuration file for Docker Compose
# It defines the services that are to be run, and how they should be configured.
# Each service is defined as a key-value pair, where the key is the name of the service,
# and the value is a dictionary of configuration options.
#
# The services section is where you define the services that you want to run.
# Each service is defined as a key-value pair, where the key is the name of the service,
# and the value is a dictionary of configuration options.
#
# The configuration options for a service can include things like the image to use,
# the command to run, the ports to expose, and the volumes to mount.
#
# In this file, we have defined four services:
# - clean: This service simply cleans out the static volume.
# - init: This service initializes the static volume with a simple HTML file.
# - pre: This service simply adds data to the HTML file
# - server: This service runs an Nginx server, which serves the content from the static volume.
#
# The volumes section is where you define the volumes that are to be used by the services.
# In this case, we have defined a single volume called "static", which is mounted by all three services.
services:
  clean:
    image: busybox
    command: rm -rf /data/*
    volumes:
      - static:/data
 
  init:
    image: busybox
    command: "sh -c 'echo Hello from init >> /data/hello.html'"
    volumes:
      - static:/data
    depends_on:
      - clean

  pre:
    image: busybox
    command: "sh -c 'echo Hello from pre >> /data/hello.html'"
    volumes:
      - static:/data
    depends_on:
      - init

  server:
    image: nginx
    volumes:
      - static:/usr/share/nginx/html
    ports:
      - "8080:80"
    depends_on:
      - pre

volumes:
  static:

This Docker Compose file defines four services and a single volume named static. The volume is shared among all services. The clean service starts by removing the index.html file from the volume. The init service then creates and populates index.html. Afterward, the pre service appends an additional line to index.html. Finally, the server service serves the content from the static volume.

Start the application with the docker-compose up command. The output should look like the following:

Container service-dependency-clean-1  Created
 Container service-dependency-init-1  Recreate
 Container service-dependency-init-1  Recreated
 Container service-dependency-pre-1  Recreate
 Container service-dependency-pre-1  Recreated
 Container service-dependency-server-1  Recreate
 Container service-dependency-server-1  Recreated
Attaching to service-dependency-clean-1, service-dependency-init-1, service-dependency-pre-1, service-dependency-server-1
service-dependency-clean-1 exited with code 0
service-dependency-init-1 exited with code 0
service-dependency-pre-1 exited with code 0
service-dependency-server-1  | /docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
service-dependency-server-1  | /docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
service-dependency-server-1  | /docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
service-dependency-server-1  | 10-listen-on-ipv6-by-default.sh: info: Getting the checksum of /etc/nginx/conf.d/default.conf
service-dependency-server-1  | 10-listen-on-ipv6-by-default.sh: info: Enabled listen on IPv6 in /etc/nginx/conf.d/default.conf
service-dependency-server-1  | /docker-entrypoint.sh: Sourcing /docker-entrypoint.d/15-local-resolvers.envsh
service-dependency-server-1  | /docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
service-dependency-server-1  | /docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh
service-dependency-server-1  | /docker-entrypoint.sh: Configuration complete; ready for start up
service-dependency-server-1  | 2024/08/13 06:53:43 [notice] 1#1: using the "epoll" event method
service-dependency-server-1  | 2024/08/13 06:53:43 [notice] 1#1: nginx/1.27.0
service-dependency-server-1  | 2024/08/13 06:53:43 [notice] 1#1: built by gcc 12.2.0 (Debian 12.2.0-14)
service-dependency-server-1  | 2024/08/13 06:53:43 [notice] 1#1: OS: Linux 5.10.102.1-microsoft-standard-WSL2
service-dependency-server-1  | 2024/08/13 06:53:43 [notice] 1#1: getrlimit(RLIMIT_NOFILE): 1048576:1048576
service-dependency-server-1  | 2024/08/13 06:53:43 [notice] 1#1: start worker processes
service-dependency-server-1  | 2024/08/13 06:53:43 [notice] 1#1: start worker process 29
service-dependency-server-1  | 2024/08/13 06:53:43 [notice] 1#1: start worker process 30
service-dependency-server-1  | 2024/08/13 06:53:43 [notice] 1#1: start worker process 31
service-dependency-server-1  | 2024/08/13 06:53:43 [notice] 1#1: start worker process 32
service-dependency-server-1  | 2024/08/13 06:53:43 [notice] 1#1: start worker process 33
service-dependency-server-1  | 2024/08/13 06:53:43 [notice] 1#1: start worker process 34
service-dependency-server-1  | 2024/08/13 06:53:43 [notice] 1#1: start worker process 35
service-dependency-server-1  | 2024/08/13 06:53:43 [notice] 1#1: start worker process 36
service-dependency-server-1  | 2024/08/13 06:53:43 [notice] 1#1: start worker process 37
service-dependency-server-1  | 2024/08/13 06:53:43 [notice] 1#1: start worker process 38
service-dependency-server-1  | 2024/08/13 06:53:43 [notice] 1#1: start worker process 39
service-dependency-server-1  | 2024/08/13 06:53:43 [notice] 1#1: start worker process 40
service-dependency-server-1  | 2024/08/13 06:53:43 [notice] 1#1: start worker process 41
service-dependency-server-1  | 2024/08/13 06:53:43 [notice] 1#1: start worker process 42
service-dependency-server-1  | 2024/08/13 06:53:43 [notice] 1#1: start worker process 43
service-dependency-server-1  | 2024/08/13 06:53:43 [notice] 1#1: start worker process 44
service-dependency-server-1  | 172.21.0.1 - - [13/Aug/2024:06:53:48 +0000] "GET / HTTP/1.1" 200 31 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36" "-"

The output shows that Docker Compose creates the containers in the order of clean, init, pre and server.

Open http://localhost:8080 in the browser:

The output shows that the clean, init and pre containers work in the expected order.

Return to your terminal and hit Ctrl + C to close the application gracefully. You will notice that in the logs, that the containers stop in the reverse order.

Gracefully stopping... (press Ctrl+C again to force)
Aborting on container exit...
 Container service-dependency-server-1  Stopping
 Container service-dependency-server-1  Stopped
 Container service-dependency-pre-1  Stopping
 Container service-dependency-pre-1  Stopped
 Container service-dependency-init-1  Stopping
 Container service-dependency-init-1  Stopped
 Container service-dependency-clean-1  Stopping
 Container service-dependency-clean-1  Stopped

In this example, we set up a Docker Compose application with interdependent services to demonstrate how Docker Compose manages the startup and operation of containers in a specific order. This feature is crucial for building and orchestrating complex multi-container applications.

Summary

In this post, we explored the use of environment variables to manage configurations in cloud-native applications, specifically within Docker Compose. We demonstrated how to configure applications using environment variables defined in Docker Compose files, shell environments, and .env files, allowing for flexible and adaptable deployments across different environments like staging, testing, and production.

We also delved into the management of service dependencies in Docker Compose, showing how to control the order in which containers start and stop. We created a Docker Compose application with four interdependent services that worked together to generate and serve a static file, highlighting the importance of managing dependencies in complex multi-container applications.

And with that we covered how to compose environments using Docker Compose. In the next few posts we are going to delve into Docker Networking.

See you next Monday!

Using Docker Compose to Build Environments

Kostas Kalafatis — Mon, 19 Aug 2024 05:23:42 +0000

In our previous posts, we explored how Docker containers and Dockerfiles can be used to package applications neatly. But what happens when your applications become more complex, with multiple components and intricate configurations? Imagine building an online store, where you have separate microservices for the frontend, backend, payment processing, order management and analytics. Each of these microservices might be developed using different programming languages and technologies, and they all need to be built, packaged, and configured correctly.

This is where Docker Compose comes to the rescue. It's a powerful tool specifically designed for managing applications that run in multiple Docker containers. You can define your entire application stack in a single YAML file, including each microservice, its configuration, and how they all interact with each other. With Docker Compose, you can spin up your entire complex application with just a single command, making it super convenient for development, testing, CI/CD pipelines, and even production environments.

The essential features of Docker Compose can be grouped into three categories:

Isolation: Imagine running multiple instances of your application, each completely isolated from the others. Docker Compose makes this possible, allowing you to replicate your entire application stack on various environments like developer machines, CI servers, or shared hosts. This not only optimizes resource utilization but also simplifies management by reducing operational complexity.
Stateful Data Management: Say your application needs to store data on disk, like a database. Docker Compose takes care of managing the data volumes associated with your containers, ensuring that your data persists across different runs. This makes it much easier to work with applications that rely on persistent storage.
Iterative Design: Docker Compose works with a clear configuration that defines all the containers in your application. You can easily add new containers to this configuration without disrupting existing ones. For example, if you have two containers running and decide to add a third, Docker Compose won't touch the first two. It will simply create and connect the new container, making it a breeze to expand your application iteratively.

Docker Compose's ability to isolate multiple instances of your application, manage stateful data, and support iterative development makes it a must-have tool for handling complex applications with multiple containers. In this chapter, we'll take a closer look at how Docker Compose can help you manage the entire lifecycle of your application, from setup to deployment.

We'll start by diving deep into the Docker Compose command-line interface (CLI) and the structure of Compose files. Then, we'll explore different ways to configure your applications using Compose and how to define dependencies between different services within your application stack.

Since Docker Compose is a core tool in the Docker ecosystem, gaining both technical knowledge and practical experience with it will be a valuable addition to your skillset. By the end of this chapter, you'll be well-equipped to handle even the most intricate multi-container applications with ease.

Docker Compose CLI

Docker Compose works hand-in-hand with Docker Engine to orchestrate multi-container applications. It uses a command-line tool called docker-compose to communicate with the Engine. On Mac and Windows, this tool is conveniently bundled with Docker Desktop. However, if you're running a Linux system, you'll need to install docker-compose separately after installing Docker Engine. The good news is that it's a single executable, making the installation quite straightforward.

You can find instructions on installing the Compose Plugin here.

Docker Compose CLI Commands

The docker-compose command can handle all aspects of an application's life cycle that use multiple containers. It is possible to start, stop, and restart services with the subcommands. You can also see what's going on with the running stacks and get their logs. The following posts will give you hands-on examples with the most important commands. In the same way, the following command can be used to see a sample of all the features:

docker-compose --help

And you should see something like the following:

Usage:  docker compose [OPTIONS] COMMAND

Define and run multi-container applications with Docker.

Options:
      --ansi string                Control when to print ANSI control
                                   characters ("never"|"always"|"auto")
                                   (default "auto")
      --compatibility              Run compose in backward compatibility mode
      --dry-run                    Execute command in dry run mode
      --env-file stringArray       Specify an alternate environment file.
  -f, --file stringArray           Compose configuration files
      --parallel int               Control max parallelism, -1 for
                                   unlimited (default -1)
      --profile stringArray        Specify a profile to enable
      --progress string            Set type of progress output (auto,
                                   tty, plain, quiet) (default "auto")
      --project-directory string   Specify an alternate working directory
                                   (default: the path of the, first
                                   specified, Compose file)
  -p, --project-name string        Project name

Commands:
  build       Build or rebuild services
  config      Parse, resolve and render compose file in canonical format
  cp          Copy files/folders between a service container and the local filesystem
  create      Creates containers for a service.
  down        Stop and remove containers, networks
  events      Receive real time events from containers.
  exec        Execute a command in a running container.
  images      List images used by the created containers
  kill        Force stop service containers.
  logs        View output from containers
  ls          List running compose projects
  pause       Pause services
  port        Print the public port for a port binding.
  ps          List containers
  pull        Pull service images
  push        Push service images
  restart     Restart service containers
  rm          Removes stopped service containers
  run         Run a one-off command on a service.
  scale       Scale services
  start       Start services
  stop        Stop services
  top         Display the running processes
  unpause     Unpause services
  up          Create and start containers
  version     Show the Docker Compose version information
  wait        Block until the first service container stops
  watch       Watch build context for service and rebuild/refresh containers when files are updated

Run 'docker compose COMMAND --help' for more information on a command.

There are three key docker-compose commands crucial for managing the lifecycle of applications. Here's a breakdown of these commands and how they fit into the overall lifecycle:

docker-compose up: The docker-compose up command initializes and starts the containers as defined in your configuration file. You can either build container images from scratch or use pre-built images available in a registry. For long-running services, like web servers, it’s often practical to run the containers in detached mode by using the -d or --detach flags. This allows the containers to run in the background, freeing up your terminal for other tasks. For a full list of options and flags available with this command, you can use docker-compose up --help.
docker-compose ps: The docker-compose ps command provides a snapshot of the containers and their current status. This is particularly useful for diagnosing issues and performing health checks on your containers. For example, if you have a two-container setup with a backend and a frontend, you can use docker-compose ps to see the status of each container. It helps you identify whether any of your containers are down, not responding to health checks, or have failed to start due to misconfiguration.
docker-compose down: The docker-compose down command stops and removes all the resources, including containers, networks, images and volumes.

Docker Compose File

The Docker Compose CLI manages and configures multi-container applications. Typically, these settings are saved in a file called docker-compose.yml. Docker Compose is a powerful tool, but its efficacy is dependent on the quality of the configuration. As a result, understanding how to create and fine-tune these docker-compose.yml files is critical, requiring careful attention to detail.

A docker-compose.yaml file consists of four main sections:

name: myapp
services:
    - ...
networks:
    - ...
volumes:
    - ...

Name

Docker Compose specifies that the top-level name property will be used as the project name if you don't set one yourself. You can change this name, and if the top-level name element is not set, Docker Compose will set a default project name that will be used. Also note that the project name is exposed for interpolation and we can use the COMPOSE_PROJECT_NAME to access this name.

name: myapp

services:
    some-service:
        image: busybox
        command: echo "I'm running ${COMPOSE_PROJECT_NAME}"

Services

A service is a general term for a computer resource in an app that can be changed or grown without affecting other parts of the app. A group of containers backs up the services. The platform runs the containers based on replication needs and placement restrictions. A Docker image and a set of runtime arguments describe a service because it is backed by a container. With these arguments, all containers in a service are made in the same way.

name: myapp

services:
    server:
        image: nginx:latest
        ports:
            - 8080:80

This section of the docker-compose.yaml file defines a service named server, which is set up to run a Nginx web server. The image: nginx:latest directive instructs Docker to utilize the latest version of the Nginx image from Docker Hub. The ports section maps port 8080 on the host system to port 80 within the container. This configuration allows you to reach the Nginx server via http://localhost:8080 on your host machine.

Networks

Networks let services communicate with each other. By default Δοψκερ Compose sets up a single network for your app. Each container for a service joins the default network and is both reachable by other containers on that network, and discoverable by the service's name. The top-level networks element lets you configure named networks that can be reused across multiple services.

services:
  frontend:
    image: example/webapp
networks:
    - front-tier
    - back-tier

This section of the docker-compose.yaml file creates a service named frontend and uses the Docker image example/webapp. This image is anticipated to include the application code and environment required to operate the frontend component of a web application. The networks section states that this service is connected to two networks: front-tier and back-tier. By connecting the frontend service to these networks, it can communicate with other services on the same networks, allowing for easier interaction across different components of a multi-container application.

Volumes

Volumes in Docker are used to handle persistent data that must survive the lifecycle of a container. Volumes in Docker Compose provide a common way for services to mount and use persistent storage areas. You can construct named volumes that are reusable across several services in your application by specifying them at the top level in a docker-compose.yaml file. This configuration not only helps to guarantee data consistency and integrity, but it also simplifies storage resource management by allowing volumes to be configured and allocated centrally.

services:
  backend:
    image: example/database
    volumes:
      - db-data:/etc/data

  backup:
    image: backup-service
    volumes:
      - db-data:/var/lib/backup/data

volumes:
  db-data:

In this docker-compose.yaml configuration, the volumes section defines a named volume called db-data. This volume is then used by two different services: backend and backup.

The backend service uses the example/database image and mounts the db-data volume to the /etc/data directory inside the container. This setup allows the backend service to store and persist data in the db-data volume.

The backup service, which uses the backup-service image, also mounts the same db-data volume but maps it to a different directory inside its container, /var/lib/backup/data. This configuration enables the backup service to access and potentially back up the data stored by the backend service.

Creating a Web Server with Docker Compose

Web servers running within containers frequently require various initial operations before they can begin serving content. These tasks could include setting up configurations, downloading files, or installing dependencies. Docker Compose makes this process easier by allowing you to specify all of these operations as part of a multi-container application configuration. In this exercise, you will build a preparation container specifically for creating static files such as index.html. Following that, a server container will be configured to serve the static files. With the proper network configuration, this server will be accessible from your host system. You'll also manage the full application lifecycle with various Docker Compose commands, which will make it easier to set up and run your containers.

Create a folder named compose-server and navigate to it:

mkdir compose-server
cd compose-server

Create a folder named init and navigate to it:

mkdir init
cd init

Create a bash script to generate a simple HTML page

#!/usr/bin/env sh

# Check if a directory named 'data' exists
if [ -d "data" ]; then
    # If it does, delete it and all its contents
    echo "Removing existing 'data' directory..."
    rm -rf data
fi

# Create a new directory named 'data'
echo "Creating a new 'data' directory..."
mkdir data

# Create a new file named 'index.html' inside the 'data' directory
echo "Creating a new file 'index.html' inside the 'data' directory..."
touch data/index.html

# Append HTML content to the 'index.html' file
echo "Appending HTML content to the 'index.html' file..."
echo "<h1>Welcome to Docker Compose! </h1>" >> data/index.html
echo "<img src='https://www.docker.com/wp-content/uploads/2021/10/Moby-logo-sm.png' />" >> data/index.html

Create a Dockerfile with the name Dockerfile and the following content:

# Use the busybox base image
FROM busybox

# Copy the prepare.sh script to /usr/bin/prepare.sh in the image
ADD prepare.sh /usr/bin/prepare.sh

# Make the prepare.sh script executable
RUN chmod +x /usr/bin/prepare.sh

# Set the entry point for the container to be the prepare.sh script
# This means that when the container starts, it will run the prepare.sh script
ENTRYPOINT ["sh", "/usr/bin/prepare.sh"]

This Dockerfile starts by using the busybox base image, which is a minimal and lightweight Linux distribution commonly used for small utilities and embedded systems. It then adds a script named prepare.sh from the build context to the /usr/bin/prepare.sh path within the image. To ensure that this script can be executed, the Dockerfile uses the RUN command to modify its permissions, making it executable with chmod +x. Finally, the ENTRYPOINT instruction is specified to define the default command that will be run when the container starts. In this case, it sets the entry point to execute the prepare.sh script using the sh shell. This setup means that every time the container is launched, it will automatically run prepare.sh, allowing for pre-defined setup or configuration tasks to be carried out as the container starts.

Change the directory to the parent folder with the cd .. command and create a docker-compose.yaml file with the following content

name: "compose-server"

services:
  init:
    build:
      context: ./init
    volumes:
      - static:/data

  server:
    image: nginx
    volumes:
      - static:/usr/share/nginx/html
    ports:
      - "8080:80"

volumes:
  static:

This Docker Compose file sets up a multi-container application with two services: init and server.

Service Definitions:
- init: This service builds an image from a Dockerfile located in the ./init directory. The Dockerfile in this context likely contains instructions to prepare or generate static files. The volumes section maps a named volume, static, to the /data directory inside the container. This allows the init service to write data to this volume, which can then be accessed by other services.
- server: This service uses the official nginx image to run an NGINX web server. It also mounts the same named volume, static, but this time to the /usr/share/nginx/html directory inside the container. This is where NGINX expects to find static files to serve. The ports section maps port 80 of the container to port 8080 on the host machine, making the web server accessible via http://localhost:8080 on the host.
Volumes:
- The volumes section at the bottom defines a named volume called static. Named volumes are managed by Docker and provide persistent storage that is shared among containers. In this case, the static volume is used to transfer static files generated by the init service to the server service.

Start the application with the following command:

docker-compose up --detach

And you should see an output similar to the following:

[+] Running 8/8
 ✔ server 7 layers [⣿⣿⣿⣿⣿⣿⣿]      0B/0B      Pulled                                                               41.0s
   ✔ efc2b5ad9eec Pull complete                                                                                   32.9s
   ✔ 8fe9a55eb80f Pull complete                                                                                   29.3s
   ✔ 045037a63be8 Pull complete                                                                                    0.9s
   ✔ 7111b42b4bfa Pull complete                                                                                    1.6s
   ✔ 3dfc528a4df9 Pull complete                                                                                    3.0s
   ✔ 9e891cdb453b Pull complete                                                                                    5.3s
   ✔ 0f11e17345c5 Pull complete                                                                                    6.2s
[+] Building 5.3s (9/9) FINISHED                                                                         docker:default
 => [init internal] load build definition from Dockerfile                                                          0.1s
 => => transferring dockerfile: 457B                                                                               0.0s
 => [init internal] load .dockerignore                                                                             0.1s
 => => transferring context: 2B                                                                                    0.0s
 => [init internal] load metadata for docker.io/library/busybox:latest                                             4.5s
 => [init auth] library/busybox:pull token for registry-1.docker.io                                                0.0s
 => [init internal] load build context                                                                             0.0s
 => => transferring context: 771B                                                                                  0.0s
 => CACHED [init 1/3] FROM docker.io/library/busybox@sha256:9ae97d36d26566ff84e8893c64a6dc4fe8ca6d1144bf5b87b2b85  0.0s
 => => resolve docker.io/library/busybox@sha256:9ae97d36d26566ff84e8893c64a6dc4fe8ca6d1144bf5b87b2b85a32def253c7   0.0s
 => [init 2/3] ADD prepare.sh /usr/bin/prepare.sh                                                                  0.0s
 => [init 3/3] RUN chmod +x /usr/bin/prepare.sh                                                                    0.5s
 => [init] exporting to image                                                                                      0.1s
 => => exporting layers                                                                                            0.0s
 => => writing image sha256:8d2660840cab8d1e01e7cb9ad823fb2a2e3141b73c6cc0cce53c91355a6d52a6                       0.0s
 => => naming to docker.io/library/compose-server-init                                                             0.0s
[+] Running 4/4
 ✔ Network compose-server_default     Created                                                                      0.3s
 ✔ Volume "compose-server_static"     Created                                                                      0.0s
 ✔ Container compose-server-init-1    Started                                                                      0.1s
 ✔ Container compose-server-server-1  Started                                                                      0.1s

This command creates and starts the containers in detached mode. It starts by creating the compose-server_default network and the compose-server_static volume. It then builds the init container using the Dockerfile from the previous step, downloads nginx, and starts the containers.

Check the status of the application with the docker-compose ps command:

docker-compose ps

and you should see the following:

NAME                      IMAGE     COMMAND                                          SERVICE   CREATED         STATUS         PORTS
compose-server-server-1   nginx     "/docker-entrypoint.sh nginx -g 'daemon off;'"   server    3 minutes ago   Up 3 minutes   0.0.0.0:8080->80/tcp

Open the http://localhost:8080/ on your browser. If everything went as planned you should see the following:

Stop and remove all the resources with the following command if you do not need the application up and running:

docker-compose down

And you should see the following output:

[+] Running 3/3
 ✔ Container compose-server-server-1  Removed                                                                      0.6s
 ✔ Container compose-server-init-1    Removed                                                                      0.0s
 ✔ Network compose-server_default     Removed                                                                      0.3s

In this exercise, we successfully created and configured a multi-container application using Docker Compose. The docker-compose.yaml file was employed to define both networking and volume options, which are crucial for inter-container communication and data persistence. We demonstrated how to use Docker Compose CLI commands to build and manage the application. These commands included starting and stopping containers, checking their status, and removing them when no longer needed.

Summary

This blog post highlighted Docker Compose's ability to manage and orchestrate multi-container applications effectively. Building on our earlier explorations of Docker containers and Dockerfiles, we looked at how Docker Compose simplifies the setup and administration of complicated applications with various components. Docker Compose, which defines your whole application stack in a single YAML file, enables you to spin up complex environments with a single command, making it ideal for development, testing, and production.

We've highlighted Docker Compose's major capabilities, such as its ability to separate container instances, handle stateful data using named volumes, and facilitate iterative development. The practical examples offered demonstrated how to effectively develop, manage, and troubleshoot applications using Docker Compose CLI commands. We also went over the key components of a docker-compose.yaml file, like service definitions, network setups, and volume management.

Looking ahead, our next post will go over the configuration choices accessible in the Docker Compose environment. We'll go over advanced settings and approaches for further optimizing and tailoring your application installations, ensuring that you can fully utilize Docker Compose's capabilities to meet your individual requirements.

See you next Monday!

Multi-Stage Dockerfiles

Kostas Kalafatis — Mon, 12 Aug 2024 06:00:00 +0000

In the previous posts, we explored Docker registries, covering both private and public options. We set up our private Docker registry to manage and store our Docker images securely. Additionally, we covered configuring access and storing Docker images on Docker Hub. Now, in this next chapter, we'll delve into the concept of multi-stage Dockerfiles. This technique allows us to streamline Docker image builds by using multiple stages to optimize build efficiency and reduce image size.

Multi-stage Dockerfiles, introduced in Docker version 17.05, are designed to enhance Docker image optimization for production environments by minimizing image size. This approach involves creating multiple intermediate Docker images during the build process. Each stage focuses on specific tasks and selectively copies essential artifacts to subsequent stages, thereby streamlining the final image composition.

Before the advent of multi-stage builds, the builder pattern was employed to achieve similar optimization goals. However, this method required two Dockerfiles and a shell script, adding complexity to the build process.

We'll begin this post by examining traditional Docker builds and the challenges they pose. Then, we'll explore the builder pattern's implementation for optimizing image size, highlighting its limitations and complexities. Finally, we'll delve into multi-stage Dockerfiles, demonstrating how they effectively address these issues, offering a more streamlined approach to Docker image optimization.

Normal Docker Builds

We can use Dockerfiles to create our own Docker images. As we discussed in the previous posts, a Dockerfile is a text file that instructs docker on how to make an image. But when we run these Docker images in production environments. it is important to have images that are as small as possible.

Consider an example where we build a simple Java application. We are going to deploy a HelloWorld application written in Java using the following Dockerfile:

# Use the official openjdk image as a base image
FROM openjdk:latest

# Set the working directory in the Docker container
WORKDIR /app

# Copy the HelloWorld.java file into the Docker container
COPY HelloWorld.java .

# Compile the HelloWorld.java file
RUN javac HelloWorld.java

# Specify the command to run when the Docker container starts
CMD ["java", "HelloWorld"]

This Dockerfile is designed to build and run a Java application inside a Docker container. It starts by specifying the base image as openjdk:latest, which provides an environment with OpenJDK 11 installed. The WORKDIR /app instruction sets the working directory inside the container to /app, ensuring all subsequent commands are executed relative to this directory.

Next, the COPY HelloWorld.java command copies the HelloWorld.java file from the host machine into the /app directory of the Docker container. This file contains the source code for the Java program.

Following that, the RUN javac HelloWorld.java command compiles the HelloWorld.java file within the Docker container using the Java compiler (javac). This step generates the corresponding bytecode (HelloWorld.class) necessary to run the Java application.

Lastly, the CMD ["java", "HelloWorld"] instruction specifies the command that should be executed when the Docker container starts. Here, it runs the Java Virtual Machine (java) with HelloWorld as the main class. This runs the compiled Java program, printing "Hello, World!" or any other specified output defined within HelloWorld.java.

The following is the content of the HelloWorld.java file. This is a simple file that will print the text "Hello World!" when executed:

public class HelloWorld {

    public static void main(String[] args) {
        System.out.println("Hello World!");
    }
}

Once the Dockerfile is ready, we can build the Docker image using the docker image build command. This image will be tagged as helloworld:v1

docker image build -t helloworld:v1 .

Now, observe the built image with the docker image ls. You will get an output similar to the following

REPOSITORY                                      TAG       IMAGE ID       CREATED        SIZE
helloworld                                      v1        3dbb42c8d45c   2 days ago     470MB

Take a look at the image size—a whopping 470MB! That's a huge Docker image for a production environment. Not only will it hog bandwidth and time when you're pushing and pulling it across networks, but it's also a potential security risk. Why? Because it likely contains a bunch of build tools that could be vulnerable to attacks.

To make your image leaner and meaner, the best practice is to strip it down to the bare essentials: the compiled code and the runtime environment it needs to function. Think of it this way: you need a hammer to build a house, but once it's built, you don't need to carry the hammer around everywhere you go.

Take Java, for instance. The Java compiler is essential for building the application, but it's unnecessary baggage once the app is up and running. Ideally, you want a minimal Docker image that only contains the Java runtime and nothing else. This reduces the attack surface, making it harder for malicious actors to exploit vulnerabilities.

The Builder Pattern

The builder pattern (which is different from the builder design pattern) is a clever technique for creating Docker images that are as small as possible. It involves two Docker images working together to selectively copy only the essential parts of your application from one image to the other.

The first image is called the build image. It's like your workshop, where you have all the tools you need to build your application from the source code. This includes things like compilers, build tools, and any libraries or dependencies your application needs during the build process.

The second image, aptly named the runtime image, is where the magic happens. It's the environment where your compiled application actually runs. This image is a minimalist, containing only the essential components: your executable files, any necessary runtime dependencies, and the runtime tools needed to execute your code.

To get these essentials from the build image to the runtime image, a shell script comes into play. It uses the docker container cp command to selectively copy only the required files, leaving behind the bulky build tools and unnecessary dependencies.

The entire process of building the image using the builder pattern consists of the following steps:

Create the Build Docker image.
Create a container from the Build Docker image.
Copy the artifacts from the Build Docker image to the local filesystem.
Build the Runtime Docker image using copied artifacts.

The diagram clearly demonstrates the builder pattern workflow:

Build Container: The Build Dockerfile is responsible for creating the build container. This container comes packed with all the tools needed to compile your source code, like compilers, build tools, and development dependencies. Think it as a well-equipped workshop.
Shell Script Action: Once the Build Container is up and running, a handy shell script takes charge. It uses the docker container cp command to carefully transfer only the necessary executable files from the build container to the Docker host (a.k.a., your local machine).
Runtime Container: Finally, the Runtime Dockerfile comes into play. This Dockerfile is specifically designed to create a lean, mean runtime container. It takes the executable files copied over from the build container and packages them together with any essential runtime dependencies. This lightweight runtime container is optimized for deployment end efficient execution of your application.

This two-step process allows us to keep the final Docker image small and secure by leaving behind the bulky build tools and unnecessary dependencies.

Building a Docker Image with the Build Pattern

In this example, we are going to optimize our Docker image using the builder pattern

Create a new directory named builder-pattern for this example

mkdir builder-pattern

Navigate to the newly created builder-pattern directory

cd builder-pattern

Within the builder-pattern directory, create a file named Welcome.java. This file will be copied to the Docker image at build time

code Welcome.java

Add the following content to the Welcome.java file, and then save and exit this file

public class Welcome { 
    public static void main(String[] args) {
        System.out.println("Hello From Docker!");
    }
}

This is a simple Hello World application written in Java. This will output "Hello From Docker!" once executed.

Within the builder-pattern directory create a file named Dockerfile.build. This file will contain all the instructions that you are going to use to create the build Docker image

code Dockerfile.build

Add the following content to the Dockerfile.build file and save the file

FROM openjdk:latest

# Set working directory inside the builder image
WORKDIR /app

# Copy the Java source file into the builder image
COPY HelloWorld.java /app

# Compile the Java source file
RUN javac HelloWorld.java

# Create the JAR file
RUN jar cvf HelloWorld.jar HelloWorld.class

It starts by selecting the openjdk:latest image from Docker Hub, which provides the latest version of OpenJDK, an open-source implementation of the Java Platform, Standard Edition. This choice ensures that the Docker environment includes all necessary tools and libraries to compile and run Java applications.

Next, the working directory inside the Docker container is set to /app using the WORKDIR /app command. This directory will serve as the context for all subsequent commands in the Dockerfile, simplifying file operations and ensuring that files are copied and created in the correct location.

The COPY HelloWorld.java /app command copies the HelloWorld.java source file from the host machine into the /app directory of the Docker container. This file contains the Java source code for the application that we want to compile and package.

To compile the Java source code into bytecode, the Dockerfile uses the RUN javac HelloWorld.java command. This command executes the Java compiler (javac) inside the Docker container, transforming the human-readable Java source code (HelloWorld.java) into machine-readable bytecode (HelloWorld.class). The resulting .class file contains the compiled Java application ready for execution.

Finally, the Dockerfile creates a JAR file (HelloWorld.jar) from the compiled HelloWorld.class file using the RUN jar cvf HelloWorld.jar HelloWorld.class command. Here, the jar command is invoked with options cvf, where:

c: create a new JAR file
v: generate verbose output (optional but helpful for debugging)
f: specify the filename for the JAR file (HelloWorld.jar) The command packages the compiled Java application into a standalone JAR file, encapsulating all necessary resources and dependencies for the application.

Next, create the Dockerfile for the runtime container. Within the builder-pattern directory, create a file named Dockerfile. This file will contain all the instructions that we are going to use to create the runtime Docker image

code Dockerfile

Add the following content to the Dockerfile and save it

FROM openjdk:11-jdk-slim

# Set working directory inside the builder image
WORKDIR /app

# Copy the Java source file
COPY HelloWorld.jar /app/HelloWorld.jar

CMD ["java", "-jar", "HelloWorld.jar"]

This Dockerfile is designed to run a Java application packaged as a JAR file within a Docker container. It begins by selecting the openjdk:11-jdk-slim image from Docker Hub. This image provides a minimalistic Java Development Kit (JDK) environment based on OpenJDK 11, optimized for smaller container size and efficient resource usage.

The working directory inside the Docker container is set to /app using the WORKDIR /app command. This directory will serve as the context for all subsequent commands in the Dockerfile, ensuring that files are copied and executed from the correct location.

The COPY HelloWorld.jar /app/HelloWorld.jar command copies the HelloWorld.jar file from the host machine into the /app directory of the Docker container. This JAR file is assumed to contain the compiled Java application (HelloWorld.class and its dependencies) that we want to run.

The Dockerfile concludes with the CMD ["java", "-jar", "HelloWorld.jar"] command. This command specifies the default command to run when the Docker container starts. Here:

java is the command to execute the Java Virtual Machine (JVM).
-jar indicates that the next argument (HelloWorld.jar) is the path to the JAR file that contains the Java application to be executed.

When the Docker container is launched, it will execute the specified java -jar HelloWorld.jar command, which starts the JVM and runs the Java application encapsulated within the HelloWorld.jar file.

Now, let's create a shell script to copy the executables between Docker containers. Within the builder-pattern directory, create a file named build.sh. This file will contain the steps to coordinate the build process between the two Docker containers

code build.sh

Add the following content to the shell script and save the file

set -x

IMAGE_NAME="welcomeapp"
BUILDER_IMAGE_NAME=$IMAGE_NAME"-builder"
IMAGE_TAG="latest"

echo "Building $BUILDER_IMAGE_NAME:$IMAGE_TAG"

# Clean up the existing build
rm -rf target

# Build the builder image
docker build -t $BUILDER_IMAGE_NAME:$IMAGE_TAG -f Dockerfile.build .
if [ $? -ne 0 ]; then
    echo "Failed to build $BUILDER_IMAGE_NAME:$IMAGE_TAG"
    exit 1
fi

# Create the container
docker create --name $BUILDER_IMAGE_NAME $BUILDER_IMAGE_NAME:$IMAGE_TAG
if [ $? -ne 0 ]; then
    echo "Failed to create $BUILDER_IMAGE_NAME:$IMAGE_TAG"
    exit 1
fi

# Copy the build output into the Docker host
docker cp $BUILDER_IMAGE_NAME:/app/HelloWorld.jar ./HelloWorld.jar

# Clean up the container
docker rm $BUILDER_IMAGE_NAME
docker rmi $BUILDER_IMAGE_NAME:$IMAGE_TAG

# Build the actual image
docker build -t $IMAGE_NAME:$IMAGE_TAG -f Dockerfile .

This script automates the creation and deployment of a Dockerized application named "welcomeapp." Here's how it works:

Setup: It defines names and tags for the Docker images involved: one for building (BUILDER_IMAGE_NAME) and one for the final application (IMAGE_NAME).
Builder Image Creation: It builds a builder image using a specific Dockerfile (Dockerfile.build), presumably containing instructions for compiling the application. It cleans up any previous build artifacts before starting and checks for build errors, stopping if any occur.
Artifact Extraction: A temporary Docker container is created from the builder image. This container is used to copy the compiled application files (from /app/) out of the container and into the host system's file system.
Cleanup: The script tidies up by removing the temporary container and deleting the builder image, as they're no longer needed.
Application Image Creation: Finally, it builds the actual application image (IMAGE_NAME) using a different Dockerfile, likely incorporating the extracted application files. This image is ready for deployment and running the "welcomeapp."

In a nutshell, this script simplifies the whole process by dividing the build into two steps (builder and application), ensuring a clean and efficient way to package and deploy the "welcomeapp" in a Docker container.

Add execution permissions to the build.sh shell script

chmod +x build.sh

Now that you have the two Dockerfiles and the shell script, build the Docker image by executing the build.sh shell script

./build.sh

You should get the following output:

++ IMAGE_NAME=helloapp
++ BUILDER_IMAGE_NAME=helloapp-builder
++ IMAGE_TAG=latest
++ echo 'Building helloapp-builder:latest'
Building helloapp-builder:latest
++ rm -rf target
++ docker build -t helloapp-builder:latest -f Dockerfile.build .
#0 building with "default" instance using docker driver

#1 [internal] load .dockerignore
#1 transferring context: 2B done
#1 DONE 0.0s

#2 [internal] load build definition from Dockerfile.build
#2 transferring dockerfile: 344B done
#2 DONE 0.0s

#3 [internal] load metadata for docker.io/library/openjdk:latest
#3 ...

#4 [auth] library/openjdk:pull token for registry-1.docker.io
#4 DONE 0.0s

#3 [internal] load metadata for docker.io/library/openjdk:latest
#3 DONE 3.6s

#5 [1/5] FROM docker.io/library/openjdk:latest@sha256:9b448de897d211c9e0ec635a485650aed6e28d4eca1efbc34940560a480b3f1f
#5 DONE 0.0s

#6 [internal] load build context
#6 transferring context: 169B done
#6 DONE 0.0s

#7 [4/5] RUN javac HelloWorld.java
#7 CACHED

#8 [2/5] WORKDIR /app
#8 CACHED

#9 [3/5] COPY HelloWorld.java /app
#9 CACHED

#10 [5/5] RUN jar cvf HelloWorld.jar HelloWorld.class
#10 CACHED

#11 exporting to image
#11 exporting layers done
#11 writing image sha256:2d5900c70ddee5681fafde163931dc1536f6812e2caeccbc172794b13c99552b done
#11 naming to docker.io/library/helloapp-builder:latest done
#11 DONE 0.0s

What's Next?
  View a summary of image vulnerabilities and recommendations → docker scout quickview
++ '[' 0 -ne 0 ']'
++ docker create --name helloapp-builder helloapp-builder:latest
bf4df7ae3e2fed2dc97b2f1952656819c87e37454108c7ef6a9e3747895f0adb
++ '[' 0 -ne 0 ']'
++ docker cp helloapp-builder:/app/HelloWorld.jar ./HelloWorld.jar
++ docker rm helloapp-builder
helloapp-builder
++ docker rmi helloapp-builder:latest
Untagged: helloapp-builder:latest
Deleted: sha256:2d5900c70ddee5681fafde163931dc1536f6812e2caeccbc172794b13c99552b
++ docker build -t helloapp:latest -f Dockerfile .
#0 building with "default" instance using docker driver

#1 [internal] load build definition from Dockerfile
#1 transferring dockerfile: 243B done
#1 DONE 0.0s

#2 [internal] load .dockerignore
#2 transferring context: 2B done
#2 DONE 0.0s

#3 [internal] load metadata for docker.io/library/openjdk:11-jdk-slim
#3 DONE 0.8s

#4 [1/3] FROM docker.io/library/openjdk:11-jdk-slim@sha256:868a4f2151d38ba6a09870cec584346a5edc8e9b71fde275eb2e0625273e2fd8
#4 DONE 0.0s

#5 [2/3] WORKDIR /app
#5 CACHED

#6 [internal] load build context
#6 transferring context: 798B done
#6 DONE 0.0s

#7 [3/3] COPY HelloWorld.jar /app/HelloWorld.jar
#7 DONE 0.0s

#8 exporting to image
#8 exporting layers 0.0s done
#8 writing image sha256:73f70ee8012a258801314c0280f1b2788afbffbf57c6235a09f00e30406e3404 done
#8 naming to docker.io/library/helloapp:latest done
#8 DONE 0.0s

Use the docker image ls command to list all the Docker images available

docker image ls

You should get a list of all the available Docker images as shown below

REPOSITORY                                      TAG       IMAGE ID       CREATED              SIZE
helloapp                                        latest    73f70ee8012a   About a minute ago   274MB

As you can see the image created has a size of 274MB, while our previous image had a size of 470 MB.

While the builder pattern is a handy way to slim down your Docker images, it does come with some overhead. You have to juggle two Dockerfiles and a shell script to make everything work smoothly. Fortunately, there's a more streamlined approach: multi-stage Dockerfiles.

In the next section, we'll ditch the multiple files and shell scripts and explore how multi-stage Dockerfiles can simplify the process while still achieving the same goal of creating leaner and more efficient images. Get ready to discover a simpler and more elegant way to optimize your Docker builds.

Introduction to Multi-Stage Dockerfiles

Multi-stage Dockerfiles offer a streamlined way to create optimized Docker images within a single file. They achieve this by incorporating multiple stages, each potentially using a different base image and serving a specific purpose. Similar to the builder pattern we explored earlier, these stages typically include a builder stage for compiling source code and a runtime stage for executing the resulting application.

The key difference lies in the use of multiple FROM directives within a single Dockerfile, where each directive marks the beginning of a new stage. This allows for greater flexibility and efficiency, as you can selectively copy only the essential files from one stage to another, leaving behind any unnecessary build tools or dependencies.

Before the introduction of multi-stage builds, the builder pattern was the go-to method for achieving this optimization. However, multi-stage builds offer a more elegant and concise solution within a single Dockerfile, eliminating the need for multiple files and external scripts.

Multi-stage Docker builds offer a significant advantage over the builder pattern by allowing you to create equally small and efficient Docker images without the hassle of managing multiple files and scripts.

While the builder pattern requires maintaining two Dockerfiles and a shell script, multi-stage builds consolidate everything into a single Dockerfile, simplifying the process considerably. Moreover, the builder pattern necessitates copying executables to the Docker host as an intermediate step before incorporating them into the final image. This extra step isn't needed with multi-stage builds, as you can directly transfer files between different stages within the same Dockerfile using the --from flag.

In essence, multi-stage builds eliminate the extra work and potential for errors associated with the builder pattern, making it a more streamlined and efficient approach to creating optimized Docker images.

The key distinction between a regular Dockerfile and a multi-stage Dockerfile is the use of multiple FROM directives. Each FROM statement kicks off a new phase or stage in the build process, starting with a fresh base image. This means that each stage is isolated from the previous one, except for the specific files or directories you explicitly copy over.

Think of it like building a house in stages. You start with a foundation (the base image), then build the frame, add walls, wiring, plumbing, and so on. Each stage builds upon the previous one, but you don't need all the tools and materials from earlier stages once you've moved on to the next.

In a multi-stage Dockerfile, you use the COPY --from=n instruction to selectively copy the necessary files (like your compiled application) from one stage to another. The number 0 refers to the first stage (stages are numbered starting from 0). This allows you to build your application in one stage with all the necessary tools and dependencies, and then create a leaner final image in a later stage that only contains the essential runtime components.

In a multi-stage Dockerfile, you can refer to each stage by a number, starting from 0. However, to make your Dockerfiles easier to read and maintain, it's often a good practice to give each stage a meaningful name.

You can do this by adding the AS keyword after the FROM directive, followed by the name you want to give the stage. For example:

FROM node:18 AS build
# ... build stage instructions
FROM nginx:alpine AS runtime
# ... runtime stage instructions

In this example, we've named the first stage build and the second stage runtime. This makes it clearer what each stage is for and can be helpful when you're referencing stages later in the Dockerfile.

When working with multi-stage Dockerfiles, you might not always need to build all the way to the final stage. Let's imagine you have a Dockerfile with two stages:

Development Stage: Packed with all the build and debugging tools you need for coding and testing.
Production Stage: A leaner version that only includes the necessary runtime tools for deployment.

During development, you might only want to build up to the development stage to test your code and catch any bugs. In this case, the --target flag in the docker build command becomes your best friend. It lets you specify which stage should be the final one for the resulting image.

For example, if you want to stop at the development stage, you'd run:

docker build --target developement -t my-image:dev .

Here, development is the name we gave to the development stage in the Dockerfile, and my-image:dev is the name and tag we're giving to the resulting image.

This way, you can create multiple images from a single Dockerfile, each tailored to a specific purpose in your development workflow.

Building a Docker Image with a Multi-Stage Docker Build

In this example, we are going to create a Go application using a Multi-Stage Docker Build. Create a new directory named multi-stage-build for this example:

mkdir multi-stage-build

And navigate to it:

cd multi-stage-build

Create the following hellodocker.go file

package main

import "fmt"

func main() {
    fmt.Println("Hello, Docker! I am built using a multi-stage build.")
}

Now create the multi-stage Dockerfile

FROM golang:latest AS builder
WORKDIR /app
COPY hellodocker.go .
RUN go build -o hello hellodocker.go

FROM scratch
WORKDIR /app
COPY --from=builder /app/hello .
ENTRYPOINT ["./hello"]

This Dockerfile employs a multi-stage build process, optimizing the resulting image size. It begins with a builder stage, leveraging the golang:latest image to create an environment suitable for compiling Go code. Within this stage, the hellodocker.go source file is copied into the /app directory, and the Go compiler (go build) generates an executable named hello.

The subsequent stage uses the scratch image, a bare-bones image devoid of an operating system, as its foundation. This minimalistic environment is ideal for running the compiled application. The hello executable, created in the previous builder stage, is copied into the /app directory of this scratch image, ensuring that only the essential binary and its runtime dependencies are included.

Finally, the ENTRYPOINT ["./hello"] instruction dictates that upon container creation, the hello binary will be executed, starting the application. This multi-stage approach effectively minimizes the final image size, making it more efficient for deployment and execution.

Build the Docker image using the following command

docker build -t hello-multistage:v1

You should get an output similar to the following

#0 building with "default" instance using docker driver

#1 [internal] load build definition from Dockerfile
#1 transferring dockerfile: 233B done
#1 DONE 0.0s

#2 [internal] load .dockerignore
#2 transferring context: 2B done
#2 DONE 0.0s

#3 [internal] load metadata for docker.io/library/golang:latest
#3 ...

#4 [auth] library/golang:pull token for registry-1.docker.io
#4 DONE 0.0s

#3 [internal] load metadata for docker.io/library/golang:latest
#3 DONE 5.1s

#5 [internal] load build context
#5 DONE 0.0s

#6 [stage-1 1/2] WORKDIR /app
#6 DONE 0.0s

#7 [builder 1/4] FROM docker.io/library/golang:latest@sha256:829eff99a4b2abffe68f6a3847337bf6455d69d17e49ec1a97dac78834754bd6
#7 resolve docker.io/library/golang:latest@sha256:829eff99a4b2abffe68f6a3847337bf6455d69d17e49ec1a97dac78834754bd6 0.0s done
#7 ...

#5 [internal] load build context
#5 transferring context: 159B done
#5 DONE 0.0s

#7 [builder 1/4] FROM docker.io/library/golang:latest@sha256:829eff99a4b2abffe68f6a3847337bf6455d69d17e49ec1a97dac78834754bd6
[...truncated...]

#8 [builder 2/4] WORKDIR /app
#8 DONE 1.0s

#9 [builder 3/4] COPY hellodocker.go .
#9 DONE 0.0s

#10 [builder 4/4] RUN go build -o hello hellodocker.go
#10 DONE 3.7s

#11 [stage-1 2/2] COPY --from=builder /app/hello .
#11 DONE 0.1s

#12 exporting to image
#12 exporting layers 0.1s done
#12 writing image sha256:6ff5d8bfd14ee1893ea5d1e315aa8c394edbc879369cd3d59326dd6256184625 done
#12 naming to docker.io/library/hello-multistage:v1 done
#12 DONE 0.1s

What's Next?
  View a summary of image vulnerabilities and recommendations → docker scout quickview

Use the docker image ls command to list all docker images available on your computer.

docker images ls

REPOSITORY                                      TAG       IMAGE ID       CREATED         SIZE
hello-multistage                                v1        6ff5d8bfd14e   9 minutes ago   1.89MB

In this example we discussed how to use a multi-stage Dockerfile to build optimized Docker images.

Summary

Multi-stage builds, introduced in Docker version 17.05, aim to streamline the Docker image creation process by using multiple stages within a single Dockerfile. Each stage focuses on specific tasks and allows for selective copying of artifacts between stages, ultimately optimizing the final image size and efficiency.

Before multi-stage builds, we often relied on the builder pattern, which involved using separate Dockerfiles and shell scripts to achieve similar optimization goals. While effective, this approach added complexity to the build process.

In this post, we started by discussing the challenges of traditional Docker builds, especially in production environments where image size and build efficiency are critical. Then, we introduced the builder pattern as a precursor to multi-stage builds, explaining its use of two Docker images—one for building and one for runtime—with selective artifact copying between them.

Finally, we delved into multi-stage Dockerfiles as a more integrated solution. Here, you define multiple stages within a single Dockerfile using FROM directives. Each stage builds upon the previous one but allows you to copy only the necessary files, eliminating the need for multiple Dockerfiles and external scripts. This approach simplifies the build process while ensuring smaller, more efficient Docker images for deployment.

Overall, multi-stage Dockerfiles offer a streamlined and elegant solution for optimizing Docker image builds, reducing complexity while enhancing efficiency—a significant benefit for anyone managing Docker images in production environments.

Storing and Publishing your Docker Images

Kostas Kalafatis — Mon, 05 Aug 2024 06:00:00 +0000

From the get-go, Docker's popularity stemmed from having a central hub where users could grab images, tweak them, and share their creations with others. Docker Hub has become a go-to resource for developers looking for pre-built images to kickstart their projects. Despite some security hiccups along the way, it's still the default starting point for many.

As a public repository, Docker Hub is a treasure trove of open-source images, perfect for streamlining your next project. It's also a great platform for companies and developers to share their work with the community.

But hold on, there's more to the image-sharing game than just Docker Hub. When it comes to production environments, especially for larger teams, relying on a public repository might not be the most secure or efficient option.

Nowadays, you've got cloud-based options like Amazon ECR or Google Container Registry, which offer more control and security for your images. And if you're feeling adventurous, I'll even show you how to set up your own local registry later on.

In this post, we'll get hands-on with transferring Docker images between different machines. We'll start by exploring how to share your creations on Docker Hub, making them publicly available to the world. Then, for a more private setup, we'll dive into setting up your own local Docker registry right on your development machine. This gives you more control and security over your images, perfect for team collaboration or sensitive projects.

Moving Docker Images Manually

In certain situations, like dealing with firewalls or network restrictions, you might need to directly transfer a Docker image between machines without using a registry. Luckily, Docker has you covered! This example will walk you through how to move an image from one system to another without relying on a registry.

We are going to use the image created in the previous post, and the docker save command. We'll save it as a .tar file with a specific filename and directory. To do that use the following command. Make sure that you replace the user, image name, and tag of the image with what you used.

docker save -o /tmp/base-image.tar base-image:1.1.0

You should now find the image you just saved sitting in your /tmp directory, packaged up as a TAR file (base-image.tar). This is because the docker save command always creates a TAR archive. But here's a fun fact: the file extension itself doesn't matter all that much. You could rename it to .tar.gz or even .myimage if you wanted to! The content inside is what's important, and Docker will recognize it as an image file regardless of the extension.

Use the du command to verify that the base-image.tar file has data in it

du -sh /tmp/base-image.tar
22M     base-image.tar

Now that you've saved your Docker image as a TAR file, you're free to move it around however you like. You can use tools like rsync, scp, or even a simple cp command. If you want to save a bit of space during the transfer, you can even compress the TAR file into a ZIP archive.

For this example, let's just remove the image from your current system to clean things up. You'll need the image ID for this

docker images

And then remove the image

docker rmi -f b5e88b0b4601                                                         

Untagged: base-image:1.1.0
Deleted: sha256:b5e88b0b4601183bb9425186cd95fee52e22be5f1ec8c58b95a50f42e78c2004

Imagine that you've successfully transferred your image to a new machine, let's load it back into Docker so you can use it again. You'll need the docker load command for this:

docker load -i /tmp/base-image.tar

You should get output like the following

Loaded image: base-image:1.1.0

Use the docker image command to bring up the image you just loaded into your local environment

docker images

You should get output similar to the following

REPOSITORY                                      TAG       IMAGE ID       CREATED             SIZE
base-image                                      1.1.0     b5e88b0b4601   About an hour ago   22MB

This example was a straightforward demonstration, but it aimed to highlight that even without access to a registry, you can still transport your Docker images across systems.

Now, let's shift our focus to the more common methods of storing, sharing, and distributing your Docker images. We'll delve into using Docker Hub and other registries to streamline your workflow and collaborate effectively.

Storing and Deleting Images from Docker Hub

While Docker Hub offers a free tier for working with Docker images, it's important to note that you only get one private repository for free. If you need more than that, you'll have to opt for a paid monthly plan.

For most teams, a single private repository is rarely enough. So, if you're considering Docker Hub, be prepared to factor in the cost of a subscription if you need multiple private repositories.

However, if your needs are limited and you're okay with your images being public, then a free account gives you an unlimited number of public repositories to work with.

Storing Docker Images in Docker Hub

Alright, let's get your base-image image up on Docker Hub. We'll walk through creating a new repository, pushing your image to it, and then deleting the repository when we're done. This will give you a taste of how to manage your Docker images in the cloud.

To follow along with these Docker shenanigans, you'll need a Docker Hub account. Don't worry, I'm not made of money either, so we'll stick to the free stuff (unless you want to sponsor me, of course). If you haven't signed up yet, head over to Docker Signup and create your free account. Then come back here and let's get this Docker party started!

First things first, head over to your Docker Hub account. Once you're logged in, look for the "Repositories" section. You should see a blue button on the right side of the screen that says "Create Repository". Click that button to start setting up a new home for your base-image image.

When creating a new repository, you'll be presented with a page like the one that follows. Fill in the Name of the repository, which is usually the name of the image or service you are storing (in this case, basic-app). You also have the option to set the repository as Public or Private, and in this instance, select Public:

Once your new repository is created, it will provide details on how to start pushing your images to your new repository. Tag your image with /:tag to let Docker know where it will be pushing the image and which repository Docker will be pushing it to:

docker tag base-image cyberduckling/base-image:v1

Great, Docker is now aware of where to send your image. Before you can push the image, you'll need to log in to your Docker Hub account through the Docker CLI. Use the docker login command and enter the same username and password you used when you logged into Docker Hub earlier. Then, let's push it to Docker Hub using the following command:

docker push cyberduckling/base-image:v1

Alright! Now, if you head back to your Docker Hub account and look at your newly created base-image repository, you should see your image version (v1) listed there. It's officially up in the cloud and ready to be shared or used in your projects

If you look at the previous screenshot, you'll see a button on the right side of the screen labeled "Public View." This lets you preview exactly what the public will see when they search for your image on Docker Hub. Go ahead and click it! You should see a screen similar to this

This exercise demonstrates how Docker Hub can be a great way to share your Docker images with the world, making it easy for others to collaborate on your projects or build upon your work. While a public repository might not always be the perfect fit for a large company, it offers a similar platform for collaboration and code sharing as GitHub does for software development. By publishing your images on Docker Hub, you're contributing to a thriving community of developers and making your own projects more accessible and impactful.

The Docker Registry

Docker Registry is like a personal storage space for your Docker images. In most cases, these registries are kept private, only accessible by your team. There are many great options out there, and one of them is a ready-made registry image maintained by Docker itself.

There are many reasons why you might want to run your own Docker registry. Maybe you have concerns about security, want to keep your work under wraps until it's ready, or simply prefer the convenience of having your images right there on your own machine.

In this section, we'll guide you through setting up a Docker registry on your development environment and start storing your valuable images there.

Creating a Local Docker Registry

In this hands-on example, we'll create a private Docker registry that runs directly on your machine. Don't worry, this won't be accessible to your team or the public – it's just for you to experiment and decide if a local registry is the right fit for your needs. We'll even set up a custom domain name to make it feel like a real registry

To give your local registry a custom domain name, you'll need to edit your system's hosts file. Here's how you can find it:

Windows: C:\Windows\System32\drivers\etc\hosts
Linux/macOS: /etc/hosts

Open the hosts file in a text editor (you might need administrator privileges on Windows) and add the following line:

127.0.0.1    my-registry.local

This line tells your computer to resolve the domain name my-registry.local to the local IP address 127.0.0.1, which is where your registry will be running.

Pull the latest registry image down from Docker Hub

docker pull registry

Great, you've set up the domain! Now, let's fire up your local Docker registry using the following command:

docker run -d -p 8954:5000 --restart=always --name my-registry registry:2

I am using port 8954 because I have other things running on 5000, but feel free to use any port you like.

Run the docker ps command to see the registry container running on your system

docker ps

You should see an output similar to the following

CONTAINER ID   IMAGE        COMMAND                  CREATED          STATUS          PORTS                    NAMES
ba187cd38a11   registry:2   "/entrypoint.sh /etc…"   42 seconds ago   Up 41 seconds   0.0.0.0:8954->5000/tcp   my-registry

Now run the docker tag command to tag existing images with the registry hostname and port my-registry:8954

docker tag cyberduckling/base-image:v1 my-registry:8954/cyberduckling/base-image:v1

Delete the original images using the docker image remove command

docker image remove my-registry:8954/cyberduckling/base-image:v1

Finally, pull down the image from your local registry

ocker pull my-registry:8954/cyberduckling/base-image:v1

This wraps up our exploration of setting up a local Docker registry on your system. While this simple registry isn't officially supported, it provides a hands-on way to understand how registries work and how they can benefit your team. It's a great stepping stone for evaluating whether a private registry is the right solution for your needs.

If you're looking for a more comprehensive, supported option, Docker also offers Docker Trusted Registry, a commercial product designed to meet enterprise-level requirements for image management and security. This could be a good next step to consider if you need a more robust registry solution.

Summary

This post dives into the world of Docker image sharing, starting with a brief overview of Docker Hub's history and benefits. It then guides you through hands-on examples of how to move Docker images between machines, both manually (using a TAR file) and by leveraging Docker Hub's repository system. You'll learn how to create a repository on Docker Hub, push your images to it, and even view your published images from a public perspective.

But the fun doesn't stop there! The post also explores setting up a local Docker registry on your development machine, giving you a taste of how to create a private repository and manage your images in a controlled environment. This is a great option for teams or individuals who want more control over their image storage and security.

Whether you're a beginner looking to share your Docker creations or a seasoned pro exploring private registry options, this post offers valuable insights and practical examples to help you master Docker image management.

Docker Image Naming and Tagging

Kostas Kalafatis — Mon, 29 Jul 2024 06:00:00 +0000

As we get more acquainted with Docker, let's explore a little bit more what image tags are all about, even if we have discussed them before. All told, a tag is essentially a label you apply to your Docker image. Like a little sticky note informing you—and everyone else using the image—something significant about it, including what version it is or what type of software it contains. Consider it as a means of maintaining easily comprehensible and orderly Docker images.

Working on our Docker images, we had assumed that we are all by ourselves. That won't always be the case, though, particularly as our development team expands. The following post will help us to understand how best to tag and name our images for a team environment. Following these guidelines will help us to maintain our Docker images clear and understandable for every project participant.

There are two basic approaches for labeling and naming Docker images. When you build an image from a Dockerfile, you can either use the -t option, or the docker tag command. When you use the docker tag command, you specify the source repository name you are going to use as the base and the target name and tag you'll be creating

docker tag <source_repository_name>:<tag> <target_repository_name>:tag

When you use the docker build command, to name your image, you will use the Dockerfile as the source and then the -t option to name and tag your images

docker build -t <target_repository_name>:tag Dockerfile

Occasionally you may find a repository name beginning with a hostname. This is only a means of informing Docker where the repository resides online; and it is absolutely optional. In future posts in this series we are going to create our own Docker Registry, and see how this works. If you're uploading your images to Docker Hub, though, you'll definitely need to add your Docker Hub username to the beginning of the repository name, like this:

docker built -t <dockerhub_user>/<target_repository_name>:tag Dockerfile

In the following example, we are going through the process of tagging Docker images.

Tagging Docker Images

In this example, we are going to work to use a new image, using the lightweight busybox image to demonstrate the process of tagging and start to implement tags in our projects. BusyBox is used to combine mini versions of many common UNIX utilities into a single small executable.

Run the docker rmi command to clear up the images you currently have on your system, so you don't get confused with a large number of images around:

docker rmi -f $(docker images -a -q)

On the command line, run the docker pull command to download the latest busybox container

docker pull busybox

Run the docker images command

docker images

This will give us the information we need to start putting some tag commands together

REPOSITORY                                      TAG       IMAGE ID       CREATED         SIZE
busybox                                         latest    65ad0d468eb1   13 months ago   4.26MB

Okay, let's give your image a name and tag it so it's easier to find and work with later. We can do this using either the image's unique ID or its repository name.

For now, we'll use the image ID. Keep in mind that your image ID will be different from the one I use in the example.

docker tag 65ad0d468eb1 mybusybox:v1

This command will create a new tag called v1 for your image in the mybusybox repository. Now you can refer to your image using either its ID or the more human-readable name mybusybox:v1.

Now that we've tagged our image, let's create a new repository using your name and a new version tag. Since we've already tagged the image as mybusybox:v1, we can use that to create a new tag with the updated version.

docker tag mybusybox:v1 kalkost/busybox:v1

Run the docker images command

docker images

You should see an output similar to the following

kalkost/busybox                                 v1        65ad0d468eb1   13 months ago   4.26MB
busybox                                         latest    65ad0d468eb1   13 months ago   4.26MB
mybusybox                                       v1        65ad0d468eb1   13 months ago   4.26MB

Let's put together a Dockerfile to create a basic image, using the mybusybox image we tagged earlier. We'll also include a tag for the new image we're creating, since Docker might try to use the latest tag if we don't specify one, and that could cause errors if it doesn't exist.

echo "FROM mybusybox:v1" > Dockerfile

Run the docker build command to create the image while naming and tagging it at the same time

docker build it built_image:v1.1.1 .

Note that when I tried the above command, the Dockerfile was created with a UTF-16 encoding causing the very user friendly ERROR: failed to solve: Internal: Internal: Internal: stream terminated by RST_STREAM with error code: INTERNAL_ERROR error to appear. If you encounter this issue, then open the Dockerfile with any editor of your choosing and change the encoding to UTF-8

Now, run the docker images command to see the four images available on your system

docker images

You should see something similar to the following

REPOSITORY                                      TAG       IMAGE ID       CREATED         SIZE
built-image                                     v1.1.1    0534e936c658   13 months ago   4.26MB
busybox                                         latest    65ad0d468eb1   13 months ago   4.26MB
mybusybox                                       v1        65ad0d468eb1   13 months ago   4.26MB
kalkost/busybox                                 v1        65ad0d468eb1   13 months ago   4.26MB

Tagging your Docker images with a meaningful version number that makes sense for your team or company is actually pretty simple, especially once you've done it a few times. The tips we've covered here will help you avoid the default latest tag, which can cause problems down the line. In the next section, we'll dive into those potential issues and why using specific tags is a smart move.

Using the Latest Tag in Docker

As we've been working with tags, I've mentioned a few times to avoid using the default latest tag. It might seem convenient, but it can actually cause a lot of headaches, especially when you're deploying images to production. Let's dive into why it's best to steer clear of latest and use specific tags instead.

The first thing to understand is that the latest tag is just a regular tag, like the v1 we used earlier. It doesn't automatically mean the newest version of your code. It actually refers to the most recent image you built without specifying a tag.

Here's the thing about using the latest tag: it can really mess things up when you have a big team constantly deploying updates to different environments. It's like trying to keep track of a moving target – you never quite know which version you're actually dealing with. And if something goes wrong, good luck rolling back to a previous version without a clear history.

Remember, Docker doesn't automatically pull the most recent image when you use latest. It just grabs the most recent one you've built or pulled without specifying a tag. This can lead to all sorts of confusion and unexpected behavior, especially in production.

Issues when Using Latest

Since you might still getting familiar with Docker and tagging, you might not have run into any problems using the latest tag yet. But don't worry, this next exercise will give you a firsthand look at how latest can throw a wrench in your development process. You'll understand why it's best to avoid it and start using specific tags instead.

We're going to build on the previous Dockerfile in this example and explore the potential pitfalls of relying on the latest tag. Get ready to see the latest tag in action (or inaction, as the case may be).

Let's update the that Dockerfile we created to include a simple version script.

FROM mybusybox:v1

RUN touch /version.sh && \ 
    echo '#!/bin/sh' >> /version.sh && \ 
    echo 'echo "Version: 1.0"' >> /version.sh && \ 
    chmod +x /version.sh

ENTRYPOINT ["sh", "/version.sh"]

Build the image and name it whatever you want

docker build -t kalkost/test .

Run the image using the docker run command

docker run kalkost/test

You should see the output of the version.sh script

Version: 1.0

Use the docker tag command to tag this image as v1

docker tag kalkost/test kalkost/test:version1

Change the value of the version of the Dockerfile in line 5

FROM mybusybox:v1

RUN touch /version.sh && \
    echo '#!/bin/sh' >> /version.sh && \
    echo 'echo "Version: 2.0"' >> /version.sh && \
    chmod +x /version.sh

ENTRYPOINT ["sh", "/version.sh"]

Build the amended Dockerfile and tag it with v2

docker build -t kalkost/test:v2 .

Run the amended image using the docker run command

docker run kalkost/test

Now you might think that you are going to see the version changed, but you will see the following

Version: 1.0

Hold on a second! This isn't the version we expected, right? It seems Docker pulled the most recent version of the image that was tagged with latest instead of the one we just built with the 1.0 tag in step 3. This is a perfect example of how using the latest tag can lead to unexpected results and confusion.

Now, run both images with latest and v2 tags

docker run kalkost/test:latest
Version: 1.0

And

docker run kalkost/test:v2
Version: 2.0

You probably guessed it - to run the updated version of the code with the version script, we need to specify the 1.0 tag that we used when building the image. If you don't, Docker defaults to the most recent image with the latest tag, which might not be what you want.

This example clearly shows how using the latest tag can create confusion, especially in a team setting where multiple developers are pushing images to a shared repository. It's like playing a guessing game with which version is actually running in production.

Things can get even messier if you're using orchestration tools (like Kubernetes) and pulling the latest image. You could end up with a mix of different versions running in your production environment, which is a recipe for unexpected behavior and potential bugs.

The key takeaway here is that using specific tags is crucial for maintaining control and predictability in your Docker workflow. It ensures that everyone is on the same page and helps avoid those frustrating latest tag surprises.

Docker Image Tagging Policies

As development teams grow and projects become more complex, having a standard tagging policy for your Docker images becomes increasingly important. Without a clear and consistent approach to tagging, you risk running into the kind of confusion and problems we've explored in earlier sections.

Deciding on a tagging policy early in the game is a smart move to avoid these headaches. There's no one-size-fits-all answer, but the key is to choose a policy that works for your team and stick to it. This section will cover some different tagging strategies you can use, along with examples of how to implement them. Remember, the goal is to find a policy that everyone on your team agrees on and can easily follow.

Semantic versioning is a popular and reliable system that can also be used for tagging your Docker images. If you haven't heard of it before, it's a simple three-part number system (major.minor.patch). For example, a version like 2.1.0 would indicate a major release of version 2, a minor release of 1, and no patches. It's a great way to keep track of your image versions, especially in automated build environments where it can be easily automated.

Another option is to use a hash value, such as the git commit hash for your code. This allows you to easily link the image tag back to the specific code changes in your repository, making it super easy for anyone to see what's been updated.

Using a date value is another simple option that can be automated. It gives you a clear timestamp for each image, which can be helpful for tracking and troubleshooting.

The key takeaway here is that automating your tagging policy is the best way to ensure consistency, clarity, and adherence across your team.

Automating Image Tagging

Alright, let's explore how we can automate tagging your Docker images. This will minimize manual intervention and streamline your workflow. We'll continue working with the familiar base-image for this example.

Create a base-image using the following Dockerfile

FROM alpine:latest

RUN apk upgrade --no-cache && apk update && apk add wget curl

ARG VERSION=0.0.0

# Copy the version.txt file to the image
COPY version.txt /version.txt

# Command to read and display the version
CMD ["cat", "/version.txt"]

Create a version.txt file with the following content

1.0.0

Now, create the following build.sh

#!/bin/bash

# Read current version from version.txt
current_version=$(cat version.txt)

# Extract the major and minor versions, set patch to 0
major_version=$(echo $current_version | awk -F. '{print $1}')
minor_version=$(echo $current_version | awk -F. '{print $2}')
new_minor_version=$((minor_version + 1))
new_version="$major_version.$new_minor_version.0"

# Update version.txt with the new version
echo $new_version > version.txt

# Build the Docker image with the new version as a build argument
docker build -t base-image:$new_version --build-arg VERSION=$new_version .

Let's break down what the script does

#!/bin/bash

# Read current version from version.txt
current_version=$(cat version.txt)

This part reads the current version number from a file named version.txt and stores it in the variable current_version. In a more real-life setting you would probably use Git to update the versions. If you are using Git, change the above command with the following

# Get the current version from Git tags 
current_version=$(git describe --tags --abbrev=0)

# Extract the major and minor versions, set patch to 0
major_version=$(echo $current_version | awk -F. '{print $1}')
minor_version=$(echo $current_version | awk -F. '{print $2}')
new_minor_version=$((minor_version + 1))
new_version="$major_version.$new_minor_version.0"

These lines use awk to parse the current_version string (which is assumed to be in MAJOR.MINOR.PATCH format, e.g., 1.2.3) and extract the MAJOR and MINOR version components into variables major_version and minor_version respectively. Then it increments the MINOR version by 1 to get the new MINOR version number.

Finally the script constructs the new_version. new_version is constructed by combining the major_version, incremented minor_version, and setting the PATCH to 0. This forms a new version number, e.g., if current_version is 1.2.3, then new_version will become 1.3.0.

# Update version.txt with the new version
echo $new_version > version.txt

This line updates the version.txt file with the newly created new_version. It overwrites the existing version number stored in version.txt with the incremented version.

# Build the Docker image with the new version as a build argument
docker build -t base-image:$new_version --build-arg VERSION=$new_version .

Finally, this command builds a Docker image named base-image with the tag $new_version (e.g., base-image:1.3.0). The --build-arg VERSION=$new_version option passes the new_version as a build argument to the Docker build process. The . at the end specifies the current directory as the build context.

Make sure that you have permissions to run the script using the chmod command

chmod +x build.sh

Now, if you run build.sh you will see that the script calculates the new version of the image and builds the Docker image

./build.sh

And the output should be something similar to the following

#0 building with "default" instance using docker driver

#1 [internal] load build definition from Dockerfile
#1 transferring dockerfile: 120B done
#1 DONE 0.0s

#2 [internal] load .dockerignore
#2 transferring context: 2B done
#2 DONE 0.0s

#3 [internal] load metadata for docker.io/library/alpine:latest
#3 ...

#4 [auth] library/alpine:pull token for registry-1.docker.io
#4 DONE 0.0s

#3 [internal] load metadata for docker.io/library/alpine:latest
#3 DONE 4.1s

#5 [1/2] FROM docker.io/library/alpine:latest@sha256:b89d9c93e9ed3597455c90a0b88a8bbb5cb7188438f70953fede212a0c4394e0
#5 DONE 0.0s

#6 [2/2] RUN apk upgrade --no-cache && apk update && apk add wget curl
#6 CACHED

#7 exporting to image
#7 exporting layers done
#7 writing image sha256:b5e88b0b4601183bb9425186cd95fee52e22be5f1ec8c58b95a50f42e78c2004 done
#7 naming to docker.io/library/base-image:1.1.0 done
#7 DONE 0.0s

View the image using the docker images command

docker images

It should reflect the name and tags created as part of the build script

REPOSITORY                                      TAG       IMAGE ID       CREATED          SIZE
base-image                                      1.1.0     b5e88b0b4601   29 minutes ago   22MB

Conclusion

We've discussed the importance of image tags in Docker and provided guidelines for tagging and naming them, especially in a team environment. We saw two approaches for labeling and naming Docker images: using the -t option or the docker tag command.

We also discussed examples of tagging Docker images and highlighted the potential issues with using the latest tag. We then saw the need for a clear tagging policy and suggested using semantic versioning, hash values, or date values for image tags.

Finally, we discussed the importance of automating the tagging policy to ensure consistency and clarity.

Dockerfile Best Practices Explained - Interlude Post 2

Kostas Kalafatis — Wed, 24 Jul 2024 06:00:00 +0000

In this post, we'll dive into the best practices for writing Dockerfiles that will help you create efficient, secure, and maintainable Docker images while reducing build times.

Using an Appropriate Parent Image

One of the most important things you can do to build efficient Docker images is to choose the right base image. It's always a good idea to start with official images from Docker Hub as your foundation when building custom images. These official images are well-maintained, follow best practices, have comprehensive documentation, and receive regular security updates.

For example, if your application needs the Java Development Kit (JDK), it's more efficient and secure to use an official Docker image like eclipse-temurin instead of starting from scratch with a generic ubuntu image and manually installing the JDK. This way, you can leverage the work that has already been done to ensure the JDK is properly configured and secure within the Docker environment.

Inefficient Dockerfile	Efficient Dockerfile
FROM ubuntu RUN apt-get update && apt-get install -y openjdk-11-jdk	FROM eclipse-temurin

Secondly, it's crucial to avoid using the latest tag when specifying the parent image in your Dockerfile, especially for production environments. The latest tag is a moving target – it always points to the most recent version of the image on Docker Hub. However, as newer versions are released, the latest tag gets updated, and those updates might not be compatible with your application. This can result in unexpected errors and failures in your production environment.

To prevent this, always use a specific versioned tag when referring to the parent image in your Dockerfile. This ensures that you're building on a stable foundation and avoids the risk of unintended changes caused by automatic updates to the latest tag.

Inefficient Dockerfile	Efficient Dockerfile
FROM eclipse-temurin:latest	FROM eclipse-temurin:8u412-b08-jre-windowsservercore-ltsc2022

And finally, to really slim down your Docker images, it's a good idea to use the minimal version of your base image whenever possible. Many official Docker images on Docker Hub offer a "slim" or "alpine" variant that's specifically designed to be lightweight.

These minimal versions are often based on Alpine Linux, a stripped-down distribution known for its small size and security focus. By using a minimal base image, you can significantly reduce the overall size of your final image, making it faster to build, push, pull, and deploy.

Inefficient Dockerfile	Efficient Dockerfile
FROM eclipse-temurin:8u412-b08-jre-windowsservercore-ltsc2022	FROM eclipse-temurin:8u412-b08-jre-alpine

The eclipse-temurin:8u412-b08-jre-alpine image will be only 51.27 MB in size, whereas eclipse-temurin:8u412-b08-jre-windowsservercore-ltsc2022 will be 2.06 GB in size.

Using a Non-Root User for Better Security

By default, Docker containers run with the power of the root user. This lets them do all sorts of things, like tweaking system settings, installing software, or using special network ports. But here's the problem: that kind of power can be dangerous. If someone manages to hack into your application running inside the container, they might also gain root access to your whole system—not good!

That's why it's considered a security best practice to run containers as a regular, non-root user whenever possible. This follows the principle of least privilege, which means only giving the application the bare minimum permissions it needs to do its job. It's like locking up your valuables when you're not using them—better safe than sorry!

There are two main ways to achieve this in Docker:

The --user (or -u) flag: You can use this flag when running a container with docker run to specify a different user ID (UID) or username to use inside the container.
The USER directive: You can add this directive to your Dockerfile to set the default user for any containers created from that image.

By using either of these methods, you can significantly reduce the risk of a security breach in case your container is compromised.

If you want to switch things up and run a Docker container as a different user than the default root user, you can use the --user (or its shorthand -u) flag when you start the container. This flag gives you the flexibility to specify either the username or the user ID (UID) of the user you want the container to run as.

docker run --user=1234 ubuntu:focal

In the previous command, we set the user ID to 1234 when starting the Docker container. If we're using a user ID instead of a username, the actual user with that ID doesn't need to exist inside the container.

Alternatively, you can use the USER directive directly in your Dockerfile to specify the default user for your containers. This can be convenient, but keep in mind that you can always override this default setting later on by using the --user flag when you run the container.

FROM ubuntu:focal

RUN apt-get update
RUN useradd demo-user

USER demo-user

CMD whoami

In the previous example, the USER directive was added to the Dockerfile to set the default user as demo-user. This indicates that any commands that follow the USER directive in the Dockerfile will be executed as if the demo-user is running them.

Using .dockerignore

The .dockerignore file is like a "do not disturb" list for Docker. It's a special text file you create in the same folder as your Dockerfile. Inside this file, you list all the files and folders you don't want Docker to include when it's building your image.

Why is this important? Well, when you run docker build, Docker first gathers everything in your project folder (called the "build context") and packages it up into a TAR archive. It then sends this archive to the Docker daemon, which actually builds the image.

The first line you usually see in the docker build output is "Sending build context to Docker daemon." This means Docker is uploading that whole package of files. If your project has a lot of files that aren't actually needed to build the image (like temporary files, logs, or large binaries), this can make the build process unnecessarily slow and bloat the size of your final image.

Sending build content to Docker daemon 18.6MB
Step 1/5 : FROM ubuntu:focal

Whenever you build a Docker image, the entire build context—that's everything in your project's directory—gets sent over to the Docker daemon. This can take time and bandwidth, so it's smart to streamline things by excluding unnecessary files.

That's where the .dockerignore file comes in handy. It acts as a filter, telling Docker which files and folders to leave behind when packaging your build context. Not only does this save time and bandwidth, but it also helps you keep sensitive information like passwords and keys out of your Docker images.

Creating a .dockerignore file is easy – just place it in the root directory of your project (the same level as your Dockerfile). Docker will automatically look for it and use its instructions to exclude unwanted files when building the image.

The following is the content of a sample .dockerignore file:

# Ignore the node_modules 
directory node_modules 

# Ignore log files 
*.log 

# Ignore test files 
**/*.test.js 

# Ignore configuration files 
.env 

# Ignore hidden files 
.* 

# Ignore build directories 
build/ 

# Ignore the Dockerfile itself (optional, but good practice) 
Dockerfile* 

# Ignore specific files 
temp.txt

This example .dockerignore file excludes:

The node_modules directory (commonly used for Node.js dependencies).
Any file ending in .log (log files).
Any JavaScript file ending in .test.js (test files).
The .env file (environment variables).
All hidden files (files starting with a dot).
The build/ directory.
Any Dockerfile (optional, but recommended to prevent accidental inclusion).
The file temp.txt.

Minimizing Layers

Every line in your Dockerfile translates to a new layer in your Docker image, and each layer adds to the overall size. To keep things lean, it's a good idea to create as few layers as possible. One way to do this is by combining multiple RUN instructions whenever you can.

Let's take a look at an example Dockerfile that installs redis-server and nginx:

FROM ubuntu:latest
RUN apt-get update
RUN apt-get install -y nginx
RUN apt-get install -y redis-server

In this case, we have three separate RUN instructions: one to update the package list, one to install nginx and another to install redis-server. This creates three separate layers in the image.

To optimize this, we can combine the three RUN instructions into one:

FROM ubuntu:latest
RUN apt-get update && apt-get install -y redis-server nginx

By using the && operator, we tell Docker to execute all three commands in a single layer, reducing the overall size of the image. This is a simple yet effective way to make your Docker images more efficient.

Don't Install Unnecessary Tools

Avoiding unnecessary tools and dependencies is a key strategy for creating efficient Docker images. For example, if you're building an image for a production environment, you likely don't need debugging tools like vim, curl, or telnet. By leaving these out, you can significantly reduce the size of your image.

Similarly, be mindful of the dependencies that your package manager installs. Some package managers, such as apt, will automatically install additional "recommended" or "suggested" packages along with the ones you explicitly request. This can bloat your image unnecessarily. To prevent this, you can use the --no-install-recommends flag with the apt-get install command. This tells apt to install only the essential packages you need and skip any optional extras.

FROM ubuntu:latest
RUN apt-get update && apt-get install --no-install-recommends -y nginx

In the previous example, using the --no-install-recommends flag when installing the nginx package already helped us trim down the image size by about 10MB. But we can squeeze even more savings out of it!

Besides avoiding recommended packages, we can also clear out the apt package manager's cache to further shrink the final Docker image. This cache stores information about available packages, and while it's helpful during installation, it's not needed once the package is installed.

To remove this cache, we can simply add a command to the end of our RUN instruction:

FROM ubuntu:latest
RUN apt-get update && apt-get install -y --no-install-recommends nginx && rm -rf /var/lib/apt/lists/*

The rm -rf /var/lib/apt/lists/* part removes the cache directory and its contents. By doing this, we're getting rid of files that take up space but aren't essential for the image to function.

Creating Base Docker Images

Kostas Kalafatis — Mon, 22 Jul 2024 06:00:00 +0000

Creating your base Docker image is quite straightforward. Just as we used the docker commit command previously to create an image from a running container, we can also create an image from a system or server where our applications are originally hosted. However, it's crucial to keep the base image small and lightweight. Simply moving existing applications from traditional servers to Docker isn't enough.

If you're using a production server, the resulting image could end up quite large. For smaller virtual machines that seem ideal for a base image, you can follow these steps to create one. Similar to the docker commit command, this method works for any system you have access to.

Creating Custom Base Docker Images

Let's take the Docker image we're already using (basic-server) and transform it into a custom base image. This is a simple demo, but the same process works for bigger, more complex setups.

First, we need to start the container and connect to it at the same time. Here's how

docker run -it basic-server sh

Use the docker image save command to create a backup of the repository:

docker save -o basebackup.tar basic-server

Create a new image using the docker load command.

docker load --input .\basebackup.tar

This example has demonstrated how to generate a base image from an existing running system or environment. However, if your goal is to create a minimal base image, the next section will explain how to utilize the scratch image.

The SCRATCH Image

The scratch image is Docker's secret weapon for building incredibly lean and efficient images. It's essentially a blank slate, devoid of any operating system or pre-installed software, making it the perfect starting point for running compiled binary applications like those written in Java or C++. These binaries are self-contained and don't rely on external libraries or dependencies, allowing them to run directly on the minimal environment provided by the scratch image.

By using the FROM scratch directive in your Dockerfile, you signal to Docker that you want to build an image with the absolute minimum footprint. This results in a container image that is remarkably small and lightweight, as it doesn't include any unnecessary overhead from an operating system or additional software layers.

Using the SCRATCH Image

In this example, we'll craft a simple C application to showcase how to build one of the most minimal Docker base images possible. While prior knowledge of C programming isn't required, this hands-on experience will demonstrate the process of creating a highly efficient and compact image using Docker's "scratch" image as our foundation. By leveraging the scratch image, we can eliminate any unnecessary dependencies and produce an image with a remarkably small footprint. Let's get started.

Try getting the scratch image using the docker pull command

docker pull scratch

You will notice that you will get an error

Using default tag: latest
Error response from daemon: 'scratch' is a reserved name

Create a C program that we will build into the image to use in our Dockerfile.

code christmas-tree.c

Add the following C code:

#include <stdio.h>

int main() {
    int height = 12;
    int i, j, spaces;

    for (i = 1; i <= height; i++) {
        for (spaces = height - i; spaces > 0; spaces--) {
            printf(" ");
        }

        for (j = 1; j <= 2 * i - 1; j++) {
            printf("*");
        }
        printf("\n");
    }

    return 0;
}

This C program crafts a Christmas tree composed of asterisks based on user-specified height. It employs a series of nested loops to achieve this. The outer loop controls the number of rows, starting from the top of the tree and iterating downwards. For each row, an inner loop determines the number of spaces preceding the asterisks, ensuring the tree's triangular shape.

Another inner loop handles the actual printing of asterisks, with the number of asterisks increasing by two with each descending row. This creates the tree's wider base. A newline character is printed after each row, moving the cursor to the next line for the subsequent row's output. The result is a beautifully constructed Christmas tree in the console, ready to spread festive cheer!

Build the image form the command line by running the following command to build the C program

gcc -o christmas-tree christmas-tree.c

Now lets create the Dockerfile. The Dockerfile will be pretty minimal but needs to start with FROM scratch. The rest of the file will add the C program and then run it

FROM scratch

ADD christmas-tree.exe /
CMD ["/christmas-tree.exe"]

Build a new image. In this instance, call the image scratch-test using the following command

docker build -t scratch-test .

Run the image from the command line

docker run scratch-test

Run the docker images command for your new image

docker images scratch-test

This will show some pretty impressive results with a size of 41.2kB in size

REPOSITORY     TAG       IMAGE ID       CREATED         SIZE
scratch-test   latest    817c50b9dbe6   3 minutes ago   41.2kB

View the layers of the image using the docker history command

docker history scratch-test

You will see a similar output to the following one

IMAGE          CREATED         CREATED BY                            SIZE      COMMENT
817c50b9dbe6   5 minutes ago   CMD ["./christmas-tree.exe"]          0B        buildkit.dockerfile.v0
<missing>      5 minutes ago   CMD ["ls"]                            0B        buildkit.dockerfile.v0
<missing>      5 minutes ago   ADD christmas-tree.exe / # buildkit   41.2kB    buildkit.dockerfile.v0

Alright, the scratch image we whipped up in this exercise shows how to make a Docker image that's both super-lean and still gets the job done. It also proves that if you put some thought into your goals, you can easily speed up your builds and shrink those images down to size.

Conclusion

Creating a base Docker image is a straightforward process, but it's important to keep the image small and lightweight. You can create a custom base image from an existing running system or environment using the docker save and docker load commands.

Additionally, Docker's "scratch" image is a powerful tool for building minimal and efficient images, especially for compiled binary applications. By using the FROM scratch directive in your Dockerfile, you can create a container image with the absolute minimum footprint. This results in a remarkably small and lightweight image that eliminates unnecessary dependencies.

Overall, by following these techniques, you can optimize your Docker images for better performance and efficiency.

In the next post, let's take a breather from building for a moment and dive into the world of naming and tagging our Docker creations. After all, organizing our work is just as important as creating it

Understanding CMD, ENTRYPOINT and RUN in Docker - An Interlude Post

Kostas Kalafatis — Fri, 19 Jul 2024 10:46:43 +0000

Disclaimer: This post is heavily inspired by this post on the Docker Blog. It is also seeks to answer questions raised by the community in previous posts in this series.

Docker is undeniably a powerful tool for containerization, but its versatility and extensive feature set can be overwhelming, especially for newcomers. The platform offers multiple ways to achieve similar goals, making it crucial for users to carefully weigh the advantages and disadvantages of each option to determine the best approach for their specific projects. This decision-making process requires a solid understanding of Docker's underlying mechanisms and the trade-offs involved in different strategies.

Docker can be a bit of a maze, especially when it comes to the nitty-gritty of instructions like RUN, CMD, and ENTRYPOINT. These three instructions often cause confusion, as they seem to have overlapping functions. But fear not! In this article, we'll unravel the mystery, clearly explaining the differences between them and highlighting when to use each one effectively. By the end, you'll be a Dockerfile ninja, wielding these instructions with confidence and precision!

The RUN Directive

In Docker, the RUN directive is a key instruction used in Dockerfiles to run commands while building your Docker image. It's your go-to tool for setting up the environment inside the image, allowing you to install packages, update dependencies, configure settings, and basically do anything you need to make your image ready to use. Think of it as the stage where you get everything in place before the curtain goes up on your running container.

Here is an example of the RUN directive in practice:

FROM ubuntu:20.04

# Update package lists and install necessary packages
RUN apt-get update && apt-get install -y \
    curl \
    wget \
    && rm -rf /var/lib/apt/lists/*

In this example, we use the RUN instruction to update the package lists on your system and install curl and wget. These are handy tools for downloading files and interacting with websites from the command line.

The && rm -rf /var/lib/apt/lists/* part is added for cleanup. It removes the cached package lists after the installation, making your final Docker image smaller and leaner. Since these lists aren't needed to run your application, it's good practice to get rid of them.

The CMD Directive

The CMD instruction in your Dockerfile sets the default command that will run when you start a container from your image. It's like a pre-set option for what your container should do when it "boots up." But here's the key: you can easily change this default behavior by providing different command-line arguments when you run the container using docker run.

CMD is perfect for situations where you want to provide a sensible default behavior for your container, but also give users the flexibility to customize it. Think of it as setting up a suggested starting point, but allowing users to take the wheel if they want to go in a different direction.

It's a common practice to use CMD in Docker images to define default parameters or configurations that can be easily overridden by the user when running the container.

For example, by default, you might want to start a web server to start, but also allow users to override this and run a shell instead:

FROM node:alpine

[...dockerfile truncated...]

CMD ["node", "server.js"]

The ENTRYPOINT Directive

The ENTRYPOINT instruction sets the main executable for your container. It's similar to CMD but with a key difference: when you run a container with docker run, the command you provide doesn't replace the ENTRYPOINT command. Instead, your command gets added to the ENTRYPOINT command as arguments.

Think of it like this: ENTRYPOINT sets the core command your container is designed to run, while any arguments you provide with docker run become extra instructions for that command.

For example, the following Dockerfile will run the webserver no matter what the users provide as arguments:

FROM node:alpine

[...dockerfile truncated...]

ENTRYPOINT ["node", "server.js"]

So CMD or ENTRYPOINT?

CMD and ENTRYPOINT are special instructions in a Dockerfile. While other instructions execute when you build the image, these two come into play when you actually run a container from that image.

Essentially, when a container starts, Docker needs to know what it should do – what program to run, and how to run it. That's where CMD and ENTRYPOINT step in. They tell Docker what the main process inside the container is and how to start it.

Now, the difference between them is a bit tricky, and many people don't fully grasp it. Luckily, most of the time, your container will work fine even if you don't use them perfectly. However, understanding the nuances can make things a lot smoother and less confusing.

To get a better handle on this, let's break down a typical Linux command:

ping -c 10 127.0.0.1

Here, ping is the command itself, and the rest (-c 10 127.0.0.1) are the parameters or arguments we're giving to that command.

Now, back to Docker:

ENTRYPOINT: This is where you define the command part of the expression. It's the core thing you want your container to do when it starts.
CMD: This is where you define the parameters for that command. They're the additional instructions you want to give it.

So, a Dockerfile that uses Alpine Linux as the base image and wants to run the ping command could look like this:

FROM alpine:latest

ENTRYPOINT ["ping"]
CMD ["-c", "10", "127.0.0.1"]

We can now build an image called pinger from the preceding Dockerfile, as follows:

docker image build -t pinger .

Now we can run a container from the pinger image we just created like this:

docker container run --rm -it pinger

PING 127.0.0.1 (127.0.0.1): 56 data bytes
64 bytes from 127.0.0.1: seq=0 ttl=64 time=0.047 ms
64 bytes from 127.0.0.1: seq=1 ttl=64 time=0.056 ms
64 bytes from 127.0.0.1: seq=2 ttl=64 time=0.038 ms
64 bytes from 127.0.0.1: seq=3 ttl=64 time=0.055 ms
64 bytes from 127.0.0.1: seq=4 ttl=64 time=0.037 ms
64 bytes from 127.0.0.1: seq=5 ttl=64 time=0.036 ms
64 bytes from 127.0.0.1: seq=6 ttl=64 time=0.053 ms
64 bytes from 127.0.0.1: seq=7 ttl=64 time=0.058 ms
64 bytes from 127.0.0.1: seq=8 ttl=64 time=0.048 ms
64 bytes from 127.0.0.1: seq=9 ttl=64 time=0.053 ms

--- 127.0.0.1 ping statistics ---
10 packets transmitted, 10 packets received, 0% packet loss
round-trip min/avg/max = 0.036/0.048/0.058 ms

The great thing about this setup is that you can easily override the default CMD parameters you've set in the Dockerfile. If you remember, we originally defined CMD ["-c", "10", "127.0.0.1"] to ping a specific address three times.

But now, when you create a new container, you can simply add different values at the end of your docker run command to change the ping target or the number of pings. This gives you a lot of flexibility while still keeping a consistent base command.

docker container run --rm -it pinger -w 5 127.0.0.1

PING 127.0.0.1 (127.0.0.1): 56 data bytes
64 bytes from 127.0.0.1: seq=0 ttl=64 time=0.034 ms
64 bytes from 127.0.0.1: seq=1 ttl=64 time=0.053 ms
64 bytes from 127.0.0.1: seq=2 ttl=64 time=0.040 ms
64 bytes from 127.0.0.1: seq=3 ttl=64 time=0.038 ms
64 bytes from 127.0.0.1: seq=4 ttl=64 time=0.056 ms

This will cause the container to ping the loopback IP address (127.0.0.1) for 5 seconds.

Directive Description and Use Cases

The following table provides an overview of these commands and use cases.

Directive	Description	Syntax Example	Use Cases
`RUN`	Executes commands during the image build process. It is used to install software packages, configure settings, or perform other setup tasks. The result is part of the image layers.	`RUN apt-get update && apt-get install -y nginx`	- Installing software packages - Configuring files or repositories - Setting up environment variables or system settings
`CMD`	Specifies the default command to run when a container starts from the image. It can be overridden by providing a command in `docker run`. It is generally used to provide default arguments to `ENTRYPOINT` or to set a default command.	`CMD ["nginx", "-g", "daemon off;"]`	- Setting a default command for the container - Providing default arguments to `ENTRYPOINT` - Running a service or application by default
`ENTRYPOINT`	Defines the main command that is always executed when a container starts. It is not overridden by arguments provided in `docker run` unless you use the `--entrypoint` flag. Useful for ensuring a specific command is always run.	`ENTRYPOINT ["nginx", "-g", "daemon off;"]`	Ensuring a specific command is always executed - Running a primary process or service - Used in combination with `CMD` to provide default arguments

Unix Signaling and the PID 1

In the world of Unix-like systems, including Docker containers, PID 1 is a special process – the very first one that starts up. Every other process inside the system is its child, creating a family tree of processes with PID 1 at the top.

In Docker, this PID 1 process is really important because it's responsible for managing everything else inside the container. One of its critical roles is handling signals (like SIGTERM) from the host system, which are essentially messages that tell the container to do something (like gracefully shut down).

Now, when you use the shell form for a Docker command, a shell process (usually /bin/sh -c) takes over as PID 1. The problem is that this shell process isn't very good at handling signals. It might not pass them along to your actual application, which can lead to issues like containers not shutting down cleanly.

On the other hand, the exec form lets you run your command directly as PID 1, without any shell in between. This way, the command itself receives and handles signals directly, making the whole thing much more reliable.

So, if your container needs to react to signals promptly and gracefully, the exec form is the way to go. It's particularly crucial for applications that need to respond to events or interruptions, ensuring that everything shuts down properly and your data stays safe.

Shell and exec forms

When you're working with Dockerfiles and setting commands for RUN, CMD, and ENTRYPOINT, you have two ways to do it: the shell form and the exec form. Each has its own strengths and weaknesses, and understanding the difference is key to effectively managing your Docker containers.

Shell Form

The shell form is like writing commands the old-school way, similar to what you'd type in a terminal. When Docker sees a command in shell form, it essentially runs it through a shell (usually /bin/sh -c for Unix-based images). This means the command gets interpreted by the shell, which unlocks some handy features like using environment variables and chaining commands together.

CMD echo "Hello, World!"

This way of writing commands, using the shell form, means that Docker executes the command echo "Hello, World!" through the default shell of your container. This can be handy for simple commands or when you need to use special shell features. However, it has a drawback: it doesn't handle signals (like those used for managing processes) the same way as the exec form.

In other words, if your command needs to respond properly to Unix signals for things like stopping or restarting processes gracefully, the shell form might not be the best choice. It's a bit like having a middleman who might not deliver your messages as reliably as you'd like.

Exec Form

Now, let's talk about the exec form. It's a more specific and reliable way to define commands in your Dockerfile. Instead of writing the whole command as a single line, you break it up into a JSON array, where each part of the command (the command itself and its arguments) is a separate element in the array.

CMD ["echo", "Hello, World!"]

Here, instead of using a shell as a middleman, Docker directly executes the command echo with the argument "Hello, World!". This direct execution method is more reliable because it doesn't depend on the shell's interpretation.

This has a few key advantages:

Better Signal Handling: The command itself can receive and respond to Unix signals directly. This is crucial for ensuring your application can gracefully shut down or restart when needed.
Precise Argument Passing: You can be confident that the arguments you provide are passed directly to your command without any potential modifications or interference from the shell.

Key Differences between Shell and Exec

	Shell Form	Exec Form
Form	Commands without square brackets (`[]`). Run by the container's shell	Commands with square brackets (`[]`). Run directly, not through a shell
Variable Substitution	Inherits environment variables from the shell, such as `$HOME` and `$PATH`	Does not inherit shell environment variables but behaves the same for `ENV` instruction variables
Shell Features	Supports sub-commands, piping output, chaining commands, I/O redirection, etc.	Does not support shell features
Signal Trapping & Forwarding	Most shells do not forward process signals to child processes.	Directly traps and forwards signals like `SIGINT`
Usage with ENTRYPOINT	Can cause issues with signal forwarding	Recommended due to better signal handling
Usage as ENTRYPOINT Params	Not possible with the shell form	If the first item in the array is not a command, all items are used as parameters for the `ENTRYPOINT`

The diagram below, will help you decide if you need RUN, CMD or ENTRYPOINT in your Dockerfile.

The diagram below, will help you decide if you need shell or exec form for your commands.

You can find high resolution diagrams here.

Running CMD and ENTRYPOINT

The following examples will walk you through the high-level differences between CMD and ENTRYPOINT, through an example

First lets create our Dockerfile

# Use the Ubuntu 20.04 image as the base image
FROM ubuntu:20.04

# Update the image and install traceroute
RUN apt-get update && apt-get upgrade -y
RUN apt-get install -y traceroute

# Set the default command
CMD traceroute

Then build your image using the docker build -t tracert command.

Run the container image with `CMD traceroute`

Without passing any arguments, we get the following output.

docker run tracert

Usage:
  traceroute [ -46dFITnreAUDV ] [ -f first_ttl ] [ -g gate,... ] [ -i device ] [ -m max_ttl ] [ -N squeries ] [ -p port ] [ -t tos ] [ -l flow_label ] [ -w MAX,HERE,NEAR ] [ -q nqueries ] [ -s src_addr ] [ -z sendwait ] [ --fwmark=num ] host [ packetlen ]
Options:
  -4                          Use IPv4
  -6                          Use IPv6
  -d  --debug                 Enable socket level debugging
  -F  --dont-fragment         Do not fragment packets
  -f first_ttl  --first=first_ttl
                              Start from the first_ttl hop (instead from 1)
  -g gate,...  --gateway=gate,...
                              Route packets through the specified gateway
                              (maximum 8 for IPv4 and 127 for IPv6)
  -I  --icmp                  Use ICMP ECHO for tracerouting
  -T  --tcp                   Use TCP SYN for tracerouting (default port is 80)
  -i device  --interface=device
                              Specify a network interface to operate with
  -m max_ttl  --max-hops=max_ttl
                              Set the max number of hops (max TTL to be
                              reached). Default is 30
[...output truncated...]

However, if we provide an IP address to test, we will get an error

docker run tracert www.dev.to                                                      docker: Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: exec: "www.dev.to": executable file not found in $PATH: unknown.

The problem is that the string you're providing on the command line, www.dev.to, is completely replacing the CMD instruction in your Dockerfile. Since that URL isn't a valid command, it's causing an error.

To fix this, you need to specify the actual command you want to run alongside the URL.

docker run tracert traceroute www.dev.to
traceroute to www.dev.to (104.18.26.242), 30 hops max, 60 byte packets
[...output truncated...]

Run the container image with `ENTRYPOINT traceroute`

In this updated version, we'll make a change to the Dockerfile. We will remove the original CMD instruction and replace it with ENTRYPOINT ["traceroute"]. This means the traceroute command is now the main command that this container will run when it starts.

The ENTRYPOINT instruction works a bit differently than CMD. With ENTRYPOINT, you can't simply override the command by typing something different after docker run. Instead, any extra arguments you add after docker run are treated as arguments for the traceroute command itself.

# Use the Ubuntu 20.04 image as the base image
FROM ubuntu:20.04

# Update the image and install traceroute
RUN apt-get update && apt-get upgrade -y
RUN apt-get install -y traceroute

# Set the default command
ENTRYPOINT ["traceroute"]

Let's see what happens when we try that

docker run tracert www.dev.to
traceroute to www.dev.to (104.18.26.242), 30 hops max, 60 byte packets
[...output truncated...]

The key point here is to use ENTRYPOINT when you want to ensure that a specific executable is always run when your container starts, regardless of any additional commands or arguments the user might provide. It gives you a way to create containers that behave like self-contained tools, with a clearly defined main purpose.

Summary

Deciding when to use RUN, CMD, or ENTRYPOINT, and whether to choose the shell or exec form, demonstrates the level of detail and flexibility Docker offers. Each of these commands plays a distinct role in the Docker ecosystem, influencing how containers are constructed, behave, and interact with their environment.

By carefully selecting the right command and form for each specific situation, developers can create Docker images that are more dependable, secure, and optimized for efficiency. Mastering these Docker commands and their formats is essential for unlocking the full potential of Docker. When these best practices are followed, applications deployed within Docker containers can achieve peak performance across diverse environments, enhancing both development workflows and production deployments.

Natural Numbers

Kostas Kalafatis — Wed, 17 Jul 2024 07:20:00 +0000

What is a number?

In mathematics, there are several ways to approach this question. We can look at it semantically, by understanding what numbers mean. Alternatively, we can take the axiomatic approach, focusing on their fundamental properties and behaviors. Or, we can answer the question constructively by looking at how numbers can be constructed from simpler objects.

The Semantic Approach

Let's begin with semantics. What do numbers really mean? Many people assume that numbers are just a tool for counting things, but their meaning is more nuanced. Numbers can serve two different purposes depending on their context.

There are two main types of numbers: cardinal and ordinal. For example, when we see the number 3, it's meaning is not immediately clear. It could mean 3 as in I have three eggplants, or it could mean 3 as in I want the third eggplant. The 3 in three eggplants is a cardinal number, and the 3 in third eggplant is an ordinal number.

A cardinal number counts how many objects there are in a group. When we say I want **three* eggplants*, that three is a cardinal. An ordinal number counts where a particular object is in a group. When we say I want the **third* eggplant*, that three is an ordinal.

The cardinal/ordinal distinction really starts to make sense when you talk about the set theoretic basis of math. For now, the basic idea is enough: cardinals count objects, ordinals position them.

The Axiomatic Approach

The axiomatic part is a lot more interesting. Here, we define concepts like numbers using a set of rules, known as axioms. These axioms lay out how the numbers (or whatever we're working with) act. Mathematicians love axiomatic definitions because they eliminate any confusion about what's possible and how it works. While they might not be as intuitive, they're incredibly precise and perfect for logical reasoning.

Let's begin our exploration of numbers with the most basic kind: the natural (or counting) numbers. These are the whole numbers we first learn as kids. They start with zero and going on forever: 0, 1, 2, 3, 4, and so on. They are like the building blocks of all numbers, and for computer scientists, they're particularly special because everything we can compute is ultimately based on them. Formally, these numbers are defined by a set of rules called Peano arithmetic, which lays out exactly how they work.

Peano arithmetic specifies a list of axioms that define the natural numbers.

Initial Value Rule: There is one special object called 0, and 0 is a natural number.
Successor Rule: Every natural number n has a successor in the natural numbers, called its successor, s(n). Zero is the only natural number that is not the successor of any natural number.
Uniqueness Rule: If the successor of two natural numbers is the same, then the two original numbers are the same.
Equality Rules: While these are not considered to be part of the Peano axioms in modern treatments, the following rules describe the equality relation:
- Reflexive: For every natural number x then x = x. Or in other words, every natural number is equal to itself.
- Symmetric: For all natural numbers x and y, if x = y then y = x.
- Transitive: For all natural numbers x, y and z, if x = y and y = z, then x = z.
- Closed under equality: For all a and b, if b is a natural number and a = b, then a is also a natural number.
Induction Rule: For some statement P, P is true for all natural numbers if
1. P is true about 0 (that is, P(0) is true)
2. If you assume P is true for a natural number n (P(n) is true), then you can prove that P is true for the successor s(n) of n (or P(s(n)) is true)

And of the above is just a fancy way of saying that natural numbers are numbers with no fractional part starting at 0. Most people get the Peano rules right away, except for the last one about induction. It's a tricky concept – it can feel a bit circular at first. But it's crucial because natural numbers go on forever. Induction is the tool that lets us take what we know about a finite number of things and apply it to the whole infinite set.

Forgetting the technical jargon, the induction rule basically says: if something works for your first number, and you have a way to describe what happens when you add 1 to it, you can apply this pattern to all the numbers that follow. This pattern lets us write proofs that hold true for all natural numbers, or even define things that work for all of them. We can use similar techniques to tackle all integers, fractions, or even all real numbers. Let's start with an example of a definition, which is easier to understand than a proof.

Let's illustrate how induction is used in a definition by looking at addition. Addition is simply a function "+" that takes two natural numbers and produces another natural number, their sum. We can formally define addition with the following rules:

Commutativity: For any pair of natural numbers a and b, $n + m = m + n$

or in simple terms, it means you can swap the numbers around, and the result stays the same.

Identity: For any natural number n, $n + 0 = 0 + n = n$

or in simple terms, adding 0 to any number doesn't change the value.

Recursion: For any natural number n, $m + s (n) = s (m + n)$

The last rule is based on a concept called recursion. It can be a bit tricky if you're not familiar with it, so let's break it down.

Essentially, we're defining addition by using the Peano arithmetic concept of adding 1 to a number. If we rewrite the rule slightly, it becomes:
$m + n = 1 + (m + (n - 1)) .$

Remember, this is a definition, not a procedure. It explains what addition means, not how to calculate it.

This rule works because of the Peano induction rule. Without it, we wouldn't have a way to define addition for any two numbers. Induction gives us a way to express the meaning of addition for any pair of natural numbers.

Now, let's tackle a proof. I know proofs can seem intimidating, but don't worry! They're not so bad, and we'll start with a very simple one.

Using Peano's Induction

Let's have some fun with a simple proof using natural numbers and addition. Suppose we have a natural number N. Can you guess the sum of all the integers from 1 to N? It's actually N times (N + 1) divided by 2. Let's prove this using induction.

First, we start with a base case, a starting point we can prove on its own. In this case, our base case is 0 because the first part of the induction rule requires us to show it works for 0. Luckily, this is easy:

(0 * (0 + 1)) /2 = 0.

So, our equation holds true when N is 0.

Now, for the inductive part. Let's assume the rule is true for a number N. We need to prove it's also true for N + 1.

This is where the magic of induction happens. We want to show that if the rule works for 0, it must work for 1, and if it works for 1, it must work for 2, and so on. We don't want to prove each of these cases individually, so we simply say, "If it's true for N, it must be true for N + 1."

By using a variable in this inductive structure, we're essentially saying, "If it's true for 0, then it's true for 1; if it's true for 1, then it's true for 2", and so on.

Here's what we want to prove:

\frac{(n + 1)(n + 2)}{2}

To begin, we know that

\frac{n(n+1)}{2}

So we can substitute that in get this

\frac{n(n+1)}{2} + n + 1 = \frac{(n+1)(n+2)}{n}

Now, we can expand the multiplication on both sides

\frac{n^2 + n}{2} + (n + 1) = \frac{n^2 + 3n + 2}{2}

Get the common denominator on the left side

\frac{n^2 + n + 2n + 2}{2} = \frac{n^2 + 3n + 2}{2}

Finally, simplify the left side

\frac{n^2 + 3n + 2}{2} = \frac{n^2 + 3n + 2}{2}

And there you have it! We've proven the equation holds true for all natural numbers. So, that's the axiomatic definition of natural numbers – numbers greater than or equal to zero, each with a successor, and where you can use induction. It's amazing how almost everything we do with natural numbers, including basic arithmetic we learn as kids, can be built upon this foundation.

So, can we now define what a number is? Well, kind of. One thing we learn in math is that numbers don't have just one meaning. There are many types of numbers: natural numbers, integers, rational numbers, real numbers, complex numbers, and so on. The entire universe of numbers begins with what we just explored: the natural numbers. And ultimately, the meaning of these numbers boils down to either quantity or position.

They're all either cardinal numbers (representing quantity) or ordinal numbers (representing position), or combinations of both. In essence, a number is a construct that represents either a quantity or a position.

Managing Docker Images

Kostas Kalafatis — Mon, 15 Jul 2024 06:00:00 +0000

Introduction

In the previous posts of the series, we discussed in depth about Docker images. As we've seen, we've been able to take existing images, provided to the general public in Docker Hub, and run them or reuse them for our purposes. The image itself helps us streamline our processes and reduce the work we need to do.

In the next few posts, we are going to take a more in-depth look at images and how to work with them on our system. We're going to discuss how images can be better organized and tagged, understand how different layers of images work, and set up registries that are both public and private to further reuse the images we have created.

Docker images are also ideal for application development because each image contains a complete, self-contained version of the application along with all its dependencies. This enables developers to create an image locally and deploy it in development or testing environments to verify compatibility with other parts of the application. If testing is successful, the same image can be pushed to the production environment for users to use. It's crucial to maintain consistency when using these images, especially when collaborating within larger developer teams.

Docker Layers and Caching

A registry is a way to store and distribute Docker images. When you pull a Docker image from a registry, you might notice that the image is pulled in pieces and not as a single image. The same thing happens when you build an image on your system.

This is because Docker images are structured in layers, each representing a stage of the image's construction. Each layer of an image represents a specific action or change made when building the image with a Dockerfile. These layers are organized on top of a base image, capturing every change to the filesystem that occurs with each instruction in the Dockerfile. This setup is structured in a way that Docker can use caching efficiently.

When you instantiate an image as a container, Docker adds a writable layer on top of the existing read-only layers. This writable layer, often referred to as the container layer, allows the container to modify and persist changes during runtime without affecting the underlying image.

As we will see in the following examples, when you build a Docker container from a Dockerfile, Docker shows the execution of each command specified in the Dockerfile. These commands contribute to creating layers in the Docker image, each represented by a unique ID generated during the build process. After successfully building the image, we can inspect the layers using the docker history command, which provides a detailed view including the image name or ID alongside the commands that formed each layer.

It's important to note, that as you setup your build environment and progress in development, the number of layers in the Docker image grows. More layers mean larger image sizes, which can lead to longer build times.

When you build an image from a Dockerfile, each instruction contributes to the creation of layers in the image. Layers are created explicitly when commands like RUN, ADD and COPY are executed. These commands make changes to the filesystem within the image, resulting in new layers being added.

On the other hand, commands like FROM, ENV, WORKDIR and CMD do not directly create filesystem changes. Instead, they modify the environment or configure settings within the image without altering the filesystem itself. As a result, these commands generate intermediate layers. These layers have a size of 0 bytes because they don't introduce any new filesystem change. They serve as metadata or configuration layers that help define how an image behaves or is structured, but they don't increase the size of the final Docker image.

When building our Docker images, we can use the docker history command and the image name or ID to see the layers used to create the image. The output will provide details on commands being used to generate the layer as well as the size of the layer.

docker history <image_name|image_id>

The docker image inspect command is useful in providing further details on where the layers of our images are located:

docker image inspect <image_id>

Working with Docker Image Layers

In this example, we are going to work with some basic Dockerfiles to see how Docker uses layers to build images. We will start by creating a Dockerfile and building a new image. We will then rebuild the image to see the advantages of caching and how the build time is reduced due to its use.

Create a file named Dockerfile and add the following directives:

FROM alpine



RUN apk update

RUN apk add wget

Save the Dockerfile and then, from the command line, make sure you are in the same directory as the Dockerfile you are created. Use the docker build command to create the new image using the -t option to name it basic-example

docker build -t basic-example .

If the image is built successfully, you should see an output similar to the following. Rach step is built as an intermediate layer and if it completes successfully, it is then transferred to a read-only layer

[+] Building 9.0s (8/8) FINISHED                                                                         docker:default
 => [internal] load .dockerignore                                                                                  0.1s
 => => transferring context: 2B                                                                                    0.0s
 => [internal] load build definition from Dockerfile                                                               0.1s
 => => transferring dockerfile: 84B                                                                                0.0s
 => [internal] load metadata for docker.io/library/alpine:latest                                                   3.6s
 => [auth] library/alpine:pull token for registry-1.docker.io                                                      0.0s
 => [1/3] FROM docker.io/library/alpine@sha256:b89d9c93e9ed3597455c90a0b88a8bbb5cb7188438f70953fede212a0c4394e0    1.1s
 => => resolve docker.io/library/alpine@sha256:b89d9c93e9ed3597455c90a0b88a8bbb5cb7188438f70953fede212a0c4394e0    0.1s
 => => sha256:a606584aa9aa875552092ec9e1d62cb98d486f51f389609914039aabd9414687 1.47kB / 1.47kB                     0.0s
 => => sha256:ec99f8b99825a742d50fb3ce173d291378a46ab54b8ef7dd75e5654e2a296e99 3.62MB / 3.62MB                     0.6s
 => => sha256:b89d9c93e9ed3597455c90a0b88a8bbb5cb7188438f70953fede212a0c4394e0 1.85kB / 1.85kB                     0.0s
 => => sha256:dabf91b69c191a1a0a1628fd6bdd029c0c4018041c7f052870bb13c5a222ae76 528B / 528B                         0.0s
 => => extracting sha256:ec99f8b99825a742d50fb3ce173d291378a46ab54b8ef7dd75e5654e2a296e99                          0.2s
 => [2/3] RUN apk update                                                                                           2.1s
 => [3/3] RUN apk add wget                                                                                         1.7s
 => exporting to image                                                                                             0.1s
 => => exporting layers                                                                                            0.1s
 => => writing image sha256:2fc965ee555abcf548a268e3622ee031366479ddefb080e6920847c46a8848b9                       0.0s
 => => naming to docker.io/library/basic-example

Use the docker history command along with the image name of basic-example to see the different layers of the image

docker history basic-example

The history gives you creation details, including the size of each layer

IMAGE          CREATED         CREATED BY                                      SIZE      COMMENT
2fc965ee555a   7 minutes ago   RUN /bin/sh -c apk add wget # buildkit          3.07MB    buildkit.dockerfile.v0
<missing>      7 minutes ago   RUN /bin/sh -c apk update # buildkit            2.32MB    buildkit.dockerfile.v0
<missing>      5 days ago      /bin/sh -c #(nop)  CMD ["/bin/sh"]              0B
<missing>      5 days ago      /bin/sh -c #(nop) ADD file:33ebe56b967747a97…   7.8MB

The docker history command shows the layer of the original image used as part of the Dockerfile FROM command as <missing>. It is showing as missing as it was created by a different system and then pulled into ours.

Run the build again without making any changes

docker build -t basic-example .

This will show you the build is done using the layers stored in the Docker image cache, thereby speeding up our build. Although this is a small image, a much larger image would show a significant increase

[+] Building 4.3s (8/8) FINISHED                                                                         docker:default
 => [internal] load .dockerignore                                                                                  0.0s
 => => transferring context: 2B                                                                                    0.0s
 => [internal] load build definition from Dockerfile                                                               0.0s
 => => transferring dockerfile: 84B                                                                                0.0s
 => [internal] load metadata for docker.io/library/alpine:latest                                                   4.1s
 => [auth] library/alpine:pull token for registry-1.docker.io                                                      0.0s
 => [1/3] FROM docker.io/library/alpine@sha256:b89d9c93e9ed3597455c90a0b88a8bbb5cb7188438f70953fede212a0c4394e0    0.0s
 => CACHED [2/3] RUN apk update                                                                                    0.0s
 => CACHED [3/3] RUN apk add wget                                                                                  0.0s
 => exporting to image                                                                                             0.0s
 => => exporting layers                                                                                            0.0s
 => => writing image sha256:2fc965ee555abcf548a268e3622ee031366479ddefb080e6920847c46a8848b9                       0.0s
 => => naming to docker.io/library/basic-example                                                                   0.0s

Lets add the curl package as part of our image creation, and modify the Dockerfile as follows

FROM alpine



RUN apk update

RUN apk add wget

RUN apk add curl

Build the image again, and now you'll see the image was created with a mix of cached and new layers

docker build -t basic-example .

The above command should create the following output

[+] Building 7.1s (9/9) FINISHED                                                                                                                                                                                                                               docker:default
 => [internal] load .dockerignore                                                                                                                                                                                                                                        0.0s
 => => transferring context: 2B                                                                                                                                                                                                                                          0.0s
 => [internal] load build definition from Dockerfile                                                                                                                                                                                                                     0.0s
 => => transferring dockerfile: 102B                                                                                                                                                                                                                                     0.0s
 => [internal] load metadata for docker.io/library/alpine:latest                                                                                                                                                                                                         4.0s
 => [auth] library/alpine:pull token for registry-1.docker.io                                                                                                                                                                                                            0.0s
 => [1/4] FROM docker.io/library/alpine@sha256:b89d9c93e9ed3597455c90a0b88a8bbb5cb7188438f70953fede212a0c4394e0                                                                                                                                                          0.0s
 => CACHED [2/4] RUN apk update                                                                                                                                                                                                                                          0.0s
 => CACHED [3/4] RUN apk add wget                                                                                                                                                                                                                                        0.0s
 => [4/4] RUN apk add curl                                                                                                                                                                                                                                               2.8s
 => exporting to image                                                                                                                                                                                                                                                   0.1s
 => => exporting layers                                                                                                                                                                                                                                                  0.1s
 => => writing image sha256:f2bb77eb9898954a27c2bd12838c3ed0fdfe19ed78a7440189c28bdb0cbfbf8d                                                                                                                                                                             0.0s
 => => naming to docker.io/library/basic-example

Run the docker image command again

docker images

You will now notice an image named and tagged as <none> to show we have now created a dangling image

REPOSITORY                                      TAG       IMAGE ID       CREATED          SIZE
basic-example                                   latest    f2bb77eb9898   52 seconds ago   18.9MB
<none>                                          <none>    2fc965ee555a   16 hours ago     13.2MB
onbuild-child-example                           latest    9fb3629a292e   5 days ago       222MB
onbuild-parent-example                          latest    4a6360882fb6   6 days ago       222MB

In Docker, dangling images are those that are represented by <none> in the image list. These images occur when a layer in the image hierarchy no longer corresponds to any tagged or referenced image in the system. Essentially, they are orphaned or unused layers that have lost their connection to any active image.

Dangling images can accumulate over time as you build and prune Docker images, and they occupy disk space without serving any meaningful purpose. Even though each individual dangling image might be relatively small, such as the example of 7.48 MB, these sizes can accumulate significantly over time, especially in development and production environments where frequent image builds and updates occur.

Run the docker image inspect command using the image ID to see the location of where the dangling images are located in the system

docker image inspect 2fc965ee555a

And you should get an output similar to the following

...
   "Data":{
      "LowerDir":"/var/lib/docker/overlay2/p9vnb2sakx8pcxhnabhpbbghs/diff:/var/lib/docker/overlay2/f41cbe299d47005328bcfbf0aa9c958ead5148eca0e0f65679aaabb38f9db96a/diff",
      "MergedDir":"/var/lib/docker/overlay2/sbm1ykpcezy3eydzy6eyxhz08/merged",
      "UpperDir":"/var/lib/docker/overlay2/sbm1ykpcezy3eydzy6eyxhz08/diff",
      "WorkDir":"/var/lib/docker/overlay2/sbm1ykpcezy3eydzy6eyxhz08/work"
   }
...

All of our images are located in the same location. So any dangling images would waste space on our system.

Run the docker images command again using the -a option

docker images -a

It will also show the intermediate layers used when our image is build

basic-example                                   latest    f2bb77eb9898   23 minutes ago   18.9MB
<none>                                          <none>    2fc965ee555a   17 hours ago     13.2MB
onbuild-child-example                           latest    9fb3629a292e   5 days ago       222MB

Run the docker image prune command to remove all the dangling images. We could use docker rmi but the docker image prune command is the easier way to do it

docker image prune

You should get an output looking like the following

WARNING! This will remove all dangling images.
Are you sure you want to continue? [y/N] y
Deleted Images:
deleted: sha256:2fc965ee555abcf548a268e3622ee031366479ddefb080e6920847c46a8848b9

Run the docker images command again

docker images

You will see we no longer have the dangling image in our list of images

REPOSITORY                                      TAG       IMAGE ID       CREATED          SIZE
basic-example                                   latest    f2bb77eb9898   28 minutes ago   18.9MB
onbuild-child-example                           latest    9fb3629a292e   5 days ago       222MB
onbuild-parent-example                          latest    4a6360882fb6   6 days ago       222MB

In this example was on smaller image sizes, but this is definitely something to keep in mind when running production and development environments. In the next example, we will look further at our layers and caching to see how they can be used to speed up the image build process.

Increasing Build Speed and Reducing Layers

We've been working with small projects up until now. As our apps get bigger and more complex, though, we'll want to start thinking about the size and number of layers in our Docker images, along with how quickly we're building them. In this example, we'll focus on speeding up build times, shrinking those image sizes, and using the --cache-from option to make things even faster.

First, let's clean up any existing images on your system. We'll use the docker rmi -f $(docker images -a -q) command, which will force-remove all images currently on your system. This will give you a clean slate to work with.

Create a new Dockerfile with the following content. It will simulate a simple web server, as well as print the output of our Dockerfile during the build process

FROM alpine

RUN apk update
RUN apk add wget curl

RUN wget -O randomdata.txt https://github.com/Kalkwst/Docker-Workshop-Repository/blob/master/Dockerfiles/create-base-image2/random_data.txt

CMD mkdir /var/www/
CMD mkdir /var/www/html/

WORKDIR /var/www/html/

COPY Dockerfile.tar.gz /tmp/

RUN tar -zxvf /tmp/Dockerfile.tar.gz -C /var/www/html
RUN rm /tmp/Dockerfile.tar.gz

RUN cat Dockerfile

Download the Alpine base image using docker pull so that we can start with the same image for each test we do

docker pull alpine

Create a TAR file to be added to our image

tar zcvf Dockerfile.tar.gz Dockerfile

Build a new image using the name of basic-server. We are going to use the time command at the start of the command to allow us to gauge the time it takes to build the image

time docker build -t basic-server .

The output will return something similar to the following

#0 building with "default" instance using docker driver

#1 [internal] load build definition from Dockerfile
#1 transferring dockerfile: 451B done
#1 DONE 0.0s

#2 [internal] load .dockerignore
#2 transferring context: 2B done
#2 DONE 0.1s

#3 [internal] load metadata for docker.io/library/alpine:latest
#3 DONE 0.0s

#4 [1/9] FROM docker.io/library/alpine
#4 DONE 0.0s

#5 [2/9] RUN apk update
#5 CACHED

#6 [internal] load build context
#6 transferring context: 396B done
#6 DONE 0.0s

#7 [3/9] RUN apk add wget curl
#7 0.323 fetch https://dl-cdn.alpinelinux.org/alpine/v3.20/main/x86_64/APKINDEX.tar.gz
#7 1.054 fetch https://dl-cdn.alpinelinux.org/alpine/v3.20/community/x86_64/APKINDEX.tar.gz
#7 2.267 (1/12) Installing ca-certificates (20240226-r0)
#7 2.393 (2/12) Installing brotli-libs (1.1.0-r2)
#7 2.638 (3/12) Installing c-ares (1.28.1-r0)
#7 2.723 (4/12) Installing libunistring (1.2-r0)
#7 3.061 (5/12) Installing libidn2 (2.3.7-r0)
#7 3.155 (6/12) Installing nghttp2-libs (1.62.1-r0)
#7 3.240 (7/12) Installing libpsl (0.21.5-r1)
#7 3.316 (8/12) Installing zstd-libs (1.5.6-r0)
#7 3.526 (9/12) Installing libcurl (8.8.0-r0)
#7 3.709 (10/12) Installing curl (8.8.0-r0)
#7 3.831 (11/12) Installing pcre2 (10.43-r0)
#7 4.008 (12/12) Installing wget (1.24.5-r0)
#7 4.157 Executing busybox-1.36.1-r29.trigger
#7 4.160 Executing ca-certificates-20240226-r0.trigger
#7 4.183 OK: 14 MiB in 26 packages
#7 DONE 4.3s

#8 [4/9] RUN wget -O randomdata.txt https://github.com/Kalkwst/Docker-Workshop-Repository/blob/master/Dockerfiles/create-base-image2/random_data.txt
#8 0.404 --2024-07-01 07:34:51--  https://github.com/Kalkwst/Docker-Workshop-Repository/blob/master/Dockerfiles/create-base-image2/random_data.txt
#8 0.414 Resolving github.com (github.com)... 140.82.121.3
#8 0.561 Connecting to github.com (github.com)|140.82.121.3|:443... connected.
#8 0.685 HTTP request sent, awaiting response... 200 OK
#8 1.109 Length: unspecified [text/html]
#8 1.109 Saving to: 'randomdata.txt'
#8 1.109
#8 1.109      0K .......... .......... .......... .......... ..........  437K
#8 1.223     50K .......... .......... .......... .......... ..........  881K
#8 1.280    100K .......... .......... .......... .......... .......... 2.02M
#8 1.304    150K .......... .......... .......... .......... ..........  720K
#8 1.373    200K .......... .......... .......... .......... .......... 35.1M
#8 1.375    250K .......... .......... .......                           481K=0.3s
#8 1.432
#8 1.432 2024-07-01 07:34:52 (858 KB/s) - 'randomdata.txt' saved [284319]
#8 1.432
#8 DONE 1.5s

#9 [5/9] WORKDIR /var/www/html/
#9 DONE 0.0s

#10 [6/9] COPY Dockerfile.tar.gz /tmp/
#10 DONE 0.0s

#11 [7/9] RUN tar -zxvf /tmp/Dockerfile.tar.gz -C /var/www/html
#11 0.365 Dockerfile
#11 DONE 0.4s

#12 [8/9] RUN rm /tmp/Dockerfile.tar.gz
#12 DONE 0.5s

#13 [9/9] RUN cat Dockerfile
#13 0.450 FROM alpine
#13 0.450
#13 0.450 RUN apk update
#13 0.450 RUN apk add wget curl
#13 0.450
#13 0.450 RUN wget -O randomdata.txt https://github.com/Kalkwst/Docker-Workshop-Repository/blob/master/Dockerfiles/create-base-image2/random_data.txt
#13 0.450
#13 0.450 CMD mkdir /var/www/
#13 0.450 CMD mkdir /var/www/html/
#13 0.450
#13 0.450 WORKDIR /var/www/html/
#13 0.450
#13 0.450 COPY Dockerfile.tar.gz /tmp/
#13 0.450
#13 0.450 RUN tar -zxvf /tmp/Dockerfile.tar.gz -C /var/www/html
#13 0.450 RUN rm /tmp/Dockerfile.tar.gz
#13 0.450
#13 0.450 RUN cat Dockerfile
#13 DONE 0.5s

#14 exporting to image
#14 exporting layers
#14 exporting layers 0.2s done
#14 writing image sha256:971477cab5a251188e82e14baaf9268de83ecdd04429e5818c617ffe0803921c done
#14 naming to docker.io/library/basic-server done
#14 DONE 0.2s

And the time will be:

...
real    0m10.468s
user    0m0.060s
sys     0m0.276s

Run the docker history command over the new basic-app image

docker history basic-server

The output should be something like the following

IMAGE          CREATED         CREATED BY                                      SIZE      COMMENT
971477cab5a2   8 minutes ago   RUN /bin/sh -c cat Dockerfile # buildkit        0B        buildkit.dockerfile.v0
<missing>      8 minutes ago   RUN /bin/sh -c rm /tmp/Dockerfile.tar.gz # b…   0B        buildkit.dockerfile.v0
<missing>      8 minutes ago   RUN /bin/sh -c tar -zxvf /tmp/Dockerfile.tar…   412B      buildkit.dockerfile.v0
<missing>      8 minutes ago   COPY Dockerfile.tar.gz /tmp/ # buildkit         350B      buildkit.dockerfile.v0
<missing>      8 minutes ago   WORKDIR /var/www/html/                          0B        buildkit.dockerfile.v0
<missing>      8 minutes ago   CMD ["/bin/sh" "-c" "mkdir /var/www/html/"]     0B        buildkit.dockerfile.v0
<missing>      8 minutes ago   CMD ["/bin/sh" "-c" "mkdir /var/www/"]          0B        buildkit.dockerfile.v0
<missing>      8 minutes ago   RUN /bin/sh -c wget -O randomdata.txt https:…   284kB     buildkit.dockerfile.v0
<missing>      8 minutes ago   RUN /bin/sh -c apk add wget curl # buildkit     8.75MB    buildkit.dockerfile.v0
<missing>      4 days ago      RUN /bin/sh -c apk update # buildkit            2.32MB    buildkit.dockerfile.v0
<missing>      10 days ago     /bin/sh -c #(nop)  CMD ["/bin/sh"]              0B
<missing>      10 days ago     /bin/sh -c #(nop) ADD file:33ebe56b967747a97…   7.8MB

As you can see there are 12 layers in our new image. As you can see, the RUN, COPY and ADD commands in our Dockerfile are creating layers of a particular size relevant to the command being run or files being added, and all of the other commands are of size 0 B.

We can slim down our image by merging some of the commands in the Dockerfile we created earlier. Combine the RUN commands from lines 3 and 4, and then merge the CMD commands from lines 8 and 9. This will reduce the number of layers in our image, making it more efficient.

After making these changes, your Dockerfile should look like this:

FROM alpine

RUN apk update && apk add wget curl
RUN wget -O randomdata.txt https://github.com/Kalkwst/Docker-Workshop-Repository/blob/master/Dockerfiles/create-base-image2/random_data.txt

CMD mkdir -p /var/www/html/

WORKDIR /var/www/html/

COPY Dockerfile.tar.gz /tmp/
RUN tar -zxvf /tmp/Dockerfile.tar.gz -C /var/www/hmtl/
RUN rm /tmp/Dockerfile.tar.gz

RUN cat Dockerfile

If we rebuild our Docker image now, we'll see that the number of layers has decreased from 12 to 9. This is because we combined several commands into single lines, even though the same actions are still being performed.

We can further optimize our Dockerfile by replacing lines 11, 12, and 13 with a single ADD command. This eliminates the need for separate COPY, RUN, and RUN commands to unzip and remove the archived file. Here's the updated Dockerfile snippet:

FROM alpine

RUN apk update && apk add wget curl
RUN wget -O randomdata.txt https://github.com/Kalkwst/Docker-Workshop-Repository/blob/master/Dockerfiles/create-base-image2/random_data.txt

CMD mkdir -p /var/www/html/

WORKDIR /var/www/html/

ADD Dockerfile.tar.gz -C /var/www/hmtl/
RUN rm /tmp/Dockerfile.tar.gz

RUN cat Dockerfile

Rebuilding our Docker image with the ADD command in place reduces the number of layers from 9 to 8, making it even more streamlined.

You might have noticed that a significant portion of the build time is spent running apk update, installing wget and curl, and fetching content from websites (as seen in lines 3 and 5 of our Dockerfile). While this isn't a major issue for a few builds, it can become a bottleneck when creating multiple images.

To address this, we can create a base image that already includes these tools and dependencies. By using this base image as our starting point, we can eliminate these lines from our Dockerfile altogether, further improving build times and image efficiency.

All right, let's create a dedicated base image to streamline our Docker builds. First, navigate to a new directory:

mkdir base-image
cd base-image

Now, create a new Dockerfile in this directory with the following contents:

FROM alpine

RUN apk update && apk add --no-cache wget curl
RUN wget -O randomdata.txt https://github.com/Kalkwst/Docker-Workshop-Repository/blob/master/Dockerfiles/create-base-image2/random_data.txt

This Dockerfile does three things:

Pulls the base image: It starts with the alpine:latest image as its foundation.
Runs the apk commands: It updates the package index and installs wget and curl using apk add --no-cache. The --no-cache option prevents apk from storing the downloaded packages in the local cache, keeping the image smaller.
Runs the wget command: It finally downloads the randomdata.txt file from the external directory.

Build the new image from the previous Dockerfile and name it basic-base

docker build basic-base .

Perfect! Now that we have a base image ready, let's update our original Dockerfile. First, head back to the directory where your original Dockerfile resides:

cd ../<original_project_directory>

Now, open your Dockerfile and make the following changes:

Remove line 3: Delete the line that starts with RUN apk update... since these commands are now handled in our base image.
Update the FROM command: Change the base image from alpine:latest to the name you'll give your custom base image (i.e., basic-base).
Remove the apk commands from line 3 (now line 2): Delete the remaining part of the line that installed wget and curl. Your updated Dockerfile should now look similar to this

FROM basic-base

CMD mkdir -p /var/www/html/

WORKDIR /var/www/html/

ADD Dockerfile.tar.gz /var/www/html/
RUN cat Dockerfile

Run the build again for your new Dockerfile. Using the time command again, you should see the build complete in just under 3 seconds

time docker build -t basic-server .

You'll likely notice a significant speed improvement compared to our previous builds. This is because we've offloaded the time-consuming package installations to our base image, streamlining the build process for our main image.

real    0m2.924s
user    0m0.000s
sys     0m0.262s

Throughout this example, we've seen firsthand how the build cache and image layers work together to significantly speed up the Docker build process. We've been starting our builds by pulling images from Docker Hub, but you have the flexibility to start with your own custom images. This gives you even more control over the build process and allows for further optimization.

By creating and maintaining your own base images, you can tailor them to your specific needs, pre-installing common dependencies and configurations. This not only reduces build times but also ensures consistency across your projects.

Summary

This post demonstrated how Docker allows users to work with images to package their applications together with a working environment to be moved across different environments. You've also seen how Docker uses layers and caching to improve build speed and ensure you can also work with these layers to reserve resources or disk space.

The Mathematics of Algorithms

Kostas Kalafatis — Fri, 12 Jul 2024 06:00:00 +0000

One of the most crucial considerations when selecting an algorithm is the speed with which it is likely to complete. Predicting this speed involves using mathematical methods. This post delves into the mathematical tools, aiming to clarify the terms commonly used in this series and the rest of the literature that describes algorithms.

Problem Instance Size

Think of the problem you are giving to an algorithm implementation as a recipe you are following. An instance of that problem is like the number of portions your recipe will create. Most of the time, the more portions (or the larger the dataset), the longer it takes to cook the dish (or for the implementation to run). On the other hand, trying to compress the dataset might include unnecessary operations that will eventually slow down the execution. Finding this sweet spot in how you represent the data to the computer is surprisingly challenging. This is because real-world problems aren't naturally in a format computers understand. They need to be translated into a suitable representation, and there are many ways to do that.

When we judge how good an implementation is at solving a problem, we want to focus on the program's core logic (the algorithm), not how the problem was presented. We don't want the way we formatted the data to be the main thing that decides how fast or well the program works. The way you represent the problem should really just depend on what the program needs to do with the data. For example, if you need to find the shortest route between cities, the best way to represent the map data is different than if you're sorting a list of names.

Designing efficient algorithms often starts with picking the right tools for the job. In this case, the tools are data structures – how you organize and store the information in the computer's memory. By choosing the right data structures, you set the stage for a program that solves the problem quickly and smoothly.

Since there's no single, universal way to define the "size" of a problem for a computer program, we usually rely on common sense and practical conventions. For instance, when dealing with a task like sorting numbers, we typically assume that each number can be stored in a single 32-bit word (a standard unit of computer memory). So, if we're sorting 100 numbers, we say the size of the problem is 100.

Now, sometimes a number might be too big to fit into one word. But if it takes a consistent, fixed number of words (say, 2 or 3), our estimate of the problem size is only off by a small factor. For instance, an algorithm that works with 64-bit numbers might take roughly twice as long as one working with 32-bit numbers, even if they're doing the same kind of sorting.

The key point is this: we make these assumptions about how data is encoded so we can compare different algorithms fairly. By focusing on the size of the problem (like the number of items to be sorted) and assuming a reasonable way to store the data, we can get a good idea of how well an algorithm performs without getting bogged down in the details of data representation.

In the world of algorithm design, researchers accept that it's impossible to perfectly predict how the choice of data representation will impact the performance of an algorithm when implemented. To simplify things, they consider performance differences that are only a constant multiple of each other as being essentially the same, especially when dealing with very large problem sizes. This concept is called asymptotic equivalence.

Let's take an example. We know that working with 64-bit integers is generally slower than 32-bit integers. But if we have a good algorithm designed for sorting a million 32-bit integers, it's reasonable to assume it will also be a good algorithm for sorting a million 64-bit integers. The difference in processing time, while existing, becomes less significant as the problem size increases.

Of course, in real-life situations, nobody would be happy to pay a bill that's 1000 times larger than expected! But this simplification, where we disregard constant multiplicative factors, is a widely accepted way to compare the efficiency of different algorithms in theory. It helps researchers focus on the core logic of the algorithm and avoid getting bogged down in the details of implementation.

Best, Average and Worst Case Analysis

For a lot of problems, there's no one-size-fits-all solution. The best algorithm to choose depends on several factors

The Problem Itself: What exactly are you trying to solve? Different problems demand different approaches.
The Data: What kind of data are you working with? How are they distributed? Algorithms can be optimized for specific types of data and distributions.
The Algorithms: How do the different algorithms your are considering behave under different circumstances? Some might be faster for datasets, while others are more appropriate for larger ones.

To help navigate this decision-making process, algorithms are usually presented with three different common scenarios in mind.

Worst Case: This identifies a category of problem instances where an algorithm performs at its absolute slowest. Instead of pinpointing a single, specific input, algorithm designers usually describe characteristics of the input data that hinder the algorithm's efficiency. This could be things like a large input size , specific data patterns, or extreme outlier values.

Average Case: This focuses on the expected performance of an algorithm when faced with typical, randomly generated problem instances. While some specific cases might take longer due to unique characteristics, most instances should fall within a predictable range of execution time. This measure reflects the performance that a typical user can anticipate when using the algorithm.

Best Case: The best-case scenario represents a special class of problem instances that allow an algorithm to shine, performing at its absolute fastest with minimal work. These are often idealized or uncommon scenarios that rarely occur in practice.

Worst Case

When dealing with a fixed-size input, a computer program's execution time can still vary greatly depending on the specific data it receives. The worst-case execution time is the absolute longest it could take to run, considering all possible input scenarios of that size.

We focus on the worst-case scenario because it represents the upper limit of a program's slowness. This is important for applications where timeliness is critical. Additionally, analyzing the worst case is often easier than dealing with the average or best cases, providing valuable insights into overall efficiency.

To express it more formally, let's consider $S_n$ as the set of all problem instances $s_i$ with a size of $n$ . The function $t(s_i)$ measures the amount of work (computational steps or time) an algorithm takes to process a specific instance $s_i$ . The worst-case performance of $S_n$ , denoted as $T_{wc}(n)$ , is the maximum value of $t(s_i)$ across all instances of $S_n$ .

In simpler terms, $T_{wc}(n)$ represents the absolute longest time the algorithm could possibly take to solve any problem instance of size n. This worst-case behavior provides an upper bound on the algorithm's running time, guaranteeing that it will never take longer than this amount of time, regardless of the specific input. The rate at which $T_{wc}(n)$ grows as n increases defines the worst-case time complexity of the algorithm.

Average Case

Imagine a massive telephone system built to handle a huge number of phones (let's say $n$ phones). In the absolute worst-case scenario, we need to be prepared for a situation where half the people ( $n2\frac{n}{2}$ ) decide to call the other half $n2\frac{n}{2}$ simultaneously. This scenario would put immense strain on the system, and while it might not crash completely, building a system to handle this extreme case would be incredibly costly.

In reality, the chances of this exact scenario happening are very, very slim. It's highly unlikely that everyone would call a completely different person at the same time. Instead of building an exorbitantly expensive system to handle this improbable worst-case scenario, we can take a more practical approach. We can design a cheaper system and then use mathematical tools to analyze the likelihood of it crashing due to overload under more realistic conditions.

To do this, we assign probabilities to different scenarios or instances of phone calls. Each instance $s_i$ has a size $n$ (representing the total number of phones), and we give each instance a probability between 0 and 1. The sum of all probabilities for all possible instances of size $n$ must equal 1.

For each set of problem instances with size $n$ (represented as $S_n$ ), we assign a probability distribution $Pr{s_i}$ . This means each individual instance $s_i$ within $S_n$ gets a probability value between 0 and 1. These probability values reflect the likelihood of encountering each specific instance. Importantly, the sum of all these probabilities across all instances in $S_n$ must add up to 1, ensuring that we've covered all possible scenarios.

More formally, we can express this relationship as:

\sum_{s_i \in S_n} Pr \lbrace s_i \rbrace = 1

This equation simply states that when you add up the probabilities of all possible instances within a given set Sn, the total probability must equal 1 (or 100%).

If $t ()$ measures the work done by an algorithm on each instance, then the average case work done by an algorithm on $S_n$ is:

T_{ac}(n) = \sum_{s_i \in S_n} t(s_i) Pr \lbrace s_i \rbrace

In simpler terms, the average-case work done by an algorithm on a set of problem instances $S_n$ is calculated by taking the work done on each specific instance ( $t(s_i)$ ) and multiplying it by the probability of that instance actually occurring ( $\lbrace s_i \rbrace$ ). If the probability of an instance is zero, it doesn't contribute to the overall average work.

The average-case work, denoted as $T_{ac}(n)$ , gives us a more realistic picture of how an algorithm performs in real-world scenarios. It's a weighted average that takes into account the likelihood of encountering different problem instances. The rate at which $T_{ac}(n)$ grows as $n$ increases defines the average-case complexity of the algorithm, indicating how the algorithm's performance scales with the size of the input.

Best Case

Understanding the best-case scenario for an algorithm, even though it rarely happens in real life, can be quite insightful. It gives us a glimpse into the ideal conditions that would allow the algorithm to perform optimally.

Let's take Sequential Search as an example. The best-case scenario for this algorithm is when the value you're looking for (let's call it v) happens to be the very first element in the list. It finds it right away, making it super efficient in this specific situation.

Now, imagine a slightly different approach called Counting Search. This algorithm counts how many times the value v appears in the list. If the count is zero, it means the item wasn't found. Otherwise, it's there.

Here's the key difference: Counting Search always goes through the entire list, even if it finds v early on. So, even though its worst-case performance is the same as Sequential Search (O(n)), its best-case performance is also O(n). This means Counting Search isn't able to take advantage of situations where it could potentially finish faster, like when the value is found at the beginning.

In essence, while the best-case scenario might be rare, it reveals the potential for an algorithm to be even more efficient under optimal conditions. Understanding this can guide us in making decisions about which algorithm to choose for a particular task, especially if we know something about the characteristics of the data we're dealing with.

Lower and Upper Bounds

To simplify how we talk about "Big O" notation in this series, we'll focus on classifying how an algorithm behaves as it tackles problems of growing size, represented by $n$ . We're going to use the notation $O (f (n))$ , where $f (n)$ is typically a function like $n$ , $n^2$ , $n^3$ , or $O^n$ .

Let's, for example, consider an algorithm where the worst-case performance grows in direct proportion to the size of the input data, $n$ , once that input gets big enough. This means there's a positive constant $c$ and a threshold size $n_0$ such that the time it takes for the algorithm to run, represented by $t (n)$ , is always less than or equal to $c$ multiplied by $n$ for all values of $n$ greater than $n_0$ , or $\leq c \times n$ , $∀n>n0\forall n \gt n_0$ . In simpler terms, the algorithm's runtime is "linear", meaning that its execution time increases proportionally with the input size. It implies that as the size of the input data grows, the time it takes to process that data also grows linearly.

In this scenario, we classify the algorithm's worst-case performance as $O (n)$ , where $n$ represents the input size. This notation, called Big O notation, describes the upper bound of the algorithm's time complexity.

Now, imagine that the same algorithm's best-case performance is also directly proportional to the input size. This means there's a different positive constant $c$ and a different threshold $n_0$ such that $t (n)$ is always greater than or equal to $O (n)$ multiplied by $O (n)$ for all $O (n)$ greater than $n_0$ , or in mathematical terms $\geq c \times n$ , $∀n≥n0\forall n \ge n_0$ . In other words, the algorithm's runtime never gets faster than linear, even in the best-case scenario.

In this situation, we classify the algorithm's best-case performance as $Ω (n)$ , using Big Omega notation. This indicates the lower bound of the algorithm's time complexity.

To summarize, the actual formal notation is as follows:

The lower bound for the execution time of an algorithm is classified as $Ω (f (n))$ and corresponds to the best-case scenario.
The upper bound for execution time of an algorithm is classified as $O (f (n))$ and corresponds to the worst-case scenario.

Summary

In this post we discussed how we analyze algorithms and what is the Big O Notation. In the next post we are going to discuss performance families of the algorithms. See you on the next post!

DEV Community: Kostas Kalafatis

Using Semaphores for Producer-Consumer Problems

Introduction

But where is my ConcurrentQueue<T>?

Understanding the Problem

What is a Shared Buffer?

What is a Consumer?

What is a Producer?

What is a Semaphore?

What is a Mutex?

Using Semaphores and Locks

Implementation

The Shared Buffer

The ProduceAsync method

The ConsumeAsync method

The Consumer Class

The Producer Class

Demonstrating the Producer-Consumer System

Enhancements

Adding Random Delays to Simulate Real-Life Behavior

The Illusion of First-Come, First-Served

Conclusion

Configuration of Services

Configuring Services with Docker Compose

Service Dependency

Service Dependency with Docker Compose

Summary

Using Docker Compose to Build Environments

Docker Compose CLI

Docker Compose CLI Commands

Docker Compose File

Name

Services

Networks

Volumes

Creating a Web Server with Docker Compose

Summary

Multi-Stage Dockerfiles

Normal Docker Builds

The Builder Pattern

Building a Docker Image with the Build Pattern

Introduction to Multi-Stage Dockerfiles

Building a Docker Image with a Multi-Stage Docker Build

Summary

Storing and Publishing your Docker Images

Moving Docker Images Manually

Storing and Deleting Images from Docker Hub

Storing Docker Images in Docker Hub

The Docker Registry

Creating a Local Docker Registry

Summary

Docker Image Naming and Tagging

Tagging Docker Images

Using the Latest Tag in Docker

Issues when Using Latest

Docker Image Tagging Policies

Automating Image Tagging

Conclusion

Dockerfile Best Practices Explained - Interlude Post 2

Using an Appropriate Parent Image

Using a Non-Root User for Better Security

Using .dockerignore

Minimizing Layers

Don't Install Unnecessary Tools

Creating Base Docker Images

Creating Custom Base Docker Images

The SCRATCH Image

Using the SCRATCH Image

Conclusion

Understanding CMD, ENTRYPOINT and RUN in Docker - An Interlude Post

The RUN Directive

The CMD Directive

The ENTRYPOINT Directive

So CMD or ENTRYPOINT?

Directive Description and Use Cases

Unix Signaling and the PID 1

Shell and exec forms

Shell Form

Exec Form

Key Differences between Shell and Exec

But where is my `ConcurrentQueue<T>`?

The `ProduceAsync` method

The `ConsumeAsync` method

Run the container image with `CMD traceroute`

Run the container image with `ENTRYPOINT traceroute`