DEV Community: Iñigo Etxaniz

Running Ollama in a Container Without Internet Access

Iñigo Etxaniz — Fri, 14 Feb 2025 11:21:47 +0000

Prerequisites

This setup has been tested on Ubuntu. To enable GPU acceleration, install the NVIDIA Container Toolkit before running the project:

sudo apt install nvidia-container-toolkit
sudo systemctl restart docker

Motivation

With the growing demand for AI-powered applications, running large language models locally is becoming increasingly common. However, in some cases, it is essential to ensure confidentiality and prevent unintended data leaks by running Ollama in a fully sandboxed environment without internet access. This setup allows organizations or individuals to maintain full control over their AI models while still leveraging their capabilities.

Additionally, this setup enables fine-grained control over networking, allowing a dedicated model-downloading instance to have internet access while keeping the model-serving instance completely isolated.

Project Overview

This project provides a Docker-based solution to run Ollama models in an isolated environment while keeping the model-downloading instance in a normal network. The architecture consists of:

Ollama Runner: Runs AI models in a sandboxed network without internet access.
Ollama Updater: Responsible for downloading models from the internet and sharing them with the runner.
Nginx Reverse Proxy: Bridges the network gap, allowing access to the runner while maintaining isolation.

Tested with VS Code's Continue Extension

To ensure practical usability, I tested this configuration with the Continue extension in Visual Studio Code, and it worked correctly. This means developers can integrate the local Ollama instance seamlessly with their workflow while keeping the model execution isolated.

Demonstrating Network Isolation

One of the key aspects of this project is network isolation, ensuring that the Ollama Runner does not have internet access while the Ollama Updater does. To verify this, we can run the following script:

bash network-isolation-test-script.sh

This script performs the following tests:

Check internet access:
- The ollama-updater container should be able to reach google.com.
- The ollama-runner container should be fully isolated and unable to reach the internet.
Check Ollama API accessibility:
- The ollama-runner should be able to access its API internally.
- The ollama-updater should be able to access its own API.

Expected output:

✓ ollama-updater network has internet access (Expected)
✓ ollama-runner network is properly isolated (Expected)
✓ Ollama API accessible from runner network
✓ Ollama API accessible from updater network

This confirms that the model execution environment remains secure and offline, while updates can still be managed efficiently.

Technical Setup

Key Features

Fully sandboxed Ollama model execution
Separate container for downloading models with internet access
Nginx reverse proxy for controlled access
Custom small Docker networks (/28) to ensure isolation
GPU acceleration via NVIDIA toolkit (if available)

Deployment Steps

Install Docker and NVIDIA Container Toolkit (if using GPU).
Clone the repository:

   git clone https://github.com/ietxaniz/ollama-local.git
   cd ollama-local

Start the services:

   docker-compose up -d

Verify running containers:

   docker ps

Use the Ollama Runner to serve models, while the Ollama Updater manages downloads.

Managing Models

To list available models:

docker exec -it ollama-runner ollama ls

To pull new models:

docker exec -it ollama-updater ollama pull deepseek-r1:14b

To check active models:

docker exec -it ollama-runner ollama ps

Future Possibilities

While this project provides a robust foundation for securely running Ollama, I am considering extending it further to explore Retrieval-Augmented Generation (RAG). This could enhance local AI capabilities by integrating external knowledge bases while keeping execution sandboxed.

If you're interested in contributing or have suggestions, feel free to open an issue in the GitHub repository.

Conclusion

This setup ensures that AI models can run securely in an isolated environment while maintaining the flexibility to update and manage them efficiently. Whether for security reasons or simply to experiment with self-hosted AI, this approach provides a reliable solution.

Would you like to see more extensions of this project? Let me know in the comments!

Enhancing Rust Error Handling: Macro to add Program Flow Trace to your applications

Iñigo Etxaniz — Sun, 21 Jul 2024 09:01:28 +0000

Introduction

When developing Rust applications, error handling is a crucial aspect that can significantly impact the debugging process and overall maintainability of your code. While Rust's error messages are generally informative about why an error occurred, they often fall short in providing insight into the program's flow leading up to that error. In complex applications with multiple layers of function calls, understanding the sequence of events that led to an error can be challenging.

To address this challenge, I have developed an approach that enhances Rust's error handling capabilities by incorporating program flow information directly into error messages. This method combines custom traits, macros, and the anyhow crate to provide a more comprehensive error reporting system.

Here's a snippet that demonstrates how clean and straightforward the implementation can be:

pub fn run() -> Result<()> {
    let input = debug_err!(parse_argument(), "Error parsing command-line argument")?;
    let result = debug_err!(intermediate_function(input))?;
    println!("Success: {}", result);
    Ok(())
}

As you can see, this approach allows you to add debug information to your error handling with minimal impact on your code's readability. The debug_err! macro seamlessly integrates with Rust's ? operator, maintaining the concise style that Rust developers appreciate. It also provides flexibility by allowing optional custom error messages:

let result = debug_err!(some_operation())?; // Adds file and line information
let result = debug_err!(some_operation(), "Custom error message")?; // Adds custom message along with file and line

Let's look at how this solution works in practice with a few examples:

Successful execution:

    $ cargo run -- 12
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 0.00s
    Running `target/debug/rust_error_handling 12`
    Success: you guessed correctly

Error in main operation:

   $ cargo run -- 89
   Finished `dev` profile [unoptimized + debuginfo] target(s) in 0.00s
   Running `target/debug/rust_error_handling 89`
   Error: 
   (at src/some_module/mod.rs:28): Error in intermediate_function
   (at src/some_module/mod.rs:23): Invalid input: 89

Error in parsing command-line argument:

   $ cargo run -- io
   Finished `dev` profile [unoptimized + debuginfo] target(s) in 0.00s
   Running `target/debug/rust_error_handling io`
   Error: Error parsing command-line argument
   (at src/some_module/mod.rs:27): Failed to parse input as integer: invalid digit found in string

As you can see, in cases where an error occurs, our enhanced error messages provide the reason for the error along with the file and line number, offering a clear picture of how the program reached the error state. The custom messages, when provided, offer additional context to aid in debugging.

In the following sections, we'll dive into the implementation details of this error handling approach, explore its benefits and potential drawbacks, and discuss best practices for incorporating it into your Rust projects.

Implementation and Example

Let's walk through the implementation of our enhanced error handling approach and demonstrate its usage with a practical example. We'll start by creating a new Rust project and then build our error handling system step by step.

Begin by creating a new Rust project and adding the necessary dependency:

cargo new rust_error_handling_example
cd rust_error_handling_example
cargo add anyhow

This will create a new project and add anyhow to your Cargo.toml file. Now, let's set up our project structure. Create the following directories and files:

src/
|-- error_handling/
|   |-- mod.rs
|-- some_module/
|   |-- mod.rs
|-- main.rs

Now that our project is set up, let's implement our core error handling logic. Create src/error_handling/mod.rs and add the following code:

use anyhow::{Context, Result};

pub trait AddDebugInfo<T> {
    fn add_debug_info(self, context: Option<&str>, file: &str, line: u32) -> Result<T>;
}

impl<T> AddDebugInfo<T> for Result<T> {
    fn add_debug_info(self, context: Option<&str>, file: &str, line: u32) -> Result<T> {
        self.with_context(move || {
            if let Some(ctx) = context {
                format!("{}\n(at {}:{})", ctx, file, line)
            } else {
                format!("\n(at {}:{})", file, line)
            }
        })
    }
}

#[macro_export]
macro_rules! debug_err {
    ($result:expr) => {
        $result.add_debug_info(None, file!(), line!())
    };
    ($result:expr, $msg:expr) => {
        $result.add_debug_info(Some(&format!($msg)), file!(), line!())
    };
}

This code defines the AddDebugInfo trait and implements it for Result<T>. The debug_err! macro provides a convenient way to add debug information to errors, with or without custom messages.

Next, let's create a module that demonstrates our error handling approach. Create src/some_module/mod.rs, with following code:

use anyhow::{Context, Result, anyhow};
use std::env;
use crate::error_handling::AddDebugInfo;
use crate::debug_err;

pub fn parse_argument() -> Result<i32> {
    let args: Vec<String> = env::args().collect();
    if args.len() != 2 {
        return Err(anyhow!("Expected exactly one argument"));
    }
    args[1].parse::<i32>()
        .context("Failed to parse input as integer")
}

pub fn some_fallible_operation(input: i32) -> Result<String> {
    match input {
        12 => Ok("you guessed correctly".to_string()),
        _ => Err(anyhow!("Invalid input: {}", input)),
    }
}

pub fn intermediate_function(input: i32) -> Result<String> {
    debug_err!(some_fallible_operation(input), "Error in intermediate_function")
}

pub fn run() -> Result<()> {
    let input = debug_err!(parse_argument(), "Error parsing command-line argument")?;
    let result = debug_err!(intermediate_function(input))?;
    println!("Success: {}", result);
    Ok(())
}

This module demonstrates various ways to use our error handling pattern. In parse_argument(), we use the standard context() method from anyhow to add a simple error message. In intermediate_function(), we use debug_err! with a custom message. The run() function shows how to chain multiple operations using debug_err!, adding contextual information at each step.

Finally, update src/main.rs to use our new module:

mod error_handling;
mod some_module;

fn main() {
    if let Err(e) = some_module::run() {
        eprintln!("Error: {:#}", e);
        std::process::exit(1);
    }
}

Now you can run the example with different inputs to see how the error handling works:

cargo run -- 12
cargo run -- 89
cargo run -- io

When an error occurs, it will be displayed with the added context and location information, making it easier to trace and debug issues in your Rust applications. This pattern allows you to add contextual information at each step, creating a trail of debug information that helps trace the error's origin and path through the program.

Reflections on the Approach

Coming from a Go background, I often find myself struggling with error handling in Rust. Despite Rust's powerful type system and the Result type, I frequently encounter situations where tracing the origin and path of errors through my codebase is more challenging than it should be. This struggle has led me to explore alternative approaches, ultimately resulting in the error handling pattern presented in this article.

The primary motivation behind developing this approach is to enhance the debugging process in my Rust applications. I want a method that can provide detailed context about where and how errors occur, significantly reducing the time and effort required to trace and resolve issues. After implementing this pattern, I have decided to change how I manage errors in my future Rust projects and potentially refactor existing codebases to incorporate this new approach.

One of the most compelling aspects of this method is the automatic inclusion of file and line information in error messages. I anticipate this feature will be particularly valuable when dealing with complex codebases, as it should immediately pinpoint the exact location where an error originates or is propagated. The flexibility to add optional custom messages also allows for the inclusion of rich, contextual information about the program's state at the time of the error.

The debug_err! macro is a key component of this new approach. I have designed it to integrate seamlessly with Rust's ? operator, allowing for concise error propagation while still maintaining detailed error context. My hope is that this will lead to more readable and maintainable code, as error handling becomes less intrusive and more informative.

However, I'm aware that this approach isn't without potential drawbacks. There might be a slight performance overhead associated with capturing and storing additional context information. While I expect this overhead to be negligible in most applications, it could become a consideration in performance-critical sections of code or in systems with extremely high throughput.

As I move forward with adopting this pattern, I plan to use it judiciously and in balance with other error handling techniques. For instance, I intend to use debug_err! with custom messages at significant points in the program flow or when crossing module boundaries. For simpler, more localized error cases, I might stick with anyhow's standard context method or even Rust's built-in ? operator without additional context.

When it comes to balancing detail and performance, I'm considering a strategy of using more detailed error messages in development and testing environments, while potentially scaling back the verbosity in production builds if performance becomes a concern. This could be achieved through conditional compilation or runtime flags.

Integrating this approach with existing error handling patterns in my projects will require careful consideration. In codebases that already have established error types or handling mechanisms, I plan to gradually introduce this method, starting with areas that would benefit most from enhanced debugging information.

While I haven't extensively used this error handling approach in production yet, I'm optimistic about its potential to improve my Rust development experience. I look forward to seeing how it performs in real-world scenarios and how it might evolve based on practical usage. As with any new technique, I expect there will be a learning curve and possibly some adjustments needed along the way. However, I believe the potential benefits in terms of improved debugging capabilities and code maintainability make it worth exploring and implementing in my future Rust projects.

Conclusion

The enhanced error handling approach I've presented in this article offers a solution to a challenge I've often encountered in Rust development. By incorporating program flow information directly into error messages, I believe I can significantly improve my debugging process and the overall maintainability of my Rust applications. This method provides detailed context by automatically including file and line information in error messages, while offering the flexibility of optional custom messages for rich, contextual information. I've designed it to integrate seamlessly with Rust's ? operator and existing error handling patterns, ensuring that I can adopt this approach without disrupting my current coding practices. I anticipate this will result in a substantial reduction in the time and effort required to trace and resolve issues, especially in the complex codebases I work with. While I'm aware of potential minor performance considerations in certain scenarios, I believe the improvements in code readability, maintainability, and debugging efficiency will make this approach a valuable addition to my Rust developer toolkit. As I prepare to implement and refine this error handling pattern in my real-world projects, I look forward to gaining practical insights and making further optimizations. My goal is to make my Rust development more efficient and enjoyable, allowing me to focus on solving problems rather than tracking down elusive errors. By sharing this approach, I hope other Rust developers might find it useful and potentially benefit from it in their own work as well.

I am not an expert in Rust, but I've adapted principles I found useful in Go to the Rust ecosystem. I hope this method proves helpful to others facing similar challenges in debugging and maintaining Rust applications. If you have more experience in Rust or suggestions for improving this approach, I would greatly appreciate your feedback. Let's continue to evolve and refine our error handling techniques together.

Introducing CurlDock: Simplify API Testing with Docker and Curl

Iñigo Etxaniz — Mon, 08 Jul 2024 18:26:33 +0000

Hey Dev.to community!

I'm excited to share CurlDock, a lightweight open-source tool I've been working on to simplify API testing and curl script management.

What is CurlDock?

CurlDock combines the power of curl with the convenience of Docker, offering a user-friendly interface for creating, editing, and executing curl commands. Built with Rust and React, it's designed for simplicity and ease of use.

Key Features

🐳 Dockerized: Easy setup and consistent environments
🖥️ User-Friendly Interface: Similar to Postman or Insomnia
📁 Git-Friendly: Store scripts as .sh files for easy version control
🔒 Network Isolation: Access endpoints within your Docker network

Quick Start

Getting started with CurlDock is as easy as running a Docker command:

docker run --name curldock --rm -e SCRIPTSFOLDER="/scripts" -v $(pwd)/scripts:/scripts -e PORT="2080" -p 2080:2080 inigoetxaniz/curldock

Then, just open http://localhost:2080 in your browser!

Why CurlDock?

Simplicity: Focus on core functionality without unnecessary complexity
Local-First: Designed for local development environments
No Authentication: Reduced complexity for improved usability
Developer Control: You manage your own security measures

Try It Out!

I'd love for you to give CurlDock a spin and share your thoughts.

Check out the GitHub repository for more details!

Happy API testing! 🚀

Deferred Redux Update Pattern

Iñigo Etxaniz — Sun, 07 Jul 2024 07:43:12 +0000

Introduction

When developing user interfaces with React, I often find myself juggling between different state management approaches. On one hand, Redux is great for handling large-scale application state, but on the other, useState is perfect for those small, component-specific details.

During a recent project, I was looking for a way to have the best of both worlds - using Redux for the big picture stuff while keeping the nitty-gritty details managed by useState. It wasn't straightforward, and I had to experiment quite a bit before I stumbled upon a pattern that seemed to work well.

After a few iterations and some refinement, I realized I had something that could be useful in future projects. This pattern basically allows components to work with complex state locally, making as many changes as needed, without bombarding Redux with updates for every little change. Instead, it waits until the right moment - like when a form is submitted or a component is about to unmount - before pushing all those changes to the global Redux store.

I thought this approach might be helpful for other developers facing similar challenges, so I decided to share it here. In this article, I'll walk you through the problem this pattern solves and how it works.

The main Use Case

The Deferred Redux Update Pattern is useful in scenarios where:

Your Redux store maintains complex state structures.
Components need to make frequent, granular updates to this state.
These updates don't need to be immediately reflected in the global store.

Consider a form with multiple fields, each represented in Redux. Traditional approaches might dispatch an action for every keystroke, leading to unnecessary re-renders and potential performance issues. With the Deferred Redux Update Pattern, we can buffer these changes locally and only update Redux when the form is submitted or the component unmounts.

Rel-World Example: curldock

To better illustrate the Deferred Redux Update Pattern, let's explore how it's implemented in curldock, an open-source project I am working on. curldock is designed to simplify API testing, allowing developers to have an user interface for interacting with curl.

Curldock's user interface (although not finished) consists of three main components:

A FileExplorer component that allows interacting with a hierarchical list of curl scripts.
A ScriptEditor component that allows users to edit and send API requests.
A ResultViewer component that will allow reviewing the response from the api call.

Here's a simplified view of the application layout:

The application's data flow and state management can be visualized as follows:

In curldock, the Deferred Redux Update Pattern is primarily used in the ScriptEditor component. Here's how it's implemented:

import { useEffect, useRef, useState } from "react";
import { useGetCurlItemByFileId, useUpdateCurlItem } from "@/store/hooks/useCurl"
import { HttpMethod } from "@/store/slices/curlSlice";

export type Header = {
  id: string;
  name: string;
  value: string;
};

export const useScriptEditorData = (fileId: number) => {
  const [initialized, setInitialized] = useState(false);
  const getCurlItem = useGetCurlItemByFileId();
  const updateCurlItem = useUpdateCurlItem();
  const [method, setMethod] = useState<HttpMethod>(HttpMethod.GET);
  const [url, setUrl] = useState('');
  const [headers, setHeaders] = useState<Header[]>([]);
  const [bodyContent, setBodyContent] = useState('');

  const exitCallbackRef = useRef(() => {});

  exitCallbackRef.current = () => {
    if (initialized) {
      console.log(`close ${fileId} - ${url}`);
      updateCurlItem({
        fileId: fileId,
        script: {
          method: method,
          url: url,
          headers: headers.map(h => [h.name, h.value]),
          data: bodyContent,
          options: {
            verbose: true,
            insecure: false,
          }
        }
      });
    }
  };


  useEffect(() => {
    const curlItem = getCurlItem(fileId);
    if (curlItem && curlItem.script) {
      console.log(curlItem);
      const { method, url, headers, data } = curlItem.script;
      setMethod(method);
      setUrl(url);
      setHeaders(headers.map((h, index) => ({ id: index.toString(), name: h[0], value: h[1] })));
      setBodyContent(data || '');
      setInitialized(true);
    }
    console.log(`open ${fileId}`);

    return () => { exitCallbackRef.current(); }

  }, [fileId]);

  return { method, setMethod, url, setUrl, headers, setHeaders, bodyContent, setBodyContent };
}

This implementation allows the ScriptEditor to work with complex state locally, making as many changes as needed without immediately updating Redux. The state is only synchronized with Redux when the component unmounts or when explicitly triggered by the user (e.g., saving the script).

By using this pattern, curldock achieves several benefits:

Better performance and a more responsive user interface, especially when dealing with complex forms and real-time data manipulation in the ScriptEditor.
Cleaner code, as detailed Redux updating function calls are not needed throughout the component logic.
Improved state management, with clear separation between local state and global (Redux) state.

The Problem in Detail

As I dug deeper into implementing this pattern, I ran into a few roadblocks that really made me scratch my head. The main issue I was grappling with was how to trigger the Redux update only when the component disappears, without causing unnecessary updates or running into stale data problems. At first, I thought I could manage everything with useEffect. Seems straightforward, right? But I quickly realized that if all the internal states were managed by useEffect, the cleanup function would fire every time an internal state changed. That wasn't what I wanted at all - it defeated the whole purpose of deferring updates! So, I tried using useCallback, thinking it might solve the problem. But, I ran into similar issues. It was like trying to plug a leak only to find another one popping up elsewhere. For a moment, I considered using useRef to store the state. But then I realized I'd essentially be keeping two copies of the state around. It felt like overkill, and I worried it might lead to synchronization headaches down the line. But, what if I used useRef, not for the state itself, but to manage a closing function? This approach turned out to be the key to solving my update timing problem.

But just when I thought I had it all figured out, React's Strict Mode threw me another curveball. In Strict Mode, useEffect gets called twice in each render. At first, this seemed like a minor inconvenience, but it quickly turned into a real problem. You see, the first cleanup call happens before the states are updated from Redux. If I didn't account for this, I risked storing empty data in Redux. Not exactly the behavior I was aiming for! After some more tinkering, I found a solution: a simple state that checks if all other states have been updated. It's a small addition, but it made all the difference in handling Strict Mode's double-call behavior.

These challenges really drove home the need for a more robust way to manage state updates in React-Redux applications. The solution I eventually landed on might not be perfect, but it addressed these pain points and gave me a pattern I could rely on. Let me show you how it all came together.

The Deferred Redux Update Pattern

The Deferred Redux Update Pattern addresses these challenges by introducing a mechanism to delay Redux updates until they're necessary. Here's an overview of the key components:

Local State Management: Use React's useState for managing state within the component.
Cleanup Function with useRef: Store the cleanup function in a useRef to ensure it always has access to the most recent state.
Initialization Flag: Implement an isInitialized flag to handle React Strict Mode's double-invocation behavior.
Deferred Redux Update: Trigger the Redux update in the cleanup function, which runs when the component unmounts or when explicitly called.

Here's a code example demonstrating the pattern:

import { useEffect, useRef, useState } from "react";
import { useDispatch } from "react-redux";

export const useDeferredReduxUpdate = (initialState, updateAction) => {
  const [state, setState] = useState(initialState);
  const [isInitialized, setIsInitialized] = useState(false);
  const dispatch = useDispatch();

  const cleanupRef = useRef(() => {});

  cleanupRef.current = () => {
    if (isInitialized) {
      dispatch(updateAction(state));
    }
  };

  useEffect(() => {
    setIsInitialized(true);
    return () => cleanupRef.current();
  }, []);

  const storeCurrentState = cleanupRef.current;

  return [state, setState, storeCurrentState];
};

Additionally, the hook exposes a storeCurrentState function, providing flexibility to trigger a Redux update on demand. This function can be called to push the local state to Redux at any time, such as when a user activates a "Save" button.

Benefits and Use Cases

The Deferred Redux Update Pattern offers several key benefits:

Improved Performance: By reducing the number of Redux dispatches, we minimize unnecessary re-renders and state calculations.
Cleaner Code: The pattern encapsulates the logic for managing local state and syncing with Redux, leading to more maintainable components.
Better User Experience: With fewer global state updates, the application can feel more responsive, especially when dealing with form inputs or real-time data manipulation.

This pattern is particularly useful in scenarios such as:

Complex forms with multiple fields
Data visualization tools with frequent updates
Real-time collaborative features where immediate global updates aren't necessary
Any situation where you need to balance local interactivity with global state consistency

By implementing the Deferred Redux Update Pattern, you can significantly optimize your React-Redux applications, especially those dealing with complex state management scenarios.

Conclusion

The Deferred Redux Update Pattern offers a powerful solution for managing complex state in React applications. By intelligently balancing local state management with global Redux updates, this pattern enhances performance, improves code clarity, and provides a smoother user experience. Whether you're working on complex forms, real-time data visualization, or collaborative features, this pattern can help enhance your state management approach. As we've seen with the curldock example, implementing this pattern can lead to more efficient and maintainable React-Redux applications. I encourage you to experiment with this pattern in your own projects and see how it can optimize your state management strategy.

How to implement HTTPS in Local Networks Using Let's Encrypt

Iñigo Etxaniz — Mon, 25 Mar 2024 10:58:10 +0000

Introduction

While browsing the internet, you may have noticed that URLs sometimes begin with "HTTPS" instead of "HTTP." This extra "S" signifies a secure connection, where your communication with the website is encrypted. This encryption ensures that only you and the website can decipher the exchanged information, protecting against "Man in the Middle" (MitM) attacks. Such attacks involve interceptors eavesdropping or altering messages without detection. HTTPS scrambles the data into an indecipherable format for anyone intercepting it.

The foundation of HTTPS is the certificate authority (CA) system. CAs act as trusted validators on the internet, verifying website identities and issuing digital certificates, much like ID cards for websites. This system hinges on global recognition and trust in CAs. When a website presents a certificate signed by a known CA, your browser accepts it as a signal that the website is secure.

Yet, when it comes to securing local networks—such as those in your home or office—using global CAs presents challenges. These authorities cannot directly verify ownership of servers within private networks, which is crucial for issuing trusted certificates. Setting up a Private CA is an alternative, but it introduces complexities, such as the need for manual trust setup on new devices and maintaining security against vulnerabilities.

Furthermore, the threat of MitM attacks isn't limited to the vast internet; it's arguably more prevalent within local networks. Shared networks, like those in offices or public Wi-Fi spots, are fertile ground for malicious users to intercept unencrypted traffic. This risk underscores the necessity of HTTPS, ensuring communications are confidential and integrity is maintained, even in internal networks.

The approach we're about to explore, inspired by Joshua's method in securing web applications on private networks, involves using Certbot and Let's Encrypt for a streamlined solution. This method sidesteps direct server connection requirements by using DNS verification, making it suitable for internal networks. We begin by securing a domain name, setting up Certbot within Docker for certificate issuance, and finally configuring an Nginx web server to utilize the SSL/TLS certificate. This process, detailed further in the article, not only secures our network but does so in a cost-effective and accessible manner.

To facilitate the implementation of the HTTPS setup described in this article, I've prepared a comprehensive repository containing all the scripts. You can find everything you need to get started at https://github.com/ietxaniz/https-setup-local-networks.

Understanding HTTPS and Its Importance

HTTPS is crucial for internet security, transforming HTTP into a secure channel using SSL/TLS protocols. These protocols encrypt data in transit, ensuring confidentiality and integrity. The process relies on asymmetric encryption, using a public key for encryption and a private key for decryption. Even if data is intercepted, it remains unreadable without the private key.

Security extends beyond encryption. An SSL/TLS certificate acts as proof that you're communicating with the intended server. Issued by trusted Certificate Authorities (CAs) after rigorous verification, it’s akin to verifying the authenticity of an ID card before granting access. This protects users from potential frauds and attacks.

At the beginning of each secure session, a handshake process lays the groundwork for a secure connection. This process includes server authentication, key exchange, and secure communication using symmetric encryption. The handshake ensures that the communication channel is private, authenticated, and untampered with, marking the transition from the computational rigor of asymmetric encryption to the efficiency of symmetric encryption.

HTTPS signifies trustworthiness, indicated by the padlock icon in the browser's address bar. This trust is crucial for websites handling sensitive information, as it assures users that their data is safe. Periodic renewal of SSL/TLS certificates acts as a continual check, ensuring encryption standards evolve to meet new security challenges and technological advances.

In the broader digital landscape, HTTPS is not just a technical specification; it is a foundational element of online security, privacy, and trust. Its role in safeguarding data integrity, authenticating website identities, and ensuring confidentiality is paramount. Understanding these principles is essential for implementing robust security measures that protect against digital vulnerabilities.

The Role of Certificate Authorities (CAs)

Certificate Authorities (CAs) are crucial to the HTTPS ecosystem, ensuring secure and trustworthy communications between browsers and websites. These trusted entities issue digital certificates, essential for establishing encrypted connections in the SSL/TLS protocol.

Digital certificates serve as electronic "passports" for websites, proving their identity to visitors. When you access a secure website, your browser checks the site's digital certificate, verifying it was issued by a CA that the browser trusts. This verification process prevents attackers from masquerading as legitimate websites, a common tactic in phishing attacks.

Understanding the distinction between global CAs and private CAs is essential. Global CAs, such as Let's Encrypt, VeriSign, and Comodo, are widely recognized and trusted, issuing certificates automatically trusted by most web browsers and operating systems. However, their use in local networks can present challenges, as they cannot directly validate servers within private networks.

Private CAs, in contrast, are internal to an organization and not automatically trusted by external systems. They issue certificates for internal servers and applications, where external validation is not feasible or necessary. While private CAs offer flexibility in securing internal communications, they require manual trust management and secure handling of the CA’s private key to prevent unauthorized certificate issuance.

In conclusion, using global Certificate Authorities for private networks simplifies the certification process and enhances security without compromising trustworthiness or administrative overhead. This approach combines the reliability of global CAs with the control of internal network management, eliminating the disadvantages traditionally associated with private CAs.

Let's Encrypt and Certbot

Let's Encrypt offers an automated, open service for obtaining digital certificates, simplifying web security. Founded on the principle of accessible privacy and security, it streamlines the process of certificate renewal. This initiative benefits both small and large organizations by securing local networks without significant costs or administrative burdens.

Certbot, a free, open-source software tool, automates obtaining, installing, and renewing certificates from Let's Encrypt. Designed to work with various web servers and operating systems, Certbot ensures that secure HTTPS is attainable regardless of the technical environment.

In this tutorial, we use the Docker version of Certbot, leveraging Docker's ability to simplify infrastructure management. Using Docker, we avoid complex installations, utilizing containers for a clean and easily replicable setup. We extend the base Certbot image to include curl, enabling more sophisticated scripts and interactions during the certificate setup process.

First, create a Dockerfile instructing Docker on how to build our customized Certbot image:

FROM certbot/certbot:latest
RUN apk add --update --no-cache jq curl

Accompany this Dockerfile with a build script, build.sh, automating the image creation process:

#!/bin/bash
docker build -t certbot-with-curl .

Save these files in a dedicated directory. Ensure you grant execution permissions to the build script by running chmod +x build.sh in your terminal. Execute the build script by navigating to the directory containing build.sh and running ./build.sh. This command builds a new Docker image named certbot-with-curl, including all necessary tools for certificate management tasks.

With these preparations complete, we proceed to the certificate generation process, ensuring a versatile and robust environment for HTTPS configuration tasks.

DNS Management

Securing your web application with HTTPS requires proving ownership of your domain, making DNS management essential. This step is particularly critical in environments with dynamic IP addresses or private networks. While various DNS providers are available, this guide focuses on using DuckDNS, a user-friendly, cost-effective option for linking your domain name to your server's IP address.

First, create an account on DuckDNS, which supports OAuth for quick sign-up using existing social media accounts. After creating your account, set up your domain. For this guide, "inner-private" is the example domain name. Follow these steps to configure your domain:

Log into DuckDNS and navigate to the domain management page. Enter your chosen domain name in the designated field and add the domain. Manually change the automatically detected public IP address to your specific private IP address. Save the changes.

To verify domain ownership, Let's Encrypt performs a DNS challenge, requiring a specific TXT record in your domain's DNS settings. The add_duckdns_txt_record.sh script automates updating your domain's TXT record via DuckDNS's API:

#!/bin/sh
DUCKDNS_TOKEN="YOUR_TOKEN"
DOMAIN="inner-private"
VALIDATION_STRING="$CERTBOT_VALIDATION"
curl "https://www.duckdns.org/update?domains=_acme-challenge.$DOMAIN&token=$DUCKDNS_TOKEN&txt=$VALIDATION_STRING"
sleep 90

Prepare and test the script by replacing "YOUR_TOKEN" with your actual DuckDNS token, making it executable, and verifying the TXT record update. This setup simplifies securing your web application with HTTPS by automating the DNS challenge process.

Creating and Storing the Certificates

Storing SSL/TLS certificates in Docker volumes offers a convenient and secure way to manage these sensitive files. For this setup, use two Docker volumes: https_keys for storing keys and https_challenge for temporary files used by Certbot. Follow these steps to generate and renew your certificates:

Create the necessary Docker volumes using the create_volumes.sh script:

#!/bin/bash
docker volume rm https_keys https_challenge
docker volume create https_keys
docker volume create https_challenge

Generate a certificate using a customized Certbot Docker container. The create_certificates.sh script initiates the certificate generation process, including hooks for DNS challenges required for domain verification:

#!/bin/bash
DOMAIN="inner-private.duckdns.org"
docker run -it --rm \
    -v https_keys:/etc/letsencrypt \
    -v https_challenge:/var/lib/letsencrypt \
    -v "$(pwd)/scripts:/scripts" \
    --env CERTBOT_DOMAIN="$DOMAIN" \
    certbot-with-curl \
    certonly \
    --staging \
    --verbose \
    --manual \
    --preferred-challenges dns \
    --manual-auth-hook "/scripts/add_duckdns_txt_record.sh $DOMAIN" \
    --manual-cleanup-hook "/scripts/remove_duckdns_txt_record.sh $DOMAIN" \
    -d "$DOMAIN" \
    --non-interactive \
    --agree-tos \
    --email me@inner-private.duckdns.org

After verifying your setup in the staging environment, remove the --staging flag from the script and re-run it to obtain a production certificate. Certificates from Let's Encrypt are valid for 90 days, so set up a cron job to automate renewal by periodically re-executing this script.

This approach to certificate management with Docker and Certbot simplifies securing applications. By automating certificate issuance and renewal, you ensure continuous protection for your domains with minimal overhead. Always test changes in the staging environment before applying them to production to avoid disruptions and ensure a smooth deployment.

Testing with Nginx

After setting up HTTPS using Certbot and Let’s Encrypt, deploy a web server to serve your content securely over HTTPS. This section provides an example of hosting a simple web server within an internal network using an Nginx container and a Let's Encrypt signed certificate.

First, create a directory named after your domain (e.g., inner-private.duckdns.org) and inside it, create an index.html file with the following content:

<!DOCTYPE html>
<html lang="en">
<head>
  <meta charset="UTF-8">
  <meta name="viewport" content="width=device-width, initial-scale=1.0">
  <title>Internal</title>
</head>
<body>
  <h1>Internal web service with HTTPS.</h1>
</body>
</html>

Next, configure Nginx to serve your website securely and redirect HTTP traffic to HTTPS. Create an Nginx configuration file named inner-private.duckdns.org.conf with the following content:

server {
  listen 80;
  server_name inner-private.duckdns.org;
  location / {
    return 301 https://$host$request_uri;
  }
}

server {
  listen 443 ssl;
  server_name inner-private.duckdns.org;

  ssl_certificate /etc/letsencrypt/live/inner-private.duckdns.org/fullchain.pem;
  ssl_certificate_key /etc/letsencrypt/live/inner-private.duckdns.org/privkey.pem;

  location / {
    root /var/www/inner-private.duckdns.org;
    index index.html index.htm;
  }

  ssl_protocols TLSv1.2 TLSv1.3;
  ssl_prefer_server_ciphers on;
  ssl_ciphers "ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384";
  ssl_session_cache shared:SSL:1m;
  ssl_session_timeout 10m;
}

Finally, use Docker to run Nginx and serve your site. The script below sets up and starts the Nginx server in a container, mounting the necessary volumes for the SSL certificates, website content, and Nginx configuration:

#!/bin/bash
docker run \
  --name rev-proxy \
  -p 80:80 \
  -p 443:443 \
  -v https_keys:/etc/letsencrypt \
  -v https_challenge:/var/lib/letsencrypt \
  -v $(pwd)/nginx:/etc/nginx/conf.d \
  -v $(pwd)/inner-private.duckdns.org:/var/www/inner-private.duckdns.org \
  nginx:1.25.1-alpine3.17-slim

This script starts an Nginx container with the necessary configurations, making your site accessible over HTTPS.

Conclusion

This article demonstrates how to combine Certbot, Let's Encrypt, and Docker with an Nginx setup to secure web applications on private networks. By highlighting the importance of HTTPS and providing a step-by-step guide, it equips readers to enhance network security. Leveraging automation and trusted global CAs, this approach makes advanced security measures accessible and manageable.

Given the dynamic nature of cybersecurity, continuous improvement and community feedback are essential. If any inaccuracies are noticed or if there are suggestions for enhancing the security aspects discussed, the community's input would be greatly appreciated.

Practical PostgreSQL Logical Replication: Setting Up an Experimentation Environment Using Docker

Iñigo Etxaniz — Sun, 11 Feb 2024 11:56:39 +0000

Introduction

In the world of database management, ensuring data consistency, availability, and scalability is paramount. Among the myriad strategies to achieve these goals, logical replication in PostgreSQL stands out as a powerful feature that allows data to be replicated to different systems in real-time. This not only enhances read performance across distributed databases but also serves as a cornerstone for disaster recovery and almost zero-downtime upgrades.

Logical replication works by replicating changes at the level of database transactions, providing the flexibility to select which databases, tables, or even rows get replicated. This granular control makes it an invaluable tool for creating highly available systems that can withstand the loss of a primary database or scale horizontally as demand increases. Although this fine-grain control can be useful for some complex scenarios, in this article, we will focus on a setup where all the database tables and rows will be replicated.

Setting up and experimenting with logical replication can seem daunting. This is where Docker comes into play. By containerizing PostgreSQL instances, Docker allows us to mimic complex, distributed database setups on a single machine, making this advanced feature more approachable for everyone.

In this article, we will embark on a hands-on journey to understand and implement PostgreSQL logical replication. We'll set up a master database and two replicas within a Dockerized environment, guiding you through the process of configuring logical replication, manipulating data, and observing how PostgreSQL handles data consistency and recovery in real-time. This exploration will not only clarify the process of setting up logical replication but also highlight its potential in real-world scenarios. For those interested in following along with code examples and configurations, the source code for this project is available on GitHub at https://github.com/ietxaniz/pg-logrepl.

What We Will Build

In this hands-on guide, we'll embark on a fascinating journey to set up a robust PostgreSQL logical replication environment using Docker. Our project is designed to simulate a real-world scenario where data consistency, availability, and scalability are key. Here's what we'll accomplish:

Create Three PostgreSQL Instances: Using Docker Compose, we will spin up three separate PostgreSQL instances. This setup will serve as the backbone of our replication environment, providing a practical insight into managing multiple databases in a contained ecosystem.
Configure Logical Replication: All instances will be configured to support logical replication. This involves setting one database as the master (publisher) and the other two as replicas (subscribers), establishing a real-time data synchronization framework.
Schema Synchronization: We'll ensure that all three databases start with the same schema, laying the foundation for seamless data replication across our setup.
Data Manipulation and Replication: Through a series of data manipulation exercises, we'll demonstrate how changes made to the master database are immediately reflected in the replica databases. This includes adding data and simulating a replica downtime.
Failover and Recovery Scenarios: We'll simulate a failover scenario by killing the master instance and promoting a replica to be the new master. This part of the project will highlight the resilience and flexibility of logical replication in handling unexpected database failures.
Data Consistency Checks: At various stages, we'll verify that data remains consistent across all instances, demonstrating the reliability of PostgreSQL's logical replication mechanism.
Role Reversal and Recovery: Finally, we'll bring the original master back online as a replica, completing our exploration of logical replication dynamics and ensuring all databases are synchronized.

Throughout this project, we will follow a sequence of steps designed to provide you with a comprehensive understanding of setting up and managing a logical replication system in PostgreSQL. By the end, you'll have hands-on experience with critical database administration tasks, including configuring replication, handling failovers, and ensuring data consistency across distributed systems.

For your convenience and to facilitate following along, all the code, configurations, and Docker Compose files used in this guide will be available in a GitHub repository. This will allow you to replicate the setup on your own machine and experiment further with PostgreSQL's logical replication features.

Working Environment

The sole prerequisite for this tutorial is having Docker installed on your machine. This guide and accompanying scripts are tailored for Linux environments, providing a smooth experience for Linux users right out of the box. However, the beauty of Docker is its cross-platform compatibility. Therefore, if you're using macOS or Windows with WSL (Windows Subsystem for Linux), you can still follow along without significant modifications. The versatility of Docker ensures that our setup can be replicated across different operating systems, making it accessible to a broader audience.

If Docker is not yet installed on your system, you can find detailed installation instructions on the official Docker website. Ensure Docker is correctly installed and running before proceeding to the next steps of this guide.

The skills and knowledge you'll gain from this tutorial are not limited to experimental setups or learning environments. The principles of PostgreSQL logical replication, when combined with the power of Docker, have wide applicability in real-world scenarios, especially in distributed environments across data centers. Whether you're looking to enhance data availability, scale your read operations, or set up a robust disaster recovery system, the concepts covered here will provide a solid foundation.

While we focus on a contained Docker environment for simplicity, the underlying principles can be extrapolated to more complex setups, including instances running on separate nodes in a data center or even across multiple data centers. This makes our guide an invaluable resource for anyone looking to implement logical replication in production environments.

For the sake of simplicity and to keep our focus on the core concepts of logical replication, this guide will not cover the setup of encryption for data in transit between the master and replica databases. However, it's important to acknowledge that encryption is a critical aspect of securing data in production environments. In real-world applications, ensuring the security of data in transit using SSL/TLS or other encryption methods is essential to protect against eavesdropping and man-in-the-middle attacks.

While this tutorial does not delve into encryption configurations, you should explore PostgreSQL's documentation on securing your database and consider encryption as an integral part of setting up logical replication in a production environment.

Setting Up the System

Docker Compose Setup

To kick off our exploration of PostgreSQL logical replication, we'll start by creating the backbone of our environment: the Docker Compose setup. This setup will include one master and two replica PostgreSQL instances, each running in separate Docker containers. What makes our setup particularly interesting is the use of different PostgreSQL versions for each instance. This approach not only tests the compatibility of logical replication across versions but also simulates a real-world use case for upgrading PostgreSQL instances with minimal downtime.

Below is the docker-compose.yml file that defines our PostgreSQL instances:

services:
  master:
    image: postgres:16.1-alpine3.19
    container_name: logrepl_pg_master
    volumes:
      - logrepl_pg_master-data:/var/lib/postgresql/data
    environment:
      POSTGRES_USER: postgres
      POSTGRES_PASSWORD: postgres
      POSTGRES_DB: postgres
    restart: unless-stopped
  replica1:
    image: postgres:15.5-alpine3.19
    container_name: logrepl_pg_replica1
    volumes:
      - logrepl_pg_replica1-data:/var/lib/postgresql/data
    environment:
      POSTGRES_USER: postgres
      POSTGRES_PASSWORD: postgres
      POSTGRES_DB: postgres
    restart: unless-stopped
  replica2:
    image: postgres:14.10-alpine3.19
    container_name: logrepl_pg_replica2
    volumes:
      - logrepl_pg_replica2-data:/var/lib/postgresql/data
    environment:
      POSTGRES_USER: postgres
      POSTGRES_PASSWORD: postgres
      POSTGRES_DB: postgres
    restart: unless-stopped
volumes:
  logrepl_pg_master-data:
    name: logrepl_pg_master-data
  logrepl_pg_replica1-data:
    name: logrepl_pg_replica1-data
  logrepl_pg_replica2-data:
    name: logrepl_pg_replica2-data

This configuration establishes a master database using PostgreSQL 16.1 and two replicas using versions 15.5 and 14.10, respectively. By employing Alpine Linux as the base image, we ensure our setup is lightweight and efficient. Each service is configured with its volume for data persistence, ensuring that data remains intact across container restarts.

To bring up the databases, create a file named docker-compose.yml with the content above. Then, in a shell window, execute the following command:

docker compose up

This command will start the PostgreSQL instances as defined in the Docker Compose file. You'll see logs in the terminal indicating that the containers are up and running, ready for the next steps of schema creation and logical replication configuration.

Once our Docker containers are up and running, the next step is crucial: creating identical schemas across all instances. This consistency is vital for ensuring smooth replication from the master to the replicas. For our project, we'll be using a simple yet illustrative schema that includes users, posts, and comments—emulating a basic blogging platform. This schema not only demonstrates the relationships between different data types but also sets the stage for understanding how logical replication handles data synchronization.

Project Schema

The schema for our project is defined as follows:

-- Schema for Users
CREATE TABLE users (
    user_id SERIAL PRIMARY KEY,
    username VARCHAR(255) NOT NULL,
    email VARCHAR(255) UNIQUE NOT NULL,
    created_at TIMESTAMP WITHOUT TIME ZONE DEFAULT (now() at time zone 'utc')
);

-- Schema for Posts
CREATE TABLE posts (
    post_id SERIAL PRIMARY KEY,
    user_id INT NOT NULL,
    title VARCHAR(255) NOT NULL,
    content TEXT NOT NULL,
    created_at TIMESTAMP WITHOUT TIME ZONE DEFAULT (now() at time zone 'utc'),
    updated_at TIMESTAMP WITHOUT TIME ZONE,
    FOREIGN KEY (user_id) REFERENCES users(user_id)
);

-- Schema for Comments
CREATE TABLE comments (
    comment_id SERIAL PRIMARY KEY,
    post_id INT NOT NULL,
    user_id INT NOT NULL,
    content TEXT NOT NULL,
    created_at TIMESTAMP WITHOUT TIME ZONE DEFAULT (now() at time zone 'utc'),
    FOREIGN KEY (post_id) REFERENCES posts(post_id),
    FOREIGN KEY (user_id) REFERENCES users(user_id)
);

This schema sets up a relational structure where users can create posts, and both users and posts can have comments, mimicking a real-world application's data model.

Schema Application Script

To simplify the schema application across our PostgreSQL instances, we've prepared a script named create-schemas.sh. This script automates the process of copying the schema.sql file to each container and executing it against the PostgreSQL service inside. Here's how the script works:

#!/bin/bash
# Define the database containers
databases=("logrepl_pg_master" "logrepl_pg_replica1" "logrepl_pg_replica2")
schema_file="./schema.sql"

for db in "${databases[@]}"; do
    echo "Copying schema to $db..."
    docker cp "$schema_file" "$db:/tmp/schema.sql"

    echo "Loading schema in $db..."
    docker exec -i "$db" psql -U postgres -d postgres -f /tmp/schema.sql

    echo "Removing schema file from $db..."
    docker exec -i "$db" rm /tmp/schema.sql

    echo "Schema loaded successfully in $db!"
done

To execute this script, ensure it has execute permissions by running chmod +x create-schemas.sh, then execute it with ./create-schemas.sh. The script will iterate over each defined database container, copy the schema file, execute it, and clean up, ensuring that each database instance is configured with the identical schema needed for our logical replication setup.

By maintaining schema consistency across the master and replica databases, we ensure that our logical replication process can proceed without any schema-related conflicts, providing a seamless data synchronization experience.

Configuration for Logical Replication

After setting up our PostgreSQL instances and ensuring they share identical schemas, the next crucial step is to configure them for logical replication. This process requires adjusting specific settings in the PostgreSQL configuration files to enable and optimize the replication process.

To streamline this configuration process, we've created a script named configure-dbs.sh. This script automates the adjustments necessary for all instances (master and replicas) to prepare them for logical replication. By configuring the same parameters across all databases, we not only prepare them for their current roles but also allow for the flexibility to change roles in the future, such as converting a replica into a master during failover scenarios or planned upgrades.

Key Configuration Changes

wal_level: Set to logical to support capturing and replicating changes at the transaction level, rather than at the physical block level.
max_replication_slots: Increased to accommodate the planned number of replicas, ensuring the master retains enough WAL logs for all replicas.
max_wal_senders: Raised to allow more concurrent WAL sender processes, facilitating efficient data streaming to replicas.
Replication Permissions in pg_hba.conf: Configured to grant necessary connection permissions for replication purposes.

Configuration Script

#!/bin/bash
# Script to configure PostgreSQL instances for logical replication

max_number_of_replicas=4
max_wal_senders=8

# Define all PostgreSQL instances
databases=("logrepl_pg_master" "logrepl_pg_replica1" "logrepl_pg_replica2")

for db in "${databases[@]}"; do
    # Apply configuration changes
    ...
done

# Reminder to execute with caution, especially in production environments.

Before running configure-dbs.sh, ensure it's executable with chmod +x configure-dbs.sh. Execute the script to apply configuration changes across all instances. Designed for efficiency and safety, this script reduces complexity and minimizes the potential for configuration errors, setting the stage for a seamless replication process.

Configure Replication

After configuring our PostgreSQL instances for logical replication, the next step is to establish the replication relationship itself. This involves defining one database as the master (or publisher) and the others as replicas (or subscribers). We achieve this through the configure-replication.sh script, which automates the setup, ensuring changes made in the master database are seamlessly replicated to logrepl_pg_replica1 and logrepl_pg_replica2.

#!/bin/bash

# Define the master database and the replicas
master_db="logrepl_pg_master"
replicas=("logrepl_pg_replica1" "logrepl_pg_replica2")

# Configure the master database
echo "Configuring $master_db as master..."
docker exec -i "$master_db" psql -U postgres -d postgres -c "DROP PUBLICATION IF EXISTS my_publication;"
docker exec -i "$master_db" psql -U postgres -d postgres -c "CREATE PUBLICATION my_publication FOR ALL TABLES;"

# Configure each replica to follow the master
for replica in "${replicas[@]}"; do
    echo "Configuring $replica to replicate from $master_db..."

    # Construct the connection string
    conninfo="host=$master_db port=5432 user=postgres password=postgres dbname=postgres"

    # Configure the subscription on the replica
    docker exec -i "$replica" psql -U postgres -d postgres -c "DROP SUBSCRIPTION IF EXISTS ${replica}_subscription;"
    docker exec -i "$replica" psql -U postgres -d postgres -c "CREATE SUBSCRIPTION ${replica}_subscription CONNECTION 'dbname=postgres host=$master_db user=postgres password=postgres' PUBLICATION my_publication;"

    echo "$replica configured to replicate from $master_db."
done

echo "Master and replicas configured successfully for logical replication."

The script starts by configuring logrepl_pg_master as the master database. It does so by dropping any existing publication named my_publication (to avoid conflicts) and then creating a new publication that includes all tables. This publication acts as the source of data changes that will be replicated to the subscribers. For each replica, the script creates a subscription to the master's publication. It first removes any existing subscription to avoid conflicts, then establishes a new subscription using the connection information to the master database. This subscription mechanism is what allows the replicas to receive and apply changes from the master.

Key Commands Explained

DROP PUBLICATION IF EXISTS my_publication;: Ensures idempotency by removing the existing publication before creating a new one.
CREATE PUBLICATION my_publication FOR ALL TABLES;: Creates a new publication on the master that includes all tables, making them available for replication.
DROP SUBSCRIPTION IF EXISTS ${replica}_subscription;: Removes any existing subscription on the replica.
CREATE SUBSCRIPTION ${replica}_subscription CONNECTION '...' PUBLICATION my_publication;: Establishes a new subscription on each replica to the master's publication, enabling data replication.

Data Manipulation and Replication Process

Now that we've set up our PostgreSQL logical replication environment, it's time to see it in action. We'll begin by demonstrating how data added to the master database is automatically replicated across all connected replicas. This real-time synchronization is the heart of logical replication, ensuring data consistency and availability across our distributed database system.

Adding Data to the Master

Let's start by inserting a new user into the users table on the master database. We'll execute the following command in a shell environment:

$ docker exec -i logrepl_pg_master psql -U postgres -d postgres -c "INSERT INTO users (username, email) VALUES ('newuser', 'newuser@example.com');"
INSERT 0 1

This command inserts a new row with the username newuser and the email newuser@example.com into the users table. The INSERT 0 1 response indicates that the operation was successful and one row was added.

Verifying Replication on the Replicas

Next, we'll verify that this new row has been replicated to both replicas. To do this, we'll query the users table on each replica:

$ docker exec -i logrepl_pg_replica1 psql -U postgres -d postgres -c "SELECT * FROM users;"
 user_id | username |        email        |         created_at         
---------+----------+---------------------+----------------------------
       1 | newuser  | newuser@example.com | 2024-02-11 10:08:08.116556
(1 row)

$ docker exec -i logrepl_pg_replica2 psql -U postgres -d postgres -c "SELECT * FROM users;"
 user_id | username |        email        |         created_at         
---------+----------+---------------------+----------------------------
       1 | newuser  | newuser@example.com | 2024-02-11 10:08:08.116556
(1 row)

As we can observe, the users table on both logrepl_pg_replica1 and logrepl_pg_replica2 instances reflects the data we inserted into the master database. This immediate synchronization showcases the efficiency and reliability of PostgreSQL's logical replication.

Switching Off One of the Replicas

To further test the resilience of our logical replication setup, let's simulate a scenario where one of the replicas (logrepl_pg_replica2) goes offline. This step will help us understand how PostgreSQL handles replication when a subscriber is temporarily unavailable.

First, we'll stop the logrepl_pg_replica2 container:

$ docker stop logrepl_pg_replica2
logrepl_pg_replica2

With logrepl_pg_replica2 offline, let's insert a new user into the users table on the master database:

$ docker exec -i logrepl_pg_master psql -U postgres -d postgres -c "INSERT INTO users (username, email) VALUES ('newuser2', 'newuser2@example.com');"
INSERT 0 1

Now, let's verify that this new data is replicated to logrepl_pg_replica1, which is still online:

$ docker exec -i logrepl_pg_replica1 psql -U postgres -d postgres -c "SELECT * FROM users;"
 user_id | username |        email         |         created_at         
---------+----------+----------------------+----------------------------
       1 | newuser  | newuser@example.com  | 2024-02-11 10:08:08.116556
       2 | newuser2 | newuser2@example.com | 2024-02-11 10:13:38.786147
(2 rows)

As expected, logrepl_pg_replica1 has successfully received and applied the new data. However, since logrepl_pg_replica2 is offline, it won't have this latest update. Attempting to query logrepl_pg_replica2 results in an error, as the container is not running:

$ docker exec -i logrepl_pg_replica2 psql -U postgres -d postgres -c "SELECT * FROM users;"
Error response from daemon: container 70cb8a4ff8f10ac047cbfd6b531b23cf88fe4def0c4ef1521187027ee343f173 is not running

This step effectively demonstrates the robustness of our logical replication setup in handling scenarios where a replica may become temporarily unavailable. Next, we'll explore how PostgreSQL catches up the offline replica once it's back online, ensuring data consistency across the entire replication system.

Switching On Replica2

After demonstrating the behavior of our logical replication setup with one replica offline, let's proceed to bring logrepl_pg_replica2 back online. This step is crucial to observe how PostgreSQL handles the synchronization process for replicas that were temporarily unavailable and ensures they catch up with the master's current state.

To restart logrepl_pg_replica2, we simply use the Docker command to start the container:

$ docker start logrepl_pg_replica2
logrepl_pg_replica2

After giving it a moment to initialize and reestablish its connection to the master, we then check the users table to confirm that logrepl_pg_replica2 has successfully synchronized and now contains the same data as the master and the other replica:

$ sleep 2
$ docker exec -i logrepl_pg_replica2 psql -U postgres -d postgres -c "SELECT * FROM users;"
 user_id | username |        email         |         created_at         
---------+----------+----------------------+----------------------------
       1 | newuser  | newuser@example.com  | 2024-02-11 10:08:08.116556
       2 | newuser2 | newuser2@example.com | 2024-02-11 10:13:38.786147
(2 rows)

As the output shows, logrepl_pg_replica2 automatically updated its data to reflect the current state of the master database upon reconnection. This behavior underscores the powerful automatic recovery and synchronization capabilities of PostgreSQL's logical replication mechanism. Without any manual intervention, the replica was able to catch up with the changes it missed while offline, ensuring data consistency across our distributed database system.

Adding More Users in the New State

With our replication setup now fully operational and having demonstrated its resilience to replica downtime, we proceed to test the continuous synchronization capabilities by adding another user to the master database. This step will help us verify that all databases, including the previously offline replica, are correctly synchronized in real time.

We add a new user, newuser3, to the users table on the master database and confirm the insertion was successful:

$ docker exec -i logrepl_pg_master psql -U postgres -d postgres -c "INSERT INTO users (username, email) VALUES ('newuser3', 'newuser3@example.com');"
INSERT 0 1

Next, we ensure that this new data is accurately reflected across both replicas, in addition to the master, by querying the users table on each database:

$ docker exec -i logrepl_pg_master psql -U postgres -d postgres -c "SELECT * FROM users;"
 user_id | username |        email         |         created_at         
---------+----------+----------------------+----------------------------
       1 | newuser  | newuser@example.com  | 2024-02-11 10:08:08.116556
       2 | newuser2 | newuser2@example.com | 2024-02-11 10:13:38.786147
       3 | newuser3 | newuser3@example.com | 2024-02-11 10:20:31.714676
(3 rows)

$ docker exec -i logrepl_pg_replica1 psql -U postgres -d postgres -c "SELECT * FROM users;"
 user_id | username |        email         |         created_at         
---------+----------+----------------------+----------------------------
       1 | newuser  | newuser@example.com  | 2024-02-11 10:08:08.116556
       2 | newuser2 | newuser2@example.com | 2024-02-11 10:13:38.786147
       3 | newuser3 | newuser3@example.com | 2024-02-11 10:20:31.714676
(3 rows)

$ docker exec -i logrepl_pg_replica2 psql -U postgres -d postgres -c "SELECT * FROM users;"
 user_id | username |        email         |         created_at         
---------+----------+----------------------+----------------------------
       1 | newuser  | newuser@example.com  | 2024-02-11 10:08:08.116556
       2 | newuser2 | newuser2@example.com | 2024-02-11 10:13:38.786147
       3 | newuser3 | newuser3@example.com | 2024-02-11 10:20:31.714676
(3 rows)

As the output indicates, newuser3 is now present in the users table across all instances (logrepl_pg_master, logrepl_pg_replica1, and logrepl_pg_replica2), demonstrating that the logical replication system efficiently synchronizes data in real-time across the entire distributed environment.

This exercise confirms the robustness and reliability of our PostgreSQL logical replication setup. It showcases not only the system's ability to handle disruptions but also its capability to maintain data consistency and availability seamlessly across multiple database instances, ensuring that each part of our system is always up-to-date with the latest data.

Switching of Master and Making Replica1 New Master

In real-world scenarios, it's essential to prepare for situations where the master database might go down or need to be taken offline for maintenance. In such cases, promoting one of the replicas to be the new master ensures continuity of operations and data availability. Our next sequence of steps demonstrates how to handle this transition smoothly, using step07.sh to automate the process.

#!/bin/bash

# Define the services
new_master="logrepl_pg_replica1"
remaining_replica="logrepl_pg_replica2"

# Stop the old master
echo "Stopping the old master: $old_master..."
docker stop "$old_master"

# Wait for a second to ensure the stop command fully processes
sleep 1

# Disable subscriptions on replica1 and replica2 without dropping them
echo "Disabling subscriptions on $new_master and $remaining_replica..."
docker exec -i "$new_master" psql -U postgres -d postgres -c "ALTER SUBSCRIPTION ${new_master}_subscription DISABLE; ALTER SUBSCRIPTION ${new_master}_subscription SET (slot_name = NONE);"
docker exec -i "$remaining_replica" psql -U postgres -d postgres -c "ALTER SUBSCRIPTION ${remaining_replica}_subscription DISABLE; ALTER SUBSCRIPTION ${remaining_replica}_subscription SET (slot_name = NONE);"


# update postgresql secuences on the new master to be
docker exec -i "$new_master" psql -U postgres -d postgres -c "SELECT setval('users_user_id_seq', COALESCE((SELECT MAX(user_id) FROM users), 0) + 1, false);"
docker exec -i "$new_master" psql -U postgres -d postgres -c "SELECT setval('posts_post_id_seq', COALESCE((SELECT MAX(post_id) FROM posts), 0) + 1, false);"
docker exec -i "$new_master" psql -U postgres -d postgres -c "SELECT setval('comments_comment_id_seq', COALESCE((SELECT MAX(comment_id) FROM comments), 0) + 1, false);"

# Ensure replica2 starts with a clean state
echo "Resetting data on $remaining_replica to prepare for replication..."
docker exec -i "$remaining_replica" psql -U postgres -d postgres -c "TRUNCATE TABLE users, posts, comments RESTART IDENTITY CASCADE;"

# Continue with the creation of the new publication and subscription as before
echo "Creating new publication on $new_master..."
docker exec -i "$new_master" psql -U postgres -d postgres -c "CREATE PUBLICATION new_master_publication FOR ALL TABLES;"

echo "Configuring $remaining_replica to replicate from $new_master..."
docker exec -i "$remaining_replica" psql -U postgres -d postgres -c "CREATE SUBSCRIPTION replica2_new_subscription CONNECTION 'dbname=postgres host=$new_master user=postgres password=postgres' PUBLICATION new_master_publication;"

echo "Replication reconfiguration complete."

Key Steps in the Transition Process

Stopping the Old Master: We begin by stopping the current master database to simulate its failure or planned downtime.
Disabling Subscriptions: Subscriptions on the replicas are temporarily disabled to prevent them from trying to replicate data from the now-offline master. This is a crucial step to avoid replication conflicts during the transition.
Updating Sequences: On the new master (logrepl_pg_replica1), we update the sequences for all tables to ensure that new records will have unique identifiers, avoiding conflicts with existing data.
Resetting Data on the Remaining Replica: To prepare logrepl_pg_replica2 for replication from the new master, we reset its data. This step is essential to ensure that it starts with a clean slate, eliminating the risk of id conflicts and data discrepancies.
Reconfiguring Replication: We create a new publication on the new master and configure the remaining replica to subscribe to this publication, effectively re-establishing the logical replication setup with logrepl_pg_replica1 now acting as the master.

After executing next sequence, we validate the replication setup by inserting a new user into the users table on the new master (logrepl_pg_replica1) and verifying that this change is replicated to logrepl_pg_replica2. The successful replication of this new record to both replicas confirms the correct reconfiguration of the logical replication setup.

$$ docker stop logrepl_pg_master
logrepl_pg_master
$$ sleep 1
$$ ./step07.sh
Stopping the old master: ...
Error response from daemon: page not found
Disabling subscriptions on logrepl_pg_replica1 and logrepl_pg_replica2...
ALTER SUBSCRIPTION
ALTER SUBSCRIPTION
ALTER SUBSCRIPTION
 setval 
--------
      4
(1 row)

 setval 
--------
      1
(1 row)

 setval 
--------
      1
(1 row)

Resetting data on logrepl_pg_replica2 to prepare for replication...
TRUNCATE TABLE
Creating new publication on logrepl_pg_replica1...
CREATE PUBLICATION
Configuring logrepl_pg_replica2 to replicate from logrepl_pg_replica1...
NOTICE:  created replication slot "replica2_new_subscription" on publisher
CREATE SUBSCRIPTION
Replication reconfiguration complete.
$ docker exec -i logrepl_pg_replica1 psql -U postgres -d postgres -c "INSERT INTO users (username, email) VALUES ('newuser4', 'newuser4@example.com');"
INSERT 0 1
$ docker exec -i logrepl_pg_master psql -U postgres -d postgres -c "SELECT * FROM users;"
Error response from daemon: container b85f8d95443abae3f898c4e28d3afc3e319730ade2a098e94cc7b0e646ee7ac8 is not running
$ docker exec -i logrepl_pg_replica1 psql -U postgres -d postgres -c "SELECT * FROM users;"
 user_id | username |        email         |         created_at         
---------+----------+----------------------+----------------------------
       1 | newuser  | newuser@example.com  | 2024-02-11 10:08:08.116556
       2 | newuser2 | newuser2@example.com | 2024-02-11 10:13:38.786147
       3 | newuser3 | newuser3@example.com | 2024-02-11 10:20:31.714676
       4 | newuser4 | newuser4@example.com | 2024-02-11 10:26:50.874549
(4 rows)

$ docker exec -i logrepl_pg_replica2 psql -U postgres -d postgres -c "SELECT * FROM users;"
 user_id | username |        email         |         created_at         
---------+----------+----------------------+----------------------------
       1 | newuser  | newuser@example.com  | 2024-02-11 10:08:08.116556
       2 | newuser2 | newuser2@example.com | 2024-02-11 10:13:38.786147
       3 | newuser3 | newuser3@example.com | 2024-02-11 10:20:31.714676
       4 | newuser4 | newuser4@example.com | 2024-02-11 10:26:50.874549
(4 rows)

Considerations and Potential Sources of Failure

Before the master goes down, it's crucial to ensure that all replicas are fully synchronized. Any discrepancy between the data on logrepl_pg_replica1 and logrepl_pg_replica2 at the time of the switch can lead to inconsistencies, as the new master will begin replicating from its current state. Updating all sequences on the new master is vital to prevent conflicts caused by overlapping identifiers. This step ensures that new inserts on the new master will not conflict with existing data. The process of stopping the master should be controlled and synchronized with the replicas to minimize the risk of data inconsistency or loss.

By carefully following these steps and considerations, we can effectively manage the transition of the master role to a replica, maintaining data integrity and availability. This process highlights the flexibility and resilience of PostgreSQL's logical replication mechanism, allowing for robust database management strategies in distributed environments.

In failover scenarios ensuring that all replicas have same data is not so easy to manage, as some replicas could not be fully updated, so a carefull strategy should be choosen for this scenarios.

Switching the Master Back On: Challenges and Solutions

The Unexpected Challenge

When attempting to reintegrate the original master (logrepl_pg_master) back into our replication setup as a subscriber, the process didn't proceed as smoothly as anticipated. Despite successfully configuring it to subscribe to logrepl_pg_replica1 (the new master), data updates from the new master weren't replicated to the old master as expected. This issue underscored the complexities involved in reversing replication roles within a PostgreSQL logical replication setup.

This was the script I used to subscribe the old master to the new master:

#!/bin/bash
old_master="logrepl_pg_master"
new_master="logrepl_pg_replica1"
old_master_subscription="my_subscription"

docker start $old_master
sleep 1

# Disable any outgoing replication from the old master
echo "Disabling any publications on $old_master..."
docker exec -i "$old_master" psql -U postgres -d postgres -c "DROP PUBLICATION IF EXISTS my_publication;"

# Create a new subscription on the old master to the new master's publication
docker exec -i "$old_master" psql -U postgres -d postgres -c "DROP SUBSCRIPTION IF EXISTS $old_master_subscription;"
docker exec -i "$old_master" psql -U postgres -d postgres -c "CREATE SUBSCRIPTION $old_master_subscription CONNECTION 'dbname=postgres host=$new_master user=postgres password=postgres' PUBLICATION new_master_publication;"

echo "Subscription setup complete. $old_master is now a subscriber of $new_master."

But, then I realised that data was not being updated in master:

$ docker exec -i logrepl_pg_master psql -U postgres -d postgres -c "SELECT * FROM users;"
 user_id | username |        email         |         created_at         
---------+----------+----------------------+----------------------------
       1 | newuser  | newuser@example.com  | 2024-02-11 10:08:08.116556
       2 | newuser2 | newuser2@example.com | 2024-02-11 10:13:38.786147
       3 | newuser3 | newuser3@example.com | 2024-02-11 10:20:31.714676
(3 rows)

$ docker exec -i logrepl_pg_replica1 psql -U postgres -d postgres -c "SELECT * FROM users;"
 user_id | username |        email         |         created_at         
---------+----------+----------------------+----------------------------
       1 | newuser  | newuser@example.com  | 2024-02-11 10:08:08.116556
       2 | newuser2 | newuser2@example.com | 2024-02-11 10:13:38.786147
       3 | newuser3 | newuser3@example.com | 2024-02-11 10:20:31.714676
       4 | newuser4 | newuser4@example.com | 2024-02-11 10:26:50.874549
(4 rows)

$ docker exec -i logrepl_pg_replica2 psql -U postgres -d postgres -c "SELECT * FROM users;"
 user_id | username |        email         |         created_at         
---------+----------+----------------------+----------------------------
       1 | newuser  | newuser@example.com  | 2024-02-11 10:08:08.116556
       2 | newuser2 | newuser2@example.com | 2024-02-11 10:13:38.786147
       3 | newuser3 | newuser3@example.com | 2024-02-11 10:20:31.714676
       4 | newuser4 | newuser4@example.com | 2024-02-11 10:26:50.874549
(4 rows)

So I checked log messages and found out the next:

logrepl_pg_replica1  | 2024-02-11 10:51:19.017 UTC [230] LOG:  logical decoding found consistent point at 0/158FF30
logrepl_pg_replica1  | 2024-02-11 10:51:19.017 UTC [230] DETAIL:  There are no running transactions.
logrepl_pg_replica1  | 2024-02-11 10:51:19.017 UTC [230] STATEMENT:  CREATE_REPLICATION_SLOT "pg_16436_sync_16385_7334283988469985309" LOGICAL pgoutput (SNAPSHOT 'use')
logrepl_pg_master    | 2024-02-11 10:51:19.079 UTC [67] ERROR:  duplicate key value violates unique constraint "users_pkey"
logrepl_pg_master    | 2024-02-11 10:51:19.079 UTC [67] DETAIL:  Key (user_id)=(1) already exists.
logrepl_pg_master    | 2024-02-11 10:51:19.079 UTC [67] CONTEXT:  COPY users, line 1
logrepl_pg_master    | 2024-02-11 10:51:19.084 UTC [1] LOG:  background worker "logical replication worker" (PID 67) exited with exit code 1
logrepl_pg_master    | 2024-02-11 10:51:24.096 UTC [68] LOG:  logical replication table synchronization worker for subscription "my_subscription", table "users" has started

So this means that now that master has subscribed to replica, the system is trying to transmit full database data. I could have reseted master subscription in order to start listening from now on using next instruction but this would mean that master would miss database changes made from the time it was off till the time it is on again:

Resolution: Starting from Scratch

To resolve the synchronization issues, a decisive approach was taken: restarting the original master from scratch. This involved removing the existing Docker container and volume for logrepl_pg_master, then recreating and reconfiguring it from the ground up. This process guaranteed that the old master, now a subscriber, was perfectly in sync with the current state of the new master, effectively making it an up-to-date replica.

#!/bin/bash
old_master="logrepl_pg_master"
new_master="logrepl_pg_replica1"
old_master_subscription="my_subscription"
max_number_of_replicas=4
max_wal_senders=8

docker stop $old_master
sleep 1
docker rm $old_master
sleep 1
docker volume rm logrepl_pg_master-data
docker compose up -d
sleep 2
docker cp ./schema.sql "$old_master:/tmp/schema.sql"
docker exec -i $old_master psql -U postgres -d postgres -f /tmp/schema.sql
docker exec -i $old_master rm /tmp/schema.sql

docker exec -i "$old_master" bash -c "sed -i 's/^#*wal_level .*$/wal_level = logical/' /var/lib/postgresql/data/postgresql.conf"
docker exec -i "$old_master" bash -c "sed -i 's/^#*max_replication_slots .*$/max_replication_slots = $max_number_of_replicas/' /var/lib/postgresql/data/postgresql.conf"
docker exec -i "$old_master" bash -c "sed -i 's/^#*max_wal_senders .*$/max_wal_senders = $max_wal_senders/' /var/lib/postgresql/data/postgresql.conf"
docker exec -i "$old_master" bash -c "grep -qxF 'host replication all all md5' /var/lib/postgresql/data/pg_hba.conf || echo 'host replication all all md5' >> /var/lib/postgresql/data/pg_hba.conf"
docker restart $old_master

sleep 2

# Disable any outgoing replication from the old master
echo "Disabling any publications on $old_master..."
docker exec -i "$old_master" psql -U postgres -d postgres -c "DROP PUBLICATION IF EXISTS my_publication;"

# Remove previous replica subscriptions of master in replica1
docker exec -it logrepl_pg_replica1 psql -U postgres -d postgres -c "SELECT pg_drop_replication_slot('my_subscription');"


# Create a new subscription on the old master to the new master's publication
docker exec -i "$old_master" psql -U postgres -d postgres -c "DROP SUBSCRIPTION IF EXISTS $old_master_subscription;"
docker exec -i "$old_master" psql -U postgres -d postgres -c "CREATE SUBSCRIPTION $old_master_subscription CONNECTION 'dbname=postgres host=$new_master user=postgres password=postgres' PUBLICATION new_master_publication;"

echo "Subscription setup complete. $old_master is now a subscriber of $new_master."

After some time:

$ docker exec -i logrepl_pg_master psql -U postgres -d postgres -c "SELECT * FROM users;"
 user_id | username |        email         |         created_at         
---------+----------+----------------------+----------------------------
       1 | newuser  | newuser@example.com  | 2024-02-11 10:08:08.116556
       2 | newuser2 | newuser2@example.com | 2024-02-11 10:13:38.786147
       3 | newuser3 | newuser3@example.com | 2024-02-11 10:20:31.714676
       4 | newuser4 | newuser4@example.com | 2024-02-11 10:26:50.874549
(4 rows)
$ docker exec -i logrepl_pg_replica1 psql -U postgres -d postgres -c "SELECT * FROM users;"
 user_id | username |        email         |         created_at         
---------+----------+----------------------+----------------------------
       1 | newuser  | newuser@example.com  | 2024-02-11 10:08:08.116556
       2 | newuser2 | newuser2@example.com | 2024-02-11 10:13:38.786147
       3 | newuser3 | newuser3@example.com | 2024-02-11 10:20:31.714676
       4 | newuser4 | newuser4@example.com | 2024-02-11 10:26:50.874549
(4 rows)
$ docker exec -i logrepl_pg_replica2 psql -U postgres -d postgres -c "SELECT * FROM users;"
 user_id | username |        email         |         created_at         
---------+----------+----------------------+----------------------------
       1 | newuser  | newuser@example.com  | 2024-02-11 10:08:08.116556
       2 | newuser2 | newuser2@example.com | 2024-02-11 10:13:38.786147
       3 | newuser3 | newuser3@example.com | 2024-02-11 10:20:31.714676
       4 | newuser4 | newuser4@example.com | 2024-02-11 10:26:50.874549
(4 rows)

And if now we add another row in replica1:

$ docker exec -i logrepl_pg_replica1 psql -U postgres -d postgres -c "INSERT INTO users (username, email) VALUES ('newuser5', 'newuser5@example.com');"
INSERT 0 1
$ sleep 1
$ docker exec -i logrepl_pg_master psql -U postgres -d postgres -c "SELECT * FROM users;"
 user_id | username |        email         |         created_at         
---------+----------+----------------------+----------------------------
       1 | newuser  | newuser@example.com  | 2024-02-11 10:08:08.116556
       2 | newuser2 | newuser2@example.com | 2024-02-11 10:13:38.786147
       3 | newuser3 | newuser3@example.com | 2024-02-11 10:20:31.714676
       4 | newuser4 | newuser4@example.com | 2024-02-11 10:26:50.874549
       5 | newuser5 | newuser5@example.com | 2024-02-11 11:17:46.730413
(5 rows)
$ docker exec -i logrepl_pg_replica1 psql -U postgres -d postgres -c "SELECT * FROM users;"
 user_id | username |        email         |         created_at         
---------+----------+----------------------+----------------------------
       1 | newuser  | newuser@example.com  | 2024-02-11 10:08:08.116556
       2 | newuser2 | newuser2@example.com | 2024-02-11 10:13:38.786147
       3 | newuser3 | newuser3@example.com | 2024-02-11 10:20:31.714676
       4 | newuser4 | newuser4@example.com | 2024-02-11 10:26:50.874549
       5 | newuser5 | newuser5@example.com | 2024-02-11 11:17:46.730413
(5 rows)
$ docker exec -i logrepl_pg_replica2 psql -U postgres -d postgres -c "SELECT * FROM users;"
 user_id | username |        email         |         created_at         
---------+----------+----------------------+----------------------------
       1 | newuser  | newuser@example.com  | 2024-02-11 10:08:08.116556
       2 | newuser2 | newuser2@example.com | 2024-02-11 10:13:38.786147
       3 | newuser3 | newuser3@example.com | 2024-02-11 10:20:31.714676
       4 | newuser4 | newuser4@example.com | 2024-02-11 10:26:50.874549
       5

This approach, while drastic, ensured data integrity and consistency across the replication setup. It highlighted a key advantage of containerized environments—the ability to rapidly rebuild and redeploy components of your infrastructure with confidence.

Reflections and Future Directions

The challenges encountered while switching the master back on have led to valuable insights:

Preparation for Role Reversal: The need for thorough preparation and clear strategies when reversing roles in a logical replication setup. This includes ensuring data consistency and understanding the implications of such changes.
Resilience through Rebuilding: The effectiveness of starting from scratch as a means to ensure an accurate and reliable replication setup. While not always ideal, this approach can be a reliable fallback strategy in complex scenarios.
Exploring Alternative Approaches: Inspired by these challenges, I am considering alternative procedures for switching off and replacing the master without the need to replicate the entire database from scratch. This exploration aims to refine our strategies for managing PostgreSQL logical replication, making the process more efficient and less disruptive.

In light of these experiences, I plan to conduct further tests and research into alternative strategies for managing master role transitions in logical replication setups. If successful, these findings will be the subject of a forthcoming article, aimed at providing even more robust and efficient methods for managing PostgreSQL replication environments.

Conclusions and Key Takeaways

Our journey through the intricacies of PostgreSQL logical replication, set within a Dockerized environment, has shed light on vital aspects of database management, including data consistency, availability, and scalability. This hands-on guide underscored the significance of logical replication in bolstering system resilience and adaptability, proving its worth in scenarios demanding disaster recovery and near-zero downtime upgrades.

The granular control offered by logical replication, enabling selective synchronization of databases, tables, or rows, empowers administrators to devise customized strategies that align with specific operational needs. Docker's role in demystifying the deployment and management of complex, distributed database systems has been pivotal, rendering sophisticated features like logical replication more approachable and manageable.

The setup's resilience in the face of replica downtimes, coupled with its capacity for smooth role reversals amid failovers, accentuates the paramount importance of foresight and meticulous planning in database management. The imperative to maintain data consistency across all instances, particularly through and beyond failover events, emerged as a central theme, highlighting the challenges that necessitate a systematic approach to managing role transitions and system updates.

While this exploration has been revealing, it also paves the way for further inquiry, especially concerning the optimization of master role transitions and the reduction of associated downtime. The difficulties experienced in reintegrating the original master as a subscriber have kindled interest in devising strategies that are both more efficient and less intrusive.

Motivated by these challenges, we are now poised to delve into alternative methodologies for handling master role changes without resorting to complete database replication from scratch. This forthcoming endeavor aims to enhance our logical replication management techniques, emphasizing operational efficiency and minimal disruption.

In wrapping up, our foray into the domain of PostgreSQL logical replication within a Docker framework has not only broadened our comprehension of this potent functionality but also underscored the perennial value of continuous learning and refinement in database management practices. This journey highlights the evolving nature of technology and our relentless pursuit of increasingly sophisticated and effective solutions within the dynamic realm of database administration.

Setting Up VSCode Development Environments with Docker on Linux

Iñigo Etxaniz — Sat, 20 Jan 2024 18:50:57 +0000

Introduction

In the evolving landscape of software development, the efficiency and reliability of your development environment can significantly impact productivity and collaboration. The VSCode in Docker project recognizes this, offering a versatile approach to set up development environments using Docker to run Visual Studio Code (VSCode) on Linux. This method excels in creating isolated, secure, and highly reproducible development spaces, tailored to the diverse needs of modern developers.

This article delves into the world of Dockerized VSCode environments, underscoring benefits like enhanced security, resource efficiency, and consistency across multiple machines. A unique advantage of this setup is its ability to segregate development environments based on project or language-specific needs. This isolation not only simplifies environment management but also reduces the clutter of unnecessary extensions, avoiding unproductive distractions or annoying notifications.

The project provides Docker images such as inigoetxaniz/vscode-base, inigoetxaniz/vscode-go, and inigoetxaniz/vscode-ext-devel, each catering to different development requirements. This setup is ideal for testing new extensions in a safe, controlled environment, without risking your primary development setup. It also allows for simultaneously running multiple VSCode instances with different configurations and access controls – imagine having separate instances for Node.js and Rust, each with its own tailored settings and themes.

In the upcoming sections, we will explore the advantages of using Dockerized VSCode environments, offer a step-by-step guide for setup, and discuss practical use cases and customization options. Join us as we dive in to discover how Docker can revolutionize your VSCode experience on Linux, enhancing productivity and offering unparalleled flexibility in your development workflow.

Advantages of Dockerized VSCode Environments

Getting Started

Before diving into the setup of Dockerized VSCode environments, it's important to cover the basic requirements. The primary requirement is a Linux-based system with Docker installed. For the purposes of this guide and the development of the Dockerized VSCode project, Ubuntu 22.04 LTS (Long-Term Support) with the GNOME desktop environment has been used as the testing platform. This version of Ubuntu provides a stable and up-to-date foundation, ensuring compatibility and performance for the Dockerized VSCode environments.

While this guide won't delve into the specifics of installing Docker on Ubuntu, users new to Docker can follow the comprehensive installation instructions provided by Docker itself at Docker Installation Guide for Ubuntu. These instructions are clear, detailed, and cater to various setup preferences, ensuring you can get Docker up and running smoothly on your system.

It's important to note that while this guide focuses on Ubuntu 22.04, the principles and steps should be applicable to other Linux distributions with minor adjustments. However, the choice of Ubuntu 22.04 comes with the advantage of long-term support and widespread use, making it a reliable choice for a development environment.

Isolation and Security

One of the standout features of using Dockerized environments for Visual Studio Code (VSCode) is the enhanced level of isolation and security it provides. This benefit is primarily due to the sandboxing nature of Docker containers. In a sandboxed environment, the application (in this case, VSCode) operates in a separate space, isolated from the host system. This isolation plays a crucial role in enhancing security, especially when dealing with untrusted extensions or experimental code.

In a typical setup without Docker, extensions installed in VSCode have access to the host system, which can pose security risks, especially if the extensions are not from trusted sources. In contrast, with Dockerized VSCode environments, VSCode instances only have access to the directories explicitly mounted as volumes in the Docker container. This design means that you, as the user, have full control over which folders the VSCode instance and its extensions can access.

A key security advantage here is that any operation, installation, or changes made within the Dockerized VSCode environment stay within the container. These changes do not affect the host machine directly. For instance, if you're experimenting with a new VSCode extension and it behaves unexpectedly or maliciously, its impact is confined within the container. The host system remains unaffected, safeguarding your primary development setup.

This containment offers a safe playground for testing and using extensions or running experimental code. If something goes wrong, the effects are limited to the container, which can be easily stopped, removed, or reset. Essentially, Docker acts as a robust barrier, ensuring that your host system's integrity and security are maintained, regardless of the activities within the VSCode container.

Furthermore, this isolation ensures that your development environment remains clean and uncluttered. Since nothing inside the Docker container is installed on the host machine, it avoids potential conflicts or pollution of the host's environment. This separation is particularly beneficial for maintaining a stable and secure development environment, especially in scenarios where multiple projects or development contexts are in play.

In summary, the isolation and security provided by Dockerized VSCode environments represent a significant step forward in safe and efficient software development practices. It allows developers to explore, experiment, and work on various projects with the assurance that their primary system remains secure and unaffected.

Resource efficiency

When we compare Docker to traditional virtual machines (VMs), Docker's superior resource efficiency becomes evident. This difference primarily stems from Docker's architecture, which contrasts starkly with that of VMs. Docker containers share the host system's kernel and only require the application and its dependencies to operate. In contrast, VMs need a full operating system to function. This architectural distinction makes Docker containers inherently less resource-intensive, allowing for more agile and responsive environments that are well-suited to modern software development needs.

Docker's lightweight nature is one of its key advantages. Containers require significantly less disk space and memory than VMs, leading to faster deployment and startup times. This quick start-up is especially beneficial in development processes, where time efficiency is critical.

In terms of resource consumption, Docker also has the upper hand. VMs typically allocate fixed resources to each instance, which can be excessive and lead to inefficient resource use. Docker's dynamic resource allocation ensures that containers use only what is necessary, optimizing system resource utilization and enabling the concurrent running of more applications.

Furthermore, Docker simplifies maintenance compared to VMs. A VM is an entire operating system that necessitates regular updates and management, whereas Docker containers, encapsulating only the necessary application components, are easier to manage and update. This simplicity extends to security patches and configuration management.

The most notable advantage, perhaps, is the reduced overhead of Docker containers. By forgoing the need to run a full guest operating system, Docker facilitates better hardware utilization. This efficiency is crucial in environments where resource constraints are a concern, allowing for more applications to be run on the same hardware.

In summary, Docker's containerization approach significantly lessens the strain on computing resources. It provides isolated, efficient, and manageable environments, making it an ideal tool for contemporary development practices that require agility and the ability to quickly adapt to new requirements. Docker's responsive and efficient environment enables developers to focus on innovation and development, unburdened by the complexities of managing weighty and cumbersome VMs.

Consistency, Portability, Version Control, and Update Management

One of the key strengths of using Dockerized environments, especially for development purposes, lies in the consistency and portability it offers across different machines. This aspect is crucial for development teams and individual developers who need to ensure that their applications behave the same way, irrespective of where they are being run.

Docker containers encapsulate not just the application but also its environment. This means that everything needed to run the application - from the operating system, libraries, and dependencies - is packaged together. This packaging ensures that the application runs in the same environment, regardless of the host system. Whether it's being developed on a local machine in Ubuntu or tested on a remote server running CentOS, the application's behavior remains consistent. This consistency eliminates the "it works on my machine" problem, a common challenge in software development.

The containerized approach of Docker also enhances portability. Containers can be easily moved from one environment to another—be it from a local development machine to a test server, or from a staging environment to production. This portability is particularly beneficial for quickly deploying applications or scaling them across different environments or cloud providers.

The practice of version-controlling Dockerfiles and associated scripts (like setup and teardown scripts) plays a pivotal role in maintaining the integrity and history of development environments. By keeping these files in a version control system, teams can track changes over time, easily roll back to previous versions if necessary, and maintain a historical record of how environments have evolved. This practice is particularly useful when parts of the toolchain are updated or when introducing new dependencies.

Having version-controlled Dockerfiles and scripts also simplifies the process of replicating environments. New team members or collaborators can quickly set up their development environments by pulling the latest versions of these files from the repository, ensuring they're working with the most up-to-date configurations.

Furthermore, the ability to version control these environment configurations aids in update management. When a new version of a tool or library is released, it can be tested in isolation, and the changes can be committed only once they are verified to work as expected. This approach minimizes disruptions and ensures that updates are systematically managed.

In conclusion, Docker's ability to provide consistent, portable, and easily replicable environments, combined with the best practices of version controlling Dockerfiles and scripts, makes it an invaluable tool in modern software development. It not only streamlines the development process but also enhances collaboration and reduces the risks associated with environment inconsistencies and updates.

Educational tool

Dockerized environments, particularly when integrated with tools like Visual Studio Code (VSCode), serve as an exceptional educational tool, offering a myriad of benefits for both teaching and learning in the realm of software development. The adoption of Docker in educational settings aligns seamlessly with the growing emphasis on hands-on, practical experience in learning.

The foremost advantage of Docker in an educational context is the creation of a consistent learning environment. This consistency ensures that all students, regardless of their individual hardware or software configurations, have access to the same development setup. Such uniformity is vital in educational settings to avoid discrepancies that can lead to confusion and hinder the learning process.

Moreover, Docker significantly eases the setup of development environments. Educators can distribute pre-configured Docker images or Dockerfiles, enabling students to swiftly and effortlessly establish their development environment. This streamlined setup is especially beneficial in time-constrained educational scenarios like workshops or short courses.

A key aspect of learning with Docker is the provision of a safe, sandboxed space for experimentation. Students can freely experiment with code and various technologies within the confines of Docker containers, without the risk of causing issues in their host system. This safety aspect encourages exploration and learning through experimentation, fostering an environment where trial and error become an integral part of the educational journey.

In addition to facilitating individual learning, Docker’s replicability and scalability are invaluable assets in classroom settings. The ability to deploy the same environment across numerous machines ensures that Docker-based learning can be scaled to accommodate a large number of students, a feature particularly useful in institutional settings.

Resource efficiency is another critical benefit of Docker in education. Its lightweight nature, compared to traditional virtual machines, makes Docker a practical choice for educational institutions that may have limited hardware resources. This efficiency ensures that Docker can be widely adopted without significant hardware investments.

Importantly, learning in Dockerized environments equips students with real-world skills that are highly sought after in the current job market. The practical experience gained with Docker and containerization prepares students for professional roles in software development, bridging the gap between academic learning and industry requirements.

Lastly, the integration of Docker with version control systems in educational projects promotes collaboration and teaches essential skills for modern software development. Students learn about managing collaborative projects and version-controlling Dockerfiles, skills that are indispensable in professional software development teams.

In essence, Docker's adaptability and relevance to industry practices make it an invaluable educational tool, providing a platform that supports both the learning of programming and development concepts and the preparation for the technological demands of the professional world.

You Almost Convinced Me, But I Don't Have Linux on My Machine

If the idea of Dockerized VSCode environments appeals to you but the absence of Linux on your machine seems like a barrier, there's a viable workaround. This involves using a virtual machine to run a Linux distribution, such as Ubuntu, on your current operating system, whether it's Windows or macOS.

The first step in this process is to install VirtualBox, a popular virtualization software. VirtualBox allows you to create a virtual environment on your existing system where you can install and run a different operating system. The process of installing VirtualBox is quite straightforward, with plenty of available tutorials to guide you through the installation.

Once you have VirtualBox installed, the next step is setting up Ubuntu as a virtual machine. This typically involves downloading the Ubuntu ISO file and using it to create a new VM in VirtualBox. The internet is replete with step-by-step guides that walk you through installing Ubuntu on VirtualBox, making it an achievable task even for those new to virtualization.

After successfully installing Ubuntu on your VM, ensure you perform a system update and install Docker and Git. These installations are crucial as they lay the groundwork for setting up the Dockerized VSCode environment. With Docker and Git installed, you can then proceed to follow the instructions provided in this article to create your Dockerized development environment.

One of the great advantages of this approach is that, despite using a VM, you can still benefit from multiple, isolated development environments. Each Docker container within your Ubuntu VM can be configured as a separate development space, complete with its own tools, settings, and dependencies. This means within a single VM, you can simultaneously run multiple development environments tailored for different projects or programming languages.

In essence, this method opens up the possibility of leveraging the benefits of Dockerized VSCode environments, even for those who don't have a Linux-based system. It's a practical solution that extends the power and versatility of Dockerized environments to a wider audience, ensuring that more developers can optimize their workflows and embrace a more efficient development process.

Related Projects

After searching antoher Dockerized VSCode environments browsing internet, several related projects offer various approaches and configurations that could be an alternative to the aproach described in this article:

colebrumley/docker-vscode: Tailored for macOS users, this project provides a Docker container specifically for VSCode. It focuses on a basic setup of VSCode within Docker, making it a straightforward option for those using macOS. GitHub Link.

beatthat/vscodify-docker: This project aims to add a VSCode development environment to any Docker base image. Its flexibility allows integration with a range of Docker base images, offering a generalized solution adaptable to various use cases. GitHub Link.

allamand/docker-vscode: Designed for developers working with Go, this project integrates Go and VSCode in a Docker environment. It's a more focused solution for those specifically developing in Go. GitHub Link.

cmiles74/docker-vscode: Offering a comprehensive development environment, this project includes VSCode along with tools like Dotnet CLI, NPM, and Emacs. It's ideal for developers needing a broad set of resources in their Dockerized VSCode setup. GitHub Link.

Each of these projects presents a unique take on Dockerizing VSCode, ranging from minimal setups to comprehensive environments with multiple tools, highlighting the versatility and adaptability of Docker in development settings.

Implementing Dockerized VSCode Environments: A Detailed Guide

In this chapter, we delve into the practical aspects of setting up your Dockerized Visual Studio Code environment. This step-by-step guide will walk you through the process using Dockerfile configurations and scripts from the GitHub repository, complete with code snippets and explanations. The focus will be on understanding the Dockerfile content and the run.sh script, which are crucial to deploying the VSCode environment within Docker.

Explaining the Dockerfile: We'll break down the Dockerfile line by line. This includes understanding the base image selection, the installation of dependencies, the addition of the Microsoft repository for VSCode, and the creation of a non-root user. Each step will be explained to provide insight into why these elements are essential for a stable and functional Dockerized VSCode environment.
Understanding the run.sh Script: The run.sh script is key to initiating and running the Dockerized VSCode. We will explore the script's functionality, including setting up X11 forwarding for GUI support, configuring Docker run commands, and mounting necessary volumes. This explanation will help you understand how the script interacts with the Docker environment to launch VSCode.
Code Snippets and Execution: Actual code snippets from the Dockerfile and run.sh script will be presented. These snippets will be accompanied by instructions on how to execute them, ensuring you can follow along and implement the setup in your own environment.
Customization Tips: Additional tips on how to customize the Dockerfile and the script for various development needs will be provided. This will enable you to tailor the Dockerized VSCode environment to fit your specific project requirements.

Explaining the Dockerfile



# Use Ubuntu Jammy as the base image
FROM ubuntu:jammy-20240111

# Update package list and install dependencies
RUN apt-get update && apt-get install -y \
    wget \
    gnupg \
    software-properties-common \
    apt-transport-https \
    git \
    iputils-ping \
    iproute2 \
    && rm -rf /var/lib/apt/lists/*

# Add the Microsoft repository for VSCode
RUN wget -qO- https://packages.microsoft.com/keys/microsoft.asc | gpg --dearmor > packages.microsoft.gpg \
    && install -o root -g root -m 644 packages.microsoft.gpg /usr/share/keyrings/ \
    && echo "deb [arch=amd64 signed-by=/usr/share/keyrings/packages.microsoft.gpg] https://packages.microsoft.com/repos/code stable main" > /etc/apt/sources.list.d/vscode.list

# Install Visual Studio Code
RUN apt-get update && apt-get install -y code

# Create a non-root user
RUN useradd -m vscodeuser
USER vscodeuser

# The command to run VSCode
CMD ["code", "--no-sandbox", "--wait", "--verbose"]

The Dockerfile starts by specifying Ubuntu Jammy (version 22.04 as of 2024) as the base image. This choice is due to Ubuntu's stability and popularity, making it an ideal choice for a consistent development environment.

The next steps involve updating the package list and installing essential dependencies. These include tools like wget for downloading files, gnupg for encryption, git for version control, iputils-ping and iproute2 for being able to execute some network related commands in visual studio console like ping or ip address, among others. This setup ensures that all necessary tools I considered are available in the Docker environment.

To install VSCode, the Dockerfile adds the Microsoft repository. This is done by downloading Microsoft's GPG keys, ensuring the authenticity of the packages, and then adding the VSCode repository to the sources list. With the repository in place, the Dockerfile proceeds to update the package list again and installs VSCode. This ensures that the latest version of VSCode when generating the image is installed in the Docker container.

The Dockerfile also takes care of creating a non-root user, vscodeuser. Running applications as a non-root user in Docker containers is a good practice for security. This user will be related to the user that runs the script on the next section (more on this later).

Finally, the Dockerfile specifies the command to run VSCode (code) with certain flags for compatibility and verbose logging. We will run docker with -d flag so logging will not be visible when running docker, but to access logs, docker logs command will be needed to run whilst the container is running. I was not able to run vscode without adding --no-sandbox, but I do not think this is important for this case as we are already controlling how vscode will be sandboxed. Finally, --wait was also needed, because otherwise the container would be killed as soon as vscode is launched.

Understanding the `run.sh` Script



#!/bin/bash
XAUTH=${XAUTHORITY:-$HOME/.Xauthority}
DOCKER_XAUTHORITY=${XAUTH}.docker
cp --preserve=all $XAUTH $DOCKER_XAUTHORITY
xauth nlist $DISPLAY | sed -e 's/^..../ffff/' | xauth -f $DOCKER_XAUTHORITY nmerge -

docker run --rm -d \
    --name vscode-base-1 \
    -u $(id -u):$(id -g) \
    -v /etc/passwd:/etc/passwd:ro \
    -e DISPLAY=unix$DISPLAY \
    -e NO_AT_BRIDGE=1 \
    -e GTK_THEME=Adwaita \
    -v /tmp/.X11-unix:/tmp/.X11-unix:ro \
    -e XAUTHORITY=$DOCKER_XAUTHORITY \
    -v $DOCKER_XAUTHORITY:$DOCKER_XAUTHORITY:ro \
    -v ./home:/home/vscodeuser \
    -v ./Documents:/home/vscodeuser/Documents \
    -v ./home:/home/$USER \
    -v ./Documents:/home/$USER/Documents \
    -e HOME=/home/vscodeuser \
    inigoetxaniz/vscode-base:latest

The run.sh script is crucial for launching the Dockerized VSCode environment. It handles several important tasks that are not so habitual when working with docker.

Perhaps, one of the most unused features of docker is forwarding X11 for GUI support. X11 is a protocol for remote graphical interfaces. It's commonly used to display GUI applications running on a server onto a client machine. In the context of Docker, X11 forwarding enables GUI applications running inside a Docker container, like VSCode, to display on the host machine's screen. The script sets up X11 forwarding by first determining the X11 authentication file location and then copying this information into a new file (DOCKER_XAUTHORITY). This step ensures that the Docker container has the necessary permissions to display its GUI on the host's screen.

Later, the docker run command includes several options to facilitate X11 forwarding. -e DISPLAY=unix$DISPLAY sets the DISPLAY environment variable inside the container, pointing it to the host's display. -v /tmp/.X11-unix:/tmp/.X11-unix:ro mounts the X11 Unix socket from the host into the container, allowing the container to communicate with the host's X server. -e XAUTHORITY=$DOCKER_XAUTHORITY and the associated volumen mount -v $DOCKER_XAUTHORITY:$DOCKER_XAUTHORITY:ro pass the authentication details to the container. This setup works well in an Ubuntu environment, but it's important to note that X11 forwarding configurations may vary based on the host's operating system and Docker version.

While X11 forwarding in Docker is an effective way to display GUI applications like VSCode from a container to the host screen, it's important to consider security implications. X11 forwarding allows the container to interact with the host's X server, which could potentially be a security concern. However, in the context of running a trusted application like VSCode, this setup does not introduce significant additional risks compared to running VSCode directly on the host.

When using X11 forwarding to run VSCode in Docker, the security risk is comparable to running it natively on your system. The isolation provided by Docker still offers a layer of protection, as the activities within the container remain separate from the host system. This means that while the container can display its output on the host, it doesn't have unrestricted access to the host system.

In essence, using X11 forwarding for trusted applications like VSCode in Docker strikes a balance between functionality and security. It allows for the convenience and benefits of Dockerized environments without adding significant security threats, especially when used in trusted and controlled development scenarios.

In this case vscode-base-1 is the name of the container. --rm and -d options make docker container to be deleted after vscode is exited and also run in a detached mode so that nothing is shown in the console and application returns to the user. Logs are still available running in the console docker logs vscode-base-1. -v ./home:/home/vscodeuser sandboxes all home related files to ./home, which allows us to store configuration options, extensions installed during ussage of vscode and other things we could install during the use. -u $(id -u):$(id -g) sets running user as current user in the host. -v ./Documents:/home/vscodeuser/Documents allows us to control which content is accessible from vscode and this is where I propose to store your working project files. Finally, -e HOME=/home/vscodeuser sets /home/vscodeuser as home folder independently of the current user name that is launching vscode from docker. I also added -v ./home:/home/$USER and -v ./Documents:/home/$USER/Documents, because in some cases docker container was refering to the user name that launched docker container. Now both /home/vscodeuser and /home/$USER point to the same location solving this issue.

This guide to setting up a Dockerized VSCode environment, particularly the run.sh script, exemplifies the balance between sophisticated functionality and user-friendliness. The script simplifies complex processes like X11 forwarding, making it accessible even to those who might be new to Docker or Linux environments. By automating intricate setup steps and providing clear explanations for each part, it significantly eases the user's journey in creating a secure, isolated, and efficient development workspace. This approach not only caters to advanced users seeking customization but also to beginners desiring a straightforward path to leveraging Docker for their development needs.

Use Cases and Customization

This section introduces two practical use cases, illustrating how you can tailor Dockerized VSCode environments for specific development needs. The first example focuses on creating an environment for Go language development, while the second caters to developing VSCode extensions with Node.js and NPM. These examples are not exhaustive but serve as a starting point for customization.

Both Dockerfiles begin with the inigoetxaniz/vscode-base image, ensuring a consistent foundation. Customizations involve switching to the root user to install necessary packages specific to each environment — Go for the first and Node.js for the second — and then reverting back to the non-root vscodeuser. This approach maintains security while allowing for necessary installations.

Environment variables are set up as needed, particularly for the Go development environment, to ensure the correct configuration of the workspace. The run.sh scripts for these environments are similar to the one detailed earlier, underscoring the ease of adapting the base setup to various requirements.

These Dockerfiles demonstrate the flexibility and adaptability of Dockerized environments, providing a blueprint for developers to create their customized setups. Whether it's for a specific programming language or a particular type of development work, these examples show how Docker can be effectively used to create efficient and tailored development environments.



# Start from the base image you created for VSCode
FROM inigoetxaniz/vscode-base:latest

USER root

# Download and Install Go
RUN wget https://go.dev/dl/go1.21.6.linux-amd64.tar.gz \
    && tar -C /usr/local -xzf go1.21.6.linux-amd64.tar.gz \
    && rm go1.21.6.linux-amd64.tar.gz

USER vscodeuser

# Set up the Go environment
ENV GOPATH /home/vscodeuser/go
ENV PATH $GOPATH/bin:/usr/local/go/bin:$PATH

# The command to run VSCode
CMD ["code", "--no-sandbox", "--wait", "--verbose"]



# Start from the base image you created for VSCode
FROM inigoetxaniz/vscode-base

USER root

# Download and install the latest LTS version of Node.js
RUN apt update && \
    wget https://nodejs.org/dist/v20.11.0/node-v20.11.0-linux-x64.tar.xz && \
    tar -xJf node-v20.11.0-linux-x64.tar.xz -C /usr/local --strip-components=1 && \
    rm node-v20.11.0-linux-x64.tar.xz

RUN npm install -g yo generator-code

# Switch back to the non-root user
USER vscodeuser

# The command to run VSCode
CMD ["code", "--no-sandbox", "--wait", "--verbose"]

Conclusion

In this guide, we've explored the versatile and secure world of Dockerized VSCode environments. Key highlights include the use of Docker for creating isolated, customizable, and resource-efficient development spaces, enhancing both security and consistency across various machines. We've dissected the Dockerfile to understand its structure and purpose and delved into the run.sh script, particularly focusing on X11 forwarding for GUI support in Docker. We've also covered practical use cases, demonstrating how to tailor Dockerized environments for specific development needs like Go and Node.js. This guide not only provides a step-by-step process for setting up these environments but also emphasizes their adaptability and ease of use, making Dockerized VSCode a valuable tool for developers seeking an efficient and secure development workflow.

Streamlining Home Entertainment: How to Use a Linux Machine for Streaming Content to Smart TVs and More

Iñigo Etxaniz — Sat, 06 Jan 2024 16:09:10 +0000

Introduction

Hey there! Ever thought about turning your old laptop or a Raspberry Pi into a personal streaming powerhouse? You're not alone. Many of us find the idea of ditching the limitations of traditional smart TV platforms (think annoying ads, blocked content, and less-than-stellar USB playback) pretty appealing.

In this guide, I'll walk you through a slightly intricate but rewarding journey to set up your own streaming service. This might be a bit of a lengthy read, but stick with me. By the end, you’ll be able to stream content directly from your laptop to your TV, bypassing those smart TV nuances and enjoying your media just the way you like it.

Let’s dive in and transform the way you experience home entertainment!

Overview of the Setup

We start by adding a new sound interface. This solution eliminates the distraction of audio playing in the speakers of the laptop, channeling all sound directly to OBS for a seamless streaming experience. Then, we install Docker, establishing isolated environments for some of the services of our system. This isolation ensures operational efficiency and prevents interference with other processes on the machine. Next, OBS takes the center stage, adeptly capturing and transmitting both audio and video. It's the key player that brings your content to life, offering user-friendly customization for an optimized streaming setup. Finally, Nginx acts as a versatile powerhouse, adeptly juggling two critical roles. It not only serves as a reverse proxy for OBS, managing the live video streams, but also doubles as a web server. This dual functionality enables Nginx to host the webpage through which your content is accessed, completing the circle of our streaming service. Together, these components interlink and work in harmony, creating a robust and versatile home streaming solution.

In the following sections, we’ll delve deeper into each component, explaining the setup process. Get ready to transform your old laptop or Raspberry Pi into a streaming powerhouse!

Requirements and Preparations

Fine-tuning our setup to match the specific environment of an old laptop, we're demonstrating that high-end hardware isn't a necessity for a functional home streaming service. The chosen machine is a modest Intel i3 with 4GB of RAM, illustrating that even older computers can effectively handle streaming tasks. This setup runs on Ubuntu, a user-friendly operating system that's ideal for such projects. While this guide primarily focuses on Ubuntu, users with different Linux distributions can still follow along, though they might encounter slight variations in the setup process.

Before diving into the streaming setup, updating your Ubuntu system is crucial. Start by opening a terminal; you can usually do this by searching for 'Terminal' in your system's applications menu or by using the shortcut Ctrl + Alt + T. In the terminal, run the following commands to update your system:

sudo apt update
sudo apt upgrade

This process ensures your system is secure and functions smoothly, paving the way for a successful setup. As we proceed, basic command line familiarity will be beneficial, but don't worry—I'll provide all the necessary commands, making it easy to follow along. We'll be configuring an external sound interface to optimize audio output for OBS and installing Docker to run Nginx in a container. This approach keeps our setup clean and modular. Embrace this project with patience and a willingness to learn; it's an exciting journey into the realm of home streaming. With these preparations, we're ready to start building our streaming system, beginning with the new sound interface installation. Let's get started!

Adding a New Sound Interface

Enhancing your streaming setup begins with the addition of a virtual sound interface, a key move for a clean audio experience. This step-by-step guide will help you create a virtual audio streamer on your Linux system, perfect for directing audio to OBS and keeping your laptop's speakers silent.

First, we need to check if PulseAudio, a powerful sound server, is already installed on your machine. Open your terminal and enter:

sudo apt update
sudo apt install pulseaudio

After confirming PulseAudio's installation, the next move is to modify its configuration to add a new virtual sound interface. Using a text editor like Nano, open the PulseAudio configuration file:

sudo nano /etc/pulse/default.pa

Scroll to the end of the file and add the following line:

load-module module-null-sink sink_name=VirtualSink sink_properties=device.description=Virtual_Audio_Streamer

This command instructs PulseAudio to create a new virtual audio output named 'VirtualSink' with the description 'Virtual_Audio_Streamer'. Save the changes and exit the editor. For these changes to take effect, a system restart is necessary.

Once your system is back up, navigate to your sound settings, accessible through the system settings or control panel. In the output section, you will now find 'Virtual_Audio_Streamer' alongside other audio interfaces. Set it as the default output device.

By setting 'Virtual_Audio_Streamer' as the default, the system audio is routed to this virtual sink, meaning you won't hear any sound from your laptop's speakers. Instead, all audio is directed to this virtual interface, ready for OBS to capture.

Installing Docker

Setting up docker on Linux is a straighforward process if you follow the official documentation. If your linux distro is ubuntu you can use the next link Docker's official installation guide ubuntu and if it is debian or raspberry pi 64 bit, you can use the next link Docker's official installation guide ubuntu.

I install following the instructions in Install using the apt repository section. I never had any issues with this method and its quite simple.

I find important to follow post-installation steps just after finishing instalation. This includes adding your user to the Docker group to avoid needing to use sudo for every Docker command. Execute the following commands in your terminal:

sudo groupadd docker
sudo usermod -aG docker $USER

These commands create a new group named 'docker' and add your user to this group. To apply these changes, restart your system. This ensures that your user permissions are updated correctly.

After restarting, you should verify the installation, check if Docker and Docker Compose are correctly installed by running:

docker version
docker compose version

These commands will display the version of Docker and Docker compose installed on your system, confirming that the installation was successful.

By following these steps, you'll have Docker up and running on your system. Docker is an extremely powerful tool for containerization, allowing you to run applications in isolated environments. This will be especially useful for our streaming service setup, where Docker will manage the Nginx server in a container.

Installing OBS Studio on Ubuntu

Installing OBS Studio on your Ubuntu/Debian system is a straightforward process, perfect for beginners, and involves using Ubuntu's or Debian's PPA to ensure easy installation and updates. Type the following command in the terminal to add the OBS Studio PPA:

sudo add-apt-repository ppa:obsproject/obs-studio

This step connects your system to the OBS Studio repository. After adding the PPA, update your system's package list and install OBS studio:

sudo apt update
sudo apt install obs-studio

After installing OBS Studio, hold off on launching it right away. We need to set up Nginx first, as it's a crucial part of our streaming setup. Once Nginx is up and running, we'll circle back to OBS Studio to configure it for streaming. This ensures a smooth integration between your streaming software and the server.

Creating a Web Interface for Streaming

In this part of our setup, we'll create a web interface for the streaming service using Docker and Nginx. This will involve setting up a directory structure and configuring files.

I will not explain how every item works. Docker compose file is quite simple and straightforward, but nginx configuration file was quite tricky as I did not have any previous experience with rtmp and I had to make many different tries till I got a working solution. Web page is also very simple, I did not dedicate much time to this one, I just add a html page with a video player. I did play with other two or three different video players and this one from https://github.com/videojs/video.js, is the one that I liked most and just works fine.

First we need to create a directory for the project. I will use ~/stream-service in my setup:

mkdir ~/stream-service

We will create three directories: ./conf for nginx configuration file, ./hls for the streaming content that will be generated by obs and ./web for hosting index.html file.

cd ~/stream-service
mkdir conf hls web

We will also add in project root folder a docker compose file. This will launch nginx service and redirect container ports to host ports. We will also add a restart: always line so that the service is started automatically when we switch on the laptop. So, create a file called ~/stream-service/docker-compose.yml with the next content:

services:
  nginx:
    image: tiangolo/nginx-rtmp:latest-2023-10-16
    container_name: obs-nginx
    restart: always
    ports:
      - "1935:1935"
      - "80:80"
    volumes:
      - ./conf/obs.conf:/etc/nginx/nginx.conf
      - ./hls:/mnt/hls
      - ./web:/web

Create also the file ~/stream-service/conf/obs.conf file with the next content:

worker_processes auto;
rtmp_auto_push on;
events {}

rtmp {
    server {
        listen 1935;

        application live {
            live on;
            record off;
            hls on;
            hls_path /mnt/hls;
            hls_fragment 3;
            hls_playlist_length 60;
        }
    }
}

http {
    sendfile off;
    tcp_nopush on;
    # aio on;
    directio 512;
    default_type application/octet-stream;

    server {
        listen 80;

        location / {
            # Disable cache
            add_header 'Cache-Control' 'no-cache';

            # CORS setup
            add_header 'Access-Control-Allow-Origin' '*' always;
            add_header 'Access-Control-Expose-Headers' 'Content-Length';

            # allow CORS preflight requests
            if ($request_method = 'OPTIONS') {
                add_header 'Access-Control-Allow-Origin' '*';
                add_header 'Access-Control-Max-Age' 1728000;
                add_header 'Content-Type' 'text/plain charset=UTF-8';
                add_header 'Content-Length' 0;
                return 204;
            }

            types {
                text/html html htm;
                application/dash+xml mpd;
                application/vnd.apple.mpegurl m3u8;
                video/mp2t ts;
            }

            index index.html;
            root /mnt/hls;
        }

        location /index.html {
            alias /web/index.html;
        }

    }
}

Lastly create the ~/stream-service/web/index.html file with the next content:

<!DOCTYPE html>
<html lang="en">
  <head>
    <meta charset="UTF-8" />
    <meta name="viewport" content="width=device-width, initial-scale=1" />
    <title>Live Stream</title>
    <link href="https://vjs.zencdn.net/7.8.4/video-js.css" rel="stylesheet" />
  </head>
  <body>
    <video id="my-video" class="video-js" controls preload="auto" width="640" height="360" data-setup="{}">
      <source src="/stream.m3u8" type="application/x-mpegURL" />
      <p class="vjs-no-js">
        To view this video please enable JavaScript, and consider upgrading to a web browser that
        <a href="https://videojs.com/html5-video-support/" target="_blank">supports HTML5 video</a>
      </p>
    </video>

    <script src="https://vjs.zencdn.net/7.8.4/video.js"></script>
  </body>
</html>

Finally, change the web pages file permission. Otherwise nginx will not be able to serve it:

chmod 777 ~/stream-server/web/index.html

Launching the services

With configuration files ready, in a terminal execute next instructions:

cd ~/stream-service
docker compose up

The second command will take some time. It needs to download nginx image and start the service. Once ready you should see a text similar to Attaching to obs-nginx in the terminal.

Once everything is up and running, your streaming service will be accessible via the web interface (index.html). As users visit your webpage, they can view the content streamed through OBS and processed by your Nginx server.

Before being able to access the web page, you will need to know the ip address of your laptop. This can be obtained with the next instruction in the terminal:

ip address

As we have docker installed you will see many networks. The network we are seeking it's typically named something like eth0 (for wired connections) or wlp2s0 or wlan0 (for wireless connections int laptop or raspberry pi). In my case it's wlp2s0 with the IP address 192.168.1.104. Now I can access to http://192.168.1.104/index.html, but the browser will just tell me that The media could not be found....

Configuring OBS for Streaming

Finally we need to configure OBS studio. Once we have the rest setup it is quite easy. We start by launching OBS stduio. If it's the first time you're opening OBS, you might see the Auto-Configuration Wizard. You can use this to set up your stream, or you can configure it manually in the settings. I will explain the second method.

In the OBS main window, find the 'Settings' button, usually located in the lower right corner. Click on it to open the settings window. Once the popup window opens, we select the stream tab (second in the left part of the window). And set the next settings:

Service: Custom
Server: rtmp://localhost/live
Stream Key: stream

After entering these details, click 'Apply' and then 'OK' to save your streaming settings.

Now, you need to set up your video and audio sources in OBS. Before doing it you can open a media and start playing it. Afterwards, click the '+' button under the 'Sources' window and select the media you are playing. OBS automatically captures your primary desktop audio and microphone. You can adjust these and add additional audio sources in the 'Audio Mixer' section. Once your sources are set up, you're ready to go live. Just click the 'Start Streaming' button in OBS. Your content should now be streaming to your Nginx server and accessible through your configured web interface.

Now you can open in a devices browser the web address configured in the previous section and you should be able to play the media in your other device.

Final Notes

If you encounter issues or the stream isn't accessible on your smart TV, make sure your TV is connected to the same network as your streaming server. Also, check if any firewall or network settings are blocking access to the server. And still, if you're having difficulties or if there's something in this guide that isn't clear, please feel free to post your questions or concerns below. Your feedback is valuable and can help improve this guide. I'll try to provide advice or update the article to address common issues.

By following this guide, you've taken a big step in enhancing your home entertainment system. This DIY approach not only offers a customized viewing experience but also expands your technical skills. Enjoy your new streaming capability, and happy streaming!

Go deadlock detection: delock library

Iñigo Etxaniz — Tue, 05 Dec 2023 14:31:58 +0000

Introduction

In the Go programming world, managing concurrency is a part of everyday life. Go makes concurrent programming relatively straightforward with its built-in primitives like goroutines and channels. However, when it comes to locking structures to ensure thread safety, things can get a bit tricky.

Typically, we use a pointer to a structure and consistently call its methods to maintain thread safety. This approach works well for most scenarios, but complex algorithms demand careful consideration to avoid concurrency issues, especially deadlocks.

While working on a large-scale project for a caching implementation in an authentication service, I encountered an intriguing problem. Occasionally, and quite rarely, the system would hang. Race conditions were unlikely culprits, as I diligently used mutexes within structures. This left me suspecting deadlocks. But how could I confirm this and, more importantly, identify where they were happening? That's where the idea for delock began – a tool specifically designed to detect and pinpoint deadlocks in Go applications.

Understanding Deadlocks in Go

Deadlocks in Go, or in any programming language for that matter, are akin to a gridlock in traffic – everything comes to a standstill. A deadlock occurs when two or more goroutines are waiting on each other to release resources, leading to a situation where none of them can proceed.

Go Runtime Deadlock Detection

This example, albeit a bit contrived, effectively illustrates a fundamental deadlock scenario. In the Execute method, we acquire a mutex lock and then call the same method recursively without ever releasing the lock. This leads to an attempt to acquire the lock again while it's still held by the same goroutine, causing a deadlock.

Though this is a simple and somewhat unrealistic example, it mirrors real-world scenarios where deadlocks occur due to improper lock handling. It’s a classic case of a process indefinitely waiting on itself. The Go runtime is adept at catching such blatant deadlocks in a single-threaded context, where the entire application grinds to a halt. This example serves as a reminder of how subtle mistakes in concurrency management can lead to significant issues, and why it's crucial to code with caution when dealing with locks.

package main

import (
    "log"
    "sync"
    "time"
)

type LockExample struct {
    mu    sync.Mutex
    Value int
}

func (l *LockExample) Execute() {
    l.mu.Lock()
    defer l.mu.Unlock()

    time.Sleep(100 * time.Millisecond)
    l.Value++
    log.Printf("value: %d", l.Value)
    l.Execute()
}

func main() {
    var l LockExample
    l.Execute()
}

Upon running the simple deadlock example with standard sync.Mutex, the Go runtime effectively detects and reports a deadlock. Here's what happens:

The log statement 2023/12/05 12:38:02 value: 1 indicates that the Execute function ran once and incremented Value.
Immediately after, the program crashes with a fatal error: all goroutines are asleep - deadlock!. This error is thrown by the Go runtime when it detects that all goroutines are waiting on each other, with no possibility of resuming.
The stack trace provided points to where the deadlock occurs, highlighting the recursive call to Execute that attempts to reacquire an already held lock.

This result demonstrates the Go runtime's ability to detect simple deadlocks where the entire program is stuck waiting for locks to be released. It serves as a practical example of how easy it is to fall into a deadlock trap and the importance of careful lock management.

go run ./examples/simple_std
2023/12/05 12:38:02 value: 1
fatal error: all goroutines are asleep - deadlock!

goroutine 1 [sync.Mutex.Lock]:
sync.runtime_SemacquireMutex(0x485040?, 0x0?, 0xc00007e050?)
        /usr/local/go/src/runtime/sema.go:77 +0x25
sync.(*Mutex).lockSlow(0xc0000120e0)
        /usr/local/go/src/sync/mutex.go:171 +0x15d
sync.(*Mutex).Lock(...)
        /usr/local/go/src/sync/mutex.go:90
main.(*LockExample).Execute(0xc0000120e0)
        /home/inigo/Documents/Projects/delock/examples/simple_std/main.go:15 +0x4c
main.(*LockExample).Execute(0xc0000120e0)
        /home/inigo/Documents/Projects/delock/examples/simple_std/main.go:21 +0x12a
main.main()
        /home/inigo/Documents/Projects/delock/examples/simple_std/main.go:26 +0x1f
exit status 2

Limitations in Complex Scenarios

In real-world programming, especially in services, it's common to have multiple goroutines performing different tasks. In such environments, a deadlock might not stop the entire application, making it more challenging to detect and resolve.

To illustrate this, let's modify our previous example to fit into a broader context:

package main

import (
    "log"
    "sync"
    "time"
)

type LockExample struct {
    mu    sync.Mutex
    Value int
}

func (l *LockExample) GetValue() int {
    l.mu.Lock()
    defer l.mu.Unlock()

    return l.Value
}
func (l *LockExample) Execute() {
    l.mu.Lock()
    defer l.mu.Unlock()

    time.Sleep(100 * time.Millisecond)
    l.Value++
    log.Printf("value: %d", l.Value)
    l.Execute()
}

func main() {
    var l LockExample
    go func() {
        l.Execute()
    }()
    readValueChan := make(chan int)
    go func() {
        readValueChan <- l.GetValue()
    }()
    for {
        timeoutChan := make(chan int)
        go func() {
            time.Sleep(1 * time.Second)
            timeoutChan <- 0
        }()

        select {
        case <-timeoutChan:
            log.Printf("waiting...")
        case <-readValueChan:
            log.Printf("value: %d", l.GetValue())
            return
        }
    }
}

In this version, we run Execute in a separate goroutine, simulating a background process in a service. Another goroutine attempts to read the Value field, while the main routine periodically checks if the value has been read.

Output and Analysis

Running this code produces an output like this:

2023/12/05 12:42:25 value: 1
2023/12/05 12:42:26 waiting...
...

The first log indicates that Value incremented once. However, the subsequent "waiting..." logs reveal the main goroutine is active but cannot read Value again due to the deadlock in Execute. This setup demonstrates a subtle deadlock that does not halt the entire program, exemplifying how deadlocks in multi-threaded applications can be more elusive and challenging to detect.

This scenario highlights the need for tools like delock, which can help identify and diagnose deadlocks in more complex, real-world applications.

Detecting Deadlocks with Delock

In our previous example, the Go runtime couldn’t detect the deadlock because other parts of the program were still running. Now, let's see how delock changes the game.

We modify our code to use delock.Mutex instead of sync.Mutex. This small change allows delock to detect and report deadlocks even in complex scenarios where the Go runtime might not.

package main

import (
    "log"
    "time"

    "github.com/ietxaniz/delock"
)

type LockExample struct {
    mu    delock.Mutex
    Value int
}

func (l *LockExample) GetValue() int {
    lockID, err := l.mu.Lock()
    if err != nil {
        panic(err)
    }
    defer l.mu.Unlock(lockID)

    return l.Value
}
func (l *LockExample) Execute() {
    lockID, err := l.mu.Lock()
    if err != nil {
        panic(err)
    }
    defer l.mu.Unlock(lockID)

    time.Sleep(100 * time.Millisecond)
    l.Value++
    log.Printf("value: %d", l.Value)
    l.Execute()
}

func main() {
    var l LockExample
    go func() {
        l.Execute()
    }()
    readValueChan := make(chan int)
    go func() {
        readValueChan <- l.GetValue()
    }()
    for {
        timeoutChan := make(chan int)
        go func() {
            time.Sleep(1 * time.Second)
            timeoutChan <- 0
        }()

        select {
        case <-timeoutChan:
            log.Printf("waiting...")
        case <-readValueChan:
            log.Printf("value: %d", l.GetValue())
            return
        }
    }
}

In the modified code, the standard locking mechanism:

l.mu.Lock()
defer l.mu.Unlock()

is transformed into:

lockID, err := l.mu.Lock()
if err != nil {
    panic(err)
}
defer l.mu.Unlock(lockID)

Here's what's happening:

lockID: This is a unique identifier for the lock instance within the delock library. It's used internally to track and manage stack data associated with each lock operation.
err: If the Lock() method takes longer than the defined timeout, it returns an error. This error contains valuable stack trace information indicating where the deadlock might be occurring.

For those concerned about verbosity, using fold-ninja, a VSCode extension I developed, can streamline the appearance in the editor. The expanded code block:

lockID, err := l.mu.Lock()
if err != nil {
    panic(err)
}
defer l.mu.Unlock(lockID)

is neatly collapsed to:

lockID, err := l.mu.Lock()
defer l.mu.Unlock(lockID)

This makes the code more manageable and less cluttered while maintaining the functionality and robustness provided by delock.

Output and Analysis

When we run this code:

2023/12/05 12:52:42 value: 1
2023/12/05 12:52:43 waiting...
panic: Deadlock detected

Unlike before, delock detects the deadlock situation and panics, providing detailed stack trace information. This shows exactly where the deadlock occurs, demonstrating delock's ability to uncover hidden issues in multi-threaded applications.

By using delock, developers can gain insights into complex deadlock situations, making it a powerful tool for debugging and ensuring smooth concurrency management in Go services.

Tackling More Complex Deadlock Scenarios

Before we dive into a more intricate deadlock scenario, let's recall the primary function of the delock tool: delock is designed to help us identify and understand deadlocks in our Go applications, especially in complex cases where traditional Go runtime detection might fall short. Its ability to provide detailed stack trace information is invaluable in diagnosing and resolving these challenging issues.

Now, let's explore a scenario involving parent-child relationships within data structures, where deadlocks can be more elusive and difficult to detect.

Not all deadlock issues are as straightforward as the ones we've explored so far. In real-world applications, especially those involving intricate parent-child relationships between data structures, deadlocks can be more elusive. Let's delve into a more challenging example, where two interconnected data structures, each with its own lock, interact in ways that could potentially lead to deadlock situations.

In this scenario, we'll examine how parent and child structures, both equipped with mutexes, can create complex locking sequences that might result in deadlocks under certain conditions. This complexity showcases the subtleties of deadlock issues in sophisticated program architectures.

The following code demonstrates the same scenario using the delock library. While delock is instrumental in detecting deadlocks, it's important to note that it's primarily designed for debugging and testing purposes rather than production use. The additional processing delock performs when acquiring and releasing locks might introduce overhead not suitable for a production environment.

package main

import (
    "fmt"
    "os"
    "strconv"
    "time"

    "github.com/ietxaniz/delock"
)

type Child struct {
    mu     delock.RWMutex
    value  string
    id     int
    parent *Parent
}

type Parent struct {
    mu         delock.RWMutex
    items      []*Child
    lastID     int
    numChanges int
}

func (p *Parent) Add(value string) {
    lockID, err := p.mu.Lock()
    if err != nil {
        panic(err)
    }
    defer p.mu.Unlock(lockID)

    time.Sleep(1 * time.Millisecond)
    p.lastID++
    if p.items == nil {
        p.items = make([]*Child, 0)
    }
    p.items = append(p.items, &Child{value: value, id: p.lastID, parent: p})
}

func (p *Parent) GetChildByID(childID int) *Child {
    lockID, err := p.mu.RLock()
    if err != nil {
        panic(err)
    }
    defer p.mu.RUnlock(lockID)

    for _, item := range p.items {
        if item.GetID() == childID {
            return item
        }
    }

    return nil
}

func (p *Parent) GetNumChanges() int {
    lockID, err := p.mu.RLock()
    if err != nil {
        panic(err)
    }
    defer p.mu.RUnlock(lockID)

    return p.numChanges
}

func (p *Parent) IncreaseNumChanges() {
    lockID, err := p.mu.RLock()
    if err != nil {
        panic(err)
    }
    defer p.mu.RUnlock(lockID)

    time.Sleep(1 * time.Millisecond)

    p.numChanges++
}

func (c *Child) GetID() int {
    lockID, err := c.mu.RLock()
    if err != nil {
        panic(err)
    }
    defer c.mu.RUnlock(lockID)

    time.Sleep(1 * time.Millisecond)

    return c.id
}

func (c *Child) SetValue(value string) {
    lockID, err := c.mu.Lock()
    if err != nil {
        panic(err)
    }
    defer c.mu.Unlock(lockID)

    time.Sleep(1 * time.Millisecond)
    c.value = value
    c.parent.IncreaseNumChanges()
}

func main() {
    var p Parent
    if len(os.Args) < 3 {
        println("Usage: complex <n> <t>")
        return
    }
    n, err := strconv.Atoi(os.Args[1])
    if err != nil {
        panic(err)
    }
    t, err := strconv.Atoi(os.Args[2])
    if err != nil {
        panic(err)
    }
    go func() {
        for i := 0; i < n; i++ {
            time.Sleep(time.Duration(t) * time.Millisecond)
            p.Add(fmt.Sprintf("child%d", i))
            go func(id int, sleepMilliseconds int) {
                for {
                    time.Sleep(time.Duration(sleepMilliseconds) * time.Millisecond)
                    child := p.GetChildByID(id)
                    if child != nil {
                        child.SetValue(fmt.Sprintf("child%d", id))
                    }
                }
            }(i, t)
        }
    }()

    lastReadedNumChanges := 0

    go func() {

        for {
            time.Sleep(1 * time.Second)
            lastReadedNumChanges = p.GetNumChanges()
        }
    }()

    for {
        time.Sleep(1 * time.Second)
        println(lastReadedNumChanges)
    }
}

It's important to note that while this example runs smoothly under normal, unstressed conditions, it can lead to a deadlock under high-stress scenarios. This behavior can be observed by adjusting the number of runner goroutines (n) and the sleep duration in milliseconds (t) in each step. When n is low and t is high, the program appears to function correctly. However, increasing n while decreasing t creates conditions ripe for a deadlock, demonstrating how system load and operation timing can impact the occurrence of deadlocks in complex systems.

With the complex example in mind, we'll run the code using various parameters to simulate different stress levels. By altering the number of goroutines (n) and the sleep duration (t), we aim to observe the program's behavior under varying conditions. Sometimes, the Value might increase steadily; other times, it may get stuck, raising suspicions of a deadlock. To confirm these suspicions, we'll switch to using delock. This will help us determine not only if a deadlock is occurring but also under which specific conditions. Once detected, we'll delve into the code to unravel the reason behind the deadlock, enhancing our understanding of these intricate scenarios.

go run ./examples/complex_std/ 50 50
0
105
395
780
780
780
780
780
780

go run ./examples/complex_std/ 50 100
28
28
136
300
496
752
1035
1407
1787
2171
2552
2937

50 tasks with 50ms of sleep time seems to have problems while 50 tasks with 100ms of sleep time seems to work properly. So lets execute 50 tasks with 10ms sleep time using delock version.

export DELOCK_TIMEOUT=10000 && go run ./examples/complex 50 10
0
...
0
panic: Deadlock detected


* * * * * READ LOCK * * * * * * * *
goroutine 52 [running]:
github.com/ietxaniz/delock.(*RWMutex).RLock(0xc0000b0120)
        /home/inigo/Documents/Projects/delock/rwmutex.go:159 +0x65
main.(*Parent).GetChildByID(0xc0000b0120, 0x3)
        /home/inigo/Documents/Projects/delock/examples/complex/main.go:42 +0x38
main.main.func1.1(0x0?, 0xa)
        /home/inigo/Documents/Projects/delock/examples/complex/main.go:119 +0x55
created by main.main.func1 in goroutine 18
        /home/inigo/Documents/Projects/delock/examples/complex/main.go:116 +0x54


* * * * * READ LOCK * * * * * * * *
goroutine 14 [running]:
github.com/ietxaniz/delock.(*RWMutex).RLock(0xc0000b0120)
        /home/inigo/Documents/Projects/delock/rwmutex.go:159 +0x65
main.(*Parent).IncreaseNumChanges(0xc0000b0120)
        /home/inigo/Documents/Projects/delock/examples/complex/main.go:68 +0x2a
main.(*Child).SetValue(0xc00007c300, {0xc000220020, 0x6})
        /home/inigo/Documents/Projects/delock/examples/complex/main.go:95 +0xb1
main.main.func1.1(0x0?, 0xa)
        /home/inigo/Documents/Projects/delock/examples/complex/main.go:124 +0x15a
created by main.main.func1 in goroutine 18
        /home/inigo/Documents/Projects/delock/examples/complex/main.go:116 +0x54

...

exit status 2

"I've omitted several lines from the output because numerous locks were acquired simultaneously in this scenario. However, I retained two critical outputs for analysis. The first one demonstrates typical behavior, but the second raises a red flag, particularly at examples/complex/main.go:95. This line is our key to understanding the deadlock. Let's examine the related code segment, with line numbers for reference:"

87     func (c *Child) SetValue(value string) {
88      lockID, err := c.mu.Lock()
89      if err != nil {
90          panic(err)
91      }
92      defer c.mu.Unlock(lockID)
93
94      c.value = value
95      c.parent.IncreaseNumChanges()
96     }

Line 95, c.parent.IncreaseNumChanges() is called while holding a lock. This is a very common mistake. The problem is that IncreaseNumChanges also acquires a lock. So, if IncreaseNumChanges is called while holding a lock, it will never be able to acquire the lock and will wait forever. This is a deadlock.

So, this demonstrates how this tool can effectively help us find deadlocks in our code.

Solution to the Found Deadlock Issue

Upon discovering the deadlock, two potential solutions come to mind. The first is straightforward, while the second aligns more with my preferred approach:

Straightforward Solution

func (c *Child) SetValue(value string) {
    lockID, err := c.mu.Lock()
    if err != nil {
        panic(err)
    }

    c.value = value
    parent := c.parent
    c.mu.Unlock(lockID)
    parent.IncreaseNumChanges()
}

This solution works by explicitly unlocking before exiting the function. However, it risks future errors if the code is modified and the lock is not properly released in all paths.

Preferred Solution

func (c *Child) setValueSafe(value string) *Parent {
    lockID, err := c.mu.Lock()
    if err != nil {
        panic(err)
    }
    defer c.mu.Unlock(lockID)

    c.value = value
    return c.parent
}

func (c *Child) SetValue(value string) {
    parent := c.setValueSafe(value)
    parent.IncreaseNumChanges()
}

This approach achieves the same goal but retains the defer statement for unlocking. This makes the code more robust against future changes and ensures that the lock is always properly released.

With these corrections applied, the program's output now aligns with our expectations, as evidenced in the terminal:

go run ./examples/complex_solved 100 1
0
404
846
1299
1763
2203
4440
7563
10670
13750
16838
19926
23012
26104
29207
32306
35417

Repository and Examples

The delock library is hosted on GitHub and can be found at https://github.com/ietxaniz/delock. The repository includes an examples folder, showcasing various scenarios:

simple and simple_std: Demonstrating deadlock detection with delock and the Go standard library, respectively.
simple2 and simple2_std: Highlighting a scenario where Go's standard deadlock detection might fail, but delock succeeds.
complex, complex_std, and complex_solved: Exploring more intricate deadlock situations with varying outcomes depending on the detection method used and whether the deadlock issue is resolved.

To run these examples, clone the repository and execute the desired example from the root folder using the command: go run pathToExample. For instance: go run ./examples/complex.

Open Invitation for Collaboration

I welcome any suggestions, feedback, or contributions to the delock project. Whether it's improving existing features, proposing new ones, or enhancing documentation and examples, your input is valuable. Feel free to raise issues, submit pull requests, or reach out for discussions on potential enhancements.

Rust Concurrency Explained: Cleaning Code with Traits and Simplifying Services

Iñigo Etxaniz — Tue, 21 Nov 2023 10:31:17 +0000

Rust Concurrency: Cleaning Code with Traits and Simplifying Services

Introduction

As I continue my journey with Rust, I've found that revisiting and reviewing my code is not just a best practice but a learning tool in itself. After getting my code to work, I like to take a step back and ponder if there are ways to enhance its readability and efficiency. This reflective practice often leads to significant improvements, making my code cleaner and more understandable.

Recently, I revisited the initial commit of my project, rust_service_example (link to first commit on GitHub). While reviewing, I noticed some areas that didn't quite sit right with me. For instance, the process of read-locking and write-locking was taking up more lines than necessary, creating repetitive patterns that I think will cumbersome during code reviews. This realization led me to explore how traits could streamline these aspects, which I'll delve into later in this article.

Another area I've been questioning is my approach to error handling. Initially, I transitioned from returning a String describing the error to returning Box<dyn Error>, which seemed like a step in the right direction. However, I'm now experimenting with the anyhow crate, seeking a more refined method, although I'm still evaluating if it's the best fit.

Lastly, I made a small change in the main code. By changing the field types in my Service struct from RwLock<T> to Arc<RwLock<T>>, I was able to implement the Clone trait. This change has simplified the code in main.rs, making it more approachable, especially from a beginner's perspective.

For those interested in seeing the code and its evolution, feel free to explore the repository on GitHub.

In this article, I'll share these refinements and discuss how they contribute to better, cleaner Rust code. Join me as I navigate through these improvements, and let's learn together how small changes can make a big difference.

Refining Error Handling

In my journey of transitioning from Go to Rust, I've come to realize the nuances in how both languages approach error handling. In my previous articles, I drew parallels between Rust and Go in terms of verbosity and explicitness in managing errors. However, with deeper exploration, I've discovered some key differences and improvements in my Rust coding practice, particularly regarding error handling.

Revisiting Rust vs. Go in Error Handling

In Go, the convention of functions returning both a value and an error is a pattern I've always appreciated for its clarity. This approach makes error handling an integral and explicit part of the function's contract. Initially, I viewed Rust's error handling through a similar lens, especially when using match statements to handle Result types.

Rust, however, offers a more streamlined approach with the ? operator. This operator allows for elegant error propagation, reducing the verbosity that comes with manual error checking. Unlike the explicit handling in Go, Rust's ? operator simplifies the code, making it less verbose while still maintaining clarity and predictability.

The Shift to `anyhow` in Rust Error Handling

My initial foray into Rust error handling involved returning strings as errors. While this method was straightforward, it wasn't in line with Rust's standard practices. I soon learned that a more idiomatic approach in Rust is to return Box<dyn Error>. This standard has its advantages, primarily its flexibility and compatibility with Rust's error propagation mechanisms.

Why `Box<dyn Error>`?

In Rust, Box<dyn Error> is commonly used for error handling, similar to how functions in Go often return an error type. Just like in Go, where a function can return a result and an error, Box<dyn Error> in Rust provides a dynamic way to handle various error types uniformly. This approach is particularly beneficial in Rust because of the language's ? operator, which enables seamless error propagation.

Using Box<dyn Error> in Rust is akin to Go's error return type in that it allows for a flexible response to different error scenarios. However, Rust's ? operator simplifies error handling further. By adding ? at the end of an operation, Rust can elegantly handle errors without the verbosity typical in Go, where each error requires explicit checking and handling.

The Challenge of Mixed Error Types

However, a challenge arises when there's a mix of error types being returned - sometimes a string, other times a boxed error. This inconsistency necessitates additional code for handling and converting errors, leading to a less streamlined and more verbose approach.

Embracing `anyhow` for Streamlined Error Handling

Recognizing this, I turned to the anyhow library for a more unified and efficient error handling strategy. anyhow simplifies the process by allowing the use of a single error type across the codebase. This aligns well with Rust's philosophy of concise and clear error handling while providing the flexibility to handle a wide range of error scenarios.

Using anyhow, I can write code that's consistent in its error handling approach, enhancing readability and maintainability. Moreover, anyhow integrates seamlessly with Rust's ? operator, further reducing verbosity and complexity.

The Importance of Consistency and Adaptability

In conclusion, the key takeaway from my experience with error handling is the importance of consistency. Whether it's using Box<dyn Error>, anyhow, or another method, sticking to a consistent approach is crucial. It's also essential to remain open to refining your error handling strategy as you gain more insight into Rust's best practices and the tools available. This flexibility allows for continuous improvement in writing more effective and idiomatic Rust code.

Practical Benefits of Using `anyhow`

Implementing anyhow in my Rust projects has brought several advantages:

Simplified Error Propagation: With anyhow, I can easily return errors from functions without worrying about their specific types. The ? operator works seamlessly with anyhow::Error, further reducing boilerplate code.
Enhanced Readability: The use of a single error type declutters the code, making it more readable and maintainable. It's easier to understand and handle errors when they're consistently represented.

Continuous Learning and Refinement

This exploration into Rust's error handling is a testament to the ongoing learning process in programming. What seemed like an established understanding can evolve as you delve deeper into a language's features and best practices. By embracing libraries like anyhow and utilizing Rust-specific features like the ? operator, I'm refining my approach to writing more idiomatic and efficient Rust code.

Refactoring the `Service` Struct for Simplified Usage

In the process of evolving the Rust codebase, a significant refactor was made to the Service struct. This change not only streamlined the code but also enhanced its readability and maintainability. Let's explore the transformation of the Service struct and how it positively impacted the usage in main.rs.

Original `Service` Struct

Initially, the Service struct was defined as follows:

pub struct Service {
    count: std::sync::RwLock<u64>,
    count_thread1: std::sync::RwLock<u64>,
    count_thread2: std::sync::RwLock<u64>,
    write_lock: std::sync::RwLock<u8>,
}

This structure effectively managed the state using Rust's RwLock for thread-safe mutable access. However, when it came to sharing Service instances across threads, the usage in main.rs required explicit handling of Arc (Atomic Reference Counting pointers) to manage the shared ownership.

The Refactored `Service` Struct

To simplify this pattern, the Service struct was refactored as follows:

#[derive(Clone)]
pub struct Service {
    count: Arc<RwLock<u64>>,
    count_thread1: Arc<RwLock<u64>>,
    count_thread2: Arc<RwLock<u64>>,
    write_lock: Arc<RwLock<u8>>,
}

By incorporating Arc<RwLock<T>> directly into the struct and deriving the Clone trait, the Service struct became more ergonomic to use. This change abstracts away the explicit handling of Arc, making the struct more straightforward to clone and share across threads.

Simplified Usage in `main.rs`

Before the Refactor

Originally, sharing an instance of Service across threads required explicitly creating an Arc and then cloning it:

let service = Arc::new(Service::new());

let service1 = Arc::clone(&service);
let service2 = Arc::clone(&service);

After the Refactor

With the refactored Service struct, the code in main.rs becomes cleaner:

let service = Service::new();

let service1 = service.clone();
let service2 = service.clone();

This refactoring makes the code more intuitive and straightforward. The cloning of the service instance is now a simple method call, enhancing the overall readability.

Using Traits for Code Simplification and Cleaning

One of the most powerful features of Rust is its trait system, which allows for code abstraction and reuse in a way that's both efficient and elegant. In this part of the article, I'll demonstrate how we can leverage traits to simplify and clean up our code. Our goal is to transform a common pattern in our codebase into a more concise and readable form, without altering the application's functionality.

The Objective: Streamlining Lock Operations

Consider this snippet from our current codebase:

let count = *self
            .count
            .read()
            .map_err(|e| format!("Failed to read-lock count: {}", e))?;

Here, we're acquiring a read lock on a resource and handling potential errors. While this code is functional, it's also somewhat verbose and repetitive, especially if similar patterns are used throughout the application.

Now, let's look at how we can streamline this with the help of a custom trait:

let count = *self.count.lock_read("count")?;

With this new approach, the code becomes much more succinct and clear. The error handling is still there, but it's abstracted away by our trait, making the main logic easier to read and maintain.

Simplifying Lock Operations with the `LockExt` Trait in Rust

In Rust, managing locks, especially with RwLock, can often involve verbose and repetitive error handling. To address this, let's explore the LockExt trait, a solution that streamlines these operations. This trait is a great example of how Rust's powerful trait system can be used to enhance code readability and efficiency.

Here's the code for the LockExt trait:

use anyhow::{anyhow, Error};
use std::sync::{Arc, RwLock};
use std::sync::{RwLockReadGuard, RwLockWriteGuard};

pub trait LockExt<T> {
    fn lock_write(&self, name: &str) -> Result<RwLockWriteGuard<T>, Error>;
    fn lock_read(&self, name: &str) -> Result<RwLockReadGuard<T>, Error>;
}

impl<T> LockExt<T> for Arc<RwLock<T>> {
    fn lock_write(&self, name: &str) -> Result<RwLockWriteGuard<T>, Error> {
        self.write()
            .map_err(|e| anyhow!("Failed to write-lock {}: {}", name, e))
    }

    fn lock_read(&self, name: &str) -> Result<RwLockReadGuard<T>, Error> {
        self.read()
            .map_err(|e| anyhow!("Failed to read-lock {}: {}", name, e))
    }
}

This trait abstracts the process of acquiring read and write locks, simplifying error handling. Instead of writing lengthy error handling every time a lock is acquired, the LockExt trait encapsulates this in two concise methods: lock_write and lock_read. And allows the simplification we where looking for at the beginning of this section.

Emphasizing the Choice of `anyhow` in the `LockExt` Trait

Having previously discussed the nuances of error handling in Rust and the advantages of utilizing libraries like anyhow, the implementation of the LockExt trait further exemplifies why anyhow was the preferred choice. This trait demonstrates the practical application of anyhow in a real-world scenario, highlighting its benefits in streamlining error handling.

Simplified Error Handling with `anyhow`

Consider the lock_write function using anyhow:

use anyhow::{Result, anyhow};
use std::sync::{RwLockWriteGuard, RwLock};

fn lock_write<T>(&self, lock: &RwLock<T>, name: &str) -> Result<RwLockWriteGuard<T>> {
    lock.write()
        .map_err(|e| anyhow!("Failed to write-lock {}: {}", name, e))
}

This implementation leverages anyhow for its concise and expressive error reporting. The anyhow! macro enables quick conversion of errors into an anyhow::Error, complete with a descriptive message. This approach significantly reduces boilerplate and enhances readability.

Contrast this with the more traditional approach without anyhow:

use std::sync::{RwLockWriteGuard, RwLock};
use std::error::Error;
use std::fmt;

struct MyError {
    details: String,
}

// Implementations for MyError...

fn lock_write<T>(&self, lock: &RwLock<T>, name: &str) -> Result<RwLockWriteGuard<T>, Box<dyn Error>> {
    lock.write()
        .map_err(|e| Box::new(MyError::new(&format!("Failed to write-lock {}: {}", name, e))) as Box<dyn Error>)
}

Here, the necessity of a custom error type (MyError) and the additional code for error management increases the complexity, making the function more verbose.

Conclusion

Consistent Error Handling with `anyhow`

The transition to using anyhow for error handling in Rust represents more than just a technical refinement; it signifies an embrace of idiomatic Rust practices that prioritize clarity, brevity, and robustness. This shift not only made the error handling in my codebase more consistent but also underscored the importance of adaptability and continuous learning in software development. As Rust continues to evolve, so too should our approaches to coding within its ecosystem. The adoption of anyhow is a testament to this philosophy, showcasing a commitment to write code that is not only functional but also clean and maintainable.

The Impact of Refactoring `Service` Struct

Embedding Arc directly into the Service struct and leveraging the Clone trait proved to be a subtle yet impactful change. This refactoring underscores the philosophy of smart struct design in Rust—where the focus is not just on the functionality but also on how the structure of the code can lead to more intuitive and elegant usage patterns. By making these adjustments, the Service struct became more aligned with Rust's design principles, offering a more streamlined and idiomatic way of handling shared state in concurrent environments.

Leveraging Traits for Code Simplification

The implementation of the LockExt trait is a prime example of Rust's powerful trait system at work. It highlights how traits can be used not just for defining shared behavior, but also for simplifying and cleaning up code. This approach is particularly useful in Rust, where managing complexity is key to writing effective programs. By abstracting repetitive patterns into a trait, the codebase becomes more organized, allowing for easier maintenance and future enhancements. The LockExt trait, therefore, is not just a utility but a representation of the Rust philosophy of making code more modular and expressive.

Rust Concurrency Explained: A Beginner's Guide to Arc and Mutex

Iñigo Etxaniz — Fri, 17 Nov 2023 19:47:32 +0000

Introduction

Jumping into Rust programming has been quite a ride. At first, it felt a bit like trying to solve a puzzle, especially when it came to understanding how Rust handles data. The borrowing concept? Got it, no big deal. But things got trickier when I started wondering how you could let two parts of your program share and change the same data without stepping on each other's toes. That's where Arc and Mutex come into play.

So, I thought, why not make an example to figure this out? The idea was to set up two threads in a Rust program, with both threads messing around with the same bunch of data. They would each bump up a couple of counters - one that they both share and another that's just for themselves. Then, right after each bump, we'd take a look at where the counters stand.

This wasn't just about writing some code; it was more about getting my head around Rust's way of handling data when multiple things are happening at once. Here's the story of what I did, what I learned, and how it all works.

The Application Idea: Playing with Counters

So, what's this example all about? Well, it's pretty straightforward:

Two Threads, Two Tasks: Imagine two little workers (threads) in our program. They're both doing a similar job but in their own way. One is labeled thread1 and the other thread2.
What They Do: Each time they get to work (thread1 and thread2 doing their thing), they do a couple of things:
- thread1 ups the numbers on two counters: a shared one (count) and its own (count_thread1). Then it shouts out (prints) the current score.
- thread2 does pretty much the same, but with count (the shared one) and count_thread2 (its own personal counter).
Keep It DRY (Don't Repeat Yourself): I wanted to be smart about this and not write the same piece of code twice for showing the score. So, both threads use the same function to print the status. It's like they're using the same megaphone to announce their results.
Playing Nice with Sharing: Here's the catch, though. When one thread is updating the counters and about to announce the score, we don't want the other one to barge in and mess things up. It's like saying, "Hold on, let me finish talking before you jump in." This means we need to be a bit clever about locking things up.

That's the gist of our little Rust adventure. Two threads, a few counters, and making sure they don't trip over each other while they're at it.

Explaining the Service Code: Mutex Magic in Rust

Alright, let's dive into the heart of our Rust code - the Service struct. This is where the magic happens, and by magic, I mean carefully managing access to shared data with mutexes. Here's the code:

pub struct Service {
    count: std::sync::RwLock<u64>,
    count_thread1: std::sync::RwLock<u64>,
    count_thread2: std::sync::RwLock<u64>,
    write_lock: std::sync::RwLock<u8>,
}

impl Service {
    pub fn new() -> Self {
        Service {
            count: std::sync::RwLock::new(0),
            count_thread1: std::sync::RwLock::new(0),
            count_thread2: std::sync::RwLock::new(0),
            write_lock: std::sync::RwLock::new(0),
        }
    }

    pub fn get_counts(&self) -> Result<(u64, u64, u64), String> {
        let count = *self
            .count
            .read()
            .map_err(|e| format!("Failed to read-lock count: {}", e))?;
        let count_thread1 = *self
            .count_thread1
            .read()
            .map_err(|e| format!("Failed to read-lock count_thread1: {}", e))?;
        let count_thread2 = *self
            .count_thread2
            .read()
            .map_err(|e| format!("Failed to read-lock write_lock: {}", e))?;
        Ok((count, count_thread1, count_thread2))
    }

    pub fn increment_counts_thread1(&self) -> Result<(u64, u64, u64), String> {
        let mut count = self
            .count
            .write()
            .map_err(|e| format!("Failed to write-lock count: {}", e))?;
        let mut count_thread1 = self
            .count_thread1
            .write()
            .map_err(|e| format!("Failed to write-lock count_thread1: {}", e))?;
        let mut write_lock = self
            .write_lock
            .write()
            .map_err(|e| format!("Failed to write-lock write_lock: {}", e))?;
        *count += 1;
        *count_thread1 += 1;
        *write_lock = 1;
        drop(count);
        drop(count_thread1);
        self.get_counts()
    }

    pub fn increment_counts_thread2(&self) -> Result<(u64, u64, u64), String> {
        let mut count = self
            .count
            .write()
            .map_err(|e| format!("Failed to write-lock count: {}", e))?;
        let mut count_thread2 = self
            .count_thread2
            .write()
            .map_err(|e| format!("Failed to write-lock count_thread2: {}", e))?;
        let mut write_lock = self
            .write_lock
            .write()
            .map_err(|e| format!("Failed to write-lock write_lock: {}", e))?;
        *count += 1;
        *count_thread2 += 1;
        drop(count);
        drop(count_thread2);
        *write_lock = 2;
        self.get_counts()
    }
}

Breaking Down the Code

Counters for Each Thread: We've got three counters here - count, count_thread1, and count_thread2. The first one is shared between both threads, while the other two are individual to each thread. Each counter is wrapped in a Mutex. Why? Because Mutex ensures that only one thread can mess with the data at a time.
The write_lock Mutex: This little guy is the key to making sure our print status function doesn't get interrupted by the other thread. We're using it to lock down the entire increment operation, from start to finish, including the print part. And to keep the compiler happy (and avoid warnings about unused variables), we're assigning a value (1 or 2) to write_lock.
Lock Order Consistency: Notice something about how we lock our counters? We always lock count first, then count_thread1 or count_thread2. This is crucial. Locking in the same order every time is a simple yet effective way to dodge deadlocks. If you start locking in different orders in different parts of your code, you're setting up a classic deadlock scenario.

Choosing RwLock Over Mutex: A Consideration of Context

In the Rust ecosystem, we often use Mutex for safe, exclusive access to data across multiple threads. Think of Mutex as a way to say, "One at a time, please," ensuring that only one thread can access the data at any given moment. It's a straightforward, foolproof approach to concurrency.

But then there's RwLock, a slightly more complex cousin of Mutex.

The Subtlety of RwLock

Reading and Writing: RwLock allows multiple threads to read data at the same time, which can be a big win for performance if you have a lot of read operations. However, it still ensures that write operations get exclusive access.
In Our Case: While our current example might not have simultaneous reads, we chose RwLock to illustrate how it could be beneficial in a broader application context. It's about understanding the tools at your disposal and choosing the right one for the right job.

The Practical Angle

Why Not Just Stick with Mutex?: You might wonder why we didn't just use Mutex since our example doesn’t explicitly require the concurrent read capabilities of RwLock. The reason is twofold: firstly, to demonstrate the capabilities of RwLock for educational purposes, and secondly, to prepare the code for potential scalability where concurrent reads might become more relevant.
Big Picture: In a larger application, where you might have numerous threads frequently reading data, the benefits of RwLock become more pronounced. By allowing concurrent reads, RwLock can significantly enhance performance, reducing the waiting time for threads that just want to read data.

Choosing RwLock in our example is a nod to these broader considerations. It’s about anticipating future needs and understanding how different concurrency tools can be leveraged in various scenarios. This is a key part of thinking like a Rustacean — not just solving the problem at hand but doing so in a way that's efficient, scalable, and idiomatic to Rust.

Explaining the `main` Function and the Role of `Arc`

In our Rust application, the main function is where we see the concurrency in action. Let's dissect how it uses Arc to enable multiple threads to interact with the same instance of our Service struct.

The `main` Function Code

Here's what our main function looks like:

mod service;

use service::Service;
use std::sync::Arc;
use std::thread;
use std::time::Duration;

fn main() {
    let service = Arc::new(Service::new());

    let service1 = Arc::clone(&service);
    let service2 = Arc::clone(&service);

    let thread1 = thread::spawn(move || {
        for i in 0..5 {
            match service1.increment_counts_thread1() {
                Ok(counts) => println!("Thread 1: Iteration {}: Counts = {:?}", i, counts),
                Err(e) => {
                    eprintln!("Thread 1: Iteration {}: Error = {}", i, e);
                    // Handle error, e.g., retry, log, or break
                    break;
                }
            }
            thread::sleep(Duration::from_millis(250));
        }
    });

    let thread2 = thread::spawn(move || {
        for i in 0..5 {
            match service2.increment_counts_thread2() {
                Ok(counts) => println!("Thread 2: Iteration {}: Counts = {:?}", i, counts),
                Err(e) => {
                    eprintln!("Thread 2: Iteration {}: Error = {}", i, e);
                    // Handle error, e.g., retry, log, or break
                    break;
                }
            }
            thread::sleep(Duration::from_millis(250));
        }
    });

    if let Err(e) = thread1.join() {
        eprintln!("Thread 1 panicked: {:?}", e);
    }

    if let Err(e) = thread2.join() {
        eprintln!("Thread 2 panicked: {:?}", e);
    }
}

Understanding `Arc` in Rust

Arc: A Smart Pointer for Concurrency: Arc stands for Atomic Reference Counting. It's a type of smart pointer in Rust, which means it keeps track of how many references exist to a certain piece of data. Once all references are gone, the data is automatically cleaned up. This is super handy in avoiding memory leaks.
Why Use Arc?: In a multi-threaded context, we need a way to safely share data between threads. Arc allows multiple threads to own a reference to the same data, ensuring that the data stays alive as long as at least one thread is using it.
Thread-Safety: Arc is thread-safe, meaning it can be used across multiple threads without the risk of causing data races. This is crucial in our example where multiple threads are accessing and modifying the shared Service instance.

Threads in Action

Creating Threads: We spawn two threads, thread1 and thread2. Each thread is given a cloned reference to our Service instance (service1 and service2). This cloning is done using Arc::clone, which increments the reference count rather than copying the actual data.
Interacting with Shared Data: Inside each thread, we call either increment_counts_thread1 or increment_counts_thread2. These methods modify the shared data in a controlled manner, thanks to our RwLock implementation in the Service struct.
Expected Outcomes: As each thread performs its operations, we expect the shared counter (count) to be incremented by both threads, whereas count_thread1 and count_thread2 are exclusive to their respective threads. The threads also print out the current state of these counters after each increment.

In summary, Arc in our main function demonstrates Rust's powerful and safe approach to concurrency. By allowing multiple threads to share ownership of data, Arc enables concurrent access while ensuring that the data lives as long as it's needed and no longer. This, combined with the thread-safe operations on our Service struct, showcases a typical pattern for managing shared state in multi-threaded Rust applications.

Seeing It in Action: Output of the Program

After understanding the roles of Arc, RwLock, and our thread setup, let's see what happens when we actually run the program. Here's the output you can expect:

cargo run .
   Compiling service_example v0.1.0 (/home/inigo/Documents/Tutorials/RUST/service_example)
    Finished dev [unoptimized + debuginfo] target(s) in 0.37s
     Running `target/debug/service_example .`
Thread 1: Iteration 0: Counts = (1, 1, 0)
Thread 2: Iteration 0: Counts = (2, 1, 1)
Thread 1: Iteration 1: Counts = (3, 2, 1)
Thread 2: Iteration 1: Counts = (4, 2, 2)
Thread 1: Iteration 2: Counts = (5, 3, 2)
Thread 2: Iteration 2: Counts = (6, 3, 3)
Thread 1: Iteration 3: Counts = (7, 4, 3)
Thread 2: Iteration 3: Counts = (8, 4, 4)
Thread 1: Iteration 4: Counts = (9, 5, 4)
Thread 2: Iteration 4: Counts = (10, 5, 5)

This output clearly shows how the counters are incremented by each thread. Notice how the shared counter (count) increases with each operation, regardless of which thread is executing, while the individual counters (count_thread1 and count_thread2) are incremented only by their respective threads.

Error Management in Rust: Embracing Verbosity for Clarity

When transitioning to Rust from a language like Go, one thing you might find familiar is the verbosity in error handling. Rust, much like Go, emphasizes explicit and clear error management, though the styles differ slightly. Let's delve into how this plays out in Rust using our example, and why embracing this verbosity can be beneficial.

Rust's Explicit Error Handling

Explicit is Better than Implicit: Rust enforces an explicit approach to error handling. Unlike languages that use exceptions, Rust requires you to acknowledge and handle potential errors at every step.
Avoiding .unwrap() in Production: While .unwrap() is convenient for quick tests or examples, it's risky in production code because it causes the program to panic in case of an error. Rust encourages handling errors gracefully to avoid unexpected crashes.

Verbosity in Rust vs. Go

Clear and Predictable Code: Rust’s verbose error handling, akin to Go's, ensures that your code is clear about how it deals with various failure scenarios. This explicitness leads to more predictable and maintainable code.
Our Code's Error Handling Approach:

match service1.increment_counts_thread1() {
    Ok(counts) => println!("Thread 1: Iteration {}: Counts = {:?}", i, counts),
    Err(e) => {
        eprintln!("Thread 1: Iteration {}: Error = {}", i, e);
        break;
    }
}

The Use of match: We use match to handle the Result type returned by our functions. This way, we explicitly define the flow for both successful and error outcomes.

Balancing Verbosity and Practicality

Context Matters: There are times when using .unwrap() might be okay, such as in prototype code or when an error is impossible by logic. However, as a general rule, explicit error handling is preferred.
Familiarity Over Time: If you're used to Go's error handling, Rust's approach will feel familiar, though it may take some time to adapt to the nuances. Eventually, you'll likely appreciate the robustness it brings to your programs.

In summary, Rust's approach to error handling, while verbose, shares similarities with Go's explicit style. This verbosity is a small price to pay for the clarity and safety it brings to your code. As you grow more accustomed to Rust's patterns, you'll find that this explicitness becomes an invaluable tool in writing reliable, bug-resistant applications.

Conclusions: Navigating the Learning Path in Rust

As we come to the end of this post, it's clear that diving into Rust's world, especially its approach to concurrency, is both challenging and rewarding. I'm by no means an expert in Rust — just someone who's starting small and grappling with concepts that initially seemed daunting.

Embracing Rust's Concurrency Model

Starting Small: My journey in Rust began with tackling small, challenging concepts. This exploration into Arc, RwLock, and threading is a part of that journey, an attempt to understand the depths of Rust's concurrency.
Safety First: One of the most significant learnings from this exercise is the importance Rust places on safety, particularly in concurrent environments. The language's design nudges you towards patterns that prevent common errors like data races.

The Beginner's Experience

From Confusion to Clarity: As a beginner, the complexity of ownership, borrowing, and concurrency in Rust can be quite overwhelming. But, as with any new skill, the confusion gradually gives way to clarity with practice and patience.
Community Support: One thing that stands out in my learning process is the Rust community's support. Whether it's through forums, documentation, or open-source contributions, there's always help available.

A Small Step in a Larger Journey

Continual Learning: This example, which I found challenging initially, represents just a small step in the broader journey of mastering Rust. It’s a testament to starting with what seems tough and breaking it down into manageable pieces.
Encouragement for Fellow Learners: To those who are also on their beginning stages with Rust, keep at it. The initial hurdles are part of the process, and every challenge overcome is a stride towards becoming a proficient Rustacean.

Explore the Code

If you're curious to see how the theory translates into code, or if you want to try your hand at modifying and playing with it, check out the repository on GitHub. It's a space for learning, experimenting, and sharing insights.

GitHub Repository Link

Wrapping Up

In summary, my journey with Rust is still in its early stages, filled with learning and discovery. This example is a reflection of that — a small piece of a much larger puzzle. As I continue to learn, I look forward to uncovering more such pieces, understanding them, and fitting them together in the beautiful tapestry that is Rust programming.

Golang Error Handling: Introducing Fold Ninja to Reduce Verbosity

Iñigo Etxaniz — Tue, 14 Nov 2023 09:03:55 +0000

Introduction to Go Error Handling

As a Go developer, one aspect of the language that I deeply appreciate is its approach to error handling. It's designed to be reliable, thorough, and encourages good programming practices. Here are some key points I particularly like about Go's error handling:

Immediate Error Handling: In Go, errors are handled as soon as they occur. This immediacy ensures that issues are addressed promptly, enhancing the stability and reliability of applications.
Comprehensive Error Consideration: When coding in Go, all potential errors are taken into account. This thoroughness ensures that we're always aware and prepared for possible failure points in our code.
Reduced Unexpected Errors: With the correct programming approach, unexpected errors (or panics) are almost a rarity in Go. This is a testament to the language's robustness and our ability to control error flows.
Ease in Tracing Errors: Finding the source of a problem in Go is generally straightforward, especially if error descriptions are clear and informative. Often, a simple search in VSCode with the text of the error leads you right to the issue.
Tools for Error Analysis: Go also provides powerful tools to understand the sequence of events leading to an error. For instance, here's a snippet showing how you can use Go's runtime package to capture a stack trace just after an error occurs:

  buffer := make([]byte, 96000)
  n := runtime.Stack(buffer, false)
  stackItems := strings.Split(string(buffer[:n]), "\n")
  stackData := make([]string, 0)
  // Further processing of stackData...

This snippet captures the current goroutine's stack trace and splits it into readable segments, allowing for a detailed analysis of the error's context.

In summary, Go's error handling is not just a feature; it's a cornerstone of the language's design philosophy, promoting robust and reliable code. However, despite these advantages, there's one aspect that can be challenging: verbosity.

The Challenge of Verbosity in Go Error Handling

While Go's error handling is effective, it introduces a challenge that I've personally grappled with: verbosity. This becomes particularly evident during the code review process. In my programming practice, I adhere to a principle where the code should largely explain itself. This is achieved through well-thought-out function and variable names, along with breaking down logic into smaller, descriptively named functions. The aim is always to reduce reliance on comments and make the code as self-explanatory as possible.

However, the verbosity inherent in Go's error handling often poses a hurdle to this approach. Let's consider a common scenario: you're reviewing code and trying to grasp the core algorithm. Each function call accompanied by error handling adds layers of additional lines. These lines, while crucial for robustness, can distract from the main logic flow. The result? It becomes more challenging to quickly understand the essence of what the code is doing.

Here's a typical example illustrating this point:

package main

import (
    "bytes"
    "compress/gzip"
    "encoding/base64"
    "fmt"
    "io"
)

// gunzipB64ToString takes a base64 encoded string that represents gzip-compressed data,
// decodes it, unzips it, and returns the resulting string.
func gunzipB64ToString(b64Data string) (string, error) {
    decodedBytes, err := base64.StdEncoding.DecodeString(b64Data)
    if err != nil {
        return "", fmt.Errorf("error decoding base64 data: %w", err)
    }

    buffer := bytes.NewBuffer(decodedBytes)
    gzipReader, err := gzip.NewReader(buffer)
    if err != nil {
        return "", fmt.Errorf("error creating new gzip reader: %w", err)
    }
    defer gzipReader.Close()

    unzippedBytes, err := io.ReadAll(gzipReader)
    if err != nil {
        return "", fmt.Errorf("error reading unzipped data: %w", err)
    }

    return string(unzippedBytes), nil
}

func main() {
    encodedString := "H4sIAAAAAAAA/8tIzcnJBwCGphA2BQAAAA=="
    result, err := gunzipB64ToString(encodedString)
    if err != nil {
        fmt.Println("Error:", err)
        return
    }
    fmt.Println("Decompressed and decoded string:", result)
}

In this snippet, the core operations – decoding a base64 string and unzipping it – are fairly straightforward. Yet, the error handling associated with each step nearly doubles the number of lines in the function. This added verbosity, while crucial for ensuring the robustness and reliability of the operation, can overshadow the main functionality, especially when viewed within the broader context of a larger application.

This isn't just a matter of aesthetics; it has practical implications for code readability and maintenance. When reviewing code, the clarity of the core logic can be clouded by the repetitive nature of error checks. This can make the review process more time-consuming and cumbersome, and sometimes, important aspects of the logic might be overlooked due to the clutter.

As a developer who values clean, self-explanatory code, I found this aspect of Go programming particularly challenging. It was important to me to find a solution that would align with my coding philosophy of minimalistic, clear code, while still respecting Go's rigorous approach to error handling.

Now, let's see how the same function looks when folded using "Fold Ninja":

package main

import (
    "bytes"
    "compress/gzip"
    "encoding/base64"
    "fmt"
    "io"
)

// gunzipB64ToString takes a base64 encoded string that represents gzip-compressed data,
func gunzipB64ToString(b64Data string) (string, error) {
    decodedBytes, err := base64.StdEncoding.DecodeString(b64Data)

    buffer := bytes.NewBuffer(decodedBytes)
    gzipReader, err := gzip.NewReader(buffer)
    defer gzipReader.Close()

    unzippedBytes, err := io.ReadAll(gzipReader)

    return string(unzippedBytes), nil
}

func main() {
    encodedString := "H4sIAAAAAAAA/8tIzcnJBwCGphA2BQAAAA=="
    result, err := gunzipB64ToString(encodedString)
    fmt.Println("Decompressed and decoded string:", result)
}

With "Fold Ninja," the error handling blocks are neatly folded away. This leaves us with a view that emphasizes the core logic of the function. The error handling is still there, just one click away if we need to inspect it, but it no longer dominates the visual landscape of the code. This folding makes it much easier to focus on and understand the primary operations of the function, especially during code reviews or when navigating through a large codebase.

The difference is clear – what was once a multi-line, error-handling-heavy function is now a concise, easily digestible piece of code. This not only enhances the readability but also aligns with the philosophy of writing self-explanatory code that focuses on the core logic.

Development of Fold-Ninja

After recognizing the challenges posed by verbosity in Go error handling, I developed "Fold Ninja," a Visual Studio Code extension tailored to enhance code readability and streamline the review process. The primary goal of Fold Ninja is to reduce the visual noise created by excessive information, particularly verbose error management sections in Go files.

Fold Ninja operates in three modes, each designed to cater to different needs during code analysis and review:

Inactive: In this mode, Fold Ninja is dormant, showing the code in its original, unaltered form.
Compact: This is where the extension truly shines. Compact mode automatically folds away verbose parts of the code, including comments and error management blocks. This mode significantly declutters the code, presenting a focused view that highlights the main logic.
Expanded: In this mode, all previously compacted sections are unfolded, making every part of the code, including comments and error handling routines, visible again.

Switching between these modes is a simple click on the status bar item, making it user-friendly. This flexibility allows developers to quickly shift views based on their current task – whether they're deep diving into a detailed analysis or doing a high-level code review.

When working in compact mode, extension is only activated when user activates a file. So, when coding the extension will not update the view.

Try Fold-Ninja and Contribute

I invite you to try Fold Ninja and experience firsthand how it can enhance your Go coding and review process. You can find the extension on the Visual Studio Marketplace and explore its source code on GitHub. Your feedback and contributions are highly valued. If you have suggestions or encounter any issues, please feel free to open an issue on the GitHub repository.

Conclusion

In summary, Fold Ninja embodies my philosophy of clean, self-explanatory code. It offers a practical solution to the verbosity challenge in Go, allowing developers to focus on what truly matters in their code. As we continue to evolve and improve our tools, I believe that we can make Go development an even more enjoyable and efficient experience. Thank you for reading, and I look forward to your thoughts and contributions to Fold Ninja.