Part 02: Building a Sovereign Software Factory: The Local Root CA & Trust Chains

GitHub: https://github.com/InfiniteConsult/0006_cicd_part02_certificate_authority

TL;DR: In this installment, we solve the "Trust Gap" inherent in self-hosted infrastructure. We reject the insecurity of HTTP and the limitations of public CAs (Let's Encrypt) by building our own Local Certificate Authority. We use openssl to mint a Root CA, automate the issuance of SAN-compliant certificates for every service, and manually inject trust into the fragmented keystores of our Host OS, Browsers, and Java applications to eliminate SSL handshake failures.

The Sovereign Software Factory Series:

1. Part 01: Building a Sovereign Software Factory: Docker Networking & Persistence
2. Part 02: Building a Sovereign Software Factory: The Local Root CA & Trust Chains (You are here)
3. Part 03: Building a Sovereign Software Factory: Self-Hosted GitLab & Secrets Management
4. Part 04: Building a Sovereign Software Factory: Jenkins Configuration as Code (JCasC)
5. Part 05: Building a Sovereign Software Factory: Artifactory & The "Strict TLS" Trap
6. Part 06: Building a Sovereign Software Factory: SonarQube Quality Gates
7. Part 07: Building a Sovereign Software Factory: ChatOps with Mattermost
8. Part 08: Building a Sovereign Software Factory: Observability with the ELK Stack
9. Part 09: Building a Sovereign Software Factory: Monitoring with Prometheus & Grafana
10. Part 10: Building a Sovereign Software Factory: The Python API Package (Capstone)

Chapter 1: The "HTTPS Everywhere" Mandate

1.1 The Problem: The "Not Secure" Pain Point

We have successfully built our foundational "city". We have a "Control Center" (DooD), "roads" (cicd-net), and "foundations" (our hybrid persistence model).

But we have a new, critical "pain point." Our services can communicate, but they do so over unencrypted http. This is insecure, unprofessional, and does not emulate a real-world environment.

Pedagogical Failure (Browsers)

If we were to deploy GitLab right now on http://gitlab.cicd.local, our browsers would immediately betray the problem. We would be faced with a "Not Secure" warning in the address bar. This breaks user trust and, for many modern web features, can block functionality entirely.

If we tried to use a "magic" self-signed certificate (which we will deconstruct later), the problem gets worse. The browser would present a full-page, "Your connection is not private" interstitial error (like NET::ERR_CERT_AUTHORITY_INVALID or ERR_INSECURE_RESPONSE). This is a hard stop for most users.

Pedagogical Failure (Tools)

This "pain point" is not just cosmetic. It's a functional "stack killer" for our specific set of tools.

• curl: Any script using curl to access an HTTPS service with a bad certificate will fail with a hard error: curl: (60) SSL certificate problem: unable to get local issuer certificate. The only way around this is to use the --insecure flag, which is a terrible security practice.
• The "Java Gotcha": This is the most critical failure for our stack. Jenkins, Artifactory, and SonarQube are all Java applications. When our Jenkins container tries to make an API call or receive a webhook from our https://gitlab.cicd.local server, the Java Virtual Machine (JVM) will reject the untrusted certificate and throw a fatal javax.net.ssl.SSLHandshakeException.

This single Java exception will bring our entire automated pipeline to a grinding halt.

The Goal

Our goal is not just to "get rid of the warning." It is to build a professional, secure, and trusted internal ecosystem. To do this, we must become our own "passport office."

Chapter 2: A "First Principles" Guide to PKI and Trust

2.1 The Solution: Public Key Infrastructure (PKI)

To solve our "Not Secure" problem, we must adopt the same solution the entire internet uses: HTTPS.

HTTPS is not a single thing; it's a combination of two: http (the protocol we already know) layered on top of SSL/TLS (Secure Sockets Layer/Transport Layer Security). This secure layer provides two fundamental guarantees:

1. Encryption: It creates a secure, private "tunnel" between the client and the server. Any data sent through this tunnel (passwords, API keys, source code) is scrambled and unreadable to eavesdroppers.
2. Identity: This is the more complex and crucial part. How does your browser know it's talking to the real gitlab and not an imposter? The server must present a form of identification.

This system of identification is called Public Key Infrastructure (PKI).

The Analogy: "The Passport Office"

PKI works exactly like the international passport system.

1. A Server Certificate is a "Passport": It's a digital file that proves a server's identity (e.g., "I am gitlab").
2. A Certificate Authority (CA) is a "Passport Office": It's a trusted entity (like a government) that issues and signs the "passport," attesting to its validity.
3. A Trust Store is a "List of Trusted Governments": Your browser and operating system come with a pre-installed "trust store" that contains a list of all the major, legitimate "passport offices" (like Let's Encrypt, DigiCert, etc.) in the world.

When your browser connects to a server, it looks at the "passport" (the certificate) and checks the "signature" on it. If it was signed by a "passport office" (CA) in its trusted list, it grants access. If not, it blocks the connection.

2.2 Deconstructing Our Options

We need a "passport" (a certificate) for our services. We have three options for getting one.

Option 1: A Public CA (e.g., Let's Encrypt)

This is the standard for all public websites. Let's Encrypt is a globally trusted "passport office." Your browser and OS already have it in their trust store.

Why we can't use it: Let's Encrypt must validate that you own a domain before issuing a certificate for it. It does this by connecting to your domain over the public internet. This process (Domain Validation or DV) is impossible for our local stack. The CA's servers cannot reach localhost or our container's hostname (gitlab). We cannot prove we "own" these private names, so the public CA will refuse to issue a certificate.

Option 2: A Simple Self-Signed Certificate

This is the "quick and dirty" method. A self-signed certificate is one that is signed by itself.

Why it fails: This is the equivalent of a "hand-drawn passport". You are telling the browser, "I am gitlab, and I vouch for myself." The browser, acting as the security guard, has no reason to trust you. It checks its list of "trusted passport offices" (the trust store) and finds no match. It correctly throws a full-page security warning (NET::ERR_CERT_AUTHORITY_INVALID) to protect you from a potential man-in-the-middle attack. This is the very SSLHandshakeException that will break our Java-based tools.

Option 3: Our Private/Local CA (The Solution)

This is the professional solution that balances our two problems. We will build our own "passport office".

1. We will become our own Certificate Authority (CA) by creating a Root CA certificate. This is our "passport office's business license."
2. We will then use our new CA to issue "passports" (service certificates) for gitlab, jenkins, and all our other services.
3. Finally, we will solve the "trust" problem by manually telling our local computers and containers: "This new 'passport office' is an official, trusted authority."

From that point on, any passport (certificate) issued by our office will be automatically trusted by our local tools, browsers, and services.

Chapter 3: The "Hard Way" - Building Our CA with openssl

3.1 Why We Use openssl (The "First Principles")

We will begin by creating our Certificate Authority using the openssl command-line tool directly. This is the "hard way."

We are doing this for a specific pedagogical reason. We could use a simpler tool like easyrsa, but easyrsa is just a shell script wrapper that hides all the complexity. It automates all the tedious, "first principles" steps that a real CA must perform.

By using openssl directly, we are forced to deconstruct those steps. We will have to manually:

1. Build the CA's "Blueprint": We will create an openssl.cnf configuration file that defines the rules and policies of our "passport office."
2. Create the "CA Database": We will manually create the index.txt "master ledger" and the serial "ticket number dispenser" that openssl needs to track the certificates it issues.
3. Manage the "Keys": We will build the secure directory structure to protect our CA's "master stamp" (its private key).

By doing this "hard way" first, you will understand what is actually happening under the hood. Only then will we introduce the "easy way" (easyrsa) as the automation tool it is.

We will build our entire PKI (Public Key Infrastructure) inside the ~/cicd_stack/ca directory we created on our host machine.

3.2 The "CA Blueprint": openssl.cnf

Our first step is to create the "blueprint" for our new "passport office." This is a configuration file, openssl.cnf, that tells openssl all of its rules. It defines the file locations, the default policies, and, most importantly, the extensions that make our root certificate a Certificate Authority.

Create this file on your host machine at ~/cicd_stack/ca/openssl.cnf.

[ ca ]
default_ca = CA_default

[ CA_default ]
# --- Directory and file locations ---
dir               = ./pki
certs             = $dir/certs
crl_dir           = $dir/crl
new_certs_dir     = $dir/newcerts
database          = $dir/index.txt
serial            = $dir/serial
RANDFILE          = $dir/private/.rand

# --- CA Key and Certificate ---
private_key       = $dir/private/ca.key
certificate       = $dir/certs/ca.pem

# --- Policy & Extensions ---
default_md        = sha256
default_days      = 3650  # 10 years
policy            = policy_strict
x509_extensions   = v3_ca

[ policy_strict ]
# Rules for signing: which fields must match the CA
countryName             = match
stateOrProvinceName     = match
organizationName        = match
organizationalUnitName  = optional
commonName              = supplied
emailAddress            = optional

[ req ]
# This section is used by the 'openssl req' command
default_bits        = 4096
distinguished_name  = req_distinguished_name
string_mask         = utf8only
default_md          = sha256
# Tell 'openssl req -x509' to use our 'v3_ca' extensions
x509_extensions     = v3_ca

[ req_distinguished_name ]
# These are the DN fields it will prompt you for
countryName                     = Country Name (2 letter code)
stateOrProvinceName             = State or Province Name
localityName                    = Locality Name (eg, city)
organizationName                = Organization Name
organizationalUnitName          = Organizational Unit Name (eg, section)
commonName                      = Common Name (e.g. server FQDN or YOUR name)
emailAddress                    = Email Address

[ v3_ca ]
# --- Extensions for the Root CA certificate itself ---
subjectKeyIdentifier = hash
authorityKeyIdentifier = keyid:always,issuer
basicConstraints = critical, CA:TRUE, pathlen:0
keyUsage = critical, digitalSignature, cRLSign, keyCertSign

Deconstruction

• [ CA_default ]: This block defines our working directory and file structure. We are telling openssl to look for its database in the pki directory (dir = ./pki), to find its "ledger" at pki/index.txt, and its "ticket dispenser" at pki/serial.
• [ policy_strict ]: This defines the rules for signing. We are telling our CA that it's allowed to sign certificates for other organizations (as long as organizationName matches) and they must supply their own commonName.
• [ v3_ca ]: This is the most important section. It defines the powers of our Root CA.
  • basicConstraints = critical, CA:TRUE: This is the master switch. It tells the world this certificate is a Certificate Authority (a "passport office") and has the power to sign other certificates.
  • keyUsage = ... keyCertSign, cRLSign: This specifies what the key can be used for. keyCertSign is the specific permission that allows it to "sign other certificates." cRLSign allows it to sign a "revocation list" (a list of "stolen passports").

3.3 The "CA Database"

Now that we have our "blueprint" (openssl.cnf), we must create the physical "office" and "filing system" that openssl will use. This is the "database" that our CA will use to track every "passport" it ever issues.

This is the second "first principles" component that wrappers like easyrsa automate and hide from you. Our openssl.cnf file told openssl to look for these files in the ./pki directory; now, we must create them.

• pki/private/: This will be our "vault." It will hold our CA's highly-sensitive private key.
• pki/certs/: This will be our "public filing cabinet" for public-facing certificates, like our main CA certificate.
• pki/newcerts/: This will be the "output tray" where openssl places all the new service certificates it signs.
• pki/index.txt: This is the CA's "master ledger." It's a simple text file that openssl uses to keep a database of every certificate it has signed, noting whether it's "valid," "revoked," or "expired."
• pki/serial: This is the "ticket number dispenser." It's a file that contains the next unique serial number (in hexadecimal) to be issued. openssl will read this file, assign the number, and then increment the file, ensuring no two certificates ever get the same serial number.

Action Plan: Creating the CA Directory

We will now create this entire structure using our first script, 01-create-ca.sh. This script will also generate our private key and our root certificate, completing our CA setup in one go.

Here is the code for 01-create-ca.sh.

#!/usr/bin/env bash

# This script creates the CA directory structure, database,
# and generates the Root CA private key and certificate.

cd ~/cicd_stack/ca || (echo "no CICD directory present" && exit)

if [ ! -f "openssl.cnf" ]; then
echo "openssl.cnf not found in ~/cicd_stack/ca. Please create it before proceeding"
exit
fi

# --- Configuration ---
# The password for your new Root CA private key.
# IMPORTANT: Change this to a strong, secure password.
CA_PASSWORD="your_secure_password"

# The Distinguished Name (DN) for your Root CA.
# Customize these values for your organization.
COUNTRY="ZA"
STATE="Gauteng"
LOCALITY="Johannesburg"
ORG_NAME="Local CICD Stack"
CA_CN="Local CICD Root CA"

# --- Setup ---
echo "--- Creating PKI directory structure ---"
mkdir -p pki/{private,certs,newcerts,crl}

# Set correct permissions for the "vault"
chmod 700 pki/private

echo "--- Creating CA database ---"
touch pki/index.txt
echo "1000" > pki/serial

# --- Generate Root CA ---
echo "--- 1. Generating Root CA Private Key (ca.key) ---"
# Use -passout to provide the password non-interactively
openssl genrsa -aes256 -passout pass:$CA_PASSWORD \
  -out pki/private/ca.key 4096

# Secure the key
chmod 400 pki/private/ca.key

echo "--- 2. Generating Root CA Certificate (ca.pem) ---"
# Use the '-subj' flag to pass DN info non-interactively
openssl req -config openssl.cnf \
      -key pki/private/ca.key \
      -passin pass:$CA_PASSWORD \
      -new -x509 -days 3650 -sha256 -extensions v3_ca \
      -subj "/C=$COUNTRY/ST=$STATE/L=$LOCALITY/O=$ORG_NAME/CN=$CA_CN" \
      -out pki/certs/ca.pem

echo "--- CA creation complete ---"
echo "CA Private Key: pki/private/ca.key"
echo "CA Certificate: pki/certs/ca.pem"

Deconstruction

Let's deconstruct the "Action" part of this script:

• mkdir -p ... & chmod 700 ...: We create our directory structure and immediately secure the "vault" (pki/private). We give it 700 (rwx------) permissions, which means only the owner (our user) can read, write, or enter it. This is the fix for the Permission denied error we discovered during our testing.
• touch pki/index.txt & echo "1000" > pki/serial: We create the "ledger" and "ticket dispenser" files.
• openssl genrsa -aes256 ...: This is Step 1: Create the CA Private Key (the "master stamp").
  • -aes256: We encrypt the key with a strong cipher.
  • -passout pass:$CA_PASSWORD: We provide the password non-interactively from our variable.
  • -out pki/private/ca.key: We save the key directly into our secure "vault."
• chmod 400 pki/private/ca.key: We immediately set the key file itself to be read-only (r--------) for the owner, protecting it from accidental modification.
• openssl req -x509 ...: This is Step 2: Create the CA Certificate (the "business license").
  • -key pki/private/ca.key & -passin ...: We specify our new "master stamp" and provide its password.
  • -x509: This is a critical flag. It tells openssl to not create a "signing request" (a CSR) but to create a self-signed public certificate. This is what makes it a "Root" CA.
  • -extensions v3_ca: This is the most important part. It tells openssl to look at our openssl.cnf file and apply the extensions from the [ v3_ca ] section. This is what flips the switch and embeds the CA:TRUE "power" into our certificate.
  • -subj "...": This provides all the "passport office" details (Country, Organization, etc.) non-interactively.

Action

Place the openssl.cnf file and the 01-create-ca.sh script in your ~/cicd_stack/ca directory on your host machine.

# (Run on HOST, inside ~/cicd_stack/ca)

# 1. Make the script executable
chmod +x 01-create-ca.sh

# 2. Run the script
./01-create-ca.sh

Result:
The script will run non-interactively. When it finishes, you will have a fully functional Certificate Authority.

--- Creating PKI directory structure ---
--- Creating CA database ---
--- 1. Generating Root CA Private Key (ca.key) ---
--- 2. Generating Root CA Certificate (ca.pem) ---
--- CA creation complete ---
CA Private Key: pki/private/ca.key
CA Certificate: pki/certs/ca.pem

Chapter 4: Issuing Service Certificates (The SAN "Gotcha")

4.1 The Goal: Creating Reusable "Passports"

We have successfully built our "passport office" (our Root CA). We have a secure "vault" for our ca.key and a public "business license" (ca.pem).

Now, we must start issuing "passports" (service certificates) for each of our 10 services. We could do this one by one, but that would be a tedious and error-prone manual process. Instead, we will immediately automate this.

We will create a single, reusable script named 02-issue-service-cert.sh. This script will be our "passport printer." We will be able to run ./02-issue-service-cert.sh gitlab or ./02-issue-service-cert.sh jenkins.cicd.local, and it will automatically:

1. Create a secure, isolated directory for the service.
2. Generate a new private key for that service.
3. Generate a Certificate Signing Request (CSR) with the correct "first principles" configuration.
4. Use our Root CA to sign the request and issue a valid, trusted "passport."

4.2 Action Plan: 02-issue-service-cert.sh

Here is the code for our "passport printer" script. Create this file on your host machine at ~/cicd_stack/ca/02-issue-service-cert.sh.

#!/usr/bin/env bash

# This script issues a new, signed certificate for a service.
#
# USAGE: ./02-issue-service-cert.sh <service_name_or_path>
#
# EXAMPLE: ./02-issue-service-cert.sh gitlab
# EXAMPLE: ./02-issue-service-cert.sh elk/elasticsearch
#
# This will create a new directory 'pki/services/gitlab'
# containing 'gitlab.key.pem' and 'gitlab.crt.pem'.

cd ~/cicd_stack/ca || (echo "no CICD directory present" && exit)

# --- Configuration ---
# This MUST match the password you used in '01-create-ca.sh'
CA_PASSWORD="your_secure_password"

# --- Script ---
SERVICE_NAME=$1
CERT_NAME=$(basename "$SERVICE_NAME")

if [ -z "$SERVICE_NAME" ]; then
  echo "ERROR: No service name provided."
  echo "USAGE: ./02-issue-service-cert.sh <service_name_or_path>"
  exit 1
fi

echo "--- Preparing environment for service: $SERVICE_NAME ---"
echo "--- Certificate simple name will be: $CERT_NAME ---"

# 1. Define all file paths
SERVICE_DIR="pki/services/$SERVICE_NAME"
KEY_FILE="$SERVICE_DIR/$CERT_NAME.key.pem"
CSR_FILE="$SERVICE_DIR/$CERT_NAME.csr"
CERT_FILE="$SERVICE_DIR/$CERT_NAME.crt.pem"
EXT_FILE="$SERVICE_DIR/v3.ext"

# 2. Create the service's private directory
mkdir -p $SERVICE_DIR
chmod 700 $SERVICE_DIR

# 3. Generate the service's Private Key
echo "--- 1. Generating Private Key ($CERT_NAME.key.pem) ---"
openssl genrsa -out $KEY_FILE 4096
chmod 400 $KEY_FILE

# 4. Create the SAN (Subject Alternative Name) config file
# This is critical for modern browser compatibility.
echo "--- 2. Creating SAN config file (v3.ext) ---"
cat > $EXT_FILE <<- EOM
[ v3_ext ]
subjectAltName = @alt_names
[alt_names]
DNS.1 = $CERT_NAME
DNS.2 = localhost
IP.1 = 127.0.0.1
EOM

# 5. Generate a Certificate Signing Request (CSR)
echo "--- 3. Generating Certificate Signing Request ($CERT_NAME.csr) ---"
openssl req -new -sha256 \
      -key $KEY_FILE \
      -subj "/C=ZA/ST=Gauteng/L=Johannesburg/O=Local CICD Stack/CN=$CERT_NAME" \
      -out $CSR_FILE

# 6. Sign the CSR with our Root CA
echo "--- 4. Signing the certificate with our Root CA ---"
openssl ca -config openssl.cnf \
      -extensions v3_ext -extfile $EXT_FILE \
      -days 365 -notext -md sha256 \
      -passin pass:$CA_PASSWORD \
      -in $CSR_FILE \
      -out $CERT_FILE \
      -batch # Use non-interactive "batch" mode

# 7. Clean up temporary files
rm $CSR_FILE
rm $EXT_FILE

echo "--- Successfully issued certificate for $SERVICE_NAME ---"
echo "Certificate: $CERT_FILE"
echo "Verifying SANs..."

# 8. Verify the certificate's SANs
openssl x509 -in $CERT_FILE -noout -text | grep -A1 "Subject Alternative Name"

4.3 Deconstruction (The SAN "Gotcha")

This script is the heart of our PKI, and it contains two critical, "first principles" lessons that we discovered during our testing.

Lesson 1: The "SAN Gotcha" (Why CN is Deprecated)

The most important part of this script is Step 4. We don't just generate a CSR; we first create a temporary config file called v3.ext:

# (Inside 02-issue-service-cert.sh)
cat > $EXT_FILE <<- EOM
[ v3_ext ]
subjectAltName = @alt_names
[alt_names]
DNS.1 = $CERT_NAME
DNS.2 = localhost
IP.1 = 127.0.0.1
EOM

This is the Subject Alternative Name (SAN) field. In the early days of the internet, browsers only checked the Common Name (CN) (which we set in Step 5 with -subj "/CN=...").

However, this CN field was ambiguous and is now deprecated. All modern browsers and tools (including Java) ignore the CN field and only validate the SAN list.

Our script fixes this by:

1. Creating a v3.ext file that defines the SAN extension (subjectAltName = @alt_names).
2. Populating the [alt_names] list with all the names this certificate should be valid for.
3. Telling openssl ca to use these extensions with the -extensions v3_ext -extfile $EXT_FILE flags.

We include three entries for maximum compatibility:

• DNS.1 = $CERT_NAME: (e.g., DNS:gitlab) For container-to-container communication on cicd-net.
• DNS.2 = localhost: For us to access the UI from our host browser via https://localhost.
• IP.1 = 127.0.0.1: To fix the curl https://127.0.0.1 error we discovered.

Lesson 2: The "Nested Path Gotcha" (Using basename)

The second "pain point" we discovered was when we tried to issue a certificate for elk/elasticsearch.cicd.local. Our script failed because it tried to create a file named .../elk/elasticsearch.cicd.local/elk/elasticsearch.cicd.local.key.pem, which is redundant and wrong.

The solution is the basename command:

# (Inside 02-issue-service-cert.sh)
SERVICE_NAME=$1
CERT_NAME=$(basename "$SERVICE_NAME")

This command is a standard Linux utility that strips the directory path from a string.

• If $SERVICE_NAME is gitlab.cicd.local, basename returns gitlab.cicd.local.
• If $SERVICE_NAME is elk/elasticsearch.cicd.local, basename returns elasticsearch.cicd.local.

This allows us to use the full $SERVICE_NAME to create the directory path (pki/services/elk/elasticsearch.cicd.local) but use the clean $CERT_NAME for all the filenames (e.g., elasticsearch.cicd.local.key.pem).

Lesson 3: The openssl ca Command

The final command, openssl ca, is the "passport office" at work.

# (Inside 02-issue-service-cert.sh)
openssl ca -config openssl.cnf \
      -extensions v3_ext -extfile $EXT_FILE \
      -days 365 -notext -md sha256 \
      -passin pass:$CA_PASSWORD \
      -in $CSR_FILE \
      -out $CERT_FILE \
      -batch

• -config openssl.cnf: "Use our main 'blueprint'."
• -passin pass:$CA_PASSWORD: "Here is the password for the 'master stamp' (the CA key)."
• -in $CSR_FILE: "Here is the 'passport application' (the CSR) to be signed."
• -extensions v3_ext -extfile $EXT_FILE: "Apply the SANs from our temporary file."
• -batch: "Don't ask me any questions ('Sign this? y/n'). Just approve it."
• -out $CERT_FILE: "Put the final, signed 'passport' here."

---

4.4 Action Plan: 03-issue-all-certs.sh

Now that we have our powerful, reusable "passport printer" script, we can create a simple automation script to issue "passports" for all 10 of our services.

Create this file on your host machine at ~/cicd_stack/ca/03-issue-all-certs.sh.

#!/usr/bin/env bash

# This script automates the issuance of certificates for all
# services in our 10-article CI/CD stack by looping
# over an array and calling our '02-issue-service-cert.sh' script.

# --- Configuration ---
# This array defines the service names. These names will be used
# for the certificate Common Name (CN) and the directory path
# inside 'pki/services/'.
#
# Note the nested paths for ELK, which our script handles automatically.
SERVICES=(
  "gitlab.cicd.local"
  "jenkins.cicd.local"
  "artifactory.cicd.local"
  "sonarqube.cicd.local"
  "mattermost.cicd.local"
  "elk/elasticsearch.cicd.local"
  "elk/logstash.cicd.local"
  "elk/kibana.cicd.local"
  "prometheus.cicd.local"
  "grafana.cicd.local"
)

ISSUER_SCRIPT="./02-issue-service-cert.sh"

# --- Pre-run Checks ---
if [ ! -f "$ISSUER_SCRIPT" ]; then
    echo "ERROR: Issuer script '$ISSUER_SCRIPT' not found."
    echo "Please ensure you are in the '~/cicd_stack/ca' directory."
    exit 1
fi

if [ ! -x "$ISSUER_SCRIPT" ]; then
    echo "ERROR: Issuer script '$ISSUER_SCRIPT' is not executable."
    echo "Please run: chmod +x $ISSUER_SCRIPT"
    exit 1
fi

echo "---=== Issuing all service certificates ===---"
echo "This will use the CA password hardcoded in '$ISSUER_SCRIPT'."

# --- Main Loop ---
for service in "${SERVICES[@]}"; do
    echo ""
    echo "------------------------------------------------"
    echo "Issuing certificate for: $service"
    echo "------------------------------------------------"

    # Call the issuer script with the service name
    ./$ISSUER_SCRIPT "$service"

    if [ $? -ne 0 ]; then
        echo "ERROR: Failed to issue certificate for $service. Aborting."
        exit 1
    fi
done

echo ""
echo "------------------------------------------------"
echo "---=== All service certificates issued successfully ===---"
echo "You can view the new directory structure:"
ls -lR ~/cicd_stack/ca/pki/services/

Action

From your host machine (inside ~/cicd_stack/ca), make both scripts executable, then run the automation script.

# (Run on HOST, inside ~/cicd_stack/ca)
chmod +x 02-issue-service-cert.sh
chmod +x 03-issue-all-certs.sh

./03-issue-all-certs.sh

Verification:
The script will loop through all 10 services. The final ls -lR command will display the complete, organized directory structure you've created, with each service's key and certificate in its own isolated folder (e.g., pki/services/gitlab/, pki/services/elk/elasticsearch/, etc.).

Chapter 5: The New "Pain Point" - Establishing Trust

5.1 The "Failure First": The "Untrusted Issuer"

We have successfully built our "passport office" and printed a full set of valid "passports" for all our services. Our pki/services directory is full of certificates that are cryptographically perfect, have a valid date range, and contain the correct Subject Alternative Names (SANs).

This leads us to a new, critical "pain point."

If we were to deploy a service right now using our new gitlab.cicd.local.key.pem certificate, our browsers would still show a full-page security warning.

However, the error would be different. It would no longer be NET::ERR_CERT_COMMON_NAME_INVALID (which our SANs fixed). Instead, it would be NET::ERR_CERT_AUTHORITY_INVALID.

The Analogy: The "Untrusted Passport Office"

This new error is a crucial distinction. The browser is no longer complaining about the "passport" itself (gitlab.cicd.local.crt.pem). It's complaining about the issuer.

The browser is acting as the border agent, looking at our valid "passport" and saying:

"This passport looks perfectly filled out. The photo matches, the name is correct, and it's not expired. But I've never heard of the 'passport office' (our 'Local CICD Root CA') that issued this. For all I know, you built that office in your garage. I don't trust it."

We have created a valid certificate from an untrusted authority. Our next and final step is to establish that trust.

5.2 The "Disjointed Trust Stores" Principle

To solve this new "untrusted" pain point, we must understand a critical, "first principles" concept: there is no single "trust" button on a modern operating system.

When we ran our 04-trust-ca-host.sh script, we discovered that curl was fixed, but Chrome and Firefox were not. This is because our "government" (the host OS) is fragmented, and different applications pledge allegiance to different "lists of trusted passport offices" (Trust Stores).

To fix this, we must manually add our ca-cert.pem to several independent trust stores:

1. The OpenSSL System Trust Store: This is the main list for the operating system, located at /etc/ssl/certs/. It is used by most command-line tools like curl, wget, git, and apt. We already fixed this with sudo update-ca-certificates.
2. The Firefox NSS Trust Store: Firefox, by design, ignores the system trust store. It maintains its own private, isolated database (a file named cert9.db) inside your user's profile directory.
3. The System-wide NSS Trust Store: Google Chrome also uses the NSS (Network Security Services) database, but it looks at a different, system-wide one (often ~/.pki/nssdb/). This is why fixing the OS store didn't fix Chrome.
4. The "Java Gotcha" (cacerts): This is the most important "gotcha" for our stack. The Java Virtual Machine (JVM) ignores all system trust stores. Java has its own separate, single-file database called cacerts. When we deploy Jenkins, Artifactory, and SonarQube, we must manually import our CA into this specific file, or they will all fail with SSLHandshakeException.

Our "Action Plan" must therefore be a multi-pronged attack, targeting each of these trust stores individually.

5.3 Action Plan: Trusting Our CA on the Host

Now that we understand the problem, we will execute the solution. Our script 04-trust-ca-host.sh is a precision tool designed to inject our ca.pem file into all three "disjointed trust stores" on our host machine.

Here is the code for 04-trust-ca-host.sh.

#!/usr/bin/env bash

# This script trusts our Root CA on the host machine.
# It will prompt for your password for 'sudo' commands.
#
# It trusts the CA in THREE places:
# 1. The Host OS (for curl, apt, git, etc.)
# 2. The system-wide NSS database (for Chrome/Chromium)
# 3. The Firefox browser (which uses its own private store)

echo "--- Navigating to CA directory ---"
# Navigate to the CA directory to find the certs
cd ~/cicd_stack/ca || (echo "ERROR: ~/cicd_stack/ca directory not found." && exit)

CA_CERT_NAME="cicd-stack-ca.crt"
CA_CERT_PATH="pki/certs/ca.pem"
CA_NICKNAME="Local CICD Root CA"

if [ ! -f "$CA_CERT_PATH" ]; then
    echo "ERROR: Root CA certificate not found at $CA_CERT_PATH"
    exit 1
fi

# --- 0. Install Prerequisites (NSS Tools) ---
echo "--- Installing 'libnss3-tools' (for certutil) ---"
sudo apt-get update -qq && sudo apt-get install -y -qq libnss3-tools

# --- 1. Trust CA in Host OS (for curl, Chrome, ...) ---
echo "--- 1. Trusting CA in Host OS (OpenSSL store) ---"
sudo cp $CA_CERT_PATH /usr/local/share/ca-certificates/$CA_CERT_NAME
sudo update-ca-certificates

# --- 2. Trust CA in System-wide NSS DB (for Chrome) ---
echo "--- 2. Trusting CA in NSS DB (for Chrome) ---"

# Create the NSS database directory if it doesn't exist
mkdir -p $HOME/.pki/nssdb

# Add the certificate to the NSS database
# -d sql:$HOME/.pki/nssdb : Specifies the database path
# -A : Add a certificate
# -n : Nickname
# -t "C,T,C" : Set trust attributes (C=SSL Client, T=SSL Server, C=Email)
certutil -A -n "$CA_NICKNAME" -t "C,T,C" -i $CA_CERT_PATH -d sql:$HOME/.pki/nssdb

# --- 3. Trusting CA in Firefox ---
echo "--- 3. Trusting CA in Firefox (private store) ---"

# Find all Firefox profile databases (cert9.db)
FF_PROFILES_DB=$(find $HOME/.mozilla/firefox -name "cert9.db")

if [ -z "$FF_PROFILES_DB" ]; then
    echo "WARNING: Could not find Firefox cert9.db. Skipping Firefox trust."
    echo "You may need to add the CA manually via Firefox settings."
else
    for db in $FF_PROFILES_DB; do
        PROFILE_PATH=$(dirname $db)
        echo "--- Found Firefox profile, adding CA to: $PROFILE_PATH ---"

        # Add the CA to the Firefox NSS database
        certutil -A -n "$CA_NICKNAME" -t "C,T,C" -i $CA_CERT_PATH -d sql:$PROFILE_PATH
    done
fi

echo "--- Host trust setup complete ---"
echo "IMPORTANT: Please restart Google Chrome for changes to take effect."

Deconstruction

This script is a perfect example of our "first principles" approach, as it surgically targets each trust store:

• Step 0: Install libnss3-tools: We first install the certutil command, which is the "first principles" tool for managing both the Chrome and Firefox NSS databases.
• Step 1: Host OS (OpenSSL):
  • sudo cp ... /usr/local/share/ca-certificates/: This copies our "passport office license" into the system's "pending applications" directory.
  • sudo update-ca-certificates: This is the system command that processes that directory, rebuilds the master trust file (/etc/ssl/certs/ca-certificates.crt), and officially trusts our CA. This fixes curl, git, apt, and other system tools.
• Step 2: Chrome (NSS System Store):
  • mkdir -p $HOME/.pki/nssdb: We ensure the "filing cabinet" for Chrome's trust store exists.
  • certutil -A ... -d sql:$HOME/.pki/nssdb: We use certutil to add our CA to this specific database, fixing the net::ERR_CERT_AUTHORITY_INVALID error in Chrome.
• Step 3: Firefox (NSS Profile Store):
  • find $HOME/.mozilla/firefox -name "cert9.db": We find the separate, private "filing cabinet" that Firefox maintains for your user profile.
  • certutil -A ... -d sql:$PROFILE_PATH: We run the same certutil command, but this time we target the Firefox-specific database path, fixing trust for Firefox.

Action

From your host machine (inside your 0006_cicd_part02_certificate_authority directory), make the script executable and run it.

# (Run on HOST)
chmod +x 04-trust-ca-host.sh
./04-trust-ca-host.sh

Result:
The script will prompt for your sudo password, then proceed to install the tools and update all three trust stores. You will see output like "1 added, 0 removed; done." from update-ca-certificates and confirmation from certutil.

5.4 Action Plan: Trusting the "Control Center" (dev-container)

We have successfully taught our host machine to trust our CA. Now we must do the same for our dev-container.

This is a critical step. Our dev-container is our "Control Center." Its tools (curl, git, and most importantly, our future Python API scripts) must be able to make secure HTTPS connections to our new services. If our container doesn't trust our CA, our entire automation plan will fail with the same SSLHandshakeException and certificate problem errors.

The "No-Rebuild" Solution

We will use the flexible architecture we built in Article 1 to solve this without rebuilding our container image.

1. The data/ Mount: Our dev-container.sh script mounts our host's data/ directory into the container.
2. The entrypoint.sh Mount: Our dev-container.sh script also mounts our host's entrypoint.sh script, which runs every time the container starts.

We will copy our public CA certificate into the host's data/ directory. Then, we will edit our entrypoint.sh script to automatically install this certificate at startup.

Action 1: Copy the CA Certificate

From your host machine, copy your public "business license" (ca.pem) into the data/ directory of your FromFirstPrinciples project.

# (Run on HOST, from your 0006_cicd... directory)
# This copies our CA cert into the shared data folder
cp ~/cicd_stack/ca/pki/certs/ca.pem ~/Documents/FromFirstPrinciples/data/ca-cert.pem

Action 2: Edit entrypoint.sh

Now, open your entrypoint.sh file (in your FromFirstPrinciples directory) with your text editor. Add the following block of code directly after the sudo service ssh restart line.

#!/bin/sh
sudo service ssh restart

# --- ADD THIS BLOCK TO TRUST THE LOCAL CA ---
CA_CERT_PATH="data/ca-cert.pem"
if [ -f "$CA_CERT_PATH" ]; then
    echo "--- Found Local CA certificate, installing to system trust store ---"
    # Copy the CA cert into the system's trust store
    sudo cp "$CA_CERT_PATH" /usr/local/share/ca-certificates/cicd-stack-ca.crt
    # Update the system's CA list
    sudo update-ca-certificates
else
    echo "--- No Local CA certificate found at $CA_CERT_PATH, skipping system trust ---"
fi
# --- END OF BLOCK ---

# Check for GPG key and gitconfig on the persistent data volume
if [ -e data/private.pgp ]; then
    gpg --import data/private.pgp
fi
...

(The rest of your entrypoint.sh file remains the same.)

Deconstruction

This logic is simple and powerful:

1. It checks if the data/ca-cert.pem file (which we just copied) is present.
2. If it is, it copies it into the container's system trust directory (/usr/local/share/ca-certificates/).
3. It runs sudo update-ca-certificates inside the container to make its system tools (like curl and git) trust our CA.

Action 3: Recreate the dev-container

Because we just changed the entrypoint.sh script (which is bind-mounted), we don't need to rebuild the image, but we do need to recreate the container to run the new script.

As we've noted, your entrypoint.sh script also handles your GPG key, which requires an interactive prompt. Therefore, we must use our docker stop/rm and ./dev-container.sh flow.

From your host machine (in the FromFirstPrinciples directory):

# (Run on HOST)
# 1. Stop and remove the old container
docker stop dev-container && docker rm dev-container

# 2. Run the new container (which will use the modified entrypoint)
./dev-container.sh

Result:
As the container starts, you will see our new messages in the log output, culminating in:

--- Found Local CA certificate, installing to system trust store ---
Updating certificates in /etc/ssl/certs...
1 added, 0 removed; done.
Running hooks in /etc/ca-certificates/update.d...
done.

This confirms our "Control Center" now also trusts our CA.

5.5 Verification: The "Success Second" (Python HTTPS Server)

We have now executed three complex, "first principles" operations:

1. We created a Root CA and a full set of service certificates, solving the "SAN Gotcha" and "Nested Path Gotcha."
2. We manually configured our Host OS to trust our CA, fixing curl, Chrome, and Firefox.
3. We configured our dev-container to trust our CA by modifying its entrypoint.sh.

The plan is complete, but we must verify it. We will now run a simple, local HTTPS server on our host machine using one of the "passports" we just printed (our gitlab.cicd.local certificate). We will then test if our browsers and our dev-container trust it.

Action 1: The Python Test Server

Here is the code for 05-verify-trust.py. This script is a simple, standard-library-only HTTPS server.

Create this file on your host machine at ~/Documents/FromFirstPrinciples/articles/0006_cicd_part02_certificate_authority/05-verify-trust.py.

#!/usr/bin/env python3

import http.server
import ssl
import os
from pathlib import Path

# --- Configuration ---
# This is the "simple name" of the certificate we want to use.
# It must match one of the subdirectories we created in pki/services/
# We will use 'gitlab' as our test certificate.
CERT_NAME = "gitlab.cicd.local"

# Find the '~/cicd_stack/ca' directory from the user's home
HOME_DIR = Path.home()
CA_DIR = HOME_DIR / "cicd_stack" / "ca"
PKI_DIR = CA_DIR / "pki"
CERT_DIR = PKI_DIR / "services" / CERT_NAME

# Paths to the certificate and private key files
CERT_FILE = CERT_DIR / f"{CERT_NAME}.crt.pem"
KEY_FILE = CERT_DIR / f"{CERT_NAME}.key.pem"

# Server configuration
HOST = "localhost"
PORT = 8443

# --- Validation ---
if not PKI_DIR.exists():
    print(f"--- ERROR ---")
    print(f"Could not find CA directory at: {CA_DIR}")
    print("Please ensure you have run '01-create-ca.sh' successfully.")
    exit(1)

if not CERT_FILE.exists() or not KEY_FILE.exists():
    print(f"--- ERROR ---")
    print(f"Could not find certificate files for '{CERT_NAME}'.")
    print(f"Expected cert at: {CERT_FILE}")
    print(f"Expected key at:  {KEY_FILE}")
    print("Please run './03-issue-all-certs.sh' first.")
    exit(1)

# --- Server Setup ---
print(f"--- Starting HTTPS server on https://{HOST}:{PORT} ---")
print(f"Loading certificate: {CERT_FILE}")
print(f"Loading private key: {KEY_FILE}")

# Create a standard TCP server
httpd = http.server.HTTPServer(
    (HOST, PORT), http.server.SimpleHTTPRequestHandler
)

# Create an SSL context
context = ssl.SSLContext(ssl.PROTOCOL_TLS_SERVER)
context.load_cert_chain(certfile=CERT_FILE, keyfile=KEY_FILE)

# Wrap the server socket with our new SSL context
httpd.socket = context.wrap_socket(httpd.socket, server_side=True)

print(f"--- Server is running. Open https://{HOST}:{PORT} in your browser. ---")
print("Press Ctrl+C to stop.")

try:
    httpd.serve_forever()
except KeyboardInterrupt:
    print("\n--- Server stopped. ---")

Deconstruction

• CERT_DIR = HOME_DIR / "cicd_stack" / "ca" ...: This Python code robustly finds the certificates we created by looking in our ~/cicd_stack/ca directory.
• context = ssl.SSLContext(...): This creates the secure SSL/TLS context for our server.
• context.load_cert_chain(...): This is the key line. It loads our service "passport" (gitlab.cicd.local.crt.pem) and its corresponding "passport key" (gitlab.cicd.local.key.pem).
• httpd.socket = context.wrap_socket(...): This "wraps" our standard HTTP server in the secure SSL context, upgrading it to HTTPS.

Action 2: Run the Server and Verify

From your host machine (inside your 0006_cicd_part02_certificate_authority directory), make the script executable and run it:

# (Run on HOST)
chmod +x 05-verify-trust.py
./05-verify-trust.py

The server is now running on https://localhost:8443.

Verification Step 1: Browsers (Host)

1. Open Google Chrome (in an Incognito window to be safe) and go to https://localhost:8443.
2. Open Firefox and go to https://localhost:8443.

Result: Both browsers should show a secure padlock icon 🔒. They will not show a "Your connection is not private" warning. This proves that our 04-trust-ca-host.sh script worked and successfully configured all three trust stores.

Verification Step 2: curl (Host)

Open a new host terminal (leaving the server running) and test with curl using both the name and the IP address.

# (Run on HOST)
# Test 1: By hostname
curl https://localhost:8443

# Test 2: By IP address
curl https://127.0.0.1:8443

Result: Both commands will succeed and print the HTML for a directory listing. They will not fail with an "SSL certificate problem." This proves our host's OpenSSL trust store is fixed and our SAN IP:127.0.0.1 entry is working.

Verification Step 3: curl (Container)

This is the final and most important test. We must prove that our dev-container—our "Control Center"—also trusts our new CA. We will do this by running our Python test server inside the container and then, from a second container shell, trying to curl it.

This is a perfect, hermetic test of the container's internal trust store.

Action 1 (Host): Copy Certs into the "Dropbox"
Our test server, 05-verify-trust.py, is configured to find certificates at ~/cicd_stack/ca/pki/services/.... This path doesn't exist inside our container, so we must first get the gitlab certificate folder into the container. We will use our mounted data/ directory as the "dropbox."

From your host machine:

# (Run on HOST)
# Copy the gitlab certs/key folder into the 'data' directory
cp -r ~/cicd_stack/ca/pki/services/gitlab.cicd.local/ ~/Documents/FromFirstPrinciples/data/

Action 2 (Container): Run the Server
Now, we'll enter our dev-container and set up the test. This requires two terminals.

Terminal 1 (dev-container shell 1):

# (Run on HOST)
docker exec -it dev-container bash

# (Inside dev-container - Terminal 1)
# 1. Re-create the path structure the script expects
mkdir -p ~/cicd_stack/ca/pki/services/

# 2. Copy the certs from our 'dropbox' to the expected path
cp -r ~/data/gitlab.cicd.local/ ~/cicd_stack/ca/pki/services/

# 3. Navigate to the article directory
cd ~/articles/0006_cicd_part02_certificate_authority/

# 4. Run the Python server
python3 05-verify-trust.py

Result:
The server will start successfully inside the container, as it can now find its certificates.

--- Starting HTTPS server on https://localhost:8443 ---
Loading certificate: /home/warren_jitsing/cicd_stack/ca/pki/services/gitlab.cicd.local/gitlab.cicd.local.crt.pem
Loading private key: /home/warren_jitsing/cicd_stack/ca/pki/services/gitlab.cicd.local/gitlab.cicd.local.key.pem
--- Server is running. Open https://localhost:8443 in your browser. ---
Press Ctrl+C to stop.

Terminal 2 (dev-container shell 2):
While the server is running in the first terminal, open a second host terminal and get another shell inside the dev-container.

# (Run on HOST - Terminal 2)
docker exec -it dev-container bash

Now, from this second shell, we will run curl to connect to the server running in our first shell.

# (Inside dev-container - Terminal 2)
# 5. Test connection to localhost
curl https://localhost:8443

# 6. Test connection to 127.0.0.1
curl https://127.0.0.1:8443

Result:
Both commands will succeed, returning the HTML of the directory listing.

Explanation:
This is the ultimate verification. curl (running inside the container) is using the container's system trust store. This test proves that:

1. Our entrypoint.sh script successfully ran at startup.
2. It correctly found data/ca-cert.pem and ran sudo update-ca-certificates.
3. The container's trust store is now fixed, and all tools inside our "Control Center" (like curl, git, and our future Python scripts) will fully trust our new CA.

Cleanup (Terminal 1):
You can now stop the Python server with Ctrl+C and remove the temporary certificate directory.

# (Inside dev-container - Terminal 1)
rm -rf ~/cicd_stack/
rm -rf ~/data/gitlab.cicd.local/

Chapter 6: The "Easy Way" - Automating with easyrsa

6.1 The "Why": Appreciating Automation

In the previous chapters, we built a fully functional, secure Certificate Authority "the hard way." We meticulously crafted our openssl.cnf, built our PKI directory structure, and manually managed our "ledger" (index.txt) and "ticket dispenser" (serial). We deconstructed the complex openssl ca command and learned, through failure, that we must manually build and pass a v3.ext file to handle the critical Subject Alternative Name (SAN) requirement.

This was an essential "first principles" exercise. Now that we have done the hard work and understand every moving part, we can appreciate the "easy button."

We will now introduce easyrsa, a tool whose sole purpose is to automate every tedious step we just learned.

6.2 The "What": easyrsa is a Wrapper

easyrsa is not a replacement for openssl. It is a shell script wrapper around openssl. It was developed by the OpenVPN project to simplify the creation of a PKI.

It is our "automated passport office clerk." It takes our simple, high-level commands (like "make a CA" or "make a server certificate") and, behind the scenes, runs the same complex openssl commands we just did, automatically managing the index.txt, serial file, and configuration templates for us.

We will now run a self-contained demo to prove this.

6.3 Action Plan: 06-run-easyrsa-demo.sh

Here is the code for 06-run-easyrsa-demo.sh. This script will install easyrsa by cloning its Git repository (as it is not available in the default Debian repositories), and then use it to build a second, separate "demo" CA.

Create this file on your host machine at ~/Documents/FromFirstPrinciples/articles/0006_cicd_part02_certificate_authority/06-run-easyrsa-demo.sh.

#!/usr/bin/env bash

# This script demonstrates the "easy way" to create a CA
# and issue a certificate using the 'easyrsa' wrapper.
#
# It is for demonstration only and does not use our
# main '~/cicd_stack/ca' directory.

DEMO_DIR="easyrsa-demo"
REPO_URL="https://github.com/OpenVPN/easy-rsa.git"
REPO_BRANCH="v3.2.4"
SERVICE_NAME="gitlab-demo"

# --- 1. Installation (via Git) ---
echo "--- 1. Installing 'git' (prerequisite) ---"
sudo apt-get update -qq && sudo apt-get install -y -qq git

# --- 2. Setup ---
echo "--- 2. Cloning easyrsa branch '$REPO_BRANCH' into demo directory: $DEMO_DIR ---"
rm -rf $DEMO_DIR
git clone --depth=1 -b $REPO_BRANCH $REPO_URL $DEMO_DIR

# Navigate into the new demo directory
cd $DEMO_DIR || (echo "Failed to clone easyrsa. Aborting." && exit 1)

# Navigate into the easyrsa3 directory where the executable is
cd easyrsa3 || (echo "Failed to find 'easyrsa3' directory. Aborting." && exit 1)


# --- 3. Create the PKI and Root CA ---
echo "--- 3. Initializing PKI and building Root CA ---"

# Initialize the PKI (creates 'pki' dir, index, serial, etc.)
./easyrsa init-pki

# Build the CA, 'nopass' = no password on the private key
# 'batch' = use defaults for the DN (Country, Org, etc.)
./easyrsa --batch build-ca nopass

# --- 4. Issue a Service Certificate ---
echo "--- 4. Issuing service certificate for '$SERVICE_NAME' (with SANs) ---"

# We MUST explicitly pass the SANs. We will add the service name,
# localhost, and 127.0.0.1, just like our "hard way" script.
SANS="DNS:$SERVICE_NAME,DNS:localhost,IP:127.0.0.1"
echo "--- Using SANs: $SANS ---"
./easyrsa --batch --san="$SANS" build-server-full $SERVICE_NAME nopass

# --- 5. Verification ---
echo "--- Demo complete. Verifying generated files: ---"

echo "1. CA Certificate:"
ls -l pki/ca.crt

echo "2. Service Private Key:"
ls -l pki/private/$SERVICE_NAME.key

echo "3. Service Certificate:"
ls -l pki/issued/$SERVICE_NAME.crt

echo ""
echo "--- Verifying SANs in the 'easyrsa' generated cert: ---"
openssl x509 -in pki/issued/$SERVICE_NAME.crt -noout -text | grep -A1 "Subject Alternative Name"

# Clean up by returning to the original directory
cd ../..

Deconstruction

This script automates the entire process, perfectly illustrating why openssl is so complex.

• git clone ...: First, we install git and clone the easyrsa repository, as it's not available in the default Debian package manager. We check out a specific, stable version (v3.2.4).
• ./easyrsa init-pki: This single command does what took us several steps in Chapter 3. It creates the entire PKI directory structure (pki/), including the index.txt "ledger" and the serial "ticket dispenser".
• ./easyrsa --batch build-ca nopass: This one command replaces our 01-create-ca.sh script. It generates the ca.key and ca.pem, automatically applying the correct CA:TRUE extensions from its own internal openssl.cnf template.
• ./easyrsa --batch --san="..." build-server-full ...: This is the most powerful command. It replaces our entire 02-issue-service-cert.sh script. It automatically handles the key generation, CSR, and signing.
• The --san Flag: This is the most critical lesson. Even this "easy" tool, when run in non-interactive batch mode, still required us to explicitly pass the SANs. This proves that our "first principles" lesson was correct: in modern PKI, SANs are non-negotiable.
• Final Verification: We use our "hard way" openssl command at the end to "check the work" of the "easy way" wrapper. The output DNS:gitlab-demo, DNS:localhost, IP Address:127.0.0.1 proves that easyrsa is just a powerful, convenient shell script running the same openssl logic we already learned.

Action

From your host machine (inside your 0006_cicd_part02_certificate_authority directory), make this new script executable and run it.

# (Run on HOST)
chmod +x 06-run-easyrsa-demo.sh
./06-run-easyrsa-demo.sh

Result:
The script will install git, clone the easyrsa repository, and then run the simplified workflow. The final openssl command will print the SAN field, proving that easyrsa is just an automated wrapper around the same "first principles" we learned.

Chapter 7: Conclusion and Next Steps

7.1 What We've Built

We have successfully built a complete, local Public Key Infrastructure (PKI) from "first principles." We started with the "pain point" of insecure http and browser errors and deconstructed the entire chain of trust.

We rejected the "magic" of simple, self-signed certificates and instead built our own "passport office"—a full Certificate Authority. We did it the "hard way" with openssl to understand every moving part:

• We created a secure openssl.cnf "blueprint."
• We manually built the CA's "filing system" (index.txt, serial).
• We solved the critical "SAN Gotcha" by creating a v3.ext file to issue modern, compliant certificates.
• We solved the "Nested Path Gotcha" using basename to make our issuer script reusable.

We then solved the "new pain point" of trust by deconstructing the "disjointed trust stores" on our host. We surgically added our CA to the OpenSSL store (for curl), the NSS store (for Chrome), and the Firefox store (for Firefox) using our 04-trust-ca-host.sh script.

Finally, we verified our entire setup end-to-end, proving that both our host machine and our dev-container now fully trust our new "passport office."

7.2 Next Steps

Our "city" is finally ready.

• We have a "Control Center" (DooD).
• We have our "roads" (cicd-net).
• We have our "foundations" (volumes and bind mounts).
• And now, we have our "security system" (our trusted CA).

We are finally ready to build our first "skyscraper." In the next article, we will deploy GitLab, the "Central Library" for our source code. We will use the docker run command to deploy it onto our foundation and use the gitlab.crt.pem and gitlab.key.pem files we just created to make it fully secure with HTTPS from the moment it starts.