DEV Community: Frederick Ollinger

Kubernetes Delete Stuck Deployment

Frederick Ollinger — Thu, 19 Feb 2026 18:29:50 +0000

Use case is a Kubernetes deployment in a namespace, but you want to keep part of the deployment such as Persistent Volume Claims (PVC) and Secrets, but you want to remove everything else.

In this example, my namespace is named "ns".

Normally, you can do this with:

terraform destroy -auto-approve

However, there are cases where the deployment gets stuck often when you are in a CrashBackoffLoop.

In these cases, deleting your namespace is the shortest path.

But this will not work if you have a situation like mine with PVC and so on as they live in our namespace.

DO NOT DO THIS IF YOU WANT TO KEEP PVC AND SECRETS:

kubectl delete namespace ns

Instead, I wrote a script which will delete all the things I want to remove but keep pvc and secrets. Feel free from the script other things you want to keep:

#!/usr/bin/bash

# This script is intended to take down everything in otel except
# the Persistent Storage

NS=$1

if [[ -z "$1" ]]; then
    echo "Warning this will take down everything in the given namespace"
    echo ""
    echo "PLEASE ONLY USES THIS IF YOU KNOW WHAT YOU ARE DOING"
    echo ""
    echo "Usage: $0 namespace_to_delete"
    exit 1
fi


kubectl scale deployment --all -n $NS --replicas=0
kubectl scale statefulset --all -n $NS --replicas=0
kubectl delete configmap --all -n $NS
kubectl delete daemonset --all -n $NS
kubectl delete statefulset --all -n $NS
kubectl delete serviceaccount --all -n $NS
kubectl delete ingress --all -n $NS
kubectl delete pod --all -n $NS
kubectl delete service --all -n $NS
# helm uninstall helm_service -n $NS

Save this script as manual-delete-deployment.sh and run it:

bash manual-deplete-deployment.sh ns

Be sure to replace the helm_service with any helm services you have as they will often stay stuck as well.

You can find them with

helm list -n ns

You can check to ensure that your deployment is gone:

kubectl get all -n ns

Happy coding!

C# HTTPS REST Connections Using Self-Signed TLS Certificates

Frederick Ollinger — Wed, 25 Jun 2025 16:03:50 +0000

RestSharp is a C# library for making client side REST connections to servers.

It can use HTTP or HTTPS, but it does not seem to like self-signed certificates to connect to a server.

Getting Shell Access to a Kubernetes Container

Frederick Ollinger — Fri, 30 May 2025 17:22:35 +0000

Problem: You have a running Kubernetes deployment, and you want to examine inside a container so you need to get shell access.

First find the pod that you are interested in.

NOTE: If you are running in a namespace, you need to include that as part of your listing with the -n flag, if not, you can skip -n. In this case, we'll name our namespace: namespace.

kubectl -n namespace get pods

This shows:

NAME      READY   STATUS    RESTARTS   AGE
fun-pod-0   1/1     Running   0          21h

To connect to fun-pod-0, we do:

kubectl exec -it pod/fun-pod-0 -n namespace -- sh

NOTE: That we use sh for our shell as it's found in most linux pods. But your milage may vary and you might want to use bash or even /bin/bash instead of sh depending upon what shell is installed and what path.

Using a Self-Signed Certificate with Git Clone Https

Frederick Ollinger — Thu, 14 Nov 2024 00:28:02 +0000

Ever have a problem when you need to clone from a git repo with https, and it fails because you have a self signed certificate?

Every other post tells you to "turn off security".

Don't do that.

First download the certificate with this script:

#!/bin/env bash

# User Variables
API_HOST=example.gitlab.com # could be an ip address
PORT=443
CRT=secret.crt

BEGIN="-----BEGIN CERTIFICATE-----"
END="-----END CERTIFICATE-----"

echo $BEGIN > $CRT
echo quit | openssl s_client -showcerts -servername "${API_HOST}" -connect "${API_HOST}":${PORT} | sed "/$BEGIN/,/$END/!d;//d" >> $CRT
echo $END >> $CRT

Use the script to download your certificate:

cd
./download-certificate.sh

Now tell git where your certificate is:

git config --global http.sslCAInfo ~/secret.crt

Done.

Now you should be able to use git clone with your https server without issues.

Deploy Postgresql with Helm

Frederick Ollinger — Thu, 26 Sep 2024 21:54:12 +0000

Helm is the package manager for deploying pre-built packages to Kubernetes.

I found the Postgresql package here: https://artifacthub.io/packages/helm/bitnami/postgresql

Follow the TL;DR

helm install my-release oci://registry-1.docker.io/bitnamicharts/postgresql

Looks like has issues:

kubectl describe pod my-release-postgresql-0


  Warning  FailedScheduling  2m33s  default-scheduler  0/1 nodes are available: pod has unbound immediate PersistentVolumeClaims. preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling

Well that did not work out. Let's figure out what's wrong.

Seems like it's related to a Persistent Volume Claim. What is that?

A Persistent Volume is a piece of storage in your Kubernetes cluster. It is provisioned by the cluster administrator or dynamically provisioned using a storage class.

It represents a physical storage resource, which can be backed by various storage systems (e.g., NFS, cloud storage like AWS EBS, etc.).

"So a persistent volume (PV) is the "physical" volume on the host machine that stores your persistent data. A persistent volume claim (PVC) is a request for the platform to create a PV for you, and you attach PVs to your pods via a PVC." [1]

References

Persistent volume vs. persistent volume claim https://stackoverflow.com/questions/48956049/what-is-the-difference-between-persistent-volume-pv-and-persistent-volume-clai

Setting Up Service Principle Deployments for Gitlab CI/CD Terraform Virtual Machine Deployments

Frederick Ollinger — Tue, 07 May 2024 22:58:09 +0000

In order to use Terraform to deploy to the cloud, you need to either be logged in to Azure all ready or you need to create a Service Principle.

How to login to Azure?

Given that you have an account:

az login

But this won't work in the cloud as it's not headless and will leak credentials. So this is a non-starter. For testing, we need to stay logged out.

az logout

That said, we do need to login to manipulate Service Principles. So log back in for this portion of this tutorial.

What is a Service Principle in Azure?

"An Azure service principal is a security identity used by user-created apps, services, and automation tools to access specific Azure resources." 1

List all your current Service Principles:

az ad sp list -o=table --show-mine

Create a new Service Principle

az ad sp create-for-rbac

This will give you the following:

{                                                                                                                                                                                                                  "appId": XXX,
  "displayName": XXX,
  "password": XXX,
  "tenant": XXX
}

Map the Service Principle JSON to Terraform Variables

In the provider block:

client_id: This is the appId from the Service Principal JSON object.
client_secret: This is the password from the Service Principal JSON object.
tenant_id: This is the tenant from the Service Principal JSON object.
subscription_id: This is your Azure subscription ID. It's optional if the Service Principal has

provider "azurerm" {
  features {}

  # Use the appId (client ID), password (client secret), and tenant ID (tenant)
  client_id       = var.client_id
  client_secret   = var.client_secret
  tenant_id       = var.tenant_id
  subscription_id = var.subscription_id  # Optional: specify your Azure subscription ID
}

Manipulate Tags on Gitlab CI/CD

Frederick Ollinger — Fri, 29 Mar 2024 00:26:43 +0000

On a Gitlab CI/CD pipeline, how to list, create, and delete tags.

Preliminaries

This assumes that you have a Gitlab project with a Docker linux container with curl and jq installed.

Create a Repository Access Token

Access token is needed to give permission to access the Gitlab api.

Navigate to your project then go to the sidebar -> Settings -> Access Tokens.

Token name is "tag". Select scope check "api" and leave the rest unchecked. Then click project access token.

Be sure to save the token locally as once it disappears, you can not get it back again and need to create a new one.

Make a new Gitlab variable called GITLAB_TOKEN.

List Tags

#!/bin/bash                                                                                                                                                                                 

 # list-tags.sh

# List all Gitlab tags for a given project.                                                                                                                                           
# $CI_PROJECT_ID is a Gitlab pre-defined variable                                                                                                                                           
# $GITLAB_TOKEN is a project access token that needs to be                                                                                                                                  
# a defined CI/CD variable.                                                                                                                                                                 

GITLAB_API_URL="https://gitlab.com/api/v4/projects/${CI_PROJECT_ID}"                                                                                                                   

curl -k --header "PRIVATE-TOKEN: $GITLAB_TOKEN" "${GITLAB_API_URL}/repository/tags"

Create Tag

#!/bin/bash

# create-tag.sh

# $CI_PROJECT_ID is a Gitlab pre-defined variable
# $GITLAB_TOKEN is a project access token that needs to be
# a defined CI/CD variable.
# TAG_NAME needs to be defined as a variable in the yaml.

GITLAB_API_URL="https://gitlab.com/api/v4/projects/${CI_PROJECT_ID}"

curl -k --request POST --header "PRIVATE-TOKEN: $GITLAB_TOKEN" $GITLAB_API_URL/repository/tags?tag_name=$TAG_NAME\&ref=$CI_COMMIT_SHA

Adding Code to Yaml

stages:
  - deploy
tag:
  stage: deploy
  image: curl/curl-docker
  variables:
    # Make TAG_NAME a more meaningful variable
    TAG_NAME: $CI_COMMIT_SHA
  script:
    # Create a Tag
    - bash tag.sh

Update Gitlab On Prem Server Running in a Docker Container

Frederick Ollinger — Mon, 12 Feb 2024 18:06:27 +0000

Problem: We had a Gitlab server with repos in it that needed to be updated which was running on prem inside a Docker container. Worse, the container is not versioned. If we restarted the Docker container for any reason, we would wind up with the latest version being run which is not the one which was run when the container was set up. This means losing all our work.

In this example, it's going to be assumed that one all ready has an instance of Gitlab server running in an official container using Docker and that one can start, stop, and connect to the running container.

Surprisingly, that while there is online guides on how to host Gitlab on prem in a Docker container, and there's an official Gitlab Docker container on Docker Hub, there were no instructions on how to update Gitlab inside a Docker container.

How to actually run Gitlab in a Docker container, is beyond the scope of this article.

NOTE: One caveat is that you can NOT make a backup on one version then restore it to another version. In this guide, we suggest backing up twice. 1st before one updates then after an update. That way if things go badly with the update, you still have your original repos in the original backup.

Solution.

1st Step is to backup the container in place so if things go badly, we can restore it:

1st we need to connect to the container:

docker exec -it $CONTAINER /bin/bash

Then backup:

mkdir -p /var/opt/gitlab/backups
/usr/bin/gitlab-backup create force=yes GZIP_RSYNCABLE=yes STRATEGY=copy CRON=1
cp /etc/gitlab/gitlab-secrets.json /var/opt/gitlab/backups
cp /etc/gitlab/gitlab.rb /var/opt/gitlab/backups

Update Gitlab:

In this example, we'll use 14.9.1, but you should pick the version that's right for you via an upgrade path. More information on this is here:

https://docs.gitlab.com/ee/update/index.html#upgrade-paths

apt update
apt install -y curl ca-certificates openssh-server apt-utils perl
curl -s https://packages.gitlab.com/install/repositories/gitlab/gitlab-ee/script.deb.sh | bash
apt update
apt install gitlab-ee=14.9.5-ee.0

Now backup again as per the steps above.

Finally, change the tag on the container. In this case, the tag should be 14.9.5-ee. then restart the Docker container.

Finally, restore the latest back which matches the new version:

BACKUPDIR=/home/git/data/backups
cd $BACKUPDIR
gitlab-ctl stop unicorn
gitlab-ctl stop sidekiq
cp ${BACKUPDIR}/gitlab-secrets.json /etc/gitlab
cp ${BACKUPDIR}/gitlab.rb /etc/gitlab
GITLAB_ASSUME_YES=1 /usr/bin/gitlab-backup restore BACKUP=${BACKUPFILE}
/usr/bin/gitlab-ctl restart

You should be able to go to your Gitlab web page and find that latest version is running and that all your repos are restored.

Pulling Git Submodules on Gitlab CI/CD Pipeline From a Different Repo

Frederick Ollinger — Mon, 24 Apr 2023 17:46:30 +0000

This is for solving the problem of building a repo with a different ssh key.

In this case, we had some legacy code on a Gerrit server, and we wanted to pull this code so we could build it on our Gitlab server.

In this example, our service account for Gerrit will be gitlab. The name shows that the Gerrit account is only for Gitlab to have access, and it is not tied to a real human's account. The linux user account on our workstation that we use for this example is testuser.

The 1st step is to create a new ssh key solely for this purpose.

I used ssh-keygen:

ssh-keygen -t rsa

When it askes for the name of the key, I use gerrit.pem.

Enter file in which to save the key (/home/testuser/.ssh/id_rsa): gerrit.pem

You should have 2 new files:

gerrit.pem - private key
gerrit.pem.pub - public key

Put the private key in your Gitlab variables and call it GERRIT_PEM:

https://docs.gitlab.com/ee/ci/variables/#define-a-cicd-variable-in-the-ui

Put the public key into Gerrit. See Register SSH public key.

https://haiyanghaiyang.gitbooks.io/gerrit-user-manual/content/sheng-cheng-yu-zhu-ce-ssh-key.html

Now you should test the ssh key on an linux account did not previously have access to the Gerrit repo. In the case, we'll call the user testuser.

Copy the gerrit.pem file to the testuser home folder and give ownership of it to testuser.

sudo cp gerrit.pem /home/testuser
sudo chown testuser /home/testuser/gerrit.pem

For this test, we will use ssh-agent.

ssh-agent is a way to allow us to put ssh keys into RAM so we can use keys which are not part of our ~/.ssh folder.

In this example, I pull one of the git submodules that I wanted to pull to make the test simple.

sudo su testuser # become testuser
eval "$(ssh-agent -s)" # start ssh-agent
ssh-add ~/gerrit.pub # add ssh key to the agent
git clone ssh://gitlab@gerrit.testexample.com/source_code # note this will change depending upon what your actual server, username, and source code is called.

This should pull the source code.

If this works, we are almost home free.

We need to add new Gitlab variables. In my case it was 2 new ones:
GERRIT_KNOWN_HOSTS1 and GERRIT_KNOWN_HOSTS2.

These are gotten from known hosts. Still as test user:

cat /home/testuser/.ssh/known_hosts

There should be 2 new lines here. Add them to Gitlab variables as the variable names above. We have to split each line into a separate variables because of the way that line breaks get broken if we don't, and the known_hosts becomes invalid due to a missing line break.

Next create a .gitlab-ci.yml file. We'll use the standard header:

build-code:
  stage: build
  image: centos:7

Next create the .ssh folder and known_hosts and change permissions. Ssh is super picky about these things:

    - mkdir ~/.ssh
    - touch ~/.ssh/known_hosts
    - chmod 600 ~/.ssh
    - chmod 644 ~/.ssh/known_hosts

Now start the ssh-agent. Note it has to be installed in the container:

    - eval "$(ssh-agent -s)"

Now add the key to ssh-agent. But this time, we don't have it on our disk, we need to pull from Gitlab variables that we set above:

    - ssh-add - <<< "${GERRIT_PEM}"

This is the magic. We need to populate known_hosts with what worked with testuser.

    - echo ${GERRIT_KNOWN_HOSTS1} >> ~/.ssh/known_hosts
    - echo ${GERRIT_KNOWN_HOSTS2} >> ~/.ssh/known_hosts

Next we need to tell ssh that we are using user gitlab instead of our container user. This will allow git submodules to work with a different user:

    - echo "Host gerrit.testexample.com" > ~/.ssh/config
    - echo "User gitlab" > ~/.ssh/config

Then we need to actually pull the submodules:

    - git submodule update --init --recursive

Finally, we need to build our code. Your build system may differ:

    - make

All in 1 script:

build-code:
  stage: build
  image: centos:7
  script:
    - mkdir ~/.ssh
    - touch ~/.ssh/known_hosts
    - chmod 600 ~/.ssh
    - chmod 644 ~/.ssh/known_hosts
    - eval "$(ssh-agent -s)"
    - ssh-add - <<< "${GERRIT_PEM}"
    - echo ${GERRIT_KNOWN_HOSTS1} >> ~/.ssh/known_hosts
    - echo ${GERRIT_KNOWN_HOSTS2} >> ~/.ssh/known_hosts
    - echo "Host gerrit.testexample.com" > ~/.ssh/config
    - echo "User gitlab" > ~/.ssh/config
    - git submodule update --init --recursive
    - make

Check this .gitlab-ci.yaml into your Gitlab repo and test it out.

Docker Same User Inside a Container Dynamically

Frederick Ollinger — Mon, 06 Mar 2023 21:38:49 +0000

In this post, we'll create a user dynamically inside a Docker container with the use UID and GID as the user on the host who ran the container.

Previously, we addressed the issue of accessing a mounted volume inside a container as user root inside the container with Docker Compose here: https://dev.to/frederickollinger/docker-compose-access-a-mounted-volume-inside-a-container-41kf

But this solution wound up creating artifacts which were owned by root which meant that accessing them as a regular user on the host was more difficult.

Thus, we actually want to have a user inside the Docker container which is the same as the host.

The 1st step is to make a bash script which will create a new user inside the container dynamically.

Create a script called entrypoint.sh:

#!/bin/env bash

USER=container

useradd -u $2 $USER -d /home/$USER
chown container /home/$USER
su $USER

This script will basically add a new user called container when the Docker container is 1st run and use the user id passed into the container.

Next create Dockerfile in the same directory:

FROM centos:7                                                                                                            
USER root

COPY entrypoint.sh /usr/sbin/entrypoint.sh
RUN chmod a+x /usr/sbin/entrypoint.sh
ENTRYPOINT ["/usr/sbin/entrypoint.sh"]

This Dockerfile will run our entrypoint.sh script when it's first run. It needs to start as root but then it will change to the container user which happens to have the same user id as whomever started the container.

Build the container, in this example as userdemo:

docker build -t userdemo .

Now run the container.

   docker \
    run \
    --rm \
    --name=userdemo \
    --hostname=userdemo \
    --workdir=/home/container \
    -v `pwd`/project:/home/developer/project \
    userdemo \
    /bin/bash `id -u`

In this example, you will have a folder called "project" which is mounted inside the container as /home/container/project. You can change this or even add more lines with the volume mount flag.

Docker Compose Access A Mounted Volume Inside a Container

Frederick Ollinger — Wed, 01 Mar 2023 18:21:44 +0000

Problem: Using a container to build my project but I can't access the mounted volume due to permission issues.

Assumptions: This tutorial assumes you are working on a reasonably up to date Linux box with Docker and Docker Compose installed, and that you have a basic familiarity with the Linux command line.

A mounted volume is when you mount a directory from your filesystem inside a container. This allows you to access files which were not available when you built the container. Also, you have access to the build artifacts outside the container after building inside the containter.

Docker Compose File:

version: '3.8'
services:
  gcc:
    image: "gcc"
    command: /bin/bash
    group_add:
        - $GID
    volumes:
      - ${PWD}/fullbuild:/root/fullbuild

This file will pull the official gcc Docker container. It expects the group user id to be $GID and the source code to build to be at fullbuild.

We usually make the $GID the same as the default group for the user who is currently running docker-compose on the host system.

If we do not have the fullbuild directory make it:

    mkdir -p fullbuild

The copy any source code in.

Before we run the function, we need to ensure that our default group has access to both fullbuild and its subdirs:

find fullbuild -type d -exec chmod g+rwx {} +

Finally, when we run and enter the container, ensure that $GID is set:

GID=`id -g` docker-compose run gcc

That's it. After we run the last command, we'll be inside a Docker container with access to gcc as well as having access to all the source code inside the container. Source code is in /root/fullbuild. Build artifacts will be available in fullbuild outside of the container.

Rust ^ cannot use the `?` operator in a function that returns `()`

Frederick Ollinger — Mon, 27 Feb 2023 21:21:18 +0000

Problem:

When I try to run code with the following function:

fn invalid_function() {
    let mut file = File::open("data/AA.csv")?;
}

This returns the following error:

$ cargo run

   Compiling voxide v0.1.0 (/rick/follinge-2020-08-10/projects/voxide)
error[E0277]: the `?` operator can only be used in a function that returns `Result` or `Option` (or another type that implements `FromResidual`)
  --> src/main.rs:10:45
   |
9  | fn invalid_function() {
   | --------------------- this function should return `Result` or `Option` to accept `?`
10 |     let mut file = File::open("data/AA.csv")?;
   |                                             ^ cannot use the `?` operator in a function that returns `()`
   |
   = help: the trait `FromResidual<Result<Infallible, std::io::Error>>` is not implemented for `()`

For more information about this error, try `rustc --explain E0277`.
error: could not compile `voxide` due to previous error

The 1st clue is in asking ourselves what does the question mark do in this context? The answer is error propagation:

https://doc.rust-lang.org/book/ch09-02-recoverable-errors-with-result.html

So basically it worked in main() because that was the top level function, and if we failed, we crashed as the error state was uncaught which is okay.

But if we use the question mark operator in a function, we need to return the Result which tells the caller that they need to handle a potential error state.

A better way to handle it is to either use the if or match keywords.


fn valid_function() -> () {
    if let Ok(file) = File::open("data/AA.csv") {
        // Do something with the file
    }
    else {
        // Handle the error
    }
}

This will open a file, and check for errors.

This pattern is used so often that Rust has a match keyword for dealing with enums:

fn better_valid_function() -> () {
    let file_result = File::open("data/AA.csv");
    let file = match file_result {
        Ok(file) => file,
        Err(error) => panic!("Problem opening the file: {:?}", error),
    };
}