If you've deployed a Helm chart that depends on Postgres lately, you've likely found yourself coming across Bitnami's chart as a dependency (or even just being used to deploy Postgres itself!). This has been very useful for others to standardize on and allows us all to spin up a Postgres instance quickly. However, I've found myself on the wrong end of this chart a handful of times in the past as I am generally looking to run things on top of OpenShift. Because of the security defaults present on OpenShift, some of the default settings of this chart cause the deployment to fail and you end up finding yourself looking at an error that looks like this:
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedCreate 1s (x11 over 7s) statefulset-controller create Pod pg-postgresql-0 in StatefulSet pg-postgresql failed error: pods "pg-postgresql-0" is forbidden: unable to validate against any security context constraint: [provider restricted: .spec.securityContext.fsGroup: Invalid value: []int64{1001}: 1001 is not an allowed group spec.containers[0].securityContext.runAsUser: Invalid value: 1001: must be in the ranges: [1000630000, 1000639999]]
To hopefully save you some of the same grief that I've went through, I figured I would write up how to get around these issues moving forward.
What Does This Error Mean
Before we go about fixing this issue, it's helpful to understand what this error is actually telling us. The key part to pay attention to is unable to validate against any security context constraint. What this is telling us is essentially: ⚠️🔥⚠️🔥. By default, OpenShift has a number of security measures in place. One of them defines the range that a UID must be within in a given project. When running an application in OpenShift, it will attempt to assign a random UID within this range to the application running within the pod. What a number of Helm charts do is attempt to set the securityContext for you. This tends to cause some issues in OpenShift as these securityContexts dont align with the rules that are in place, which cause the application to not be able to start up.
In some well designed charts (such as those in the Bitnami collection), there are parameters that allow you to to configure this appropriately, which is what we will look at next!
How To Resolve It
By looking at the values.yaml file provided by the Postgres chart, Bitnami actually gives us a few hints as to what we need to do.
## Init container Security Context
## Note: the chown of the data folder is done to securityContext.runAsUser
## and not the below volumePermissions.securityContext.runAsUser
## When runAsUser is set to special value "auto", init container will try to chwon the
## data folder to autodetermined user&group, using commands: `id -u`:`id -G | cut -d" " -f2`
## "auto" is especially useful for OpenShift which has scc with dynamic userids (and 0 is not allowed).
## You may want to use this volumePermissions.securityContext.runAsUser="auto" in combination with
## pod securityContext.enabled=false and shmVolume.chmod.enabled=false
What this tells us is that we want to have something that looks like the following included in our own values.yaml:
volumePermissions:
enabled: false
securityContext:
runAsUser: "auto"
securityContext:
enabled: false
shmVolume:
chmod:
enabled: false
So let's give this a shot. We'll first want to configure our environment so that we can deploy the Postgres chart. To do this, we just want to let our Helm client know where to find the chart:
helm repo add bitnami https://charts.bitnami.com/bitnami
Once this is done, we can create our own my-values.yaml with the above values. This will allow us to override what is provided by the Bitnami chart defaults. After we have this saved to our file, we can run things into our cluster with the following:
helm install our-release -f my-values.yaml
Afterwards you should see mention of a successful installation and some instructions on how to get logged in, interact with the cluster, etc. What we want to check though is the status of the StatefulSet. This will look something like:
╰─ oc get sts
NAME READY AGE
pg-postgresql 0/1 5m
At this point, it looks things still aren't up and running. By running the following we'll also confirm that we see a similar error:
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedCreate 1s (x10 over 4s) statefulset-controller create Pod pg-postgresql-0 in StatefulSet pg-postgresql failed error: pods "pg-postgresql-0" is forbidden: unable to validate against any security context constraint: [spec.containers[0].securityContext.runAsUser: Invalid value: 1001: must be in the ranges: [1000630000, 1000639999]]
If we take a look at the original comment in the default values.yaml, we seem to have disabled all of the appropriate securityContext options to allow OpenShift to start configuring this itself. However, what we've discovered is a common problem with comments. They're static 💀. The longer they stick around, the more likely to become out of date they are. A quick search for securityContext in the values file shows us the following:
containerSecurityContext:
enabled: true
runAsUser: 1001
This is also confirmed if you search the ArtifactHub page for this chart and search OpenShift.
For OpenShift, one may either define the runAsUser and fsGroup accordingly, or try this more dynamic option: volumePermissions.securityContext.runAsUser="auto",securityContext.enabled=false,containerSecurityContext.enabled=false,shmVolume.chmod.enabled=false
So to clean this up, all we need to do is add the above to our my-values.yaml file:
volumePermissions:
enabled: false
securityContext:
runAsUser: "auto"
securityContext:
enabled: false
shmVolume:
chmod:
enabled: false
containerSecurityContext:
enabled: false
Once this is in place, we can just clean up after our previous installation and then reinstall our chart to make sure things are in good shape:
╰─ helm uninstall our-release
╰─ helm install our-release -f my-values.yaml
Afterwards, we should see similar messages that we saw during our previous install. But now when we go to look at the status of our StatefulSet, we should be in a much better place.
╰─ oc describe sts
<snip>
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal SuccessfulCreate 9s statefulset-controller create Pod pg-postgresql-0 in StatefulSet pg-postgresql successful
We should now be able to see the pod being created (assuming that we don't hit any odd dockerhub limitations):
╰─ oc get pods
NAME READY STATUS RESTARTS AGE
pg-postgresql-0 1/1 Running 0 87s
And just like that, we have a functional Postgres instance that we're able to interact with. Hopefully this was helpful to you and will help you in the future with other charts that act in a similar manner! If you run into any issues or see any issue with what is noted above, feel free to reach out!
Update:
About halfway through this article, I noticed that this was really rooted in a documentation error and only hits folks like myself who don't venture away from the terminal too much. If you look at the README for this chart (which I believe also feeds the ArtifactHub landing page), you'll see a note that says that this additional securityContext settings needs set to false. I've updated the article to call this out in the appropriate places.
Top comments (3)
note most modern charts have the following section:
global.compatibility.openshift.adaptSecurityContext=auto
which automates the parts you were talking about, no need to change the runasuser/fsgroup for every container. As some charts have dozens of containers, and that could be a huge problem.
artifacthub.io/packages/helm/bitna...
I spent a lot of time to resolve this issue and the simplest way to resolve this is, you can simply edit your values.yaml file and disable the security context for pod and container both. Also comment the fsgroup and runAsUser parameters.
For instance;
podSecurityContext:
enabled: false
# fsGroup: 1001
containerSecurityContext:
enabled: false
# runAsUser: 1001
ARGH :(
With OpenShift 4.11 not really working:
W0118 16:10:41.358143 84716 warnings.go:67] would violate PodSecurity "restricted:v1.24": allowPrivilegeEscalation != false (container "postgresql" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (container "postgresql" must set securityContext.capabilities.drop=["ALL"]), runAsNonRoot != true (pod or container "postgresql" must set securityContext.runAsNonRoot=true), seccompProfile (pod or container "postgresql" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")
NAME: postgresql-bitnami
LAST DEPLOYED: Wed Jan 18 16:10:40 2023
NAMESPACE: deptrack
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
CHART NAME: postgresql
CHART VERSION: 12.1.9
APP VERSION: 15.1.0
Any Idea?