Use OpenShift secret securely without any third party tool

Note: This approach does not use any OpenShift specific features (apart from oc CLI). So it should work in any Kubernetes cluster as well.

You might have used OpenShift secrets many times by defining them on the cluster and using it inside the pods. There different ways in OpenShift to make the secrets available to the pods. For example, using the volume mounts or environment variables. Those are documented here.

Have you ever wondered how much secure these secrets are once you make them available inside the pod? Anyone who can sneak into the container can see these secrets. If your application dumps the environment variables as part of logging, the secrets could be exposed to the outside world in exponentially. This is exactly what we will be discussing here.

You have a use case where you need to define OpenShift secrets to keep the sensitive information used by the application pods. It could be the database credentials, certificate files or any other sensitive information.

Here we will use very simple example of using the secret. There is a pod that runs a Java application. This application connects to the database which is running external to the application pod. It can be on another pod or outside of the OpenShift cluster. The application just needs the database username and password to connect to the database (along with other obvious details like host, port and so on). We will treat database password as the only sensitive information. The application reads the database password from the configuration file stored on the file system accessible to the application.

Lets try to implement this in the OpenShift in a simple way.

Simple way

The simplest way to use OpenShift secret in the pod is as below.

• Create a OpenShift secret that will contain database password. ```

oc create secret generic db-passwd-secret --from-literal=DB_PASS=password123

- This secret is then used in the application pod as environment variable. Below is one of the few ways to load secret as environment variables.

containers:

• name: app-container image: k8s.gcr.io/busybox command: [ "/bin/sh", "-c", "env" ] envFrom:
  • secretRef: name: db-passwd-secret

- Container startup script reads the database password from environment variable, and writes it to the configuration file in plaintext.
-  Application reads the database password from the configuration file and uses it to connect to the database.

Though it is very simple implementation using the features supported by OpenShift, it has many security concerns. The most important being the database password is visible in plaintext as environment variable and in the configuration file. Anyone who has access to the container can see the database password.

Les see how we can do it in better way...

## Secure way

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/0s7erh72yhzhvg12eqmw.png)

No one can make the application secure in OpenShift if the application itself does not support handling of the sensitive data securely. In this example, keeping the database password in plaintext in the configuration file is basic security flaw in the application. Such flaws cannot be handled at the infrastructure or platform layer (for example in OpenShift). The application requires certain changes to adopt to the enhanced security model.

### Application changes to use encrypted config
- There are some pre-requisite changes required in the application to remove the dependency on plaintext password. The application should be updated to allow keeping database password encrypted in the configuration file and decrypting it at runtime using the provided passphrase at the time of startup. 
For that, it should support a way to encrypt and decrypt the password using a passphrase. We can keep it simple by using symmetric encryption where same passphrase is used to encrypt and decrypt. There should not be any need to have plaintext password available in the application container (for example to make database connectivity checks in the readiness probe).
- This passphrase is also kept in the OpenShift secret. This is a different secret than the secret used to keep the database password.
So create two secrets, one for database password and other for passphrase.
- Create a OpenShift secret that will contain database password. 

oc create secret generic db-passwd-secret --from-literal=DB_PASS=password123

- Create a OpenShift secret that will contain passphrase. 

oc create secret generic passphrase-secret --from-literal=PASSPHRASE=secretpassphrase

### Init container processing
- Init container for loading application secrets -
  - An init container is created to perform the processing of secrets. 
  - The OpenShift secrets `db-passwd-secret` and `passphrase-secret` are made available to init container using volume mounts. 
  - Init container shares the *emptyDir* volume with application container. This data in this volume is kept in-memory and never written to the disk (with `medium: Memory`).

initContainers:
volumeMounts:

• name: init-secret-shared mountPath: /mnt/init-secret .... containers: volumeMounts:
• name: init-secret-shared mountPath: /mnt/init-secret .... volumes:
  • name: secret-db-pwd secret: secretName: db-passwd-secret
  • name: secret-passphrase secret: secretName: passphrase-secret
  • name: init-secret-shared emptyDir: medium: Memory

-   Init container’s startup script reads both the secrets from volume mounts. It copies the plaintext database password (retrieved from OCP secret `db-passwd-secret`) and passphrase (retrieved from OCP secret `passphrase-secret`) into a file on shared volume `init-secret-shared`.
-   The database password and passphrase are written to the init-shared-secret volume in `/mnt/init-secret/secrets` file.

$ cat /mnt/init-secret/secrets
DB_PASS=password123
PASSPHRASE=secretpassphrase

-   The init container completes it's execution and exits. It is no more accessible to get inside the container.

### Application container processing
-   The startup script of application container is updated to read the file from shared volume. Inside the application container, the file is available at `/mnt/init-secret/secrets`. The startup script reads the database password and passphrase from the shared file.
-   The application container has the `encrypt.sh` script included which encrypts the plaintext database password using the passphrase.

encypt.sh --passphrase secretpassphrase password123

- This command generates the output as encrypted value of the password.

Rh75ggs4s0#j@1

Note: There are various tools to perform symmetric encryption and decryption. One example is [gpg](https://www.gnupg.org/gph/en/manual/x110.html).
-   It writes it to the configuration file (in encrypted format) and deletes the secrets file `/mnt/init-secret/secrets` (or makes is empty). The startup script then starts the application by passing the passphrase to it.
-   The application then reads the encrypted database password from configuration file, decrypts it using the provided passphrase, and uses it to connect to the database.

**Note:** Make sure the application container has permissions to delete the `/mnt/init-secret/secrets` file created by init container. 

## What have we achieved with this approach?
-   The application now supports using encrypted sensitive data implicitly.
-   No password or passphrase is written to anywhere on the disk or available as environment variable.
-   Init container that process the mounted OpenShift secrets is not accessible to read the files from the secret mount path once it's execution completes.
-   Application container reads the password and passphrase, and deletes the file from the shared volume (in-memory) so that even if someone logs into the application container, the secret file containing password and passphrase is not accessible. 

This is one of the possible ways to securely consume OpenShift secrets in the application pods.