<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: Deyan7 GmbH &amp; Co.KG</title>
    <description>The latest articles on DEV Community by Deyan7 GmbH &amp; Co.KG (@deyan7).</description>
    <link>https://dev.to/deyan7</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F1011292%2F67c98eba-5873-4de6-8c05-4b009fef1029.png</url>
      <title>DEV Community: Deyan7 GmbH &amp; Co.KG</title>
      <link>https://dev.to/deyan7</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/deyan7"/>
    <language>en</language>
    <item>
      <title>SOPS (Secrets OPerationS - Kubernetes Operator): Secure your sensitive data, while maintaining ease of use</title>
      <dc:creator>Deyan7 GmbH &amp; Co.KG</dc:creator>
      <pubDate>Thu, 23 Feb 2023 09:14:01 +0000</pubDate>
      <link>https://dev.to/deyan7/sops-secrets-operations-kubernetes-operator-secure-your-sensitive-data-while-maintaining-ease-of-use-8el</link>
      <guid>https://dev.to/deyan7/sops-secrets-operations-kubernetes-operator-secure-your-sensitive-data-while-maintaining-ease-of-use-8el</guid>
      <description>&lt;h2&gt;
  
  
  Intention / Goal
&lt;/h2&gt;

&lt;p&gt;When using an Infrastructure-as-Code approach to populate your Kubernetes cluster, e.g. with &lt;a href="https://www.terraform.io" rel="noopener noreferrer"&gt;Terraform&lt;/a&gt;, a question is how secrets are handled that need to be injected into the cluster (The words secrets, sensistive or confidential data are used interchangeably here). An example would be a database password which is needed by a backend instance to be able to operate.&lt;br&gt;&lt;br&gt;
Usually the IaC code is stored in a remote git repository (e.g. on GitHub or GitLab), which could potentially leak the password to the public when the password is stored as plain text in code. Even if your git repository is not publicly available, it will drastically multiply the radius of your secrets.&lt;br&gt;&lt;br&gt;
Additionally to the git problem, Terraform also uses a State file where it keeps track of the deployed resources. Kubernetes Secrets will also be included, hence the State file will leak your sensitive data, too, when remote state is used (Storing the state e.g. in GitLab or Amazon S3).&lt;/p&gt;

&lt;p&gt;Finding the right solution to keep your unencrypted secrets out of the git repo (and out of any other systems than the cluster itself) can be a tediuous research process.  &lt;/p&gt;

&lt;p&gt;In this article we will present a solution, storing secrets as encrypted data in your git repo and only decrypting it inside of your Kubernetes cluster. We have test-driven this approach under various circumstances and are convinced that it is one of the best solutions for most applications.&lt;/p&gt;

&lt;p&gt;Only a small subset of your team, like some Site Realiability Engineers should be able to decrypt the credentials locally.&lt;/p&gt;
&lt;h2&gt;
  
  
  SOPS
&lt;/h2&gt;
&lt;h3&gt;
  
  
  &lt;a&gt;&lt;/a&gt;How does SOPS(-Operator) work from a meta perspective:
&lt;/h3&gt;

&lt;p&gt;SOPS is used to encrypt / decrypt data of type YAML, JSON, ENV, INI and BINARY (source: &lt;a href="https://github.com/mozilla/sops#sops-secrets-operations" rel="noopener noreferrer"&gt;Mozilla sops Github&lt;/a&gt;) by using mechanisms like AWS KMS, GCP KMS, Azure Key Vault, age, and PGP. The &lt;a href="https://github.com/isindir/sops-secrets-operator#sops-secrets-operations---kubernetes-operator" rel="noopener noreferrer"&gt;SOPS-Operator&lt;/a&gt; was inspired by this mechanism and extends it into Kubernetes clusters.  &lt;/p&gt;

&lt;p&gt;SOPS Operator takes a Kubernetes Custom Resource Definition (CRD) called &lt;code&gt;SopsSecret&lt;/code&gt; as input, whose sensitive components are encrypted, decrypts these and creates (updates, deletes) usual Kubernetes &lt;code&gt;Secret&lt;/code&gt; Resources. These resources in turn can be consumed by your Kubernetes entities (like Pods). &lt;/p&gt;

&lt;p&gt;Here we will use &lt;code&gt;age&lt;/code&gt; for encryption and decryption (see Age Section). How these CRDs look like and how they are encrypted / decrypted will be explained in How to / Step-by-Step Guide.&lt;/p&gt;
&lt;h3&gt;
  
  
  Why SOPS(-Operator)?
&lt;/h3&gt;

&lt;p&gt;You might ask yourself: why do I need to encrypt my secrets / sensitive data, when everything is stored in a private repo and it is not planned to change anything here anytime soon?&lt;br&gt;&lt;br&gt;
Also, not the entire team is allowed to access this repo, but just a small part (e.g. SREs). Who really needs access to these files? A look in the past, where incidents did occur (&lt;a href="https://blog.adafruit.com/2022/03/04/a-github-repository-was-public-viewable/" rel="noopener noreferrer"&gt;Adafruit&lt;/a&gt;, &lt;a href="https://www.bleepingcomputer.com/news/security/slacks-private-github-code-repositories-stolen-over-holidays/" rel="noopener noreferrer"&gt;Slack&lt;/a&gt;, &lt;a href="https://rewind.com/blog/is-github-still-safe-to-use/" rel="noopener noreferrer"&gt;Github&lt;/a&gt;), shows that you should best keep your attack surface small.&lt;/p&gt;

&lt;p&gt;By using SOPS-Operator (Secrets OPerationS - Kubernetes Operator), we add one more layer of security and also one more fine-grained control to our system. In addition we are able to hide sensitive information / secrets from cloud providers. Still everybody should be able to access a source code repository and use it normally.&lt;br&gt;&lt;br&gt;
&lt;strong&gt;Warning:&lt;/strong&gt; Keep in mind that persons with cluster access can still (if not further restricted) list the clusters secrets and use them. &lt;/p&gt;
&lt;h3&gt;
  
  
  What encryption to use with SOPS?
&lt;/h3&gt;

&lt;p&gt;&lt;code&gt;AWS KMS&lt;/code&gt;, &lt;code&gt;GCP KMS&lt;/code&gt;, &lt;code&gt;Azure Key Vault&lt;/code&gt; can be used with &lt;code&gt;sops&lt;/code&gt;, but they are not open source like &lt;code&gt;age&lt;/code&gt; and tie your implementation to one provider only. Additionally, secrets in Vaults are handled centrally and do not live alongside your code. Hence, under some circumstances manipulations in the Vault need to be synced with code deployment. This is not a problem when using file-based encryption.&lt;br&gt;&lt;br&gt;
And, last but not least, Vaults usually do cost a small amount of money while file-based encryption is for free.&lt;/p&gt;

&lt;p&gt;So &lt;code&gt;GPG&lt;/code&gt; and &lt;code&gt;age&lt;/code&gt; remain. Whereas in the past GPG was used to encrypt the entire repo and distribute the keys of each team member, it misses ease of use or at least it has room for improvement. GPG is a mature solution and is still widely used. But it is not very lightweight, due to the possibility to use it in a lot of scenarios like mail etc. &lt;/p&gt;
&lt;h4&gt;
  
  
  Flaws in / with GPG
&lt;/h4&gt;

&lt;p&gt;GPG is the open-source alternative of Symantec´s PGP and has been on the market for more than 20 years. It is used on a variety of topics, and hence is prone to be being targeted. Of course GPG is constantly improved and secured, but however, the following security vulnerabilities exist(ed): &lt;a href="https://www.cvedetails.com/vulnerability-list/vendor_id-4711/Gnupg.html" rel="noopener noreferrer"&gt;Gnupg&lt;/a&gt;, &lt;a href="https://www.cvedetails.com/vulnerability-list/vendor_id-76/product_id-24665/Symantec-Pgp-Desktop.html" rel="noopener noreferrer"&gt;Symantec PGP Desktop&lt;/a&gt;, &lt;a href="https://www.cvedetails.com/vulnerability-list/vendor_id-76/product_id-23042/Symantec-Pgp-Universal-Server.html" rel="noopener noreferrer"&gt;Symantec PGP Universal Server&lt;/a&gt;. &lt;/p&gt;
&lt;h4&gt;
  
  
  &lt;a&gt;&lt;/a&gt;AGE
&lt;/h4&gt;

&lt;p&gt;Along comes &lt;a href="https://github.com/FiloSottile/age/" rel="noopener noreferrer"&gt;age&lt;/a&gt;, which is written in GO and was invented by a Google Engineer in &lt;a href="https://github.com/FiloSottile/age/commit/06cbe4f91ea984306996d1f7dbde1bb5ffd67fec" rel="noopener noreferrer"&gt;2019&lt;/a&gt;. It tries to learn from past mistakes of other tools like GPG. At the same time, it tries to be very compact, lightweight and concise. This keeps the attack surface as small as possible. Hereby, it follows the typical UNIX approach solving only one single problem.&lt;br&gt;
Age is a secure process for encrypting / decrypting any data regardless of the filetype.&lt;/p&gt;

&lt;p&gt;To support its credibility, &lt;a href="https://github.com/mozilla/sops#22encrypting-using-age" rel="noopener noreferrer"&gt;Mozilla&lt;/a&gt; recommends AGE over PGP.&lt;/p&gt;

&lt;p&gt;Of the two available encryption approaches for age, asymmetric encryption based on X25519 and passphrase encryption type based on scrypt (see &lt;a href="https://github.com/C2SP/C2SP/blob/main/age.md" rel="noopener noreferrer"&gt;AGE Specification&lt;/a&gt;) are offered. For SOPS asymmetric encryption via &lt;code&gt;X25519&lt;/code&gt; is used, which is a mechanism that can also be choosen with e.g. &lt;code&gt;ssh&lt;/code&gt;. &lt;/p&gt;

&lt;p&gt;First of all a key-pair is need, which can be created with &lt;a href="https://htmlpreview.github.io/?https://raw.githubusercontent.com/FiloSottile/age/main/doc/age-keygen.1.html" rel="noopener noreferrer"&gt;age-keygen&lt;/a&gt; (comparable to ssh-keygen) that creates a public and private key pair.&lt;/p&gt;

&lt;p&gt;In Age the private key is called &lt;code&gt;Identity&lt;/code&gt;. It allows to decrypt a file encrypted to its corresponding &lt;code&gt;Recipient&lt;/code&gt;. The &lt;code&gt;Recipient&lt;/code&gt; is like the public key of ssh that files can be encrypted to (see:&lt;br&gt;
&lt;a href="https://htmlpreview.github.io/?https://github.com/FiloSottile/age/blob/main/doc/age.1.html" rel="noopener noreferrer"&gt;age man page&lt;/a&gt;)&lt;/p&gt;

&lt;p&gt;The Identity / private key starts with &lt;code&gt;AGE-SECRET-KEY-&lt;/code&gt; and must be kept secret / private, like your ssh private key.&lt;br&gt;&lt;br&gt;
The counterpart is the &lt;code&gt;Recipient&lt;/code&gt; / public key that is written as a comment line in the file generated by &lt;code&gt;age-keygen&lt;/code&gt;. It starts with '&lt;code&gt;age&lt;/code&gt;' and defines where a file is encrypted to. &lt;/p&gt;

&lt;p&gt;
&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdeyan7.de%2Fwp-content%2Fuploads%2F2023%2F02%2Fsecrets_subdivision-1024x1012.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdeyan7.de%2Fwp-content%2Fuploads%2F2023%2F02%2Fsecrets_subdivision-1024x1012.png" width="800" height="400"&gt;&lt;/a&gt;
&lt;/p&gt;

&lt;p&gt;The Password Vault, as shown in the image could be 1Password or similar. It is good practive that users should store the created key-pair file / AGE-SECRET-KEY in a password vault. In this example only an admin team is allowed to access the AGE Secret key file in an admin vault (blue rectangle) and other means to access the cluster itself. &lt;/p&gt;

&lt;p&gt;Now &lt;code&gt;SOPS&lt;/code&gt; / &lt;code&gt;SOPS-Operator&lt;/code&gt; come into play. As explained in How does SOPS(-Operator) work from a meta perspective, SOPS-Operator is inspired by SOPS and is used to manage &lt;a href="https://kubernetes.io/docs/concepts/configuration/secret/" rel="noopener noreferrer"&gt;Kubernetes Secret Resources&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;You first need to encrypt your sensitive information locally, like shown in the picture, using the &lt;code&gt;AGE recipient Public Key&lt;/code&gt; and the unencrypted &lt;code&gt;SopsSecret&lt;/code&gt; as input to &lt;code&gt;sops&lt;/code&gt;. Sops hands this to the AGE process, which outputs the encrypted SopsSecret. The encrypted SopsSecret must then be supplied to your cluster.&lt;/p&gt;

&lt;p&gt;
&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdeyan7.de%2Fwp-content%2Fuploads%2F2023%2F02%2Fsops_encrypt-563x1024.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdeyan7.de%2Fwp-content%2Fuploads%2F2023%2F02%2Fsops_encrypt-563x1024.png" width="" height=""&gt;&lt;/a&gt;
&lt;/p&gt;

&lt;p&gt;If you receive a SopsSecret with encrypted keys, the decryption flow is quite similar, but instead of the public key the Identity of the AGE-SECRET-KEY file is used.&lt;/p&gt;

&lt;p&gt;
&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdeyan7.de%2Fwp-content%2Fuploads%2F2023%2F02%2Fsops_decrypt-779x1024.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdeyan7.de%2Fwp-content%2Fuploads%2F2023%2F02%2Fsops_decrypt-779x1024.png" width="800" height="400"&gt;&lt;/a&gt;
&lt;/p&gt;

&lt;p&gt;
&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdeyan7.de%2Fwp-content%2Fuploads%2F2023%2F02%2Fsops-operator.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdeyan7.de%2Fwp-content%2Fuploads%2F2023%2F02%2Fsops-operator.png" width="800" height="400"&gt;&lt;/a&gt;
&lt;/p&gt;

&lt;p&gt;When we look at the cluster (see image above), the process is comparable. The cluster receives an encrypted &lt;code&gt;SopsSecret&lt;/code&gt; Custom Resource Definition as input (see top left of the image). This &lt;code&gt;SopsSecret&lt;/code&gt; CRD is created locally (it can be written as &lt;code&gt;kubernetes_manifest&lt;/code&gt; with terraform). It has the same structure as usual Kubernetes Secret Resources. The values are the plain secrets, which are in the next step encrypted using SOPS. This encrypted manifest file is then applied to your cluster, via kubectl or e.g. terraform (it has to be converted to a terraform &lt;code&gt;kubernetes_manifest&lt;/code&gt; Resource first in this case). Afterwards it is available in your cluster and the &lt;code&gt;sops-secrets-operator&lt;/code&gt; Pod listens to state change (new or adjusted &lt;code&gt;SopsSecret&lt;/code&gt;-Crd). It takes this CRD together with the &lt;code&gt;sops-age-key-file&lt;/code&gt; Secret as input and decrypts the encrypted values via age, using the &lt;code&gt;Identity&lt;/code&gt; from the &lt;code&gt;sops-age-key-file&lt;/code&gt;. As a next step it creates a Kubernetes Secret from this SopsSecret CRD, so it can be consumed by other pods.&lt;/p&gt;

&lt;p&gt;Data is hereby decrypted as close as possible to the resource that is using it (like the database password of your PostgreSQL database or similar). This resource does not need to know anything about SOPS, since it can consume the password via the Kubernetes Secret resource, as usual.&lt;br&gt;&lt;br&gt;
We now have established a solution that does not carry sensitive, plain secrets in any parts of the system where they shouldn´t be.&lt;/p&gt;

&lt;p&gt;
&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdeyan7.de%2Fwp-content%2Fuploads%2F2023%2F02%2Fci_cd-530x1024.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdeyan7.de%2Fwp-content%2Fuploads%2F2023%2F02%2Fci_cd-530x1024.png" width="800" height="400"&gt;&lt;/a&gt;
&lt;/p&gt;

&lt;p&gt;A typical workflow could look like it can be seen in the image above. You locally create an encrypted SopsSecret and push this to your git remote repository like Gitlab or Github. The pipeline picks up your changes and deploys this secret to your cluster.&lt;/p&gt;

&lt;h2&gt;
  
  
  &lt;a&gt;&lt;/a&gt;How to / Step-by-Step Guide
&lt;/h2&gt;

&lt;h3&gt;
  
  
  Prepare your cluster and install SOPS-Operator
&lt;/h3&gt;

&lt;ol&gt;
&lt;li&gt;Encode your secret in base64 format: &lt;code&gt;sed -n -e '/^AGE-SECRET-KEY/p' PATH_TO_YOUR_AGE_KEY_FILE | base64&lt;/code&gt;
&lt;/li&gt;
&lt;li&gt;Supply a secret (age identity) to your cluster (Needed to decrypt the SopsSecret CRDs)
&lt;/li&gt;
&lt;/ol&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nb"&gt;cat&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;&amp;lt;&lt;/span&gt;&lt;span class="no"&gt;EOF&lt;/span&gt;&lt;span class="sh"&gt; | kubectl apply -f -
apiVersion: v1
data:
  key: YOUR_BASE64_ENCODED_IDENTITY_PRIVATE_KEY
kind: Secret
metadata:
  name: sops-age-key-file
  namespace: YOUR_NAMESPACE
type: Opaque
&lt;/span&gt;&lt;span class="no"&gt;EOF
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Here we do not use Terraform on purpose in order to not make the secret part of your terraform state, as this state would leak your secrets (You can also generate a &lt;code&gt;kubernetes_secret&lt;/code&gt;, but some regulatory might force you to not leak this secret to your cloud provider).&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Add the &lt;a href="https://github.com/isindir/sops-secrets-operator#aws" rel="noopener noreferrer"&gt;sops-secrets-operator&lt;/a&gt; to your cluster, using &lt;code&gt;helm&lt;/code&gt; (Alternative below):
&lt;/li&gt;
&lt;/ol&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;
helm repo add sops https://isindir.github.io/sops-secrets-operator/
helm upgrade &lt;span class="nt"&gt;--install&lt;/span&gt; &lt;span class="nt"&gt;--create-namespace&lt;/span&gt; sops sops/sops-secrets-operator &lt;span class="nt"&gt;--namespace&lt;/span&gt; YOUR_NAMESPACE
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Or use e.g. Terraform:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight terraform"&gt;&lt;code&gt;&lt;span class="k"&gt;resource&lt;/span&gt; &lt;span class="s2"&gt;"helm_release"&lt;/span&gt; &lt;span class="s2"&gt;"sops-secrets-operator"&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="nx"&gt;name&lt;/span&gt;             &lt;span class="p"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;"sops-secrets-operator"&lt;/span&gt;
  &lt;span class="nx"&gt;chart&lt;/span&gt;            &lt;span class="p"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;"sops-secrets-operator"&lt;/span&gt;
  &lt;span class="nx"&gt;repository&lt;/span&gt;       &lt;span class="p"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;"https://isindir.github.io/sops-secrets-operator/"&lt;/span&gt;
  &lt;span class="nx"&gt;namespace&lt;/span&gt;        &lt;span class="p"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;"YOUR_NAMESPACE"&lt;/span&gt;
  &lt;span class="nx"&gt;version&lt;/span&gt;          &lt;span class="p"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;"0.14.0"&lt;/span&gt;
  &lt;span class="nx"&gt;create_namespace&lt;/span&gt; &lt;span class="p"&gt;=&lt;/span&gt; &lt;span class="kc"&gt;true&lt;/span&gt;

  &lt;span class="nx"&gt;values&lt;/span&gt; &lt;span class="p"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;
    &lt;span class="o"&gt;&amp;lt;&amp;lt;&lt;/span&gt;&lt;span class="no"&gt;EOF&lt;/span&gt;&lt;span class="sh"&gt;
extraEnv:
- name: SOPS_AGE_KEY_FILE
  value: /etc/sops-age-key-file/key
secretsAsFiles:
- mountPath: /etc/sops-age-key-file
  name: sops-age-key-file
  secretName: sops-age-key-file
&lt;/span&gt;&lt;span class="no"&gt;EOF
&lt;/span&gt;  &lt;span class="p"&gt;]&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h3&gt;
  
  
  Create SopsSecret &amp;amp; Deploy it to your cluster
&lt;/h3&gt;

&lt;ol&gt;
&lt;li&gt;Install &lt;code&gt;sops&lt;/code&gt;, &lt;code&gt;age&lt;/code&gt; and optionally &lt;code&gt;tfk8s&lt;/code&gt;, if you want to use terraform, via a package manager. Here we use brew:
&lt;/li&gt;
&lt;/ol&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;brew &lt;span class="nb"&gt;install&lt;/span&gt; &lt;span class="se"&gt;\&lt;/span&gt;
sops &lt;span class="se"&gt;\&lt;/span&gt;
age &lt;span class="se"&gt;\&lt;/span&gt;
tk8s
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;ol&gt;
&lt;li&gt;Generate a public-private-keypair: &lt;code&gt;age-keygen -o age-key.txt&lt;/code&gt;. [&lt;strong&gt;Optional&lt;/strong&gt;: Store the age-key.txt in a password vault, like 1Password. We highly recommend to store it somewhere save.]

&lt;ol&gt;
&lt;li&gt;Encrypt your secrets that should be used in your cluster
&lt;/li&gt;
&lt;/ol&gt;


&lt;/li&gt;

&lt;/ol&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;sops &lt;span class="nt"&gt;--encrypt&lt;/span&gt; &lt;span class="nt"&gt;--age&lt;/span&gt; &lt;span class="s1"&gt;'YOUR_AGE_RECIPIENT_PUBLIC_KEY_STARTING_WITH_age'&lt;/span&gt; &lt;span class="nt"&gt;--encrypted-suffix&lt;/span&gt; Templates SOPS_SECRET_FILE_YOU_WANT_TO_ENCRYPT.yml &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; SOPS_SECRET_FILE_ENCRYPTED.yml.enc
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;





&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight yaml"&gt;&lt;code&gt;
&lt;span class="na"&gt;apiVersion&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;isindir.github.com/v1alpha3&lt;/span&gt;
&lt;span class="na"&gt;kind&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;SopsSecret&lt;/span&gt;
&lt;span class="na"&gt;metadata&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;your-secrets-name&lt;/span&gt;
  &lt;span class="na"&gt;namespace&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;YOUR_NAMESPACE&lt;/span&gt;
&lt;span class="na"&gt;spec&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;secretTemplates&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
    &lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;your-secrets-name&lt;/span&gt;
      &lt;span class="na"&gt;stringData&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
         &lt;span class="na"&gt;NAME_OF_YOUR_SECRET&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;SECRET_ITSELF&lt;/span&gt;
         &lt;span class="na"&gt;NAME_OF_ANOTHER_SECRET&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;SECRET_ITSELF&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;ol&gt;
&lt;li&gt;Deploy this SopsSecret to your infrastructure via
&lt;/li&gt;
&lt;/ol&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;  kubectl apply &lt;span class="nt"&gt;-f&lt;/span&gt; SOPS_SECRET_FILE_ENCRYPTED.enc.yml
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;or if you want to use terraform, you can convert it with&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nb"&gt;cat &lt;/span&gt;SOPS_SECRET_FILE_ENCRYPTED.yml.enc | tfk8s &lt;span class="nt"&gt;--strip&lt;/span&gt; &lt;span class="nt"&gt;-o&lt;/span&gt; your-secrets-name.tf
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;This produces a &lt;a href="https://registry.terraform.io/providers/hashicorp/kubernetes/latest/docs/resources/manifest" rel="noopener noreferrer"&gt;kubernetes_manifest resource&lt;/a&gt;. Afterwards you can use &lt;code&gt;terraform plan&lt;/code&gt; and &lt;code&gt;terraform apply&lt;/code&gt; as you would normally do or use your CI / CD pipeline to do so.&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;With the following Terraform datasource you are able to reference the secrets in other resources. Watch out for the values of your keys in the binary_data, they should be empty as they are populated by terraform itfself, without storing these inside your terraform state file .
&lt;/li&gt;
&lt;/ol&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight terraform"&gt;&lt;code&gt;&lt;span class="k"&gt;data&lt;/span&gt; &lt;span class="s2"&gt;"kubernetes_secret"&lt;/span&gt; &lt;span class="s2"&gt;"your-secrets-name"&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="nx"&gt;metadata&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="nx"&gt;name&lt;/span&gt;      &lt;span class="p"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;"your-secrets-name"&lt;/span&gt;
    &lt;span class="nx"&gt;namespace&lt;/span&gt; &lt;span class="p"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;"YOUR_NAMESPACE"&lt;/span&gt;
  &lt;span class="p"&gt;}&lt;/span&gt;

  &lt;span class="nx"&gt;binary_data&lt;/span&gt; &lt;span class="p"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="nx"&gt;NAME_OF_YOUR_SECRET&lt;/span&gt;                               &lt;span class="p"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;""&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;You can e.g. define an output from it:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight terraform"&gt;&lt;code&gt;&lt;span class="k"&gt;output&lt;/span&gt; &lt;span class="s2"&gt;"NAME_OF_YOUR_SECRET"&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="nx"&gt;value&lt;/span&gt;       &lt;span class="p"&gt;=&lt;/span&gt; &lt;span class="nx"&gt;base64decode&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="k"&gt;data&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;kubernetes_secret&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;your-secrets-name&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;binary_data&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;NAME_OF_YOUR_SECRET&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
  &lt;span class="nx"&gt;sensitive&lt;/span&gt;   &lt;span class="p"&gt;=&lt;/span&gt; &lt;span class="kc"&gt;true&lt;/span&gt;
  &lt;span class="nx"&gt;description&lt;/span&gt; &lt;span class="p"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;"FooBar"&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;and use this output in any other module &lt;code&gt;module.secrets.NAME_OF_YOUR_SECRET&lt;/code&gt;.&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Check if everything went fine, by checking if the Kubernetes Secret Resource was created:
&lt;code&gt;kubectl get secrets your-secrets-name -n YOUR_NAMESPACE -o yaml&lt;/code&gt; .
If the output is &lt;code&gt;No resources found in YOUR_NAMESPACE namespace.&lt;/code&gt;, something went wrong. So best is to consult the logs of the operator first:
&lt;code&gt;kubectl logs  -l  app.kubernetes.io/name=sops-secrets-operator -n YOUR_NAMESPACE&lt;/code&gt;
&lt;/li&gt;
&lt;/ol&gt;

&lt;h3&gt;
  
  
  Decrypt and edit SopsSecret
&lt;/h3&gt;

&lt;ol&gt;
&lt;li&gt;If you want to edit existing secrets that one of your colleagues did encrypt, then get the secret identity key file from your teams password vault.&lt;/li&gt;
&lt;li&gt;Specify the location of your key file:  &lt;code&gt;export SOPS_AGE_KEY_FILE=./key.txt&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;Decrypt the file &lt;code&gt;sops -d SOPS_SECRET_FILE_ENCRYPTED.yml.enc&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;Edit the contents.&lt;/li&gt;
&lt;li&gt;Repeat the steps to encrypt and deploy the secret again.&lt;/li&gt;
&lt;/ol&gt;

&lt;h2&gt;
  
  
  Conclusion
&lt;/h2&gt;

&lt;p&gt;Handling secrets in a safe way is one of the hardest tasks in software development.&lt;br&gt;&lt;br&gt;
Using &lt;code&gt;SOPS Operator&lt;/code&gt; with &lt;code&gt;age&lt;/code&gt; allows you to completely free your code versioning repositories, CICD pipelines and other systems involved in the deployment of Kubernetes clusters from secrets.&lt;br&gt;&lt;br&gt;
Our approach still uses whichever workflow you are used to deploy and does not interfer with any tooling you might be using for these steps. The usual &lt;code&gt;Kubernetes Secret&lt;/code&gt; resources can be used within the cluster, enabling a high compatibility with existing mechanisms.&lt;/p&gt;

</description>
      <category>tooling</category>
    </item>
    <item>
      <title>SOPS (Secrets OPerationS – Kubernetes Operator): Secure your sensitive data, while maintaining ease of use</title>
      <dc:creator>Deyan7 GmbH &amp; Co.KG</dc:creator>
      <pubDate>Wed, 22 Feb 2023 13:37:14 +0000</pubDate>
      <link>https://dev.to/deyan7/sops-secrets-operations-kubernetes-operator-secure-your-sensitive-data-while-maintaining-ease-of-use-2j55</link>
      <guid>https://dev.to/deyan7/sops-secrets-operations-kubernetes-operator-secure-your-sensitive-data-while-maintaining-ease-of-use-2j55</guid>
      <description>&lt;h2&gt;
  
  
  Intention / Goal
&lt;/h2&gt;

&lt;p&gt;When using an Infrastructure-as-Code approach to populate your Kubernetes cluster, e.g. with &lt;a href="https://www.terraform.io"&gt;Terraform&lt;/a&gt;, a question is how secrets are handled that need to be injected into the cluster (The words secrets, sensistive or confidential data are used interchangeably here). An example would be a database password which is needed by a backend instance to be able to operate.&lt;br&gt;&lt;br&gt;
Usually the IaC code is stored in a remote git repository (e.g. on GitHub or GitLab), which could potentially leak the password to the public when the password is stored as plain text in code. Even if your git repository is not publicly available, it will drastically multiply the radius of your secrets.&lt;br&gt;&lt;br&gt;
Additionally to the git problem, Terraform also uses a State file where it keeps track of the deployed resources. Kubernetes Secrets will also be included, hence the State file will leak your sensitive data, too, when remote state is used (Storing the state e.g. in GitLab or Amazon S3).&lt;/p&gt;

&lt;p&gt;Finding the right solution to keep your unencrypted secrets out of the git repo (and out of any other systems than the cluster itself) can be a tediuous research process.  &lt;/p&gt;

&lt;p&gt;In this article we will present a solution, storing secrets as encrypted data in your git repo and only decrypting it inside of your Kubernetes cluster. We have test-driven this approach under various circumstances and are convinced that it is one of the best solutions for most applications.&lt;/p&gt;

&lt;p&gt;Only a small subset of your team, like some Site Realiability Engineers should be able to decrypt the credentials locally.&lt;/p&gt;
&lt;h2&gt;
  
  
  SOPS
&lt;/h2&gt;
&lt;h3&gt;
  
  
  &lt;a&gt;&lt;/a&gt;How does SOPS(-Operator) work from a meta perspective:
&lt;/h3&gt;

&lt;p&gt;SOPS is used to encrypt / decrypt data of type YAML, JSON, ENV, INI and BINARY (source: &lt;a href="https://github.com/mozilla/sops#sops-secrets-operations"&gt;Mozilla sops Github&lt;/a&gt;) by using mechanisms like AWS KMS, GCP KMS, Azure Key Vault, age, and PGP. The &lt;a href="https://github.com/isindir/sops-secrets-operator#sops-secrets-operations---kubernetes-operator"&gt;SOPS-Operator&lt;/a&gt; was inspired by this mechanism and extends it into Kubernetes clusters.  &lt;/p&gt;

&lt;p&gt;SOPS Operator takes a Kubernetes Custom Resource Definition (CRD) called &lt;code&gt;SopsSecret&lt;/code&gt; as input, whose sensitive components are encrypted, decrypts these and creates (updates, deletes) usual Kubernetes &lt;code&gt;Secret&lt;/code&gt; Resources. These resources in turn can be consumed by your Kubernetes entities (like Pods). &lt;/p&gt;

&lt;p&gt;Here we will use &lt;code&gt;age&lt;/code&gt; for encryption and decryption (see Age Section). How these CRDs look like and how they are encrypted / decrypted will be explained in How to / Step-by-Step Guide.&lt;/p&gt;
&lt;h3&gt;
  
  
  Why SOPS(-Operator)?
&lt;/h3&gt;

&lt;p&gt;You might ask yourself: why do I need to encrypt my secrets / sensitive data, when everything is stored in a private repo and it is not planned to change anything here anytime soon?&lt;br&gt;&lt;br&gt;
Also, not the entire team is allowed to access this repo, but just a small part (e.g. SREs). Who really needs access to these files? A look in the past, where incidents did occur (&lt;a href="https://blog.adafruit.com/2022/03/04/a-github-repository-was-public-viewable/"&gt;Adafruit&lt;/a&gt;, &lt;a href="https://www.bleepingcomputer.com/news/security/slacks-private-github-code-repositories-stolen-over-holidays/"&gt;Slack&lt;/a&gt;, &lt;a href="https://rewind.com/blog/is-github-still-safe-to-use/"&gt;Github&lt;/a&gt;), shows that you should best keep your attack surface small.&lt;/p&gt;

&lt;p&gt;By using SOPS-Operator (Secrets OPerationS - Kubernetes Operator), we add one more layer of security and also one more fine-grained control to our system. In addition we are able to hide sensitive information / secrets from cloud providers. Still everybody should be able to access a source code repository and use it normally.&lt;br&gt;&lt;br&gt;
&lt;strong&gt;Warning:&lt;/strong&gt; Keep in mind that persons with cluster access can still (if not further restricted) list the clusters secrets and use them. &lt;/p&gt;
&lt;h3&gt;
  
  
  What encryption to use with SOPS?
&lt;/h3&gt;

&lt;p&gt;&lt;code&gt;AWS KMS&lt;/code&gt;, &lt;code&gt;GCP KMS&lt;/code&gt;, &lt;code&gt;Azure Key Vault&lt;/code&gt; can be used with &lt;code&gt;sops&lt;/code&gt;, but they are not open source like &lt;code&gt;age&lt;/code&gt; and tie your implementation to one provider only. Additionally, secrets in Vaults are handled centrally and do not live alongside your code. Hence, under some circumstances manipulations in the Vault need to be synced with code deployment. This is not a problem when using file-based encryption.&lt;br&gt;&lt;br&gt;
And, last but not least, Vaults usually do cost a small amount of money while file-based encryption is for free.&lt;/p&gt;

&lt;p&gt;So &lt;code&gt;GPG&lt;/code&gt; and &lt;code&gt;age&lt;/code&gt; remain. Whereas in the past GPG was used to encrypt the entire repo and distribute the keys of each team member, it misses ease of use or at least it has room for improvement. GPG is a mature solution and is still widely used. But it is not very lightweight, due to the possibility to use it in a lot of scenarios like mail etc. &lt;/p&gt;
&lt;h4&gt;
  
  
  Flaws in / with GPG
&lt;/h4&gt;

&lt;p&gt;GPG is the open-source alternative of Symantec´s PGP and has been on the market for more than 20 years. It is used on a variety of topics, and hence is prone to be being targeted. Of course GPG is constantly improved and secured, but however, the following security vulnerabilities exist(ed): &lt;a href="https://www.cvedetails.com/vulnerability-list/vendor_id-4711/Gnupg.html"&gt;Gnupg&lt;/a&gt;, &lt;a href="https://www.cvedetails.com/vulnerability-list/vendor_id-76/product_id-24665/Symantec-Pgp-Desktop.html"&gt;Symantec PGP Desktop&lt;/a&gt;, &lt;a href="https://www.cvedetails.com/vulnerability-list/vendor_id-76/product_id-23042/Symantec-Pgp-Universal-Server.html"&gt;Symantec PGP Universal Server&lt;/a&gt;. &lt;/p&gt;
&lt;h4&gt;
  
  
  &lt;a&gt;&lt;/a&gt;AGE
&lt;/h4&gt;

&lt;p&gt;Along comes &lt;a href="https://github.com/FiloSottile/age/"&gt;age&lt;/a&gt;, which is written in GO and was invented by a Google Engineer in &lt;a href="https://github.com/FiloSottile/age/commit/06cbe4f91ea984306996d1f7dbde1bb5ffd67fec"&gt;2019&lt;/a&gt;. It tries to learn from past mistakes of other tools like GPG. At the same time, it tries to be very compact, lightweight and concise. This keeps the attack surface as small as possible. Hereby, it follows the typical UNIX approach solving only one single problem.&lt;br&gt;
Age is a secure process for encrypting / decrypting any data regardless of the filetype.&lt;/p&gt;

&lt;p&gt;To support its credibility, &lt;a href="https://github.com/mozilla/sops#22encrypting-using-age"&gt;Mozilla&lt;/a&gt; recommends AGE over PGP.&lt;/p&gt;

&lt;p&gt;Of the two available encryption approaches for age, asymmetric encryption based on X25519 and passphrase encryption type based on scrypt (see &lt;a href="https://github.com/C2SP/C2SP/blob/main/age.md"&gt;AGE Specification&lt;/a&gt;) are offered. For SOPS asymmetric encryption via &lt;code&gt;X25519&lt;/code&gt; is used, which is a mechanism that can also be choosen with e.g. &lt;code&gt;ssh&lt;/code&gt;. &lt;/p&gt;

&lt;p&gt;First of all a key-pair is need, which can be created with &lt;a href="https://htmlpreview.github.io/?https://raw.githubusercontent.com/FiloSottile/age/main/doc/age-keygen.1.html"&gt;age-keygen&lt;/a&gt; (comparable to ssh-keygen) that creates a public and private key pair.&lt;/p&gt;

&lt;p&gt;In Age the private key is called &lt;code&gt;Identity&lt;/code&gt;. It allows to decrypt a file encrypted to its corresponding &lt;code&gt;Recipient&lt;/code&gt;. The &lt;code&gt;Recipient&lt;/code&gt; is like the public key of ssh that files can be encrypted to (see:&lt;br&gt;
&lt;a href="https://htmlpreview.github.io/?https://github.com/FiloSottile/age/blob/main/doc/age.1.html"&gt;age man page&lt;/a&gt;)&lt;/p&gt;

&lt;p&gt;The Identity / private key starts with &lt;code&gt;AGE-SECRET-KEY-&lt;/code&gt; and must be kept secret / private, like your ssh private key.&lt;br&gt;&lt;br&gt;
The counterpart is the &lt;code&gt;Recipient&lt;/code&gt; / public key that is written as a comment line in the file generated by &lt;code&gt;age-keygen&lt;/code&gt;. It starts with '&lt;code&gt;age&lt;/code&gt;' and defines where a file is encrypted to. &lt;/p&gt;

&lt;p&gt;
&lt;a href="https://res.cloudinary.com/practicaldev/image/fetch/s--frmh0PJb--/c_limit%2Cf_auto%2Cfl_progressive%2Cq_auto%2Cw_880/https://deyan7.de/wp-content/uploads/2023/02/secrets_subdivision-1024x1012.png" class="article-body-image-wrapper"&gt;&lt;img src="https://res.cloudinary.com/practicaldev/image/fetch/s--frmh0PJb--/c_limit%2Cf_auto%2Cfl_progressive%2Cq_auto%2Cw_880/https://deyan7.de/wp-content/uploads/2023/02/secrets_subdivision-1024x1012.png" width="880" height="870"&gt;&lt;/a&gt;
&lt;/p&gt;

&lt;p&gt;The Password Vault, as shown in the image could be 1Password or similar. It is good practive that users should store the created key-pair file / AGE-SECRET-KEY in a password vault. In this example only an admin team is allowed to access the AGE Secret key file in an admin vault (blue rectangle) and other means to access the cluster itself. &lt;/p&gt;

&lt;p&gt;Now &lt;code&gt;SOPS&lt;/code&gt; / &lt;code&gt;SOPS-Operator&lt;/code&gt; come into play. As explained in How does SOPS(-Operator) work from a meta perspective, SOPS-Operator is inspired by SOPS and is used to manage &lt;a href="https://kubernetes.io/docs/concepts/configuration/secret/"&gt;Kubernetes Secret Resources&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;You first need to encrypt your sensitive information locally, like shown in the picture, using the &lt;code&gt;AGE recipient Public Key&lt;/code&gt; and the unencrypted &lt;code&gt;SopsSecret&lt;/code&gt; as input to &lt;code&gt;sops&lt;/code&gt;. Sops hands this to the AGE process, which outputs the encrypted SopsSecret. The encrypted SopsSecret must then be supplied to your cluster.&lt;/p&gt;

&lt;p&gt;
&lt;a href="https://res.cloudinary.com/practicaldev/image/fetch/s--r33Y-JbR--/c_limit%2Cf_auto%2Cfl_progressive%2Cq_auto%2Cw_880/https://deyan7.de/wp-content/uploads/2023/02/sops_encrypt-563x1024.png" class="article-body-image-wrapper"&gt;&lt;img src="https://res.cloudinary.com/practicaldev/image/fetch/s--r33Y-JbR--/c_limit%2Cf_auto%2Cfl_progressive%2Cq_auto%2Cw_880/https://deyan7.de/wp-content/uploads/2023/02/sops_encrypt-563x1024.png" width="563" height="1024"&gt;&lt;/a&gt;
&lt;/p&gt;

&lt;p&gt;If you receive a SopsSecret with encrypted keys, the decryption flow is quite similar, but instead of the public key the Identity of the AGE-SECRET-KEY file is used.&lt;/p&gt;

&lt;p&gt;
&lt;a href="https://res.cloudinary.com/practicaldev/image/fetch/s--n2yvGZTr--/c_limit%2Cf_auto%2Cfl_progressive%2Cq_auto%2Cw_880/https://deyan7.de/wp-content/uploads/2023/02/sops_decrypt-779x1024.png" class="article-body-image-wrapper"&gt;&lt;img src="https://res.cloudinary.com/practicaldev/image/fetch/s--n2yvGZTr--/c_limit%2Cf_auto%2Cfl_progressive%2Cq_auto%2Cw_880/https://deyan7.de/wp-content/uploads/2023/02/sops_decrypt-779x1024.png" width="779" height="1024"&gt;&lt;/a&gt;
&lt;/p&gt;

&lt;p&gt;
&lt;a href="https://res.cloudinary.com/practicaldev/image/fetch/s--ZHzLaBFZ--/c_limit%2Cf_auto%2Cfl_progressive%2Cq_auto%2Cw_880/https://deyan7.de/wp-content/uploads/2023/02/sops-operator.png" class="article-body-image-wrapper"&gt;&lt;img src="https://res.cloudinary.com/practicaldev/image/fetch/s--ZHzLaBFZ--/c_limit%2Cf_auto%2Cfl_progressive%2Cq_auto%2Cw_880/https://deyan7.de/wp-content/uploads/2023/02/sops-operator.png" width="880" height="700"&gt;&lt;/a&gt;
&lt;/p&gt;

&lt;p&gt;When we look at the cluster (see image above), the process is comparable. The cluster receives an encrypted &lt;code&gt;SopsSecret&lt;/code&gt; Custom Resource Definition as input (see top left of the image). This &lt;code&gt;SopsSecret&lt;/code&gt; CRD is created locally (it can be written as &lt;code&gt;kubernetes_manifest&lt;/code&gt; with terraform). It has the same structure as usual Kubernetes Secret Resources. The values are the plain secrets, which are in the next step encrypted using SOPS. This encrypted manifest file is then applied to your cluster, via kubectl or e.g. terraform (it has to be converted to a terraform &lt;code&gt;kubernetes_manifest&lt;/code&gt; Resource first in this case). Afterwards it is available in your cluster and the &lt;code&gt;sops-secrets-operator&lt;/code&gt; Pod listens to state change (new or adjusted &lt;code&gt;SopsSecret&lt;/code&gt;-Crd). It takes this CRD together with the &lt;code&gt;sops-age-key-file&lt;/code&gt; Secret as input and decrypts the encrypted values via age, using the &lt;code&gt;Identity&lt;/code&gt; from the &lt;code&gt;sops-age-key-file&lt;/code&gt;. As a next step it creates a Kubernetes Secret from this SopsSecret CRD, so it can be consumed by other pods.&lt;/p&gt;

&lt;p&gt;Data is hereby decrypted as close as possible to the resource that is using it (like the database password of your PostgreSQL database or similar). This resource does not need to know anything about SOPS, since it can consume the password via the Kubernetes Secret resource, as usual.&lt;br&gt;&lt;br&gt;
We now have established a solution that does not carry sensitive, plain secrets in any parts of the system where they shouldn´t be.&lt;/p&gt;

&lt;p&gt;
&lt;a href="https://res.cloudinary.com/practicaldev/image/fetch/s--E4-hS8k8--/c_limit%2Cf_auto%2Cfl_progressive%2Cq_auto%2Cw_880/https://deyan7.de/wp-content/uploads/2023/02/ci_cd-530x1024.png" class="article-body-image-wrapper"&gt;&lt;img src="https://res.cloudinary.com/practicaldev/image/fetch/s--E4-hS8k8--/c_limit%2Cf_auto%2Cfl_progressive%2Cq_auto%2Cw_880/https://deyan7.de/wp-content/uploads/2023/02/ci_cd-530x1024.png" width="530" height="1024"&gt;&lt;/a&gt;
&lt;/p&gt;

&lt;p&gt;A typical workflow could look like it can be seen in the image above. You locally create an encrypted SopsSecret and push this to your git remote repository like Gitlab or Github. The pipeline picks up your changes and deploys this secret to your cluster.&lt;/p&gt;

&lt;h2&gt;
  
  
  &lt;a&gt;&lt;/a&gt;How to / Step-by-Step Guide
&lt;/h2&gt;

&lt;h3&gt;
  
  
  Prepare your cluster and install SOPS-Operator
&lt;/h3&gt;

&lt;ol&gt;
&lt;li&gt;Encode your secret in base64 format: &lt;code&gt;sed -n -e '/^AGE-SECRET-KEY/p' PATH_TO_YOUR_AGE_KEY_FILE | base64&lt;/code&gt;
&lt;/li&gt;
&lt;li&gt;Supply a secret (age identity) to your cluster (Needed to decrypt the SopsSecret CRDs)
&lt;/li&gt;
&lt;/ol&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nb"&gt;cat&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;&amp;lt;&lt;/span&gt;&lt;span class="no"&gt;EOF&lt;/span&gt;&lt;span class="sh"&gt; | kubectl apply -f -
apiVersion: v1
data:
  key: YOUR_BASE64_ENCODED_IDENTITY_PRIVATE_KEY
kind: Secret
metadata:
  name: sops-age-key-file
  namespace: YOUR_NAMESPACE
type: Opaque
&lt;/span&gt;&lt;span class="no"&gt;EOF
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Here we do not use Terraform on purpose in order to not make the secret part of your terraform state, as this state would leak your secrets (You can also generate a &lt;code&gt;kubernetes_secret&lt;/code&gt;, but some regulatory might force you to not leak this secret to your cloud provider).&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Add the &lt;a href="https://github.com/isindir/sops-secrets-operator#aws"&gt;sops-secrets-operator&lt;/a&gt; to your cluster, using &lt;code&gt;helm&lt;/code&gt; (Alternative below):
&lt;/li&gt;
&lt;/ol&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;
helm repo add sops https://isindir.github.io/sops-secrets-operator/
helm upgrade &lt;span class="nt"&gt;--install&lt;/span&gt; &lt;span class="nt"&gt;--create-namespace&lt;/span&gt; sops sops/sops-secrets-operator &lt;span class="nt"&gt;--namespace&lt;/span&gt; YOUR_NAMESPACE
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Or use e.g. Terraform:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight terraform"&gt;&lt;code&gt;&lt;span class="k"&gt;resource&lt;/span&gt; &lt;span class="s2"&gt;"helm_release"&lt;/span&gt; &lt;span class="s2"&gt;"sops-secrets-operator"&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="nx"&gt;name&lt;/span&gt;             &lt;span class="p"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;"sops-secrets-operator"&lt;/span&gt;
  &lt;span class="nx"&gt;chart&lt;/span&gt;            &lt;span class="p"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;"sops-secrets-operator"&lt;/span&gt;
  &lt;span class="nx"&gt;repository&lt;/span&gt;       &lt;span class="p"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;"https://isindir.github.io/sops-secrets-operator/"&lt;/span&gt;
  &lt;span class="nx"&gt;namespace&lt;/span&gt;        &lt;span class="p"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;"YOUR_NAMESPACE"&lt;/span&gt;
  &lt;span class="nx"&gt;version&lt;/span&gt;          &lt;span class="p"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;"0.14.0"&lt;/span&gt;
  &lt;span class="nx"&gt;create_namespace&lt;/span&gt; &lt;span class="p"&gt;=&lt;/span&gt; &lt;span class="kc"&gt;true&lt;/span&gt;

  &lt;span class="nx"&gt;values&lt;/span&gt; &lt;span class="p"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;
    &lt;span class="o"&gt;&amp;lt;&amp;lt;&lt;/span&gt;&lt;span class="no"&gt;EOF&lt;/span&gt;&lt;span class="sh"&gt;
extraEnv:
- name: SOPS_AGE_KEY_FILE
  value: /etc/sops-age-key-file/key
secretsAsFiles:
- mountPath: /etc/sops-age-key-file
  name: sops-age-key-file
  secretName: sops-age-key-file
&lt;/span&gt;&lt;span class="no"&gt;EOF
&lt;/span&gt;  &lt;span class="p"&gt;]&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h3&gt;
  
  
  Create SopsSecret &amp;amp; Deploy it to your cluster
&lt;/h3&gt;

&lt;ol&gt;
&lt;li&gt;Install &lt;code&gt;sops&lt;/code&gt;, &lt;code&gt;age&lt;/code&gt; and optionally &lt;code&gt;tfk8s&lt;/code&gt;, if you want to use terraform, via a package manager. Here we use brew:
&lt;/li&gt;
&lt;/ol&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;brew &lt;span class="nb"&gt;install&lt;/span&gt; &lt;span class="se"&gt;\&lt;/span&gt;
sops &lt;span class="se"&gt;\&lt;/span&gt;
age &lt;span class="se"&gt;\&lt;/span&gt;
tk8s
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;ol&gt;
&lt;li&gt;Generate a public-private-keypair: &lt;code&gt;age-keygen -o age-key.txt&lt;/code&gt;. [&lt;strong&gt;Optional&lt;/strong&gt;: Store the age-key.txt in a password vault, like 1Password. We highly recommend to store it somewhere save.]

&lt;ol&gt;
&lt;li&gt;Encrypt your secrets that should be used in your cluster
&lt;/li&gt;
&lt;/ol&gt;


&lt;/li&gt;
&lt;/ol&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;sops &lt;span class="nt"&gt;--encrypt&lt;/span&gt; &lt;span class="nt"&gt;--age&lt;/span&gt; &lt;span class="s1"&gt;'YOUR_AGE_RECIPIENT_PUBLIC_KEY_STARTING_WITH_age'&lt;/span&gt; &lt;span class="nt"&gt;--encrypted-suffix&lt;/span&gt; Templates SOPS_SECRET_FILE_YOU_WANT_TO_ENCRYPT.yml &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; SOPS_SECRET_FILE_ENCRYPTED.yml.enc
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;





&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight yaml"&gt;&lt;code&gt;
&lt;span class="na"&gt;apiVersion&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;isindir.github.com/v1alpha3&lt;/span&gt;
&lt;span class="na"&gt;kind&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;SopsSecret&lt;/span&gt;
&lt;span class="na"&gt;metadata&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;your-secrets-name&lt;/span&gt;
  &lt;span class="na"&gt;namespace&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;YOUR_NAMESPACE&lt;/span&gt;
&lt;span class="na"&gt;spec&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;secretTemplates&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
    &lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;your-secrets-name&lt;/span&gt;
      &lt;span class="na"&gt;stringData&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
         &lt;span class="na"&gt;NAME_OF_YOUR_SECRET&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;SECRET_ITSELF&lt;/span&gt;
         &lt;span class="na"&gt;NAME_OF_ANOTHER_SECRET&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;SECRET_ITSELF&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;ol&gt;
&lt;li&gt;Deploy this SopsSecret to your infrastructure via
&lt;/li&gt;
&lt;/ol&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;  kubectl apply &lt;span class="nt"&gt;-f&lt;/span&gt; SOPS_SECRET_FILE_ENCRYPTED.enc.yml
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;or if you want to use terraform, you can convert it with&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nb"&gt;cat &lt;/span&gt;SOPS_SECRET_FILE_ENCRYPTED.yml.enc | tfk8s &lt;span class="nt"&gt;--strip&lt;/span&gt; &lt;span class="nt"&gt;-o&lt;/span&gt; your-secrets-name.tf
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;This produces a &lt;a href="https://registry.terraform.io/providers/hashicorp/kubernetes/latest/docs/resources/manifest"&gt;kubernetes_manifest resource&lt;/a&gt;. Afterwards you can use &lt;code&gt;terraform plan&lt;/code&gt; and &lt;code&gt;terraform apply&lt;/code&gt; as you would normally do or use your CI / CD pipeline to do so.&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;With the following Terraform datasource you are able to reference the secrets in other resources. Watch out for the values of your keys in the binary_data, they should be empty as they are populated by terraform itfself. &lt;strong&gt;Attention (UPDATE: 23.2.2023):&lt;/strong&gt; Doing so stores this plain secrets in your state again and as of now there seems to be no way that they do not end up in your state, when you want to reference them (See: &lt;a href="https://github.com/hashicorp/terraform/issues/30469"&gt;Ability to not store certain attributes in TF state&lt;/a&gt;)
&lt;/li&gt;
&lt;/ol&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight terraform"&gt;&lt;code&gt;&lt;span class="k"&gt;data&lt;/span&gt; &lt;span class="s2"&gt;"kubernetes_secret"&lt;/span&gt; &lt;span class="s2"&gt;"your-secrets-name"&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="nx"&gt;metadata&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="nx"&gt;name&lt;/span&gt;      &lt;span class="p"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;"your-secrets-name"&lt;/span&gt;
    &lt;span class="nx"&gt;namespace&lt;/span&gt; &lt;span class="p"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;"YOUR_NAMESPACE"&lt;/span&gt;
  &lt;span class="p"&gt;}&lt;/span&gt;

  &lt;span class="nx"&gt;binary_data&lt;/span&gt; &lt;span class="p"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="nx"&gt;NAME_OF_YOUR_SECRET&lt;/span&gt;                               &lt;span class="p"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;""&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;You can e.g. define an output from it:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight terraform"&gt;&lt;code&gt;&lt;span class="k"&gt;output&lt;/span&gt; &lt;span class="s2"&gt;"NAME_OF_YOUR_SECRET"&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="nx"&gt;value&lt;/span&gt;       &lt;span class="p"&gt;=&lt;/span&gt; &lt;span class="nx"&gt;base64decode&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="k"&gt;data&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;kubernetes_secret&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;your&lt;/span&gt;&lt;span class="err"&gt;-&lt;/span&gt;&lt;span class="nx"&gt;secrets&lt;/span&gt;&lt;span class="err"&gt;-&lt;/span&gt;&lt;span class="nx"&gt;name&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;binary_data&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;NAME_OF_YOUR_SECRET&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
  &lt;span class="nx"&gt;sensitive&lt;/span&gt;   &lt;span class="p"&gt;=&lt;/span&gt; &lt;span class="kc"&gt;true&lt;/span&gt;
  &lt;span class="nx"&gt;description&lt;/span&gt; &lt;span class="p"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;"FooBar"&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;and use this output in any other module &lt;code&gt;module.secrets.NAME_OF_YOUR_SECRET&lt;/code&gt;.&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Check if everything went fine, by checking if the Kubernetes Secret Resource was created:
&lt;code&gt;kubectl get secrets your-secrets-name -n YOUR_NAMESPACE -o yaml&lt;/code&gt; .
If the output is &lt;code&gt;No resources found in YOUR_NAMESPACE namespace.&lt;/code&gt;, something went wrong. So best is to consult the logs of the operator first:
&lt;code&gt;kubectl logs  -l  app.kubernetes.io/name=sops-secrets-operator -n YOUR_NAMESPACE&lt;/code&gt;
&lt;/li&gt;
&lt;/ol&gt;

&lt;h3&gt;
  
  
  Decrypt and edit SopsSecret
&lt;/h3&gt;

&lt;ol&gt;
&lt;li&gt;If you want to edit existing secrets that one of your colleagues did encrypt, then get the secret identity key file from your teams password vault.&lt;/li&gt;
&lt;li&gt;Specify the location of your key file:  &lt;code&gt;export SOPS_AGE_KEY_FILE=./key.txt&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;Decrypt the file &lt;code&gt;sops -d SOPS_SECRET_FILE_ENCRYPTED.yml.enc&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;Edit the contents.&lt;/li&gt;
&lt;li&gt;Repeat the steps to encrypt and deploy the secret again.&lt;/li&gt;
&lt;/ol&gt;

&lt;h2&gt;
  
  
  Conclusion
&lt;/h2&gt;

&lt;p&gt;Handling secrets in a safe way is one of the hardest tasks in software development.&lt;br&gt;&lt;br&gt;
Using &lt;code&gt;SOPS Operator&lt;/code&gt; with &lt;code&gt;age&lt;/code&gt; allows you to completely free your code versioning repositories, CICD pipelines and other systems involved in the deployment of Kubernetes clusters from secrets.&lt;br&gt;&lt;br&gt;
Our approach still uses whichever workflow you are used to deploy and does not interfer with any tooling you might be using for these steps. The usual &lt;code&gt;Kubernetes Secret&lt;/code&gt; resources can be used within the cluster, enabling a high compatibility with existing mechanisms.&lt;/p&gt;

</description>
    </item>
    <item>
      <title>Top 5 fails when developing a DiGA backend (German Digital Therapeutics – DTx)</title>
      <dc:creator>Deyan7 GmbH &amp; Co.KG</dc:creator>
      <pubDate>Thu, 19 Jan 2023 11:10:13 +0000</pubDate>
      <link>https://dev.to/deyan7/top-5-fails-when-developing-a-diga-backend-german-digital-therapeutics-dtx-4o3n</link>
      <guid>https://dev.to/deyan7/top-5-fails-when-developing-a-diga-backend-german-digital-therapeutics-dtx-4o3n</guid>
      <description>&lt;p&gt;Navigating the jungle of ISO norms and DiGAV annexes can be frustratingly hard and sometimes you still feel you didn´t get a real direction or answer. Legal texts are mostly quite blurry when it comes to concrete technical implications.&lt;/p&gt;

&lt;p&gt;Since we are working with &lt;strong&gt;DiGA&lt;/strong&gt; s on a daily basis and do know their challenges, we will show some of the most important &lt;strong&gt;pitfalls&lt;/strong&gt; you will encounter when developing a DiGA. We will also &lt;strong&gt;present our solutions and technology stacks&lt;/strong&gt; to avoid the mentioned pitfalls.&lt;/p&gt;

&lt;h2&gt;
  
  
  &lt;strong&gt;1. Choose correct Storage Location&lt;/strong&gt;
&lt;/h2&gt;

&lt;p&gt;In a nutshell, we do &lt;strong&gt;not recommend to use any major cloud services provider or BaaS offerings&lt;/strong&gt; (like Firebase, AWS, Azure or GCP) and hence we will have to find an alternative to host your personal data.&lt;/p&gt;

&lt;p&gt;Many mobile developers starting from scratch will choose &lt;a href="https://docs.flutter.dev" rel="noopener noreferrer"&gt;Flutter&lt;/a&gt; as their platform of choice (as do we). And with Flutter, &lt;a href="https://firebase.google.com" rel="noopener noreferrer"&gt;Firebase&lt;/a&gt; is heavily marketed and well-supported as the &lt;strong&gt;Backend-As-A-Service&lt;/strong&gt; solution.&lt;/p&gt;

&lt;p&gt;Sadly, there is &lt;a href="https://www.gesetze-im-internet.de/digav/anlage_1.html" rel="noopener noreferrer"&gt;DiGAV Annex I, data protection 38&lt;/a&gt;, which states that &lt;strong&gt;personal data can only be processed in the following locations&lt;/strong&gt; :&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;In Germany&lt;/li&gt;
&lt;li&gt;In another member state of the EU&lt;/li&gt;
&lt;li&gt;In accordance with an adequacy resolution of EU 2016/679, article 45&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;strong&gt;Keeping it short: USA and any services originating from there are not covered&lt;/strong&gt;. The only way to store outside of the aforementioned locations is to either encrypt or to anonymize the personal data before it leaves the EU.&lt;/p&gt;

&lt;h3&gt;
  
  
  Should we just encrypt?
&lt;/h3&gt;

&lt;p&gt;&lt;strong&gt;Encrypting&lt;/strong&gt; opens up a completely new problem space for backend developers, as even simple database operations like &lt;strong&gt;selections, joins or ordering might become unfeasible&lt;/strong&gt;. The encryption key is not allowed to be used in systems that are non-EU based and of course &lt;strong&gt;decrypting the personal data&lt;/strong&gt; , no matter where, &lt;strong&gt;drastically reduces performance&lt;/strong&gt; of those operations that should be finished in a millisecond time frame.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Anonymizing&lt;/strong&gt; is likely not what you want, as the personal data would be lost for the user.&lt;/p&gt;

&lt;p&gt;Performing the encryption in-app leaves the possibility for “skilled users” to get access to the encryption key.&lt;/p&gt;

&lt;p&gt;So if you want to store &lt;strong&gt;unencrypted personal or person-relatable data&lt;/strong&gt; in a backend, you need to make sure that:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Personal data is stored in the EU&lt;/li&gt;
&lt;li&gt;The company providing the storage solution is not a subsidiary of any US-based company&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;This &lt;strong&gt;rules out all major cloud services like Firebase, AWS, Azure, Google Cloud Services&lt;/strong&gt; , even if you choose a european storage location.&lt;/p&gt;

&lt;p&gt;What you will choose as an alternative will depend on your technology stack. Do you only run everything on one Virtual Machine? Do you prefer a &lt;strong&gt;Kubernetes&lt;/strong&gt; cluster? Would you like to go fully &lt;strong&gt;serverless&lt;/strong&gt;?&lt;/p&gt;

&lt;p&gt;For our customers, &lt;strong&gt;we have chosen &lt;a href="https://www.scaleway.com" rel="noopener noreferrer"&gt;https://www.scaleway.com/&lt;/a&gt; as our provider of choice&lt;/strong&gt;. Located in France, it contains a &lt;strong&gt;fully managed Kubernetes&lt;/strong&gt; offering and everything else you might need to develop a DiGA.&lt;/p&gt;

&lt;p&gt;Other alternatives are:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;&lt;a href="https://open-telekom-cloud.com" rel="noopener noreferrer"&gt;https://open-telekom-cloud.com&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://www.ovhcloud.com" rel="noopener noreferrer"&gt;https://www.ovhcloud.com/&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://www.hetzner.com/cloud" rel="noopener noreferrer"&gt;https://www.hetzner.com/cloud&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  &lt;strong&gt;2. Mitigate Broken Access Control&lt;/strong&gt;
&lt;/h2&gt;

&lt;p&gt;Especially for a DiGA it is vital that the stored personal and health data is &lt;strong&gt;secure&lt;/strong&gt;. The data should &lt;strong&gt;only be accessible by those that are accredited to use it&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;It seems like a no-brainer, but in the past there have been reports about existing DiGAs where &lt;strong&gt;one user could read the (quite sensitive) data of another user&lt;/strong&gt; (&lt;a href="https://zerforschung.org/posts/datenabfluss-auf-rezept/" rel="noopener noreferrer"&gt;https://zerforschung.org/posts/datenabfluss-auf-rezept/&lt;/a&gt;), just by incrementing or decrementing a number in the request.&lt;/p&gt;

&lt;p&gt;There are &lt;strong&gt;several ways to mitigate&lt;/strong&gt; this, even before the app reaches the public:&lt;/p&gt;

&lt;h3&gt;
  
  
  &lt;strong&gt;Solve authorization by reusing code&lt;/strong&gt;
&lt;/h3&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;def authorize_user(user_id: str, user: auth_schemas.AuthenticatedUser = Depends(authenticated_user)) -&amp;gt; auth_schemas.AuthorizedUser:
    if user.user_id == user_id:
        return user
    raise authorization_exception()
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;If you can refactor authorization code into &lt;strong&gt;building blocks that can be reused&lt;/strong&gt; , chances to ‘forget’ pieces of the puzzle are reduced.&lt;/p&gt;

&lt;h3&gt;
  
  
  &lt;strong&gt;Have Peer Reviews&lt;/strong&gt;
&lt;/h3&gt;

&lt;p&gt;By having a &lt;strong&gt;second developer review the code&lt;/strong&gt; , flaws are much more likely to be discovered. The reviewing dev should also check if there are tests securing the behavior and inspect if the newly introduced feature covers all edge cases.&lt;/p&gt;

&lt;p&gt;We also recommend to automatically add &lt;strong&gt;checklists&lt;/strong&gt; for the reviewer to every Merge/Pull Request.&lt;/p&gt;

&lt;p&gt;No excuses, &lt;strong&gt;every branch should be reviewed&lt;/strong&gt; before being merged to the main line of code!&lt;/p&gt;

&lt;h3&gt;
  
  
  &lt;strong&gt;Introduce System Tests&lt;/strong&gt;
&lt;/h3&gt;

&lt;p&gt;&lt;strong&gt;System Tests&lt;/strong&gt; do secure the behavior of your system as a whole, different than &lt;strong&gt;Unit Tests&lt;/strong&gt; and &lt;strong&gt;Integration Tests&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;Usually this is achieved by starting up a new instance of the whole backend system, containing the new feature to be introduced. A piece of software then acts as the client and uses the backend.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Useful tests&lt;/strong&gt; that answer the following questions for all introduced endpoints could be:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Is the data of the test user accessible without authentication?&lt;/li&gt;
&lt;li&gt;Is the data accessible with authentication?&lt;/li&gt;
&lt;li&gt;Can the data of another user be fetched/manipulated/deleted without being accredited to do so?&lt;/li&gt;
&lt;li&gt;Do aggregated endpoints only return data for the current user or do they leak information that should not be accessible to that user?&lt;/li&gt;
&lt;/ul&gt;

&lt;h3&gt;
  
  
  &lt;strong&gt;Define guidelines&lt;/strong&gt;
&lt;/h3&gt;

&lt;p&gt;One part of this guideline should be, that &lt;strong&gt;no enumerable ID is handed out&lt;/strong&gt; of the backend &lt;strong&gt;or accepted as input&lt;/strong&gt; to retrieve a selection of data.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Not good:&lt;/strong&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;http://backend.com/v1/user/1
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;strong&gt;Better:&lt;/strong&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;http://backend.com/v1/user/ADDC955C-E420-48D2-941E-70A9C45CB9E4
http://backend.com/v1/user/hello[at]mail.com
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Instead, you should opt for a &lt;strong&gt;UUID&lt;/strong&gt; or a similar, non-enumerable ID. Reasons:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;If your API contains enumerable IDs, it is quite easy fetch all of the data, just by incrementing the number to infinity&lt;/li&gt;
&lt;li&gt;It´s also easy to &lt;strong&gt;guess your total number of users&lt;/strong&gt; , overall entries of users and so on, because usually numerical IDs start with number 1.&lt;/li&gt;
&lt;li&gt;If your access control is broken, your API is instantly &lt;strong&gt;attackable&lt;/strong&gt;  &lt;strong&gt;by incrementing/decrementing IDs&lt;/strong&gt;. Guessing UUIDs is much harder than incrementing a number.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  &lt;strong&gt;3. Use approved Identity Provider&lt;/strong&gt;
&lt;/h2&gt;

&lt;p&gt;Lots of developers are &lt;strong&gt;tempted to go with the usual solutions&lt;/strong&gt; for user login and authentication, for example &lt;a href="https://firebase.google.com" rel="noopener noreferrer"&gt;Firebase&lt;/a&gt; or &lt;a href="https://auth0.com" rel="noopener noreferrer"&gt;Auth0&lt;/a&gt;. Maybe they even are convinced to develop their &lt;strong&gt;own user management&lt;/strong&gt; , directly built into the backend.&lt;/p&gt;

&lt;p&gt;No developer wants to spend weeks of developing on a feature or software package that &lt;strong&gt;has been solved thousands of times before&lt;/strong&gt;. If we are developing a &lt;strong&gt;DiGA&lt;/strong&gt; though, there´s &lt;strong&gt;rules to what an Identity Provider should fulfil&lt;/strong&gt; and there are even &lt;strong&gt;OpenSource&lt;/strong&gt; projects that are recommended to use.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://www.gesetze-im-internet.de/digav/anlage_1.html" rel="noopener noreferrer"&gt;DiGAV Annex I, data security 11&lt;/a&gt; states:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;The DiGA should use a centralized authentication/authorization component&lt;/li&gt;
&lt;li&gt;It is built using established standard components&lt;/li&gt;
&lt;li&gt;The DiGA can verify trustworthiness of the auth component&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;An identity provider also uses &lt;strong&gt;some token of personal data&lt;/strong&gt; (for example e-mail address or phone number) to identify the user´s existence. This in turn means that we have to look into the &lt;strong&gt;storage location&lt;/strong&gt; of our chosen Identity Provider, as &lt;strong&gt;personal data receives special treatment&lt;/strong&gt; in the DiGA and GDPR context.&lt;/p&gt;

&lt;p&gt;Their storage location &lt;strong&gt;rules out Firebase, Auth0&lt;/strong&gt; and other comparable products instantly. ‘ &lt;strong&gt;Homegrown&lt;/strong&gt; ‘ implementations are usually ruled out because they &lt;strong&gt;don´t fulfil the ‘established standard component’ rule&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;Thankfully, in the newest &lt;a href="https://www.bfarm.de/SharedDocs/Downloads/DE/Medizinprodukte/DiPA_Leitfaden.html" rel="noopener noreferrer"&gt;DiPA FastTrack Leitfaden&lt;/a&gt;, BfArM acknowledges &lt;strong&gt;three Identity Providers&lt;/strong&gt; as established and proven components:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Keycloak&lt;/li&gt;
&lt;li&gt;OpenIAM&lt;/li&gt;
&lt;li&gt;CAS&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;These are &lt;strong&gt;battle-proven&lt;/strong&gt; over years and constantly being updated to ensure that &lt;strong&gt;no vulnerabilities&lt;/strong&gt; are left open for attackers.&lt;/p&gt;

&lt;p&gt;Our choice is &lt;a href="https://www.keycloak.org" rel="noopener noreferrer"&gt;Keycloak&lt;/a&gt;, which also &lt;strong&gt;fulfils lots of other DiGA requirements&lt;/strong&gt; all in one go, for example:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Password policy&lt;/li&gt;
&lt;li&gt;Brute Force protection&lt;/li&gt;
&lt;li&gt;Two-Factor authentication&lt;/li&gt;
&lt;li&gt;Audit logs&lt;/li&gt;
&lt;li&gt;Established components like OpenID Connect and OAuth2&lt;/li&gt;
&lt;li&gt;Configurable JWT token expiry times&lt;/li&gt;
&lt;li&gt;And many more&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  &lt;strong&gt;4. Implement silent fails&lt;/strong&gt;
&lt;/h2&gt;

&lt;p&gt;What would you say if you are a customer of a DiGA where its usage might have an &lt;strong&gt;impact on social expectations&lt;/strong&gt; placed upon you? Would you tell your colleagues about it?&lt;/p&gt;

&lt;p&gt;Examples could be apps where the focus is on sexual or psychological topics.&lt;/p&gt;

&lt;p&gt;Of course &lt;strong&gt;we do not welcome&lt;/strong&gt; that certain diagnoses do attract more attention than others, but with the current state of society I am sure the majority would &lt;strong&gt;not be pleased&lt;/strong&gt; if a diagnose becomes public. That´s exactly why we want to &lt;strong&gt;implement silent fails&lt;/strong&gt; to stop leaking personal information.&lt;/p&gt;

&lt;p&gt;The usual place where this kind of leak can happen are the backend or identity provider endpoints for &lt;strong&gt;user creation, password reset and login.&lt;/strong&gt;&lt;/p&gt;

&lt;h3&gt;
  
  
  &lt;strong&gt;Example for existing user&lt;/strong&gt;
&lt;/h3&gt;

&lt;p&gt;&lt;strong&gt;Request:&lt;/strong&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;POST https://backend.com/auth/login
{
  "email": "marc-geheim[at]deyan7.de",
  "password": "asdfqwertz"
}
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;strong&gt;Response:&lt;/strong&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;{
  "error": "Unauthorized",
  "message": "Username and password do not match",
  "statusCode": 401
}
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h3&gt;
  
  
  &lt;strong&gt;Example for nonexisting user&lt;/strong&gt;
&lt;/h3&gt;

&lt;p&gt;&lt;strong&gt;Request:&lt;/strong&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;POST https://backend.com/auth/login
{
  "email": "marcus.roedderus[at]deyan7.de",
  "password": "asdfqwertz"
}
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;strong&gt;Response:&lt;/strong&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;{
  "error": "Bad Request",
  "message": "Account does not exist.",
  "statusCode": 400
}
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;As you can see, if your backend or Identity Provider is implemented in a way that it responds with &lt;strong&gt;too detailed error information&lt;/strong&gt; , it is &lt;strong&gt;quite easy to crawl your backend&lt;/strong&gt; and to find out who is a customer of yours.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;The solution here is to return the exact same response for all error cases.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;This will make it harder for the User Experience of the app, but it won´t leak information.&lt;/p&gt;

&lt;h2&gt;
  
  
  &lt;strong&gt;5. Do not underestimate or misinterpret the interoperable export requirement&lt;/strong&gt;
&lt;/h2&gt;

&lt;p&gt;One DiGA on its own might already be beneficial for the patient, but the &lt;strong&gt;real power&lt;/strong&gt; comes into play when the data of all the patient´s apps, diagnoses etc. could be &lt;strong&gt;inspected together as a whole&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;For this to happen, a &lt;strong&gt;data format&lt;/strong&gt; needs to be introduced that also includes &lt;strong&gt;medical terminologies&lt;/strong&gt; like &lt;a href="https://www.snomed.org" rel="noopener noreferrer"&gt;SNOMED-CT&lt;/a&gt; and &lt;a href="https://loinc.org" rel="noopener noreferrer"&gt;LOINC&lt;/a&gt; to fit into the whole context of medical data. &lt;a href="https://www.hl7.org/fhir/" rel="noopener noreferrer"&gt;HL7 FHIR R4&lt;/a&gt; is the weapon of choice here, and although it is not explicitly mentioned in the &lt;strong&gt;DiGAV&lt;/strong&gt; , it´s factually the only export format that´s accepted by &lt;a href="https://www.bfarm.de" rel="noopener noreferrer"&gt;BfArM&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;It´s explicitly &lt;strong&gt;not sufficient&lt;/strong&gt; to just export your own format as &lt;strong&gt;JSON, XML or CSV by serializing your objects&lt;/strong&gt; , although the DiGAV might sound a bit like it!&lt;/p&gt;

&lt;p&gt;If you don´t have experience with &lt;strong&gt;HL7 FHIR R4 profiling&lt;/strong&gt; , it may be time to either look for an external expert to create it (shameless plug: &lt;strong&gt;we can help!&lt;/strong&gt; ), or to reserve a fair bit of time to learn.&lt;/p&gt;

&lt;p&gt;We recommend using &lt;a href="https://github.com/FHIR/sushi" rel="noopener noreferrer"&gt;https://github.com/FHIR/sushi&lt;/a&gt; and &lt;a href="https://hl7.org/fhir/uv/shorthand/" rel="noopener noreferrer"&gt;https://hl7.org/fhir/uv/shorthand/&lt;/a&gt; to create your &lt;strong&gt;Implementation Guides&lt;/strong&gt;. An &lt;strong&gt;example&lt;/strong&gt; of an ImplementationGuide that was created by &lt;strong&gt;Deyan7&lt;/strong&gt; can be found here: &lt;a href="https://simplifier.net/medipee-uroli-export" rel="noopener noreferrer"&gt;https://simplifier.net/medipee-uroli-export&lt;/a&gt;.&lt;/p&gt;




&lt;p&gt;Of course there is lots more ground to cover to really reach the state of a DiGA that is approved by Germany´s BfArM (federal institute for drugs and medical products), e.g. proving your medical efficacy or acquiring several certifications, but in this article we only want to focus on what you need to have in mind when writing software that should become a DiGA later.&lt;/p&gt;

&lt;p&gt;The post &lt;a href="https://deyan7.de/top-5-fails-when-developing-a-diga-backend-german-digital-therapeutics-dtx/" rel="noopener noreferrer"&gt;&lt;strong&gt;Top 5 fails when developing a DiGA backend (German Digital Therapeutics – DTx)&lt;/strong&gt;&lt;/a&gt; was first published at &lt;a href="https://deyan7.de" rel="noopener noreferrer"&gt;Deyan7&lt;/a&gt;.&lt;/p&gt;

</description>
      <category>backend</category>
      <category>python</category>
      <category>testing</category>
      <category>security</category>
    </item>
  </channel>
</rss>
