DEV Community: Lucas Severo Alves

Yet another rant in favor of Kubernetes Secrets

Lucas Severo Alves — Tue, 01 Aug 2023 04:58:37 +0000

Translation to pt-br at: https://knela.dev/blog/mais-um-desabafo-em-favor-dos-segredos-do-kubernetes

Back in the day, Kubernetes secrets had many engineers scratching their heads, like "Is this a joke"? Secrets are implemented in base64 encoding which is not an encryption system at all but a simple encoding. Of course, anybody could then decode it. I must admit I was one of those engineers. My opinion changed, but I can understand the feeling most people have with regard to these objects, however wrong this premise is.

Why are Kubernetes secrets base64 encoded? (And why that's ok)

The base64 encoding of Kubernetes secrets was never intended as a security measure. It does not exist to obscure or hide any values. The decision to implement this encoding method was initially made to enable the secrets to accommodate binary data. If the secrets were structured as simple maps, perhaps there would be less confusion. However, this isn't the point. When observers notice the base64 encoding, they often assume it to be a flawed attempt to achieve the protection of a value, even going so far as to regard it as a mistake. This interpretation, as we just saw, isn't accurate.

How to make Kubernetes Secrets secure?

Initially, RBAC (Role Base Access Control): Restrict users or teams that have access to secrets, or block that access completely. Give access of a specific secret only to service accounts that need it.

If you have any policy engine: Include policies that won't allow secrets being globally accessible. Come up with policies that make sense with your context, blocking or alerting when someone is violating your guidelines. Maybe you just want to avoid teams having access to other team's secrets. Maybe your control is more fine grained, and that is for you to figure out.

Encrypt etcd at rest: This is the method to safeguard secret values stored within etcd and you shouldn't worry about it being easy to decode when you get the value. When you retrieve a value from kube API server with curl or kubectl, and it feels odd due to it merely being base64, pause to consider. When you are authenticated and authorized to do so, isn't obtaining a secret from a source you deem secure essentially the same? For instance, if you curl Hashicorp Vault (or Azure Key Vault, or any other equivalent), armed with your proper keys and certificates, isn't the response plain text (I know it is obvious, I think we forget about this)? The value needs to be in plain text at some point, to the consumer — to the application that will use that credential — to do something. Our primary objective should be to ensure the secure storage and transmission of these values (though the latter won't be the focus in this article). We shouldn't concern ourselves unnecessarily with the data being displayed in plain text, provided the request comes from an authenticated and authorized entity that requires this information.

Encrypting etcd alone is not enough

Most guides that teach you how to encrypt etcd at rest will get you into a predicament. Encryption keys, in a lot of cases, are still stored in the same machines where your control plane components are running. If you consider an etcd instance to be compromised, and let's say the attacker has access to files on that machine, it is super simple to just decrypt the whole database.

The preferred method of encrypting etcd values is by using a Key Management Service. Here we have some extra levels of indirection and also another layer of enforcement on authorization. You store the encryption key, encrypted, in a remote service. And only authorized services or users can access and decrypt the key and then the data. This can still be mimicked by an attacker, and reverse engineered into what they need, but it is still the recommended method as of this writing (you can always come up with more indirections, but complexity can also impact negatively security incident response, so better stick to supported approaches).

A few other adjacent measures will be listed at the end of the article.

Who uses Secrets?

Kubernetes 😂. Your control plane, right now, is using Kubernetes Native Secrets, and those secret values are of course stored in etcd. It does not matter the flavor of Kubernetes, which cloud provider. Attackers with access to those secrets can already cause a lot of damage if you don't protect those secrets.

My reasoning for this is as follows: if you believe Kubernetes Secrets should be avoided, does this mean you are also discarding the secrets that inherently come with using Kubernetes? You are probably not doing that. It might be more pragmatic to acknowledge their existence and strategize ways to secure them, along with your other secrets, within this context.

Furthermore, Cloud Native tools from the Cloud Native Landscape will likely anticipate your secret values to be Kubernetes Native Secrets. If you wish to leverage the plug-and-play capabilities of the Kubernetes Ecosystem, avoiding secrets could lead to unnecessary difficulties. This choice would probably result in an excessive reliance on custom in-house solutions, which not only increase complexity but also introduce additional security risks.

So, etcd is compromised

Let's try to explore a few premises and scenarios with regard to avoiding secrets.

The relevant assumptions when avoiding Kubernetes Secrets

The biggest argument against Kubernetes Secrets comes from the fact that if etcd is compromised, you basically have access to any secret value on it. And I can see how that can be very dangerous. But I would argue this is only relevant in a very unrealistic scenario:

Attacker has only read access to etcd (biggest gotcha in my opinion)
Attacker has encryption key (or etcd is unencrypted)
(Optional) Attacker has kubeconfg/credentials of a dev with limited permissions
- can't pod exec into a container
- can't create pods in important namespaces
- can only get a few resources, logs, and some other minor permissions
Cluster is updated (above 1.24), and does not have any legacy token secrets for service accounts

If this scenario came to fruition (an attack happening exactly like this), avoiding Kubernetes Secrets in fact ended up protecting your environment. That's good.

So, why do I say this is an unrealistic scenario? Have you ever heard of a security breach where an attacker got only read access to etcd, without write? This would require an attacker to get access to an etcd instance or endpoint, and have access to credentials of an user that only has read (but they have the encryption key). Another option could be: if the attacker has access to the instance and for some reason they have access only to the encryption key and not the credentials. In my experience, and opinion, this is not the scenario that actually happens. When etcd is compromised, it is completely compromised (I will touch on what happens when etcd is completely compromised later in this article).

The other requirements of this scenario are also important. Avoiding secrets would not help if ANY of those are not true. If an Attacker has pod exec into any pod consuming a secret value (being Kubernetes Native Secret or not) that attacker now has a secret value (if they can grab a shell — if this is not protected or you are not using distro-less). Similarly if attackers can create pods using whatever k8s-secret-avoiding-mechanism you came up with, with a secret value mounted, again, they got a secret.

Last but not least, if you are on Kubernetes <1.24, service accounts used to store tokens in secrets (not using TokenRequests). Whichever k8s-secret-avoiding-mechanism you might been using, probably used service account tokens or other credentials to talk to external Vaults. An attacker with access to a compromised etcd could just use that token to get a remote secret.

Given that all these factors must align for avoiding Kubernetes Native Secrets to bear significance from a pure secret value protection standpoint, I would consider it niche, and not really relevant to the majority of setups around the globe.

The non-arguments

Previous scenario is a good example of when you are actually protected by not having Kubernetes Native Secrets, even though it is unrealistic. These other ones listed here are, in my opinion, not good arguments against using Secrets:

Saying you can get the value easily decoded through the kube api-server: That's also easily protected by RBAC. And even considering any other way to mount a secret value into a pod, through an unprotected API you can still pod exec into it and get secret values.
Saying Secrets as Env vars are easily inspected: Kubernetes Native Secrets can be mounted as volumes.
Saying that compromises that are not etcd related could also leave secrets exposed: The risk still exists with any other k8s-secret-avoiding-mechanism. Root access to nodes with RAM dumping or memory scan, network sniffing, or anything like that. As we said before, applications have the final plain text secret because they need it like that. A memory scan (or similar breach), due to its intrusive nature, could reveal secret values regardless of whether they are Kubernetes Native Secrets or not.

The write on etcd

If an attacker has write on etcd (and access to any kubeconfig even with a role without permissions), they have you. It does not matter if you are avoiding Kubernetes Native secrets. It does not matter if you are using complex policies with popular policy engines.

A few ways to get secrets (or almost anything) with write on etcd:

Insert a RoleBinding with any privileged role to get access to pods with secret values mounted.
Insert a pod or Deployment directly through etcd with inline secret value mounting.
If a pod has a service account that lets it get secrets from a remote Vault: Insert a pod in etcd and make it use the same service account that can talk with the external Vault and write a simple code to call the TokenRequest API. This would also work on Kubernetes >1.24. You can log the token and get all secrets before it expires.
There are a few other more complicated ways to get secrets even if the attacker does not have access to kube api-server. Shifting roles and role bindings around, getting privileged pods inside the cluster, acting and getting responses all from etcd. It is harder, but possible.

So, nothing is secure?

That is not the point. The point is that you have very similar threat models while avoiding Kubernetes Native Secrets and while not avoiding them. So, with similar threat models, the best option is the simplest option. To just use what you are used to, and figure out how to protect that.

So..., we don't have any reason to avoid Kubernetes Secrets?

Now, that is the right question! Do you? There are plenty of reasons to avoid Kubernetes Secrets. Maybe you have a company policy to follow. Perhaps you have to adhere to multiple types of compliance. Maybe the complexity of avoiding Kubernetes Native Secrets is worth it in your case, because it will make your life easier during auditing and some other recurring processes.

My point is more on the side of: Avoiding Kubernetes Secrets is not intrinsically more secure all the time. On the other hand, hashing passwords when storing them into a database table is, and it is simple enough to always be worth it, and this is clearly not the same thing. You HAVE to check your requirements, and see what works for you, in this case.

What is the best way to protect your secrets then?

A few of the best practices were listed in the beginning of the article when we talked about RBAC, policies and encrypting and protecting ETCD. But there are other common security practices that are maybe even more important:

Avoid initial attack vectors and using vulnerable software/dependencies (supply chain security). If your frontend has a vulnerable lib that lets people inject arbitrary code and somehow grab a shell of that frontend container. Then if somehow they can escape the container and escalate privileges to root, and then do a memory scan... This person has all your secrets on this node. It does not matter if you are avoiding Kubernetes Native Secrets or which solution you are using for synchronizing your secrets. Besides avoiding vulnerable software/dependencies, there are a bunch of other ways to protect yourself, like phishing training for non technical employees, WAF, and many others.
Be mindful about Access Management all around. Not only RBAC on Kubernetes, but also to your cloud accounts, sub-accounts and your whole infrastructure.
Design good identity and security boundaries for your system. Isolate tenants or teams of your company in a way that makes sense with regard to their daily work, your software architecture and your team structure, giving permissions and accesses only to people that need it.

I know, these sound generic and basically best practices to protect your system in general. My point is that exactly. You have to do that first.

Additionally:

Make sure your approach takes into consideration security response, and not only prevention. It is not only about protecting secrets. It also about what to do when (not if) they leak.
Being one of the maintainers of External Secrets Operator (ESO) and getting most of the general advice out of the way, I also want to give you some reasons why ESO helps with security and facilitates your processes and I think you should also consider it (or some other tools that also deliver on some of these points):
- you can give your developers only references to secrets, and not secret values directly.
- ESO integrates into developers GitOps processes facilitating adoption and debugability.
- it facilitates the toll of auto-rotation and secret generation, helping with prevention and response.
- ESO already considers multi-tenant and multi cluster setups, then if that's your case you don't have to improvise
- ESO can use another service-account identity, per provider configuration, so you don't have to give permissions to the operator pod on all your providers. This also allows the pod itself to not have any permissions to the provider, just a detached service account.
- Accepting Kubernetes Native Secrets as the final destination to secret values simplifies your solution (most of the time). Complexity will mostly hurt the security of your system.

Summarizing everything

The understanding and proper management of Kubernetes secrets are critical, and base64 encoding, often misunderstood as a weak security measure, is intended to handle binary data. Using RBAC and policies, and encrypting etcd at rest are ways to make secrets secure. However, the total avoidance of Kubernetes secrets is not necessarily safer and could add complexity. Always evaluate your specific needs to choose the right approach. Focus not only on prevention but also on effective response strategies for potential leaks. Tools like the External Secrets Operator (ESO) can assist in securing Kubernetes secrets, especially by making it easy to perform rotations. Essentially, system security requires a comprehensive view rather than focusing only on individual elements.

References:

Unfortunately the reasoning behind base64 encoding on secrets is not shared on the documentation, or PRs/issues, as this of course feels obvious to core Kubernetes developers. However, Jordan Liggitt, one of Kubernetes top contributors, shares an answer here:

https://stackoverflow.com/questions/49046439/why-does-k8s-secrets-need-to-be-base64-encoded-when-configmaps-does-not

With regard to encryption at rest and KMS:

Considering my statement about any Kubernetes Control Plane using secrets to work under the hood: If you are an admin you can just kubectl get secrets -A within a fresh provisioned cluster to see all the secrets being used. A few managed solutions won't even hide some from you. If it's self managed you are going to see hundreds of them.

Given my opinion and experience on the read/write aspect of etcd during incidents. I would be curious to hear about other people experiences. Maybe I am completely mistaken here. Feel free to reach out!

It is relevant to also have a look over documentation for Secrets and RBAC on official docs:

Looking back at the part we talked about injecting pods or other resources into etcd directly, here you can see a simple step by step example of injecting a pod:

https://lobuhisec.medium.com/using-etcd-to-inject-resources-and-bypass-rbac-and-admission-controller-restrictions-f240ae31e7f0

Even though I don't necessarily agree with all statements on the following article (as I still think it is worth it to use external providers), this was one of the blogs that resonated with me some time ago when we were having these kinds of discussions within the community: https://www.macchaffee.com/blog/2022/k8s-secrets/, written by Mac Chaffee. I get some analogies and threat modeling way of thinking from it.

Finally, since we mentioned ESO, I would recommend looking at ESO's docs and the talk given at Kubecon:

And would also like to thank the whole external-secrets community for trusting ESO when handling their secrets! :D

As a follow up blog post I'm thinking about sharing some lab examples of how to get secrets in multiple scenarios, and how to protect from those attacks. Please let me know if that would be interesting to you, as it would be a lot of work, and I can do it on demand! 😁

True Secrets Auto Rotation with ESO and Vault

Lucas Severo Alves — Thu, 27 Jul 2023 20:40:37 +0000

Translation to pt-br at https://knela.dev/blog/auto-rotao-de-segredos-de-verdade-com-eso-e-vault

Requirements

A Kubernetes cluster that you can use (kind, minikube, something managed) and kubectl to connect to it
Vault CLI
External Secrets Operator (ESO) installed.
Vault installed through the helm chart

What we want to achieve

This guide aims to establish an automatic hourly rotation of a database connection secret. Following these steps, an administrator sets up the process once, ensuring that the secret refreshes/changes every hour. Simultaneously, the application will always maintain new valid credentials for seamless database interactions.

ESO secret Generators

⚠️ As of this writing this feature is in alpha state, and we want more people to help test it, so we can make improvements, and eventually promote to stable.

Documentation around it is a bit limited, that's why I am getting this guide out in my blog, while we figure better ways to bring these into our documentation.

Getting Started

Let's make sure we start with the same setup locally:

I have installed Vault in a namespace named vault.
- You can use helm install vault hashicorp/vault -n vault --create-namespace command instead of the one provided in the guide.
- Follow all steps in there to init Vault, unseal, and get the cluster-keys.json with the token.
- You can skip other steps.
I have installed ESO in the default namespace.

In this guide we are going to use Vault token authentication just for the sake of simplicity. However, please never use this in real setups. Prefer service account auth.

After properly starting Vault and unsealing it, take note of your auth token. Let's do a port forward and authenticate in our work desktop so we don't have to exec into Vault every time we need to run commands.

In a new terminal (this terminal will be blocked)



kubectl -n vault port-forward service/vault 8200:8200

In another terminal you can run.



export VAULT_ADDR=http://127.0.0.1:8200
vault login
## type your auth token

Simple Deployment of PostgreSQL

To have an interesting example, let's deploy psql and configure it so we can let Vault and other workloads connect to it.

Let's first create a configmap with an admin user and password for this psql instance (just for simplicity and to get to the other part of the guide quickly).



cat <<EOF > postgres-config.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: postgres-config
  labels:
    app: postgres
data:
  POSTGRES_DB: postgresdb
  POSTGRES_USER: admin
  POSTGRES_PASSWORD: psltest
EOF

Apply it.



kubectl apply -f postgres-config.yaml

Now create the postgres-deployment.yaml.



cat <<EOF > postgres-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: postgres  # Sets Deployment name
spec:
  replicas: 1
  selector:
    matchLabels:
      app: postgres
  template:
    metadata:
      labels:
        app: postgres
    spec:
      containers:
        - name: postgres
          image: postgres:10.1 # Sets Image
          imagePullPolicy: "IfNotPresent"
          ports:
            - containerPort: 5432  # Exposes container port
          envFrom:
            - configMapRef:
                name: postgres-config

EOF

Apply it.



kubectl apply -f postgres-deployment.yaml

And finally, let's create a service, so other workloads can access it:



cat <<EOF > postgres-service.yaml
apiVersion: v1
kind: Service
metadata:
  name: postgres # Sets service name
  labels:
    app: postgres # Labels and Selectors
spec:
  type: NodePort # Sets service type
  ports:
    - port: 5432 # Sets port to run the postgres application
  selector:
    app: postgres

EOF

Apply it.



kubectl apply -f postgres-service.yaml

Preparing DB with new readonly role

Exec into the psql pod.



kubectl get pods # get pod name
kubectl exec -it <postgres-pod-name> -- bash

Change into postgres user, and run commands to create the new role.



su postgres
psql -c "CREATE ROLE \"ro\" NOINHERIT;"
psql -c "GRANT SELECT ON ALL TABLES IN SCHEMA public TO \"ro\";"

We are going to use this role when configuring Vault to use Dynamic Secrets with psql plugin.

Vault Dynamic Secrets

Vault Dynamic secrets are in fact meant to be used as a way to get short lived credentials. However, there is nothing stopping us from using them in our auto-rotation process. There are various other plugins that integrate with other systems, like AWS credentials, or certificate issuing systems. Most of these are also interesting in the context of ESO, but I wanted a self-contained example with no need to create external accounts for you to try it out.

Lets first enable the database engine.



vault secrets enable database

After that, let's configure PostgreSQL secrets engine, with the admin creds we had before (we are passing credentials into the connection url here, never do that outside of test labs).



## POSTGRES_URL with name of the service and namespace
export POSTGRES_URL=postgres.default.svc.cluster.local:5432

vault write database/config/postgresql \
     plugin_name=postgresql-database-plugin \
connection_url="postgresql://admin:psltest@$POSTGRES_URL/postgres?sslmode=disable" \
     allowed_roles=readonly \
     username="root" \
     password="rootpassword"

Create an SQL file container the templated command that will be used by Vault when dynamically creating roles.



tee readonly.sql <<EOF
CREATE ROLE "{{name}}" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}' INHERIT;
GRANT ro TO "{{name}}";
EOF

Write that into Vault and configure default expiration of new requested roles and other fields (this will fail if you did not create the ROLE 'ro' correctly while setting up psql).



vault write database/roles/readonly \
      db_name=postgresql \
      creation_statements=@readonly.sql \
      default_ttl=1h \
      max_ttl=24h

You can already check within Vault if you can get the temporary credentials before setting up other steps.



vault read database/creds/readonly
## response
Key                Value
---                -----
lease_id           database/creds/readonly/CPqcUrG55f8qfrA9QKMV3peO
lease_duration     1h
lease_renewable    true
password           5p-xDWSC5Iu9z-hlZPrs
username           v-root-readonly-SQjhNhGxxmKx9QaRKsxM-1690473242

ESO Generator and ExternalSecret

Before next steps we are going to base64 encode the token so we can apply it with a secret. Grab you Vault token and echo it into base64.



echo "somethinsomething" | base64

Now we can use the new External Secrets Operator CRD, the Generator. Use the value outputted above for the auth token secret (vault-token).



cat <<EOF > vaultDynamicSecret.yaml
apiVersion: generators.external-secrets.io/v1alpha1
kind: VaultDynamicSecret
metadata:
  name: "psql-example"
spec:
  path: "/database/creds/readonly" ## this is how you choose which vault dynamic path to use
  method: "GET" ## this path will only work with GETs
  # parameters: ## no needed parameters 
  # ...
  provider:
    server: "http://vault.vault.svc.cluster.local:8200" ## vault url. In this case vault service on the vault namespace
    auth:
      # points to a secret that contains a vault token
      # https://www.vaultproject.io/docs/auth/token
      tokenSecretRef: ## reference to the secret holding the Vault auth token
        name: "vault-token"
        key: "token"
---
apiVersion: v1
kind: Secret
metadata:
  name: vault-token
data:
  token: aHZzLkM4M0o2UWNQSW1YQkRJVU96aWNNNzVHdwo= ## token base64 encoded
EOF

Apply this file.



kubectl apply -f vaultDynamicSecret.yaml

And finally we can now create our ExternalSecret that in the end will let the operator create the final Kubernetes Secret.



cat <<EOF > vaultDynamicSecret.yaml
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: "psql-example-es"
spec:
  refreshInterval: "1h" ## the same as the expiry time on the dynamic config of Vault, or lower, so apps have always new valid credentials
  target:
    name: psql-example-for-use ## the final name of the kubernetes secret created in your cluster
  dataFrom:
  - sourceRef:
      generatorRef:
        apiVersion: generators.external-secrets.io/v1alpha1
        kind: VaultDynamicSecret
        name: "psql-example" ## reference to the generator
EOF

Apply this and check if the status of the ExternalSecret is ok.



kubectl get externalsecret
## response
NAME              STORE   REFRESH INTERVAL   STATUS         READY
psql-example-es           1h                 SecretSynced   True

If you get errors here, verify that you used the right path in the Generator. Also check that you created the right roles inside psql and you can ping vault from a pod in the ESO namespace.

Checking the final secret

You should get a secret containing new users and passwords with read-only access to the database every hour.



kubectl get secrets psql-example-for-use -o jsonpath="{.data}"
## response
{"password":"V2lSWUlqZzdvQS1yOTFaV2N1SWE=","username":"di1yb290LXJlYWRvbmx5LVlXQ3kzZ01hbkhSbGtuY3FqTUg2LTE2OTA0NzIwMzc="}

To check one of the values you can get it and base64 decode it.



kubectl get secrets psql-example-for-use -o jsonpath="{.data.password}" | base64 -d

Now your application can use this secret, it will be automatically auto-rotated, and still be a valid credential to the database.

Caveats

If you use secrets as Environment Variables you will need to use something to make workloads get the new credentials, if they just loose connection. You can use the Reloader project for that.

If you use secrets as volumes, pods will get that update automatically, and you won't have problems connecting, as long as your application can get the new values.

Conclusion

That's it! We've set up an auto-rotating secret for a database connection using ESO and Vault. The magic is, as we said, you can set it once and forget. Your secret refreshes every hour and your app stays connected to the database with new valid credentials. It is secure, you follow best practices with regard to rotation, and you avoid manual intervention if that is not needed.

As a Dev, you should know Ops

Lucas Severo Alves — Mon, 03 Jun 2019 23:13:44 +0000

Cause DevOps.

Some would say that you should learn it simply because you should, end of story. But it’s not that simple. Regardless of your reasons, I think that you should learn it to be a better dev and to boost your technical career.

Lets quickly look at DevOps

Think about the traditional Dev vs Ops goals that existed “long ago” (Hopefully, you’re past dealing with this today): Devs were rewarded for shipping features and making changes, while Ops were rewarded by keeping things stable, and thus change was an enemy.

A massive number of methodologies were created, or borrowed, to get rid of this conflict, like Blameless Postmortems and Just Culture. Another example is error-budgeting, in which SLAs and SLOs define how stable something has to be, and also how unstable it 'must' be. This is done to cut costs while encouraging change.

These things, paired with the LEAN practices, reflect the cultural and organizational pillars of DevOps. They are crucial, but often disregarded by some people. The other pillars use automation to cut down on repetitive tasks, and improve intelligent decision making with better data through monitoring, observing, and using this data as a guide.

With all that in mind, I would say that there is a lot more about DevOps than just Devs knowing Ops.

So, why would you, as a Dev, need to know some Ops, if not because of DevOps ? I’ll now try to give you four reasons that answer this question in a more career-oriented vibe. Maybe your company or your current position doesn’t require this knowledge, but knowing it will make you a better Dev overall, helping you write better services and giving you more “power”.

1- You can be part of important technical decisions/discussions

Ops knowledge can open many doors for you. If your company has an architecture-steering committee or something similar, having such knowledge may play a big role in you becoming part of it, or even in bringing things for the committee to discuss. You may have a say in decisions related to stack, architecture, and even business.

Let’s say your company has a bit of a delay when it comes to response in some situations and in some services. One possible solution is to re-implement some of these services and part of the architecture. However, this could be very time consuming and demand a lot of effort. A better option might be to do a mix of software and infrastructure architecture change, hopefully with less effort. You might arrive at this solution by yourself from having a “mixed bag” of knowledge. You can then bring more people in for discussion on the topic.

2- You can write better services

In the same way that you can provide better answers to problems that emerge in your services regarding its architecture and code, Ops knowledge may allow you to foresee problems that are bound to happen.

Your development can be really modular and focused on the thing your service is solving. But the ability to visualize problems regarding how your service interacts with other services and with the systems holding it is crucial for applications with really high SLOs and SLAs. This makes your services both reliable and available.

Hopefully, with some Ops or System Administrator experience under your belt, you will also feel the need to keep your services observable. With operability and observability you make possible to have people on call with less stress.

3- Being on call can make you foolproof when coding

Here I am not talking about just Ops knowledge, but also Ops experience with being on call. With enough of it, you start to see the weak links and parts most prone to failure in your system. This gets to you, you start feeling it under your skin, and when you go back to coding , it should influence you to write better code.

This one is almost the same as the second item, but I believe that the experience of being on call brings something that only it can bring: empathy for service operability. The second item refers to Ops knowledge you can acquire by reading or listening to people. This one refers to the knowledge acquired by your own experience and how this can later be used in writing code.

4- You can be part of all the life cycle of your code

You can write your code, and test it; you know how to CI/CD it, and you know how to operate it in the production. You also know how to observe everything. I’m not saying you will be doing all these things every time, but you have the know-how, and may have done it before.

Your company shouldn’t expect everyone to know all of these things. But knowing them might reward you greatly. And thus, this pursuit should be encouraged by your company.

So, how do we go about this?

I hope I have convinced you to learn some Ops. If you don’t know how, here are some options:

References

I like the opsschool readthedocs. There are a lot of topics missing content, but you can still use it as a list of topics to further research on the internet, and maybe get some books and articles on.

There are lots of specific tools-focused books that I could suggest, but you’d be better off looking for them based on what your company uses and what you want to learn.

For more general learning you could get The Practice of Cloud System Administration and UNIX Linux System Administration Handbook.

For Linux, Linux Programming Interface System Handbook is a great book. Mostly to be used as reference.

For networking you can read Computer Networks by Tanenbaum.

You can read this references about observability: Honeycombio Guide to Achieving Observability and Distributed Systems Observability.

I know this seems like a lot, but don’t be intimidated by it all! Remember, there’s no need to read all of it. And here’s the actual best way to learn Ops:

Just do it!

See if your company gives you the option to try other positions and go for it. Ask for guidance and try to stay on call sometimes. If you have people around you who know something, talking to them and sharing what you know might be the best way to learn. So long as you have solid development skills, the addition of Ops experience could help you achieve even greater things!