🇧🇷 Versão em português aqui

Building an On-Premise Kubernetes Cluster - Part 7: Essential Production Features

In previous parts of this series, we built the cluster from scratch and learned the basic structure of YAML manifests. But there's a real gap between "a Deployment that works" and "a Deployment ready for production." In this article, I cover a set of features I consider essential for that second scenario: Namespaces, Secrets, private registry authentication (regcred), RollingUpdate, SecurityContext, and Requests/Limits. At the end, we bring everything together in a real, complete Deployment example.

Namespace

A Namespace is a way to logically divide a single physical cluster into multiple "virtual clusters", useful for separating environments (production, staging, development) or different applications, preventing them from growing mixed together in a single space.

Creating a namespace:

$ kubectl create namespace aplicacao-prod

or

$ kubectl create ns aplicacao-prod

Or via manifest:

apiVersion: v1
kind: Namespace
metadata:
  name: aplicacao-prod

Listing existing namespaces:

$ kubectl get namespaces

or

$ kubectl get ns

From there, practically every object created (Deployments, Services, Secrets) can, and generally should, declare which namespace it lives in, through the metadata.namespace field or using the -n flag if it using by command line. This brings important benefits:

• Name isolation: two Deployments named api can coexist peacefully, as long as they're in different namespaces;
• Access control (RBAC): you can restrict what a user or an application can do, scoped to a specific namespace;
• Organization: commands like kubectl get pods -n aplicacao-prod make it clear what belongs to which environment or application.

If you don't specify a namespace, Kubernetes uses the default namespace, which works for quick tests, but isn't recommended for real environments where multiple applications share the cluster.

Secrets

Secrets store sensitive data: passwords, tokens, API keys, in a slightly more protected way than a regular ConfigMap (the content is base64-encoded, and access can be restricted via RBAC).

Creating a Secret via the command line, from literal values:

kubectl create secret generic aplicacao-secret \
  --namespace=aplicacao-prod \
  --from-literal=DB_PASSWORD=mypassword \
  --from-literal=API_KEY=abc123

Or via a YAML manifest:

apiVersion: v1
kind: Secret
metadata:
  name: aplicacao-secret
  namespace: aplicacao-prod
type: Opaque
data:
  DB_PASSWORD: bXlwYXNzd29yZA==
  API_KEY: YWJjMTIz

Values in the data field need to be base64-encoded (echo -n 'mypassword' | base64). This isn't encryption, it's just an encoding, so treat the Secret's YAML file with the same care you'd treat a plain-text password file. Never commit (Git) a Secret with real values without some additional layer of protection, like Sealed Secrets or a secrets vault (Vault).

Consuming the Secret in the application

The most common approach is injecting the Secret's entire content as environment variables, using envFrom:

containers:
  - name: aplicacao
    image: mycompany/api:1.0.0
    envFrom:
      - secretRef:
          name: aplicacao-secret

This way, every key in the Secret (DB_PASSWORD, API_KEY) automatically becomes an environment variable inside the container, without needing to list them one by one.

Regcred - Credentials for a Private Registry

When an application's images are stored in a private registry, whether a self-hosted registry, or a paid plan on Docker Hub or another provider, Kubernetes needs credentials to be able to pull those images when creating pods. That's what the imagePullSecret is for, commonly created under the name regcred.

The safest approach is to use a GitLab Deploy Token (instead of your personal password), it can be scoped to read-only permission (read_registry) and restricted to a specific project or group.

Creating the secret with the Deploy Token:

There are two ways to create this secret: via the command line (faster) or via a YAML file (more suitable when you need to version or automate its creation).

Option 1: creating via command line

kubectl create secret docker-registry regcred \
  -n aplicacao-prod \
  --docker-server=registry.yourcompany.com \
  --docker-username=<deploy-token-username> \
  --docker-password=<deploy-token-password> \
  --docker-email=[e-mail]

Where:

• -n namespace: namespace where the secret will be created (the secret is scoped to that namespace);
• --docker-server: private registry address;
• --docker-username / --docker-password: access credentials;
• --docker-email: email associated with the account (required by the command, even if not always used in practice).

Option 2: creating via YAML file

If you'd rather (or need to) version this configuration, you can manually build the secret as a YAML file.

1. Create a config.json file with the registry credentials:

{
    "auths": {
        "registry.yourcompany.com": {
      "username": "<deploy-token-username>",
      "password": "<deploy-token-password>",
      "email": "[e-mail]"
    }
    }
}

2. Convert the file to base64, since that's how Kubernetes expects the secret content:

$ cat config.json | base64 -w 0

3. Create the regcred.yaml file, pasting the result of the command above into the .dockerconfigjson field:

apiVersion: v1
kind: Secret
metadata:
  name: regcred
  namespace: namespace
type: kubernetes.io/dockerconfigjson
data:
  .dockerconfigjson: ewoJImF1dGhzIjogewoJCSJyZWdpc3RyeS5zdWFlbXByZXNhLmNvbSI6IHsKICAgICAgInVzZXJuYW1lIjogIjxkZXBsb3ktdG9rZW4tdXNlcm5hbWU+IiwKICAgICAgInBhc3N3b3JkIjogIjxkZXBsb3ktdG9rZW4tcGFzc3dvcmQ+IiwKICAgICAgImVtYWlsIjogInNldS1lbWFpbEBzdWFlbXByZXNhLmNvbSIKICAgIH0KCX0KfQo=

Never commit (Git) a regcred.yaml file with real credentials: even though it's base64-encoded, that's not encryption, just an encoding: anyone with access to the file can decode it and recover the original password. Treat this value as a genuinely sensitive secret, using tools like Sealed Secrets, Vault, or environment variables from your CI/CD pipeline.

4. Apply the secret to the cluster:

$ kubectl apply -f regcred.yaml

Using regcred in a Deployment

Reference the secret in the Deployment through imagePullSecrets:

spec:
  imagePullSecrets:
    - name: regcred
  containers:
    - name: aplicacao
      image: registry.gitlab.com/organization/aplicacao:v1.0.0

The imagePullSecrets field points to the regcred secret created earlier, and it needs to be in the same namespace as the Pod or Deployment that will use it.

RollingUpdate

By default, a Deployment already uses the RollingUpdate strategy when updating an application's version, gradually replacing old pods with new ones, without full downtime. But the exact parameters of this process can (and should) be tuned:

spec:
  minReadySeconds: 10
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 1

• maxSurge: how many pods above the desired replica count can be temporarily created during the update (here, 1 extra pod);
• maxUnavailable: how many pods can be unavailable at the same time during the update (here, at most 1);
• minReadySeconds: the minimum time (in seconds) a new pod needs to stay "ready" (passing health checks) before being considered available, this prevents the rollout from moving too fast with pods that came up but haven't stabilized yet.

These values should be tuned based on the application's criticality: more conservative values (maxUnavailable: 0) guarantee zero downtime during deployment, at the cost of requiring more temporary capacity in the cluster (via maxSurge).

I have encountered downtime issues with Java applications because the new replica took too long to become available.

SecurityContext

The securityContext defines security restrictions on how a Pod or container can run, reducing the attack surface in case the application is compromised. It can be set both at the Pod level (applied to all containers) and at the container level (more specific, overrides the Pod's setting).

spec:
  securityContext:
    runAsUser: 5000
    runAsGroup: 5000
  containers:
    - name: aplicacao
      securityContext:
        allowPrivilegeEscalation: false

• runAsUser / runAsGroup (Pod level): forces processes to run with a specific UID/GID, instead of root, reduces potential damage if an attacker manages to execute code inside the container;
• allowPrivilegeEscalation: false (container level): prevents a process inside the container from gaining more privileges than it already has (for example, through binaries with the setuid bit).

Other common options worth considering, depending on the application:

• readOnlyRootFilesystem: true: makes the container's root filesystem read-only, forcing any writes to go to explicitly mounted volumes;
• capabilities.drop: ["ALL"]: removes all Linux capabilities by default, adding back only the ones strictly necessary.

Requests / Limits

Without defining requests and limits, a container could, in theory, consume all resources available on the node it's running on, hurting other applications on the same cluster (or even bringing down the node itself).

resources:
  requests:
    memory: "128Mi"
    cpu: "100m"
  limits:
    memory: "512Mi"
    cpu: "1000m"

• requests: the amount of resources Kubernetes reserves for the container when deciding which node to schedule it on. The scheduler only places the pod on a node that has that amount available;
• limits: the maximum ceiling the container can consume. Exceeding the memory limit results in the container being killed (OOMKilled); exceeding the CPU limit results in throttling (the container is slowed down, but not killed).

100m of CPU means 100 millicores, i.e., 0.1 of a core. Setting realistic requests (based on the application's observed usage) matters, requests that are too high waste cluster capacity; requests that are too low can lead to more pods running simultaneously than a node can actually support.

Real-world application Deployment example

Bringing together everything covered in this part: a dedicated namespace, secrets, private registry authentication, a tuned rolling update, a restrictive securityContext, and defined requests/limits, plus the Service and HorizontalPodAutoscaler already covered in previous parts:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: aplicacao
  namespace: aplicacao-prod
spec:
  minReadySeconds: 10
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 1
  replicas: 2
  selector:
    matchLabels:
      app: aplicacao
  template:
    metadata:
      labels:
        app: aplicacao
    spec:
      securityContext:
        runAsUser: 5000
        runAsGroup: 5000
      containers:
      - name: aplicacao
        image: registry.yourcompany.com/organization/api:v9346
        envFrom:
          - secretRef:
              name: aplicacao-secret
        securityContext:
          allowPrivilegeEscalation: false
        resources:
          requests:
            memory: "128Mi"
            cpu: "100m"
          limits:
            memory: "512Mi"
            cpu: "1000m"
        ports:
        - containerPort: 8080
      imagePullSecrets:
        - name: regcred
---
apiVersion: v1
kind: Service
metadata:
  name: aplicacao
  namespace: aplicacao-prod
spec:
  selector:
    app: aplicacao
  ports:
  - protocol: TCP
    port: 8080
    targetPort: 8080
  type: NodePort
---
apiVersion: autoscaling/v1
kind: HorizontalPodAutoscaler
metadata:
  name: aplicacao-hpa
  namespace: aplicacao-prod
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: aplicacao
  minReplicas: 2
  maxReplicas: 5
  targetCPUUtilizationPercentage: 75

The three objects work together:

• The Deployment runs the application with 2 replicas, authenticating with the private registry via regcred and injecting sensitive configuration via aplicacao-secret;
• The Service exposes those replicas in a stable way on port 8080;
• The HorizontalPodAutoscaler monitors those replicas' CPU usage and automatically adjusts between 2 and 5, based on demand.

Final thoughts

These features: namespaces, secrets, secure registry authentication, a tuned rollout strategy, security restrictions, and resource limits, are typically what separates a standard manifest from one ready to run a application in production. It's worth reviewing each of these points whenever you're about to promote an application from staging to production.