DEV Community

Yash Sonawane
Yash Sonawane

Posted on

Kubernetes in Production: 10 Lessons I Wish I Knew Before Deploying My First Application

Kubernetes is easy to understand in a tutorial.

You create a Deployment:

replicas: 3
Enter fullscreen mode Exit fullscreen mode

Create a Service:

kind: Service
Enter fullscreen mode Exit fullscreen mode

Run:

kubectl apply -f .
Enter fullscreen mode Exit fullscreen mode

And everything works.

Then you deploy a real application.

Suddenly you have questions:

  • What happens when a Pod crashes?
  • What if the node goes down?
  • How do I update the application without downtime?
  • How do I manage secrets?
  • How do I control CPU and memory?
  • How do I monitor the cluster?
  • How do I troubleshoot networking?
  • How do I safely deploy a new version?

This is where Kubernetes changes from a learning topic into an engineering discipline.

In this article, let's go beyond kubectl get pods and look at some practical lessons for running Kubernetes workloads.


1. Don't Treat Pods Like Servers

One of the biggest mindset changes in Kubernetes is this:

A Pod is disposable.

In a traditional server environment, you might think:

Server
 ↓
Application
 ↓
Keep it alive
Enter fullscreen mode Exit fullscreen mode

In Kubernetes, think:

Desired State
      ↓
Deployment
      ↓
Pods
      ↓
Containers
Enter fullscreen mode Exit fullscreen mode

If a Pod disappears, Kubernetes can create another one.

Therefore, your application should not depend on the identity of a particular Pod.

Don't design your system around:

"Pod abc123 must always exist."
Enter fullscreen mode Exit fullscreen mode

Instead:

"At least 3 healthy replicas should exist."
Enter fullscreen mode Exit fullscreen mode

That's a completely different mindset.


2. Always Define Resource Requests and Limits

One of the easiest mistakes is deploying workloads without thinking about resources.

You might write:

containers:
  - name: app
    image: my-app:latest
Enter fullscreen mode Exit fullscreen mode

But how much CPU does it need?

How much memory?

What happens when several applications compete for resources?

A better starting point is:

resources:
  requests:
    cpu: "250m"
    memory: "256Mi"

  limits:
    cpu: "500m"
    memory: "512Mi"
Enter fullscreen mode Exit fullscreen mode

Requests help Kubernetes make scheduling decisions.

Limits establish boundaries for container resource usage.

The exact values should come from measurement rather than blindly copying numbers from tutorials.


3. Never Use latest Blindly

You might see:

image: myapp:latest
Enter fullscreen mode Exit fullscreen mode

It looks convenient.

But production deployments benefit from predictable image versions.

For example:

image: myapp:1.4.2
Enter fullscreen mode Exit fullscreen mode

or a controlled immutable identifier such as a digest.

Why?

Because you want to know exactly what you're deploying.

Imagine:

Monday:
latest → version 1.4

Tuesday:
latest → version 1.5
Enter fullscreen mode Exit fullscreen mode

The same Kubernetes manifest can now deploy different application code.

That makes debugging harder.

Predictability matters.


4. Health Checks Are Not Optional in Serious Deployments

Your container being alive doesn't necessarily mean your application is healthy.

Imagine:

Container: Running
Application: Database connection broken
Enter fullscreen mode Exit fullscreen mode

Kubernetes sees a running process.

Users see a broken application.

This is where probes become useful.

Readiness

Answers:

"Can this Pod receive traffic?"

Liveness

Answers:

"Should this container continue running?"

Example:

readinessProbe:
  httpGet:
    path: /health
    port: 8080

livenessProbe:
  httpGet:
    path: /health
    port: 8080
Enter fullscreen mode Exit fullscreen mode

The endpoints and timings should be designed around your actual application behavior.

Don't simply copy probe configurations from another project.


5. Use Services Instead of Depending on Pod IPs

Pod IPs are not permanent.

Imagine:

backend-pod
10.0.1.15
Enter fullscreen mode Exit fullscreen mode

The Pod crashes.

Kubernetes creates:

new-backend-pod
10.0.2.27
Enter fullscreen mode Exit fullscreen mode

If your frontend was directly calling:

10.0.1.15
Enter fullscreen mode Exit fullscreen mode

your architecture breaks.

Instead, use a Service:

Frontend
   ↓
backend-service
   ↓
┌──────┬──────┬──────┐
│ Pod  │ Pod  │ Pod  │
└──────┴──────┴──────┘
Enter fullscreen mode Exit fullscreen mode

The Service provides a stable abstraction while Pods can come and go.


6. Separate Configuration From Application Code

Your application code shouldn't need to change just because you're moving from:

Development
Enter fullscreen mode Exit fullscreen mode

to:

Production
Enter fullscreen mode Exit fullscreen mode

Configuration such as:

DATABASE_HOST
API_URL
LOG_LEVEL
Enter fullscreen mode Exit fullscreen mode

can be managed separately.

Kubernetes provides:

  • ConfigMaps
  • Secrets

For example:

Application Image
       +
Configuration
       ↓
Running Container
Enter fullscreen mode Exit fullscreen mode

This separation makes deployments easier to manage.

For sensitive credentials, also consider dedicated secrets-management solutions when appropriate.


7. Deployments Should Be Reversible

Imagine you deploy version 2.0.

Then users start reporting errors.

Your deployment process should give you a controlled way to recover.

Kubernetes Deployments maintain rollout history.

You can inspect it:

kubectl rollout history deployment/my-app
Enter fullscreen mode Exit fullscreen mode

And, when appropriate, roll back:

kubectl rollout undo deployment/my-app
Enter fullscreen mode Exit fullscreen mode

A production deployment process should answer:

"What happens if the new version is broken?"

before the incident happens.


8. Don't Ignore Observability

If your application is running in production, eventually someone will ask:

"Why is it slow?"

or:

"Why did requests start failing?"

If all you have is:

kubectl get pods
Enter fullscreen mode Exit fullscreen mode

you're going to struggle.

You need visibility into things such as:

Logs
Metrics
Events
Latency
Error rates
CPU
Memory
Request volume
Enter fullscreen mode Exit fullscreen mode

A common observability stack in Kubernetes environments can include:

Prometheus
     ↓
Metrics

Grafana
     ↓
Visualization

Application Logs
     ↓
Log Aggregation
Enter fullscreen mode Exit fullscreen mode

The exact stack depends on the organization and workload.

The important principle is:

You can't reliably operate what you can't observe.


9. Security Should Be Designed, Not Added at the End

A Kubernetes cluster isn't automatically secure just because it's Kubernetes.

Think about:

RBAC
Network Policies
Secrets
Container Images
Pod Security
Service Accounts
API Access
Image Scanning
Enter fullscreen mode Exit fullscreen mode

For example, don't give every workload unnecessary permissions.

Use the principle:

Least privilege.

If an application only needs to read a particular resource, don't automatically give it broad administrative access.

Security should be part of the architecture from the beginning.


10. Automate Repetitive Deployments

Imagine manually running:

kubectl apply -f deployment.yaml
kubectl apply -f service.yaml
kubectl apply -f ingress.yaml
Enter fullscreen mode Exit fullscreen mode

every time an application changes.

It works.

But eventually you want automation.

A common workflow is:

Developer
    ↓
Git Push
    ↓
CI Pipeline
    ↓
Tests
    ↓
Build Image
    ↓
Security Scan
    ↓
Push Image
    ↓
Deploy
    ↓
Kubernetes
Enter fullscreen mode Exit fullscreen mode

Tools such as GitHub Actions, Jenkins, GitLab CI, Argo CD, and others can participate in different parts of this workflow.

The goal isn't to use every tool.

The goal is to make deployments:

repeatable, observable, and controlled.


🧠 The Real Kubernetes Mental Model

After learning all these pieces, step back.

A production application might look like:

                       Users
                         │
                         ▼
                     Ingress
                         │
                         ▼
                      Service
                         │
              ┌──────────┼──────────┐
              ▼          ▼          ▼
            Pod        Pod        Pod
              │          │          │
              └──────────┼──────────┘
                         │
                  Application
                         │
              ┌──────────┴──────────┐
              ▼                     ▼
          Database                Cache
Enter fullscreen mode Exit fullscreen mode

And around the application you have:

Monitoring
Logging
Security
Autoscaling
CI/CD
Configuration
Secrets
Networking
Enter fullscreen mode Exit fullscreen mode

That's when Kubernetes starts to make sense as a platform rather than just a list of commands.


🔥 Build This Project Yourself

If you're learning Kubernetes, don't stop at tutorials.

Build a production-style project.

For example:

Cloud-Native E-Commerce Platform

Architecture:

                       Internet
                           │
                           ▼
                        Ingress
                           │
             ┌─────────────┴─────────────┐
             ▼                           ▼
        Frontend Service            API Service
             │                           │
             ▼                           ▼
        Frontend Pods               Backend Pods
                                         │
                              ┌──────────┼──────────┐
                              ▼          ▼          ▼
                           PostgreSQL  Redis      Queue
Enter fullscreen mode Exit fullscreen mode

Then add:

✓ Docker
✓ Kubernetes
✓ Helm
✓ ConfigMaps
✓ Secrets
✓ Ingress
✓ Resource Limits
✓ Health Probes
✓ Horizontal Pod Autoscaling
✓ RBAC
✓ Prometheus
✓ Grafana
✓ CI/CD
Enter fullscreen mode Exit fullscreen mode

Now you have something much more valuable than a collection of tutorial commands.

You have a project you can discuss in an interview.


🎯 How to Explain Kubernetes in an Interview

Don't say:

"Kubernetes is a container orchestration platform."

and stop.

Explain the architecture.

For example:

"Kubernetes manages containerized workloads using a desired-state model. The control plane manages cluster state, while worker nodes run workloads in Pods. Deployments manage replicated application Pods, Services provide stable networking, and components such as probes, resource controls, and autoscaling help operate applications reliably."

That's a much stronger understanding.


📚 Want to Learn Kubernetes Properly?

If you're preparing for Kubernetes administration or the Certified Kubernetes Administrator (CKA), I've created:

CKA Complete Study Guide — Certified Kubernetes Administrator

The goal is to give you a structured learning path instead of forcing you to jump between hundreds of disconnected tutorials.

📘 Get the CKA Complete Study Guide

CKA Complete Study Guide

The best approach is:

Learn
 ↓
Build
 ↓
Break
 ↓
Troubleshoot
 ↓
Automate
Enter fullscreen mode Exit fullscreen mode

🛠️ More Resources for Cloud & DevOps

🐳 Docker Mastery

Docker Mastery

🏗️ Terraform Associate Crash Course

Terraform Associate Crash Course

🔀 Git Mastery

Git Mastery

⚙️ DevOps Complete Pack

DevOps Complete Pack

🐹 Mastering Go

Mastering Go


Final Thoughts

Kubernetes isn't difficult because the commands are difficult.

It's difficult because you're learning to think about distributed systems.

You need to understand:

Containers
    ↓
Pods
    ↓
Deployments
    ↓
Services
    ↓
Networking
    ↓
Storage
    ↓
Security
    ↓
Observability
    ↓
Automation
Enter fullscreen mode Exit fullscreen mode

And eventually:

Application
     ↓
Infrastructure
     ↓
Automation
     ↓
Reliability
Enter fullscreen mode Exit fullscreen mode

That's the real journey from Kubernetes beginner → DevOps engineer.

So don't spend months only memorizing:

kubectl get
kubectl describe
kubectl delete
kubectl apply
Enter fullscreen mode Exit fullscreen mode

Build something.

Deploy it.

Break it.

Fix it.

Monitor it.

Automate it.

That's where Kubernetes knowledge turns into engineering skill.

Top comments (0)