DEV Community

Janak Shrestha
Janak Shrestha

Posted on

Day 64: Fix Python App Deployed on Kubernetes Cluster

#kubernetes #devops

One of the DevOps engineers was trying to deploy a python app on Kubernetes cluster. Unfortunately, due to some mis-configuration, the application is not coming up. Please take a look into it and fix the issues. Application should be accessible on the specified nodePort.

  1. The deployment name is python-deployment-datacenter, its using poroko/flask-demo-app image. The deployment and service of this app is already deployed.

  2. nodePort should be 32345 and targetPort should be python flask app's default port.

Understanding the Problem

Initial Status

When we first examined the deployment, we found the following issues:

Component Status Issue
Deployment 0/1 READY Pod failing to start
Pod ImagePullBackOff Cannot pull container image
Service Active Wrong port configuration

Root Causes

  1. Incorrect Image Name: The deployment was using poroko/flask-app-demo, but the correct image is poroko/flask-demo-app
  2. Wrong Service Port: The service was configured on port 8080, but Flask applications listen on port 5000 by default

Diagnostic Process

Step 1: Check Deployment Status

First, we examined the current state of the deployment:

kubectl get deployments
Enter fullscreen mode Exit fullscreen mode

Output:

NAME                           READY   UP-TO-DATE   AVAILABLE   AGE
python-deployment-datacenter   0/1     1            0           86s
Enter fullscreen mode Exit fullscreen mode

The deployment was created but had zero ready replicas.

Step 2: Check Pod Status 

kubectl get pods
Enter fullscreen mode Exit fullscreen mode

Output:

NAME                                           READY   STATUS             RESTARTS   AGE
python-deployment-datacenter-65648dc9d-jqxmw   0/1     ImagePullBackOff   0          103s
Enter fullscreen mode Exit fullscreen mode

The pod was in ImagePullBackOff status, indicating Kubernetes could not pull the container image.

Step 3: Check Service Status 

kubectl get services
Enter fullscreen mode Exit fullscreen mode

Output:

NAME                        TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)          AGE
python-service-datacenter   NodePort    10.43.165.47   <none>        8080:32345/TCP   110s
Enter fullscreen mode Exit fullscreen mode

The service was exposed on port 8080, but Flask applications use port 5000.

Step 4: Investigate Pod Details 

kubectl describe pod python-deployment-datacenter-65648dc9d-jqxmw
Enter fullscreen mode Exit fullscreen mode

Key Output:

Containers:
  python-container-datacenter:
    Image:          poroko/flask-app-demo
    State:          Waiting
      Reason:       ImagePullBackOff
Events:
  Warning  Failed  Failed to pull image "poroko/flask-app-demo": 
           pull access denied, repository does not exist
Enter fullscreen mode Exit fullscreen mode

The error message clearly indicated that the image poroko/flask-app-demo does not exist on Docker Hub.

Step 5: Verify Deployment Image 

kubectl describe deployment python-deployment-datacenter | grep -i image
Enter fullscreen mode Exit fullscreen mode

Output:

Image: poroko/flask-app-demo
Enter fullscreen mode Exit fullscreen mode

The deployment was configured with the incorrect image name.

Implementing the Fix

Fix 1: Correct the Image Name

The image name was misspelled. The correct image is poroko/flask-demo-app.

kubectl edit deployment python-deployment-datacenter
Enter fullscreen mode Exit fullscreen mode

Change:

image: poroko/flask-app-demo
Enter fullscreen mode Exit fullscreen mode

To:

image: poroko/flask-demo-app
Enter fullscreen mode Exit fullscreen mode

Fix 2: Correct the Service Port

The service was configured on port 8080, but Flask applications listen on port 5000.

kubectl edit service python-service-datacenter
Enter fullscreen mode Exit fullscreen mode

Change:

ports:
  - port: 8080
    targetPort: 8080
    nodePort: 32345
Enter fullscreen mode Exit fullscreen mode

To:

ports:
  - port: 5000
    targetPort: 5000
    nodePort: 32345
Enter fullscreen mode Exit fullscreen mode

Verification

Step 1: Check Pod Status 

kubectl get pods
Enter fullscreen mode Exit fullscreen mode

Output:

NAME                                            READY   STATUS    RESTARTS   AGE
python-deployment-datacenter-57d654488b-fq6w9   1/1     Running   0          20m
Enter fullscreen mode Exit fullscreen mode

The pod is now running successfully.

Step 2: Check Service Status 

kubectl get services
Enter fullscreen mode Exit fullscreen mode

Output:

NAME                        TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)          AGE
python-service-datacenter   NodePort    10.43.165.47   <none>        5000:32345/TCP   27m
Enter fullscreen mode Exit fullscreen mode

The service is now correctly configured on port 5000.

Step 3: Check Deployment Status 

kubectl get deployments
Enter fullscreen mode Exit fullscreen mode

Output:

NAME                           READY   UP-TO-DATE   AVAILABLE   AGE
python-deployment-datacenter   1/1     1            1           27m
Enter fullscreen mode Exit fullscreen mode

The deployment is now fully ready.

Step 4: Test the Application 

curl https://32345-port-ryzfgyyrjvx3b7jf.labs.kodekloud.com/
Enter fullscreen mode Exit fullscreen mode

Output:

Hello World Pyvo 1!
Enter fullscreen mode Exit fullscreen mode

The application is now accessible and returning the expected response.

Troubleshooting Commands Reference

Diagnostic Commands

Command Purpose
kubectl get deployments Check deployment status
kubectl get pods Check pod status
kubectl get services Check service status
kubectl describe pod <pod-name> Detailed pod information
kubectl describe deployment <deployment-name> Detailed deployment information
kubectl describe service <service-name> Detailed service information
kubectl get events --sort-by='.lastTimestamp' View recent events
kubectl logs <pod-name> View pod logs

Fix Commands

Command Purpose
kubectl edit deployment <name> Edit deployment configuration
kubectl edit service <name> Edit service configuration
kubectl rollout restart deployment <name> Restart deployment
kubectl rollout status deployment <name> Check rollout status

Common Kubernetes Issues and Solutions

ImagePullBackOff

Symptom Possible Causes Solutions
Pod in ImagePullBackOff Wrong image name Verify image name and tag
Image does not exist Check Docker Hub for image
Authentication required Configure imagePullSecrets
Network issues Check registry connectivity

Wrong Port Configuration

Symptom Possible Causes Solutions
Service not routing traffic Wrong targetPort Match targetPort with containerPort
Wrong port Use correct application port
Label mismatch Ensure labels match selector

CrashLoopBackOff

Symptom Possible Causes Solutions
Pod continuously restarts Application errors Check pod logs
Wrong command Verify command and arguments
Missing dependencies Check application configuration

Best Practices

1. Image Management

  • Use specific image tags, not latest
  • Verify image names before deployment
  • Use private registries with proper authentication

2. Service Configuration

  • Match targetPort with containerPort
  • Use meaningful port numbers
  • Document port mappings

3. Troubleshooting Approach

  1. Check Status: Start with kubectl get all
  2. Investigate: Use kubectl describe for details
  3. Review Logs: Check application logs
  4. Verify Configuration: Compare with expected values
  5. Apply Fixes: Make targeted changes
  6. Validate: Confirm the fix resolves the issue

4. Deployment Strategies

  • Use rolling updates for zero downtime
  • Test changes in staging environments
  • Have rollback plans ready

Summary

In this challenge, we successfully:

  1. Identified the root causes of the deployment failure:

    • Incorrect image name (poroko/flask-app-demo should be poroko/flask-demo-app)
    • Wrong service port (8080 should be 5000)

  2. Applied targeted fixes:

    • Updated the deployment with the correct image name
    • Corrected the service port configuration

  3. Verified the fixes:

    • Pod is now running
    • Service is correctly configured
    • Application is accessible

This exercise demonstrates the importance of careful configuration and systematic troubleshooting when deploying applications on Kubernetes. The skills learned here are essential for maintaining production workloads in containerized environments.

Top comments (0)