DEV Community

Janak Shrestha
Janak Shrestha

Posted on

Resolve VolumeMounts Issue in Kubernetes

We encountered an issue with our Nginx and PHP-FPM setup on the Kubernetes cluster this morning, which halted its functionality. Investigate and rectify the issue:

The pod name is nginx-phpfpm and configmap name is nginx-config. Identify and fix the problem.

Once resolved, copy /home/thor/index.php file from the jump host to the nginx-container within the nginx document root. After this, you should be able to access the website using Website button on the top bar.

Introduction

Have you ever deployed a multi-container pod in Kubernetes only to have it fail mysteriously? You're not alone! One of the most common challenges when working with multi-container pods is correctly configuring shared volumes. In this comprehensive guide, we'll walk through a real-world scenario where an Nginx-PHP-FPM setup was broken due to volume mount misconfiguration, and learn how to fix it.

Table of Contents

  1. Understanding the Problem
  2. The Scenario: Nginx-PHP-FPM Setup
  3. Step-by-Step Troubleshooting
  4. Identifying the Root Cause
  5. The Solution
  6. Verification and Testing
  7. Common Mistakes and Lessons Learned
  8. Best Practices
  9. Conclusion

Understanding the Problem

The Challenge

When you have multiple containers in a single pod, they often need to share data. This is where Volumes come in. However, a common pitfall is mounting the same volume at different paths in different containers, leading to confusing errors.

The Symptoms

In our scenario, the application was returning "File not found" errors, even though:

  • ✅ The pod was running (2/2 READY)
  • ✅ The ConfigMap was correct
  • ✅ The Service was properly configured
  • ✅ The file existed in the container

Why This Happened 

┌─────────────────────────────────────────────────────────────┐
│                    The Problem                             │
├─────────────────────────────────────────────────────────────┤
│                                                             │
│  Volume: shared-files (same underlying storage)           │
│                                                             │
│  ┌─────────────────────────────────────────────────────┐   │
│  │  Nginx Container                                   │   │
│  │  Mount Path: /var/www/html  ✅                     │   │
│  │  root: /var/www/html        ✅                     │   │
│  └─────────────────────────────────────────────────────┘   │
│                                                             │
│  ┌─────────────────────────────────────────────────────┐   │
│  │  PHP-FPM Container                                  │   │
│  │  Mount Path: /usr/share/nginx/html  ❌              │   │
│  │                                                    │   │
│  │  ❌ Mismatch!                                     │   │
│  └─────────────────────────────────────────────────────┘   │
│                                                             │
│  Result: PHP-FPM can't find index.php → 404 Error        │
└─────────────────────────────────────────────────────────────┘
Enter fullscreen mode Exit fullscreen mode

The Scenario: Nginx-PHP-FPM Setup

What We Had

Component Name Details
Pod nginx-phpfpm Multi-container pod
Container 1 php-fpm-container PHP 7.2 FPM Alpine
Container 2 nginx-container Nginx latest
ConfigMap nginx-config Nginx configuration
Volume shared-files EmptyDir volume

The Goal

The task was to:

  1. Fix the broken Nginx-PHP-FPM setup
  2. Copy index.php to the correct location
  3. Make the website accessible via the "Website" button

Initial Symptoms 

$ kubectl get pods
NAME           READY   STATUS    RESTARTS   AGE
nginx-phpfpm   2/2     Running   0          44s

$ curl http://localhost:30008/
File not found.
Enter fullscreen mode Exit fullscreen mode

The pod was healthy, but the website was returning "File not found"!

Step-by-Step Troubleshooting

Step 1: Check Pod Status 

kubectl get pod nginx-phpfpm
kubectl describe pod nginx-phpfpm
Enter fullscreen mode Exit fullscreen mode

Output:

NAME           READY   STATUS    RESTARTS   AGE
nginx-phpfpm   2/2     Running   0          44s
Enter fullscreen mode Exit fullscreen mode

✅ Both containers are running. No errors in the Events section.

Step 2: Check the ConfigMap 

kubectl get configmap nginx-config -o yaml
Enter fullscreen mode Exit fullscreen mode

Key Findings:

nginx.conf: |
  server {
    listen 8099 default_server;    # Nginx listens on port 8099
    root /var/www/html;            # Document root is /var/www/html
    location ~ \.php$ {
      fastcgi_pass 127.0.0.1:9000; # PHP-FPM on port 9000
      fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
    }
  }
Enter fullscreen mode Exit fullscreen mode

✅ The ConfigMap is correct. Nginx is looking for files in /var/www/html.

Step 3: Check the Service 

kubectl get svc
kubectl describe svc nginx-service
Enter fullscreen mode Exit fullscreen mode

Output:

Name:         nginx-service
Type:         NodePort
Port:         8099/TCP
TargetPort:   8099/TCP
NodePort:     30008/TCP
Endpoints:    10.22.0.9:8099
Enter fullscreen mode Exit fullscreen mode

✅ The Service correctly targets port 8099 (matching Nginx's listen port).

Step 4: Check Volume Mounts 

kubectl get pod nginx-phpfpm -o yaml
Enter fullscreen mode Exit fullscreen mode

Key Findings:

containers:
- name: php-fpm-container
  volumeMounts:
  - mountPath: /usr/share/nginx/html   # ❌ WRONG
    name: shared-files

- name: nginx-container
  volumeMounts:
  - mountPath: /var/www/html           # ✅ CORRECT
    name: shared-files
Enter fullscreen mode Exit fullscreen mode

🚨 THE PROBLEM IS FOUND!

Both containers mount the same volume but at different paths:

  • Nginx: mounts at /var/www/html (matches ConfigMap root)
  • PHP-FPM: mounts at /usr/share/nginx/html (DOESN'T match)

Step 5: Check the Logs 

kubectl logs nginx-phpfpm -c nginx-container
Enter fullscreen mode Exit fullscreen mode

Output: Clean Nginx startup, no errors.

kubectl logs nginx-phpfpm -c php-fpm-container
Enter fullscreen mode Exit fullscreen mode

Output:

127.0.0.1 -  24/Aug/2026:10:16:46 +0000 "GET /index.php" 404
Enter fullscreen mode Exit fullscreen mode

🚨 PHP-FPM is returning 404! This confirms the file can't be found.

Identifying the Root Cause

The Flow of a PHP Request

When a request comes in for index.php:

1. Nginx receives request on port 8099
   ↓
2. Nginx looks for file in /var/www/html
   ↓
3. Nginx passes to PHP-FPM:
   SCRIPT_FILENAME = /var/www/html/index.php
   ↓
4. PHP-FPM looks for file at /var/www/html/index.php
   ↓
5. But PHP-FPM has the volume mounted at /usr/share/nginx/html
   ↓
6. PHP-FPM can't find the file
   ↓
7. Returns 404 Not Found ❌
Enter fullscreen mode Exit fullscreen mode

The Volume Mount Mismatch

Container Mount Path ConfigMap Root Matches?
Nginx /var/www/html /var/www/html ✅ Yes
PHP-FPM /usr/share/nginx/html /var/www/html ❌ No

The Problem: PHP-FPM is looking in the wrong place because it has the volume mounted at a different path than where Nginx is telling it to look.

The Solution

Step 1: Fix the PHP-FPM Mount Path

Change the php-fpm-container's volumeMount from /usr/share/nginx/html to /var/www/html.

Before (WRONG):

- name: php-fpm-container
  volumeMounts:
  - mountPath: /usr/share/nginx/html
    name: shared-files
Enter fullscreen mode Exit fullscreen mode

After (CORRECT):

- name: php-fpm-container
  volumeMounts:
  - mountPath: /var/www/html
    name: shared-files
Enter fullscreen mode Exit fullscreen mode

Step 2: Apply the Fix 

# 1. Get the pod YAML
kubectl get pod nginx-phpfpm -o yaml > nginx-phpfpm.yaml

# 2. Edit the file (fix the mountPath)
nano nginx-phpfpm.yaml

# 3. Delete the existing pod
kubectl delete pod nginx-phpfpm

# 4. Create the pod with the fixed YAML
kubectl apply -f nginx-phpfpm.yaml

# 5. Wait for pod to be ready
kubectl wait --for=condition=Ready pod/nginx-phpfpm --timeout=60s
Enter fullscreen mode Exit fullscreen mode

Step 3: Copy the PHP File 

# Copy index.php to the correct location
kubectl cp /home/thor/index.php nginx-phpfpm:/var/www/html/index.php -c nginx-container

# Verify the file exists
kubectl exec nginx-phpfpm -c nginx-container -- ls -la /var/www/html/
Enter fullscreen mode Exit fullscreen mode

Step 4: Verify It Works 

# Test the website
curl http://localhost:30008/
Enter fullscreen mode Exit fullscreen mode

Expected Output: Full phpinfo() HTML! ✅

Verification and Testing

Complete Verification Commands 

echo "=== FINAL VERIFICATION ==="

echo -e "\n1. Pod Status:"
kubectl get pod nginx-phpfpm

echo -e "\n2. Both containers running:"
kubectl get pod nginx-phpfpm -o jsonpath='{.status.containerStatuses[*].ready}'
echo ""

echo -e "\n3. PHP File Location:"
kubectl exec nginx-phpfpm -c nginx-container -- ls -la /var/www/html/

echo -e "\n4. Nginx Configuration:"
kubectl exec nginx-phpfpm -c nginx-container -- grep -E "listen|root" /etc/nginx/nginx.conf

echo -e "\n5. Testing Website:"
curl http://localhost:30008/ | head -20

echo -e "\n6. Nginx Logs:"
kubectl logs nginx-phpfpm -c nginx-container | tail -3

echo -e "\n7. PHP-FPM Logs:"
kubectl logs nginx-phpfpm -c php-fpm-container | tail -3
Enter fullscreen mode Exit fullscreen mode

Expected Output 

$ curl http://localhost:30008/ | head -10
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml"><head>
<style type="text/css">
body {background-color: #fff; color: #222; font-family: sans-serif;}
pre {margin: 0; font-family: monospace;}
a:link {color: #009; text-decoration: none; background-color: #fff;}
a:hover {text-decoration: underline;}
table {border-collapse: collapse; border: 0; width: 934px; box-shadow: 1px 2px 3px #ccc;}
...
Enter fullscreen mode Exit fullscreen mode

Common Mistakes and Lessons Learned

Mistake 1: Inconsistent Volume Mount Paths

What We Did Wrong Why It Failed
PHP-FPM mounted at /usr/share/nginx/html ConfigMap root is /var/www/html
Nginx mounted at /var/www/html PHP-FPM couldn't find the file

Lesson: Both containers must mount shared volumes at the same path, OR the configuration must align.

Mistake 2: Not Checking Logs Early Enough

What We Did: Assumed the file copy worked because the pod was running.

What We Should Have Done: Checked the PHP-FPM logs immediately.

Lesson: Always check ALL container logs, not just the main one.

Mistake 3: Overlooking the FastCGI Flow

What We Missed: How Nginx passes SCRIPT_FILENAME to PHP-FPM.

The Flow:

Nginx: SCRIPT_FILENAME = /var/www/html/index.php
PHP-FPM: Looks for /var/www/html/index.php
But: PHP-FPM volume is at /usr/share/nginx/html
Result: 404 Not Found
Enter fullscreen mode Exit fullscreen mode

Lesson: Understand the data flow between containers.

Best Practices

1. Consistent Volume Mount Paths 

# ✅ GOOD - Both containers use the same path
containers:
- name: container-1
  volumeMounts:
  - mountPath: /app/data
    name: shared-storage
- name: container-2
  volumeMounts:
  - mountPath: /app/data
    name: shared-storage
Enter fullscreen mode Exit fullscreen mode

2. Document Your Volume Mounts 

# ✅ GOOD - Clear comments
volumeMounts:
- mountPath: /var/www/html
  name: shared-files
  # This is the Nginx document root. 
  # PHP-FPM must mount the same volume at the same path.
Enter fullscreen mode Exit fullscreen mode

3. Test After Every Change 

# Always verify after making changes
kubectl get pods
kubectl logs <pod-name> -c <container-name>
curl <endpoint>
Enter fullscreen mode Exit fullscreen mode

4. Use the Same Image with Proper Configuration 

# ✅ GOOD - Consistent with pod definition
apiVersion: v1
kind: ConfigMap
metadata:
  name: nginx-config
data:
  nginx.conf: |
    root /var/www/html;  # Must match volume mount
Enter fullscreen mode Exit fullscreen mode

5. Label Everything Clearly 

# ✅ GOOD - Clear labels
metadata:
  labels:
    app: web-app
    component: nginx-phpfpm
    environment: production
Enter fullscreen mode Exit fullscreen mode

Quick Reference Commands

Diagnostic Commands

Command Purpose
kubectl get pods Check pod status
kubectl describe pod <name> Get pod details
kubectl logs <pod> -c <container> Check container logs
kubectl exec <pod> -c <container> -- <command> Run commands in container
kubectl get pod <pod> -o yaml Get pod YAML

Fix Commands

Command Purpose
kubectl get pod <pod> -o yaml > pod.yaml Export pod YAML
kubectl delete pod <pod> Delete pod
kubectl apply -f pod.yaml Recreate pod
kubectl cp <source> <pod>:<dest> -c <container> Copy files to pod

Conclusion

What We Learned

  1. Shared volumes must be mounted consistently - Both containers must mount the volume at the same path, or the configuration must align.

  2. Logs are your best friend - The PHP-FPM logs revealed the 404 error, pointing us to the problem.

  3. Understand the data flow - Knowing how Nginx passes requests to PHP-FPM helped us identify the path mismatch.

  4. Always test after changes - Verifying with curl after each step would have caught the issue earlier.

The Working Configuration

After the fix:

┌─────────────────────────────────────────────────────────────┐
│                    Working Setup                           │
├─────────────────────────────────────────────────────────────┤
│                                                             │
│  Volume: shared-files (emptyDir)                          │
│                                                             │
│  ┌─────────────────────────────────────────────────────┐   │
│  │  Nginx Container                                   │   │
│  │  ✅ Mount Path: /var/www/html                     │   │
│  │  ✅ root: /var/www/html                          │   │
│  │  ✅ listen: 8099                                 │   │
│  └─────────────────────────────────────────────────────┘   │
│                      │                                     │
│                      │ (Same volume)                      │
│                      ▼                                     │
│  ┌─────────────────────────────────────────────────────┐   │
│  │  PHP-FPM Container                                  │   │
│  │  ✅ Mount Path: /var/www/html                     │   │
│  │  ✅ SCRIPT_FILENAME matches                        │   │
│  └─────────────────────────────────────────────────────┘   │
│                                                             │
│  ✅ Both containers can access the same files!             │
└─────────────────────────────────────────────────────────────┘
Enter fullscreen mode Exit fullscreen mode

Final Result

  • ✅ Pod running (2/2 READY)
  • ✅ Both containers healthy
  • ✅ File accessible in both containers
  • ✅ Website returns phpinfo() correctly
  • ✅ "Website" button works!

Resources

Top comments (0)