DEV Community

Ahmed Rakan
Ahmed Rakan

Posted on

Setting Up a Scalable MongoDB on Kubernetes: A Technical Guide

In this post, we’ll walk through deploying a highly available MongoDB replica set on Kubernetes using StatefulSet, headless service, and RBAC. We'll also cover how to initialize the replica set, troubleshoot DNS issues, and configure user authentication using Kubernetes secrets.

🔐 Step 1: Creating a Kubernetes Secret for MongoDB Credentials

Before deploying MongoDB, you should securely store the root credentials using a Kubernetes secret:

kubectl -n databases create secret generic mongo-secret \
  --from-literal=mongo-user=123\
  --from-literal=mongo-password=321
Enter fullscreen mode Exit fullscreen mode

👤 Step 2: Define a ServiceAccount

MongoDB pods need access to service discovery. We'll create a service account in the same namespace:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: mongo
  namespace: databases
Enter fullscreen mode Exit fullscreen mode

⚙️ Step 3: Role Binding for DNS Resolution

Allow the service account to resolve pod and service endpoints via a ClusterRoleBinding:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: system:serviceaccount:databases:mongo
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: read-pod-service-endpoint
subjects:
- kind: ServiceAccount
  name: mongo
  namespace: databases
Enter fullscreen mode Exit fullscreen mode

🌐 Step 4: Define a Headless Service

Headless services enable direct pod-to-pod communication, which MongoDB requires for replication:

apiVersion: v1
kind: Service
metadata:
  name: mongo
  labels:
    name: mongo
spec:
  selector:
    app: mongo
  ports:
    - protocol: TCP
      port: 27017
      targetPort: 27017
  clusterIP: None
Enter fullscreen mode Exit fullscreen mode

📦 Step 5: Deploy MongoDB with StatefulSet

Here's a complete StatefulSet manifest for deploying a 3-node MongoDB replica set:

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: mongo
  labels:
    app: mongo
spec:
  serviceName: "mongo"
  replicas: 3
  selector:
    matchLabels:
      app: mongo
  template:
    metadata:
      labels:
        app: mongo
    spec:
      nodeSelector:
        node.longhorn.io/create-default-disk: "true"
      serviceAccountName: mongo
      automountServiceAccountToken: true
      containers:
      - name: mongodb
        image: mongo:5.0
        command: ["mongod", "--replSet=rs0", "--bind_ip=0.0.0.0"]
        ports:
          - containerPort: 27017
        env:
          - name: MONGO_INITDB_ROOT_USERNAME
            valueFrom:
              secretKeyRef:
                name: mongo-secret
                key: mongo-user
          - name: MONGO_INITDB_ROOT_PASSWORD
            valueFrom:
              secretKeyRef:
                name: mongo-secret
                key: mongo-password
        volumeMounts:
          - name: mongo-persistent-storage
            mountPath: /data/db
      - name: mongo-sidecar
        image: morphy/k8s-mongo-sidecar
        env:
          - name: KUBERNETES_POD_LABELS
            value: "role=mongo,environment=test"
          - name: KUBERNETES_SERVICE_NAME
            value: "mongo"
  volumeClaimTemplates:
    - metadata:
        name: mongo-persistent-storage
      spec:
        accessModes: ["ReadWriteOnce"]
        resources:
          requests:
            storage: 50Gi
Enter fullscreen mode Exit fullscreen mode

🧠 Step 6: Initialize the Replica Set

Once the pods are up and running, initialize the replica set:

kubectl exec -it mongo-0 -n databases -- mongo
Enter fullscreen mode Exit fullscreen mode

Then run:

rs.initiate({
  _id: "rs0",
  version: 1,
  members: [
    { _id: 0, host: "mongo-0.mongo.databases.svc.cluster.local:27017" },
    { _id: 1, host: "mongo-1.mongo.databases.svc.cluster.local:27017" },
    { _id: 2, host: "mongo-2.mongo.databases.svc.cluster.local:27017" }
  ]
})
Enter fullscreen mode Exit fullscreen mode

Verify with:

rs.status()
Enter fullscreen mode Exit fullscreen mode

🧪 Step 7: Verify Replication 

use testDB
db.testCollection.insert({ name: "Replication Test" })
db.testCollection.find({})
Enter fullscreen mode Exit fullscreen mode

👮 Step 8: Create Admin User (Workaround)

Until you integrate authentication with secrets fully, manually create a user:

use admin
db.createUser({
  user: "123",
  pwd: "321",
  roles: [
    { role: "readWrite", db: "Alhumdillah" },
    { role: "dbAdmin", db: "Alhumdillah" }
  ]
})
Enter fullscreen mode Exit fullscreen mode

🧭 Step 9: Troubleshooting DNS

To resolve headless service DNS issues, deploy a debugging pod:

apiVersion: v1
kind: Pod
metadata:
  name: dnsutils
  namespace: default
spec:
  containers:
    - name: dnsutils
      image: registry.k8s.io/e2e-test-images/agnhost:2.39
      imagePullPolicy: IfNotPresent
  restartPolicy: Always
Enter fullscreen mode Exit fullscreen mode

Then run:

kubectl exec -i -t dnsutils -- nslookup mongo-2.mongo.databases.svc.cluster.local
Enter fullscreen mode Exit fullscreen mode

You should get multiple IPs for each pod.

🧪 Step 10: Test External Connection 

mongo "mongodb://<user>:<password>@mongo-0,mongo-1,mongo-2.mongo.databases.svc.cluster.local/Alhumdillah?replicaSet=rs0&authSource=admin"
Enter fullscreen mode Exit fullscreen mode

Example:

mongo "mongodb://123:321@mongo-0.mongo.databases.svc.cluster.local:27017,mongo-1.mongo.databases.svc.cluster.local:27017,mongo-2.mongo.databases.svc.cluster.local:27017/Alhumdillah?replicaSet=rs0&authSource=admin"
Enter fullscreen mode Exit fullscreen mode

🔁 Step 11: Reconfigure Replica Set (If Needed)

If your replica set was initialized with incorrect hosts (e.g., different namespaces), you must reconfigure it:

  1. Fetch current config: 
cfg = rs.conf();
Enter fullscreen mode Exit fullscreen mode
  1. Modify config: 
cfg.version = 2;
cfg.members = [
  { _id: 0, host: "mongo-0.mongo.databases.svc.cluster.local:27017" },
  { _id: 1, host: "mongo-1.mongo.databases.svc.cluster.local:27017" },
  { _id: 2, host: "mongo-2.mongo.databases.svc.cluster.local:27017" }
];
Enter fullscreen mode Exit fullscreen mode
  1. Apply: 
rs.reconfig(cfg);
Enter fullscreen mode Exit fullscreen mode

⚠️ Important: Never re-initiate an already initialized replica set. Use rs.reconfig() instead.

Next Steps :

1. Using Longhorn for Persistent Storage

Longhorn is a lightweight, distributed block storage system designed for Kubernetes. It provides resilient, high-availability volumes — critical for databases like MongoDB.

In our StatefulSet, we’re assuming Longhorn is already installed in your cluster. To ensure MongoDB pods use Longhorn volumes:

Enable a default disk on the node (if not already created):

kubectl label node <your-node-name> node.longhorn.io/create-default-disk=true
Enter fullscreen mode Exit fullscreen mode

Verify Longhorn volumes:
After deploying MongoDB, check the Longhorn UI (http://:) or run:

kubectl get pv
Enter fullscreen mode Exit fullscreen mode

Confirm replication settings:
In Longhorn settings, ensure each volume has at least 2 replicas for HA.

✅ Longhorn ensures that if a node or disk fails, your MongoDB data remains intact across replicas.

2. Backup and disaster recovery

The last thing you want to encounter when running highly scalable, highly available databases is data loss. You might want to consider kasten10

helm install k10 kasten/k10 --namespace kasten-io --create-namespace
Enter fullscreen mode Exit fullscreen mode

✅ Conclusion

You’ve now deployed a robust, secure, and scalable MongoDB replica set on Kubernetes using best practices including:

  • Kubernetes Secrets
  • StatefulSets with persistent storage
  • RBAC and DNS debugging
  • Authentication and replica set configuration
  • High Availability

This setup is production-grade and extensible—ideal for running databases like MongoDB in a cloud-native stack.

Top comments (0)