DEV Community

suresh devops
suresh devops

Posted on • Edited on

Hidden gems of Kubernetes

#devops #kubernetes #docker #ai

While CRDs and API extensions are well-known, Kubernetes has many powerful but underutilized features. Here are some that even experienced DevOps engineers often overlook:

  1. Advanced Scheduling Features

    • Pod Topology Spread Constraints : Fine-grained control over pod distribution across zones, nodes, etc.
    • Pod Overhead : Account for runtime/daemon overhead when scheduling (critical for Kata Containers, gVisor).
    • Scheduler Profiles & Extenders : Custom scheduler behavior without writing a custom scheduler.
    • NodeResourceFit with Pod Overhead : Actually considers runtime overhead in scheduling decisions.

  2. Networking Deep Cuts

    • Network Policy Port Ranges : (K8s 1.25+) Specify port ranges like 30000-32767 in NetworkPolicies.
    • Service internalTrafficPolicy: Local : Prefer routing traffic to pods on same node for NodePort/LoadBalancer.
    • IPVS Session Affinity Fine-Tuning : Timeout settings, scheduling algorithms beyond round-robin.
    • EndpointSlice : More scalable alternative to Endpoints (automatic since 1.21, but few leverage its full API).

  3. Storage Gems

    • Volume Populators : (Alpha→Beta) Create PVC content from custom sources (like snapshots, http) pre-attachment.
    • Read-Write-Many (RWX) for block storage : Some CSI drivers now support it via filesystem layer magic.
    • Volume Health Monitoring : CSI driver can report volume issues to Kubernetes events.
    • Generic Ephemeral Volumes : Request temporary storage without creating StorageClass/PVC definitions.

  4. Security Obscurities

    • Pod Security Admission (PSA) Exemptions : Namespace-level exemptions for specific service accounts.
    • Seccomp/AppArmor Annotations for Windows : Wait, they exist (some work on Windows Server 2022+).
    • TokenRequest API : Short-lived service account tokens with audience binding.
    • CSIVolumeFSGroupPolicy : Control how CSI drivers handle fsGroup ownership changes.

  5. API Machinery & Admission Magic

    • API Priority and Fairness (APF) : Prevent noisy neighbors in API server with flow control.
    • ValidatingAdmissionPolicy : (K8s 1.26+) CEL-based policies without webhook complexity.
    • Server-Side Apply Field Management : Track which manager owns which field for conflict resolution.
    • API Aggregation Layers : Not just for CRDs - aggregate multiple API servers transparently.

  6. Node & Runtime Features

    • Kubelet Credential Providers : Plugins for dynamic registry credential fetching (ECR, GCR, ACR).
    • RuntimeClass Scheduling : Schedule pods to specific container runtimes (runc, Kata, gVisor).
    • Node System Swap Support : (K8s 1.22+) Yes, swap can now be enabled with performance caveats.
    • Pod Memory QoS : Memory throttling for containers using cgroups v2.

  7. Debugging & Observability

    • Kubelet Tracing : Built-in OpenTelemetry traces for kubelet operations.
    • Dynamic Kubelet Configuration : (Deprecated but interesting) Concept lives on in KEPs.
    • Pod Ready++ : Startup probes are known, but ReadinessGate for custom conditions isn't.
    • Pod Disruption Budget with Unhealthy Pod Exclusion : Auto-exclude unhealthy pods from PDB calculations.

  8. Workload Features

    • Pod Lifecycle Sleep Action : In postStart/preStop hooks - sleep 60 is more common than you think.
    • Pod Deletion Cost : (K8s 1.21+) Annotation to control pod deletion order during downscaling.
    • Pod Topology Spread Constraints by Pod Label : Spread based on pod labels, not just topology.
    • Suspend CronJobs : Temporarily disable without deleting.

  9. CLI & Client Hidden Gems

    • kubectl alpha debug : Node debugging with ephemeral containers (now stable as kubectl debug).
    • kubectl events : The sorted, combined event view everyone should use but doesn't.
    • Client-Side Apply Server-Side Diff : kubectl apply --server-side --dry-run=server.
    • Custom Columns with JSONPath : kubectl get pods -o custom-columns=... with complex expressions.

  10. Ecosystem Integration Points

    • Container Device Interface (CDI) : Standard for exposing hardware (GPUs, FPGAs, etc.) to containers.
    • Service Binding Specification : (K8s 1.21+) Standardized way to bind services to workloads.
    • Cluster Trust Bundles : Distribute CA certificates to workloads trustlessly.

Why These Are Unknown:

  1. Version Skew : Many use older K8s versions
  2. Cloud Provider Abstractions : Managed services hide complexity
  3. Documentation Depth : Features exist but aren't emphasized
  4. Complexity vs. Benefit : Some are too niche for general use
  5. Gradual Rollouts : Features exist for years in alpha/beta before attention

Top comments (0)