DEV Community

Cover image for Architecting Kubernetes Deployments with Python
Joaquin Menchaca
Joaquin Menchaca

Posted on

Architecting Kubernetes Deployments with Python

#devops #python #cloud #kubernetes

Python is an excellent language for automating cloud infrastructure, but the official Kubernetes Python client leaves developers with an important architectural decision:

Where should Kubernetes manifests live?

Should they be constructed directly with Python objects? Embedded as multiline strings? Or stored as external files and rendered at runtime?

Each approach works, but they have very different implications for readability, maintainability, and long-term operational cost.

The key is recognizing that deployment logic and platform configuration evolve on different lifecycles. Your deployment code, the part that authenticates to Kubernetes, renders templates, and applies resources, may remain unchanged for months. Your manifests, however, often change weekly as applications evolve, resource limits are tuned, cloud-provider annotations are added, or networking requirements change.

When those two concerns are tightly coupled, even a configuration, only change forces you to modify, test, and redeploy the delivery or application code itself. Over time, this increases maintenance costs, slows platform changes, and makes configuration drift and production mistakes more likely.

This is a familiar software engineering principle: separate concerns that evolve independently. The same thinking that keeps application configuration separate from executable code also applies to Kubernetes manifests. Treating manifests as first-class configuration artifacts allows them to evolve independently from the Python code that delivers them.

In this article we'll compare three ways of deploying Kubernetes resources with the official kubernetes-python-client, ranging from tightly coupled implementations to a design that cleanly separates deployment logic from platform configuration.

The Landscape at a Glance

The comparison below assumes a common application deployment scenario, where the desired state is largely known ahead of time. Controllers and Operators have fundamentally different requirements where the desiredstate is determined dynamically at runtime.

Method¹ Maintainability² Readability³ Flexibility⁴ Best Used For
Direct API
Object Calls		 ⭐⭐⭐ Medium ⭐ Low ⭐⭐⭐⭐⭐ High Dynamic,
complex operators,
custom controllers
Embedded Strings ⭐⭐ Low ⭐⭐⭐ Medium ⭐⭐ Low Small utilities, remediation scripts, one-off automation
External YAML
Templates (Jinja2)		 ⭐⭐⭐⭐⭐ High ⭐⭐⭐⭐⭐ High ⭐⭐⭐ Medium Declarative,
Standardized application deployments,
CI/CD pipelines
  1. Method refers to the overall approach used to define and deploy Kubernetes resources, not just features of the official kubernetes-python-client.
  2. Maintainability measures the cost to safely modify the solution over time.
  3. Readability measures ability of the engineer quickly determine the desired state and confidently modify it?
  4. Flexibility measures the ability to determine or modify the desired state dynamically at runtime.

1. The Most Verbose: Direct API Object Calls

The official client libary (kubernetes-python-client) exposes the Kubernetes API as native Python objects. This provides complete programmatic control over every resource, but that flexibility comes at the cost of verbose, deeply nested object construction.

Instead of expressing the desired state directly as a Kubernetes manifest, it is represented through a hierarchy of Python objects that are ultimately translated back into the Kubernetes API.

This additional abstraction makes the desired state less obvious during code reviews and collaboration, since engineers must mentally reconstruct the resulting Kubernetes resource instead of reading it directly as a manifest.

The example below shows simple web application deployment and a service (type: LoadBalancer) with some annotations for AWS NLB.

from kubernetes import client, config

config.load_kube_config()
apps_v1 = client.AppsV1Api()
core_v1 = client.CoreV1Api()

# Define the Deployment Object
deployment = client.V1Deployment(
    metadata=client.V1ObjectMeta(name="demo-nlb-app-deployment"),
    spec=client.V1DeploymentSpec(
        replicas=3,
        selector=client.V1LabelSelector(match_labels={"app": "demo-nlb-app"}),
        template=client.V1PodTemplateSpec(
            metadata=client.V1ObjectMeta(labels={"app": "demo-nlb-app"}),
            spec=client.V1PodSpec(
                containers=[client.V1Container(name="web", image="nginx:alpine")]
            )
        )
    )
)

# Define the Service Object with AWS NLB Annotations
service = client.V1Service(
    metadata=client.V1ObjectMeta(
        name="demo-nlb-app-service",
        annotations={
            "service.beta.kubernetes.io/aws-load-balancer-type": "external",
            "service.beta.kubernetes.io/aws-load-balancer-scheme": "internet-facing",
            "service.beta.kubernetes.io/aws-load-balancer-nlb-target-type": "instance"
        }
    ),
    spec=client.V1ServiceSpec(
        type="LoadBalancer",
        selector={"app": "demo-nlb-app"},
        ports=[client.V1ServicePort(port=80, target_port=80)]
    )
)

# Apply to Cluster
apps_v1.create_namespaced_deployment(namespace="default", body=deployment)
core_v1.create_namespaced_service(namespace="default", body=service)
Enter fullscreen mode Exit fullscreen mode

Notice that the desired state is no longer represented as a Kubernetes manifest. Instead, it is distributed across a hierarchy of Python object constructors. Understanding the resulting Deployment requires understanding both the Kubernetes resource model and the Python API simultaneously. This makes the deployed infrastructure less transparent during reviews, since the desired state must first be reconstructed from the implementation.

The Verdict: This approach excels when the desired state cannot be known until runtime, such as Kubernetes Controllers, Operators, or other applications that continuously reconcile resources based on external events. For conventional application deployments, however, the additional flexibility rarely outweighs the increased verbosity and maintenance cost compared to declarative manifests.

Segue: Community Client Libraries

As the official client is generated directly from the Kubernetes OpenAPI specification, its object model closely mirrors the Kubernetes API rather than idiomatic Python. This has led the community to develop alternative client libraries that provide a more Pythonic developer experience while still interacting with the same Kubernetes APIs.

  • pykube-ng - A lightweight client that models Kubernetes resources much like a Python ORM such as SQLAlchemy or Django's ORM. Resources are manipulated through familiar CRUD-style operations rather than constructing deeply nested API objects.

  • kr8s - Lightweight client whose API closely mirrors the kubectl experience while abstracting away much of the connection setup and boilerplate required by the official client. It also includes native asyncio support for asynchronous applications.

2. The Hardcoded Copy-Paste: Embedded Strings

The easiest shortcut is copying raw YAML directly into a Python file using triple-quoted """

As the manifest remains recognizable YAML, this approach initially feels like the best of both worlds: the readability of Kubernetes manifests combined with the convenience of keeping everything in one Python file. However, that convenience comes from embedding platform configuration directly inside the application or delivery logic, creating a coupling that becomes increasingly expensive as the application evolves.

import yaml
from kubernetes import client, config

config.load_kube_config()

# Hardcoded variables injected via Python f-strings
app_name = "demo-nlb-app"
image_name = "nginx:alpine"
replicas = 3

manifest_string = f"""
apiVersion: apps/v1
kind: Deployment
metadata:
  name: {app_name}-deployment
spec:
  replicas: {replicas}
  selector:
    matchLabels:
      app: {app_name}
  template:
    metadata:
      labels:
        app: {app_name}
    spec:
      containers:
      - name: web
        image: {image_name}
---
apiVersion: v1
kind: Service
metadata:
  name: {app_name}-service
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-type: "external"
    service.beta.kubernetes.io/aws-load-balancer-scheme: "internet-facing"
    service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: "instance"
spec:
  type: LoadBalancer
  selector:
    app: {app_name}
  ports:
  - port: 80
    targetPort: 80
"""

# Split the string and apply each document sequentially
apps_v1 = client.AppsV1Api()
core_v1 = client.CoreV1Api()

for doc in yaml.safe_load_all(manifest_string):
    if doc['kind'] == 'Deployment':
        apps_v1.create_namespaced_deployment(namespace="default", body=doc)
    elif doc['kind'] == 'Service':
        core_v1.create_namespaced_service(namespace="default", body=doc)
Enter fullscreen mode Exit fullscreen mode

The Verdict: Embedded manifests are a pragmatic choice for small utilities, remediation scripts, or one-off automation where the manifest is simply an implementation detail. For long-lived deployment tooling, however, embedding configuration inside the application or delivery logic tightly couples two concerns that often evolve independently, increasing maintenance costs over time.

Segue: The Cost of Tethering Configuration to Delivery Logic

Embedding manifests directly into Python couples two concerns that frequently evolve at different rates: the deployment logic and the platform configuration. The deployment engine may remain unchanged for months, while manifests evolve regularly as applications grow, cloud-provider annotations change, or operational requirements shift.

  • Asymmetric Velocity: Deployment logic and platform configuration rarely evolve together. A deployment script may remain unchanged for months while Kubernetes manifests are updated frequently to adjust resource limits, networking policies, or cloud-provider annotations. Coupling them forces both to follow the same release lifecycle.

  • The Testing Tax: If your configuration is baked inside the application code, modifying a simple JSON block or YAML string often requires rebuilding, retesting, and redeploying the full application, which can slow your release down and impact time-to-market and revenue. If testing is bypass to run a fix out, your blast radius explodes.

  • The Silent Outage: Finding configuration buried inside hundreds of lines of code logic increases cognitive load. In practice, this exact pattern frequently leads to real-world outages because a developer updates an application specification but forgets to dig through the delivery codebase to update the embedded string matching it.

For a simple atomic web application deployment, the desired state should be immediately visible during a code review. Reviewers should not have to mentally search through delivery logic to understand what Kubernetes will ultimately receive. Keeping manifests as first-class artifacts preserves that transparency.

3. The Gold Standard: External YAML Templates (Jinja2)

When the desired state is largely known ahead of time, the cleanest approach is to keep Kubernetes manifests as first-class artifacts and render only the values that must be determined dynamically. Standard .yaml files remain in your repository—preserving IDE support, schema validation, and the familiar Kubernetes representation, while Jinja2 provides lightweight parameter substitution at runtime.

This allows you to link your service and deployment fluidly using shared variables like {{ app_name }}.

manifest.yaml.j2 (The Clean Template): 

apiVersion: apps/v1
kind: Deployment
metadata:
  name: {{ app_name }}-deployment
spec:
  replicas: {{ replicas }}
  selector:
    matchLabels:
      app: {{ app_name }}
  template:
    metadata:
      labels:
        app: {{ app_name }}
    spec:
      containers:
      - name: web
        image: {{ image_name }}
---
apiVersion: v1
kind: Service
metadata:
  name: {{ app_name }}-service
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-type: "external"
    service.beta.kubernetes.io/aws-load-balancer-scheme: "internet-facing"
    service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: "instance"
spec:
  type: LoadBalancer
  selector:
    app: {{ app_name }}
  ports:
  - port: 80
    targetPort: 80
Enter fullscreen mode Exit fullscreen mode

deploy.py (The Clean Script) 

import yaml
from jinja2 import Template
from kubernetes import client, config

config.load_kube_config()

# 1. Read the external Jinja2 template file
with open("manifest.yaml.j2", "r") as f:
    raw_template = f.read()

# 2. Render variables into the template seamlessly
rendered_yaml = Template(raw_template).render(
    app_name="demo-nlb-app",
    replicas=3,
    image_name="nginx:alpine"
)

# 3. Parse and safely push to Kubernetes
apps_v1 = client.AppsV1Api()
core_v1 = client.CoreV1Api()

for doc in yaml.safe_load_all(rendered_yaml):
    if doc['kind'] == 'Deployment':
        apps_v1.create_namespaced_deployment(namespace="default", body=doc)
    elif doc['kind'] == 'Service':
        core_v1.create_namespaced_service(namespace="default", body=doc)
Enter fullscreen mode Exit fullscreen mode

Notice that the deployment logic and the desired state are now represented independently. The Python code is responsible only for supplying runtime values and applying resources, while the manifest remains a transparent description of what Kubernetes will receive. Each artifact can evolve independently without forcing changes to the other.

The Verdict: For conventional application deployments, this approach provides the best balance of maintainability, readability, and flexibility. The deployment implementation remains focused on supplying runtime values, while the manifests continue to express the desired state as first-class configuration artifacts. This separation reduces the cost of change, improves collaboration across engineering teams, and leaves the door open to adopting tools such as Helm or GitOps workflows in the future.

Beyond the Official Client

The approaches discussed in this article intentionally stay close to the official Kubernetes Python client (kubernetes-python-client). As your applications grow, higher-level frameworks such as Kopf, cdk8s, and Pulumi provide different programming models that may better fit specific classes of problems.

  • Kopf — A Python framework for building Kubernetes Operators and Controllers. Rather than manually watching resources and reconciling state, Kopf provides an event-driven framework designed specifically for applications whose desired state is determined at runtime.

  • cdk8s — Generates Kubernetes manifests from object-oriented Python code. This allows infrastructure to be modeled using software abstractions while still producing standard Kubernetes YAML as the final artifact.

  • Pulumi — Treats infrastructure as software with state management, dependency tracking, and cloud resource provisioning. Unlike cdk8s, Pulumi manages infrastructure lifecycle rather than simply generating Kubernetes manifests.

Conclusion

When automating Kubernetes deployments with Python, the goal isn't to eliminate YAML or force every deployment through an object-oriented API. The goal is to choose a representation that matches the nature of the problem.

If the desired state is largely known ahead of time, keep it declarative. External manifests preserve Kubernetes as the source of truth while allowing the Python implementation to focus on supplying runtime values and interacting with the Kubernetes API. Because the deployment logic and platform configuration evolve independently, treating manifests as first-class artifacts reduces the cost of change, improves transparency during code reviews, and encourages collaboration across engineering disciplines.

When the desired state cannot be known until runtime, however, the trade-offs change. Controllers, Operators, and other event-driven applications continuously reconcile resources based on external events. In these cases, the flexibility of the Kubernetes Python API justifies the additional complexity because the desired state must be synthesized rather than declared.

Ultimately, there isn't a single "best" approach. Each representation optimizes for a different class of problems. Understanding those trade-offs, and choosing the abstraction that matches the problem—is far more important than the particular library or templating engine you use.

Practical Guidance

  • Use external manifest templates when the desired state is largely known ahead of time and only a subset of values must be determined dynamically.
  • Use embedded manifests for small utilities, remediation scripts, or one-off operational automation where the manifest is simply an implementation detail.
  • Use the Kubernetes Python API directly when resources must be created or modified dynamically in response to runtime events, such as Controllers or Operators.

Further Reading

Here are some articles and resources I came across in research for this article.

Client Libraries

  • pykube-ng - A highly lightweight client tailored for simple scripts and basic cluster interactions.
  • kr8s - A simple, extensible Python client library for Kubernetes that feels familiar for folks who already know how to use kubectl
  • kubernetes-python-client - Official, comprehensive library that provides full access to all Kubernetes API endpoints, authentication mechanisms, and resource types.

Higher Level Frameworks

  • kopf - A Python framework to write Kubernetes operators in just a few lines of code
  • cdk8s - Define Kubernetes native apps and abstractions using object-oriented programming
  • Pulumi - Infrastructure as Code in any programming language
    • pulumi-kubernetes - A Pulumi resource provider for Kubernetes to manage API resources and workloads in running clusters

Articles

Top comments (0)