DEV Community: Marco

Daily Dose of DevOps — Kubernetes scheduling: with multi-tenant isolation

Marco — Fri, 25 Sep 2026 12:29:54 +0000

Kubernetes scheduling: with multi-tenant isolation

A platform capability becomes dangerous when ownership is implicit but authority is broad. For kubernetes scheduling with multi-tenant isolation, the decisive question is not whether a team can demonstrate the technology once. It is whether the organisation can operate it repeatedly, audit its decisions, and recover when assumptions fail.

The operating problem

Treat kubernetes scheduling as part of a multi-tenant Kubernetes platform with declarative reconciliation and bounded workload privileges. Define the consumer, owner, support boundary, change policy, and recovery objective before selecting implementation details. When operating kubernetes scheduling with multi-tenant isolation, undocumented authority and ambiguous ownership create more risk than a missing feature.

A credible design measures saturation, scheduling latency, error-budget burn, reconciliation failure, and recovery time. Those measures should be visible to both the platform owner and consuming teams. If the measurements cannot distinguish adoption from coercion, or reliability from mere activity, the operating model is not yet falsifiable.

A practical control

Start with a narrow contract that can be tested automatically. The implementation should encode tested admission policy, workload identity, resource envelopes, progressive delivery, and explicit tenancy boundaries. The following fragment is illustrative; production values must be derived from workload evidence and organisational policy.

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: critical-api
spec:
  minAvailable: 80%
  selector:
    matchLabels:
      app: critical-api

Roll the control out to one representative service, observe failure behaviour, and exercise rollback before broad adoption. Record exceptions as expiring decisions with an accountable owner, not permanent bypasses.

Trade-offs

Standardisation reduces cognitive load and makes controls observable, but an excessively rigid path displaces complexity into workarounds. Flexibility improves local fit, but every variant expands the support surface and weakens fleet-wide guarantees. For kubernetes scheduling with multi-tenant isolation, prefer a small mandatory safety kernel surrounded by replaceable implementation choices.

The organisation must also pay for operability. More validation increases feedback time; more telemetry increases cost and cardinality risk; stronger isolation can reduce utilisation. Make those costs explicit and compare them with the blast radius and recovery cost they buy down.

What teams get wrong

The recurrent anti-pattern is treating Kubernetes defaults as an operating model and allowing every team to invent cluster policy. Teams then measure task completion instead of production outcomes, accumulate exceptions without expiry, and discover during an incident that the nominal control has no tested recovery path.

Another error is adopting a reference architecture without its assumptions. Validate identity boundaries, dependency failure, capacity pressure, partial rollout, rollback, and audit reconstruction in the environment that will actually carry production traffic.

Key Takeaways

Treat kubernetes scheduling as an owned product and control system, not a tool installation.
Design for the constraints captured by “with multi-tenant isolation”; document the authority, failure domain, and recovery objective.
Measure saturation, scheduling latency, error-budget burn, reconciliation failure, and recovery time.
Automate tested admission policy, workload identity, resource envelopes, progressive delivery, and explicit tenancy boundaries.
Test degraded operation and rollback before scaling adoption.

Daily Dose of DevOps — Golden path: for service ownership models

Marco — Thu, 24 Sep 2026 12:28:28 +0000

Golden path: for service ownership models

Enterprise reliability deteriorates when automation accelerates change without strengthening evidence. For golden path for service ownership models, the decisive question is not whether a team can demonstrate the technology once. It is whether the organisation can operate it repeatedly, audit its decisions, and recover when assumptions fail.

The operating problem

Treat golden path as part of a product-oriented internal platform with explicit service ownership and paved-road contracts. Define the consumer, owner, support boundary, change policy, and recovery objective before selecting implementation details. When operating golden path for service ownership models, undocumented authority and ambiguous ownership create more risk than a missing feature.

A credible design measures adoption, successful self-service completion, lead time, and platform-induced toil. Those measures should be visible to both the platform owner and consuming teams. If the measurements cannot distinguish adoption from coercion, or reliability from mere activity, the operating model is not yet falsifiable.

A practical control

Start with a narrow contract that can be tested automatically. The implementation should encode versioned golden paths, discoverable ownership, policy guardrails, and measured developer outcomes. The following fragment is illustrative; production values must be derived from workload evidence and organisational policy.

apiVersion: backstage.io/v1alpha1
kind: Component
metadata:
  name: payments-api
  annotations:
    platform.example/owner: payments
spec:
  type: service
  lifecycle: production

Trade-offs

Standardisation reduces cognitive load and makes controls observable, but an excessively rigid path displaces complexity into workarounds. Flexibility improves local fit, but every variant expands the support surface and weakens fleet-wide guarantees. For golden path for service ownership models, prefer a small mandatory safety kernel surrounded by replaceable implementation choices.

What teams get wrong

The recurrent anti-pattern is centralising delivery behind a ticket queue while calling the result self-service. Teams then measure task completion instead of production outcomes, accumulate exceptions without expiry, and discover during an incident that the nominal control has no tested recovery path.

Key Takeaways

Treat golden path as an owned product and control system, not a tool installation.
Design for the constraints captured by “for service ownership models”; document the authority, failure domain, and recovery objective.
Measure adoption, successful self-service completion, lead time, and platform-induced toil.
Automate versioned golden paths, discoverable ownership, policy guardrails, and measured developer outcomes.
Test degraded operation and rollback before scaling adoption.

Daily Dose of DevOps — Telemetry design: during migration programs

Marco — Wed, 23 Sep 2026 12:29:18 +0000

Telemetry design: during migration programs

Scale converts an informal convention into an outage mechanism. For telemetry design during migration programs, the decisive question is not whether a team can demonstrate the technology once. It is whether the organisation can operate it repeatedly, audit its decisions, and recover when assumptions fail.

The operating problem

Treat telemetry design as part of an SLO-led telemetry platform that connects service behaviour to customer-visible outcomes. Define the consumer, owner, support boundary, change policy, and recovery objective before selecting implementation details. When operating telemetry design during migration programs, undocumented authority and ambiguous ownership create more risk than a missing feature.

A credible design measures error-budget burn, tail latency, cardinality growth, telemetry loss, and diagnostic time. Those measures should be visible to both the platform owner and consuming teams. If the measurements cannot distinguish adoption from coercion, or reliability from mere activity, the operating model is not yet falsifiable.

A practical control

Start with a narrow contract that can be tested automatically. The implementation should encode stable semantic conventions, tiered retention, cardinality budgets, sampling policy, and telemetry pipeline SLOs. The following fragment is illustrative; production values must be derived from workload evidence and organisational policy.

processors:
  memory_limiter:
    limit_mib: 512
  batch:
    send_batch_size: 1024
exporters:
  otlp:
    endpoint: telemetry-gateway:4317

Trade-offs

Standardisation reduces cognitive load and makes controls observable, but an excessively rigid path displaces complexity into workarounds. Flexibility improves local fit, but every variant expands the support surface and weakens fleet-wide guarantees. For telemetry design during migration programs, prefer a small mandatory safety kernel surrounded by replaceable implementation choices.

What teams get wrong

The recurrent anti-pattern is collecting every available signal without ownership, retention economics, or an incident hypothesis. Teams then measure task completion instead of production outcomes, accumulate exceptions without expiry, and discover during an incident that the nominal control has no tested recovery path.

Key Takeaways

Treat telemetry design as an owned product and control system, not a tool installation.
Design for the constraints captured by “during migration programs”; document the authority, failure domain, and recovery objective.
Measure error-budget burn, tail latency, cardinality growth, telemetry loss, and diagnostic time.
Automate stable semantic conventions, tiered retention, cardinality budgets, sampling policy, and telemetry pipeline SLOs.
Test degraded operation and rollback before scaling adoption.

Daily Dose of DevOps — Workload identity: for identity-driven access

Marco — Tue, 22 Sep 2026 17:09:00 +0000

Workload identity: for identity-driven access

Scale converts an informal convention into an outage mechanism. For workload identity for identity-driven access, the decisive question is not whether a team can demonstrate the technology once. It is whether the organisation can operate it repeatedly, audit its decisions, and recover when assumptions fail.

The operating problem

Treat workload identity as part of a zero-trust delivery system in which identity, provenance, and policy are evaluated continuously. Define the consumer, owner, support boundary, change policy, and recovery objective before selecting implementation details. When operating workload identity for identity-driven access, undocumented authority and ambiguous ownership create more risk than a missing feature.

A credible design measures control coverage, policy exceptions, credential lifetime, remediation latency, and provenance verification. Those measures should be visible to both the platform owner and consuming teams. If the measurements cannot distinguish adoption from coercion, or reliability from mere activity, the operating model is not yet falsifiable.

A practical control

Start with a narrow contract that can be tested automatically. The implementation should encode short-lived identity, least privilege, immutable dependencies, signed provenance, and policy-as-code enforcement. The following fragment is illustrative; production values must be derived from workload evidence and organisational policy.

permissions:
  contents: read
  id-token: write
steps:
  - uses: actions/checkout@<immutable-sha>
  - run: cosign verify --certificate-identity-regexp='^https://github.com/example/' artifact

Trade-offs

Standardisation reduces cognitive load and makes controls observable, but an excessively rigid path displaces complexity into workarounds. Flexibility improves local fit, but every variant expands the support surface and weakens fleet-wide guarantees. For workload identity for identity-driven access, prefer a small mandatory safety kernel surrounded by replaceable implementation choices.

What teams get wrong

The recurrent anti-pattern is adding ceremonial approval gates without constraining capabilities or verifying the produced artifact. Teams then measure task completion instead of production outcomes, accumulate exceptions without expiry, and discover during an incident that the nominal control has no tested recovery path.

Key Takeaways

Treat workload identity as an owned product and control system, not a tool installation.
Design for the constraints captured by “for identity-driven access”; document the authority, failure domain, and recovery objective.
Measure control coverage, policy exceptions, credential lifetime, remediation latency, and provenance verification.
Automate short-lived identity, least privilege, immutable dependencies, signed provenance, and policy-as-code enforcement.
Test degraded operation and rollback before scaling adoption.

Daily Dose of DevOps — Observability strategy: for platform reliability reviews

Marco — Sat, 19 Sep 2026 11:44:25 +0000

Observability strategy: for platform reliability reviews

Enterprise reliability deteriorates when automation accelerates change without strengthening evidence. For observability strategy for platform reliability reviews, the decisive question is not whether a team can demonstrate the technology once. It is whether the organisation can operate it repeatedly, audit its decisions, and recover when assumptions fail.

The operating problem

Treat observability strategy as part of an SLO-led telemetry platform that connects service behaviour to customer-visible outcomes. Define the consumer, owner, support boundary, change policy, and recovery objective before selecting implementation details. When operating observability strategy for platform reliability reviews, undocumented authority and ambiguous ownership create more risk than a missing feature.

A practical control

processors:
  memory_limiter:
    limit_mib: 512
  batch:
    send_batch_size: 1024
exporters:
  otlp:
    endpoint: telemetry-gateway:4317

Trade-offs

Standardisation reduces cognitive load and makes controls observable, but an excessively rigid path displaces complexity into workarounds. Flexibility improves local fit, but every variant expands the support surface and weakens fleet-wide guarantees. For observability strategy for platform reliability reviews, prefer a small mandatory safety kernel surrounded by replaceable implementation choices.

What teams get wrong

Key Takeaways

Treat observability strategy as an owned product and control system, not a tool installation.
Design for the constraints captured by “for platform reliability reviews”; document the authority, failure domain, and recovery objective.
Measure error-budget burn, tail latency, cardinality growth, telemetry loss, and diagnostic time.
Automate stable semantic conventions, tiered retention, cardinality budgets, sampling policy, and telemetry pipeline SLOs.
Test degraded operation and rollback before scaling adoption.

Daily Dose of DevOps — What is CI/CD and why it matters

Marco — Fri, 18 Sep 2026 12:01:00 +0000

CI/CD as a Control System: From Commit Entropy to Production Evidence

CI/CD is often reduced to automation, but its deeper purpose is epistemic: it converts uncertain changes into evidence about whether a system remains safe to operate. A pipeline is a feedback controller. Source changes are disturbances; tests, policy checks, and telemetry are sensors; deployment strategies are actuators; service-level objectives define acceptable operating bounds.

The control loop

Continuous integration reduces integration entropy by keeping change sets small and repeatedly testing their composition. Continuous delivery preserves a deployable state; continuous deployment automatically promotes changes after policy gates succeed. These are distinct maturity levels, and conflating them creates unsafe expectations.

A defensible pipeline evaluates more than functional correctness:

Provenance: Can every artifact be traced to reviewed source and a reproducible build?
Security: Were dependencies, secrets, permissions, and artifact signatures evaluated?
Operability: Do latency, saturation, error-rate, and rollback signals exist before promotion?
Change risk: Is rollout scope proportional to uncertainty?

promote:
  needs: [unit, integration, policy, sbom]
  environment: production
  steps:
    - run: cosign verify --key cosign.pub artifact.example/app:$GIT_SHA
    - run: deploy --strategy=canary --initial-traffic=1%
    - run: verify-slo --window=15m --max-error-budget-burn=2

Why small batches dominate

If each changed component has some independent probability of introducing a defect, larger batches increase both the probability of failure and the diagnostic search space. Independence is an imperfect assumption—software dependencies are correlated—but the conclusion survives: small batches shorten feedback latency and improve causal attribution. Deployment frequency is therefore valuable only when paired with fast recovery and trustworthy verification.

Failure modes

Pipeline success is not proof of production safety. Tests may encode incomplete specifications; staging traffic rarely matches production; mutable tags can sever provenance; and approval gates can become ceremonial. A mature design treats every gate as a falsifiable claim and continuously measures its predictive power. Flaky tests, for example, are not harmless noise: they weaken the statistical meaning of a green build and train operators to ignore alarms.

Key Takeaways

CI/CD is a socio-technical feedback system, not merely a collection of scripts.
Optimize for evidence quality, small batch size, bounded blast radius, and reversible change.
A green pipeline is a risk estimate; production telemetry must close the control loop.
Measure lead time and deployment frequency alongside change-failure rate and recovery time.

Daily Dose of DevOps — GitHub Actions basics for DevOps

Marco — Thu, 17 Sep 2026 12:21:20 +0000

GitHub Actions as a Capability System: Secure Automation by Construction

GitHub Actions combines an event system, workflow scheduler, ephemeral compute, and credential broker. Its primary security question is not “Does the YAML run?” but “What authority can untrusted input exercise?” A workflow triggered by pull-request content sits on a trust boundary: branch names, commit contents, issue text, and third-party action outputs may all be attacker-controlled.

Minimize authority

Permissions should be explicit and job-scoped. Build jobs usually need read-only repository access; deployment jobs can receive stronger rights only after protected-environment controls succeed. OpenID Connect is preferable to long-lived cloud credentials because it exchanges a short-lived, claim-bound identity at runtime.

permissions:
  contents: read

jobs:
  deploy:
    environment: production
    permissions:
      contents: read
      id-token: write
    steps:
      - uses: actions/checkout@<immutable-commit-sha>
      - run: npm ci --ignore-scripts
      - run: npm test
      - run: ./scripts/deploy.sh

Pin third-party actions to immutable commit SHAs, review their provenance, and use dependency automation to propose controlled updates. Tags are readable but mutable. Treat workflow logs and artifacts as potential exfiltration channels; masking is not a substitute for preventing secret exposure.

Reproducibility and concurrency

Use lockfiles and deterministic installation commands. Separate build from deployment, promote the same verified artifact, and attach provenance rather than rebuilding per environment. Apply concurrency groups to prevent stale deployments from racing newer commits, while choosing cancellation semantics carefully for stateful operations.

Key Takeaways

Model workflows as programs executing with capabilities and untrusted inputs.
Grant the minimum token permissions at the narrowest job scope.
Prefer short-lived federated identity and immutable dependencies.
Build once, verify provenance, and promote the identical artifact through environments.

Daily Dose of DevOps — Terraform remote state explained

Marco — Wed, 16 Sep 2026 12:21:49 +0000

Terraform Remote State: A Consistency Boundary for Infrastructure Control

Terraform state is not a cache that can be casually regenerated. It is the controller’s mapping between declarative addresses and real provider objects, including dependency metadata and sensitive attributes. Remote state turns that mapping into a shared consistency boundary for teams and automation.

State, locking, and serializability

Concurrent applies are competing writers. Without coordination, each run may calculate a valid plan from the same prior snapshot and then overwrite the other’s observations—a lost-update anomaly. A backend lock approximates single-writer serializability, but only if every writer honors it and lock leases are handled safely.

terraform {
  backend "s3" {
    bucket       = "org-terraform-state"
    key          = "production/network.tfstate"
    region       = "eu-west-1"
    encrypt      = true
    use_lockfile = true
  }
}

Encryption at rest is necessary but insufficient. Use narrowly scoped identities, transport encryption, access logs, object versioning, retention controls, and tested recovery. State often contains credentials or connection material even when configuration marks outputs as sensitive; “sensitive” primarily controls presentation, not storage.

Partition by failure domain

A single monolithic state increases lock contention, plan latency, privilege breadth, and blast radius. Excessive fragmentation, however, produces brittle cross-state dependencies and coordination overhead. Prefer boundaries aligned with ownership, lifecycle, privilege, and failure domains. Exchange stable identifiers through explicit interfaces rather than exposing entire state snapshots.

Recovery discipline

Never repair state by editing JSON under pressure. First stop writers, preserve the current object and its versions, compare state with provider reality, and use supported operations such as import, moved blocks, or state move. A backend backup is only credible after a restore exercise demonstrates recovery point and recovery time objectives.

Key Takeaways

Remote state is a critical consistency and security boundary.
Locking prevents competing writers only when all automation uses the same backend discipline.
Partition state along operational boundaries, not arbitrary directory structure.
Version, audit, encrypt, restrict, and regularly test restoration.

Daily Dose of DevOps — Kubernetes readiness vs liveness probes

Marco — Tue, 15 Sep 2026 12:24:00 +0000

Kubernetes Probes as Failure Detectors: Semantics, Timing, and Cascading Risk

Kubernetes probes are distributed-systems failure detectors with different control effects. A readiness failure removes a Pod from Service endpoints; a liveness failure asks the kubelet to restart the container. The distinction matters because detection is necessarily imperfect: aggressive thresholds reduce detection latency but increase false positives under transient load.

Readiness protects traffic; liveness repairs deadlock

Readiness should answer: “Can this replica safely accept new work now?” It may include critical local state and indispensable downstream dependencies, but indiscriminately probing every dependency can create a cascade: one database slowdown marks every replica unready, eliminating all capacity precisely when graceful degradation is needed.

Liveness should answer a narrower question: “Is the process irrecoverably stuck such that restart is the best available remediation?” It should not fail because a remote dependency is unavailable. Restarting healthy processes during a network partition adds cold-start pressure without repairing the dependency.

startupProbe:
  httpGet: { path: /health/startup, port: 8080 }
  periodSeconds: 5
  failureThreshold: 30
readinessProbe:
  httpGet: { path: /health/ready, port: 8080 }
  periodSeconds: 5
  failureThreshold: 2
livenessProbe:
  httpGet: { path: /health/live, port: 8080 }
  periodSeconds: 10
  failureThreshold: 3

The startup probe creates a temporal firewall: until initialization succeeds, liveness checks are suppressed. This prevents slow but valid startup from entering a restart loop.

Deriving thresholds

Probe timing should follow measured distributions, not folklore. Let the check interval be p, timeout t, and failure threshold f. Approximate worst-case detection latency is p × f, plus request timeout effects. Choose this against the service’s recovery-time objective and the cost of a false positive. Then validate under CPU throttling, garbage-collection pauses, dependency latency, and node pressure.

Key Takeaways

Readiness changes routing; liveness triggers restart; startup protects initialization.
Keep liveness local and conservative.
Design readiness for graceful degradation rather than dependency-amplified outages.
Derive timings from latency distributions and test them under realistic resource pressure.

Daily Dose of DevOps — What is CI/CD and why it matters

Marco — Mon, 14 Sep 2026 13:33:02 +0000

CI/CD as a Control System: From Commit Entropy to Production Evidence

The control loop

A defensible pipeline evaluates more than functional correctness:

Provenance: Can every artifact be traced to reviewed source and a reproducible build?
Security: Were dependencies, secrets, permissions, and artifact signatures evaluated?
Operability: Do latency, saturation, error-rate, and rollback signals exist before promotion?
Change risk: Is rollout scope proportional to uncertainty?

promote:
  needs: [unit, integration, policy, sbom]
  environment: production
  steps:
    - run: cosign verify --key cosign.pub artifact.example/app:$GIT_SHA
    - run: deploy --strategy=canary --initial-traffic=1%
    - run: verify-slo --window=15m --max-error-budget-burn=2

Why small batches dominate

Failure modes

Key Takeaways

CI/CD is a socio-technical feedback system, not merely a collection of scripts.
Optimize for evidence quality, small batch size, bounded blast radius, and reversible change.
A green pipeline is a risk estimate; production telemetry must close the control loop.
Measure lead time and deployment frequency alongside change-failure rate and recovery time.

Daily Dose of DevOps — GitHub Actions basics for DevOps

Marco — Sun, 13 Sep 2026 12:29:13 +0000

GitHub Actions as a Capability System: Secure Automation by Construction

Minimize authority

permissions:
  contents: read

jobs:
  deploy:
    environment: production
    permissions:
      contents: read
      id-token: write
    steps:
      - uses: actions/checkout@<immutable-commit-sha>
      - run: npm ci --ignore-scripts
      - run: npm test
      - run: ./scripts/deploy.sh

Reproducibility and concurrency

Key Takeaways

Model workflows as programs executing with capabilities and untrusted inputs.
Grant the minimum token permissions at the narrowest job scope.
Prefer short-lived federated identity and immutable dependencies.
Build once, verify provenance, and promote the identical artifact through environments.

Daily Dose of DevOps — Terraform remote state explained

Marco — Sat, 12 Sep 2026 11:25:02 +0000

Terraform Remote State: A Consistency Boundary for Infrastructure Control

State, locking, and serializability

terraform {
  backend "s3" {
    bucket       = "org-terraform-state"
    key          = "production/network.tfstate"
    region       = "eu-west-1"
    encrypt      = true
    use_lockfile = true
  }
}

Partition by failure domain

Recovery discipline

Key Takeaways

Remote state is a critical consistency and security boundary.
Locking prevents competing writers only when all automation uses the same backend discipline.
Partition state along operational boundaries, not arbitrary directory structure.
Version, audit, encrypt, restrict, and regularly test restoration.