DEV Community: Dean

I Built Agentflow Because More Coding Agents Were Making Me Slower

Dean — Fri, 14 Aug 2026 08:56:09 +0000

I started with a simple assumption: if one coding agent was useful, several
coding agents working in parallel would be even better.

Like most, I figured out that works until a point where you are just copying and pasting outputs between sessions, and sessions would get worse on delivering as time and context grew.

Codex, Claude Code, and GitHub Copilot could each write, research, review, or test a bounded piece of work. The problem was everything between those pieces.

Agents duplicated effort, missed repository-specific instructions, reviewed stale changes, and returned individually plausible results that did not form a coherent whole.

I had created more execution capacity, but I had also made myself the
scheduler, message bus, status dashboard, and integration queue.

That problem eventually became Agentflow: an open-source, provider-neutral CLI for coordinating coding agents around an approved goal.

The problem was coordination, not intelligence

The first failure mode was easy to spot. If I divided one content creation task among several writers, every individual edit could look reasonable while the complete experience lost its voice, transitions, or teaching sequence.

My issue was the lack of true shared state between agents and models that I was running.

A long-running piece of work was spread across:

the controller's chat history;
Markdown plans and handoff files;
agent sessions and terminal panes;
Git branches and worktrees;
pull requests and CI; and
whatever I could remember when an agent stopped or a chat disconnected.

None of those was a reliable workflow graph. A terminal output could tell me that an agent had exited, but not that its result was accepted. A claimed task did not prove that a worker was alive. A passing test did not prove that every acceptance condition had evidence.

The natural response is often to write a larger prompt, which was never going to solve the issue. It made the controller more expensive and turned its context window into an increasingly fragile database.

The solution I chose

I decided to separate coordination from execution.

Agentflow uses Beads as durable task state: goals, dependencies, claims, decisions, review stages, and acceptance evidence. Agentflow adds the control plane around that graph:

one approved root goal;
one persistent, resumable controller lease;
exact, atomic task claims;
bounded provider-neutral handoffs;
model, skill, base-commit, scope, and budget preflight;
structured worker results and review dispositions;
deterministic waiting for CI or other external state; and
explicit completion or human-action-required outcomes.

The coding agents remain disposable execution contexts. Git and GitHub remain the integration boundary. Repository instructions and domain skills remain the source of truth for the work itself.

human-approved goal
        |
        v
Agentflow controller <----> Beads task and evidence graph
        |
        v
typed preflight and bounded handoff
        |
        +----> Codex
        +----> Claude Code
        +----> GitHub Copilot CLI
        |
        v
isolated worktree or bounded directory
        |
        v
structured result -> review -> validation -> integration

Agentflow is not trying to become another coding agent, and it does not replace project expertise. It decides what is ready, who may
own it, what evidence must come back, and whether the overall workflow may advance.

Originally I wondered if just having beads was enough, but unfortunately I needed that external harness outside of all the model specific tooling, that could become a co-ordinator. Plus some other features.

Why not just use agent teams or more terminal tabs?

Native subagents are useful, and observable terminal sessions are useful. I use both, but found neither is durable project state.

Agentflow can launch observable external sessions through Herdr, or work with native provider capabilities, but a session is only an execution lane. The controller still has to validate the launch, reconcile the result, disposition review findings, and decide whether the root goal is complete.

This also means a dropped IDE connection, a sleeping laptop, or a provider error does not require a new plan. A fresh chat can reattach to the same root, inspect the durable state, reconcile completed work, and continue without replaying the old transcript.

I had a ChatGPT outage during a particular long session, a lot of things in the session folder were just left in a weird state that was a nightmare to get the existing session to recover from. Again, with agentflow and beads, I'd hope to avoid those issues in the future.

What Agentflow became

The project grew from a set of personal prompts and scripts into a public CLI with:

Git-backed and Gitless workflows;
adapters for Codex, Claude Code, and GitHub Copilot CLI;
configurable local or project skills;
persistent controller recovery and context rotation;
optional Herdr session tracking;
pull-based workers and staged review;
acceptance-to-evidence matrices;
exact-base worktrees and safe integration checks;
deterministic CI watchers;
local usage and provenance indexes that do not copy full transcripts;
fail-closed model and asset policy; and
a conditional prose-quality lane for reader-facing content.

The prose lane came from another practical lesson. A model can write correct documentation while still producing far too much of it. I'm looking at you Claude Opus 5 who likes to spout absolute word-salad into anything it produces.

Agentflow can first run free deterministic checks, then—only when the workflow opts in—route one bounded editing pass through a configured model. The original writer does not get an unlimited rewrite loop.

The result is not “press a button and trust a swarm.” It is a way to let several agents contribute without giving any worker authority over the whole outcome.

Install the public preview

Agentflow is available from
GitHub.

At the time of writing, 0.0.4 is a public preview intended for development and testing—not unattended privileged automation.

Let's be honest, I'll never claim production-ready for something I built on the back of a late night idea, and nobody is paying me to support. =D

You need Python 3.10 or later, Beads 1.1 or later, Git for Git-backed projects, and at least one supported coding-agent CLI. Herdr and the GitHub CLI are optional.

Install the versioned preview with uv or pipx:

uv tool install "git+https://github.com/saintdle/agentflow.git@v0.0.4"

# Alternatively:
pipx install "git+https://github.com/saintdle/agentflow.git@v0.0.4"

Verify it, then preview and install its provider-neutral workflow assets:

agentflow --version
agentflow doctor
agentflow install --dry-run
agentflow install

Agentflow preserves existing provider files for review instead of silently
replacing them.

A quick terminal workflow

In an existing Git repository:

cd /path/to/project
agentflow init . --beads
agentflow beads status .
agentflow doctor

Initialization creates missing workflow adapters, keeps runtime state out of Git (by configuring a .gitignore), and leaves existing agent instructions and hooks in place.

Once you have created and approved a Beads root and its task graph, start the persistent controller from the terminal:

agentflow controller start \
  --root "$PWD" \
  --workflow-root <approved-root-id> \
  --controller agentflow-controller

Without --once, the controller continues through ready work, result
collection, review, remediation, validation, and local integration until it reaches GOAL_COMPLETE, USER_ACTION_REQUIRED, a durable task block, or its deadline.

The repository includes a complete
first-workflow tutorial
showing how to create the root and task, attach acceptance evidence, inspect a halt, and complete the workflow.

Use it from a Codex or Claude chat

You do not have to operate the lower-level commands yourself. And this is how I consume agentflow most days.

Open the target project in your IDE, start an agent-mode Codex/ChatGPT or Claude session with
terminal access, and ask it to prepare the workflow:

Use the installed shape-goal, to-tickets, and orchestrate-agents skills with Agentflow for this workspace.

Goal: [describe the outcome]

Inputs: [files, issues, feedback, or specifications]

Constraints: [write boundaries, allowed tests, model/provider choices, and any
push, merge, deployment, hosted test, or external action requiring approval]

Inspect the repository and its relevant instructions and domain skills. Create
one measurable goal contract and a dependency-aware Beads graph with exact
ownership, model routes, budgets, review lanes, acceptance evidence, and halt
conditions. Persist the proposal, but do not claim tasks, launch workers, edit
implementation files, push, merge, or mutate hosted state. Explain the plan in
ordinary language and finish with one exact approval message.

To be honest I've had much success with being less verbose than this such as the below in codex chat session:

I want you to research all the VMware topics at veducate.co.uk and compare to recent posts on reddit.com/r/vmware, tell me which new blog posts would be useful for veducate. 

Use $orchestrate-agents for this research work, save output in $directory

The response will provide you the opportunity to review the proposed goal and boundaries. If they are correct, approve the root
in the same chat or a fresh controller chat:

Approved. Use Agentflow root [root-id] in the workspace currently open in the
IDE. Treat its persisted contract, graph, permissions, model routes, evidence
lanes, and halt conditions as authoritative.

Acquire or reattach the single root-controller lease and run the persistent
controller without --once. Continue through ready implementation, result
collection, independent review, accepted remediation, deterministic validation,
integration, and local delivery. Stop only at GOAL_COMPLETE,
USER_ACTION_REQUIRED, a durable task block that needs my decision, or the
persisted deadline. Do not push, merge, deploy, run hosted tests, or mutate
hosted state unless the contract explicitly authorizes it. Never make me relay
routine worker output between chats.

That last sentence addresses the original problem directly. The human approves
the goal and consequential actions; the controller handles routine coordination.

Or again, you could work in a simplier format

goal approved

The full
chat-first guide
also includes prompts for reconnect-safe resume, read-only status, context
rotation, and a separately authorized pull-request delivery step.

Bring your own skills

Agentflow's bundled skills deal with workflow rather than your product domain.
You can register a repository skill or an explicit local skill source without repackaging the CLI:

agentflow skills add ./skills/my-domain-skill
agentflow skills sync
agentflow skills list
agentflow skills doctor

This was an important design correction for me. The general orchestration skill implemented by Agentflow should never outrank the repository's own technical or editorial knowledge.

Agentflow routes the correct domain skill into the bounded task and pins what the worker actually received.

The longer build story

I documented the design as it evolved on vEducate:

Building a Coding-Agent Scheduler with Codex, Claude Code and GitHub Copilot covers the original coordination problem, early controller design, work allocation, review, CI, and integration.
How to Build a Durable Coding-Agent Scheduler with Beads covers the move from Markdown and chat state to a durable dependency graph, pull-based work, resumability, deterministic waiting, and stronger evidence boundaries.

Agentflow is the culmination of those experiments: the reusable CLI, skills, policy, tests, documentation, and release process extracted from the workflow I had been using locally.

It is still early software. The repository is explicit about its AI-assisted authorship and its development/testing status (Yes I used agentflow to help develop itself further).

If you try it, review the plan, permissions, diffs, and evidence just as carefully as you would review the work of any other automation.

The project, installation guide, examples, and issue tracker are all available
at github.com/saintdle/agentflow.

Kubernetes Metric Server – cannot validate certificate because it doesn’t contain any IP SANs

Dean — Thu, 27 Jul 2023 17:09:32 +0000

The Issue

Whilst trying to install the Metric’s server:

kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml

so I could use kubectl top node for it’s metrics on Node resource useage, I found the pods were not loading, and upon inspection found the following:

> kubectl logs -n kube-system metrics-server-6f6cdbf67d-v6sbf 

I0717 12:19:32.132722 1 server.go:187] "Failed probe" probe="metric-storage-ready" err="no metrics to serve"
E0717 12:19:39.159422 1 scraper.go:140] "Failed to scrape node" err="Get \"https://192.168.49.2:10250/metrics/resource\": x509: cannot validate certificate for 192.168.49.2 because it doesn't contain any IP SANs" node="minikube"

The Cause

The issue here was due to the installation of Cert-Manager and setting up some TLS configurations within the CNI and Self-Signed certificates, the metric’s server wasn’t able to validate the authority of the Kubernetes API

The Fix

As this is communication within the cluster, I could simply fix this by telling Metric Server container to trust the insecure certificates from the API using the below

kubectl patch command:

kubectl patch deployment metrics-server -n kube-system --type='json' -p='[{"op": "add", "path": "/spec/template/spec/containers/0/args/-", "value": "--kubelet-insecure-tls"}]'

Regards

Follow @Saintdle

Dean Lewis

The post Kubernetes Metric Server – cannot validate certificate because it doesn’t contain any IP SANs appeared first on vEducate.co.uk.

Interview with Daniel Bryant, Ambassador Labs – Kubernetes, PaaS, Err what’s next?

Dean — Thu, 21 Jul 2022 23:05:12 +0000

After KubeCon EU 2022, I had the chance to connect with Daniel Bryant, Head of DevRel at Ambassador Labs, and expand on his extremely popular KubeCon Session further.

I wanted to take his session, and also give it a platform/infra focus, as this is the background and world I'm from in IT, along with a lot of customer teams I work with, and I assume I'm not alone in this. I think we gave enough coverage of the new cloud native concepts and linking this back to the changing skills of platform admin.
My 25 minute marker soon ran over, and we recorded 47 minutes or so. Rather than cut it back or things out, I decided to release the full interview as two parts, to be enjoyed over some extended coffee breaks (tell your boss to blame me if your late back to work 😉 ).

Hopefully for those of you whom are interested, you enjoy the recordings!

vEducate.co.uk - Interview with Daniel Bryant, Ambassador Labs - Kubernetes, PaaS, Err what's next? with a Platform/Infra point-of-view

Quick Fix – AWS Console – Current user or role does not have access to Kubernetes objects on this EKS Cluster

Dean — Tue, 25 Jan 2022 13:30:41 +0000

The Issue

Once you’ve deployed an EKS cluster, and try to view this in the AWS Console, you are presenting the following message:

Your current user or role does not have access to Kubernetes objects on this EKS Cluster

The Cause

This is because you need to run some additional configuration on your cluster to allow your AWS user IAM to access the cluster.

AWS EKS Docs – Enabling IAM user and role access to your cluster

The Fix

Grab your User ARN from the Identity and Access Management (IAM) page.

Download this template YAML file for configuring the necessary ClusterRole and ClusterRoleBinding and then apply it to your EKS cluster.

curl -o eks-console-full-access.yaml https://amazon-eks.s3.us-west-2.amazonaws.com/docs/eks-console-full-access.yaml

kubectl apply -f eks-console-full-access.yaml

Now edit the following configmap:

kubectl edit configmap/aws-auth -n kube-system

Add in the following under the data tree:

mapUsers: |
  - userarn: arn:aws:iam::3xxxxxxx7:user/dean@veducate.co.uk
    username: admin
    groups:
      - system:masters

After a minute or so, once you revisit the EKS Cluster page in the AWS console, you will see all the relevant details.

Regards

Follow @Saintdle

Dean Lewis

The post Quick Fix – AWS Console – Current user or role does not have access to Kubernetes objects on this EKS Cluster appeared first on vEducate.co.uk.

Using the new vSphere Kubernetes Driver Operator with Red Hat OpenShift via Operator Hub

Dean — Wed, 05 Jan 2022 10:36:03 +0000

What is the vSphere Kubernetes Driver Operator (VDO)?

This Kubernetes Operator has been designed and created as part of the VMware and IBM Joint Innovation Labs program. We also talked about this at VMworld 2021 in a joint session with IBM and Red Hat. With the aim of simplifying the deployment and lifecycle of VMware Storage and Networking Kubernetes driver plugins on any Kubernetes platform, including Red Hat OpenShift.

This vSphere Kubernetes Driver Operator (VDO) exposes custom resources to configure the CSI and CNS drivers, and using Go Lang based CLI tool, introduces validation and error checking as well. Making it simple for the Kubernetes Operator to deploy and configure.

The Kubernetes Operator currently covers the following existing CPI, CSI and CNI drivers, which are separately maintained projects found on GitHub.

This operator will remain CNI agnostic, therefore CNI management will not be included, and for example Antrea already has an operator.

vSphere Antrea CNI Driver

Below is the high level architecture, you can read a more detailed deep dive here.

Installation Methods

You have two main installation methods, which will also affect the pre-requisites below.

If using Red Hat OpenShift, you can install the Operator via Operator Hub as this is a certified Red Hat Operator. You can also configure the CPI and CSI driver installations via the UI as well.

Alternatively, you can install the manual way and use the vdoctl cli tool, this method would also be your route if using a Vanilla Kubernetes installation.

This blog post will cover the UI method using Operator Hub.

Pre-requisites

Kubernetes and vSphere environment must meet the following:

vSphere 6.7U3(or later) is supported for VDO
Virtual Machine hardware version should be version 15(or later)
Enable Disk UUID(disk.EnableUUID) on all node vm’s
K8s master nodes should be able to communicate with vCenter management interface
Disable Swap(swapoff -a) on all Kubernetes nodes at the Guest Operating System level.

If you are going to deploy this on a Vanilla Kubernetes instance or want to use the CLI tooling:

Clone the VDO GitHub Repo or download the files from the release page. This installation method will be covered separately in another blog post.

git clone https://github.com/vmware-tanzu/vsphere-kubernetes-drivers-operator

Install Go, so that we can use the vdoctl command line tool.

https://go.dev/doc/install

Installing and configuring the vSphere Kubernetes Driver Operator

Installation via Red Hat Operator Hub UI

Create a project to install the Operator into.
- In this example I have used “vsphere-kubernetes-drivers-operator” and will match the below SecurityContext example

Create the Security Context Constraint, so that the Service Account can access the relevant resources.
- Administration > Custom Resource Definitions > SecurityContextConstraints > Instance > Create

# Example SCC [Source and further details](https://github.com/vmware-tanzu/vsphere-kubernetes-drivers-operator/blob/main/docs/getting-started/getting-started-from-operator-hub.md#pre-requisites)
# Used for CSI 2.3.0 and later, ensure the namespace in bold below matches the one you have created earlier

apiVersion: security.openshift.io/v1
kind: SecurityContextConstraints
metadata:
  name: example
allowPrivilegedContainer: true
allowHostDirVolumePlugin: true
allowHostNetwork: true
allowHostPorts: true
defaultAddCapabilities:
- SYS_ADMIN
runAsUser:
  type: RunAsAny
seLinuxContext:
  type: RunAsAny
fsGroup:
  type: RunAsAny
users:
- system:serviceaccount: **vsphere-kubernetes-drivers-operator** :vdo-controller-manager
- system:serviceaccount:vmware-system-csi:vsphere-csi-node

Now we will install the Operator.

Go to OperatorHub and search for “vsphere-kubernetes-driver-operator”
Click the Operator to install it
Ensure it is installed to the namespace you have created, where the SCC is linked.

Note: I recommend that you ensure you are installing/running Operator version 0.1.5 or higher. This blog post used 0.1.3 in some of the images. However, a number of enhancements were introduced, and I upgraded to 0.1.5 part way through (see [this section for upgrades](https://veducate.co.uk/vsphere-kubernetes-operator-openshift#Performing_upgrades_of_the_Operator_and_deployed_CPICSI))

Once completed, click to view the Operator, and we will continue to configure the drivers.

Configuring the CPI and CSI via the Operator in the OpenShift Cluster UI

You will need the following pieces of information

IP address or FQDN of vCenter
- If using a secure connection to the vCenter, you will need to provide the SSL thumbprint
Credentials for the vCenter with the appropriate permissions.
Datacenter(s) Names within the vCenter. This is required by CPI and CSI to manage the cluster

We now need to create a source secret to hold our credentials for our vCenter.

In the OpenShift Cluster UI > Workloads > Secrets
Ensure you are in the namespace of “kube-system”
Create > Source Secret

Provide a name for the secret
Authentication type should be set to “basic”
Provide the username and password and click create

Go back to the Installed Operators Page, set the namespace where you installed the operator and select it.

First we will create the vSphere Cloud Config, which is the connection and credential data for our vCenter.

Click Create Instance under “VsphereCloudConfig”

Provide a name for the configuration and any labels as necessary.
Credentials – provide the name of the source secret created earlier in the kube-system namespace
Provide your datacenter names
Select Insecure configuration if necessary
- If unticked, you need to provide a thumbprint for the vCenter SSL
VC IP – you can provide either IP or FQDN here.

Click create.

We will now configure the VDOConfig, which will control which drivers are deployed and the credentials to use.

Click to create a VDOConfig instance.

Provide a name for the configuration and any labels as necessary.
Then open the Storage Provider heading.

Provide the vSphere Cloud Config instance name we have just created
Provide the ClusterDistribution Name – for this blog it will of course be OpenShift
But remember this VDO is available for any vanilla K8s setup
Provide a custom kubelet path if applicable
Provide configuration for VSAN File Services volumes access if applicable

Now open the Cloud Provider configuration. If you have already installed the cloud provider in your environment, you do not need to configure this section. However, the Cloud Provider is a mandatory requirement to be installed when using the vSphere CSI Driver. So, if you don’t have it installed already, install it as part of this configuration.

Provide any topology information if applicable. You can read more about deploying the vSphere CSI and CPI in a topology aware mode here.
Provide the name for the vSphere Cloud Config
Click Create

You can monitor the Operator performing its actions by going to the “vdo-controller-manager” pod, and viewing the logs from the “manager” container.

Below you can see the reconciler has picked up the configuration instances and is now attempting to install the CPI. You can follow the logs for the full status.

Another Option is to view the VDOConfig instance in the Operator to see the status.

Once completed, you can check the status of the Pods for the configured CPI and CSI via the UI in the highlighted projects, kube-system for CPI, vmware-system-csi for CSI.

Or by running the following command:

# CPI
oc get pods -l k8s-app=vsphere-cloud-controller-manager

# CSI
oc get pods -n vmware-system-csi

Testing and validating the installation

To test the installation, we will configure a StorageClass and then a Persistent Volume Claim.

To go StorageClasses under Storage
Click Create StorageClass

Either use the form to fill in, and select the provisioner as “csi.vmware.com” or use the “Edit YAML” view and paste in your configuration such as the below example.

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: csi-sc-vmc
  annotations:
    storageclass.kubernetes.io/is-default-class: "false"
provisioner: csi.vsphere.vmware.com
parameters:
  StoragePolicyName: "vSAN Default Storage Policy"
  datastoreURL: "ds:///vmfs/volumes/vsan:3672d400f5fa4515-8a8cb78f6b972f74/"

To create a Persistent Volume Claim (PVC)

Under Storage navigation heading select PersistentVolumeClaims
Click Create PersistentVolumeClaim

Provide the necessary details, such as selecting your Storage Class and the correct Volume mode. Again you have the ability to use the “Edit YAML” option and provide the configuration such as the below example:

kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: veducate-blog-test-pvc
  labels:
    name: veducate-blog-test-pvc
  annotations:
    volume.beta.kubernetes.io/storage-class: veducate-csi
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi

And we are looking for a status of Bound.

Performing upgrades of the Operator and deployed CPI/CSI

The operator will follow the upgrade option you provided during install, either automatically when a new version is released, or manually. You can read more about this behaviour on the Red Hat OpenShift Documentation site.

The CPI and CSI will be installed and aligned to the compatibility matrix in use. You can check which file is in use by going to:

Workloads > Config Maps > Ensure you are in the vSphere Kubernetes Driver Operator namespace > compat-matrix-config

Drivers can be updated by updating the compatibility matrix using the following command:

vdoctl update compatibility-matrix <path-to-updated-compat-matrix>

# Existing pods of CloudProvider and StorageProvider are terminated and new pods are spawned according to the compatible versions of CSI and CPI

# You can either provide your own file, such as one which is edited with the locations of your own modified vSphere CSI deployment files, or a later file from the [GitHub Repo releases page.](https://github.com/vmware-tanzu/vsphere-kubernetes-drivers-operator/releases)

Summary

This new operator does what it sets out to achieve, to simplify the deployment, configuration and lifecycle of the vSphere Kubernetes Drivers. And for the Red Hat OpenShift customers, it’s fully supported and certified.

The VMware team have made the tooling and ability to consume this new “Master Operator of the Drivers” flexible in terms of consumption, and simple.

Regards

Follow @Saintdle

Dean Lewis

The post Using the new vSphere Kubernetes Driver Operator with Red Hat OpenShift via Operator Hub appeared first on vEducate.co.uk.

Deleting AWS EKS Cluster fails – Cannot evict pod as it would violate the pod’s disruption budget

Dean — Wed, 22 Dec 2021 11:38:05 +0000

The Issue

I had to remove a demo EKS Cluster where I had screwed up an install of a Service Mesh. Unfortunately, it was left in a rather terrible state to clean up, hence the need to just delete it.

When I tried the usual eksctl delete command, including with the force argument, I was hitting errors such as:

2021-12-21 23:52:22 [!] pod eviction error ("error evicting pod: istio-system/istiod-76f699dc48-tgc6m: Cannot evict pod as it would violate the pod's disruption budget.") on node ip-192-168-27-182.us-east-2.compute.internal

With a final error output of:

Error: Unauthorized

The Cause

Well, the error message does call out the cause, moving the existing pods to other nodes is failing due to the configured settings. Essentially EKS will try and drain all the nodes and shut everything down nicely when it deletes the cluster. It doesn’t just shut everything down and wipe it. This is because inside of Kubernetes there are several finalizers that will call out actions to interact with AWS components (thanks to the integrations) and nicely clean things up (in theory).

To get around this, I first tried the following command, thinking if delete the nodegroup without waiting for a drain, this would bypass the issue:

 eksctl delete nodegroup standard --cluster veducate-eks --drain=false --disable-eviction

This didn’t allow me to delete the cluster however, I still got the same error messages.

The Fix

So back to the error message, and then I realised it was staring me in the face!

Cannot evict pod as it would violate the pod's disruption budget

What is a Pod Disruption Budget? It’s essentially a way to ensure availability of your pods from someone killing them accidentality.

A PDB limits the number of Pods of a replicated application that are down simultaneously from voluntary disruptions. For example, a quorum-based application would like to ensure that the number of replicas running is never brought below the number needed for a quorum. A web front end might want to ensure that the number of replicas serving load never falls below a certain percentage of the total.

To find all configured Pod Disruption Budgets:

kubectl get poddisruptionbudget -A

Then delete as necessary:

kubectl delete poddisruptionbudget {name} -n {namespace}

Finally, you should be able to delete your cluster.

Regards

Follow @Saintdle

Dean Lewis

The post Deleting AWS EKS Cluster fails – Cannot evict pod as it would violate the pod’s disruption budget appeared first on vEducate.co.uk.

Deploying Nvidia GPU enabled Tanzu Kubernetes Clusters

Dean — Thu, 09 Dec 2021 23:01:04 +0000

In this blog post I’m going to detail how deploy and configure a Nvidia GPU enabled Tanzu Kubernetes Grid cluster in AWS. The method will be similar for Azure, for vSphere there are a number of additional steps to prepare the system. I’m going to essentially follow the official documentation, then run some of the Nvidia tests. Like always, it’s good to get a visual reference and such for these kinds of deployments.

Pre-Reqs

Nvidia today only support Ubuntu deployed images in relation to a TKG deployment
For this blog I’ve already deployed my TKG Management cluster in AWS

Deploy a GPU enabled workload cluster

It’s simple, just deploy a workload cluster that for the compute plane nodes (workers) that uses a GPU enabled instance.

You can create a new cluster YAML file from scratch, or clone one of your existing located in:

~/.config/tanzu/tkg/clusterconfigs

Below are the four main values you will need to change. As mentioned above, you need a GPU enabled instance, and for the OS to be Ubuntu. The OS version will default if not set to 20.04.

CONTROL_PLANE_MACHINE_TYPE: t3.large
NODE_MACHINE_TYPE: g4dn.xlarge
OS_ARCH: amd64
OS_NAME: ubuntu
OS_VERSION: "20.04

The rest of the file you configure as you would for any workload cluster deployment.

Create the cluster.

tanzu cluster create {name} -f {cluster.yaml}

You can retrieve the kubeadmin file to login by running.

tanzu cluster kubeconfig get {cluster_name} --admin

Deploying the Nvidia Kubernetes Operator

Change the kubectl context to your newly deployed cluster.

Deploying the Nvidia operator couldn’t be easier, you can either download the files from the Cluster API for AWS github repo, or directly install them.

kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/cluster-api-provider-aws/de4fd54e6f988ca7fd3f94bce46867ba0523e23b/test/e2e/data/infrastructure-aws/gpu/clusterpolicy-crd.yaml

kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/cluster-api-provider-aws/de4fd54e6f988ca7fd3f94bce46867ba0523e23b/test/e2e/data/infrastructure-aws/gpu/gpu-operator-components.yaml

Validate the installation

Validate the operator pods in the default namespace, and then “nvidia” pods in the namespace “gpu-operator-resources”.

kubectl get pods

kubectl get pods -n gpu-operator-resources

If you scale out your cluster with additional nodes, the Nvidia operator will ensure the additional pods run on the new nods.

Running the Sample Applications

From here to further validate, I am running the sample applications from the Nvidia documentation.

So rather than copy the exact configs here, I’m just showing the outputs.

CUDA VectorAdd

CUDA load generator

If you want to look at further examples, Nvidia have some fantastic Deep Learning examples in this repository.

https://github.com/NVIDIA/DeepLearningExamples

Wrap-up and Resources

Hopefully you can see that to use the GPU support with a Tanzu Kubernetes Grid cluster is quick and simple to setup and consume.

Blog – VMware Tanzu Kubernetes Grid Now Supports GPUs Across Clouds
Documentation
- Deploy Tanzu Kubernetes Clusters to Amazon EC2
- Deploy a GPU-Enabled Cluster

Regards

Follow @Saintdle

Dean Lewis

The post Deploying Nvidia GPU enabled Tanzu Kubernetes Clusters appeared first on vEducate.co.uk.

Upgrading the vSphere CSI Driver (Storage Container Plugin) from v2.1.0 to latest

Dean — Mon, 15 Nov 2021 21:54:10 +0000

In this post I’m just documenting the steps on how to upgrade the vSphere CSI Driver, especially if you must make a jump in versioning to the latest version.

Upgrade from pre-v2.3.0 CSI Driver version to v2.3.0

You need to figure out what version of the vSphere CSI Driver you are running.

For me it was easy as I could look up the Tanzu Kubernetes Grid release notes. Please refer to your deployment manifests in your cluster. If you are still unsure, contact VMware Support for assistance.

Then you need to find your manifests for your associated version. You can do this by viewing the releases by tag.

Then remove the resources created by the associated manifests. Below are the commands to remove the version 2.1.0 installation of the CSI.

kubectl delete -f https://raw.githubusercontent.com/kubernetes-sigs/vsphere-csi-driver/v2.1.0/manifests/latest/vsphere-7.0u1/vanilla/deploy/vsphere-csi-controller-deployment.yaml

kubectl delete -f https://raw.githubusercontent.com/kubernetes-sigs/vsphere-csi-driver/v2.1.0/manifests/latest/vsphere-7.0u1/vanilla/deploy/vsphere-csi-node-ds.yaml

kubectl delete -f https://raw.githubusercontent.com/kubernetes-sigs/vsphere-csi-driver/v2.1.0/manifests/latest/vsphere-7.0u1/vanilla/rbac/vsphere-csi-controller-rbac.yaml

Now we need to create the new namespace, “vmware-system-csi”, where all new and future vSphere CSI Driver components will run.

kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/vsphere-csi-driver/v2.3.0/manifests/vanilla/namespace.yaml

Next, we migrate the existing vSphere configuration secret from its location in the “kube-system” namespace to the new “vmware-system-csi” namespace.

kubectl get secret vsphere-config-secret --namespace=kube-system -o yaml | sed 's/namespace: .*/namespace: vmware-system-csi/' | kubectl apply -f -

Delete the original secret in the “kube-system” namespace.

kubectl delete secret vsphere-config-secret --namespace=kube-system

Now to deploy the manifests for the vSphere CSI Driver version 2.3.0

kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/vsphere-csi-driver/v2.3.0/manifests/vanilla/vsphere-csi-driver.yaml

Below you can see all the commands running in my environment.

You can scale the deployment of the vSphere CSI Controller to match the number of Control-Plane nodes in your environment.

kubectl scale deployment vsphere-csi-controller --replicas=1 -n vmware-system-csi

You can check the pods status by running the following command.

kubectl get pods -n vmware-system-csi

Now to check that I can successfully create a PVC and associated Persistent Volume on the vSphere environment still. I used my trusty Pac-Man application for this test.

Upgrade from v2.3.0 to the latest

Now you can upgrade to the latest version, currently v2.4.0, by running the below command.

kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/vsphere-csi-driver/v2.4.0/manifests/vanilla/vsphere-csi-driver.yaml

Summary and wrap-up

The steps do follow the documentation, the main points to remember, if you are running a version below v2.3.0, you need to get to v2.3.0 before then upgrading to the latest version. There will be no changes to your PVCs or PVs.

But if you are unsure about any configuration changes or the status of your environment, log a support call with VMware Support for assistance and validation.

Regards

Follow @Saintdle

Dean Lewis

The post Upgrading the vSphere CSI Driver (Storage Container Plugin) from v2.1.0 to latest appeared first on vEducate.co.uk.

First Look – Setup Tanzu Build Services and rebuilding Pac-Man

Dean — Thu, 11 Nov 2021 00:44:56 +0000

This blog post will detail how to setup Tanzu Build Services in a test environment, and then create a container image from a dockerfile, fixing several vulnerabilities compared to the current container image.

What is Tanzu Build Service?

Tanzu Build Service uses the open-source [Cloud Native Buildpacks](https://buildpacks.io) project to turn application source code into [container images](https://github.com/opencontainers/image-spec/blob/master/spec.md). 

Build Service executes reproducible builds that align with modern container standards, and additionally keeps image resources up-to-date. It does so by leveraging Kubernetes infrastructure with [kpack](https://github.com/pivotal/kpack), a Cloud Native Buildpacks Platform, to orchestrate the image lifecycle. 

Build Service helps you develop and automate containerized software workflows securely and at scale.

You can read more about the Tanzu Build Services concepts here.

Pre-Reqs

Have an accessible Image Registry to both your local client and your Kubernetes cluster.

I used Dockerhub for my lab environment.

Install the Carvel tools.

kapp is a deployment tool that allows users to manage Kubernetes resources in bulk.
ytt is a templating tool that understands YAML structure.
kbld is needed to map relocated images to k8s config.
imgpkg is tool that relocates container images and pulls the release configuration files.

brew tap vmware-tanzu/carvel

brew install ytt kbld kapp imgpkg kwt vendir

Install the kp cli tool.

# Download from the Tanzu Network pages

chmod +x kp-linux-0.4.0 
sudo mv kp-linux-0.4.0 /usr/bin/local/kp

# Install using Brew

brew tap vmware-tanzu/kpack-cli
brew install kp

# Download from GitHub Releases Page
curl -LJO https://github.com/vmware-tanzu/kpack-cli/releases/download/v0.4.2/kp-linux-0.4.2
chmod +x kp-linux-0.4.2
sudo mv kp-linux-0.4.2 /usr/bin/local/kp

Installing Tanzu Build Services

docker login registry.tanzu.vmware.com

docker login **{repo\_url}**

Copy the images from the Tanzu Registry to your registry suing the imgpkg tool.

imgpkg copy -b "registry.tanzu.vmware.com/build-service/bundle: **{version}**" --to-repo **{repo\_url}**

# Example
imgpkg copy -b "registry.tanzu.vmware.com/build-service/bundle:1.3.0" --to-repo saintdle/tbs

Pull the image manifests locally.

imgpkg pull -b **{repo\_url}** -o **{location}**

# Example
imgpkg pull -b "saintdle/tbs:1.3.0" -o /tmp/bundle

Now to deploy Tanzu Build Services, we’ll use the yyt tooling to map values across as needed into the various files:

Local Image Repo URL
- Username and Password
Tanzu Net username and password

ytt -f bundle/values.yaml \
     -f bundle/config/ \
 -v kp_default_repository='{repo_url}' \
 -v kp_default_repository_username='{username}' \
 -v kp_default_repository_password='{password}' \
 -v pull_from_kp_default_repo=true \
 -v tanzunet_username='' \
 -v tanzunet_password='' \
 | kbld -f bundle/.imgpkg/images.yml -f- \
 | kapp deploy -a tanzu-build-service -f- -y

Below is a concatenated output once the command is run.

To check the installation

kp clusterbuilder list

(Re)Building the Pac-Man Application

Now that Tanzu Build Services is deployed within my cluster. Let’s look at taking an existing application and repacking it into a container using the functions of Tanzu Build Services to inject various layers into the container and resolve issues such as vulnerabilities.

I use the same Pac-Man application in my demos, as it’s fun, but also does somewhat mirror a real-world application, as it has a web front end and a database backend.

GitHub Repo for Pac-Man for Kubernetes/Tanzu

The Web Front end is a container built by Ivan Font.

quay.io/ifont/pacman-nodejs-app:latest

Ivan has also provided his Dockerfile for this container here:

https://github.com/font/pacman

I forked this repo, and then without making any changes currently. I ran the commands for Tanzu Build Services to build me a new container and fix some of the dependency issues straight away.

First, we need to create a secret for the kp tool to use to upload our images to the registry.

kp secret create my-dockerhhub-creds --dockerhub saintdle

# you will be prompted to enter the password for the account

Next, we run the command to compile the new image.

kp image create pacman-test3 --tag saintdle/pacmantest:0.1 --git https://github.com/saintdle/pacman.git

# specify a certain branch
kp image create {name} --tag {repository location to create image} --git {git url} --git-revision {git branch or commit}

We can then monitor the build process with the following commands.

kp build list pacman-test3

# you can add the watch command infront of this to cycle the command/response

watch kp build list pacman-test3

You can view further build details by viewing the logs.

kp build logs pacman-test3

The build process goes through the following stages, and you check out the full output in my example in the image below:

Setup CA certs for the repo that the process is pushing the image to
Prepare the source files
Detect the build packs that will be needed for this image build run
Analyze if any elements reuse existing cached data
Restore any cached data as needed
Build our new image
Export our image to our container image registry
Completion – finalisation messaging

Testing the new Image

As I’m doing this as a first look and not really reading any documentation and jumping straight in, I don’t have this as part of a CI/CD pipeline.

I just ran the new container image in docker locally on it’s own to make sure it works and exposed the port 8080 to port 80.

Locally on my machine Pac-Man loads. However, fitting this into a wider deployment, there are more considerations to think about. And that is for another day and another blog post.

Comparing the original and new images

I uploaded the original container image and new container image to a Harbor repository, so that I could use the Trivvy Scanner functions to see the vulnerabilities in both container images.

The below screenshot shows the original container image that I’ve been using. As we can see there are a high number of issues that need to be addressed.

Below is my new container image built by Tanzu Build Services. As you can see there is far less CVEs reported, and the container image size is also a lot smaller as well.

The key thing here to note is, I have changed nothing here with the original dockerfile used to create the containers. I’m simply run this through Build Services, which has discovered the components, and made decisions about how this is packaged together.

Wrap up

In this blog post, I’ve covered a very quick setup in a lab environment to look at how Tanzu Build Services works and run a dockerfile of an existing application through it.

There is a lot more to cover here though:

How to troubleshoot builds,
How to tailor builds to specific needs
Updating images when the sources are changed or have new commits
Bringing this into a full CI/CD pipeline and tool chain

And I assume even more items I’m not currently considering.

Regards

Follow @Saintdle

Dean Lewis

The post First Look – Setup Tanzu Build Services and rebuilding Pac-Man appeared first on vEducate.co.uk.

Quick Tip – Kubernetes – Delete all evicted pods across all namespaces

Dean — Mon, 08 Nov 2021 11:15:57 +0000

I’m currently troubleshooting an issue with my Kubernetes clusters where pods keep getting evicted, and this is happening across namespaces as well.

The issue now that I am faced with, is being able to keep ontop of the issues. When I run:

kubectl get pods -A | grep Evicted

I’m presented with 100’s of returned results.

So to quickly clean this up, I can run the following command:

kubectl get pods -A | grep Evicted | awk '{print $1,$2,$4}' | xargs kubectl delete pod $2 -n $1

Breaking down the command:

Get all pods across all namespaces
Filter by term “Evicted”
Manipulate the output by selecting the data in field 1, 2 and 4
Use xargs to read from the standard output to place the data from the previous pipe into the “kubectl” command.

This command will cycle and remove everything you need. You can use this command line for other status of pods if needed.

Now I need to get back to troubleshooting my cluster issues.

Regards

Follow @Saintdle

Dean Lewis

The post Quick Tip – Kubernetes – Delete all evicted pods across all namespaces appeared first on vEducate.co.uk.

Kasten K10 – Air gap installation using Harbor Image Registry

Dean — Tue, 26 Oct 2021 19:03:14 +0000

In this blog post, I will cover the steps for an air-gap installation for Kasten K10. For situations where your Kubernetes cluster doesn’t have available internet access to pull down the container images directly from their online locations.

Pre-requisites

Image Registry that is accessible by your Kubernetes cluster
- In this example I am using the Harbor Image Registry.
Client that has access to download the container images and then to the Image Registry
- In this example, I am using my local machine which has docker installed.
Helm downloaded
- Run the following to get the helm files locally for the install.

helm repo update && \
    helm fetch kasten/k10 --version=<k10-version>

**Example for Kasten K10 4.5.0**

helm repo update && \ 
    helm fetch kasten/k10 --version=4.5.0

This will download a file, for example "k10-4.5.0.tgz"

Log into your Image Registry

First you need to ensure that your docker client (or similar) has authenticated to your Image Registry which your air-gap Kubernetes cluster can access.

When using Harbor and Docker, I typically use this method with a robot account for programmatic access.

However, when running the Kasten tooling which we’ll discuss next, I kept hitting an error.

54e42005468d: Waiting  File=kasten.io/k10/kio/tools/k10offline/k10offline.go Function=kasten.io/k10/kio/tools/k10offline.PushK10Images Line=179 hostname=3ffc0162e190

Error: {"message":"Failed to push K10 container images to harbor-repo.veducate.co.uk/deanl","function":"main.pullImages","linenumber":171,"cause":{"message":"Failed to push","function":"kasten.io/k10/kio/tools/k10offline.PushK10Images","linenumber":181,"fields":[{"name":"image","value":"harbor-repo.veducate.co.uk/deanl/kanister-tools:k10-0.69.0"}],"cause":{"Stderr":" **dW5hdXRob3JpemVkOiB1bmF1dGhvcml6ZWQgdG8gYWNjZXNzIHJlcG9zaXRvcnk6IGRlYW5sL2thbmlzdGVyLXRvb2xzLCBhY3Rpb246IHB1c2g6IHVuYXV0aG9yaXplZCB0byBhY2Nlc3MgcmVwb3NpdG9yeTogZGVhbmwva2FuaXN0ZXItdG9vbHMsIGFjdGlvbjogcHVzaAo=**"}}}

**Base64 decode the message above in light bold provides you the response:**

unauthorized: unauthorized to access repository: deanl/kanister-tools, action: push: unauthorized to access repository: deanl/kanister-tools, action: push

To resolve this, I had to remove the credsStore line from my ~/.docker/config.json file. Then log into my Harbor registry using the above method.

This does mean your auth account details are stored in a JSON file locally.

Pull down the Kasten K10 images locally and Push to internal Air-Gap Image Registry

Kasten has provided an easy-to-use tool which can run locally on your docker client to make pulling the necessary images simple.

docker run --rm -ti -v /var/run/docker.sock:/var/run/docker.sock \
    -v ${HOME}/.docker:/root/.docker \
    gcr.io/kasten-images/k10offline: **{TAG}** pull images

# Example with Tag

docker run --rm -ti -v /var/run/docker.sock:/var/run/docker.sock \
    -v ${HOME}/.docker:/root/.docker \
    gcr.io/kasten-images/k10offline:4.5.0 pull images

By specifying the appropriate tag, you this will use a container image to pull down all the containers and store them within your docker client. To push to an internal repo, you simply add the argument:

--newrepo {repo url}

For example:

docker run --rm -ti -v /var/run/docker.sock:/var/run/docker.sock \
    -v ${HOME}/.docker:/root/.docker \
    gcr.io/kasten-images/k10offline:4.5.0 pull images --newrepo harbor-repo.veducate.co.uk/deanl

The below image shows the tool download containers and pushing them to the repo.

Here are my images in Harbor.

Installing Kasten K10 with a local Helm Chart and Container Images

Create your Kasten namespace

kubectl create namespace kasten-io

Then run the following Helm command

helm install k10 k10-4.5.0.tgz --namespace kasten-io \
--set global.airgapped.repository= **{registry URL}**

# Example

helm install k10 k10-4.5.0.tgz --namespace kasten-io \
  --set global.airgapped.repository=harbor-repo.veducate.co.uk/deanl

We can now watch the pods start by using the command:

kubectl get pods -n kasten-io -w

We can check these are coming from our local image repository by running the describe command against one of our pods.

Finally, we can also see pulls in our Harbor image registry as well.

Summary

Kasten has made it easy to perform an air-gap internet restricted installation of their software in a Kubernetes cluster, made especially easy thanks to the little k10offline tool you run in your docker client to grab the necessary images for you.

I did hit that little authentication issue where I had to remove the credsStore in docker, due to the way the tool reads the Image Registry auth details. I messaged the Kasten support team about this and they were quick to give me the workaround I documented earlier.

I’m not going to say outside of this I did anything more than follow the Kasten docs on the subject. But I always feel sometimes it’s good to add some more context and colour with screenshots and using real environments to demonstrate these capabilities and configurations.

Regards

Follow @Saintdle

Dean Lewis

The post Kasten K10 – Air gap installation using Harbor Image Registry appeared first on vEducate.co.uk.

Kubernetes – Kubelet Unable to attach or mount volumes – timed out waiting for the condition

Dean — Thu, 30 Sep 2021 11:40:00 +0000

The Issue

When I updated my Kasten application in my Kubernetes cluster, I found that one of the pods was stuck in “init” status.

dean@dean [~] (⎈ |tkg-wld-01-admin@tkg-wld-01:default) # k get pods -n kasten-io -w
NAME READY STATUS RESTARTS AGE
aggregatedapis-svc-78564d4697-wl9wg 1/1 Running 0 3m9s
auth-svc-7977b9684b-zph27 1/1 Running 0 3m11s
catalog-svc-7ff7779b75-kmvsr 0/2 Init:0/2 0 2m43s

Running a describe on that pod pointed to the fact the volume could not be attached.

Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 2m58s default-scheduler Successfully assigned kasten-io/catalog-svc-7ff7779b75-kmvsr to tkg-wld-01-md-0-54598b8d99-rpqjf
Warning FailedMount 55s kubelet Unable to attach or mount volumes: unmounted volumes=[catalog-persistent-storage], unattached volumes=[k10-k10-token-lbqpw catalog-persistent-storage]: timed out waiting for the condition

The Cause

Some where along the line I found some stale volumeattachmentslinked to Kubernetes node that no longer exist in my cluster. This looks to be causing some confusion in the cluster who should be attaching the volume

The image below shows:

Find the Persistent Volume name linked to the associated claim for the failure in the pod events
Map this to the available VolumeAttachments
Reference VolumeAttachments for each node to available nodes in the cluster
- I’ve highlighted the missing node in the red box

The Fix

The fix is to remove the stale VolumeAttachment.

kubectl delete volumeattachment [volumeattachment_name]

After this your pod should eventually pick up and retry, or you could remove the pod and let Kubernetes replace it for you (so long as it’s part of a deployment or other configuration managing your application).

Regards

Follow @Saintdle

Dean Lewis

The post Kubernetes – Kubelet Unable to attach or mount volumes – timed out waiting for the condition appeared first on vEducate.co.uk.