Foreman 101: agentic coding as Kubernetes resources

Foreman is an agentic coder that runs as Kubernetes resources. You describe work as a Workload, it decomposes into tasks, agents running on your nodes pick them up, and a branch comes out the other end with something deterministic standing between that branch and your main.

This is the walkthrough. Four objects to understand, an install, an agent, a verifier, and a real run. Every command and every output below is from a working cluster.

The four objects

Foreman is deliberately small. Almost everything you do is one of these.

• Agent is a worker definition: which model it talks to, which tools it may call, and what budget it gets. An Agent has a role, and the two that matter here are coder and verifier.
• Workload is the unit of work you actually author. It carries an intent, a repository, and which agents to use.
• AgenticTask is what a Workload decomposes into. You rarely write one by hand; you read them to see what is happening.
• FleetNode is a node that has advertised itself as able to run tasks. The scheduler matches a task's required capabilities against these.

The shape of a run is: you apply a Workload, the controller synthesizes AgenticTasks, the scheduler routes each to a FleetNode whose agent can serve it, the agent runs the model in a loop with tools, and the result lands as a branch plus a verdict.

The idea underneath it

Worth stating plainly, because it shapes every design decision: the model is not trusted, and specifically its claim to have succeeded is not trusted.

A coder agent finishes by calling a tool that says "I am done, verdict GO." Foreman treats that as a request, not a result. If the model says GO and produced no diff, the run is recorded as NO-GO. If the verifier's checks do not pass, the work does not land, no matter how confident the summary was.

That is the difference between an agent that writes code and a system you can leave running. Everything else in this post is plumbing around that idea.

Install

Foreman ships as a Helm chart that depends on LLMKube core. Any cluster will do; a throwaway kind cluster is fine, and no GPU is required to follow along.

kind create cluster --name foreman-101

helm repo add llmkube https://defilantech.github.io/LLMKube
helm repo update

helm upgrade --install llmkube llmkube/llmkube \
  --namespace llmkube-system --create-namespace

helm upgrade --install foreman llmkube/foreman \
  --namespace foreman-system --create-namespace \
  --set agent.mode=native \
  --set 'agent.roles={worker,coder,verifier}'

Two flags on that second install are worth understanding rather than copying.

agent.mode=native runs the real agent loop. The chart's default is stub, which registers a FleetNode and does nothing, so that a chart install can be smoke-tested on a cluster with no model and no credentials.

agent.roles is what the scheduler matches against. A node advertising only worker will never be given coder work. Advertise the roles this node should actually serve.

You should end up with three pods and a registered node:

$ kubectl get pods -n llmkube-system
NAME                                          READY   STATUS    RESTARTS   AGE
llmkube-controller-manager-777d9f756b-xb7xr   1/1     Running   0          56s

$ kubectl get pods -n foreman-system
NAME                                READY   STATUS    RESTARTS   AGE
foreman-agent-568b74c49f-mnkc8      1/1     Running   0          27s
foreman-operator-7c687499f9-k96mb   1/1     Running   0          27s

$ kubectl get fleetnodes
NAME                             PHASE   VERSION   HEARTBEAT   AGE
foreman-agent-568b74c49f-mnkc8   Ready   0.9.12    20s         20s

Credentials

Foreman needs two kinds of credential, and they do different jobs.

A git credential lets the agent read the issue and push the branch. A model credential lets it talk to a hosted API, and is not needed at all if you serve the model yourself.

# git: the API token, plus the credential used for the push
kubectl create secret generic foreman-github \
  --from-literal=GITHUB_TOKEN="$GITHUB_TOKEN" -n foreman-system

kubectl create secret generic foreman-git-credentials \
  --from-literal=token="$GITHUB_TOKEN" -n foreman-system

helm upgrade foreman llmkube/foreman -n foreman-system --reuse-values \
  --set agent.githubToken.secretName=foreman-github \
  --set coder.gitCredentialsSecret=foreman-git-credentials \
  --set agent.gitRemoteURL=https://github.com/<you>/<repo>

# model: read from the AGENT's namespace, not the operator's
kubectl create secret generic anthropic-key \
  --from-literal=apiKey="$ANTHROPIC_API_KEY" -n default

agent.gitRemoteURL is where branches are pushed. Pointing it at your fork keeps agent output off the upstream repository, which is the arrangement you want while you are still learning what it does.

Note the namespaces differ, and it is not arbitrary. The git secrets are consumed by the agent process, so they live alongside it in foreman-system. The model credential is resolved per Agent, from that Agent's namespace, so it belongs wherever you create your Agents. Here that is default.

Define a coder

An Agent needs a model. The cloud-proxy provider speaks to any OpenAI-compatible endpoint, so this path works with a hosted API and no GPU.

apiVersion: foreman.llmkube.dev/v1alpha1
kind: Agent
metadata:
  name: claude-coder
  namespace: default
spec:
  role: coder
  provider: cloud-proxy
  providerConfig:
    # /chat/completions is appended, so give the /v1 prefix.
    baseURL: https://api.anthropic.com/v1
    model: claude-sonnet-5
    apiKeySecretRef:
      name: anthropic-key
      key: apiKey
  tools:
    - read_file
    - write_file
    - str_replace
    - grep
    - bash
    - submit_result
  maxTurns: 12
  contextWindowTokens: 30000
  maxOutputTokens: 4096
  requiredCapability:
    roles: [coder]
  systemPrompt: |
    You are a senior staff software engineer.

    You fix one issue per run. The repository is already cloned in your
    workspace, checked out on a branch created for you.

    You communicate only by calling tools. Every action is a tool call.

    Work in this order:
      1. read_file / grep to find the change site. Do not survey the whole
         repository; find the file and start.
      2. str_replace or write_file to make the change. Make your first edit
         early, even if incomplete, then refine it.
      3. bash to verify. Run the narrowest check that proves your change.
      4. submit_result to finish.

    submit_result is terminal. Call it with verdict=GO and a commit_message
    when the change is complete and verified, or verdict=INCOMPLETE if you
    cannot finish, saying plainly what is missing.

    Make the minimum change that fixes the issue. Do not refactor adjacent
    code, and do not fix unrelated problems you notice on the way.

Three fields there deserve more than a glance.

The budgets are a cost control. Defaults are 120 turns and a 90k context, and every turn re-sends the whole context to a metered API. Twelve turns and 30k is the difference between a run that costs pennies and one that does not.

The tool list is the agent's authority. It cannot do anything not on that list. Removing bash gives you an agent that can edit but not execute; removing the write tools gives you one that can only read and report.

The system prompt is the behaviour. The ordering instructions above are not decoration. Left to explore, a coding model will survey a repository for a long time before touching it, so "find the file and start" and "make your first edit early" are load bearing. So is the scope fence at the end, which is what keeps a one-line fix from arriving as a refactor.

Define a verifier

This is the object that makes Foreman different, and it is the smallest one.

apiVersion: foreman.llmkube.dev/v1alpha1
kind: Agent
metadata:
  name: gate
  namespace: default
spec:
  role: verifier
  provider: local
  tools:
    - run_gate_job
  requiredCapability:
    accelerator: any
    roles: [verifier]

There is no model here. No provider config, no API key, no system prompt. An Agent with an empty inferenceServiceRef is a deterministic agent: Foreman skips the model loop entirely and just runs the tool.

run_gate_job submits a Kubernetes Job that clones the branch and runs your checks. In a Go repository that is your build, vet, linter and tests, plus a bite check that reverts the production change and re-runs the new tests to confirm they fail without it. A test that still passes with the fix reverted is not a test.

The component that decides whether the AI's work is acceptable is not itself AI. That is the whole point.

Give it work

A Workload is what you author day to day. The only strictly required field is intent.

apiVersion: foreman.llmkube.dev/v1alpha1
kind: Workload
metadata:
  name: abtest-s1
  namespace: default
spec:
  repo: defilantech/LLMKube
  issues: [1311]
  coderAgentRef:
    name: claude-coder
  verifierAgentRef:
    name: gate
  openPullRequest: false
  intent: |
    Fix #1311: HPA-driven scale-up is not gated by GPUQuota, and
    status.usedGPUCount under-reports because it counts the admitted replica
    figure, not the autoscaling maxReplicas the service can reach. Charge
    autoscaling.maxReplicas (not spec.replicas) toward GPUQuota usage so the
    cap reflects the worst case, and reject admission when maxReplicas would
    exceed the quota. Add a regression test.

The intent is a brief to a competent engineer who has not seen your codebase. Being exact about the change, charge maxReplicas and not spec.replicas, and naming the deliverable (a regression test) does more for the outcome than any model setting. Where there is one obvious file to start in, name it; here the fix spans two accounting paths, so the brief names both.

Apply it and watch:

$ kubectl get workload,agentictask
NAME                                     PHASE        REPO                  AGE
workload.foreman.llmkube.dev/abtest-s1   Dispatched   defilantech/LLMKube   8s

NAME                                                  PHASE     KIND        VERDICT
agentictask.foreman.llmkube.dev/abtest-s1-code-1311   Running   issue-fix

The Workload synthesized a task named abtest-s1-code-1311: the Workload name, then the step, then the issue. With a verifier configured you also get a abtest-s1-verify-1311 that waits on the coder finishing.

Read the result

A finished task carries its verdict and the evidence for it.

verdict: GO
summary: "Charge autoscaling.maxReplicas (not spec.replicas) toward
          GPUQuota usage in both the admission webhook and the GPUQuota
          status reconciler, so HPA scale-up is gated at admission and
          status.usedGPUCount reflects the worst-case headroom. Added
          regression tests."
turnCount: 64
elapsedSec: 1608.38
branch: foreman/abtest-s1/issue-1311
commitSHA: 87a0d70c804a6fc4a51fba6550083e976d0d2c06

Sixty-four turns and about twenty-seven minutes: a real change across two accounting paths with a regression test, not a one-line edit, and the branch is on the fork. The full transcript is persisted as a ConfigMap, referenced from the task status, so you can read every tool call the model made rather than guessing at it.

The verdicts you will actually see, and what each means:

• GO the coder finished and produced a diff.
• NO-GO it did not, or it claimed success without one.
• INCOMPLETE it ran out of turns or could not verify its own work.
• GATE-PASS the verifier's checks passed on the branch.
• GATE-FAIL they did not, and the branch does not land.

The gate, concretely

A verifier runs whatever you tell it to. You can point one at an existing branch to see the mechanism on its own.

apiVersion: foreman.llmkube.dev/v1alpha1
kind: AgenticTask
metadata:
  name: gate-check
  namespace: default
spec:
  kind: verify
  agentRef:
    name: gate
  gateProfile:
    # "generic" runs the commands below instead of the Go make targets.
    language: generic
    image: golang:1.26
    commands:
      # Swap this line to see the other verdict.
      format: "grep -q Foreman docs/site/foreman/README.md"
      build: "true"
      lint: "true"
      test: "true"
  payload:
    repo: defilantech/LLMKube
    branch: main

Run it once as written, then once with a string that is not in the file, and the Job logs show exactly what happened:

=== clone defilantech/LLMKube @ main ===
=== grep -q Foreman docs/site/foreman/README.md ===
GATE PASS

=== clone defilantech/LLMKube @ main ===
=== grep -q NO_SUCH_STRING_IN_THIS_REPO docs/site/foreman/README.md ===
GATE FAIL

Same agent, same branch, one command different, opposite verdicts. The task status carries deterministic: true, because no model was consulted about whether the work was good. A grep decided.

Running the model yourself

Everything above used a hosted API because it is the version anyone can try today. The reason LLMKube exists is the other path: the model runs on hardware you own and nothing leaves the building.

The Agent barely changes. No API key and no external URL, just a reference to an InferenceService that LLMKube is already serving.

spec:
  role: coder
  provider: local
  inferenceServiceRef:
    name: laguna-s-2.1-strix
  model: laguna-s-2.1-q4km-strix
  # tools, budgets and systemPrompt are identical to the cloud-proxy Agent

Foreman resolves that to the service's in-cluster address at task time. The agent that runs the loop must be able to reach it, which means an in-cluster agent for an in-cluster service. Everything downstream, the tools, the verdicts, the gate, is identical.

The model behind that reference in our own cluster is a 118B-parameter, roughly 8B-active open-weight coder on an AMD Strix Halo box with 128GB of unified memory. Four of its pull requests merged into LLMKube while this post was being written, each through the full gate.

Where the line is

Being straight about scope, because it determines whether this is useful to you.

Mechanical, well-scoped work lands: a documented bug with a known change site, a refactor with a clear contract, a missing test, a docs correction. Give it the file and a scope fence and it will do the job and stop.

Architectural work does not, yet. A change that requires holding two packages in your head and deciding which of two components should own a behaviour is still yours. When a run fails on that kind of task it usually fails honestly, as INCOMPLETE with a summary, rather than landing something plausible and wrong. That is the failure mode you want, but it is still a failure.

Start here

A kind cluster, the two Helm charts, one coder Agent against whatever API key you already have, one verifier, and a genuinely small issue. Twenty minutes end to end.

Then add the verifier before you add anything else. An agent without one is autocomplete with a Kubernetes API; an agent with one is a system you can leave running.

The docs are at llmkube.com/docs/foreman and the code is on GitHub. Both are Apache 2.0.