An Overview of Agent Substrate

A Kubernetes Pod is a useful unit of deployment. An agent session has a different lifetime: it may run briefly, wait for minutes, then need its process and filesystem state back for the next turn. Keeping a dedicated environment resident through every wait can consume capacity that other sessions could use.

Agent Substrate separates those lifetimes. Kubernetes runs a pool of worker Pods; Substrate places sandboxed agent environments onto them, persists their state, and restores them when work arrives. This article follows one request through that system, then examines the components and a Counter demo you can try locally.

TL;DR

• Kubernetes manages Workers; Substrate manages Actors. Actor state and assignments live behind a separate API backed by PostgreSQL.
• Workers can serve many Actors over time. Capacity-based concurrency is combined with suspend/resume to reuse compute.
• atelet and ateom save and restore sandbox state. They use gVisor or microVMs; the snapshot fidelity determines what is preserved.
• Credentials can stay outside the Actor. An egress gateway and credential provider can inject API credentials into outgoing requests.
• Check the implemented behavior, not just the design. Authorization enforcement is experimental and disabled by default for ordinary Control RPCs. Idle detection and performance claims also require context.

Implementation details target Agent Substrate v0.5.0. See Components and versions for the full scope.

This walkthrough assumes familiarity with Pods, Deployments and basic Kubernetes networking. Start with the request lifecycle, jump to the kind demo, or use the evaluation questions to assess the fit for your platform.

The design decision: keep the workers, move the sessions

An agent may execute code briefly, wait for inference or an external tool, then wait indefinitely for another user turn. Running untrusted code also creates a need for isolated environments, often with one tenant per sandbox.

The combination matters: a large fleet can contain many more waiting environments than actively executing ones. Reserving CPU and memory for every waiting agent wastes capacity. Snapshot-based multiplexing is intended to let those environments retain state without remaining assigned to running compute.

Source: docs/architecture.md#L17-L23.

What Kubernetes still does—and what Substrate moves elsewhere

The architecture document identifies three pressures behind its design:

Pressure	Why it matters for agents
API resource count and write traffic	Millions of frequently changing agent records are a different workload from ordinary Kubernetes reconciliation
Activation latency	Pod scheduling, reconciliation, network hops and image preparation add overhead to very short executions
State movement	Rapidly saving and restoring many environments is a different problem from attaching conventional persistent volumes

These are the project's design judgments for its intended scale, rather than universal Kubernetes limits or a claim that every Pod takes seconds to start.

Kubernetes still handles infrastructure provisioning and worker Pod lifecycles. Substrate introduces a separate control plane for Actor state, assignment and resumption, so an activation does not need to wait for a fresh Pod lifecycle.

Sources: docs/architecture.md#L153, docs/architecture.md#L160-L172.

The important boundary is the activation path: a new turn can restore an Actor onto an existing Worker without provisioning another Pod. Database write throughput, reconciliation and environment preparation all contribute to that choice.

Follow a request into a suspended Actor

The execution layer combines routing, state persistence and sandbox isolation:

Source: Agent Substrate on GKE, Google Cloud Blog (September 16, 2026).

Two lifecycles, two APIs

Resource	Meaning	Managed through
ActorTemplate	Immutable workload definition, including images and snapshot settings	Substrate API and database
WorkerPool	Defines a pool that atecontroller reconciles into a Deployment	Kubernetes CRD
Atespace	Logical grouping and identity scope for Actors; not a Kubernetes namespace	Substrate API and database
Actor	An instance of an ActorTemplate and the unit of suspend/resume	Substrate API and database
Worker	A record corresponding to a worker Pod; can host multiple Actors concurrently	Substrate API and database

Sources: docs/glossary.md#L10-L23, docs/glossary.md#L38-L52.

An Actor is the stateful environment that belongs to a session. A Worker is the reusable execution capacity. Suspending an Actor releases its assignment; it does not require destroying and recreating the worker Pod. Worker capacity can also permit several Actors to run concurrently.

Restore first, forward second

For an HTTP request targeting a suspended Actor, the activation path has four steps:

1. The router identifies the Actor from ate-target-actor.
2. The control plane assigns a Worker and coordinates restoration of the Actor's snapshot.
3. Once the assignment is ready, the router forwards the request through the Worker's tunnel to the Actor.
4. The Actor handles the request and returns its response.

For an Actor that is already running, ResumeActor returns the existing assignment. The call is idempotent.

Source: workflow_resume.go.

Returning a response does not automatically suspend the Actor. The application or higher-level system must choose a safe suspension boundary, taking account of outstanding work and network calls.

Suspend and resume

Source: Agent Substrate on GKE, Google Cloud Blog (September 16, 2026).

Here, “idle” means state has been persisted and the environment removed from running compute, rather than a process simply sleeping. The illustration is conceptual: it does not imply a fixed one-to-one relationship between Workers and Actors.

When a request arrives, atenet-router asks the control plane to resume the target Actor and resolve its Worker assignment. A suspended Actor is restored from its latest snapshot before traffic is forwarded. It may resume on a different Worker.

The router itself does not decide whether the Actor is running; that decision belongs to the control plane. Suspending the Actor saves state and releases its assignment for reuse.

Source: docs/architecture.md#L193-L204.

What runs where

Component	Responsibility	Relationship to Kubernetes
ate-api-server (ateapi)	Actor lifecycle, scheduling and snapshot coordination	Separate API, scheduler and workflow layer backed by PostgreSQL
atecontroller	Reconciles WorkerPool and related CRDs	Standard controller-runtime controller
atelet / ateom	Coordinates and executes sandbox checkpoint/restore	Additional execution layer inside worker Pod boundaries
atenet router and egress	Resumption-aware ingress and controlled outbound traffic	Routing plus Actor lifecycle coordination
podcertcontroller	Issues short-lived certificates for component mTLS	Signer controller for PodCertificateRequest

The threat-model diagram shows how these components connect:

Source: docs/threat-model.md#L30-L40.

Repository map: controllers, sidecars and tools

The repository also includes supporting programs:

Directory	Role
cmd/ateapi	gRPC control plane
cmd/atecontroller	Kubernetes controllers
cmd/atelet	Node supervisor deployed as a DaemonSet
cmd/ateom-gvisor	Drives gVisor inside worker Pods
cmd/ateom-microvm	Runs Actors with Cloud Hypervisor microVMs
cmd/objectstore-plugin	Snapshot transfer sidecar for node and control-plane operations
cmd/atenet	Networking daemon and proxy integration
cmd/dataplane/envoy	Envoy image with Rust Dynamic Modules for egress enforcement
cmd/credential-provider/kubernetes-secrets	Reference credential provider backed by Kubernetes Secrets
cmd/podcertcontroller	Pod certificate signers
cmd/kubectl-ate	Actor and other Substrate resource operations
cmd/ate-setup	Installation and removal
cmd/benchmarking	Synthetic resource-consuming test workloads
tools/setup-gcp	GKE, storage and IAM provisioning

The README's tour is not an exhaustive inventory; the sidecar, provider, dataplane and setup entries are supplemented from their own source and documentation.

Sources: README.md#L260-L272, cmd/objectstore-plugin/main.go#L15-L19, cmd/ate-setup/README.md#L3, cmd/dataplane/envoy/README.md.

Resource ownership and assignment

Source: Bringing you Agent Sandbox on GKE and Agent Substrate, Google Cloud Blog (May 20, 2026).

The green sandbox-isolation block is the foundation for the checkpoint and restore operations. Agent Substrate incorporates Agent Sandbox's secure runtime and snapshot capabilities into a separate execution layer.

classDiagram
    namespace kube-apiserver {
        class WorkerPool {
            <<CRD>>
        }
        class Deployment
        class WorkerPod {
            ateom
            runsc
        }
    }

    namespace ate-api-server {
        class ActorTemplate {
            <<record>>
        }
        class Actor {
            <<record>>
            status
            snapshotRefs
        }
        class Worker {
            <<record>>
            actorAssignments
            podIP
        }
    }

    ActorTemplate ..> WorkerPool : worker_selector (labels)
    WorkerPool ..> Deployment : reconciled by atecontroller
    Deployment "1" *-- "*" WorkerPod : manages
    Actor "*" --> "1" ActorTemplate : actor_template
    Actor "0..*" --> "0..1" Worker : runs on
    Worker "1" --> "1" WorkerPod : maps to

Adapted from docs/architecture.md#L268-L303; the multiplicity reflects the implementation's support for multiple Actors per Worker.

The split between API servers is central. Infrastructure resources stay in Kubernetes, while frequent Actor/Worker assignment changes go through ate-api-server and its state store.

Two resource layers

The architecture gives three reasons for this split:

• Keep high-frequency Actor updates away from the Kubernetes control plane.
• Use low-latency lookups and atomic assignments for resumption. The document's 100ms figure is a design target, not a result measured here.
• Retain Kubernetes RBAC, auditing and policies for infrastructure resources such as WorkerPools and SandboxConfigs.

One detail in that document is stale: it says authorization has not been implemented. The same v0.5.0 tag contains OpenFGA-based authorization. --experimental-enable-authz defaults to false and controls enforcement for ordinary Control RPCs. AccessPolicy RPCs are checked regardless of that flag.

Sources: docs/architecture.md#L247-L261, cmd/ateapi/main.go#L84-L88, cmd/ateapi/internal/authz/interceptor.go#L28-L60.

ate-api-server: the Actor control plane

ateapi owns Actor lifecycles, schedules them onto Workers and coordinates snapshots. Its gRPC server connects to PostgreSQL, creates Kubernetes clients and configures its mTLS credentials.

Sources: docs/glossary.md#L56-L58, cmd/ateapi/main.go#L509-L521, cmd/ateapi/main.go#L560-L586.

templateReconciler := controlapi.NewActorTemplateReconciler(persistence, controlSrv, *templateResyncInterval)
templateReconciler.Start(shutdownCtx)

workerAssignmentReconciler := controlapi.NewWorkerAssignmentReconciler(persistence, workerCache)
workerAssignmentReconciler.Start(shutdownCtx)

Source: cmd/ateapi/main.go#L326-L330.

Despite their names, these reconcilers are internal Substrate loops rather than controller-runtime reconcilers watching custom resources. ActorTemplateReconciler prepares golden snapshots; WorkerAssignmentReconciler handles Actors lost when ateom restarts.

This component does not map one-to-one to kube-apiserver: it combines Actor-specific API handling, scheduling and workflows.

atecontroller: conventional Kubernetes reconciliation

atecontroller uses ctrl.NewManager and registers four reconcilers, including WorkerPoolReconciler. Its core structure is familiar:

func (r *WorkerPoolReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
    log := log.FromContext(ctx)

    wp := &atev1alpha1.WorkerPool{}
    if err := r.Get(ctx, req.NamespacedName, wp); err != nil {
        if k8errors.IsNotFound(err) {
            return ctrl.Result{}, nil
        }
        return ctrl.Result{}, fmt.Errorf("failed to get worker pool %q: %w", req.NamespacedName, err)
    }

    if !wp.GetDeletionTimestamp().IsZero() {
        log.Info("WorkerPool is being deleted")
        return ctrl.Result{}, nil
    }

    if err := r.reconcileWorkerPool(ctx, wp); err != nil {
        log.Error(err, "Failed to reconcile worker pool")
        return ctrl.Result{}, err
    }

    return ctrl.Result{}, nil
}

Sources: cmd/atecontroller/main.go#L233-L276, cmd/atecontroller/internal/controllers/workerpool_controller.go#L105-L129.

Reconciliation does more than produce a Deployment. The WorkerPool reconciler protects referenced SandboxConfigs with finalizers; a separate SandboxConfigProtectionReconciler releases protection. Updating the recorded SandboxConfig uses two status writes so that status never references a configuration whose protection has already been released.

Owns(&Deployment{}) and Watches(&SandboxConfig{}, ...) connect these resources through normal controller-runtime mechanisms.

Sources: cmd/atecontroller/internal/controllers/workerpool_controller.go#L134-L136, cmd/atecontroller/internal/controllers/workerpool_controller.go#L416-L425.

atelet and ateom: checkpoint and restore

atelet runs on each node as a DaemonSet. ateom runs inside worker Pods and exposes operations such as RunWorkload, CheckpointWorkload and RestoreWorkload.

atelet coordinates worker supervision, image caching and CSI volumes. It also reuses kubelet's image credential-provider mechanism through --image-credential-provider-config.

Sources: docs/architecture.md#L324-L326, cmd/atelet/main.go#L103-L104.

Snapshot upload and download are delegated to objectstore-plugin sidecars over a Unix-socket gRPC API:

Plugin role	Operations
NodeProvider, alongside atelet	Upload and fetch snapshot files
ControlProvider, alongside ate-api-server	Copy and delete whole snapshots

The main processes do not directly transfer snapshots to object storage. This does not mean they hold no credentials at all: mTLS and database authentication remain separate concerns.

Source: docs/architecture.md#L330.

For gVisor, cmd/ateom-gvisor/runsc.go invokes runsc checkpoint and runsc restore. The microVM implementation uses Kata integration and Cloud Hypervisor, with userfaultfd demand paging when restoring memory.

Sources: docs/architecture.md#L328-L338.

Kubelet and the container runtime still run the worker Pod itself. Substrate manages sandboxed Actors inside it. Kubernetes already has a Kubelet Checkpoint API; Substrate's distinction is coordinating persistence, assignment, restore and reconnection as an Actor lifecycle.

atenet: routing that can resume its destination

Ingress requests identify an Actor with the ate-target-actor header, formatted as <atespace>/<actor>. In the Envoy configuration described here, an ext_proc processor calls the control plane's ResumeActor RPC before resolving the destination.

Once the assignment is available, the router opens an mTLS CONNECT tunnel to the Worker's atunnel listener on port 8443. atunnel, hosted by ateom, forwards traffic over the Actor's private veth interface. Worker Pod port 80 is not a direct Actor ingress endpoint.

Sources: docs/architecture.md#L344-L354.

The Kubernetes analogy is Service/Ingress/Gateway routing with an extra orchestration step: restore the destination if necessary before forwarding to it.

podcertcontroller: Pod certificate signers

podcertcontroller provides three signers:

Signer	Purpose
servicedns.podcert.ate.dev/identity	Service DNS certificates backed by a local CA
podidentity.podcert.ate.dev/identity	Pod identity certificates equivalent to Kubernetes service account identity
postgres.podcert.ate.dev/identity	PostgreSQL login certificates for ate-api-server

Source: cmd/podcertcontroller/main.go#L15-L22.

It watches certificates.k8s.io/v1 PodCertificateRequest objects. The API became GA in Kubernetes v1.37:

PodCertificateRequest: {
    {Version: version.MustParse("1.34"), Default: false, PreRelease: featuregate.Alpha},
    {Version: version.MustParse("1.35"), Default: false, PreRelease: featuregate.Beta},
    {Version: version.MustParse("1.37"), Default: true, PreRelease: featuregate.GA}, // LockToDefault in 1.38
},

Source: kube_features.go.

The API lets kubelet request certificates and project them into Pods. A signer must still process those requests. The well-known kubernetes.io/kube-apiserver-client-pod signer described in the API comments is unimplemented in this version.

Substrate fills its own needs with the three signers above. Its README calls the controller a “polyfill” for the missing signers.

Sources: certificate API types, README.md#L268-L269.

Keep session state and API credentials separate

Isolation, identity and credentials are separate controls

Source: Agent Substrate on GKE, Google Cloud Blog (September 16, 2026).

Kernel isolation does not by itself protect external API credentials. That also requires the gateway and credential-provider configuration described below.

The security document distinguishes Met, In Progress, Future and Out of Scope. Selected principles in v0.5.0 are:

Principle	Meaning	Documented status
Always Sandboxed	Run Actors in gVisor or Cloud Hypervisor microVMs rather than relying on ordinary Linux containers alone	Met
Zero Ambient Credentials	Do not implicitly give Actors infrastructure permissions or external API keys	Met; distinct from demos that explicitly pass keys in environment variables
No Access by Default	Restrict Actor access to the control plane, separate from required infrastructure traffic	Loopback/private netns: Met; DNS filtering: In Progress
Fail-Closed Policy-Enforced Egress Gateway	Control outbound traffic through an egress proxy	Gateway: Met; fail-closed behavior: In Progress, with UDP passthrough noted

Sources: docs/security.md#L30-L33, docs/security.md#L41, docs/security.md#L59, docs/security.md#L75-L77.

Warning: The same document lists Atespace authorization as In Progress, and node blast-radius containment and storage encryption as Future. The security architecture is not evidence that every control is fully implemented. Ordinary Control RPC authorization still requires the experimental flag.

Inject credentials outside the Actor

With egress credential injection, an Actor sends an HTTPS request containing a placeholder header. The gateway obtains the real credential and substitutes it outside the sandbox:

1. The Actor sends the request with a placeholder value.
2. The egress gateway terminates TLS and matches an EgressPolicy header-replacement rule.
3. The credential provider authorizes access and reads the secret.
4. The gateway replaces the header and makes a separate TLS connection to the destination.

The Actor must trust the gateway's CA. If the request does not contain the header to replace, credential lookup and replacement do not occur.

This keeps the API key outside Actor memory and snapshots. It does not prevent misuse of an allowed API; destination restrictions and application authorization still matter.

Sources: docs/egress-credential-injection.md#L1-L15, docs/egress-credential-injection.md#L41-L54.

The reference kubernetes-secrets provider starts with an empty deny-all policy. Readable namespaces must be explicitly allowed for each Atespace. Actors do not read Kubernetes Secrets themselves.

Source: README.md#L132-L136.

EgressPolicy defines destinations and header replacement

An EgressPolicy specifies allowed traffic and header replacement rules. The credential provider separately decides whether the Actor can use the referenced secret. The API defines the policy as an unordered set of allows: traffic matching no rule is denied.

Source: pkg/proto/ateapipb/ateapi.proto#L609-L610.

For example, allow HTTPS to api.example.com and replace its Authorization header:

rules:
- https:
    hostnames: ["api.example.com"]
    effects:
      replaceHeaders:
      - header: Authorization
        prefix: "Bearer "
        credentialUri: ate-secret://k8s.io/default/ns1/example-api/token

Source: docs/egress-credential-injection.md#L69-L76.

The policy is associated with an Actor rather than embedded in its ActorTemplate:

# Create an egress policy.
kubectl ate create egress-policy <actor-name> -a <atespace> -f policy.yaml

Source: cmd/kubectl-ate/README.md#L247-L253.

Note: The Claude Code demo discussed later directly places ANTHROPIC_API_KEY in the ActorTemplate environment. That differs from gateway injection. In either case, the Actor needs an egress policy permitting its HTTPS destination, such as api.anthropic.com.

The complete request path

With routing and credential injection in place, an HTTP-triggered agent can resume and call an external LLM as follows. This combines the documented flows; Counter itself does not call an LLM.

sequenceDiagram
    actor User as User / higher-level system
    participant Router as atenet-router
    participant API as ate-api-server
    participant Atelet as atelet
    participant Tunnel as ateom / atunnel
    participant Agent as Actor
    participant Egress as atenet-egress
    participant Cred as credential provider
    participant LLM as External LLM API

    User->>Router: HTTP request (ate-target-actor)
    Router->>API: ResumeActor
    alt Actor is suspended
        API->>Atelet: Restore request
        Atelet->>Tunnel: RestoreWorkload
        Tunnel-->>Atelet: Restore complete
        Atelet-->>API: Complete
    else Actor is running
        Note over API: Return existing assignment
    end
    API-->>Router: Worker assignment
    Router->>Tunnel: Forward through mTLS CONNECT
    Tunnel->>Agent: Forward request
    Agent->>Egress: HTTPS (placeholder header)
    Egress->>Cred: Request credential
    Cred-->>Egress: Return API key after authorization
    Egress->>LLM: Replace header and send over a separate TLS connection
    LLM-->>Egress: Response
    Egress-->>Agent: Response
    Agent-->>Tunnel: Result
    Tunnel-->>Router: Response
    Router-->>User: Response
    Note over User,API: A higher-level system can request SuspendActor to save state and release the assignment

Sources: Resume sequence, credential injection. Cold boot without a snapshot and other branches are omitted.

The Actor-to-gateway and gateway-to-provider connections use separate TLS sessions. The gateway must terminate the first to replace headers in the second.

Try the Counter demo with kind

The Counter workload makes state preservation visible: each request increments counters in process memory, a file and the root filesystem. The example below uses gVisor and includes recorded output from a local kind run.

Set up the cluster and demo

Building v0.5.0 requires Go 1.27.0 and a working Docker-based development environment. The microVM option additionally requires /dev/kvm and suitable virtualization support.

Sources: go.mod#L1-L5, hack/create-kind-cluster.sh#L74-L87.

Use a dedicated checkout and cluster name. The creation script deletes and recreates an existing cluster with the same name, so choose one that does not overlap with an environment you want to keep.

git clone --branch v0.5.0 --depth 1 https://github.com/agent-substrate/substrate.git substrate-v0.5.0
cd substrate-v0.5.0
export KIND_CLUSTER_NAME=ate-overview

# create cluster and local registry (IPv4; IP_FAMILY=dual|ipv6 overrides)
hack/create-kind-cluster.sh

# install ate, PostgreSQL, rustfs
hack/install-ate-kind.sh --deploy-ate-system --credential-provider='{"name":"k8s.io"}'

# install counter demo
hack/install-ate-kind.sh --deploy-demo-counter

# install kubectl-ate
go install ./cmd/kubectl-ate

# create a counter actor in the demo's atespace (--template names the
# actor template, resolved in the actor's atespace)
kubectl ate create actor my-counter-1 -a ate-demo-counter --template counter

# port-forward the network router to bind to local port `8000`
kubectl port-forward -n ate-system svc/atenet-router 8000:80

Source: README.md#L111-L129.

The creation script sets up a local registry and a kind cluster with kindest/node:v1.37.0. It explicitly enables certificate-related feature gates, including ClusterTrustBundleProjection; some comments still describe v1.36 requirements.

Source: hack/create-kind-cluster.sh#L104-L113.

The system installation uses ko to build and push its Go components and deploys PostgreSQL, RustFS, CRDs and SandboxConfig resources. The Counter installation adds its WorkerPool and ActorTemplate and waits for a golden snapshot.

WorkerPool and ActorTemplate

The demo's WorkerPool includes the following settings. The installer resolves template placeholders and ko:// references before applying the configuration.

apiVersion: ate.dev/v1alpha1
kind: WorkerPool
metadata:
  name: counter
  namespace: ate-demo-counter
  labels:
    workload: counter
spec:
  replicas: 3
  workerImage: ko://github.com/agent-substrate/substrate/cmd/ateom-gvisor
  sandboxClasses:
  - name: gvisor
  template:
    resources:
      limits:
        cpu: "1"
        memory: 1Gi
      requests:
        cpu: 250m
        memory: 1Gi

Source: demos/counter/counter.yaml.tmpl#L26-L63.

An ActorTemplate is not a Kubernetes object. This YAML maps to Substrate's protobuf fields and is passed through kubectl ate create actor-template:

metadata:
  atespace: ate-demo-counter
  name: counter
workerSelector:
  matchLabels:
    workload: counter
containers:
- name: counter
  image: ko://github.com/agent-substrate/substrate/demos/counter
  wakeupProbe:
    httpGet:
      path: /readyz
      port: 80
  volumeMounts:
  - name: data
    mountPath: /home/counter
resources:
  limits:
  - name: cpu
    quantity: "1"
  - name: memory
    quantity: 512Mi
snapshotConfig:
  preferredFidelity: SNAPSHOT_FIDELITY_MEMORY
  storageLocation: gs://${BUCKET_NAME}/ate-demo-counter/
sandboxConfig:
  sandboxClass: gvisor
  configName: gvisor-default
volumes:
- name: data
  durableDir: {}

Source: demos/counter/counter-template.yaml.tmpl#L22-L69.

SNAPSHOT_FIDELITY_MEMORY requests preservation of process memory. gvisor-default names a cluster-scoped SandboxConfig whose assets specify where to download the runtime binaries:

apiVersion: ate.dev/v1alpha1
kind: SandboxConfig
metadata:
  name: gvisor-default
  annotations:
    sandboxconfig.ate.dev/is-class-default: "true"
spec:
  sandboxClass: gvisor
  defaultVersion: v3
  versions:
  - name: v3
    state: Enabled
    pauseImage: "registry.k8s.io/pause:3.10.2@sha256:f548e0e8e3dc1896ca956272154dde3314e8cc4fde0a57577ee9fa1c63f5baf4"
    assets:
      arm64:
        gvisor:
          url: "gs://gvisor/releases/nightly/2026-10-10/aarch64/gvisor.tar.zstd"
          sha256: "aaf414f06eea7960238e43165c691f0a36cefc0593fb141e011e96211c7b6f57"

Source: manifests/ate-install/sandboxconfig-gvisor.yaml#L22-L48. Only the arm64 assets are shown; amd64 assets are omitted.

Observe Actor state and preserved counters

A newly created Counter Actor starts suspended: its snapshot exists, but it has no Worker assignment yet.

$ kubectl ate create actor my-counter-1 -a ate-demo-counter --template counter
ATESPACE           NAME           TEMPLATE                   STATE                   WORKER POD   WORKER IPS   VERSION   AGE
ate-demo-counter   my-counter-1   ate-demo-counter/counter   ACTOR_STATE_SUSPENDED   <none>                    1         0s

Send a request through the router. It resumes the Actor before forwarding, and the recorded response is HTTP 200:

$ curl -X POST -H "ate-target-actor: ate-demo-counter/my-counter-1" -i http://localhost:8000/
HTTP/1.1 200 OK
Content-Length: 111
Content-Type: text/plain; charset=utf-8

hello from: 169.254.17.2 | preserved memory count: 1 | preserved file counter: 1 | preserved rootfs counter: 1

A second POST increments the counter to 2. The Actor is now running and assigned to a worker Pod:

$ curl -X POST -H "ate-target-actor: ate-demo-counter/my-counter-1" http://localhost:8000/

$ kubectl ate get actor my-counter-1 -a ate-demo-counter
ATESPACE           NAME           TEMPLATE                   STATE                 WORKER POD                                  WORKER IPS    VERSION   AGE
ate-demo-counter   my-counter-1   ate-demo-counter/counter   ACTOR_STATE_RUNNING   ate-demo-counter/counter-796d677995-v55tv   10.244.0.23   3         20s

Explicitly suspend it, then send another request:

$ kubectl ate suspend actor my-counter-1 -a ate-demo-counter
$ curl -X POST -H "ate-target-actor: ate-demo-counter/my-counter-1" http://localhost:8000/
hello from: 169.254.17.2 | preserved memory count: 3 | preserved file counter: 3 | preserved rootfs counter: 3

The memory, file and root-filesystem counters continue at 3. This illustrates the documented snapshot-and-resume lifecycle, including preserved process memory.

Stop the port-forward when finished, then remove the dedicated cluster. The kind-registry container is separate and remains after cluster deletion.

hack/kind.sh delete cluster --name ate-overview

Plan the deployment around more than Pods

Worker Pods and CRDs are only part of the deployment. The following table distinguishes infrastructure that must be supplied from components the installer can deploy:

Dependency	Purpose	Installation considerations
PostgreSQL	Actor/Worker state store	The kind setup includes a development database; production operators supply database identities, schema and connection settings
GCS or S3-compatible object storage	Durable snapshot storage	Configure storage.bucketName / BUCKET_NAME; kind uses ate-snapshots and an in-cluster RustFS service
Egress credential provider	Resolve credentials outside Actors	Provider selection is required, including explicit {"enabled":false} to disable injection; {"name":"k8s.io"} deploys the reference provider
Sandbox runtime binaries	Run Actors with gVisor or microVMs	Selected through a cluster-scoped SandboxConfig; default gVisor assets come from a public GCS bucket
Proxy dataplane image	Router and egress proxies	--atenet-dataplane selects Envoy, requiring its custom build, or agentgateway, using a published image
Object-storage permissions	Let plugin sidecars transfer snapshots	For GCS, configure the required Workload Identity/IAM bindings; the kind substitute uses local credentials
CSI driver, when needed	Attach CSI-backed persistent storage to Actors	Not needed for the demo's durableDir; Substrate imposes additional controller/node deployment requirements

External databases are not provisioned by ateapi or ate-setup. Snapshot-bucket selection and credential-provider selection also need explicit configuration, even when other defaults are available.

Sources: docs/postgres.md#L8, docs/operator-install.md#L314, docs/egress-credential-injection.md#L128-L131.

Missing GCS IAM bindings can produce a 403 on the first suspend/resume, and also prevent control-plane snapshot operations and image pulls.

Source: tools/setup-gcp/README.md#L154-L159.

Note: Substrate directly orchestrates CSI operations from ateapi and atelet. A CSI controller must be reachable as a cluster-network gRPC service, unlike the usual controller-side Unix socket arrangement. See docs/csi-deployment.md#L5.

For the Kubernetes v1.37 setup covered here, Pod certificate and ClusterTrustBundle features are GA and enabled by default. Substrate also includes its certificate controller. Provider-specific installation requirements should still be checked separately.

Source: ClusterTrustBundle feature definition.

GKE availability

Google announced Agent Substrate availability on GKE on September 16, 2026.

The GKE offering installs open-source Substrate onto GKE Standard; Autopilot is not supported. Evaluation and non-production use are available, while production support is allowlisted under a private GA program. Substrate has no additional product charge, but the GKE and other resources created still incur their normal charges.

Source: Install Agent Substrate on GKE.

The announcement also describes ComputeClasses and optional Filestore agent volumes. Shared workspace storage serves a different purpose from snapshots of execution state.

Google reports sub-500ms resumption, more than 500 suspend/resume activations per second and 10x density. These are Google's published figures, separate from upstream design targets and the Counter example.

Source: Agent Substrate on GKE, Google Cloud Blog (September 16, 2026).

What to evaluate in your own workload

Substrate is worth examining when a fleet has many isolated, stateful environments but only a fraction need to execute at once. The useful question is how much resident capacity suspend/resume can release, and whether restoration meets the application's latency needs.

Start with three checks:

• Suspension boundaries: identify when an environment can safely stop, including outstanding network calls and application work.
• State requirements: decide what must survive, then select snapshot fidelity and any persistent workspace storage accordingly.
• Operational ownership: account for the database, snapshot backend, identity and egress services as well as the Workers.

The Counter demo exposes the core mechanism. An agent platform must then connect it to real session events and measure its own activation latency and capacity use.

Further implementation details

Claude Code demo: workload execution versus UI state

What the Claude Code demo actually implements

claude-code-multiplex defines three Claude Code Actors—luna, mars and orion—with a two-Pod WorkerPool. It uses the control API rather than sending workload HTTP through atenet-router, so it does not use the routing header.

Source: demos/claude-code-multiplex/README.md#L14-L21.

An ActorTemplate supplies the task and interval as environment variables:

  env:
  - name: ACTOR_NAME
    value: "luna"
  - name: TASK
    value: "Tell me one short, surprising fact about the Moon. One sentence."
  - name: INTERVAL_SECONDS
    value: "45"
  - name: ANTHROPIC_API_KEY
    value: "${ANTHROPIC_API_KEY}"

Source: demos/claude-code-multiplex/agent-luna-template.yaml.tmpl#L34-L42.

The workload repeatedly runs claude --print and then sleeps:

while true; do
  ...
  claude --print "${TASK}" 2>&1 || echo "[demo-actor:${ACTOR_NAME}] claude exited non-zero"
  ...
  sleep "${INTERVAL_SECONDS}"
done

Source: demos/claude-code-multiplex/workload/run.sh#L17-L49.

That idle interval is an opportunity for multiplexing. However, the v0.5.0 UI does not implement everything its README's task-flow description suggests:

UI behavior	Implementation
“Give a task”	Records the selected Actor and task in UI-server assignment state
queued / running / completed	Derived from timestamps and randomized waiting/execution durations
Actor state and logs	Retrieved from the Substrate and Kubernetes APIs

giveTask does not send a task to the Actor or call ResumeActor / SuspendActor. A comment in computeState identifies the task state as a time-based UI display rather than Substrate state.

Source: demos/claude-code-multiplex/ui/server.go#L252-L315.

A changing UI status therefore does not establish that a real LLM task completed or automatic multiplexing occurred. The useful pieces are the ActorTemplates, the periodic Claude Code workload and the state-viewing UI. Verify suspend/resume through real API operations and Actor state, as in the Counter demo.

RustFS: the local snapshot backend

RustFS

RustFS is a separate, Rust-based S3-compatible object store. The kind installation uses it as its local snapshot backend. The v0.5.0 manifest selects rustfs/rustfs:1.0.0-beta.3.

This development setup does not replace production storage availability, persistence or operational design. objectstore-plugin uses RustFS through its S3 backend; GCS uses the same plugin interface with a different backend.

Sources: manifests/ate-install/kind/rustfs.yaml#L55-L75, rustfs/rustfs.

ReBAC: deriving permissions from relationships

ReBAC and Kubernetes RBAC

Substrate's API authorization uses OpenFGA's relationship-based access control model. In this version, ordinary Control RPC enforcement remains experimental and disabled by default.

Source: docs/security.md#L46.

Kubernetes RBAC checks applicable RoleBindings and ClusterRoleBindings against Role/ClusterRole rules: API groups, resources, verbs, namespaces and sometimes resource names. It does not automatically derive access from ownership or arbitrary parent-child relationships.

Source: Using RBAC Authorization.

ReBAC derives access from relations between subjects and objects, and potentially relations between objects. Inheritance is explicit in the authorization model, rather than automatic for every parent-child association.

Source: OpenFGA: Modeling—Getting Started.

Substrate's .fga model makes a Worker's read permission depend on its parent global object's viewer or controller relation:

type worker
  relations
    define parent_global: [global]
    define host_node: [node]

    define can_get: viewer from parent_global or controller from parent_global

Source: cmd/ateapi/internal/authz/model.fga#L29-L34.

An Atespace can inherit its owner from its parent global object:

type atespace
  relations
    define parent_global: [global]
    define owner: [user] or owner from parent_global

An ActorTemplate can derive read permission from the viewer relation on its parent Atespace:

type actor_template
  relations
    define parent_atespace: [atespace]

    define can_get: viewer from parent_atespace

Sources: cmd/ateapi/internal/authz/model.fga#L71-L77, cmd/ateapi/internal/authz/model.fga#L101-L106.

The relation data connects global → atespace → actor_template and global → worker. Permission checks follow those modelled relations. The practical difference from ordinary Kubernetes RoleBindings is the ability to derive child access from parent relations according to an explicit model.

Components and versions

Component	Version	Repository	Scope
Agent Substrate	v0.5.0	agent-substrate/substrate	Implementation, configuration and source examples
Kubernetes	v1.37.1	kubernetes/kubernetes	Pod certificate APIs and feature gates
kind	v0.33.0	kubernetes-sigs/kind	Local example using kindest/node:v1.37.0

GKE availability information reflects the official documentation checked on October 20, 2026. The GKE installer does not necessarily distribute the same version as the upstream tag used here.

Note: Agent Substrate is pre-1.0 and does not guarantee backward compatibility. Unless another repository is named, source links in this article refer to agent-substrate/substrate v0.5.0.

References

• Kubernetes Podcast #272: Agent Substrate
• Agent Substrate repository
• Bringing you Agent Sandbox on GKE and Agent Substrate
• Agent Substrate on GKE announcement
• Install Agent Substrate on GKE
• Architecture
• Threat model
• Glossary
• Security guide
• Egress credential injection
• Claude Code multiplex demo
• Kubernetes Pod Certificates KEP
• Kubernetes RBAC
• OpenFGA modeling
• RustFS