Learn Kubernetes by Building a Mini Container Orchestrator

You start three copies of your application. One crashes while you are away from your laptop. Two are still running, but you asked for three.

Something needs to notice the missing copy and replace it. That is a useful first step toward understanding Kubernetes.

The idea has two parts: what you want, called desired state, and what is actually happening, called observed state. Reconciliation means comparing the two and taking a useful next step. In this example, the next step is to start one replacement.

Figure 1: Compare the saved request with current observations, act, and check again.

This article starts with that idea, explains the Kubernetes components behind it, and gives you a small Node.js exercise you can run with Docker. We then add the harder questions: where work should run, what happens during a crash, and how to recover safely.

You can follow the explanation without Kubernetes experience. For the exercise, you should be comfortable creating a file, running a terminal command, and reading basic JavaScript. The result is a learning orchestrator, a program that coordinates containers. It is a starting point for understanding a cluster, not a replacement for production Kubernetes.

1. Why Kubernetes exists

From one application to many machines

Docker lets you package an application and its dependencies into an image, then run that image as a container. This makes deployment more repeatable.

Now imagine that your application needs several copies across several machines. You need to choose machines with enough capacity, replace failed copies, and send requests to instances that are ready. You also need a way to roll out a new version.

Kubernetes coordinates these tasks. A cluster is the collection of machines and components working together. You describe the result you want, and the system keeps working toward it. This is called a declarative approach.

For example, “keep three copies running” describes a result. It lets the platform decide which actions are needed now and after a later failure.

Everyday problem	What Kubernetes contributes
A machine has no room for another workload	Scheduling considers resource requests and placement rules
An application copy disappears	Controllers can create a replacement
Replacement copies have different addresses	A Service provides a stable way to reach them
A new version needs to be deployed	A Deployment coordinates application replicas and rollouts

These features still require decisions from you. Kubernetes does not automatically choose suitable resource settings or scale every app with traffic. Autoscaling needs its own configuration. For one small application on one host, the additional platform work may be unnecessary.

How Kubernetes developed

Kubernetes drew on Google's experience with cluster management, including Borg. Google open-sourced Kubernetes in 2014. Its donation to the CNCF accompanied the 1.0 release in 2015. Kubernetes is a separate project rather than an open-source copy of Borg.

The useful historical lesson is that packaging an application and coordinating many applications are different responsibilities. Containers helped with the first. Cluster management addresses the second. The Kubernetes history and Borg paper provide the background.

2. Understand the real Kubernetes architecture first

Figure 2: The control plane coordinates the cluster. Worker nodes run Pods. The cloud provider API manages external infrastructure.

Read the image from left to right

The large outer box represents one Kubernetes cluster. Inside it, the control plane appears on the left, and two worker nodes appear toward the middle. The cloud provider API sits outside the cluster on the right.

The image shows where the components belong and how they communicate. Its arrows do not represent a single application request travelling through every component.

Left: the control plane coordinates the work

The control plane records what you want, decides where new Pods should run, and checks whether the cluster is moving toward the requested state.

Component shown	Its job	A simple example
kube-apiserver	Provides the shared API for reading and changing cluster objects.	Receives your request for three application replicas.
etcd	Persists the cluster's API state.	Stores the requested configuration and reported status.
Scheduler	Chooses a suitable node for an unscheduled Pod.	Assigns a new Pod to Node 2 because it meets the requirements.
Controller manager	Runs controllers that reconcile different resources.	A ReplicaSet controller creates a replacement Pod when one is missing.
Cloud controller manager	Connects supported cluster operations to cloud APIs.	Coordinates a load balancer through the provider integration.

The cloud controller manager is optional. A cluster that does not need that cloud integration may not run it.

etcd stores cluster records, while application processes run on the nodes. It does not replace your application's database or store its container images. The Kubernetes component overview describes these roles.

Middle: Node 1 and Node 2 run your applications

Each worker node is a machine capable of running Pods. A Pod contains one or more containers that run together. The picture uses several blue Pod boxes to show that one node can host multiple Pods. It does not mean every node must have the same number.

The kubelet is the local agent. It learns which Pods are assigned to its node, asks the container runtime to run them, and reports status back through the API server.

The CRI label stands for Container Runtime Interface. This is how the kubelet communicates with a runtime such as containerd or CRI-O. The runtime itself is not drawn as a separate box here. CRI is an interface, rather than a storage area or a container that holds Pods.

kube-proxy handles a different responsibility: it usually configures network rules for Service traffic. It does not create Pods or start their containers. Some networking implementations replace kube-proxy's role. Read more about the runtime interface and kube-proxy.

Right: the cloud provider supplies infrastructure

The cloud-shaped box represents a provider's API, not another worker node. Kubernetes integrations can use it to manage supported infrastructure features, such as cloud load balancers or node information.

The dotted connection illustrates that integration. Ordinary requests to your web application do not travel through the cloud controller manager.

Where kubectl fits

kubectl is not drawn in this picture. It runs on your laptop, a CI system, or another client machine and communicates with kube-apiserver.

Its kubeconfig tells it how to connect and authenticate. A context selects a cluster, user, and optional namespace. A YAML manifest describes the object you want to create or update, such as a Deployment. See kubeconfig and contexts.

Follow one deployment through the image

Suppose you request three copies of a web application using a Deployment.

1. kubectl sends the manifest to kube-apiserver.
2. After access checks and validation, the API server persists accepted state in etcd.
3. The Deployment and ReplicaSet controllers arrange the required Pod objects.
4. The scheduler selects suitable nodes for the unscheduled Pods.
5. Each selected node's kubelet asks its runtime to start the containers.
6. The kubelets report status, and controllers continue checking for differences.

If you also create a Service, its configuration and backend information guide the network implementation in forwarding traffic. A Service does not start the Pods. These operations happen asynchronously, so accepting the manifest does not mean the application is already ready. See Deployments.

Services, DNS, and network routing

Figure 3: A typical ClusterIP Service provides stable discovery while network rules route traffic.

When a Pod is replaced, its network address can change. Clients need a stable way to find the application.

A typical ClusterIP Service supplies a virtual IP address. Cluster DNS lets clients use a name for that address. Labels, which are key-value tags on objects, help a Service select the relevant Pods.

EndpointSlices record the discovered backend addresses and their conditions. You can think of them as the backend directory used by the networking implementation.

kube-proxy usually configures node network rules for Service traffic. Some networking implementations provide an alternative. Application packets normally travel through this network data plane, the machinery that forwards traffic, without passing through etcd or the API server.

Service type	First use to remember
ClusterIP	Access inside the cluster through a virtual IP
NodePort	Access through a port on reachable nodes
LoadBalancer	Request an external load balancer from a supporting integration

A Service does not restart failed Pods. It solves access and discovery. Controllers and node agents handle workload lifecycle. Headless Services and special endpoint policies add variations to this basic model. See the Service documentation.

Infrastructure integrations

The optional cloud controller manager connects Kubernetes with cloud provider APIs, such as those used for load balancers and node information.

You may also encounter CNI, the Container Network Interface, and CSI, the Container Storage Interface. CNI integrations help provide Pod networking. CSI integrations connect storage systems. For this first build, remember their purpose. You do not need to implement them yet.

Example: what happens after kubectl apply

Imagine submitting a Deployment for three web application replicas and a Service for reaching them.

1. The API server accepts valid, authorized objects and persists the changes.
2. The Deployment controller manages the appropriate ReplicaSet.
3. The ReplicaSet controller creates the required Pod objects.
4. The scheduler assigns each new Pod to a suitable node.
5. The selected kubelets ask their runtimes to start the containers.
6. Readiness information helps determine which backends receive Service traffic.
7. Controllers keep checking as the application and cluster change.

These steps happen asynchronously and can overlap. An accepted API request does not mean all three replicas are ready. The Deployment documentation explains rollout status in more detail.

3. Design the system around durable intent

Figure 4: The larger learning project adds an API, database, placement, and node agents.

We will start with a finite job: a container that does some work and exits. A web service is expected to keep running, while a successful job should stay finished. This difference matters when deciding whether to restart a container.

The diagram shows a later version with an API, PostgreSQL, a scheduler, and a node agent. First, we can learn the central loop with just a saved job file and Docker on one machine.

Run your first reconciliation loop

You need Node.js 22 or newer, a running Docker Engine, and the Docker command available in your terminal. The example uses only Node's built-in modules, so there is no npm installation step.

Create a folder named mini-k8s-starter. Inside it, create these two files:

File	Purpose
job.json	The job you want to run
orchestrator.mjs	A loop that inspects Docker and takes the next step

Save this as job.json:

{
  "id": "demo-1",
  "image": "alpine:3.22",
  "command": ["sleep", "60"]
}

This job runs Alpine Linux's sleep command for 60 seconds. It gives you time to stop and restart the orchestrator while the container is still running.

Save this as orchestrator.mjs:

import { readFile } from 'node:fs/promises'
import { execFile } from 'node:child_process'
import { promisify } from 'node:util'
import { setTimeout as pause } from 'node:timers/promises'

const exec = promisify(execFile)
const owner = 'mini-k8s.owner=article-demo'
const once = process.argv.includes('--once')

async function docker(...args) {
  const { stdout } = await exec('docker', args, { timeout: 30_000 })
  return stdout.trim()
}

async function reconcile() {
  const job = JSON.parse(await readFile(new URL('./job.json', import.meta.url), 'utf8'))
  if (!/^[a-z0-9-]{1,40}$/.test(job.id) || typeof job.image !== 'string' ||
      !Array.isArray(job.command) || job.command.length === 0 ||
      !job.command.every(value => typeof value === 'string')) {
    throw new Error('Use a short lowercase ID, an image, and a command array')
  }

  const name = `mini-k8s-${job.id}`
  const names = (await docker(
    'container', 'ls', '-a', '--filter', `label=${owner}`, '--format', '{{.Names}}'
  )).split('\n')

  if (!names.includes(name)) {
    await docker('container', 'create', '--name', name, '--label', owner,
      '--memory', '64m', '--cpus', '0.5', job.image, ...job.command)
  }

  const [container] = JSON.parse(await docker('container', 'inspect', name))
  if (container.Config.Labels?.['mini-k8s.owner'] !== 'article-demo' ||
      container.Config.Image !== job.image ||
      JSON.stringify(container.Config.Cmd) !== JSON.stringify(job.command)) {
    throw new Error('Existing container does not match this job. Use a new job ID')
  }

  const state = container.State
  if (state.Status === 'created') {
    await docker('container', 'start', name)
    console.log(`${job.id}: start requested, checking status next pass`)
  } else if (state.Status === 'exited') {
    const result = state.ExitCode === 0 ? 'SUCCEEDED' : 'FAILED'
    console.log(`${job.id}: ${result}, exit code ${state.ExitCode}`)
  } else {
    console.log(`${job.id}: ${state.Status.toUpperCase()}`)
  }
}

do {
  try {
    await reconcile()
  } catch (error) {
    console.error(`Reconciliation error: ${error.message}`)
    if (once) process.exitCode = 1
  }
  if (!once) await pause(3000)
} while (!once)

From the folder containing these files, pull the image and start the loop:

docker pull alpine:3.22
node orchestrator.mjs

The loop first requests a start, then reports the observed state. After about a minute, the job should report SUCCEEDED with exit code 0.

Each pass reads your saved intent, lists the containers belonging to this exercise, and inspects the matching container. If it is missing, the loop creates it. If it has been created but not started, the loop starts it. If it has finished, the loop reports its result and leaves it finished.

Docker's create and inspect commands provide these separate lifecycle operations. Node's execFile passes arguments to the Docker command without constructing a shell command string.

Try the restart experiment

While the job is running, press Ctrl+C to stop the Node process. Start it again with node orchestrator.mjs.

The container name, mini-k8s-demo-1, stays the same. The restarted loop finds that container and resumes observation. Stopping this Node process does not stop a detached Docker container.

Inspect the result in another terminal:

docker container inspect --format '{{.Id}} {{.State.Status}}' mini-k8s-demo-1

You can also run node orchestrator.mjs --once to perform one check and exit. When finished experimenting, stop the loop before removing the demo container:

docker container rm -f mini-k8s-demo-1

Understand this exercise's limits

The file preserves intent, and the retained Docker container preserves observable execution history. There is no separate result database. If you delete the completed container while the loop still requests that job, the loop can create it again.

Use a new job ID for different work. This example does not support editing an existing job's image or command, automatic retries, cancellation, multiple nodes, or multiple orchestrator processes. Keep it on a trusted development machine because access to Docker gives substantial control over the host.

A useful detail is how errors behave: a failed Docker listing does not become an empty list. The pass reports an error and tries later. “I could not inspect the machine” and “the container does not exist” require different decisions.

Grow the exercise into a small control plane

For the next version, replace the hand-edited job file with an API and database. Store one durable record per job and one record per attempt. An attempt is a particular execution of that job.

Kubernetes responsibility	Learning implementation	Simplification
API server	Express API	A small job API
Persisted API state backed by etcd	PostgreSQL records	A relational model suited to the exercise
Controller loops	Job reconciler	One workload type
Scheduler	Placement worker	A few capacity checks
Kubelet	Node agent	Basic runtime operations and observation
Runtime integration	Docker CLI or Dockerode	Docker API access rather than CRI

Redis and BullMQ can later wake workers promptly. The database should still record which jobs exist. A missed queue notification must not make a saved job disappear.

4. Make reconciliation the center of the design

Reconciliation becomes useful when repeating a pass is safe. If the first pass already created a container, the next pass should find it and continue.

The technical word for a repeat-safe operation is idempotent. You do not need to remember the word before using the idea: checking twice should not accidentally launch the same work twice.

Job lifecycle

Figure 5: Each job state records a specific fact about placement or execution.

As the project grows, give each state a clear meaning:

State	What you know
PENDING	The job is saved and waiting for placement
ASSIGNED	A node and capacity reservation have been recorded
STARTING	The agent has claimed the attempt and is preparing execution
RUNNING	The runtime has confirmed that the container started
SUCCEEDED or FAILED	A terminal result and reason have been recorded

Sending a start request is not enough to claim RUNNING. A request can fail, or its reply can be lost after it succeeds. Your status should describe what you observed.

If a job may be retried, preserve the failed attempt and create a new attempt record. Otherwise, the history becomes difficult to explain.

A concrete failure example

Docker creates a container, but the worker crashes before saving the container ID in PostgreSQL. After a restart, the database appears to contain unfinished work.

Starting again without checking Docker could create a duplicate. Stable job and attempt labels let the worker discover the existing container, save its ID, and continue.

That is the same discovery idea used in the first exercise. The database makes more history durable, but it also adds another place where a crash can interrupt progress.

Use events to react quickly and occasional scans to repair missed work. After repeated failures, wait longer between retries. This backoff prevents a broken dependency from receiving a flood of requests.

5. Schedule capacity before you launch

Figure 6: Reserve CPU capacity before another scheduler can claim it.

With one node, placement is easy. With two nodes, the scheduler needs a trustworthy view of available resources.

First, rule out nodes that are unhealthy or cannot satisfy the job's requirements. Then choose among the remaining nodes. If none fits, leave the job pending and explain why.

Consider this example:

Item	CPU capacity or request
Node A	2 units available
Node B	1 unit available
Job 1	Requests 1.5 units
Job 2	Requests 1.5 units

Place Job 1 on A. A now has only 0.5 units available, so Job 2 must wait.

Two schedulers can get this wrong if both read A's old capacity before either records a reservation. In the database version, selecting the node and reserving its capacity must be one guarded transaction. A transaction groups database changes so competing workers cannot both claim the same capacity.

In Kubernetes, resource requests help decide placement, while limits constrain runtime use under applicable enforcement. Successful scheduling does not guarantee that a process will never exhaust memory or encounter node pressure. See resource management and node-pressure eviction.

6. Treat external side effects as crash windows

Figure 7: Different interruptions require different recovery actions.

A crash window is a gap where one operation succeeds, but the next has not yet completed. Starting a container changes the outside world. Saving its ID changes the database. Those operations do not share one database transaction.

Failure	What might still exist	Next safe step
API stops after saving a job	Pending database record	Find it during the next scan
Worker stops after reserving a node	Reservation without a container	Check the runtime, then resume or release it
Docker succeeds but its reply is lost	Running or stopped container	Discover it before creating another
Agent loses network contact	Container may still be running	Mark observations stale and investigate
Node becomes unreachable	Work may still be executing	Decide whether another execution is safe

That last row matters for jobs that send emails, charge cards, or update another system. Losing contact with a node does not prove the node stopped working. Retrying elsewhere can repeat the external action.

An application-level idempotency key gives repeated requests for the same logical operation a stable identity, allowing a supporting service to recognize duplicates. Systems requiring a single active writer may also need fencing, which prevents an old worker from continuing to modify a protected resource. The scheduler alone cannot guarantee exactly once execution.

Protect access to the runtime

Restrict who can submit jobs and which privileges those jobs receive. A container is not automatically a safe place to run code from an unknown user. The Kubernetes security checklist covers the broader production considerations.

7. Add Kubernetes features only when the previous layer works

Figure 8: Add capabilities in stages, testing recovery as you go.

Use the first exercise as a checkpoint. Then build in this order:

1. Add an API and database that preserve requested jobs and their results.
2. Recover existing attempts after a worker restart.
3. Add a second node and atomic capacity reservations.
4. Add retries with attempt history and a clear retry budget.
5. For continuously running applications, add replicas, readiness checks, and routing.
6. Add storage, access policy, backups, and upgrade procedures as the project needs them.

Separate startup, readiness, and liveness

Figure 9: Use separate checks for initialization, traffic readiness, and restart decisions.

A process can be running without being ready to serve users. Kubernetes distinguishes three checks:

Check	Meaning	Main effect
Startup	Initialization has completed	Gates readiness and liveness checks until it succeeds
Readiness	This instance can receive traffic	Failed readiness normally removes it from eligible Service backends
Liveness	The container needs a restart to recover	Repeated failure can trigger a restart

For example, an application loading initial data may need time before it is ready. Restarting it repeatedly during that work can prevent it from ever finishing. This is why the checks have separate jobs. See the probe documentation.

Persistent storage is another separate concern. A container restart, a Pod replacement, and a lost node can affect data differently. Kubernetes PersistentVolumes have a lifecycle independent of individual Pods, but you still need a suitable storage implementation and recovery plan. See PersistentVolumes.

How to test that you learned the right lesson

Run a failure drill whenever you add a responsibility. Stop a worker after it reserves capacity. Stop it after a container starts. Deliver the same queue notification twice. Make an image download fail. Disconnect an agent while its container keeps running.

For each drill, record the saved intent, what the runtime actually did, and what the next worker can safely do. The goal is a truthful result and explainable recovery, even when errors occur.

8. Compare the approach with alternatives

Approach	Advantage	Limitation	Useful when
Docker Compose on one host	Simple deployment and debugging	No cluster-wide placement	One host meets your needs
Managed container platform	Less infrastructure to operate	Provider-specific behavior and costs	Delivering the application is the priority
Custom learning orchestrator	Makes scheduling and recovery visible	Considerable work to make production-ready	You want to understand the mechanisms
Kubernetes	Established APIs and integrations	Requires platform knowledge and operations	Your workloads justify those capabilities

Kubernetes makes some configuration portable, but networking, storage, load balancers, and operating practices still differ across environments. Moving providers can require work. A custom orchestrator also creates maintenance responsibilities that outlast the first working demo.

Common mistakes

• Treating a queue notification as the only record of a job.
• Calling every background worker a scheduler.
• Assuming an unreachable node has stopped its containers.
• Restarting a successful finite job as if it were a web service.
• Reporting an application as running before the runtime confirms it.
• Adding features before testing the failure paths they introduce.

9. What changes as you approach a real cluster

The next challenges include authenticating users, deciding who can change which objects, keeping the control plane available, supporting upgrades, and observing failures across machines. Kubernetes implements these concerns through multiple components and extensions.

One example is an API watch, a stream of changes that helps a component maintain a local view of objects. A reliable client must handle interruptions and refresh its view correctly. The Kubernetes API concepts reference explains this in depth.

The specific integrations can evolve. The habit you are learning remains useful: preserve what was requested, inspect what happened, and choose a recovery action that is safe to repeat.

Practical checklist

• [ ] Explain desired state and observed state in your own words.
• [ ] Run the first job and observe its exit result.
• [ ] Restart the loop while the container is active and confirm it finds the same container.
• [ ] State where intent and execution history are stored.
• [ ] Give jobs and attempts stable identities.
• [ ] Make capacity reservations safe under competing workers.
• [ ] Test the gaps between database writes and runtime operations.
• [ ] Document when retries could repeat external side effects.
• [ ] State which Kubernetes features the project does not implement.

Start with one job whose behavior you can explain after a restart. Once you can trace its intent, execution, and result, you have a practical foundation for understanding the larger system.