Microservices at Edge with K3s and Fleet

Imagine you're building a coffee shop application with three core microservices:

• order-service: Handles customer orders.
• payment-service: Processes transactions.
• inventory-service: Tracks beans, milk, cups, and other inventory.

Now scale this across hundreds or thousands of physical store locations—each with its own on-site device like a Raspberry Pi or Intel NUC. How do you maintain consistency, ensure reliability, and keep deployments secure across all sites?

That’s where K3s (a lightweight Kubernetes distribution for edge) and Rancher Fleet(GitOps engine) are designed to manage thousands of clusters to help you.

In this guide, you learn how to build a resilient, scalable edge infrastructure using:

• K3s to run Kubernetes clusters on resource-constrained devices,
• Fleet to manage deployments across hundreds or thousands of clusters,
• OpenFaaS and MQTT for lightweight event-driven automation and telemetry,
• And strategies for handling air-gapped stores, remote access, logging, and storage.

What Is a Container Image?

Before diving into Kubernetes or K3s, it’s essential to understand container image

A container image is a lightweight, portable unit that includes everything needed to run a service that includes:

• Application code
• Runtime (like Python, Java, or Node.js)
• System libraries and binaries
• Configuration and dependencies

For example, in the coffee shop app each service: order-service, payment-service, and inventory-service is built as a container image:

• order-service: Python + Flask + order logic
• payment-service: Node.js + Stripe SDK
• inventory-service: Go + SQLite + inventory tracker

Kubernetes (K8s) is an open-source system that automates the deployment, scaling, and management of containerized applications. It provides a robust, extensible platform for orchestrating containers across clusters of machines, simplifying the management of distributed, cloud-native systems.

Note: For more information, refer to Introduction to Container Images and Orchestration.

Understanding Edge Computing

Edge computing refers to placing compute resources as close as possible to the data source or end-user. In our case, this means deploying services directly into each coffee shop rather than relying on centralized cloud data centers.

Benefits of Edge Computing:

• Real-time inventory decisions: Milk running low, auto order more.
• Low-latency UX: Instant response at self-service kiosks.
• Data sovereignty: Payment data stays local to comply with regulations.

Challenges at the Edge: While beneficial, edge environments present unique challenges:

• Resource constraints: Devices at the edge, such as those in your coffee shops, often have lower compute and memory capacity.
• Intermittent connectivity: Edge devices may not maintain a constant 24/7 internet connection.
• Remote management difficulty: You cannot manually SSH into hundreds of individual store servers to update software.

K3s: Kubernetes Optimized for the Edge

K3s is a lightweight, CNCF-certified Kubernetes distribution built specifically for edge and IoT environments. Developed by Rancher, K3s simplifies cluster setup while remaining fully compatible with Kubernetes tooling and APIs. It’s ideal for resource-constrained devices—such as Raspberry Pis or Intel NUCs deployed in coffee shops or retail branches.

Key K3s Features for Edge Deployments:

• Single binary distribution (~100MB) for fast installs and minimal overhead.
• Embedded components: containerd (container runtime), runc, and kubectl.
• Supports multiple storage backends: SQLite (via Kine), embedded etcd, and external SQL (MySQL/PostgreSQL).
• Works on ARM and x86_64 platforms.
• Pre-configured defaults: Includes Flannel for networking, Traefik for ingress, and metrics-server for observability.

How K3s Optimizes for the Edge

• Unified Binary: Embeds everything needed—container runtime, CRI tools, control plane in one downloadable file. Makes updates easy via binary replacement.
• Kine (SQLite integration): Reduces overhead by emulating etcd with SQLite, ideal for single-node clusters without consensus requirements.
• Minimal System Requirements: Can run in 512MB RAM, uses minimal CPU.

Comparing K3s and Kubernetes

Capability	K3s	Upstream Kubernetes
Installation	Single binary, HTTP tunnel	kubeadm, multiple binaries
Control plane	Embedded etcd or SQLite	etcd only
Runtime	Embedded containerd + runc	External container runtime
System requirements	~500MB RAM, low CPU	1GB+ RAM, higher CPU
HA options	SQLite, etcd, SQL	etcd only
Targets	Edge, IoT, hobbyists	Data centers, cloud

There are several personas or users of Kubernetes. Refer how K3s may affect each of them:

• Developers: Write standard manifests. No changes needed.
• Platform Engineers: Use pre-bundled defaults to reduce setup time.
• SREs/SecOps: Need to understand K3s-specific HA, upgrades, and bootstrap.

K3s renames Kubernetes control plane nodes to "servers" and worker nodes to "agents". It uses an HTTP tunnel for simplified server-agent communication, Flannel for networking by default, and Traefik for ingress.

• Server: Control plane node
• Agent: Worker node

K3s Architecture

K3s is purpose-built for edge and resource-constrained environments. Unlike upstream Kubernetes, which requires multiple binaries and external dependencies, K3s is distributed as a single binary. Internally, K3s embeds several core components:

• containerd and runc: Used as the default container runtime.
• Embedded SQLite or etcd: For storing Kubernetes cluster state. SQLite is ideal for single-node clusters; etcd enables high availability.
• Built-in defaults: Includes Flannel (CNI), Traefik (Ingress), metrics-server, and Helm controller.

K3s uses a simplified bootstrapping process where agents (workers) connect to the control plane (server) via an encrypted tunnel (k3s-agent uses reverse tunnels).

Note: In smaller stores, a single-node K3s cluster can run all three services—order, payment, and inventory—on one device.

High Availability (HA) with K3s

If you want your cluster to tolerate failures (e.g., a power outage at one device), K3s supports HA using embedded etcd. This mode is recommended for clusters that run critical systems and must remain available even during node failures.

For resilience, K3s supports two HA modes:

• Embedded etcd (Raft consensus): Suitable for production. Requires 3 or 5 server nodes.
• External SQL DB (PostgreSQL/MySQL): Lightweight but harder to scale.

What's up with k3sup?

k3sup (pronounced "ketchup") is a community tool that simplifies K3s installation and joining agents to a cluster over SSH, making remote setup trivial. k3sup can also update your kubectl configuration file.

Why Use k3sup?

• Remote installation via SSH: great for air-gapped or low-access environments.
• One-line cluster provisioning:ideal for bootstrapping multiple stores quickly.
• kubectl auto-merge: automatically adds new K3s clusters to your KUBECONFIG file.

Fleet: Centralized Multi-Cluster Management

Fleet is Rancher’s GitOps controller purpose-built to manage thousands of Kubernetes clusters—ideal for widespread edge deployments like your coffee shop application. While K3s runs a lightweight cluster at each store, Fleet offers centralized visibility, version control, and deployment across all of them.

Why Use Fleet with K3s?

• Central control, local autonomy: You manage manifests, Helm charts, and configurations from a single Git repository. Each K3s store fetches its configuration independently.
• Designed for scale: Fleet supports up to 1 million clusters and uses lightweight agents with minimal overhead.
• Pull-based sync: Suitable for remote stores with dynamic IPs or unreliable connectivity.

How Fleet Works?

1. Git Repository as Source of Truth You store application definitions (YAML, Helm, Kustomize, or OCI bundles) in a Git repo. These include:
   • fleet.yaml for configuration
   • Manifests for each microservice (order.yaml, payment.yaml, inventory.yaml)
2. Fleet Manager (Controller) The Fleet controller runs in a central management cluster (on-prem or in the cloud). It watches Git repositories and generates bundles for deployment.
3. Fleet Agents in Edge Clusters Each K3s cluster runs a Fleet agent. This agent:

• Connects back to the Fleet manager securely.
• Pulls the correct workload bundle.
• Applies the manifests locally.
• Reports status.

Coffee Shop Deployment: fleet.yaml

defaultNamespace: coffee-shop  
  targets:  
  \- name: edge-shops  
    clusterSelector:  
      matchLabels:  
        location: edge  
    yaml:  
      \- path: manifests/order.yaml  
      \- path: manifests/payment.yaml  
      \- path: manifests/inventory.yaml

Benefits of Using Fleet in Edge Deployments

• Resilient by Design: If a cluster goes offline, Fleet retries syncing when it's back online. No manual recovery needed.
• Declarative GitOps Workflow: All changes are Git-driven and reproducible. Every store gets the same, tested configuration.
• Security & Separation: Since clusters pull workloads from Fleet, they don't need inbound access or public IPs.

Remote Access

Because edge locations (coffee shops) often lack:

• Static IPs
• VPNs
• SSH access

Fleet enables secure pull-based management, where the cluster reaches out to Fleet rather than requiring a central controller to initiate communication.

Alternatives and Tradeoffs

Method	Pros	Cons
Fleet (GitOps)	Secure, scalable, automatic retries	Requires pre-registration
VPN	Secure tunneling	Complex to set up at scale
SSH	Quick setup, compatible with k3sup	Hard to scale, brittle
Inlets	Easy tunneling of HTTP/TCP	Requires cloud relay or tunnel server

When managing 1,000+ coffee shops, you can't SSH into each branch to make changes. Most shops lack static IPs or stable internet. Instead of managing clusters by pushing changes from a central control plane, Fleet agents run inside each K3s cluster and initiate outbound connections to the central Fleet controller. This helps you:

• Avoid the need for public or routable IP addresses.
• Work over common outbound ports (e.g., HTTPS).
• Enable auto-recovery when clusters reboot or reconnect

For example, in coffee shop app:

• A barista at "Store 042" restarts their Raspberry Pi.
• It boots K3s, pulls the latest manifests from Fleet via Git.
• Fleet ensures order-service, payment-service, and inventory-service run correctly without manual intervention.

Air-Gapped Environment

Many remote stores may lack internet access. K3s and Fleet support full air-gapped deployments.

In these setups, you prepare everything ahead of time using USB drives or local staging networks.

Resource	Where to Load	Notes
K3s binary	Flash drive or staging laptop	Install manually
Container images	/var/lib/rancher/k3s/agent/images/	Use ctr images export and import
fleet.yaml and app manifests	Local Git repo clone or OCI registry mirror	Load from USB or portable device
Fleet agent	Bundled with app or installed offline	Uses local bundle and syncs when connected

Note: For air-gapped shops, you can use a portable Git repo or OCI registry mirror hosted temporarily on a laptop or USB stick.

Fleet still works offline by syncing from this local source. Later, when connectivity returns, these nodes can reconnect and pull updates as usual.

What Is MQTT and Why Use It at the Edge?

MQTT (Message Queuing Telemetry Transport) is a lightweight, publish-subscribe messaging protocol designed for constrained devices and unreliable networks—making it ideal for edge computing.

For example, in the Coffee Shop app, each store has:

• A temperature sensor inside a fridge.
• An espresso machine that counts daily cycles.
• A grinder that logs wear or usage hours.

MQTT enables these devices to publish events (like temp=97°F) to a local broker running in the store (e.g., Mosquitto). Meanwhile, your backend services (e.g., an OpenFaaS function or logging collector) subscribe to relevant topics (/store104/fridge/temp).

Why MQTT at the Edge?

• Offline Resilience: Devices buffer messages even when the internet is unavailable.
• Loose Coupling: Producers don’t need to know who the consumers are.
• Asynchronous: Ideal for non-blocking sensor data or logs.
• Bandwidth Efficient: Optimized for unreliable or metered networks.

When connectivity returns, MQTT relays buffered messages to:

• A central cloud MQTT broker (e.g., EMQX, HiveMQ, or AWS IoT)
• A data lake, time-series database, or dashboard via Kafka or Prometheus exporters

Using OpenFaaS for Functions at the Edge

OpenFaaS (Function-as-a-Service) is a serverless framework designed to run lightweight, containerized functions on Kubernetes—including K3s at the edge.

Why Use It at the Edge?

• Reduces the overhead of running full-blown apps
• Fast deployment via Kubernetes CRDs
• Supports HTTP, MQTT, and CRON as event sources
• Integrates with container registries and GitOps workflows

For example, in a coffee Shop app consider store 14 has a Raspberry Pi that:

• Receives real-time fridge temperature via MQTT.
• Triggers a Python-based function when data is published.

def handle(req):  
    temp \= float(req)  
    if temp \> 95:  
        return "Warning: Too hot\!"  
    return "Temperature OK"

Other Edge Functions You Can Run:

• Detect low inventory thresholds and alert inventory-service.
• Sanitize incoming sensor data and write to local disk.
• Detect anomalies and forward to a machine-learning model.

OpenaaS wraps this as a function, builds a container image, and deploys it via Kubernetes CRDs. Supports MQTT, HTTP, and CRON-based triggers.

Integrating MQTT and OpenFaaS

Integrating MQTT and OpenFaaS provides a powerful, event-driven model for edge computing. MQTT acts as the event transport layer, while OpenFaaS serves as the event processor. Together, they create a reactive system that processes local data in real time and only syncs to the cloud when necessary.

Consider Store 14's Fridge Monitoring in a coffee shop app:

1. Sensor detects fridge temperature and publishes a message:
   • Topic: /store104/fridge/temp
   • Payload: {"temp": 97.4, "unit": "F"}
2. Mosquitto (local MQTT broker) receives the message.
3. OpenFaaS MQTT Connector subscribes to the topic, reads the payload, and invokes a temperature-check function.
4. The function evaluates the temperature:
   • Logs a warning if it's too high.
   • Optionally triggers an alert (e.g., send webhook, write to disk, store in SQLite).
5. If the internet is offline, logs and events are buffered locally.
   • MQTT persists the message.
   • OpenFaaS writes logs to a mounted persistent volume.
6. When connectivity is restored:
   • Logs or alerts can be forwarded to a centralized logging system or metrics database.

Logging and Storage in Edge Scenarios

In edge environments—like individual coffee shop branches—connectivity isn't guaranteed. Devices must operate independently for logging, metrics collection, and temporary data storage. Here's how to plan for these constraints.

Local Storage Considerations

Each store’s K3s device should have storage for:

• System: K3s binary, operating system, configuration files.
• Runtime: Container logs, telemetry, ephemeral workloads.
• Persistent Data: Customer orders, payment events, inventory state.

This ensures store operations can continue even if cloud connectivity is lost.

Logging Options

K3s includes metrics-server for lightweight resource monitoring. For advanced use cases:

• Prometheus/Grafana can run on each edge cluster for real-time observability.
• Use Loki or Elasticsearch to forward logs when reconnected.
• Local logs can be written using Kubernetes emptyDir or persistent volumes.

Note: For more information, refer to Understanding Observability with OpenTelemetry and Coffee.

Syncing Logs and Events

Logs and events generated at the edge can be buffered and forwarded later:

• Use MQTT to publish sensor data or application events to a central broker.
• Store logs on local disk or persistent volume.
• Sync to the cloud (object storage, log aggregation system) when a connection becomes available.

Best Practices

• Cache locally: Ensure services like inventory-service and payment-service write to disk.
• Use queues: MQTT or Redis queues help avoid data loss when offline.
• CSI Plugins: Use Kubernetes-compatible storage interfaces suited for edge devices.
• Backups: Use the 3-2-1 rule: 3 copies, 2 local (disk + USB/flash), 1 remote (cloud or data center).

Addressing Common Edge Challenges with K3s and Fleet

The combined power of K3s and Fleet helps you overcome typical edge computing hurdles:

• Connectivity: K3s nodes might go offline, but Fleet handles retries and synchronizes changes when network connectivity is restored.
  • For scenarios where continuous connectivity is a concern, MQTT can be used to buffer and forward messages from edge devices to a central network.
• Upgrades: K3s offers a simplified one-binary upgrade process, and Fleet centrally manages the redeployment of your applications across clusters.
• Storage: At the edge, you need to consider storage for the operating system, K3s itself, logged data, and container images.
  • Local Data: For local logs or transactions, you can use persistent volumes (if available) or write data to local disk and sync to the cloud when connected.
  • Concerns: Be mindful of latency, capacity, and reliability when designing your storage solution.
  • Backups: Employ the 3-2-1 backup method: three copies of data, on two different media, with one copy off-site.