DEV Community: hridyesh bisht

Your Telemetry Has Data. Does It Have Context?

hridyesh bisht — Fri, 13 Nov 2026 18:01:09 +0000

Imagine you're investigating a slow checkout experience. Your application generates metrics, logs, and traces, but connecting the information across these signals isn't always straightforward.

For example, one component identifies a service as checkout-api, while another reports it as checkout. Although both refer to the same logical service, inconsistent naming makes it harder to connect related telemetry and understand what's happening.

This highlights an important challenge in distributed systems: collecting telemetry is only the first step. You also need consistent naming, shared context, and reliable ways to navigate between signals.

In the previous article, we explored how Prometheus, Jaeger, OpenSearch, and Perses fit into an open-source observability stack. Now, let's look at how to make telemetry from different services and signals work together.

We'll explore three concepts that make this possible: semantic conventions, resource attributes, and context propagation.

Semantic conventions

OpenTelemetry semantic conventions define standard names and meanings for telemetry attributes, events, and operations. They help instrumentation libraries and applications describe common concepts consistently, making telemetry easier for observability tools to interpret.

For example, OpenTelemetry defines standard HTTP attributes:

Attribute	Example value	What it describes
`http.request.method`	`POST`	The HTTP request method
`http.route`	`/payments/{payment_id}`	The matched route template
`http.response.status_code`	`201`	The HTTP response status code

However, semantic conventions don't enforce consistency automatically. Your instrumentation and telemetry pipeline must use and preserve the defined conventions. Custom attributes might still be necessary for application-specific details.

Note: Pay attention to attribute cardinality, especially when using attributes as metric dimensions. Use route templates such as /payments/{payment_id} rather than individual URLs such as /payments/12345. Recording unique values for every request can create high-cardinality metric data, increasing storage and query costs.

Resource attributes vs. span attributes

Resource attributes

A resource represents an entity that produces telemetry, such as a service, process, or container. Its attributes provide identifying information that can accompany the entity's metrics, logs, and traces.

For example, a payment-service instance might report the following resource:

{
  "service.name": "payment-service",
  "service.namespace": "checkout",
  "service.version": "2.4.1",
  "deployment.environment.name": "production",
  "service.instance.id": "payment-service-instance-01"
}

Each attribute provides a different piece of information:

service.name identifies the logical service.
service.namespace groups related services.
service.version identifies the deployed version.
deployment.environment.name identifies the deployment environment.
service.instance.id distinguishes this instance from other instances of the same service.

For example, multiple instances of payment-service can share the same service.name, service.version, and environment while reporting different service.instance.id values.

Span attributes

A span represents an individual operation within a trace. Its attributes describe what happened during that operation.

For example, a span created when payment-service handles an HTTP request might contain:

{
  "http.request.method": "POST",
  "http.route": "/payments/{payment_id}",
  "http.response.status_code": 201
}

These attributes describe the request handled by the service, rather than the service itself.

Note: This distinction helps you investigate telemetry at different levels. Resource attributes identify the source of telemetry, while span attributes provide details about individual operations.

Context propagation

A distributed request often passes through several services. To understand the complete request, you need to connect the spans generated by each service into one distributed trace.

OpenTelemetry context propagation carries information between services and execution units. For distributed tracing, this context includes identifiers such as the trace ID and span ID. The receiving service extracts the incoming context and uses it when creating its own span.

OpenTelemetry commonly uses the W3C Trace Context specification to propagate trace context across HTTP requests.

For HTTP requests, services commonly transmit this context through the traceparent header. Its format is:

<version>-<trace-id>-<parent-id>-<trace-flags>

For example:

00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01

Each field serves a specific purpose:

Field	Description
`version`	The Trace Context format version
`trace-id`	The identifier shared by all spans in the distributed trace
`parent-id`	The span ID of the calling operation
`trace-flags`	Trace-related flags, including the sampling decision

When a service receives a request, it extracts the incoming context and uses it as the parent context for its new span. Before calling another service, it propagates the updated context. This process connects spans across the request path.

The following sequence shows how the API gateway, order service, and payment service propagate context.

The resulting trace represents the request across all three services. You can inspect it in a tracing backend such as Jaeger to follow the request and identify where an operation takes longer than expected.

What context propagation does not do

Context propagation carries context information, not telemetry data. It does not transfer log records or metric measurements between services. Each service must still generate and export its own telemetry.

It also does not guarantee that every signal will automatically appear together in an observability tool. Correlation requires the relevant trace context, resource information, instrumentation, and backend support.

Propagation also requires attention at service boundaries:

Asynchronous communication: Message queues and event-driven systems need to carry context through message headers or other supported metadata. Depending on the instrumentation, you might need additional configuration or manual propagation.
External services: Avoid propagating sensitive information to untrusted services. In particular, review what you include in baggage and define clear propagation boundaries.

Correlating logs, metrics, and traces

Two mechanisms are particularly useful for connecting logs and metrics to traces: trace identifiers in log records and exemplars in metrics.

Linking logs to traces

A log record can include the trace and span IDs active when the application emits it. These identifiers help an observability backend associate the log with the operation that generated it.

For example, the payment-service might emit the following log record:

{
  "timestamp": "2026-11-10T10:30:00Z",
  "severity_text": "ERROR",
  "body": "Payment provider request timed out",
  "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736",
  "span_id": "00f067aa0ba902b7"
}

This is an illustrative JSON representation. OpenTelemetry's log data model defines the corresponding fields as TraceId and SpanId. Exporters and logging integrations can map them to different field names and formats.

With these identifiers, you can open a trace, locate the relevant span, and inspect the associated log records. For example, a timeout message linked to a payment span can help you investigate why a checkout request failed.

This correlation requires a logging integration or another mechanism that captures the active trace context and adds the identifiers to log records. Not every logging framework does this automatically.

Linking metrics to traces with exemplars

Metrics help you identify patterns and changes in system behavior, such as increased request latency or error rates. However, an aggregated metric usually doesn't show which individual request contributed to a measurement.

Exemplars provide a way to investigate individual measurements. An exemplar associates a metric measurement with additional context, which can include a trace ID and span ID.

Consider an HTTP request duration histogram. The following is a simplified representation of the histogram and one of its exemplars:

Metric: http.server.request.duration

Aggregated measurements:
  Request count: 1,250
  Duration distribution: ...

Exemplar:
  Observed value: 2.4 seconds
  Trace ID: 4bf92f3577b34da6a3ce929d0e0e4736
  Span ID: 00f067aa0ba902b7

The histogram helps you identify a latency problem across many requests. The exemplar provides a reference to a particular measurement and its trace context, so you can investigate the corresponding request.

Exemplars are optional. Their availability depends on instrumentation, SDK and exporter configuration, and backend support. The referenced trace must also be available in your tracing backend.

Avoid adding trace IDs or span IDs as metric labels. Because these identifiers can vary for every request, using them as labels creates high-cardinality time series and increases storage and query costs. Exemplars provide a more suitable way to link metric measurements to traces.

How the signals fit together

The following diagram distinguishes shared resource information from direct links between signals.

" width="594" height="622">

Resource attributes help identify and group telemetry from the same entity. Logs can link directly to traces through trace and span identifiers, while metric exemplars provide references to trace context.

Together, these mechanisms help you move from an aggregated view of system health to the details of an individual request.

Common correlation gaps

Even with consistent instrumentation, implementation and configuration issues can make cross-signal correlation unreliable.

Context propagation stops at service boundaries: HTTP instrumentation might handle propagation automatically, but asynchronous messaging, custom protocols, and unsupported libraries can require additional configuration.
Logs don't contain trace context: A logging framework might not automatically include trace and span IDs. Verify that your logging integration captures the active context and exports the identifiers with log records.
Exemplars aren't available or preserved: Exemplar support depends on the SDK, exporter, and observability backend. Verify that your pipeline preserves exemplars and that your backend can use them to navigate to traces.
A missing trace is mistaken for an instrumentation failure: Sampling decisions, export failures, and retention policies can affect trace availability. Check these settings before concluding that instrumentation failed.
Separate backends don't support cross-signal navigation: Exporting metrics, logs, and traces to different destinations doesn't guarantee that you can navigate between them. Check whether your observability tools support the required correlation features and whether you have configured them correctly.

Validate telemetry correlation

Use this checklist when testing a new service or reviewing an existing telemetry pipeline:

[ ] Send a test request that crosses at least two services.
[ ] Verify that the resulting spans share a trace ID and have the expected parent-child relationships.
[ ] Confirm that the corresponding log records contain the expected trace and span identifiers.
[ ] Check whether metric exemplars link to traces that are available for inspection.
[ ] Test cross-signal navigation in your observability backend.
[ ] Repeat these checks across asynchronous and external service boundaries where applicable.

Dash0

Dash0 is an OpenTelemetry-native observability platform that brings metrics, logs, traces, and resource information into a unified interface. It also provides service monitoring, dashboards, alerting, and query capabilities.

Dash0 extends OpenTelemetry semantic conventions with its own dash0.* attributes, metrics, and events. It maintains a public registry that documents the meaning and usage of these fields.

Dash0 also supports semantic convention migrations during ingestion. These migrations help align incoming telemetry with a selected conventions version. For example, Dash0 can rename http.status_code to http.response.status_code or move service.name from span attributes to resource attributes.

Consider the checkout application from our earlier example. Users report slow checkout requests, and the operations team needs to identify the affected service.

Dash0 provides several tools to investigate the issue:

Identify the problem: The Service Catalog helps identify checkout services with increased latency or error rates.
Find the dependency: The Service Map shows service relationships and request flows, helping identify which downstream services need investigation.
Inspect the request: The Trace Explorer lets you examine a representative trace and locate slow operations across the service path.
Find supporting evidence: Inspect related logs for errors, such as payment provider timeouts, associated with the affected request.
Verify the fix: After resolving the issue, compare latency and error metrics to confirm that the service has recovered.

The following diagram summarizes the workflow.

A unified observability platform doesn't remove the need for correct instrumentation. Dash0 cannot reconstruct missing trace context or generate application-specific telemetry that your instrumentation doesn't capture. You still need to configure your OpenTelemetry SDKs and Collectors correctly.

You should also evaluate data residency, deployment requirements, telemetry volume, and compatibility with existing tools and workflows before adopting the platform.

Dash0 is worth considering when you want a managed, OpenTelemetry-native experience without operating multiple observability backends yourself. However, teams with specific infrastructure, compliance, or backend requirements should validate those needs before migrating.

For more information, see the Dash0 documentation.

Conclusion

Collecting metrics, logs, and traces gives you visibility into your applications. Connecting those signals gives you a more complete picture of system behavior.

OpenTelemetry provides several building blocks for doing this:

Semantic conventions establish common names and meanings for telemetry.
Resource attributes identify the entities producing telemetry.
Context propagation connects operations across service boundaries.
Log trace context and metric exemplars provide ways to navigate between signals.

These mechanisms don't remove the need for correct instrumentation or backend configuration. They provide the shared structure and context that make correlation possible.

When telemetry follows consistent conventions and carries the right context, you spend less time figuring out what each data point represents and more time understanding what happened.

Beyond the Collector: How the Open-Source Observability Stack Fits Together

hridyesh bisht — Fri, 06 Nov 2026 07:13:59 +0000

In Understanding Observability with OpenTelemetry and Coffee, I introduced the fundamentals of observability. In Native OpenTelemetry, I explored how OpenTelemetry generates and processes telemetry, how the OpenTelemetry Collector manages telemetry pipelines, and why understanding those pipelines matters when troubleshooting production issues.

But what happens after telemetry leaves the Collector? Which tools store and analyze it? How do you connect these tools, and what should you consider before operating them in production?

This article explores the open-source observability ecosystem, from the responsibilities of individual tools to the engineering decisions involved in running them together.

OpenTelemetry isn't your observability backend

OpenTelemetry is an open-source, vendor-neutral observability framework. It provides APIs, SDKs, instrumentation libraries, and tools for generating, collecting, and exporting telemetry such as metrics, logs, and traces.

It also defines common data formats and semantic conventions that help applications describe telemetry consistently. However, OpenTelemetry isn't an observability backend. It doesn't replace the storage, query interfaces, and dashboards that teams use to investigate production issues. Those capabilities come from other tools.

Think of OpenTelemetry as the system that standardizes how information leaves each part of the coffee shop. The observability tools help you examine that information and understand what happened.

Let's look at what each tool contributes to an observability stack.

Prometheus

Prometheus commonly scrapes metrics endpoints exposed by applications and infrastructure. This pull-based model lets Prometheus collect measurements at configured intervals and discover targets through supported service-discovery mechanisms.

Prometheus is useful when you need to answer questions such as:

Is request latency increasing?
Are error rates rising after a deployment?
Is an application approaching its CPU or memory limits?
Does the system meet its latency or availability targets?

Note: You can also use alerting rules to detect conditions that require investigation.

When integrating Prometheus with OpenTelemetry, decide how Prometheus will receive metrics. A common approach is to scrape an endpoint exposed by an application or the OpenTelemetry Collector's Prometheus exporter.

Resources

Documentation: Prometheus documentation
Community: Prometheus community channels, including the #prometheus channel on CNCF Slack.
Discussions: Prometheus user forum

Jaeger

Jaeger provides a platform for exploring distributed traces and understanding how requests move through services.

Jaeger is useful when metrics tell you that a service is slow but don't explain which operation causes the delay. You can inspect individual traces to investigate service dependencies, slow operations, and errors.

When integrating Jaeger with OpenTelemetry, configure a compatible trace ingestion path, such as an appropriate OTLP endpoint. Also plan for trace storage and retention based on your deployment requirements. Jaeger uses a configured storage backend, so you must account for that backend in your architecture.

Resources

Documentation: Jaeger documentation
Community: Jaeger community channels, including the #jaeger channel on CNCF Slack.
Discussions: Jaeger GitHub Discussions

OpenSearch

OpenSearch provides search and analytics capabilities for observability data, including application logs. Structured fields make it easier to filter records by service, environment, timestamp, or trace ID. For example, a log record might contain:

{
"service.name": "payment-service",

"severity": "ERROR",

"message": "Payment provider timed out",

"trace_id": "example-trace-id"

}

Fields such as service.name and trace_id can help you narrow a search and connect a log entry to a related request. That connection depends on your instrumentation, log format, ingestion configuration, and the way you query the data.

OpenSearch provides flexible search and analytics, but you need to configure ingestion, indexing, storage, and retention. For example, OpenSearch Data Prepper can process telemetry before sending it to OpenSearch.

Resources

Documentation: OpenSearch observability documentation
Community: Connect with the OpenSearch community, including its public Slack workspace.
Discussions: OpenSearch community forum

Perses

Perses is an open-source dashboarding tool with an open dashboard specification and support for managing dashboards as code.

Perses doesn't replace the systems that store telemetry. It queries configured data sources and displays their results. Its plugins determine which backends it can query. This separation lets teams manage dashboards independently of telemetry storage. However, dashboard definitions still depend on compatible data sources, queries, and configuration.

For example, a team might configure Prometheus as a Perses data source to display request rates, error rates, and latency measurements. Other data sources depend on the available plugins and configuration.

Resources

Documentation: Perses documentation
Community: Perses contact and community channels, including the #perses-dev channel on CNCF Slack.
Discussions: Perses GitHub Discussions

How do these tools fit together?

An observability stack isn't one application. Installing these components doesn't automatically create a working observability stack. You must decide how data reaches each backend, which components process it, and how engineers access the results.

The following diagram illustrates one possible architecture. It distinguishes the Prometheus scrape process from the trace and log ingestion paths.

This architecture has three important parts.

Ingestion: Choose how each backend receives data. The Collector supports multiple receivers and exporters, but not every backend uses the same ingestion method.
Storage and analysis: Decide which systems handle each data set. You might store metrics in Prometheus, configure a storage backend for Jaeger, and store logs in OpenSearch.
- Routing data to multiple destinations is also possible when your requirements justify the additional processing and storage.
Visualization: Decide where engineers will explore the data. Jaeger and OpenSearch provide their own interfaces. Perses supports reusable dashboards, but you must configure compatible data sources.

OpenTelemetry provides a foundation for telemetry transport. It doesn't automatically configure every backend, unify their interfaces, or correlate all the data they contain. Correlation also depends on consistent attributes, trace context, and compatible ingestion and query capabilities.

What happens when an incident occurs?

Consider a production service whose response latency suddenly increases. An engineer might follow this workflow:

Metrics help you detect and scope the problem. Traces help you locate the delay, and logs provide events that may explain it. After applying a fix, check the relevant telemetry to verify that the system has recovered.

Start with one use case

You don't need to deploy every tool at once. Start with one observability question and build the smallest setup that can help you answer it.

Define the question. For example, determine whether a service's request latency is increasing.
Choose a backend. Select a tool that supports the type of data you need to investigate.
Send test telemetry. Configure the required ingestion path and generate a small amount of test data.
Verify the results. Query the data and confirm that the expected measurements and attributes appear.
Document what you learned. Record the configuration, useful queries, and any limitations you encountered.

For example, you could start with Prometheus and query request latency for one service. Once you need to investigate individual requests, add tracing. If you also need to search detailed application events, introduce a log backend.

This approach helps you validate each component before expanding the stack. It also makes integration problems easier to identify while the system is still small.

What should you consider before choosing your stack?

A successful experiment doesn't guarantee that a stack is ready for production. Before deploying it across services and environments, evaluate the integration work and operational responsibilities.

Integration and query models

Prometheus uses PromQL. OpenSearch supports its own query and analytics capabilities. Jaeger provides an interface for exploring traces, while Perses uses configured data sources and their respective query languages.

Engineers may need to switch between tools to investigate the same incident. You also need to configure data sources, align resource attributes, and preserve trace context in logs when you want to connect related events.

Reliability, storage, and security

Each deployed component introduces operational requirements. Plan for:

Storage and retention: Estimate how data volume will grow and define retention policies for each signal.
Availability: Decide how ingestion and investigation should behave when a backend becomes unavailable.
Scaling: Determine how to handle increasing telemetry volume and query load.
Security: Control access to telemetry and protect sensitive information.
Maintenance: Manage upgrades, configuration changes, backups, and recovery.

These concerns become more important as the number of services, environments, and engineering teams grows.

Licensing and long-term flexibility

Open-source projects provide different levels of freedom depending on their licenses. Some products follow an open-core model, offering an open-source base while reserving some capabilities for commercial editions.

Before adopting a tool, review its license, governance, community activity, commercial feature boundaries, and data export options. Consider how difficult it would be to replace the component if your requirements change.

The goal isn't to choose open-source software at any cost. It's to make an informed decision about flexibility, operational effort, and long-term sustainability.

Self-managed stack or unified observability platform?

You can assemble an observability stack from individual open-source tools or use a platform that brings telemetry together in one place. The right choice depends on your team's requirements, desired control, and capacity to operate the infrastructure.

Consideration	Self-managed stack	Unified observability platform
Control	Choose and configure individual components.	Use the capabilities and configuration options provided by the platform.
Integration	Your team connects and maintains the components.	The platform provides an integrated experience for supported signals.
Operations	Your team manages the infrastructure, storage, upgrades, and maintenance.	The platform can reduce the work required to operate separate backends.
Flexibility	Customize components to meet specific requirements.	Evaluate supported integrations, export options, and platform limitations.
Best fit	Teams that need direct control and have the capacity to maintain the stack.	Teams that prioritize a unified investigation experience and reduced operational work.

Note: Neither approach is inherently better. A self-managed stack gives you control over individual components, while a unified platform can reduce integration and maintenance work, which allows you to focus on your business logic.

Dash0: Explore telemetry in one platform

Dash0 is an OpenTelemetry-native observability platform that brings metrics, logs, traces, and resource information together for analysis and investigation.

For example, when a service's latency increases, you can examine metrics to understand the impact, inspect traces to identify slow operations, and investigate related logs to find potential causes within the same platform.

Dash0 supports PromQL and uses the Perses dashboard specification. These open standards help teams reuse familiar query capabilities and dashboard workflows. Dash0 isn't a connector that automatically combines an existing Prometheus, Jaeger, OpenSearch, and Perses deployment. It's an alternative approach that can reduce the need to operate these components separately, depending on your requirements.

Choose a self-managed stack when you need control over individual components and can maintain the infrastructure. Consider Dash0 when you want a unified place to investigate telemetry with less operational overhead.

Learn more in the Dash0 documentation.

Conclusion

An open-source observability stack is an architecture you design, not a set of tools that automatically work together.

Prometheus, Jaeger, OpenSearch, and Perses address different needs. OpenTelemetry provides a foundation for instrumentation and telemetry transport, but you must configure the data paths, storage, integrations, and investigation workflows.

You can assemble and maintain those components yourself, or evaluate a platform such as Dash0 to bring telemetry into a common observability experience. The best choice is the one that meets your requirements without introducing more operational complexity than your team can manage.

Your telemetry doesn't go straight to the backend: Understanding the OpenTelemetry Collector

hridyesh bisht — Sun, 01 Nov 2026 06:11:34 +0000

Imagine you run a coffee shop with an online ordering system. A customer orders a cappuccino, completes the payment, and waits for a pickup notification. But the notification never arrives.

Your application generates telemetry as it processes the order. Traces capture the request's journey, logs record events, and metrics show how the system behaves. But generating telemetry is only part of the problem. You also need to decide where it goes, what information it contains, and how to deliver it reliably so you can investigate issues.

The OpenTelemetry Collector helps you manage these tasks outside your application code. It receives, processes, and exports telemetry, giving you a dedicated layer for controlling your telemetry pipeline.

In Understanding Observability with OpenTelemetry and Coffee, I introduced OpenTelemetry's fundamentals through a coffee-shop example. In Native OpenTelemetry, I explored why visibility into telemetry processing matters. I also discussed the risks of sharing telemetry with AI tools in Before you paste telemetry data into an AI agent.

This article builds on those ideas to explain how the Collector manages telemetry, why you might need one, and what to consider when operating it in production.

Do you need a Collector?

In our coffee shop. When a customer places an order, the application checks ingredient availability, records the order, and contacts the payment service. OpenTelemetry instrumentation can generate telemetry for these operations.

You can export that telemetry directly to an observability backend or route it through a Collector.

However, imagine that your coffee shop expands to multiple locations. You now have several services running in different environments, and you want consistent metadata, filtering rules, and export configurations across all of them. Managing these responsibilities independently in every application can become difficult.

A Collector is particularly useful when you need to:

Apply consistent processing rules across multiple services.
Enrich telemetry with additional context.
Filter data, batch exports, or control telemetry volume.
Route telemetry to multiple destinations.
Manage telemetry delivery independently of application code.

How does the Collector process telemetry?

The Collector organizes telemetry processing into pipelines. A typical pipeline uses three primary component types: receivers, processors, and exporters.

Each component has a specific responsibility.

Receivers: Accept telemetry

Receivers accept telemetry from applications and other sources.

For example, the coffee shop's ordering system can use the OpenTelemetry SDK to export traces, metrics, and logs through the OpenTelemetry Protocol (OTLP). The Collector's OTLP receiver accepts this data and passes it to the appropriate pipeline.

The Collector also supports other protocols and data sources, depending on the components available in your deployment.

Processors: Process telemetry

Processors transform, enrich, filter, or otherwise process telemetry before export.

Suppose the coffee shop has separate staging and production environments. You want to identify the environment that generated each measurement so engineers can distinguish test activity from production activity.

A resource processor can add an environment attribute. A batch processor groups telemetry before export, while a memory limiter helps protect the Collector from excessive memory use.

Other processors can filter telemetry or modify attributes. Processor order matters.

Each processor receives the output of the preceding processor. To activate a processor, you must include it in the relevant pipeline. Adjust the order to suit your pipeline. For example, if a filter depends on an attribute added by a resource processor, make that attribute available before the filter runs.

Exporters: Send telemetry to a destination

Exporters send processed telemetry to backends or other destinations.

For example, you can export traces to Jaeger over OTLP, expose metrics through an endpoint that Prometheus scrapes, or use a supported Prometheus Remote Write exporter with a compatible destination. You can also send logs to a compatible logging backend, such as OpenSearch, when the required integration is available.

Alternatively, you can send telemetry to an integrated observability platform such as Dash0.

The exact configuration depends on the Collector distribution and destination. This approach lets you manage downstream export configuration separately from your application's instrumentation.

Where should you deploy the Collector?

You can deploy the Collector near your applications, run it as a shared gateway, or combine both approaches.

Two common deployment patterns are the agent and gateway patterns.

Agent pattern

In the agent pattern, a Collector runs close to the application or host that generates telemetry. On Kubernetes, you can deploy it as a DaemonSet or sidecar, depending on your requirements.

For the coffee shop, an agent could collect telemetry from the ordering service or its underlying infrastructure. This provides a local collection point for application and infrastructure telemetry.

Gateway pattern

In the gateway pattern, applications or local Collectors send telemetry to a shared Collector endpoint. The gateway processes the telemetry and exports it to the backend.

Imagine that the coffee shop operates several branches, each with its own services and infrastructure. A shared gateway provides a central place to manage processing rules, credentials, and export traffic.

You can combine the two patterns. Agents collect telemetry near workloads, while gateway Collectors handle shared processing and export policies. Neither pattern is always better. The decision depends on your deployment architecture, scale, security boundaries, processing requirements, and operational capacity.

A gateway centralizes control but also concentrates traffic. You need to provide sufficient capacity and configure appropriate availability controls.

What happens to telemetry inside the pipeline?

The Collector can enrich and filter telemetry before exporting it. However, these decisions can affect the information available during an investigation.

The following YAML examples are configuration fragments, not complete Collector configurations. Define each processor and include it in the relevant pipeline under service.pipelines to apply it. Validate the configuration against the Collector distribution and version you use.

Add context to telemetry

Suppose the coffee shop has separate staging and production environments. You want to identify the environment associated with each service's telemetry.

This approach helps standardize metadata at the collection layer. However, application-specific details, such as why the system applied a particular discount, might still require instrumentation in the application.

Filter sensitive information

Suppose the ordering system includes a customer's email address in a span attribute. The coffee shop doesn't want to send this information to its observability backend.

This removes the specified attribute from the telemetry the processor handles. It doesn't automatically remove the same information if it appears in a log body, an event, or another attribute.

In Before you paste telemetry data into an AI agent, I explored the risks of exposing sensitive information and untrusted content when using AI tools to investigate production issues.

Manage telemetry volume

A busy coffee shop might process hundreds of orders during peak hours. Its ordering system can generate substantial telemetry, especially when requests pass through multiple services.

Batching, filtering, and sampling can help manage the volume of data sent to downstream systems, but they serve different purposes:

Batching groups telemetry before export.
Filtering removes selected telemetry records or attributes.
Trace sampling determines which traces or spans to retain, depending on the sampling strategy.

Sampling can occur in the SDK or Collector, depending on your configuration and requirements.

Reducing telemetry volume involves trade-offs. Filtering might remove useful attributes, while sampling might exclude a trace that an engineer needs to investigate. The goal is to retain useful diagnostic information while managing resource usage and cost.

In Native OpenTelemetry we focus on understanding how telemetry is transformed or discarded helps engineers make informed decisions about data quality.

What happens when the backend becomes unavailable?

Suppose a customer places an order, but the observability backend becomes unavailable. The ordering service continues generating telemetry, and the Collector continues receiving it. Depending on its configuration, the Collector can queue outgoing telemetry and retry failed exports. This can help the pipeline recover when the destination becomes available again.

However, retries don't guarantee delivery. If the backend remains unavailable and the queue reaches capacity, the Collector can drop telemetry. In-memory queues can also lose pending data if the Collector restarts.

When supported and configured, persistent storage can preserve queued telemetry across restarts. However, it doesn't prevent every type of data loss.

Before deploying a Collector in production, consider how much telemetry you need to buffer, how long to retry failed exports, and whether persistent storage is necessary. Monitor queue capacity and export failures so you can detect problems before they affect telemetry delivery.

Monitor the Collector itself

Imagine that the coffee shop's ordering service continues sending telemetry, but the Collector can't export it because the backend is unavailable. How would you know that the pipeline is falling behind?

The Collector exposes internal telemetry, including metrics for accepted and refused data, exporter failures, queue sizes, and resource usage. For example, a growing exporter queue might indicate that the backend can't process incoming data quickly enough. An increase in export failures might indicate connectivity or configuration problems.

Monitor these metrics to identify issues with the Collector and investigate problems in the telemetry pipeline. Monitor both the applications that generate telemetry and the infrastructure that collects and exports it.

Where does Dash0 fit?

Dash0 is an observability platform that accepts OpenTelemetry data. You can configure the Collector to process telemetry before exporting it to Dash0. This keeps your application's instrumentation separate from your backend configuration.

In our coffee shop example, the ordering system continues to generate telemetry through OpenTelemetry, while the Collector manages how that data is processed and delivered.

Alternatively, you can build a self-managed, open-source observability stack using tools such as Prometheus, Jaeger, and OpenSearch. This approach offers flexibility and control, but your team must maintain the individual components and their integrations.

An integrated observability platform can reduce the effort required to bring telemetry together and investigate issues. Choose the approach that best fits your organization's requirements, resources, and operational needs.

For setup instructions, refer to the Dash0 OpenTelemetry Collector integration guide.

Conclusion: Treat telemetry as a pipeline, not an afterthought

OpenTelemetry standardizes how applications generate and export telemetry. The Collector provides a separate layer for managing how that telemetry is processed and delivered.

You can use the Collector to enrich data, batch exports, filter records, manage data volume, and route telemetry to one or more destinations. These capabilities become increasingly valuable as your architecture grows and observability becomes a shared responsibility across engineering teams.

However, introducing a Collector also means operating another component. You must configure its pipelines, secure its endpoints, allocate sufficient resources, handle failures, and monitor its health.

You don't always need a Collector. Introduce one when you need greater control over your telemetry pipeline. The goal is to make telemetry collection and delivery consistent, manageable, and adaptable as your systems grow.

Serverless Computing in Kubernetes: A Developer’s Guide

hridyesh bisht — Sun, 10 Aug 2025 10:40:46 +0000

Serverless computing allows you to focus on writing business logic without managing infrastructure. While often confused with Functions as a Service (FaaS), serverless is broader. It includes event-driven execution, auto-scaling, stateless workloads, and billing based on usage rather than uptime.

This guide explores serverless from a developer's point of view using a coffee shop application deployed on Kubernetes. You also learn about cover setup, advanced use cases, observability, deployment strategies, and production readiness.

For this blog, consider a coffee shop app with the following services:

order-service (handles orders)
payment-service (processes payments)
inventory-service (manages beans and stock)

Combine Serverless and Kubernetes?

Kubernetes is designed to run containers continuously. This works well for workloads that must always be available. However, many workloads are event-driven and only need to run when triggered.

OpenFaaS extends Kubernetes to run workloads only when needed, scaling them down to zero when idle. This approach saves costs, improves resource efficiency, and accelerates development.

OpenFaaS does not replace Kubernetes, it enhances it with:

Scale to zero: No pods running = zero resource cost during inactivity.
Function templates: Developers write the logic; OpenFaaS handles packaging, networking, scaling, and observability.
Standard Kubernetes integration: Works with any Kubernetes distribution; no special infrastructure required.
Event-driven triggers: Supports HTTP, Kafka, MQTT, cron, and more.
Built-in monitoring: Integrates with Prometheus and Grafana out of the box.

For example, in the coffee shop app, the payment-service is only used during checkout. Running it all day wastes resources. With OpenFaaS:

The payment-service function spins up only when a customer checks out.
During busy hours (e.g., morning rush), it automatically scales to handle high demand.
After hours, it scales down to zero, using no resources.

Serverless in Kubernetes

Kubernetes is not inherently serverless, but open-source projects like OpenFaaS bring serverless capabilities to it. These platforms provide abstraction layers over Kubernetes primitives like Pods, Services, and Deployments.

To run a serverless function on Kubernetes, the following components are typically required:

A container image containing your function or application
A container registry to store the image
A Pod to run the container
A Service or Ingress to expose it
An autoscaler (e.g., HPA, KEDA) to handle scale
ConfigMaps and Secrets to store configuration and credentials

Note: OpenFaas supports Serverless 2.0 out of the box.

OpenFaaS

OpenFaaS enables developers to run functions and microservices on Kubernetes using Rancher or containerd. It supports:

Build templates for languages like Python, Go, Node.js
Scale to zero using Prometheus or Kubernetes HPA v2
Event triggers: HTTP, cron, Kafka, SQS, MQTT
CLI and web UI for deployment and monitoring
Secrets management
OpenFaaS Cloud for CI/CD and team-based management

Note: OpenFaaS also offers faasd, a minimal single-node alternative to Kubernetes using containerd and CNI.

For the coffee-shop app, in a real coffee shop:

An order is placed by the user
order-service validates and sends the request to inventory-service
inventory-service checks stock and updates it
payment-service processes the transaction

You could write a function to check inventory:

def handle(req):

    order \= json.loads(req)

    if order\['quantity'\] \> inventory.get\_stock(order\['item'\]):

        return "Out of stock"

    inventory.decrement(order\['item'\], order\['quantity'\])

    return "Order accepted"

And trigger it through HTTP or MQTT when new orders arrive.

Architecture of OpenFaaS

OpenFaaS is composed of the following layers:

Gateway: Handles all incoming requests
Watchdog: Converts HTTP to stdin/stdout and back
Function containers: Stateless business logic
Autoscaler: Monitors metrics and adjusts replicas
Connector SDK: Connects external events (e.g., MQTT, Kafka)
Prometheus: Collects metrics for observability
faas-cli: CLI tool for development and CI/CD

The function deployment flow with OpenFaas is:

Write your function using a template (e.g., Python, Go, Node.js).
1. Write handler logic for your function.
Package it as a container image.
Deploy to Kubernetes using faas-cli.
Invoke it through HTTP, MQTT, cron, or a message queue.
Prometheus gathers metrics; logs go to stdout.
Scale automatically based on usage or metrics.

Interacting with OpenFaaS

You can manage OpenFaaS functions in three ways:

faas-cli (recommended for scripting and CI)
Web UI (good for demos or quick insights)
REST API (custom app integration)

OpenFaaS supports various trigger mechanisms:

HTTP (default)
MQTT (great for IoT devices)
Apache Kafka
cron (time-based)
AWS SQS
MinIO
RabbitMQ

Most of these use the connector-sdk, allowing custom event bridges.

Accessing the OpenFaaS Gateway

After installing OpenFaaS (using Helm or arkade), you can access the Gateway. An HTTP API and UI that manages all deployed functions.

Forward the OpenFaaS Gateway to your local machine

kubectl rollout status \-n openfaas deploy/gateway

kubectl port-forward \-n openfaas svc/gateway 8080:8080 &

This exposes the Gateway at http://127.0.0.1:8080.

Note: If the port becomes unavailable later, rerun the port-forward command.

Some of the key features are:

Deploy New Function: From the store or using custom Rancher images
Invoke Functions: Test your functions manually with input data
Monitor Logs and Metrics: Includes basic Prometheus metrics and live logs
Manage Deployments: Delete or update existing functions

OpenFaaS CLI

The faas-cli is the primary developer interface for building, deploying, and managing OpenFaaS functions. It communicates directly with the Gateway.

Use faas-cli --help to learn about available options for each command. You can also find help for some of the commands in the OpenFaaS documentation.

For example, each store tracks daily espresso counts. An OpenFaaS function reads an MQTT message, then pushes usage stats to a central dashboard.

def handle(req):

    count \= int(req)

    if count \> 100:

        return "Daily threshold reached\!"

    return "Usage normal"

Function and Template Stores

OpenFaaS simplifies function development with two built-in stores:

Function Store

The Function Store is a curated catalog of ready-to-deploy serverless functions. These functions follow reusable patterns such as:

Image conversion
Sentiment analysis
Slack notifications
PDF generation

For example, in a coffee ordering app. You can search for functions that relate to coffee logic, like coffee-order, inspect their behavior, and deploy instantly. A deployed function could receive order data (e.g., drink type, size, customer name) and respond with a formatted confirmation receipt.

Template Store

The Template Store provides scaffolding to build your own functions using supported languages and frameworks (e.g., Python, Flask, Node.js, Go).

Templates handle:

HTTP input and response setup
Boilerplate build and deploy logic
Language-specific structure

For example, you could scaffold a payment-service function using a Python template and extend it to:

Parse JSON order data
Validate payment information
Return a payment confirmation status

Templates are extensible, you can add packages like jinja2 for HTML rendering or numpy for calculations by modifying the template's dependency file.

Observability: Prometheus + Grafana

OpenFaaS integrates with Prometheus by default to enable real-time observability.

For example, you can track:

Number of coffee orders processed
Payment success vs. failure rate
Low-stock alerts for ingredients

To access Prometheus (hidden by default for security), use port-forwarding:

kubectl \-n openfaas port-forward deployment/prometheus 9090:9090 &

Each function automatically exposes a /metrics endpoint. Prometheus scrapes this and Grafana can visualize metrics on dashboards.

Create Your First Function

OpenFaaS offers templates that scaffold functions, handling HTTP entry, code wiring, and build scripts automatically.

You can source templates from:

OpenFaaS official repo
OpenFaaS incubator or community stores
Custom template repos

To build a function from scratch, you’ll:

Choose a template (e.g., python3-flask-debian)
Generate your function using the CLI
Edit logic and dependencies
Build, push, and deploy to OpenFaaS

Templates are pulled from the Template Store. You can use community-curated templates or your own custom versions. Each function includes:

lang: the template type
handler: the path to your business logic
image: the container image to publish

Once set up, the CLI can build, push, and deploy your function in a single command. This creates a Kubernetes deployment behind the scenes, ready to accept HTTP requests.

You can run each step individually or use faas-cli up:

faas-cli up -f order-service.yml

This does the following:

builds the container locally via Rancher.
pushes the image to your registry.
deploys via OpenFaaS API → Kubernetes → Pod.

Templating with Jinja2

For functions that return HTML, Jinja2 can render dynamic content using variables. In the coffee app, a receipt template could include placeholders for:

Customer name
Coffee type
Timestamp

This improves user-facing responses without hardcoding the output.

Note: To include large or compiled packages (e.g., NumPy or Flask), use a Debian-based template like python3-debian. These templates support native compilation and pip installs that Alpine-based templates might not.

Controlling HTTP Responses in OpenFaaS

When you need precise control over status codes, headers, and response types (like JSON or binary), OpenFaaS offers flexible templates, especially python3-flask and python3-http. These allow you to build rich APIs with familiar HTTP semantics.

For example, consider the payment-service in a coffee shop. It needs to return:

A 201 Created status for successful payments
A 400 Bad Request for invalid inputs
Custom headers with order IDs and trace IDs
Structured JSON for frontend integration

These templates allow you to define all of the above without additional tools.

Serving Static Sites and Microservices

With HTTP-based templates, you can also serve static content or build lightweight services.

For example, a function in Coffee Shop menu could serve:

menu.html for your store’s website.
Promotional flyers as PDFs.
Static assets such as HTML, CSS, or JSON.

Functions with Secrets

To protect sensitive operations like payment validation or admin APIs, OpenFaaS supports secret management.

You can integrate common HTTP API authentication methods:

API Token in Header: A shared API key is sent in the request header.
HMAC (Hash-based Message Authentication Code): Used by providers like GitHub, PayPal, and Stripe to sign payloads with a shared key.
OAuth2: Delegates authentication to a third-party identity provider.

For example, your payment-service might require an API key passed as a header. The function reads the key from a mounted secret and compares it with the request input. This ensures only trusted clients can access sensitive endpoints such as payment processing or refunds.

Asynchronous Invocations

High-traffic periods, such as the morning coffee rush, can cause latency spikes. OpenFaaS supports asynchronous function calls to mitigate this.

For example, when rendering large receipt PDFs or syncing inventory with external systems, your function can be invoked asynchronously.

Async calls return an immediate acknowledgment while processing jobs in the background. You can optionally send results to a callback endpoint.

Autoscaling Functions

OpenFaaS supports both horizontal scaling and scale-to-zero based on real-time demand.

The minimum (initial) and maximum replica count can be set at deployment time by adding a label to the function.

com.openfaas.scale.min: by default, this is set to 1, which is also the lowest value and unrelated to scale-to-zero.
com.openfaas.scale.max: the current default value is 20 for 20 replicas.
com.openfaas.scale.factor: by default, this is set to 20% and has to be a value between 0-100 (including borders).

For example, if you want a function to have at least 5 replicas at all times, but to scale up to 15 when under load, set it as follows in your stack.yml file:

labels:

  com.openfaas.scale.min: 5

  com.openfaas.scale.max: 15

Horizontal Scaling

You can configure functions with:

Minimum replicas (for readiness)
Maximum replicas (to conserve resources)
Scale factor to control how fast functions scale out

For example, during peak morning hours, the coffee shop scales order-service from 2 to 10 replicas to meet demand. In off-peak hours, the function scales back down.

Scale-to-Zero and Cold Starts

You can enable cold starts by setting minimum replicas to zero. This reduces idle costs for functions like inventory-audit that run infrequently.

Kubernetes is also called “eventually consistent” and requires some tuning to get the cold-start. Cold starts in Kubernetes can take 1–2 seconds without tuning. Keep 1–5 replicas to avoid delays or use asynchronous calls to hide scaling latency.

TLS and Production Readiness

TLS is optional for local testing because kubectl port-forward already provides an encrypted tunnel. For production:

Install Ingress with TLS.
Use cert-manager for certificate management.
Route traffic over HTTPS.

Once set up, you can log in with the CLI using a secure gateway.

Advanced Use Cases

Custom HTTP Responses

Using templates like python3-http or python3-flask, you can control:

HTTP status codes (e.g., 201 Created, 500 Internal Server Error)
Custom headers (e.g., Content-Type)
JSON-formatted responses for frontend apps

For example, your function could return {"error": "Insufficient balance"} with a 402 Payment Required code.

Binary Data Handling

To support raw byte input/output (e.g., uploading a receipt image), enable RAW_BODY: True in the function’s environment.

For example, in a coffee shop's self-ordering kiosk, a function could:

Receive a JPEG from a camera
Convert it to grayscale
Return the processed image as a binary payload

Serving Static Pages

You can serve a micro-site using the python3-http template.

For example, a function named homepage could return static HTML pages like /about.html or /menu.html.

Combining OpenFaaS and MQTT for Edge Use Cases

MQTT (Message Queuing Telemetry Transport) is a lightweight, pub-sub messaging protocol designed for unreliable or constrained networks. It’s ideal for edge use cases like IoT and retail.

Some benefits of integrating OpenFaas and MQTT are:

Low bandwidth and power usage
Decouples producers and consumers
Buffers messages locally when offline
Reliable delivery once reconnected

In edge computing scenarios, OpenFaaS and MQTT work together:

MQTT brokers handle sensor data (e.g., temperature, order count).
OpenFaaS functions are triggered by these MQTT events.
Responses are logged, alerts are triggered, or orders are adjusted.

Note: For more information, refer to https://programmerprodigy.code.blog/2025/07/09/microservices-at-edge-with-k3s-and-fleet/

Balancing Containers and OpenFaaS

Choosing between a traditional cloud native app and a serverless approach with OpenFaaS is not not an "either-or" choice. The most effective cloud-native solutions often combine both to balance their strengths.

In a coffee shop app, Kubernetes container workloads are ideal for services that must always be available.

The core order-service runs continuously to ensure customers can place orders anytime.
For event-driven or infrequently used workloads, such as payment-service and inventory-service, OpenFaaS offers a more efficient, cost-effective option. It can scale these services to zero when idle, reducing unnecessary resource use.

A hybrid approach delivers the best of both worlds:

Optimize costs by running resource-intensive services only when needed.
Improve resource efficiency by reducing idle workloads.
Accelerate development by breaking down logic into small, manageable functions.
Scale intelligently to handle unpredictable traffic spikes without over-provisioning.

The key is to use the right tool for each job. Kubernetes provides persistence and control for always-on workloads, while OpenFaaS adds event-driven, scalable, and cost-efficient capabilities. Together, they enable a resilient, adaptable, and optimized cloud-native architecture.

Microservices at Edge with K3s and Fleet

hridyesh bisht — Thu, 10 Jul 2025 14:29:07 +0000

Imagine you're building a coffee shop application with three core microservices:

order-service: Handles customer orders.
payment-service: Processes transactions.
inventory-service: Tracks beans, milk, cups, and other inventory.

Now scale this across hundreds or thousands of physical store locations—each with its own on-site device like a Raspberry Pi or Intel NUC. How do you maintain consistency, ensure reliability, and keep deployments secure across all sites?

That’s where K3s (a lightweight Kubernetes distribution for edge) and Rancher Fleet(GitOps engine) are designed to manage thousands of clusters to help you.

In this guide, you learn how to build a resilient, scalable edge infrastructure using:

K3s to run Kubernetes clusters on resource-constrained devices,
Fleet to manage deployments across hundreds or thousands of clusters,
OpenFaaS and MQTT for lightweight event-driven automation and telemetry,
And strategies for handling air-gapped stores, remote access, logging, and storage.

What Is a Container Image?

Before diving into Kubernetes or K3s, it’s essential to understand container image

A container image is a lightweight, portable unit that includes everything needed to run a service that includes:

Application code
Runtime (like Python, Java, or Node.js)
System libraries and binaries
Configuration and dependencies

For example, in the coffee shop app each service: order-service, payment-service, and inventory-service is built as a container image:

order-service: Python + Flask + order logic
payment-service: Node.js + Stripe SDK
inventory-service: Go + SQLite + inventory tracker

Kubernetes (K8s) is an open-source system that automates the deployment, scaling, and management of containerized applications. It provides a robust, extensible platform for orchestrating containers across clusters of machines, simplifying the management of distributed, cloud-native systems.

Note: For more information, refer to Introduction to Container Images and Orchestration.

Understanding Edge Computing

Edge computing refers to placing compute resources as close as possible to the data source or end-user. In our case, this means deploying services directly into each coffee shop rather than relying on centralized cloud data centers.

Benefits of Edge Computing:

Real-time inventory decisions: Milk running low, auto order more.
Low-latency UX: Instant response at self-service kiosks.
Data sovereignty: Payment data stays local to comply with regulations.

Challenges at the Edge: While beneficial, edge environments present unique challenges:

Resource constraints: Devices at the edge, such as those in your coffee shops, often have lower compute and memory capacity.
Intermittent connectivity: Edge devices may not maintain a constant 24/7 internet connection.
Remote management difficulty: You cannot manually SSH into hundreds of individual store servers to update software.

K3s: Kubernetes Optimized for the Edge

K3s is a lightweight, CNCF-certified Kubernetes distribution built specifically for edge and IoT environments. Developed by Rancher, K3s simplifies cluster setup while remaining fully compatible with Kubernetes tooling and APIs. It’s ideal for resource-constrained devices—such as Raspberry Pis or Intel NUCs deployed in coffee shops or retail branches.

Key K3s Features for Edge Deployments:

Single binary distribution (~100MB) for fast installs and minimal overhead.
Embedded components: containerd (container runtime), runc, and kubectl.
Supports multiple storage backends: SQLite (via Kine), embedded etcd, and external SQL (MySQL/PostgreSQL).
Works on ARM and x86_64 platforms.
Pre-configured defaults: Includes Flannel for networking, Traefik for ingress, and metrics-server for observability.

How K3s Optimizes for the Edge

Unified Binary: Embeds everything needed—container runtime, CRI tools, control plane in one downloadable file. Makes updates easy via binary replacement.
Kine (SQLite integration): Reduces overhead by emulating etcd with SQLite, ideal for single-node clusters without consensus requirements.
Minimal System Requirements: Can run in 512MB RAM, uses minimal CPU.

Comparing K3s and Kubernetes

Capability	K3s	Upstream Kubernetes
Installation	Single binary, HTTP tunnel	kubeadm, multiple binaries
Control plane	Embedded etcd or SQLite	etcd only
Runtime	Embedded containerd + runc	External container runtime
System requirements	~500MB RAM, low CPU	1GB+ RAM, higher CPU
HA options	SQLite, etcd, SQL	etcd only
Targets	Edge, IoT, hobbyists	Data centers, cloud

There are several personas or users of Kubernetes. Refer how K3s may affect each of them:

Developers: Write standard manifests. No changes needed.
Platform Engineers: Use pre-bundled defaults to reduce setup time.
SREs/SecOps: Need to understand K3s-specific HA, upgrades, and bootstrap.

K3s renames Kubernetes control plane nodes to "servers" and worker nodes to "agents". It uses an HTTP tunnel for simplified server-agent communication, Flannel for networking by default, and Traefik for ingress.

Server: Control plane node
Agent: Worker node

K3s Architecture

K3s is purpose-built for edge and resource-constrained environments. Unlike upstream Kubernetes, which requires multiple binaries and external dependencies, K3s is distributed as a single binary. Internally, K3s embeds several core components:

containerd and runc: Used as the default container runtime.
Embedded SQLite or etcd: For storing Kubernetes cluster state. SQLite is ideal for single-node clusters; etcd enables high availability.
Built-in defaults: Includes Flannel (CNI), Traefik (Ingress), metrics-server, and Helm controller.

K3s uses a simplified bootstrapping process where agents (workers) connect to the control plane (server) via an encrypted tunnel (k3s-agent uses reverse tunnels).

Note: In smaller stores, a single-node K3s cluster can run all three services—order, payment, and inventory—on one device.

High Availability (HA) with K3s

If you want your cluster to tolerate failures (e.g., a power outage at one device), K3s supports HA using embedded etcd. This mode is recommended for clusters that run critical systems and must remain available even during node failures.

For resilience, K3s supports two HA modes:

Embedded etcd (Raft consensus): Suitable for production. Requires 3 or 5 server nodes.
External SQL DB (PostgreSQL/MySQL): Lightweight but harder to scale.

What's up with k3sup?

k3sup (pronounced "ketchup") is a community tool that simplifies K3s installation and joining agents to a cluster over SSH, making remote setup trivial. k3sup can also update your kubectl configuration file.

Why Use k3sup?

Remote installation via SSH: great for air-gapped or low-access environments.
One-line cluster provisioning:ideal for bootstrapping multiple stores quickly.
kubectl auto-merge: automatically adds new K3s clusters to your KUBECONFIG file.

Fleet: Centralized Multi-Cluster Management

Fleet is Rancher’s GitOps controller purpose-built to manage thousands of Kubernetes clusters—ideal for widespread edge deployments like your coffee shop application. While K3s runs a lightweight cluster at each store, Fleet offers centralized visibility, version control, and deployment across all of them.

Why Use Fleet with K3s?

Central control, local autonomy: You manage manifests, Helm charts, and configurations from a single Git repository. Each K3s store fetches its configuration independently.
Designed for scale: Fleet supports up to 1 million clusters and uses lightweight agents with minimal overhead.
Pull-based sync: Suitable for remote stores with dynamic IPs or unreliable connectivity.

How Fleet Works?

Git Repository as Source of Truth You store application definitions (YAML, Helm, Kustomize, or OCI bundles) in a Git repo. These include:
- fleet.yaml for configuration
- Manifests for each microservice (order.yaml, payment.yaml, inventory.yaml)
Fleet Manager (Controller) The Fleet controller runs in a central management cluster (on-prem or in the cloud). It watches Git repositories and generates bundles for deployment.
Fleet Agents in Edge Clusters Each K3s cluster runs a Fleet agent. This agent:

Connects back to the Fleet manager securely.
Pulls the correct workload bundle.
Applies the manifests locally.
Reports status.

Coffee Shop Deployment: fleet.yaml

defaultNamespace: coffee-shop  
  targets:  
  \- name: edge-shops  
    clusterSelector:  
      matchLabels:  
        location: edge  
    yaml:  
      \- path: manifests/order.yaml  
      \- path: manifests/payment.yaml  
      \- path: manifests/inventory.yaml

Benefits of Using Fleet in Edge Deployments

Resilient by Design: If a cluster goes offline, Fleet retries syncing when it's back online. No manual recovery needed.
Declarative GitOps Workflow: All changes are Git-driven and reproducible. Every store gets the same, tested configuration.
Security & Separation: Since clusters pull workloads from Fleet, they don't need inbound access or public IPs.

Remote Access

Because edge locations (coffee shops) often lack:

Static IPs
VPNs
SSH access

Fleet enables secure pull-based management, where the cluster reaches out to Fleet rather than requiring a central controller to initiate communication.

Alternatives and Tradeoffs

Method	Pros	Cons
Fleet (GitOps)	Secure, scalable, automatic retries	Requires pre-registration
VPN	Secure tunneling	Complex to set up at scale
SSH	Quick setup, compatible with k3sup	Hard to scale, brittle
Inlets	Easy tunneling of HTTP/TCP	Requires cloud relay or tunnel server

When managing 1,000+ coffee shops, you can't SSH into each branch to make changes. Most shops lack static IPs or stable internet. Instead of managing clusters by pushing changes from a central control plane, Fleet agents run inside each K3s cluster and initiate outbound connections to the central Fleet controller. This helps you:

Avoid the need for public or routable IP addresses.
Work over common outbound ports (e.g., HTTPS).
Enable auto-recovery when clusters reboot or reconnect

For example, in coffee shop app:

A barista at "Store 042" restarts their Raspberry Pi.
It boots K3s, pulls the latest manifests from Fleet via Git.
Fleet ensures order-service, payment-service, and inventory-service run correctly without manual intervention.

Air-Gapped Environment

Many remote stores may lack internet access. K3s and Fleet support full air-gapped deployments.

In these setups, you prepare everything ahead of time using USB drives or local staging networks.

Resource	Where to Load	Notes
K3s binary	Flash drive or staging laptop	Install manually
Container images	/var/lib/rancher/k3s/agent/images/	Use ctr images export and import
fleet.yaml and app manifests	Local Git repo clone or OCI registry mirror	Load from USB or portable device
Fleet agent	Bundled with app or installed offline	Uses local bundle and syncs when connected

Note: For air-gapped shops, you can use a portable Git repo or OCI registry mirror hosted temporarily on a laptop or USB stick.

Fleet still works offline by syncing from this local source. Later, when connectivity returns, these nodes can reconnect and pull updates as usual.

What Is MQTT and Why Use It at the Edge?

MQTT (Message Queuing Telemetry Transport) is a lightweight, publish-subscribe messaging protocol designed for constrained devices and unreliable networks—making it ideal for edge computing.

For example, in the Coffee Shop app, each store has:

A temperature sensor inside a fridge.
An espresso machine that counts daily cycles.
A grinder that logs wear or usage hours.

MQTT enables these devices to publish events (like temp=97°F) to a local broker running in the store (e.g., Mosquitto). Meanwhile, your backend services (e.g., an OpenFaaS function or logging collector) subscribe to relevant topics (/store104/fridge/temp).

Why MQTT at the Edge?

Offline Resilience: Devices buffer messages even when the internet is unavailable.
Loose Coupling: Producers don’t need to know who the consumers are.
Asynchronous: Ideal for non-blocking sensor data or logs.
Bandwidth Efficient: Optimized for unreliable or metered networks.

When connectivity returns, MQTT relays buffered messages to:

A central cloud MQTT broker (e.g., EMQX, HiveMQ, or AWS IoT)
A data lake, time-series database, or dashboard via Kafka or Prometheus exporters

Using OpenFaaS for Functions at the Edge

OpenFaaS (Function-as-a-Service) is a serverless framework designed to run lightweight, containerized functions on Kubernetes—including K3s at the edge.

Why Use It at the Edge?

Reduces the overhead of running full-blown apps
Fast deployment via Kubernetes CRDs
Supports HTTP, MQTT, and CRON as event sources
Integrates with container registries and GitOps workflows

For example, in a coffee Shop app consider store 14 has a Raspberry Pi that:

Receives real-time fridge temperature via MQTT.
Triggers a Python-based function when data is published.

def handle(req):  
    temp \= float(req)  
    if temp \> 95:  
        return "Warning: Too hot\!"  
    return "Temperature OK"

Other Edge Functions You Can Run:

Detect low inventory thresholds and alert inventory-service.
Sanitize incoming sensor data and write to local disk.
Detect anomalies and forward to a machine-learning model.

OpenaaS wraps this as a function, builds a container image, and deploys it via Kubernetes CRDs. Supports MQTT, HTTP, and CRON-based triggers.

Integrating MQTT and OpenFaaS

Integrating MQTT and OpenFaaS provides a powerful, event-driven model for edge computing. MQTT acts as the event transport layer, while OpenFaaS serves as the event processor. Together, they create a reactive system that processes local data in real time and only syncs to the cloud when necessary.

Consider Store 14's Fridge Monitoring in a coffee shop app:

Sensor detects fridge temperature and publishes a message:
- Topic: /store104/fridge/temp
- Payload: {"temp": 97.4, "unit": "F"}
Mosquitto (local MQTT broker) receives the message.
OpenFaaS MQTT Connector subscribes to the topic, reads the payload, and invokes a temperature-check function.
The function evaluates the temperature:
- Logs a warning if it's too high.
- Optionally triggers an alert (e.g., send webhook, write to disk, store in SQLite).
If the internet is offline, logs and events are buffered locally.
- MQTT persists the message.
- OpenFaaS writes logs to a mounted persistent volume.
When connectivity is restored:
- Logs or alerts can be forwarded to a centralized logging system or metrics database.

Logging and Storage in Edge Scenarios

In edge environments—like individual coffee shop branches—connectivity isn't guaranteed. Devices must operate independently for logging, metrics collection, and temporary data storage. Here's how to plan for these constraints.

Local Storage Considerations

Each store’s K3s device should have storage for:

System: K3s binary, operating system, configuration files.
Runtime: Container logs, telemetry, ephemeral workloads.
Persistent Data: Customer orders, payment events, inventory state.

This ensures store operations can continue even if cloud connectivity is lost.

Logging Options

K3s includes metrics-server for lightweight resource monitoring. For advanced use cases:

Prometheus/Grafana can run on each edge cluster for real-time observability.
Use Loki or Elasticsearch to forward logs when reconnected.
Local logs can be written using Kubernetes emptyDir or persistent volumes.

Note: For more information, refer to Understanding Observability with OpenTelemetry and Coffee.

Syncing Logs and Events

Logs and events generated at the edge can be buffered and forwarded later:

Use MQTT to publish sensor data or application events to a central broker.
Store logs on local disk or persistent volume.
Sync to the cloud (object storage, log aggregation system) when a connection becomes available.

Best Practices

Cache locally: Ensure services like inventory-service and payment-service write to disk.
Use queues: MQTT or Redis queues help avoid data loss when offline.
CSI Plugins: Use Kubernetes-compatible storage interfaces suited for edge devices.
Backups: Use the 3-2-1 rule: 3 copies, 2 local (disk + USB/flash), 1 remote (cloud or data center).

Addressing Common Edge Challenges with K3s and Fleet

The combined power of K3s and Fleet helps you overcome typical edge computing hurdles:

Connectivity: K3s nodes might go offline, but Fleet handles retries and synchronizes changes when network connectivity is restored.
- For scenarios where continuous connectivity is a concern, MQTT can be used to buffer and forward messages from edge devices to a central network.
Upgrades: K3s offers a simplified one-binary upgrade process, and Fleet centrally manages the redeployment of your applications across clusters.
Storage: At the edge, you need to consider storage for the operating system, K3s itself, logged data, and container images.
- Local Data: For local logs or transactions, you can use persistent volumes (if available) or write data to local disk and sync to the cloud when connected.
- Concerns: Be mindful of latency, capacity, and reliability when designing your storage solution.
- Backups: Employ the 3-2-1 backup method: three copies of data, on two different media, with one copy off-site.

Simplifying Microservices with Istio and Service Mesh Architecture

hridyesh bisht — Thu, 03 Jul 2025 07:04:23 +0000

As apps shift from monoliths to microservices, managing service-to-service communication becomes complex. Developers must handle traffic routing, retries, timeouts, load balancing, TLS encryption, metrics, and logs for each service. This leads to duplicated code and operational complexity.

Service mesh is an infrastructure layer that manages service-to-service communication transparently. Istio, a popular open-source service mesh, addresses these challenges by deploying Envoy proxies as sidecars to your pods. The proxies intercept traffic and apply consistent policies without requiring code changes.

I use a coffee shop microservices app as a running example, since I love coffee. The app includes:

order-service (handles customer orders)
payment-service (processes payments)
inventory-service (manages coffee inventory)

Core Capabilities of Istio

Istio provides powerful capabilities for traffic management, security, and observability within a microservices environment.

Capability	Description
Traffic Management	Define routing, fault injection, and load balancing using CRDs like `VirtualService`, `DestinationRule`, and `ServiceEntry`
Traffic Routing	Flexible traffic routing configurations using `VirtualService` and `DestinationRule` resources.
Resiliency	Enforce timeouts, retries, circuit breaking, and failover without changing app logic
Mesh Extension	Integrate external services, Virtual Machines (VMs), and custom Envoy configurations Apply strong authentication (AuthN) and authorization (AuthZ) using mTLS and identity-based policies
Security	Apply strong authentication (AuthN) and authorization (AuthZ) using mTLS and identity-based policies
Observability	Export telemetry (metrics, traces, logs) using integrations like SigNoz, Prometheus, Jaeger, and Kiali

Installing Istio

Istio provides flexible installation options to support different environments and use cases. You can install Istio using:

istioctl CLI
Helm charts
Istio Operator (for GitOps and declarative installs)

Istio includes predefined installation profiles optimized for different scenarios. Each profile configures control and data plane components through the IstioOperator resource.

istioctl profile list

Istio provides different profiles:

Profile	Description
default	Production-ready; installs control plane and ingress gateway.
demo	Best for demos and learning; enables tracing, logging, ingress, and egress.
minimal	Control plane only; no gateways.
external	Used in remote clusters for multi-cluster mesh; installs nothing.
empty	Baseline config for custom setups.
preview	Includes experimental features.
ambient	Sets up sidecar-less ambient mesh (Alpha; not for production).

To install Istio using the Istio CLI, you can use the --set flag and specify the profile like this:

istioctl install \--set profile=demo

Note: Use the demo profile during development to enable full telemetry. For production, switch to default to improve performance and security.

Get full configuration of a profile:

istioctl profile dump demo

Compare two profiles:

istioctl profile diff demo default

Combining Helm and Operator

You can use the IstioOperator resource alongside Helm:

Use Helm to install base components.
Use IstioOperator to apply profile-level and mesh-level configurations.

This modular setup is useful in environments where GitOps or CI/CD pipelines manage different aspects of configuration.

Istio Architecture

Istio consists of:

Data Plane: Lightweight Envoy proxies injected as sidecars to each pod. These proxies handle all ingress and egress traffic for the pod.
Control Plane: The Istiod component configures and manages the behavior of Envoy proxies by pushing policies and configuration dynamically.

The overall architecture of an Istio-based application.

Image credits: Istio documentation.

For example, each pod in the Coffee Shop app has a sidecar Envoy proxy that intercepts all traffic. This enables Istio to provide seamless:

Traffic routing
mTLS encryption
Metrics and tracing

Sidecar

Manually modifying manifests to add sidecars is error-prone and not scalable. Istio supports two approaches:

Manual Sidecar: This method involves using the istioctl CLI to manually inject sidecars into YAML manifests. You can use the CLI to inject sidecars into your YAML manifests. Run the following command:

istioctl kube-inject \-f deployment.yaml | kubectl apply \-f \-

Automatic Sidecar: This is the recommended approach for most use cases. Istio uses a Mutating Admission Webhook to inject sidecars into all pods created in a namespace labeled with istio-injection=enabled.

Note: All new pods in that namespace get sidecars injected automatically.

To enable for a namespace, run:

kubectl label namespace coffee-shop istio-injection=enabled

Routing Traffic Through Sidecars

Istio uses iptables rules or CNI plugins to transparently route traffic through the Envoy sidecar.

An init container sets up the iptables rules before the application starts. This ensures:

Outbound traffic from the app is redirected to the Envoy sidecar
Inbound traffic hits the sidecar before reaching the app

For example, a request from order-service to payment-service is transparently routed via sidecars.

Configuring Envoy Proxies

When an app makes a call to another service, that call is now intercepted by its sidecar. The job of configuring the proxies with all the information they need to handle both incoming and outgoing traffic falls to the Istio control plane.

The Istio control plane configures all Envoy proxies dynamically using xDS APIs:

Feature	Delivered via Istio Control Plane
Route discovery	`VirtualService` and `DestinationRule` updates
Load balancing	Weighted or subset-based routing
Retry and timeout	Policy enforcement without app changes
mTLS and security	Dynamic certificate provisioning

Istio automates all sidecar configurations according to the current mesh topology. Each time services are added, removed, or updated, Istio ensures that the latest configurations—network, routing, or security policies—are distributed to the appropriate sidecars.

For example, order-service wants to call payment-service:

Istio discovers endpoints of payment-service
Pushes routing config to order-service’s proxy
Applies retries, timeouts, load balancing, and TLS

Traffic Management

Istio's traffic management relies on Custom Resources (CRDs) like VirtualService, DestinationRule, and ServiceEntry to define fine-grained routing, resiliency, and fault injection policies

Ingress and Egress Gateways

Istio uses the Gateway resource to manage how Envoy proxies handle inbound and outbound traffic. Unlike Kubernetes Ingress, Istio Gateways provide richer Layer 7 routing.

Gateway Type	Purpose	Example Scenario
Ingress Gateway	Accepts external traffic into the mesh	Client → Ingress Gateway → order-service
Egress Gateway	Manages outbound traffic to external APIs	payment-service → Egress Gateway → Payment API

Example: Gateway and VirtualService Configuration

apiVersion: networking.istio.io/v1beta1

kind: Gateway

metadata:

  name: coffee-ingress

spec:

  selector:

    istio: ingressgateway

  servers:

    \- port:

        number: 80

        name: http

        protocol: HTTP

      hosts:

        \- "order.coffee.com"

VirtualService with Gateway

To route external traffic to internal services, a Gateway must be used in conjunction with a VirtualService. If a VirtualService is not bound to a Gateway, Envoy returns an HTTP 404, indicating no route has been defined.

Create a corresponding VirtualService that binds to the Gateway:

apiVersion: networking.istio.io/v1beta1

kind: VirtualService

metadata:

  name: order-route

spec:

  hosts:

  \- "\*"

  gateways:

  \- coffee-gateway

  http:

  \- route:

    \- destination:

        host: order-service.default.svc.cluster.local

        port:

          number: 80

Apply the configuration:

kubectl apply \-f order-route.yaml

Test the route:

curl \-v http://$GATEWAY_IP/

Note: Usually, istio-ingressgateway service is exposed using the Kubernetes LoadBalancer type, which assigns an external IP to receive HTTP(S) traffic.

How the LoadBalancer Kubernetes service type works depends on how and where you run the Kubernetes cluster.

Platform	LoadBalancer Behavior
AWS, GCP, Azure	Provisions a cloud load balancer and assigns external IP.
Minikube	Requires minikube tunnel to simulate external access.

For example, In coffee shop app, gateways are essential to expose services like order-service, payment-service, and inventory-service to the outside world or external systems.

Traffic Routing and Resiliency

Istio allows flexible traffic routing configurations using VirtualService and DestinationRule resources.

Resource	Purpose
VirtualService	Defines traffic routing rules to one or more destinations.
DestinationRule	Configures policies for routed traffic, such as load balancing and TLS.
ServiceEntry	Adds external services to the mesh registry.

For example, consider a coffee-shop app with these services:

Web-frontend: the UI for customers.
Customer-service: handles customer profiles.
- Two versions of customer-service: v1 and v2.

You can define different subsets of a service, typically based on labels in the pod spec (e.g., version: v1 or version: v2 for a customer-service). Pods are labeled with version: v1 or version: v2. You set subsets in a DestinationRule:

apiVersion: networking.istio.io/v1beta1

kind: DestinationRule

metadata:

  name: customer-service

spec:

  host: customer-service.default.svc.cluster.local

  subsets:

  \- name: v1

    labels:

      version: v1

  \- name: v2

    labels:

      version: v2

Routing Traffic with VirtualService

In the VirtualService, you can specify the traffic matching and routing rules that decide which destinations traffic is routed to.

Note: To generate some traffic, open a separate terminal window and start making requests to the GATEWAY_IP in an endless loop:

export GATEWAY_IP=$(kubectl get svc \-n istio-system istio-ingressgateway \-ojsonpath='{.status.loadBalancer.ingress\[0\].ip}')

while true; do curl http://$GATEWAY_IP/; done

Weight-Based Routing: Distributes traffic across different subsets of the same service based on assigned weights (e.g., 70% to v1 and 30% to v2).

apiVersion: networking.istio.io/v1beta1

kind: VirtualService

metadata:

  name: customer-service

spec:

  hosts:

  \- customer-service.default.svc.cluster.local

  http:

  \- route:

    \- destination:

        host: customer-service.default.svc.cluster.local

        subset: v1

      weight: 70

    \- destination:

        host: customer-service.default.svc.cluster.local

        subset: v2

      weight: 30

Match-Based Routing: Routes traffic based on specific conditions, such as HTTP headers (e.g., User-Agent) or URI paths.

http:

\- match:

  \- headers:

      user-agent:

        regex: ".\*Firefox.\*"

  route:

  \- destination:

      host: customer-service.default.svc.cluster.local

      subset: v1

\- route:

  \- destination:

      host: customer-service.default.svc.cluster.local

      subset: v2

Redirect and Rewrite: Redirects traffic (HTTP 301) to a different URI or hostname, or rewrites path prefixes before forwarding. Note that redirect and destination fields are mutually exclusive.

http:

\- match:

  \- uri:

      exact: /v1/hello

  redirect:

    uri: /v2/hello

    authority: hello.default.svc.cluster.local

Rewrite path prefix:

http:

\- match:

  \- uri:

      prefix: /v1/api

  rewrite:

    uri: /v2/api

  route:

  \- destination:

      host: customer-service.default.svc.cluster.local

The redirect and destination fields are mutually exclusive. If we use the redirect, there is no need to set the destination.

Mirroring Traffic: Sends a copy of live traffic to another service version (e.g., mirroring 100% of traffic sent to v1 to v2). This "fire and forget" mechanism is useful for testing and debugging with production traffic.

http:

\- route:

  \- destination:

      host: customer-service.default.svc.cluster.local

      subset: v1

    weight: 100

  mirror:

    host: customer-service.default.svc.cluster.local

    subset: v2

  mirrorPercentage:

    value: 100.0

Header Manipulation: Allows you to add, set, or remove request and response headers, either for individual destinations or all destinations within a VirtualService.

http:

\- headers:

    request:

      set:

        debug: "true"

  route:

  \- destination:

      host: customer-service.default.svc.cluster.local

      subset: v2

    weight: 30

  \- destination:

      host: customer-service.default.svc.cluster.local

      subset: v1

    headers:

      response:

        remove:

        \- x-api-key

    weight: 70

In the above example, you set a request header debug: true for all traffic sent to the host. You are removing a response header called x-api-key. So, whenever the traffic reaches the subset v1, the response from the service will not include the x-api-key header.

AND Matching: Rules can combine multiple conditions using AND logic (e.g., matching a URI prefix and a specific header).

match:

\- uri:

    prefix: /v1

  headers:

    user:

      exact: debug

OR Matching: Rules can combine multiple conditions using OR logic (matching either a URI prefix or a header).

match:

\- uri:

    prefix: /v1

\- headers:

    user:

      exact: debug

If the first match does not evaluate to true, the algorithm moves to the second match field and tries to match the header. If you omit the match field on the route, it will continually evaluate to true.

Note: When using either of the two options, make sure you provide a fallback route if applicable. That way, if traffic doesn’t match any of the conditions, it could still be routed to a “default” route.

Resiliency Patterns

Istio enables the application of resiliency policies at the network layer, reducing the need for application code changes

Note: Both retries and timeouts happen on the client side.

Timeouts : If a request exceeds the timeout, Envoy responds with HTTP 408.

http:

\- route:

  \- destination:

      host: customer-service.default.svc.cluster.local

      subset: v1

  timeout: 5s

Retries: If the first pod fails, Envoy retries with a different healthy endpoint.

retries:

  attempts: 3

  perTryTimeout: 2s

  retryOn: gateway-error,connect-failure,reset

Circuit Breaking with Outlier Detection: This prevents cascading failures by automatically rejecting requests to overloaded or unhealthy services.

Istio implements circuit breaking using outlier detection, a passive health-checking mechanism. Envoy doesn't actively probe services but observes runtime metrics such as failure rate, latency, and connection health.

apiVersion: networking.istio.io/v1beta1

kind: DestinationRule

metadata:

  name: customer-service

spec:

  host: customer-service

  trafficPolicy:

    outlierDetection:

      consecutive5xxErrors: 1

      interval: 1s

      baseEjectionTime: 3m

      maxEjectionPercent: 100

consecutive5xxErrors: Number of consecutive 5xx responses before ejection.
interval: How often Envoy checks pod health.
baseEjectionTime: Initial duration a pod remains ejected. This increases with repeated failures.
maxEjectionPercent: Caps the percentage of pods that can be ejected.

When thresholds are met, Envoy temporarily removes the unhealthy pod from the load-balancing pool. Over time, the pod is gradually reintroduced if it recovers.

Failure Injection: This allows you to simulate network failures or delays. This helps validate your service's resilience and fallback mechanisms.

Istio supports two types of fault injection in the VirtualService:

Abort: Simulate HTTP errors by terminating requests with a specified status code.
Delay: Introduce artificial latency before forwarding requests.

Abort 30% of requests:

fault:

  abort:

    percentage:

      value: 30

    httpStatus: 404

Note: If you omit the percentage field, all matching requests will be aborted.

Inject delay to 5% of requests:

fault:

  delay:

    percentage:

      value: 5

    fixedDelay: 3s

Fault injection only affects services matched by the VirtualService. It does not impact other consumers.

Extending the Istio Mesh

Istio provides mechanisms to bring external services and Virtual Machines (VMs) into the mesh, and to customize Envoy proxies.

Bringing External Services into the Mesh

Istio tracks internal services automatically. To include external or non-Kubernetes services, use the ServiceEntry custom resource. This allows you to manage traffic and apply policies like retries, timeouts, mirroring, and fault injection to external endpoints.

For example, the Coffee Shop microservices application:

payment-service needs to call an external payment API (mesh-external)
rewards-service communicates with an internal legacy database (mesh-internal)

MESH_EXTERNAL: Used for services outside the mesh (e.g., www.googleapis.com), typically with resolution: DNS.

apiVersion: networking.istio.io/v1beta1  
kind: ServiceEntry  
metadata:  
  name: googleapis-svc-entry  
spec:  
  hosts:  
  \- www.googleapis.com  
  location: MESH_EXTERNAL  
  resolution: DNS  
  ports:  
  \- number: 443  
    name: https  
    protocol: TLS

location: MESH_EXTERNAL: Specifies the service is outside the mesh.
resolution: DNS: Istio uses DNS to resolve the host.

MESH_INTERNAL: Used for services within the mesh that do not have DNS, requiring resolution: STATIC and explicit IP addresses. The hosts field is optional with STATIC resolution. You can also use workloadSelector for endpoint selection.

apiVersion: networking.istio.io/v1beta1  
kind: ServiceEntry  
metadata:  
  name: legacy-loyalty-db  
spec:  
  addresses:  
  \- 192.192.192.192/24  
  ports:  
  \- number: 27018  
    name: mongodb  
    protocol: MONGO  
  location: MESH_INTERNAL  
  resolution: STATIC  
  endpoints:  
  \- address: 10.0.0.2  
  \- address: 10.0.0.3

Note: The hosts field is optional when using STATIC resolution.

Outbound Traffic Policy: The REGISTRY_ONLY outbound traffic policy can be configured to ensure traffic is only allowed to known services registered in the mesh.

Configure Mesh to Registry-Only:

istioctl install \--set profile=demo \--set meshConfig.outboundTrafficPolicy.mode=REGISTRY_ONLY

Confirm Configuration:

kubectl get cm \-n istio-system istio \-o yaml | grep outboundTrafficPolicy

Centralized Egress via Gateway

Use an Egress Gateway to manage and monitor all outbound traffic. This setup enables centralized TLS termination, access control, and observability.

Required resources:

AuthorizationPolicy
Gateway
VirtualService
DestinationRule

Onboarding VMs into the Mesh

VMs can join the mesh using the WorkloadEntry and WorkloadGroup resources. Istio treats VMs similarly to Kubernetes pods, assigning identities based on namespace and service account.

The general procedure for onboarding a VM can be summarized by the following steps:

Install the Istio sidecar using .deb or .rpm
Define a WorkloadGroup:

  apiVersion: networking.istio.io/v1beta1  
  kind: WorkloadGroup  
  metadata:  
    name: barista-vm  
    namespace: coffee-shop  
  spec:  
    metadata:  
      labels:  
        app: barista-service  
    template:  
      serviceAccount: barista-account

Configure the east-west gateway: ./samples/multicluster/gen-eastwest-gateway.sh \--single-cluster | istioctl install \-y \-f \-
Expose istiod: kubectl apply \-n istio-system \-f ./samples/multicluster/expose-istiod.yaml
Generate and copy configs: istioctl x workload entry configure \-f barista.yaml \-o ./output-dir
Place files in the correct locations and start the sidecar on the VM.

Communication in extending to VM:

Services in the cluster can reach VMs using DNS.
VMs can access services inside Kubernetes using mesh DNS.

An east-west gateway is necessary to enable communication between the sidecar that will be running on the VM and istiod, the Istio control plane (see the Istio documentation).

To Install the East-West Gateway and Expose Istiod

Install the east-west gateway: A./samples/multicluster/gen-eastwest-gateway.sh --single-cluster | istioctl install -y -f -
- If you list the pods in the istio-system namespace you’ll notice the istio-eastwestgateway instance was created.
Expose istiod though the east-west gateway: kubectl apply -n istio-system -f ./samples/multicluster/expose-istiod.yaml

For example, consider

apiVersion: networking.istio.io/v1beta1  
kind: WorkloadGroup  
metadata:  
  name: barista-vm  
  namespace: coffee-shop  
spec:  
  metadata:  
    labels:  
      app: barista-service  
  template:  
    serviceAccount: barista-account

Customizing and Extending Envoy Proxies

Istio automatically generates the Envoy configuration for each proxy. However, for advanced use cases, you can customize this configuration and extend Envoy's functionality.

Envoy's configuration is structured into several key components:

Listeners: Network locations (IP and port) where Envoy listens for incoming connections and requests. Istio generates multiple listeners for each sidecar.
Filters: Ordered lists of processing logic that a request flows through (Listener, Network, and HTTP filters). The router filter is typically the last HTTP filter and is responsible for routing traffic.
Routes: URI/path-based traffic routing rules defined within the route configuration. These rules match incoming requests and specify where traffic should be sent.
Clusters: Groups of similar upstream hosts (destinations or servers), analogous to Kubernetes Services, that accept traffic.
Endpoints: Concrete IP:port pairs within a cluster, representing the specific addresses where traffic can be sent.

For example, when a request reaches coffee-frontend:

Envoy listens on port 15001.
Filters inspect and process the request.
Routing sends it to the barista-service cluster.
One of the barista pods (endpoint) handles the request.

You can inspect the Envoy configuration using the istioctl proxy-config command. For example:

istioctl proxy-config clusters coffee-frontend-xyz \--namespace coffee-shop

The EnvoyFilter resource allows you to customize portions of the auto-generated Envoy proxy configuration by patching existing settings. This enables updating values, adding or removing filters, or creating new listeners and clusters.

Application Scope: EnvoyFilter resources can be applied at three levels: globally (affecting all proxies in the mesh), per namespace, or to specific workloads.
Patch Location (applyTo): You can target specific configuration sections, such as LISTENER, HTTP_FILTER, NETWORK_FILTER, or CLUSTER.
Patch Target (match): The scope can be narrowed using context (e.g., SIDECAR_INBOUND, SIDECAR_OUTBOUND, GATEWAY), listener properties, route configuration, or cluster properties.
Patch Action (patch): Defines how the patch is applied, with operations like MERGE, ADD, REMOVE, INSERT_BEFORE, or INSERT_AFTER.

Example to patch with EnvoyFilter

\- applyTo: EXTENSION_CONFIG  
  patch:  
    operation: ADD  
    value:  
      name: custom-metrics  
      typed_config:  
        "@type": type.googleapis.com/envoy.extensions.filters.http.wasm.v3.Wasm  
        config:  
          root_id: metrics-root  
          vm_config:  
            vm_id: metrics-vm  
            runtime: envoy.wasm.runtime.v8  
            code:  
              remote:  
                http_uri:  
                  uri: http://wasm-module-uri

Extending Envoy with WebAssembly

Envoy's functionality can be extended using custom filters written in different languages:

C++: Offers native, high-performance extensions but requires rebuilding Envoy.
Lua: Script-based, suitable for simpler use cases.
WebAssembly (Wasm): Enables run-time loaded plugins compiled from languages like Rust, Go, or AssemblyScript. Wasm plugins run in a sandboxed virtual machine (VM), providing isolation and memory safety.

Wasm allows dynamic extensibility of the Envoy data plane without needing to rebuild Envoy or manually modify its configurations. Istio's istio-agent handles the distribution of Wasm plugins, fetching them from registries and mounting them into Envoy's file system.

For example, in the Coffee Shop app: Use a WASM filter to collect metrics on espresso orders handled by barista-service. This plugin runs inside the Envoy proxy and logs telemetry data.

In Istio, Wasm enables customization of the Envoy data plane without rebuilding or manually modifying Envoy configurations. It introduces dynamic extensibility to the mesh.

Wasm Plugin Deployment Workflow:

Compile Plugin
- Use your SDK to generate .wasm file
Publish to Registrydocker build -t registry.io/barista-metrics:v1 . docker push registry.io/barista-metrics:v1
Deploy with WasmPlugin

apiVersion: extensions.istio.io/v1alpha1  
kind: WasmPlugin  
metadata:  
  name: barista-metrics  
  namespace: coffee-shop  
spec:  
  selector:  
    labels:  
      app: barista-service  
  url: oci://registry.io/barista-metrics:v1  
  pluginConfig:  
    trackEspresso: true  
    debug: true

Plugin Source Options:

oci://: OCI registry
http://: Direct HTTP URL
/path/to/local: Local file path

Using WasmPlugin is preferred over EnvoyFilter as it simplifies deployment.

Security

Istio enhances security by enforcing strong authentication (AuthN) and authorization (AuthZ) policies.

Authentication

Istio issues X.509 SPIFFE-compliant certificates to each pod, based on Kubernetes ServiceAccounts.

mTLS: Ensures both client and server verify each other’s identities.
Certificates: Automatically rotated and managed by Istio agent using SDS.

Automated Identity Provisioning: Istio automates workload identity through these components:

Component	Role
Istio Agent	Runs in the sidecar, manages certificates and bootstraps Envoy
SDS	Envoy’s Secret Discovery Service; fetches certs dynamically
Istiod	Acts as the Certificate Authority (CA); issues and rotates certs

When a sidecar starts, the Istio agent sends a Certificate Signing Request (CSR) to istiod. Once verified, istiod returns a signed certificate. This identity is used for secure communication between services. Certificates are rotated automatically.

For example, in the coffee shop microservices app:

barista-service runs with a sidecar.
Istio agent requests a certificate for the barista-service account.
CA authenticates the request and returns a signed certificate.
barista-service uses this identity for secure communication.

Mutual TLS (mTLS)

Mutual TLS ensures encrypted and authenticated communication. Both client and server validate each other using their certificates. Envoy sidecars handle this process transparently.

PeerAuthentication (Inbound): Configures the mTLS mode for incoming traffic to a service or workload.

apiVersion: security.istio.io/v1beta1

kind: PeerAuthentication

metadata:

  name: default

spec:

  mtls:

    mode: STRICT

DestinationRule (Outbound): Configures the mTLS mode for outgoing traffic from a service or workload. This also applies to outgoing traffic through an egress gateway.

trafficPolicy:  
 tls:  
   mode: ISTIO_MUTUAL

mTLS Modes Overview:

Mode	Description
PERMISSIVE	Accepts both plain text and mTLS connections (default for onboarding)
STRICT	Only mTLS connections are allowed
ISTIO_MUTUAL	Uses Istio-managed certificates for mTLS (recommended default)
SIMPLE	One-way TLS (client verifies server)
MUTUAL	mTLS using custom certificates
PASSTHROUGH	Routes encrypted TLS traffic without termination
AUTO_PASSTHROUGH	Automatically forwards TLS based on SNI (no VirtualService required)

Note: You can apply mTLS at mesh, namespace, workload, or port level and these modes apply to both Ingress and Egress gateways.

Example: Set STRICT mTLS for payment-service:

apiVersion: security.istio.io/v1beta1

kind: PeerAuthentication

metadata:

  name: payment-service-mtls

  namespace: coffee-shop

spec:

  selector:

    matchLabels:

      app: payment-service

  mtls:

    mode: STRICT

Request Authentication (User Authentication)

Use RequestAuthentication to verify JWT tokens from end users. If a token is invalid or missing, the request is rejected. Valid tokens yield an authenticated principal for policy enforcement.

Example: Require JWT for customer-service:

apiVersion: security.istio.io/v1  
kind: RequestAuthentication  
metadata:  
  name: customer-jwt  
  namespace: coffee-shop  
spec:  
  selector:  
    matchLabels:  
      app: customer-service  
  jwtRules:  
  \- issuer: "https://auth.coffeeshop.com"  
    jwksUri: "https://auth.coffeeshop.com/.well-known/jwks.json"

Authorization (Access Control)

Use the AuthorizationPolicy resource to enforce fine-grained control over what services or users can access. Policies use service identities (via mTLS) and user identities (via JWT).

For example, only allow authenticated users to call customer-service:

apiVersion: security.istio.io/v1

kind: AuthorizationPolicy

metadata:

  name: require-jwt

  namespace: coffee-shop

spec:

  selector:

    matchLabels:

      app: customer-service

  action: ALLOW

  rules:

  \- from:

    \- source:

        requestPrincipals: \["\*"\]

Match Conditions: Rules can match requests based on:

Field	Description
`from`	Source identity: service accounts, IPs, JWT principals
`to`	Operation match: HTTP methods, ports, paths
`when`	Additional conditions: headers, claims, IPs

Example: Allow DELETE only from admin-service:

rules:  
\- from:  
  \- source:  
      principals: \["spiffe://cluster.local/ns/coffee-shop/sa/admin-service"\]  
  to:  
  \- operation:  
      methods: \["DELETE"\]  
      paths: \["/customers/\*"\]

Action Types in AuthorizationPolicy:

Action	Purpose
`ALLOW`	Permit matching requests
`DENY`	Block matching requests
`CUSTOM`	Delegate evaluation to a custom extension
`AUDIT`	Log matching requests without enforcing access decisions

Note: Istio evaluates policies in this order: CUSTOM → DENY → ALLOW

Best Practices

Start with a DENY-all policy and incrementally allow access using ALLOW rules.
Assign dedicated ServiceAccounts per workload to ensure identity isolation.
Use STRICT mTLS once workloads are mesh-ready.
Combine PeerAuthentication, RequestAuthentication, and AuthorizationPolicy for zero-trust enforcement.

Observability

Observability is essential for understanding and operating microservices in production. Istio provides out-of-the-box observability by capturing telemetry at the network layer through sidecar proxies.

Istio enables deep insights across services by capturing:

Metrics: Quantitative measurements such as request latency or error rates.
Traces: End-to-end request flow across services.
Logs: Context-rich records for debugging.

These signals work together. For example, a spike in latency (metrics) leads you to a specific service call (trace), and the logs explain the failure.

Note: For more information, refer to Understanding Observability with OpenTelemetry and Coffee.

For the coffee shop example,the coffee shop app has three microservices:

order-service
payment-service
inventory-service

Each service includes an injected Envoy sidecar that automatically collects and exposes telemetry.

Setup for Observability

Install Istio using the demo profile to enable full telemetry:

istioctl install \--set profile=demo \-y

This profile enables 100% trace sampling—ideal for development. In production, reduce sampling to 1% to balance overhead.

Envoy sidecars expose Prometheus scrape endpoints. Metrics can also be accessed via each pod’s Envoy admin dashboard.

Tracing and Logs with SigNoz

SigNoz is an OpenTelemetry-compatible observability tool that integrates seamlessly with Istio:

helm install signoz signoz/signoz -n platform

You can use the SigNoz UI to:

Search for traces by service (e.g., order-service)
Visualize trace duration and latency
Correlate logs and spans

Refer to SigNoz Installation Guide for setup instructions.

Optimization and Advanced Deployments

In large meshes, every sidecar receives service discovery updates for all mesh services. This can lead to:

Excessive configuration updates
Increased startup time for proxies

To limit this, use the Sidecar resource to restrict which services a workload can see.

For example, a Sidecar resource can restrict outbound traffic (egress) from a coffee-frontend workload to only communicate with order-service and payment-service within its namespace.

apiVersion: networking.istio.io/v1beta1  
kind: Sidecar  
metadata:  
  name: coffee-frontend  
  namespace: coffee-shop  
spec:  
  workloadSelector:  
    labels:  
      app: coffee-frontend  
  egress:  
  \- hosts:  
    \- "order-service.coffee-shop.svc.cluster.local"  
    \- "payment-service.coffee-shop.svc.cluster.local"

This configuration allows coffee-frontend to communicate only with order-service and payment-service, reducing its load.

Multi-Cluster Deployments

Deploying Istio across multiple Kubernetes clusters offers benefits such as high availability, failover capabilities, and organizational separation.

Network Models:
- Single Network: Pods across different clusters can communicate directly.
- Multiple Networks: East-west gateways are used to facilitate communication between clusters.
Control Plane Models:
- Single Control Plane: A single Istiod instance manages all clusters. While simpler, it represents a single point of failure.
- Per-Cluster Control Plane: Each cluster has its own Istiod instance, providing better high availability and isolation.
Mesh Models:
- Single Mesh: A unified trust domain and configuration across all clusters.
- Multi-Mesh Federation: Separate meshes can share trust bundles (root certificates), define shared ServiceEntry resources, and apply AuthorizationPolicy for secure cross-mesh communication.
Tenancy Models:
- Soft Tenancy: Achieves isolation at the namespace level.
- Hard Tenancy: Provides isolation at the cluster level with separate meshes.

Locality-Aware Load Balancing:

Use localityLbSetting to steer traffic based on geographic proximity (region, zone, sub-zone).

Failover Example: If all endpoints in us-west are unavailable, traffic fails over to us-east.

trafficPolicy:

  localityLbSetting:

    enabled: true

    failover:

      \- from: us-west

        to: us-east

Weighted Distribution: This routes 50% of the traffic to local zone us-west1-a, 30% to the neighboring zone, and 20% to a remote zone.

trafficPolicy:

  localityLbSetting:

    enabled: true

    distribute:

      \- from: "us-west1/us-west1-a/\*"

        to:

          "us-west1/us-west1-a/\*": 50

          "us-west1/us-west1-b/\*": 30

          "us-east1/us-east1-a/\*": 20

Wrapping Up: Istio as Your Mesh Barista

Istio helps transform your distributed microservices from a tangle of complexity into a well-orchestrated system. Whether it's fine-grained traffic routing, observability across services, enforcing security through mTLS, or extending the mesh with WebAssembly.

Like a well-run coffee shop, every component of your system needs to collaborate in real time. Istio acts as the operations manager behind the scenes, ensuring communication flows smoothly, issues are detected early, and only trusted interactions are allowed.

Understanding Observability with OpenTelemetry and Coffee

hridyesh bisht — Wed, 04 Jun 2025 16:39:38 +0000

Solutions are increasingly built using microservices architecture, leading to complex distributed systems. Monitoring these systems becomes challenging due to the diversity of tools, protocols, and data formats.

This blog focuses on:

Explaining the basics of OpenTelemetry, its role in observability, and the current state of observability in the industry.
Explaining how to instrument code and identify when to use manual and automatic instrumentation.
Discussing the OpenTelemetry Collector and Connector, which are responsible for processing and forwarding telemetry data.

What is OpenTelemetry?

OpenTelemetry addresses these challenges by providing a unified framework for collecting, processing, and exporting telemetry data, enabling you to gain deep insights into their apps’ behavior.

For this guide, consider a modern coffee shop app with the following microservices:

Order Service: Handles customer orders.
Payment Service: Processes payments.
Inventory Service: Manages stock levels.
Notification Service: Sends order confirmations.

OTel Collector -> Observability Backend" width="800" height="297">

Each service operates independently, possibly written in different languages and deployed across various environments. When a customer places an order, the request traverses multiple services, making it essential to have a comprehensive observability solution to monitor and troubleshoot the system effectively.

Key Benefits of OpenTelemetry:

Unified Instrumentation: Instrument your code once and send telemetry data to multiple backends without re-instrumentation.
Vendor-Neutral: Avoid vendor lock-in by using standard APIs and protocols. Since you can switch platforms without having to re-instrument your entire solution.
Unified telemetry: Combines tracing, logging, and metrics into a single framework enabling correlation of all data and establishing an open standard for telemetry data.
- Linking these parameters helps you make better decisions.
Community-Driven: Benefit from a vibrant open-source community contributing to continuous improvements.
Improved Correlation: Easily correlate data across different telemetry signals for better insights.

Three pillars of Observability

Telemetry is collected via instrumentation and flows through a pipeline that enriches, batches, and stores it for later analysis. Most observability tooling revolves around three categories of telemetry: logs, metrics, and traces.

While they share architectural similarities, such as instrumentation, ingestion, storage, and visualization. Each type presents unique challenges and is best suited to answer different types of questions.

Logs

Logs are immutable, timestamped records of discrete events. Each log entry typically contains a message and optional structured metadata. However, coming up with a standardized log format is no easy task, since different pieces of information are critical for different types of software.

You can also build logging agents and protocols to forward logs to a central location for efficient storage. For example, consider a user placing an order in a microservices-based coffee shop app. The order-service logs a line like:

{  
  "timestamp": "2025-06-01T08:43:12Z",  
  "level": "INFO",  
  "service": "order-service",  
  "message": "New order placed: latte",  
  "order\_id": "ORD-20250601-001"  
}

Metrics

Metrics help you understand a high-level view of the current state of your system. A metric is a single numerical value derived by applying a statistical measure to a group of events.

In other words, metrics represent an aggregate. This is useful because their compact representation allows us to graph how a system changes over time.

Different Metric Types:

Counters: Total number of orders placed
Gauges: Current number of in-progress orders
Histograms: Distribution of order preparation times
Summaries: Quantiles of response times

Coffee Shop Example*:* The order-service emits a metric would be displayed.

orders_placed_total{beverage=”latte”} 1560

A Prometheus dashboard may show a sharp spike in latte orders, suggesting a promotional campaign is working or an anomaly is occurring.

Traces

To understand the larger context in distributed solution, you must identify other related events, such as the specific requests or transactions that initiated the log entry and the sequence of services or microservices involved in processing that request across the system.

Traces visualize the full journey of a single request across services. A trace consists of multiple spans, each representing a step in the request’s lifecycle. This makes it possible to reconstruct the journey of requests in the system.

Coffee Shop Example*:* A user places an order. The request flows through UI -> order-service -> payment-service -> inventory-service.

order-service -> payment-service -> inventory-service." width="800" height="326">

Each service adds a span with trace and span IDs, allowing you to view:

Total request duration
Which service caused a delay
Any failed steps in the chain

Problems with the Current Observability Approach

Logs, metrics, and traces typically live in separate systems, with different formats and tooling. This fragmentation forces you to jump between dashboards and correlate data manually. Even with shared metadata like timestamps or service names, stitching information together remains time-consuming and error-prone.

Coffee Shop Example: Imagine a spike in failed order-service requests. You check metrics and see a high error rate. You then switch to logs, scan for failures, and try to match logs with trace IDs. Without consistent context, root cause analysis becomes guesswork.

Lack of Built-in Instrumentation in Open Source Software

Many open source libraries expose hooks but do not include native telemetry support. You must build and maintain custom adapters.

Problems this causes:

Version Compatibility: Library updates may break adapters.
Telemetry Loss: Converting data between formats can degrade signal quality.
Engineering Overhead: Teams spend time wiring telemetry instead of building features.

Coffee Shop Example*:* If the inventory-service uses a third-party stock manager with no OpenTelemetry support, you must manually instrument it or depend on its observability hooks.

What OpenTelemetry is NOT

OpenTelemetry simplifies telemetry collection and export, but it doesn’t offer end-to-end observability out of the box. It’s a toolkit, not a monitoring platform.

Not OpenTelemetry’s Job	Description
Data storage	OpenTelemetry exports data; it doesn’t store it. You’ll need systems like SigNoz, Prometheus, or Elasticsearch.
Visualization	No dashboards or charts are included. Use tools like Grafana, Jaeger, or Datadog.
Alerting	OpenTelemetry doesn’t generate alerts. Integrate it with systems that support alert rules.
Monitoring out-of-the-box	It doesn’t auto-instrument everything or provide prebuilt dashboards. You must configure and integrate.
Performance optimization	It helps identify bottlenecks, but doesn’t tune your app.

OpenTelemetry standardizes how you collect logs, metrics, and traces. It enables observability, but doesn’t deliver it on its own. You still need storage, visualization, alerting, and analysis platforms to complete the picture.

Signals in OpenTelemetry

OpenTelemetry organizes observability data into three core signals:

Signal	Purpose
Traces	Capture the lifecycle and flow of a request across services.
Metrics	Measure system and app performance over time.
Logs	Record discrete events and state changes in the app.

Each signal is independent but can be correlated to provide richer observability. OpenTelemetry’s architecture ensures signal consistency and interoperability across programming languages through its official OpenTelemetry Specification.

OpenTelemetry Specification Components

Common Terminology: Ensures a consistent vocabulary across implementations.
API Specification: Provides language-agnostic interfaces to generate telemetry (traces, metrics, logs). APIs are backend-agnostic and enable portable instrumentation.
- For more information, refer to Tracing API, Metrics API, and OpenTelemetry Logging.
SDK Specification: Defines how SDKs process, sample, and export telemetry. Ensures consistent behavior across languages.
- For more information, refer to Tracing SDK, Metrics SDK, and Logs SDK.
Semantic Conventions: Standardizes names and attributes for telemetry data (e.g., HTTP status codes, DB queries).
OpenTelemetry Protocol (OTLP): Describes a vendor-neutral transport protocol to send telemetry

Why separate API from SDK?

The API–SDK split improves modularity, portability, and vendor neutrality:

Library safety: A shared library (e.g., database driver) can safely include only the API, avoiding heavy SDK dependencies and avoiding conflicts in user apps.
Portability: You can ship apps with OpenTelemetry APIs baked in, and let platform teams decide which SDK/exporter to use later.
Flexibility: You can write your own SDK or replace components (e.g., use a custom sampler or exporter).

OpenTelemetry API vs SDK

Feature	OpenTelemetry API	OpenTelemetry SDK
Purpose	Defines interfaces to generate telemetry	Implements logic to process and export telemetry
Responsibility	Exposes functions to create spans, metrics, logs	Manages batching, sampling, context, and export
Language-specific?	Yes	Yes
Included by default?	Yes, lightweight	No, must be explicitly added and configured
Default behavior	No-op	Active when configured
Used by	App and library developers	DevOps, SREs, platform engineers
Stability	High	May evolve with backends and exporter needs
Customizable	No	Yes (exporters, processors, samplers)

For example, consider the following scenarios:

Scenario	Best choice
Open-source library with tracing support	API only (lightweight, no deps)
Production microservice exporting to Grafana	API + SDK + OTLP Exporter
CLI tool needing optional debug tracing	API (enabled conditionally with SDK)

How to Instrument Code with OpenTelemetry

OpenTelemetry supports multiple instrumentation approaches to capture telemetry from apps. Understanding these methods helps choose the right approach based on your app’s complexity, development stage, and observability goals.

OpenTelemetry classifies instrumentation into three categories, often overlapping in practice:

Category	Effort	Control	Customization	Code Changes
Automatic Instrumentation (Zero-Code)	Low	Limited	Minimal	None
Instrumentation Libraries	Moderate	Medium	Moderate	Minimal to moderate
Manual Instrumentation (Fully Code-Based)	High	Full	Full	Extensive

OpenTelemetry provides three ways to capture telemetry from your app:

1. Automatic Instrumentation

Auto-instrumentation in .NET 8 is available via the OpenTelemetry .NET Auto-Instrumentation Agent, which instruments common libraries like ASP.NET Core, HttpClient, and SQL clients at runtime.

Ideal use case: Use this to quickly add observability to .NET services without modifying source code.

Example: orders-service (.NET 8, ASP.NET Core)

Download and install the auto-instrumentation binaries .NET Auto-Instrumentation GitHub
Run the app with the auto-instrumentation profiler

set OTEL_SERVICE_NAME=orders-service
set OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4317
set CORECLR_ENABLE_PROFILING=1
set CORECLR_PROFILER_PATH=C:\otel-dotnet auto\OpenTelemetry.AutoInstrumentation.Native.dll
dotnet run

What it captures:

HTTP requests and responses
Outgoing HTTP/gRPC calls
SQL queries via ADO.NET

Pros:

No code changes required
Fast onboarding
Works well for ASP.NET Core, Entity Framework, and HttpClient

Cons:

Limited to supported libraries
Less control over span names and metadata

2. Library-Based Instrumentation

Library-based instrumentation uses the OpenTelemetry SDK and prebuilt instrumentations like AddAspNetCoreInstrumentation.

Ideal use case: You want to customize configuration and capture high-value signals without full manual control.

Example: menu-service (.NET 8, ASP.NET Core)

Install NuGet packages:

dotnet add package OpenTelemetry.Extensions.Hosting
dotnet add package OpenTelemetry.Instrumentation.AspNetCore
dotnet add package OpenTelemetry.Instrumentation.Http
dotnet add package OpenTelemetry.Exporter.OpenTelemetryProtocol

Configure in Program.cs:

using OpenTelemetry.Metrics;
using OpenTelemetry.Resources;
using OpenTelemetry.Trace;

var builder = WebApplication.CreateBuilder(args);

builder.Services.AddOpenTelemetry()
    .WithTracing(tracerProviderBuilder =>
    {
        tracerProviderBuilder
            .SetResourceBuilder(
                ResourceBuilder.CreateDefault()
                    .AddService("menu-service"))
            .AddAspNetCoreInstrumentation()
            .AddHttpClientInstrumentation()
            .AddOtlpExporter(otlp =>
            {
                otlp.Endpoint = new Uri("http://otel-collector:4317");
            });
    });

var app = builder.Build();
app.MapGet("/", () => "Hello from Menu Service");
app.Run();

What it captures:

Inbound ASP.NET Core request spans
Outbound calls (HttpClient, gRPC)
Custom span and resource metadata

Pros

Easy to configure
Integrates well with DI and hosting model
Supports enrichment and filtering

Cons

Requires adding code/configuration
Less flexible than full manual instrumentation

3. Manual Instrumentation

Manual instrumentation lets you define custom spans for critical business logic (e.g., awarding loyalty points or calculating discounts).

Ideal use case: You need to trace domain-specific logic not covered by auto or library-based methods.

Example: loyalty-service (.NET 8 Worker Service)

Install packages:

dotnet add package OpenTelemetry
dotnet add package OpenTelemetry.Trace
dotnet add package OpenTelemetry.Exporter.OpenTelemetryProtocol

Configure tracing in Program.cs:

using OpenTelemetry.Trace;
using OpenTelemetry.Resources;
using System.Diagnostics;

var builder = Host.CreateApplicationBuilder(args);

builder.Services.AddOpenTelemetry()
    .WithTracing(tracerProviderBuilder =>
    {
        tracerProviderBuilder
            .SetResourceBuilder(ResourceBuilder.CreateDefault().AddService("loyalty-service"))
            .AddSource("LoyaltyService")
            .AddOtlpExporter(options =>
            {
                options.Endpoint = new Uri("http://otel-collector:4317");
            });
    });

var app = builder.Build();

// Start a custom span manually
var source = new ActivitySource("LoyaltyService");

using var activity = source.StartActivity("AwardLoyaltyPoints", ActivityKind.Internal);
activity?.SetTag("customer.id", "cust-123");
activity?.SetTag("points.awarded", 20);

// Simulate business logic
Console.WriteLine("Loyalty points awarded.");

What it captures:

Custom spans for logic like point calculations
Rich metadata (tags)
Correlation with other telemetry (metrics/logs)

Pros:

Full control over telemetry
Capture domain-specific operations
High value for debugging or performance tuning

Cons:

Requires development effort
Must manage span lifecycle correctly
Potential for inconsistent usage without guidelines

Overlaps and Clarifications

Instrumentation libraries sometimes provide automatic instrumentation after import, blurring the line between zero-code and code-based.
Under the hood, all approaches use some form of libraries.
Zero-code is broad and quick; libraries add customization; manual is full control.

Recommended Approach and Strategy

Start with automatic instrumentation to gain immediate insight with minimal effort.
Add instrumentation libraries where you need more coverage or framework-specific tracing.
Use manual instrumentation for critical business logic or custom metrics requiring fine-grained control.

Why use OpenTelemetry Collector?

The OpenTelemetry Collector is a vendor-agnostic, standalone service that simplifies telemetry management in production. It decouples telemetry generation from ingestion and export, offering the following benefits:

The Collector provides three core capabilities:

Function	Description
Receive	Accepts telemetry from apps, agents, or other Collectors via OTLP or other supported protocols.
Process	Filters, enriches, transforms, batches, or samples telemetry data.
Export	Sends processed data to one or more observability backends.

Key benefits

Without Collector	With Collector
Apps must export data directly to each backend	Central point of control for all telemetry
Risk of tight coupling to backend protocols	Decouples app logic from backend details
Difficult to enforce consistent processing	Apply transformations consistently
No central routing or batching	Route and batch data efficiently

Understanding OpenTelemetry Protocol (OTLP)

OTLP is the native telemetry transport used across OpenTelemetry. It standardizes how telemetry is serialized, transmitted, and received.

Key benefits:

Unified: Handles traces, metrics, and logs in one format.
Vendor-neutral: Reduces backend lock-in and removes custom exporters.
Efficient: Uses gRPC and Protobuf for high-performance streaming.
Extensible: Schema evolves without breaking compatibility.
Integrated: Collector and most observability tools support OTLP out of the box.

OTLP transport options

Transport	Encoding	Use case
gRPC	Protobuf	Default for performance and bi-directional streaming
HTTP/1.1	JSON	Debugging, human-readable payloads
HTTP/2	Protobuf	Efficient, firewall-friendly alternative to gRPC

Example: Pre-OTLP vs OTLP

Before OTLP	With OTLP
Prometheus exporter, Zipkin exporter, Fluentd plugin	One OTLP exporter and one Collector instance
Multiple exporters in each service	Centralized, simplified telemetry pipeline

In OpenTelemetry, logs are a critical signal for observability. Any data that isn’t a trace or metric is categorized as a log. Events, for instance, are specialized log entries.

Unlike traces and metrics, which OpenTelemetry implements via dedicated APIs and SDKs, logging is designed to integrate with existing logging frameworks in various programming languages. Instead of requiring a brand-new logging API, OpenTelemetry provides a Logs Bridge API that links traditional logging systems with telemetry signals such as traces and metrics.

How Logging Works in OpenTelemetry

You instrument logging using the Logs Bridge API, which connects popular logging frameworks (like Serilog, ILogger, or log4net in .NET) to OpenTelemetry’s pipeline.

Key Components

LoggerProvider: Factory for creating loggers.
Logger: Used to create log entries (LogRecord).
LogRecord: Represents a single log entry with metadata.
LogRecordExporter: Sends logs to destinations like the OpenTelemetry Collector.
LogRecordProcessor: Processes logs before they’re exported.

LogRecord Structure

A LogRecord typically includes:

timestamp: When the log occurred.
trace_id, span_id: Links to a trace/span for correlation.
severity_text: e.g., INFO, WARNING, ERROR.
body: The log message or structured content.
attributes: Custom metadata (e.g., user.id, http.method).

Example Use Case: Coffee app has a /get_coffee endpoint. When a coffee request fails due to a missing ID, the app logs this event.

logger.error(“Missing coffee ID”, extra={“http.status_code”: 400, “coffee_id”: None})

This log entry can be linked to the trace of the request, helping correlate the failure with upstream service calls and backend metrics.

Collector Configuration

The OpenTelemetry Collector decouples telemetry generation from backend concerns. It processes logs, traces, and metrics independently.

Collector Pipeline Example

receivers:
  otlp:
    protocols:
      grpc:


processors:
  batch: {}


exporters:
  logging:
    loglevel: debug


service:
  pipelines:
    logs:
      receivers: [otlp]
      processors: [batch]
      exporters: [logging]

Collector Deployment Topologies

The OpenTelemetry Collector supports multiple deployment models, allowing you to tailor observability pipelines based on your architecture and scalability needs. Each topology serves different use cases—from tightly coupled microservices to centralized processing in large-scale environments.

Sidecar Deployment : OpenTelemetry Collector runs as a sidecar alongside each application instance. This setup is common in containerized environments like Kubernetes, where the Collector is injected into each Pod.

Advantages:

Low latency: The Collector runs on the same host or Pod, reducing network overhead for exporting telemetry data.
Isolation: Each service has a dedicated Collector instance, ensuring telemetry data stays service-specific and avoids cross-contamination.
Simplified trace correlation: Local logs, traces, and metrics can be more easily linked.

Ideal for microservices architectures where services operate independently and require individual telemetry pipelines.

Node Agent Deployment: a single Collector instance runs per host or node. This is typically implemented as a Kubernetes DaemonSet or similar system service in virtual machine environments.

Advantages:

Centralized control per node: One Collector handles telemetry for all services on the same node.
Resource-efficient: Fewer Collector instances are required compared to the sidecar model.
System metrics access: Can collect host-level metrics (CPU, memory, disk, etc.) in addition to application telemetry.

Ideal Use Case:

Suitable for clusters with many lightweight services that share node resources.
Often used to monitor node-level infrastructure and runtime metrics alongside service-level data.

Standalone or Gateway Deployment: The Collector runs as a dedicated service, often behind a load balancer. Applications send telemetry data remotely to this central Collector (typically over OTLP).

Advantages:

Scalability: A centralized Collector cluster can scale independently from application workloads.
Simplified configuration management: Telemetry pipelines and transformations are managed in one place.
Decoupling from application logic: Developers don’t need to worry about backend changes or exporter configurations.

Ideal Use Case:

Best suited for large-scale systems with high telemetry volume.
Useful for teams that want to offload all processing from applications and maintain a consistent observability architecture.

Benefits of OpenTelemetry Collectors:

Separation of concerns: Developers emit logs; operators manage pipelines.
Centralized management: All configuration is in one place.
Resource efficiency: Offloads processing from app.
No redeployments needed: Change pipelines without touching app code.

SigNoz with OpenTelemetry

SigNoz is a powerful observability platform built specifically for OpenTelemetry. It provides a seamless experience for collecting, storing, visualizing, and querying telemetry data, without vendor lock-in.

With OpenTelemetry, you collect signals (logs, metrics, and traces) from the coffee shop services. These signals are sent to the OpenTelemetry Collector, which processes and forwards them to SigNoz.

In our coffee shop microservices example, SigNoz plays the role of the observability backend, giving your team full visibility into traces, metrics, and logs generated by the app. Here’s how SigNoz helps the coffee shop:

Traces: Visualize how an order moves through the system, from frontend-service to payment-service and inventory-service. Identify latency bottlenecks or failed calls.
Metrics: Monitor key service-level indicators like espresso_orders_per_minute, latency, and error_rate without writing custom dashboards.
Logs: Correlate logs with trace IDs and span IDs to troubleshoot order failures (e.g., inventory out-of-stock or payment declined).

For more information, refer to:

Introduction to Container Images and Orchestration

hridyesh bisht — Wed, 07 May 2025 08:09:02 +0000

As modern apps shift toward microservices and cloud-native architectures, containers have become the standard for packaging and deploying software. However, running containers in production requires more than just building images—it demands scalable orchestration and intelligent management.

This blog introduces container images. It explains why orchestration is essential in production environments. The blog also explores Kubernetes, the industry-standard platform for container orchestration.

What Are Container Images?

A container image packages app code, its runtime, libraries, and all dependencies in a predefined, portable format. This enables consistent deployment across various environments.

Container runtimes—such as containerd, runC, and CRI-O—use these prebuilt images to create and run one or more containers. While these runtimes are effective on a single host, they lack scalability and fault tolerance required for production environments.

In production scenarios, apps must meet several critical requirements:

Fault tolerance: Automatically recover from failures.
Scalability: Adjust resources based on demand.
Efficient resource utilization: Optimize hardware usage.
Service discovery: Enable components to find each other dynamically.
External accessibility: Expose services to external clients.
Seamless updates and rollbacks: Deploy new versions without downtime.

Managing containers manually or through scripts becomes impractical as the number of containers grows. This is where container orchestrators come into play.

What Is a Container Orchestrator?

A container orchestrator automates the deployment, scaling, networking, and management of containers across multiple hosts. It treats a group of systems as a single cluster, providing:

High availability: Ensures services are always accessible.
Distributed workloads: Balances tasks across nodes.
Optimized resource allocation: Efficiently utilizes system resources.
Automated health checks and restarts: Maintains application health.

Common features of orchestrators include:

Cluster management: Combine multiple hosts into a unified cluster.
Container scheduling: Deploy containers based on resource availability.
Service discovery: Enable communication across containers, regardless of the host.
Storage binding: Attach persistent storage volumes to containers.
Load balancing: Distribute traffic across containers.
Security policies: Control access to containerized applications.
Resource optimization: Automatically manage and scale resources based on demand.

Popular container orchestrators and services:

Kubernetes: Open-source and cloud-agnostic; the industry standard for container orchestration.
Amazon ECS: A fully managed service by AWS for running Docker containers at scale.
Amazon EKS: A managed Kubernetes service by AWS.
Azure Kubernetes Service (AKS): Microsoft's managed Kubernetes offering.
Google Kubernetes Engine (GKE): Google's managed Kubernetes service.
HashiCorp Nomad: A flexible orchestrator for containers and other workloads.

Container orchestrators are platform-agnostic and can be deployed on:

Bare metal servers
Virtual machines (VMs)
On-premises infrastructure
Public clouds (AWS, Azure, Google Cloud, etc.)
Hybrid cloud environments

For instance, Kubernetes can be deployed on a local machine, in a private data center, or across public cloud services like AWS EC2, Google Compute Engine, or OpenStack.

Understanding Kubernetes

What Is Kubernetes?

Key features of Kubernetes:

Automated scheduling: Assigns containers to nodes based on resource requirements and constraints.
Extensibility: Supports custom resources and controllers without modifying the core codebase.
Self-healing: Monitors container health and replaces failed or unresponsive containers automatically.
Service discovery and load balancing: Assigns stable DNS names and IP addresses to services, distributing network traffic evenly across pods.
Automated rollouts and rollbacks: Manages application updates and configuration changes incrementally, with automatic rollbacks on failure.
Secret and configuration management: Separates sensitive data and configuration from application code, injecting secrets securely into the runtime environment.
Storage orchestration: Mounts persistent storage from various sources dynamically, based on declarative configuration.
Batch and job processing: Supports batch jobs, cron jobs, and long-running tasks with automatic retries and failure handling.

Managed Kubernetes-as-a-Service (KaaS)

Managed Kubernetes offerings simplify setup and operations, allowing you to provision production-grade clusters with minimal effort. Examples include:

Amazon EKS
Azure Kubernetes Service (AKS)
Google Kubernetes Engine (GKE)

These platforms handle cluster provisioning, scaling, patching, and security, enabling teams to focus on application development.

Kubernetes Architecture Overview

A Kubernetes cluster consists of two main node types:

Control plane nodes: Manage the cluster and maintain its desired state.
Worker nodes: Run the containerized applications.

Image credits : https://trainingportal.linuxfoundation.org

Control Plane Node

The control plane is the brain of the Kubernetes cluster, managing cluster state, responding to user requests, scheduling workloads, and ensuring the desired state matches the actual state. Users interact with the control plane using the Kubernetes API—through the CLI (kubectl), a web UI (Dashboard), or external tools.

Core components:

API Server (kube-apiserver): Exposes the Kubernetes API, validating and processing requests, and communicating with etcd.
Scheduler (kube-scheduler): Assigns pods to nodes based on resource availability and constraints.
Controller Manager (kube-controller-manager): Runs background reconciliation loops to maintain the desired cluster state.
Cloud Controller Manager: Integrates the cluster with cloud provider APIs for storage, load balancing, and node management.
etcd: Stores all configuration and state data for the cluster, using the Raft consensus algorithm for leader election and fault tolerance.

Image credits : https://trainingportal.linuxfoundation.org

For high availability(HA), replicate control plane nodes and configure them in HA mode. In HA setups, one node acts as the leader, while others remain synchronized and ready to take over if needed.

Worker Node

Pods are the smallest deployable units in Kubernetes. A pod can contain one or more containers sharing the same network and storage context.

Image credits : https://trainingportal.linuxfoundation.org

Worker nodes host the containerized applications in pods. Each worker node contains the necessary services to run and manage these pods:

Container Runtime: Executes containers. Supported runtimes include containerd, CRI-O, and Docker (via cri-dockerd).
Kubelet: Agent that communicates with the control plane and manages pods on the node.
CRI Shim: Interfaces between the Kubelet and container runtime using the Container Runtime Interface (CRI).
kube-proxy: Manages network rules and forwards traffic to the correct pods based on Kubernetes services.
Add-ons: Optional services like DNS, logging, monitoring, and dashboards.

Networking in Kubernetes

Kubernetes networking supports four main types of communication:

Container-to-container: Containers in the same pod communicate over localhost.
Pod-to-pod: Uses the "IP-per-pod" model, with each pod receiving a unique IP.
Service-to-pod: Enables load-balanced access to pods using stable service endpoints.
External-to-service: Routes external traffic into the cluster via NodePorts, Ingress, or LoadBalancers.

Container Network Interface (CNI)

Kubernetes relies on the CNI specification to configure networking. Common CNI plugins include:

Flannel
Calico
Cilium
Weave

These plugins handle IP allocation, routing, and network policies.

Kubernetes Extensibility and Ecosystem

Kubernetes has a modular, pluggable architecture, supporting the development of:

Custom resources and operators
Custom APIs and admission controllers
Custom scheduling rules and plugins

This flexibility enables you to tailor Kubernetes to your specific needs, especially in complex microservices environments.

Installing Kubernetes

You can install Kubernetes using several cluster configurations, each serving different use cases:

All-in-One Single-Node Installation: Installs both control plane and worker components on a single node. Ideal for learning, development, and testing. Not recommended for production due to lack of high availability and scalability.
Single-Control Plane and Multi-Worker Installation: Includes a single control plane node running a stacked etcd instance, managing multiple worker nodes. Suitable for small-scale environments but introduces a single point of failure.
Single-Control Plane with External etcd and Multi-Worker Installation: The control plane runs independently from an external etcd instance, improving data durability. The single control plane manages multiple worker nodes.
Multi-Control Plane and Multi-Worker Installation: High-availability setup with multiple control plane nodes, each running a stacked etcd instance forming an HA etcd cluster. Offers better fault tolerance.
Multi-Control Plane with External etcd and Multi-Worker Installation: The most robust and production-ready configuration. Each control plane node connects to a dedicated external etcd instance, all configured in a highly available cluster. Ensures maximum resilience and scalability.

As cluster complexity increases, so do the hardware and infrastructure requirements. For production environments, use a multi-node setup with high availability and redundant control planes.

When planning infrastructure, consider:

Environment: Bare metal, public cloud, private cloud, or hybrid cloud?
Operating System: Red Hat-based, Debian-based, or Windows OS?
Networking: Which CNI plugin best fits your needs?

Next Steps

For more information, refer to:

A Developer's Guide to Kubernetes Components

hridyesh bisht — Wed, 07 May 2025 07:56:08 +0000

Kubernetes is the backbone of modern cloud-native applications. It simplifies deploying, scaling, and managing containerized workloads. But for developers, understanding its core concepts—like Pods, Deployments, and Services—is essential to building scalable and resilient apps.

In this guide, you learn about Kubernetes components from a developer’s point of view, complete with real-life use cases and visual diagrams.

Understanding Kubernetes Objects

Kubernetes uses a declarative object model. Each object’s spec defines the desired state. The status reflects the current state. Common fields are:

apiVersion: API version (e.g., v1)
kind: Object type (e.g., Pod, Service)
metadata: Name, namespace, labels
spec: Desired configuration
status: System-managed status

For certain objects like Secrets and ConfigMaps, data and stringData fields are used to store key-value pairs.Kubernetes API Server accepts object definitions in a JSON format, most often such definition manifests in a YAML format which is converted by kubectl in a JSON payload and sent to the API Server.

Enable Autocompletion (Optional)

Autocompletion enhances your CLI experience. For example, in Bash:

source <(minikube completion bash)

Nodes

A node is a physical or virtual machine in a Kubernetes cluster. Each node runs the following:

kubelet: Ensures containers in a Pod are running.
kube-proxy: Manages network rules for communication.
Container runtime: Runs containers (e.g., Docker, containerd).

In a cloud-native e-commerce app, each node could run pods handling different services—payments, inventory, or recommendations.

There are two types of nodes:

Control plane nodes: Manage the cluster. They run the API server, scheduler, controller manager, and etcd.
Worker nodes: Run application workloads.

Image credits : https://trainingportal.linuxfoundation.org

Namespaces

Namespaces partition cluster resources and isolate workloads. They allow teams to share a cluster without interfering with each other.

Default namespaces:

default: For user-defined resources.
kube-system: For system-level components.
kube-public: Readable by all users.
kube-node-lease: Used for node heartbeats.

If multiple users and teams use the same Kubernetes cluster you can partition the cluster into virtual sub-clusters using Namespaces. The names of the resources/objects created inside a Namespace are unique, but not across Namespaces in the cluster.

To list all namespaces:

kubectl get namespaces

To create a namespace:

kubectl create namespace \<namespace-name\>

Namespaces support:

Unique resource names within each namespace
Resource isolation by team, project, or environment
Resource quotas and limits

A good practice, however, is to create additional Namespaces, as desired, to virtualize the cluster and isolate users, developer teams, apps, or tiers.

Pods

A Pod is the smallest deployable unit in Kubernetes. It can contain one or more containers that share:

The same network namespace
Storage volumes

Pods are ephemeral and typically managed by higher-level objects like Deployments.

Image credits : https://trainingportal.linuxfoundation.org

Example, nginx-pod.yaml:

apiVersion: v1  
kind: Pod  
metadata:  
  name: nginx-pod  
spec:  
  containers:  
  \- name: nginx  
    image: nginx:1.22.1  
    ports:  
    \- containerPort: 80

Apply the manifest:

kubectl apply \-f nginx-pod.yaml

Generate the manifest without creating the Pod:

kubectl run nginx-pod \--image=nginx:1.22.1

Labels and Selectors

Labels are key-value pairs used to organize, select, and manage Kubernetes objects. They are used by controllers and services to organize and manage resources, hence many objects can have the same Label(s).

Label Selectors:

Equality-based: Select resources matching specific key-value pairs.
- kubectl get pods -l env=dev
Set-based: Select resources matching a set of values.
- kubectl get pods -l 'env in (dev, qa)'

Image credits : https://trainingportal.linuxfoundation.org

Use labels to group resources logically (e.g., by environment, app version).

ReplicaSets

A ReplicaSet ensures a specific number of identical Pods are always running.

Key features:

Self-healing: Replaces failed Pods
Scalable: Supports manual and automated scaling
Uses label selectors to identify Pods

For example, consider a ReplicaSet with a replica count set to 3 for a specific Pod template. Pod-1, Pod-2, and Pod-3 are identical, running the same app container image, being cloned from the same Pod template.

Although the three Pod replicas are said to be identical, they have unique Pod name and IP address. The Pod object ensures that the application can be individually placed on any worker node of the cluster as a result of the scheduling process.

YAML Example

apiVersion: apps/v1  
kind: ReplicaSet  
metadata:  
  name: frontend  
  labels:  
    app: guestbook  
spec:  
  replicas: 3  
  selector:  
    matchLabels:  
      app: guestbook  
  template:  
    metadata:  
      labels:  
        app: guestbook  
    spec:  
      containers:  
      \- name: php-redis  
        image: gcr.io/google\_samples/gb-frontend:v3

To create the ReplicaSet:

kubectl create \-f redis-rs.yaml

Assume that one of the Pods is forced to unexpectedly terminate (due to insufficient resources, timeout, its hosting node has crashed, etc.), causing the current state to no longer match the desired state.

The ReplicaSet detects that the current state is no longer matching the desired state and triggers a request for an additional Pod to be created, thus ensuring that the current state matches the desired state.

Image credits : https://trainingportal.linuxfoundation.org

Deployments

Deployments manage the creation, deletion, and updates of Pods. A Deployment automatically creates a ReplicaSet, which then creates a Pod.

There is no need to manage ReplicaSets and Pods separately, the Deployment will manage them on our behalf.

YAML Example

apiVersion: apps/v1  
kind: Deployment  
metadata:  
  name: nginx-deployment  
  labels:  
    app: nginx-deployment  
spec:  
  replicas: 3  
  selector:  
    matchLabels:  
      app: nginx-deployment  
  template:  
    metadata:  
      labels:  
        app: nginx-deployment  
    spec:  
      containers:  
      \- name: nginx  
        image: nginx:1.20.2  
        ports:  
        \- containerPort: 80

** Apply the manifest:**

kubectl apply \-f def-deploy.yaml

When you update the Pod template (for example, a container image), Kubernetes performs a rolling update. Each update creates a new ReplicaSet and marks it as a new revision.

Once the rolling update has completed, the Deployment will show both ReplicaSets A and B, where A is scaled to 0 (zero) Pods, and B is scaled to 3 Pods. This is how the Deployment records its prior state configuration settings, as Revisions.

Image credits : https://trainingportal.linuxfoundation.org

When you update the Pod template (for example, a container image), Kubernetes performs a rolling update. Each update creates a new ReplicaSet and marks it as a new revision.

DaemonSets

A DaemonSet ensures a Pod runs on all (or some) Nodes in the cluster. It's ideal for running background agents (e.g., log collectors, monitoring tools).

In contrast, the ReplicaSet and Deployment operators by default have no control over the scheduling and placement of multiple Pod replicas on the same Node.

YAML Example

apiVersion: apps/v1  
kind: DaemonSet  
metadata:  
  name: fluentd-agent  
  labels:  
    k8s-app: fluentd-agent  
spec:  
  selector:  
    matchLabels:  
      k8s-app: fluentd-agent  
  template:  
    metadata:  
      labels:  
        k8s-app: fluentd-agent  
    spec:  
      containers:  
      \- name: fluentd  
        image: quay.io/fluentd\_elasticsearch/fluentd:v4.5.2

Create the DaemonSet:

kubectl create \-f fluentd-ds.yaml

Whenever a Node is added to the cluster, a Pod from a given DaemonSet is automatically placed on it. When any one Node crashes or it is removed from the cluster, the respective DaemonSet operated Pods are garbage collected. If a DaemonSet is deleted, all Pod replicas it created are deleted as well.

Authentication, Authorization, and Admission Control

To manage Kubernetes resources, all API requests go through three control stages:

Authentication: Authenticate a user based on credentials provided as part of API requests.
Authorization: Authorizes the API requests submitted by the authenticated user.
Admission Control: Software modules that validate and/or modify user requests.

Image credits : https://trainingportal.linuxfoundation.org

Authentication

Authentication verifies the identity of a user or service making a request to the API server. Kubernetes doesn’t store user objects but supports various authentication methods:

Different user types:

Normal users: Managed externally (e.g., client certificates, static token files, OIDC).
Service accounts: Used by in-cluster processes. Automatically created per namespace and mount credentials into pods. The Service Accounts are tied to a particular Namespace and mount the respective credentials to communicate with the API server as Secrets.

If properly configured, Kubernetes can also support anonymous requests, along with requests from Normal Users and Service Accounts.

Authorization

Authorization determines whether an authenticated user is allowed to perform an action.

More than one module can be configured for one Kubernetes cluster, and each module is checked in sequence. If any authorizer approves or denies a request, then that decision is returned immediately.

Supported Modes

Node Authorization: Grants kubelet access to node- and pod-related APIs.
Attribute-Based Access Control (ABAC): Policy-based access using user attributes.
- To enable ABAC mode, you must start the API server with the --authorization-mode=ABAC option, while specifying the authorization policy with --authorization-policy-file=PolicyFile.json.
Webhook: Sends authorization requests to an external service.
- To enable the Webhook authorizer, we need to start the API server with the --authorization-webhook-config-file=SOME_FILENAME option, where SOME_FILENAME is the configuration of the remote authorization service.
Role-Based Access Control (RBAC) (Recommended).

Example: Role granting pod read access in lfs158 namespace

apiVersion: rbac.authorization.k8s.io/v1  
kind: Role  
metadata:  
  namespace: lfs158  
  name: pod-reader  
rules:  
\- apiGroups: \[""\]  
  resources: \["pods"\]  
  verbs: \["get", "watch", "list"\]

In Kubernetes, multiple Roles can be attached to subjects like users, service accounts, etc. In RBAC, you can create two kinds of Roles:

Role: Grants namespace-scoped permissions.
ClusterRole: Grants cluster-wide permissions.

Once the role is created, youcan bind it to users with a RoleBinding object. There are two kinds of RoleBindings:

RoleBinding: Binds a Role or ClusterRole to users/groups/service accounts in a namespace.
ClusterRoleBinding: Binds a ClusterRole at the cluster scope.

Example: RoleBinding for user bob

apiVersion: rbac.authorization.k8s.io/v1  
kind: RoleBinding  
metadata:  
  name: pod-read-access  
  namespace: lfs158  
subjects:  
\- kind: User  
  name: bob  
  apiGroup: rbac.authorization.k8s.io  
roleRef:  
  kind: Role  
  name: pod-reader  
  apiGroup: rbac.authorization.k8s.io

Admission Control

Admission controllers validate or modify API requests after authentication and authorization but before persistence.

Controller Types

Validating: Check request validity.
Mutating: Modify request objects.

Examples

LimitRanger: Enforces resource limits.
ResourceQuota: Enforces resource quotas.
DefaultStorageClass: Sets default storage class.
AlwaysPullImages: Forces images to always be pulled.

Enable admission controllers with:

\--enable-admission-plugins=NamespaceLifecycle,ResourceQuota,PodSecurity,DefaultStorageClass

Custom controllers can be created as admission webhooks to support dynamic, external validation or mutation.

Accessing Application Pods

Each Pod is assigned a dynamic IP address. If a Pod is restarted, Kubernetes assigns a new IP. If you're connecting directly to a Pod IP, you'll lose access when the Pod is replaced.

Image credits : https://trainingportal.linuxfoundation.org

Example: Accessing Pods by IP

For example, a client accesses Pods using their individual IPs. If one Pod fails, a new Pod is created with a different IP. The client must then detect and update its target IPs, which adds complexity and increases overhead.

To overcome this situation, Kubernetes provides a higher-level abstraction called Service, which logically groups Pods and defines a policy to access them. This grouping is achieved via Labels and Selectors.

**apiVersion: apps/v1**  
**kind: Deployment**  
**metadata:**  
  **labels:**  
    **app: frontend**  
  **name: frontend**  
**spec:**  
  **replicas: 3**  
  **selector:**  
    **matchLabels:**  
      **app: frontend**  
    **template:**  
      **metadata:**  
        **labels:**  
          **app: frontend**  
      **spec:**  
        **containers:**  
        **\- image: frontend-application**  
        **name: frontend-application**  
        **ports:**  
        **\- containerPort: 5000**

Services

A Kubernetes Service provides a stable network endpoint for a group of Pods. It automatically routes traffic to healthy Pods and load balances requests.

Services use label selectors to identify target Pods.

Create a Service to Expose Pods

First, define a Deployment that runs your application and labels the Pods:

apiVersion: apps/v1  
kind: Deployment  
metadata:  
  name: frontend  
spec:  
  replicas: 3  
  selector:  
    matchLabels:  
      app: frontend  
  template:  
    metadata:  
      labels:  
        app: frontend  
    spec:  
      containers:  
        \- name: frontend-application  
          image: frontend-application  
          ports:  
            \- containerPort: 5000

Then, expose these Pods using a Service:

apiVersion: v1  
kind: Service  
metadata:  
  name: frontend-svc  
spec:  
  selector:  
    app: frontend  
  ports:  
    \- port: 80  
      targetPort: 5000  
      protocol: TCP

By default, the Service type is ClusterIP, which exposes the Service only within the cluster.

To apply the Service:

kubectl apply \-f frontend-svc.yaml

Or use kubectl expose:

kubectl expose deployment frontend \--name=frontend-svc \\  
  \--port=80 \--target-port=5000`

How Services Group Pods

Services use label selectors to identify groups of Pods. You can define separate Services for each group:

frontend-svc — targets Pods with app=frontend
db-svc — targets Pods with app=db

Image credits : https://trainingportal.linuxfoundation.org

When you create a Service, Kubernetes:

Assigns it a ClusterIP (accessible only inside the cluster)
Maps that ClusterIP to a list of Pod IPs and ports (called Endpoints)
Uses kube-proxy to route traffic based on IP rules

To view the Service and its endpoints:

kubectl get service,endpoints frontend-svc

The client connects to the Service via its ClusterIP. The Service forwards traffic to one of the selected Pods and performs load balancing.

The user/client now connects to a Service via its ClusterIP, which forwards traffic to one of the Pods attached to it. A Service provides load balancing by default while selecting the Pods for traffic forwarding.

Load Balancing and Failover

A Service balances traffic across all healthy Pods. When a Pod is replaced, the Service updates its endpoints list and redirects traffic to the new Pod—no changes needed in client configuration.

Note: Each endpoint includes the Pod's IP and its target port.

Kube-proxy

In Kubernetes, each node runs kube-proxy, a network proxy that maintains network rules on nodes. It enables communication to your Pods from network sessions inside or outside of your cluster.

Each node runs kube-proxy, which:

Service Management: kube-proxy watches the Kubernetes API for changes in Service and Endpoint objects, updating the node's network rules accordingly.
Traffic Routing: It uses either iptables or IPVS to handle traffic routing. By default, iptables is used, which is simple and well-supported but less efficient than IPVS.

For each new Service, kube-proxy configures iptables rules on each node to capture traffic for the Service's ClusterIP and forward it to one of the Service's endpoints. This enables any node to receive external traffic and route it internally based on the iptables rules. When a Service is removed, kube-proxy deletes the corresponding iptables rules from all nodes.

The kube-proxy agent runs on every node, and iptables rules are redundantly populated across nodes. Each iptables instance stores routing rules for the entire cluster, ensuring that Service objects implement distributed load balancing.

Image credits : https://trainingportal.linuxfoundation.org

Traffic Policies

Kubernetes Services support traffic policies that influence routing decisions:

Cluster (Default): Routes traffic to all ready endpoints, regardless of their node.
Local: Routes traffic only to endpoints on the same node as the client. If no local endpoints are available, the traffic is dropped.

You can configure these policies in your Service manifest:

apiVersion: v1  
kind: Service  
metadata:  
  name: frontend-svc  
spec:  
  selector:  
    app: frontend  
  ports:  
    \- protocol: TCP  
      port: 80  
      targetPort: 5000  
  internalTrafficPolicy: Local  
  externalTrafficPolicy: Local

Service Discovery

Kubernetes supports two service discovery mechanisms:

Environment Variables: For each active Service, Kubernetes injects environment variables into new Pods. For example: REDIS_MASTER_SERVICE_PORT.
- Note: These variables are set only when the Pod starts.
DNS-Based Discovery: Kubernetes DNS creates names like: my-svc.my-namespace.svc.cluster.local

This allows Services to be discoverable within the cluster.

ServiceType

Kubernetes Services can be exposed in different ways, defined by the type field:

Is only accessible within the cluster.
Is accessible from within the cluster and the external world.
Maps to an entity which resides either inside or outside the cluster.

Access scope is decided by ServiceType property, defined when creating the Service.

ClusterIP (default)

Exposes the Service on an internal IP, making it accessible only within the cluster.

Image credits : https://trainingportal.linuxfoundation.org

NodePort

Exposes the Service on a static port on each node’s IP. A ClusterIP Service, to which the NodePort Service routes, is automatically created.

LoadBalancer

Exposes the Service externally using a cloud provider's load balancer. NodePort and ClusterIP Services, to which the external load balancer routes, are automatically created.

Image credits : https://trainingportal.linuxfoundation.org

Note: Requires cloud provider support.

ExternalIP

Maps a Service to an external IP address. Traffic that is ingressed into the cluster with the ExternalIP (as destination IP) on the Service port, gets routed to one of the Service endpoints

Image credits : https://trainingportal.linuxfoundation.org

The administrator must configure external routing.

ExternalName

Maps the Service to an external DNS name using a CNAME record. No proxying occurs.

Multi-Port Services

Expose multiple ports in a single Service:

apiVersion: v1  
kind: Service  
metadata:  
  name: my-service  
spec:  
  selector:  
    app: myapp  
  type: NodePort  
  ports:  
    \- name: http  
      protocol: TCP  
      port: 8080  
      targetPort: 80  
      nodePort: 31080  
    \- name: https  
      protocol: TCP  
      port: 8443  
      targetPort: 443  
      nodePort: 31443

This is a helpful feature when exposing Pods with one container listening on more than one port, or when exposing Pods with multiple containers listening on one or more ports.

Port Forwarding for Local Testing

Use kubectl to forward a local port to a Service for testing:

kubectl port-forward svc/frontend-svc 8080:80

This is useful for debugging applications without exposing the Service externally.

Kubernetes Ingress

Kubernetes Services support internal routing, but defining routing logic per Service leads to duplication and limited flexibility. Ingress decouples routing rules from individual Services and acts as a centralized entry point for external traffic.

Ingress defines HTTP and HTTPS routing rules and acts as a single entry point for external traffic into your cluster. It configures a Layer 7 (application layer) load balancer and supports the following capabilities:

TLS termination: Offload SSL at the edge.
Name-based virtual hosting: Route traffic by hostname.
Fanout routing: Route traffic by URL path.
Custom routing rules: Use annotations to enable advanced behaviors.
Load balancing: Distribute traffic across Service backends.

Image credits : https://trainingportal.linuxfoundation.org

Instead of accessing a Service directly, clients connect to the Ingress endpoint. The Ingress resource defines routing rules that forward requests to the appropriate Service based on hostnames and URL paths.

Note: The Ingress resource itself does not handle traffic. An Ingress Controller—such as NGINX—interprets the rules and manages request forwarding.

Example: Name-Based Virtual Hosting

Use this pattern to route traffic based on the request hostname.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: virtual-host-ingress
  namespace: default
  annotations:
    nginx.ingress.kubernetes.io/service-upstream: "true"
spec:
  ingressClassName: nginx
  rules:
  - host: blue.example.com
    http:
      paths:
      - path: /
        pathType: ImplementationSpecific
        backend:
          service:
            name: webserver-blue-svc
            port:
              number: 80
  - host: green.example.com
    http:
      paths:
      - path: /
        pathType: ImplementationSpecific
        backend:
          service:
            name: webserver-green-svc
            port:
              number: 80

In this example, requests to blue.example.com and green.example.com are routed to their respective backend Services.

Image credits : https://trainingportal.linuxfoundation.org

Example: Path-Based Fanout Routing

Use this pattern to route traffic based on the URL path.

apiVersion: networking.k8s.io/v1  
kind: Ingress  
metadata:  
  name: fan-out-ingress  
  namespace: default  
  annotations:  
    nginx.ingress.kubernetes.io/service-upstream: "true"  
spec:  
  ingressClassName: nginx  
  rules:  
  \- host: example.com  
    http:  
      paths:  
      \- path: /blue  
        pathType: ImplementationSpecific  
        backend:  
          service:  
            name: webserver-blue-svc  
            port:  
              number: 80  
      \- path: /green  
        pathType: ImplementationSpecific  
        backend:  
          service:  
            name: webserver-green-svc  
            port:  
              number: 80

Requests to example.com/blue and example.com/green are routed to the corresponding Services.

The ingress is fulfilled by an Ingress Controller, which is a reverse proxy responsible for traffic routing based on rules defined in the Ingress resource.

Ingress Controller

The Ingress resource only defines routing rules. It does not route traffic on its own. An Ingress Controller is responsible for fulfilling these rules.

An Ingress Controller:

Monitors the Kubernetes API for changes to Ingress resources
Configures the Layer 7 load balancer
Acts as a reverse proxy for external traffic

Popular Ingress Controllers

NGINX Ingress Controller
AWS Load Balancer Controller
GCE L7 Load Balancer
Istio Ingress

Note: Each controller may require specific annotations. Always specify the correct ingressClassName and annotations for compatibility.

Deploy an Ingress Resource

After enabling the Ingress Controller, deploy your Ingress resource using:

kubectl create \-f virtual-host-ingress.yaml

Annotations

Annotations allow you to store non-identifying metadata on Kubernetes objects in key-value pairs. They're not used for selection but provide auxiliary information to tools.

Common use cases:

Store build/release IDs, Git branch names.
Reference logging or monitoring tools.
Annotate ingress controller data.

Example: Add annotations during Deployment creation

apiVersion: apps/v1  
kind: Deployment  
metadata:  
  name: webserver  
  annotations:  
    description: "Deployment PoC \- 2 Mar 2022"  
spec:  
  ...

Resource Quotas and Limit Ranges

In multi-tenant Kubernetes clusters, it's essential to prevent any single user or team from consuming excessive resources. Kubernetes provides ResourceQuota and LimitRange objects to enforce such constraints.

Resource Quotas

ResourceQuota objects limit the aggregate resource consumption per namespace. They can restrict:

Object counts (Pods, Services, ConfigMaps, etc.)
Compute resources (CPU, memory).
Storage resources (PersistentVolumeClaims).

Example:

apiVersion: v1  
kind: ResourceQuota  
metadata:  
  name: compute-resources  
  namespace: devspace  
spec:  
  hard:  
    requests.cpu: "1"  
    limits.cpu: "2"  
    requests.memory: 1Gi  
    limits.memory: 2Gi

Limit Ranges

LimitRange objects set default request and limit values for Pods or Containers within a namespace. They ensure that containers don't consume excessive resources and help maintain cluster stability.

Example:

apiVersion: v1
kind: LimitRange
metadata:
 name: cpu-limits
 namespace: devspace
spec:
 limits:
 - default:
     cpu: 500m
   defaultRequest:
     cpu: 500m
   max:
     cpu: "1"
   min:
     cpu: 100m
   type: Container

Autoscaling

Autoscaling in Kubernetes adjusts the number of running objects based on resource utilization, availability, and requirements. There are several types of autoscalers:

Horizontal Pod Autoscaler (HPA)

HPA automatically scales the number of pod replicas based on CPU utilization or other select metrics.

kubectl autoscale deploy myapp \--min=2 \--max=10 \--cpu-percent=80

Vertical Pod Autoscaler (VPA)

VPA adjusts the CPU and memory requests and limits for containers based on usage. It helps optimize resource allocation for individual pods.

Cluster Autoscaler

The Cluster Autoscaler adjusts the number of nodes in your cluster when pods fail to launch due to insufficient resources or when nodes in the cluster are underutilized. In Azure Kubernetes Service (AKS), it's recommended to let the Kubernetes Cluster Autoscaler manage the required scale settings.

Job Scheduling

Jobs

A Job creates one or more Pods to perform a specific task and ensures that the specified number of Pods successfully terminate. Jobs are useful for batch processing tasks.

Configuration Options:

parallelism: Number of Pods to run in parallel.
completions: Number of successful completions needed.
activeDeadlineSeconds: Duration in seconds the Job may be active.
backoffLimit: Number of retries before marking the Job as failed.
ttlSecondsAfterFinished: Time to retain the Job after completion.

Example:

apiVersion: batch/v1  
kind: Job  
metadata:  
  name: data-cleanup  
spec:  
  template:  
    spec:  
      containers:  
      \- name: cleaner  
        image: busybox  
        command: \["sh", "-c", "cleanup.sh"\]  
      restartPolicy: Never  
  backoffLimit: 4

CronJobs

CronJobs schedule Jobs to run periodically at fixed times, dates, or intervals. They are useful for recurring tasks like backups or report generation .

Configuration Options:

schedule: Cron format schedule string.
startingDeadlineSeconds: Deadline in seconds for starting the Job if it misses its scheduled time.
concurrencyPolicy: Specifies how to treat concurrent executions.

Example:

apiVersion: batch/v1  
kind: CronJob  
metadata:  
  name: db-backup  
spec:  
  schedule: "0 1 \* \* \*"  
  jobTemplate:  
    spec:  
      template:  
        spec:  
          containers:  
          \- name: backup  
            image: backup-tool  
          restartPolicy: OnFailure

StatefulSets

StatefulSets manage the deployment and scaling of Pods with unique, persistent identities. Unlike Deployments, StatefulSets guarantee the ordering and uniqueness of Pods, making them ideal for stateful workloads.

Key Features

Persistent storage: Each Pod in a StatefulSet gets its own PersistentVolume. This volume is retained across Pod restarts or rescheduling.
Stable network identity: Each Pod receives a unique and consistent DNS name, allowing predictable network communication (for example, pod-0.service-name).
Ordered operations: Pods are created, updated, and deleted in a defined order, one at a time. This ensures safe startup, updates, and shutdowns.

Real-World Example

Use StatefulSets to deploy clustered databases or distributed systems where each node must retain its identity and storage.

Use Case: Cassandra or Redis Clusters

In a Cassandra cluster:

Each node (Pod) requires a stable hostname for cluster gossip protocol.
Each node needs its own storage volume to persist data across rescheduling.

apiVersion: apps/v1  
kind: StatefulSet  
metadata:  
  name: redis  
spec:  
  serviceName: "redis"  
  replicas: 3  
  selector:  
    matchLabels:  
      app: redis  
  template:  
    metadata:  
      labels:  
        app: redis  
    spec:  
      containers:  
        \- name: redis  
          image: redis:7.0  
          volumeMounts:  
            \- name: redis-data  
              mountPath: /data  
  volumeClaimTemplates:  
    \- metadata:  
        name: redis-data  
      spec:  
        accessModes: \[ "ReadWriteOnce" \]  
        resources:  
          requests:  
            storage: 1Gi

Custom Resources

Custom Resources are user-defined API objects that allow you to store and retrieve structured data in Kubernetes. Combined with controllers, they help automate custom workflows or represent external systems inside your cluster.

Custom Resource Definitions (CRDs)

CRDs are the most common way to add custom resources. They allow you to define custom objects without modifying the Kubernetes source code .

apiVersion: apiextensions.k8s.io/v1  
kind: CustomResourceDefinition  
metadata:  
name: databases.example.com  
spec:  
group: example.com  
versions:  
\- name: v1  
served: true  
storage: true  
schema:  
openAPIV3Schema:  
type: object  
properties:  
spec:  
type: object  
properties:  
engine:  
type: string  
scope: Namespaced  
names:  
plural: databases  
singular: database  
kind: Database  
shortNames:  
\- db

Once registered, you can create resources like:

apiVersion: example.com/v1  
kind: Database  
metadata:  
name: my-postgres-db  
spec:  
engine: postgres

Use a CRD to manage custom services like Database, Cache, or Queue, with a controller automating provisioning tasks across your infrastructure.

API Aggregation

API Aggregation is an advanced extension mechanism. It lets you run a separate API server behind the Kubernetes API, delegating requests to custom endpoints.

Your API server must implement Kubernetes-style authentication, authorization, and admission control.
You write and deploy your own API server.

This method is more complex but offers greater flexibility.

Security Contexts

Security Contexts define privilege and access controls for Pods and containers. You can use them to enforce non-root execution, set file system permissions, and limit privilege escalation.

Example: Secure Pod Configuration

apiVersion: v1  
kind: Pod  
metadata:  
  name: secure-pod  
spec:  
  securityContext:  
    runAsUser: 1000       \# Runs Pod as non-root user  
    fsGroup: 2000         \# Shared file system group  
  containers:  
  \- name: app  
    image: busybox  
    securityContext:  
      allowPrivilegeEscalation: false  \# Prevents gaining extra privileges

Note: Always run containers as a non-root user unless absolutely necessary.

Pod Security Admission

Pod Security Admission (PSA) is a built-in admission controller in Kubernetes. It enforces security standards at the namespace level by applying predefined policies.

Restricted: Strictest, enforces non-root and drops capabilities.
Baseline: Reasonably secure defaults.
Privileged: Allows full capabilities—use with caution.

Example: Enable Restricted Policy

kubectl label namespace my-namespace \\ pod-security.kubernetes.io/enforce=restricted

Network Policies

Network Policies control traffic flow to and from Pods. By default, all traffic is allowed unless restricted by a policy.

Example: Allow Only Frontend to Access Database

apiVersion: networking.k8s.io/v1  
kind: NetworkPolicy  
metadata:  
  name: allow-frontend  
spec:  
  podSelector:  
    matchLabels:  
      role: db  
  ingress:  
  \- from:  
    \- podSelector:  
        matchLabels:  
          role: frontend  
    ports:  
    \- port: 5432

Combine multiple policies to fine-tune network security for microservices.

Metrics Server

The Metrics Server is a lightweight resource monitoring component. It provides CPU and memory usage data for Pods and nodes.

Example Commands:

kubectl top pods  
kubectl top nodes

Prometheus

Prometheus is a robust monitoring tool that collects and queries time-series metrics.

Scrapes metrics from applications and Kubernetes components.
Supports alerting and visualizations via Grafana.

Example Use Case:

Monitor HTTP request rates and latency in a web application. Integrate alerts when request rates spike or response times degrade.

Helm: Kubernetes Package Manager

Helm is the de facto package manager for Kubernetes. It enables developers to deploy and manage complex applications using templated YAML files—called charts—that follow DRY (Don't Repeat Yourself) principles.

Package and deploy complex applications like WordPress, NGINX Ingress, or custom APIs using reusable charts. A chart includes:

Template files for resource definitions
Configuration values
Metadata (e.g., chart name, version)

Charts can be stored in repositories—similar to how .deb or .rpm packages are stored for Linux distributions—or in container registries.

Key Benefits

Reusable templates: Use Helm charts to define, install, and upgrade Kubernetes applications.
Version control: Roll back to previous releases with a single command.
GitOps friendly: Integrates well with tools like Argo CD and Flux for continuous delivery.

How Helm Works

Helm is a command-line tool that runs alongside kubectl and uses your existing kubeconfig file to connect securely to your cluster. It performs the following actions:

Searches chart repositories based on your criteria.
Downloads the selected chart to your local system.
Uses the Kubernetes API to deploy the resources defined in the chart.

You can also use Helm to upgrade or delete deployments.

Benefits:

DRY deployment configs
Easy upgrades/rollbacks
Integrates with GitOps workflows

Service Mesh: Advanced Service Communication

A service mesh abstracts communication between microservices into a dedicated infrastructure layer. It helps manage:

Secure communication: Enforces mutual TLS (mTLS) between Pods.
Advanced traffic routing: Supports strategies like canary releases and A/B testing.
Observability: Collects traffic metrics, latency data, and failure insights without modifying app code.

Architecture Overview

Each Pod includes a sidecar proxy that handles communication and policy enforcement. These proxies form the data plane, while a central control plane manages configuration and telemetry.

Data Plane: Handles service traffic. This usually includes sidecar proxies injected into each Pod, or node-level proxies in some implementations.
Control Plane: Manages configuration, service discovery, policy enforcement, and telemetry.

The sidecar proxy intercepts all inbound and outbound traffic to the Pod, enabling consistent policy enforcement and observability without modifying the application code.

Decoupling of networking logic from application code

Popular Service Meshes

Name	Notable Feature
Istio	Full-featured, widely adopted
Linkerd	Lightweight and CNCF-supported

Next steps

For more information, refer to:

Getting Started with minikube for Kubernetes

hridyesh bisht — Tue, 06 May 2025 15:57:10 +0000

Minikube is a lightweight tool that runs a full Kubernetes cluster on your local machine. It's ideal for developers who want to test applications, simulate production behavior, or learn Kubernetes without using cloud resources. Whether you’re developing microservices, testing Ingress routing, or deploying with Helm, Minikube provides a fast, isolated environment with full Kubernetes features.

Key Features:

Installs and manages Kubernetes components
Supports single-node and multi-node clusters
Offers both CLI and web-based access
Enables custom profiles and driver selection

Hardware Requirements

Minikube provisions local resources for cluster nodes. Minimum recommended per node:

CPU: 2 cores
Memory: 2 GB (4–8 GB recommended)
Storage: 20 GB
Internet: Required for initial setup and image downloads

Ensure your system has enough resources for both Minikube and any workloads.

Software Requirements

Minikube requires a supported Type-2 hypervisor or container runtime. It uses these to isolate Kubernetes from your host OS.

OS	Hypervisors	Container Runtimes
Linux	VirtualBox, KVM2, QEMU	Docker, Podman
macOS	VirtualBox, HyperKit, VMware Fusion, Parallels	Docker, Podman
Windows	VirtualBox, Hyper-V, VMware Workstation, QEMU	Docker, Podman

Note: On Linux, you can use --driver=none to run Minikube directly on the host. This bypasses isolation and requires root access.

Create and Manage Clusters

Run the following to create your first cluster:

minikube start

This command:

Detects or uses the specified driver
Creates a VM or container with:
- 2 CPUs
- 6 GB memory
- 20 GB storage
Bootstraps Kubernetes using kubeadm
Installs Docker as the default container runtime
Creates a default profile to track cluster state

To create a multi-node cluster:

minikube start \--profile custom-cluster \--nodes 3 \--kubernetes-version=v1.28.1 \--driver=docker

Use the --profile flag to create and manage multiple clusters.

Cluster Profiles

Minikube uses profiles to store cluster configuration. By default, all commands apply to the default profile. To work with a custom profile:

minikube stop \--profile custom-cluster  
minikube start  \# Uses the default profile

Use --profile in commands to manage multiple environments.

Enable Autocompletion (Optional)

Autocompletion enhances your CLI experience. For example, in Bash:

source <(minikube completion bash)

Accessing Minikube

kubectl (CLI)

Minikube includes a bundled version of kubectl:

minikube kubectl \-- get pods

However, for convenience, install kubectl separately. It automatically detects your Minikube cluster.

Kubernetes Dashboard (Web UI)

To use the web-based UI:

addons enable metrics-server  
minikube addons enable dashboard  
minikube dashboard

This opens a browser UI for inspecting deployments, pods, and logs.

Kubernetes API Access

The API server is the entry point to the cluster. You can access it via:

kubectl and CLI tools
Web Dashboard
Custom automation using HTTP API calls

Use kubectl proxy to expose the API server:

kubectl proxy

This opens access at http://localhost:8001/.

To explore the API: curl http://localhost:8001/api/v1

To run the proxy in the background:

kubectl proxy &

Access Without a Proxy

To access the Kubernetes API directly, use an authentication token or certificate credentials.

For example, you must generate a token and grant access:

export TOKEN=$(kubectl create token default)  
export APISERVER=$(kubectl config view \--minify \-o jsonpath='{.clusters\[0\].cluster.server}')
curl $APISERVER \--header "Authorization: Bearer $TOKEN" \--insecure

Deploy and Access an NGINX Application Using the Minikube Dashboard

You’ll use the nginx container image from Docker Hub.

Start Minikube: To launch your local Kubernetes cluster, run:
- minikube start
Verify the status:
- minikube status
Launch the Dashboard :Start the web-based Kubernetes Dashboard:
- minikube dashboard

This command opens the dashboard in your default browser. By default, it connects to the default namespace, where all operations occur unless you change the context.

Note: If you reboot your machine or log out and log back in, rerun the minikube dashboard command to reopen the dashboard.

Deploying an Application - Accessing the Dashboard

To deploy the NGINX application:

In the Dashboard, select Deploy.
Choose Show Advanced Options to set:
- Labels
- Namespace
- Resource Requests
Use nginx as the container image.
Select Deploy.

The deployment creates:

A Deployment resource (for example, web-dash)
A ReplicaSet (e.g., web-dash-74d8bd488f)
A Pod (e.g., web-dash-74d8bd488f-dwbzz)

Note: Resource names are unique and may vary in your cluster. The naming pattern follows Kubernetes conventions.

Use the left navigation panel to explore the Deployment, ReplicaSet, and Pod resources.

You can:

View properties by selecting resource names.
Scale the deployment from the vertical three-dots menu.
Delete individual Pods and observe them automatically recreated.
Delete the deployment to remove all Pods.

Once we create the web-dash Deployment, we can use the resource navigation panel from the left side of the Dashboard to display details of Deployments, ReplicaSets, and Pods in the default Namespace.

Access the Application

To access the application: get the Minikube IP

minikube ip

In your browser, go to http://192.168.99.100:\. Replace <NodePort> with the actual port assigned to your service (e.g., 31074).

Minikube opens the application in your browser. You should see the default NGINX welcome page.

We can see the Nginx welcome page, displayed by the webserver application running inside the Pods created. Our requests could be served by either one of the three endpoints logically grouped by the Service since the Service acts as a Load Balancer in front of its endpoints.

Enable Ingress for Routing

Minikube includes the NGINX Ingress Controller as a built-in add-on. To enable it, run:

minikube addons enable ingress

Apply an Ingress manifest:

kubectl apply \-f virtual-host-ingress.yaml

Add entries to `/etc/hosts`:

192.168.99.100 blue.example.com green.example.com

Deploy an Ingress Resource

After enabling the Ingress Controller, deploy your Ingress resource using:

kubectl create \-f virtual-host-ingress.yaml

Configure Liveness, Readiness, and Startup Probes

Probes monitor and manage container health. If a container becomes unresponsive, the kubelet uses probes to take action (e.g., restart the container).

Rather than restarting it manually, we can use a Liveness Probe. Liveness Probe checks on an application's health, and if the health check fails, kubelet restarts the affected container automatically.

Liveness Probes

Use Liveness Probes to detect and recover from unresponsive applications.

Supported types:

Exec command
HTTP GET
TCP Socket
gRPC

Liveness Probe (Exec)

livenessProbe:  
  exec:  
    command: \["cat", "/tmp/healthy"\]  
  initialDelaySeconds: 15  
  periodSeconds: 5

Simulate file removal after startup to trigger restarts. Until the container reports "ready," the Pod will not receive traffic from a Service.

Startup Probe (HTTP)

Use Startup Probes for applications that take a long time to initialize. They prevent premature liveness and readiness checks.

startupProbe:  
  httpGet:  
    path: /startup  
    port: 8080  
  failureThreshold: 30  
  periodSeconds: 10 `

Integrating Low-Code & High-Code Solutions Effectively

hridyesh bisht — Mon, 11 Nov 2024 10:39:38 +0000

Low and high-code solutions address different needs in today's development landscape. Low-code platforms enable fast development with minimal coding, making them ideal for non-developers or projects with tight timelines. High code requires more technical expertise, offering greater customization and control over complex features. Combining low and high-code approaches can provide flexibility and speed, but successful integration and monitoring require a clear strategy.

With over three years of experience in low-code solutions, I often get questions at meetups and hackathons like:

Why should I use low-code solutions?
Can I integrate low-code and high-code solutions?
How do I architect infrastructure that supports low- and high-code?
How can I unify logs and traces to monitor my entire infrastructure?

This blog addresses these questions and helps you:

Decide when to choose low code or high code.
Integrate both solutions seamlessly.
Architect an infrastructure that supports both approaches.
Enable unified logging and tracing with tools like SigNoz.

When to Use Low-Code vs. High-Code

Low-code platforms allow you to build applications with minimal coding, letting you focus on business logic. They offer visual tools, drag-and-drop interfaces, and pre-built components that simplify development. Low-code solution enables you to:

Build and deploy applications quickly using reusable components.
Adapt to changing business needs with flexible, scalable applications.
Automate workflows, reducing the overhead of traditional software development.

Low-code is often the best choice for fast development or projects with limited technical resources. High-code, however, is ideal when customization, scalability, and control are critical. Sometimes, combining both approaches provides an effective balance for different project components. In some cases, combining both approaches can leverage their strengths, effectively balancing different project components.

For example, consider an employee onboarding app that collects employee details, generates tasks, assigns managers, and integrates with HR systems. You have two options:

Low-Code Approach: Use drag-and-drop components for the UI, integrate APIs, automate task generation, and set up notifications.
High-Code Approach: Manually code the UI, APIs, and business logic in Java or Python for full customization.

Benefits of Low-Code for Front-End Components:

Pre-built components: Forms, task management, and notifications are ready-made.
Centralized dashboard: Easily monitor updates, reducing time on code changes and bug fixes.
Built-in connectors: Common HR integrations like SAP reduce custom code requirements.
Rapid prototyping: Build and adapt prototypes based on feedback.

Integrating Low Code and High Code Solutions

Combining low-code and high-code solutions lets you leverage both rapid development and customization. This approach creates a flexible, scalable architecture supporting speed and control. Key benefits include:

Flexibility: Design each component with the best approach for its needs. Use low-code for user interfaces and workflows and high-code for backend logic.
Scalability: Quickly iterate on low-code components for front-end changes while handling complex business logic with high-code.
Efficient collaboration: Non-developers can build front-end features in low-code while developers focus on high-code elements, maintaining functionality.

Despite the benefits, integrating low and high-code solutions comes with challenges that require careful planning:

Compatibility: Low-code platforms may have limited interoperability with custom high-code solutions.
Data synchronization: Ensure seamless data flow between components, maintaining consistent formats and structures.
API limitations: Restricted APIs in low-code platforms can complicate consistent bidirectional integration with high-code systems.

In the earlier example about employee onboarding, the low code for the UI and the high code for backend functions were combined. In an employee onboarding system,

Low-code: Manages front-end tasks, onboarding workflows, and notifications.
High-code: Handles backend processing, complex integrations, analyzing data to track metrics and generating documents.

This approach lets non-technical users manage the onboarding workflow while developers focus on implementing secure and efficient backend operations.

Architecting an Infrastructure with Low-Code and High-Code Solutions

When architecting an infrastructure with both low-code and high-code, address compatibility, data flow, and scalability:

Separation of concerns: Use microservices or modular components to separate low-code and high-code tasks, allowing flexibility in updates and maintenance.
Data flow and APIs: Establish clear data pathways using APIs, creating a bridge between low-code and high-code.
Scalability: Design the infrastructure to scale specific components independently, supporting growth without overhauling the system.

To integrate both, you can use Middleware to manage communication between low- and high-code components using an API gateway. An API gateway centralizes communication, manages authentication, and routes requests to the appropriate services.

The benefits of architecting an Infrastructure with Low-Code and High-Code Solutions are:

Scalability: By separating low-code and high-code tasks, each component can be scaled based on specific usage requirements, optimizing resource allocation.
Efficient Collaboration: Developers and non-developers can collaborate efficiently, with each team focusing on the parts that best align with their expertise.

For example, in the employee onboarding app. You add an extra component:

Middleware: An API gateway centralizes communication, manages authentication, and routes requests to the appropriate services. Unified Logging and Tracing

Combining low-code and high-code solutions adds flexibility but can complicate monitoring. Unified logging and tracing with observability tools like SigNoz provide a centralized view of application performance across both environments. SigNoz’s APM (Application Performance Monitoring) features enable monitoring and troubleshooting through a single dashboard.

Unified logging and tracing across both approaches lets you:

Monitor application performance: View metrics and logs across all components.
Detect issues: Identify bottlenecks in both low-code and high-code elements.
Simplify troubleshooting: Trace issues end-to-end for faster incident resolution.
Optimize infrastructure: Use insights to allocate resources more effectively.

Implementing SigNoz for Unified Observability

Instrument High Code Applications:
1. Integrate OpenTelemetry libraries in your high code applications to capture traces, metrics, and logs. This includes configuring custom traces for critical processes such as API calls, external integrations, and custom logic.
2. High code applications can leverage SigNoz’s SDKs in languages like Python, Java, and Node.js to capture detailed telemetry data.
Connect Low Code Platforms: 1.If your low-code platform supports custom logging or webhooks, configure it to send logs to SigNoz. Alternatively, integrate through REST APIs to stream logs, events, and metrics from the low-code environment into SigNoz.

Use SigNoz’s UI to view real-time logs and traces across low- and high-code components. A unified dashboard allows you to correlate data from different sources, providing comprehensive insights and aiding in root-cause analysis.

Consider an employee onboarding app, and you need to add another layer to your infrastructure. An observability layer to:

Aggregates all telemetry data from low and high code layers, allowing centralized monitoring and troubleshooting.

Present real-time performance insights, helping teams monitor low and high code components.

Best Practices for Unified Observability

Standardize log formats: Ensure consistent log formats across components for seamless parsing.
Use trace IDs: Connect transactions across low-code and high-code environments for end-to-end traceability.
Monitor and adjust metrics regularly: Track key metrics and adjust infrastructure as needed.

Combining low-code and high-code solutions can empower your development strategy, providing the agility to build while retaining control over complex features. You can optimize performance and streamline collaboration across teams by architecting a unified infrastructure and leveraging robust observability tools like SigNoz. Whether you're building rapid prototypes or integrating with advanced systems, a thoughtful blend of low-code and high-code approaches offers the best of both worlds, making your applications adaptable, scalable, and resilient.

How to effectively store and ANALYSE logs in AWS CLOUD

hridyesh bisht — Wed, 22 Mar 2023 15:12:53 +0000

Services and applications typically create logs that contain significant amounts of information. This information is logged and stored on persistent storage, allowing it to be reviewed and analyzed at any time.

By monitoring the data within your logs, you can quickly identify potential issues you want to be made aware of as soon as they occur. Resolving an incident as quickly as possible is paramount for developing real-life solutions.

Having more data about how your environment is running far outweighs the disadvantage of needing more information, especially when it matters to your business in the case of incidents and security breaches.

Some logs can be monitored in real-time, allowing automatic responses to be carried out depending on the data contents of the log. Logs often contain vast amounts of metadata, including date stamps and source information such as IP addresses or usernames.

This blog aims to help you understand how to store and analyze logs in AWS and some recommended practices.

Storing and analyzing logs in AWS can be done using various services and tools, depending on your requirements and use case. Here are some services you can consider:

A. AWS CloudWatch

AWS CloudWatch provides valuable insights into the health and performance of your applications and resources, which can help you optimize their performance, increase availability, and improve the overall customer experience. Various components of Amazon CloudWatch include:

1. Dashboards: You can quickly and easily design different dashboards to represent the data by building your views. For example, you can view all performance metrics and alarms from resources relating to a specific customer.

Once you have built your Dashboards, you can easily share them with other users, even those who may not have access to your AWS account.

Note: The resources within your customised dashboard can be from multiple different regions.

2. Metrics: You can monitor a specific element of an application or resource over time, for example, the number of DiskReadson in an EC2 instance.

Anomaly detection allows CloudWatch to implement machine learning algorithms against your metric data to help detect any activity generally expected outside the normal baseline parameters.

Note: Different services will offer different metrics.

3. Amazon CloudWatch Alarms: You can implement automatic actions based on specific thresholds that you can configure relating to each metric.

For example, you could set an alarm to activate an auto-scaling operation if your CPU utilization of an EC2 instance peaked at 75% for more than 2 minutes.

There are three different states for any alarm associated with a metric,

OK – The metric is within the defined configured threshold.
ALARM – The metric has exceeded the thresholds set.
INSUFFICIENT_DATA – There is insufficient data for the metric to determine the alarm state.

Image credits: https://docs.aws.amazon.com/images/AmazonCloudWatch/latest/monitoring/images/CW-Overview.png

4. CloudWatch EventBridge: You connect applications to various targets, allowing you to implement real-time monitoring and respond to events in your application.

The significant benefit of using CloudWatch EventBridge is that it allows you to implement the event-driven architecture in a real-time decoupled environment.

Various elements of this feature include:

Rules: A rule acts as a filter for incoming streams of event traffic and then routes these events to the appropriate target defined within the rule. The target must be in the same region.
Targets: Targets are where the Rules direct events, such as AWS Lambda, SQS, Kinesis, or SNS. All events received are in JSON format.
Event Buses: It receives the event from your applications, and your rules are associated with a specific event bus.

5. CloudWatch Logs: You have a centralized location to store your logs from different AWS services that provide logs as an output, such as CloudTrail, EC2, VPC Flow logs, etc., in addition to your own applications.

An added advantage of CloudWatch logs comes with the installation of the Unified CloudWatch Agent, which can collect logs and additional metric data from EC2 instances as well as from on-premise services running either a Linux or Windows operating system. This metric data is in addition to the default EC2 metrics that CloudWatch automatically configures for you.

Various types of insights within CloudWatch include Log Insights, Container Insights, and Lambda Insights.

B. AWS Cloud Trail

AWS CloudTrail records and tracks all AWS API requests. It captures an API request made by a user as an event and logs it to a file it stores on S3.

CloudTrail captures additional identifying information for every event, including the requester's identity, the initiation timestamp, and the source IP address.

You can use the data captured by CloudTrail to help you enhance your AWS environment in several ways.

Security analysis tool.
Help resolve and manage day-to-day operational issues and problems.

C. AWS COnfig

AWS Config records and captures resource changes within your environment, allowing you to perform several actions against the data that help optimize resource management in the cloud.

AWS Config can track changes made to a resource and store the information, including metadata, in a Configuration Item (CI) file. This file can also serve as a resource inventory.

It can provide information on who made the change and when through AWS CloudTrail integration. AWS CloudTrail is used with AWS Config to help you identify who made the change and when and with which API.

Note: AWS Config is region specific.

Image credits: https://static.us-east-1.prod.workshops.aws/public/5659531a-ebcf-42ca-bd3f-f5b15e64cda5/static/images/logsinsights/Cloud-Watch-Insights-Query-Results.PNG

D. AWS Cloud front logs

Enabling CloudFront access logs allows you to track each user's request for accessing your website and distribution. These logs contain information about the requests made to your CloudFront distributions, such as the request's date and time, the requester's IP address, the URL path, and the response's status code.

Amazon S3 stores these logs, similar to S3 access logs, providing a durable and persistent storage solution. Although enabling logging is free, S3 will charge you for the storage used.

E. AWS VPC Flow logs

VPC Flow Logs allow you to capture IP traffic information between the network interfaces of resources within your VPC. This data aids in resolving network communication incidents and monitoring security by detecting prohibited traffic destinations.

Note: VPC Flow Logs do not store data in S3 but transmit data to CloudWatch logs.

Before creating VPC Flow Logs, you should know some limitations that may affect their implementation or configuration.

If you have a VPC peered connection, you can only view flow logs of peered VPCs within the same account.
You cannot modify its configuration once you create a VPC Flow Log. Instead, you need to delete it and create a new one to make changes.

You need an IAM role with the appropriate permissions to send your flow log data to a CloudWatch log group. You select this role during the setup configuration of VPC Flow Logs.

Recommended practices for storing and analyzing logs in AWS:

By following these best practices, you can effectively store and analyze logs in AWS and improve the reliability and performance of your applications.

Define a consistent logging format such as JSON or Apache Log Format.
Use log rotation to prevent from taking up too much storage space. You can use AWS Elastic Beanstalk helps to automate log rotation.
Set up alerts to notify you of any critical issues in your logs to quickly resolve any issues. Amazon CloudWatch helps to set up alerts based on predefined thresholds or custom metrics.
Use encryption to protect your log data in transit and at rest. Amazon S3 server-side encryption or Amazon CloudFront field-level encryption helps protect your data,
You should regularly review and analyze your logs to help identify potential issues. AWS services like Amazon Athena, Amazon Elasticsearch Service, and AWS Glue help you gain insights.
Use a log aggregation service to centralize your logs across many AWS services. AWS CloudTrail or Amazon CloudWatch Logs helps to centralize your logs.
Implement a data retention policy that specifies how long you need to keep your log data. To help avoid unnecessary storage costs and ensure compliance with regulatory requirements.

By monitoring the data within your logs, you can quickly identify potential issues you want to be made aware of as soon as they occur. In addition, by combining this monitoring of logs with thresholds and alerts, you can receive automatic notifications of potential issues, threats, and incidents, before they become production issues.

By logging what's happening within your applications, network, and other cloud infrastructure, you can build a performance baseline and establish what's routine and what isn't.

DEV Community: hridyesh bisht

Your Telemetry Has Data. Does It Have Context?

Semantic conventions

Resource attributes vs. span attributes

Resource attributes

Span attributes

Context propagation

What context propagation does not do

Correlating logs, metrics, and traces

Linking logs to traces

Linking metrics to traces with exemplars

How the signals fit together

Common correlation gaps

Validate telemetry correlation

Dash0

Conclusion

Beyond the Collector: How the Open-Source Observability Stack Fits Together

OpenTelemetry isn't your observability backend

Prometheus

Jaeger

OpenSearch

Perses

How do these tools fit together?

What happens when an incident occurs?

Start with one use case

What should you consider before choosing your stack?

Integration and query models

Reliability, storage, and security

Licensing and long-term flexibility

Self-managed stack or unified observability platform?

Dash0: Explore telemetry in one platform

Conclusion

Your telemetry doesn't go straight to the backend: Understanding the OpenTelemetry Collector

Do you need a Collector?

How does the Collector process telemetry?

Receivers: Accept telemetry

Processors: Process telemetry

Exporters: Send telemetry to a destination

Where should you deploy the Collector?

Agent pattern

Gateway pattern

What happens to telemetry inside the pipeline?

Add context to telemetry

Filter sensitive information

Manage telemetry volume

What happens when the backend becomes unavailable?

Monitor the Collector itself

Where does Dash0 fit?

Conclusion: Treat telemetry as a pipeline, not an afterthought

Further reading

Serverless Computing in Kubernetes: A Developer’s Guide

Combine Serverless and Kubernetes?

Serverless in Kubernetes

OpenFaaS

Architecture of OpenFaaS

Interacting with OpenFaaS

Accessing the OpenFaaS Gateway

OpenFaaS CLI

Function and Template Stores

Function Store

Template Store

Observability: Prometheus + Grafana

Create Your First Function

Templating with Jinja2

Controlling HTTP Responses in OpenFaaS

Serving Static Sites and Microservices

Functions with Secrets

Asynchronous Invocations

Autoscaling Functions

Horizontal Scaling

Scale-to-Zero and Cold Starts

TLS and Production Readiness

Advanced Use Cases

Custom HTTP Responses

Binary Data Handling

Serving Static Pages

Combining OpenFaaS and MQTT for Edge Use Cases

Balancing Containers and OpenFaaS

Microservices at Edge with K3s and Fleet

What Is a Container Image?