DEV Community: VIREN PATEL

Building a Production Observability Pipeline with AWS, Prometheus and Grafana

VIREN PATEL — Tue, 06 Oct 2026 00:06:12 +0000

1. The Problem

In a production environment, knowing that an application is deployed isn't enough — you also need to know whether the devices and services running at the edge are actually healthy. In my case, that meant thousands of edge devices deployed across many customer locations, each running local software that depends on network connectivity, hardware health, and correct configuration.

A device can silently go offline, lose connectivity to its local network, run low on storage, or fall out of compliance with the fleet's expected OS version — and none of that is visible unless something is actively watching for it. We needed a way to answer, at any moment, for any location: "Is this device alive, and is it healthy?"

2. Requirements

Every device should report its liveness on a predictable cadence (a heartbeat), scoped to a specific location.
The ingestion path must be resilient to bursts, retries, and duplicate/late messages without losing data.
Failed or malformed messages must not be silently dropped — they need a dead-letter path for investigation.
In addition to "is it alive," we wanted richer telemetry (CPU, memory, storage, battery, compliance) without requiring every device to push it directly.
All ingested data needed to land in a time-series store so it could be queried, graphed, and alerted on.
Operators needed dashboards for fleet-wide and per-location visibility, plus alerts when something goes wrong — without needing to read logs.

3. Architecture

End-to-End System Diagram

Two ingestion paths feed the same metrics backend:

Direct heartbeats — each edge device pushes a small heartbeat payload on its own cadence.
Scheduled telemetry pull — a Lambda polls an external device management API on a fixed schedule to enrich the picture with hardware and compliance data the devices don't push themselves.

Both paths converge on the same time-series backend, so a single set of dashboards and alerts covers "is it alive" and "is it healthy."

4. AWS Components

Component	Role
API Gateway + Lambda authorizer	Authenticates and routes incoming heartbeat requests, scoped per location
Lambda (enqueue)	Validates the heartbeat payload and pushes it onto SQS
SQS + DLQ	Buffers and batches heartbeats; isolates malformed/failed messages after repeated delivery failures instead of dropping them
Lambda (dequeue)	Consumes SQS batches, converts each heartbeat into Prometheus-format metrics, and writes them to the metrics backend
Lambda (scheduled telemetry sync)	Runs on a fixed EventBridge schedule, pulls device and management telemetry from an external device-management API, and writes it as metrics
S3	Stores a raw, timestamped backup of every metrics payload as an audit trail, with lifecycle expiry
SSM Parameter Store	Holds service credentials and sync-state (e.g. last successful poll timestamp) outside of source code
CloudWatch Alarms + SNS	Watches queue age, DLQ depth, and Lambda error rates — the "is the pipeline itself healthy" layer, separate from device health

Keeping the ingestion pipeline serverless meant we didn't need to run or patch any always-on infrastructure just to accept heartbeats — it scales automatically with the size of the fleet and the queue absorbs bursts.

5. Heartbeat Design

The heartbeat payload is intentionally small and cheap to send. At minimum, a device only needs to identify itself and increment a counter:

{
  "serialNumber": "XXXXXXXXXXXXXXXXXXXXXXXXXX",
  "count": 14
}

Optionally, a device can include local network connectivity state, since a healthy device that's lost its local network connection is a different failure mode from an offline device:

{
  "serialNumber": "XXXXXXXXXXXXXXXXXXXXXXXXXX",
  "count": 14,
  "isConnected": true,
  "lastPublished": "2025-07-29T20:12:26Z",
  "connectionType": "local",
  "messageCount": 5
}

A few deliberate design choices here:

The device serial number is validated against the authenticated caller, not trusted blindly — a device can't report a heartbeat under an identity it isn't authorized for.
The schema is strict (additionalProperties: false) — unexpected fields fail validation loudly instead of being silently ignored.
The payload has no dependency on any specific transport; it's just JSON over HTTPS, which keeps the device-side implementation trivial.

6. Metrics and Monitoring

Rather than storing heartbeats as raw events in a database, every heartbeat is immediately converted into Prometheus-format gauges at ingestion time. That decision shaped everything downstream: querying "which devices haven't reported in the last N minutes" becomes a standard PromQL query instead of a custom aggregation job.

Two families of metrics exist:

Liveness metrics (from the heartbeat path): heartbeat counter, network connection status, last publish time, message count — labeled by device serial number.
Deep telemetry metrics (from the scheduled sync path): CPU utilization and temperature, memory and storage capacity, battery health, network diagnostics, OS version and compliance status, enrollment state.

Because both are just labeled Prometheus gauges in the same store, a dashboard or alert doesn't need to care which pipeline produced a given metric.

7. Prometheus / VictoriaMetrics

We chose VictoriaMetrics as the metrics backend because it speaks the Prometheus remote-write/import protocol (so nothing upstream needs to know it isn't "real" Prometheus) while being cheaper to run at scale for a high cardinality of devices and locations.

The dequeue Lambda batches metrics and writes them via VictoriaMetrics' /api/v1/import/prometheus endpoint, gzip-compressed, over HTTPS with basic auth. The scheduled telemetry sync does the same, but chunks its output into sub-1.5MB batches (to stay under payload limits) with retry and exponential backoff, since a single sync run can cover the entire device fleet.

Locally, the same VictoriaMetrics image runs in Docker so the ingestion path can be tested end-to-end without touching production metrics.

8. Grafana Dashboards

VictoriaMetrics is queried by Grafana for both fleet-wide and per-location visibility:

A fleet overview dashboard shows aggregate liveness across all locations/devices — how many are currently reporting heartbeats, how many have gone silent, and overall compliance/health trends.
Per-location dashboards drill into a single location's devices — heartbeat recency, network connectivity, hardware telemetry (CPU/memory/storage/battery) — so an operator investigating a specific customer complaint can go straight to the relevant view.

Because everything is labeled consistently by device serial number, the same panels/queries generalize across dashboards just by changing the label filter.

9. Alerting

Alerting is defined as Grafana alert rules evaluated directly against the VictoriaMetrics data — for example, a device with no heartbeat metric update within an expected window, or a rising rate of network disconnections at a location. This keeps alerting logic close to the same PromQL used for the dashboards, rather than as a separate system with its own definition of "healthy."

This is layered with CloudWatch alarms that watch the pipeline itself rather than the devices — SQS queue age, DLQ message visibility, and Lambda error rates — because a healthy-looking dashboard doesn't help if the ingestion pipeline is actually the thing that's broken.

10. Challenges and Trade-offs

Two ingestion paths, one data model. Direct heartbeats and polled telemetry arrive on completely different cadences and shapes, but both had to converge into the same labeled metric format so dashboards and alerts didn't need to special-case the source.
Batching vs. payload limits. The scheduled telemetry sync can produce a large volume of metrics in one run across the whole fleet; naively posting them in one request risks hitting the metrics backend's payload limits, so batching with retry/backoff was necessary.
Trusting device-reported data. A device claiming a given serial number can't be taken at face value — it has to be checked against what the caller is actually authorized for, otherwise one compromised or misconfigured device could pollute another device's metrics.
Losing data safely. Anything that fails to enqueue or process shouldn't just vanish — the DLQ and the raw S3 backup of every metrics payload exist specifically so a bad batch can be replayed or inspected after the fact.
Two definitions of "healthy." Device health (via Grafana/VictoriaMetrics) and pipeline health (via CloudWatch/SNS) are deliberately separate concerns, alerting through different channels, because conflating them makes it harder to tell "the fleet is unhealthy" from "our monitoring is broken."

11. What We Learned

Converting data into Prometheus format as early as possible in the pipeline (at ingestion, not at query time) simplified everything downstream — dashboards, alerts, and ad-hoc debugging all speak the same query language.
Treating the metrics pipeline itself as a first-class thing to monitor (via CloudWatch) is just as important as monitoring the devices it reports on.
Keeping the heartbeat payload minimal paid off — it made the device-side integration trivial and kept the ingestion Lambda's job simple and fast.
A raw payload backup (S3) turned out to be cheap insurance — being able to replay or inspect exactly what was received, even non-fatally, saved investigation time more than once.

12. Possible Improvements

Push more of the "deep telemetry" fields into liveness-style alerting (e.g. combining "hasn't heartbeated" with "was last seen non-compliant") for higher-signal alerts.
Explore anomaly-detection style alerting rather than fixed thresholds for fleet-wide trends.
Reduce the polling cadence gap between direct heartbeats and scheduled telemetry sync so both liveness and deep health data are closer to real-time.

13. Conclusion

Observability at the edge isn't just about collecting data — it's about designing the smallest possible signal (a heartbeat) that reliably answers "is this thing alive," and complementing it with richer telemetry pulled on our own terms rather than depending on every device to push everything. By normalizing both into the same time-series format early, we got a single, consistent place — Grafana on top of VictoriaMetrics — to answer both "is the fleet healthy" and "is our monitoring pipeline itself healthy."

Building a Reusable AWS SAM CI/CD Pipeline for Multi-Account Deployments

VIREN PATEL — Tue, 14 Jul 2026 18:56:04 +0000

Introduction & The Problem
I am an Intermediate Software Engineer at one of New Zealand’s global QSR enterprises.

One challenge kept showing up in my day-to-day work: every new AWS SAM project in our team came with yet another deployment pipeline.

Different repositories used different AWS CLI and AWS SAM CLI versions. Some pipelines deployed to a single account, others to multiple accounts. Even small improvements to the deployment process had to be repeated repo by repo.

After maintaining this for a while, I realized we weren’t solving deployment problems anymore. We were solving the same tooling problem over and over again.

I wanted every project to use one reusable deployment engine instead of maintaining dozens of nearly identical pipelines. So, this article is aimed at teams that deploy multiple AWS SAM applications and want to standardize deployments across AWS accounts without duplicating CI/CD logic.

Architecture Overview

Because this article is about solving a repeated deployment problem, I am showing two architecture views:

The common fragmented model many teams start with.
The standardized deployment-engine model I implemented with sam-pipeline.

Architecture 1: Fragmented Team-by-Team Deployments (Current State)

In this model, each repository carries its own deployment logic, tool versions, and account assumptions. It works initially, but maintenance scales badly and drift becomes unavoidable.

Architecture 2: Shared Deployment Engine with sam-pipeline (Solution)

Here, every project uses the same deployment engine. Runtime setup, SAM build/deploy flow, and multi-account behavior are standardized, which reduces duplication and makes deployments more predictable.

Building the Docker Deployment Image

To solve this consistently, I built sam-multi-account-pipeline, a reusable deployment image that teams can pull from Amazon ECR Public.

The idea is simple: developers use a shared public ECR image to deploy AWS SAM stacks across multiple AWS accounts and regions, without rebuilding deployment logic in every repository.

Image: public.ecr.aws/z7q5l7x5/sam-pipeline:1.0.0

Publishing to Amazon ECR Public

I published the image to Amazon ECR Public so anyone can consume it directly in their CI/CD pipeline.

Complete setup and usage instructions are available in the repository:
https://github.com/Viren1993-web/sam-multi-account-pipeline

Multi-Account Deployment with AssumeRole
Here’s how multi-account deployment works in this model:

For every AWS account you want to deploy to, create a deployment role in that account and grant it the permissions required by your SAM template resources.

The pipeline then assumes that role and performs the deployment securely in the target account. In my repository, I’ve included an example using a DeployerGithub role to demonstrate this pattern.

What I Wanted to Achieve
I wasn’t trying to build “another CI script.” I wanted a deployment foundation the team could trust and reuse everywhere.

My goals were:

One consistent deployment engine for all SAM projects.
Support for both single-account and multi-account deployments.
No repository-specific setup drift for AWS/SAM tooling.
Easy onboarding for new services.
A safer security model using role assumption instead of long-lived credentials.
In short, I wanted deployment to be boring, predictable, and easy to maintain.

Where This Fits Alongside AWS Services

AWS provides excellent services such as AWS CodePipeline and CloudFormation Change Sets for building and managing deployment workflows. My goal wasn't to replace these services, but to solve a different challenge: standardizing how AWS SAM applications are deployed across multiple repositories and AWS accounts.

By packaging the deployment tooling into a reusable, versioned Docker image, every project uses the same AWS CLI, AWS SAM CLI, and deployment process regardless of the CI/CD platform. This approach reduces duplication, simplifies maintenance, and provides a consistent deployment experience whether the pipeline runs in Bitbucket Pipelines, GitHub Actions, Jenkins, or even AWS CodePipeline itself.

Supporting Node.js and Python
One practical requirement was runtime flexibility. Some of our services are Node.js, others are Python, and the deployment pipeline needed to support both without branching into separate systems.

So I designed the image and scripts to be runtime-aware:

Node.js projects can use the right Node version.
Python projects can use the right Python version.
The SAM deployment process stays consistent.
That means teams can keep their service language choices while still using a common deployment pipeline.

Repository Walkthrough
The repository is structured to keep responsibilities clear:

Core pipeline orchestration logic.
Scripts for runtime setup and SAM build/deploy execution.
Example projects (Node.js and Python) showing expected usage.
Tests for parsing, validation, and deployment flow behavior.
Sample CI workflows to demonstrate real integration.
This structure helped keep the project approachable, especially for teams adopting it for the first time.

How to Use the Pipeline
Using the pipeline follows a simple pattern:

Configure your CI workflow to use the public ECR image.

image:
  name: public.ecr.aws/z7q5l7x5/sam-pipeline:1.0.0

pipelines:
  branches:
    main:
      - step:
          script:
            - ./deploy.sh

Provide deployment variables (accounts, regions, runtime language, stack name, working directory, and optional SAM arguments).
Ensure target AWS accounts have the deploy role and trust policy configured.
Trigger the pipeline.
After that, the container handles runtime setup, account role assumption, and SAM build/deploy flow in a standardized way.

Benefits & Results
Moving to this model gave immediate benefits:

Standardization: same deployment behavior across repositories.
Lower maintenance: no repeated fixes in every pipeline.
Faster onboarding: new projects plug into an existing pattern.
Better security posture: role assumption with temporary credentials.
More confidence in releases: fewer environment/toolchain surprises.
The biggest gain was reducing operational noise. We now spend more time shipping features and less time fixing deployment plumbing.

Lessons Learned
A few lessons stood out while building this:

Reusability matters more than cleverness in CI/CD tooling.
Most deployment pain comes from inconsistency, not complexity.
Security and developer productivity can improve together.
Good examples are just as important as good code.
Treat deployment tooling like a product: version it, document it, test it.
Also, making one good shared pipeline is usually easier than maintaining ten almost-identical ones.

Future Improvements
There are still a few things I want to improve:

Better deployment reporting per account/region.
Optional pre-deployment checks and guardrails.
Stronger observability around failures and retries.
More advanced runtime customization for edge cases.
Cleaner developer experience for local dry-run validation.
The long-term goal is to keep scaling this without increasing operational complexity.

Conclusion
The biggest lesson from this work is that the real value is not just a Docker image, but a reusable deployment model.

Instead of maintaining similar deployment logic across many repositories, teams can standardize on one runtime and one SAM execution flow, then securely deploy to multiple accounts using AssumeRole.

If you want quick adoption, you can use my public image directly:
public.ecr.aws/z7q5l7x5/sam-pipeline:1.0.0

If your organization requires tighter governance, use the same architecture and publish your own private deployment image.

Either way, the outcome is the same: less deployment drift, lower maintenance costs, and faster delivery across environments/ multi-account/ regions.