Paulo Victor Leite Lima Gomes

Posted on Aug 8

prometheus collectors are cloud control plane now

#aws #observability #prometheus #cloudwatch

Prometheus used to have a nice emotional contract: you ran the thing, scraped the thing, and cursed at the thing when a target disappeared because somebody changed a label during a deployment and now the dashboard looks like a crime scene.

It was not pleasant, but at least ownership was obvious.

Then AWS announced CloudWatch managed Prometheus collectors, and the shape changed. CloudWatch can now run fully managed collectors that discover Prometheus-compatible targets in AWS infrastructure, scrape them through VPC connectivity, and deliver the metrics into CloudWatch. EKS, EC2, ECS, MSK, OpenSearch. The usual suspects.

This is genuinely useful.

It also means the scrape layer is not just a daemonset, sidecar, or collector you forgot to upgrade. It is becoming another cloud control plane.

That is worth paying attention to.

nobody enjoys operating scrapers

Let us be honest. Maintaining metrics collection infrastructure is not the part of observability that makes people feel alive. Somebody has to run collectors, patch them, scale them, give them network access, keep configs synced, notice when service discovery breaks, and stop the high-cardinality endpoint from eating the month.

Prometheus is excellent. Prometheus also makes you own the scraping path.

AWS taking more of that burden makes sense. The new collectors are agentless from the customer's point of view. You provide scrape configuration and a connection to the resources. CloudWatch provisions and scales the machinery. The docs say the collector creates ENIs in selected subnets, scrapes through them using OTLP, and delivers metrics to a CloudWatch dataset through a VPC endpoint.

That removes operational work and moves it into the platform contract.

the scrape config is production code

Every observability system eventually teaches the same annoying lesson:

The dashboard is not the product.

The pipeline is.

If the scraper misses a target, the dashboard lies politely. If relabeling is wrong, the alert routes to the wrong team. If a metric selector is too wide, the bill grows. If it is too narrow, the incident starts with everyone saying "that is weird, the graph is flat."

With managed collectors, scrape configuration is production code. It decides what gets seen, how it is named, where it lands, and how much it costs. That means it needs the same discipline as other production configuration:

code review
owners
environment separation
rollback
change history
label conventions
cost expectations
tests or at least validation queries

Yes, tests for observability config sound boring. So does checking a parachute.

managed does not mean ownerless

The trap with managed services is the little mental shortcut where "AWS runs it" becomes "we do not have to think about it." That shortcut is expensive.

Managed collectors can remove the need to operate collector hosts, in-cluster scrapers, or a pile of OpenTelemetry collector deployments. Great. Please remove toil. Tired humans deserve mercy.

But the collector still needs network permissions. It still needs scrape targets. It still needs security groups. It still needs subnets. It still needs metric selection. It still emits billable data. It still becomes part of the answer when the incident commander asks, "can we trust this graph?"

Observability is not decoration. Autoscalers, rollback decisions, SLOs, anomaly detection, agent workflows, and very tired humans at 03:17 all depend on it. Once the metrics pipeline becomes managed infrastructure, platform teams need to treat it as a shared dependency.

labels are where good intentions go to get weird

The hard part of metrics collection is rarely "can I scrape /metrics?"

It is usually:

what is this service called?
which team owns it?
is this production?
which customer, region, tenant, cluster, namespace, queue, model, or job does this represent?
is this cardinality useful or did we just create a tiny money printer pointed at ourselves?

Labels are the observability version of naming things and blaming DNS. They look simple until your query returns twelve answers and none of them are the one you need.

Managed collectors do not remove that problem. They make the boundary cleaner.

For ECS, AWS describes DNS-based discovery through Cloud Map so the collector follows changing task IPs. For EKS, discovery can follow Kubernetes targets. For EC2, direct instance scraping still needs ports and sane target definitions.

But if every team brings its own labels, the centralized collector becomes a centralized junk drawer.

The right platform move is to publish a small label contract and enforce it where possible:

service
team
environment
region
workload_type
slo_name when it exists

Then be suspicious of labels with unbounded values. User IDs, request IDs, raw URLs, pod hashes, prompt IDs, session IDs, and "temporary" fields are how observability turns into a billing incident wearing a Grafana badge.

cost is part of the signal now

The CloudWatch docs are clear that managed collectors are charged by the hour and CloudWatch OpenTelemetry metric ingestion pricing applies. They also mention VPC data transfer can show up when scrapers pull metrics from targets, and suggest compressing /metrics responses to reduce transfer volume.

That is architecture.

When metrics were collected by something your team operated, the cost was often hidden inside nodes, daemonsets, storage, and people pretending a giant Prometheus was "fine for now." With managed collectors, the bill can become more explicit. This is good if you measure it. It is surprising if you do not.

Metric volume needs a budget. Scrape interval needs a budget. Cardinality needs a budget. Retention and query patterns need a budget. Before anyone says "but it is just metrics", remember that "just logs" already bought many companies a very educational invoice.

The useful question is not "is managed collection cheaper?"

What is the cost per service of the metrics we actually use to operate the system?

observability has a supply chain too

We talk a lot about software supply chains now. Images, dependencies, SBOMs, signatures, provenance, all that healthy paranoia.

Metrics have a supply chain too.

Exporter. Endpoint. Network path. Scraper. Relabeling. Transport. Storage. Query. Alert. Dashboard. Human decision. If one link is wrong, the final graph may still look professional.

Managed collectors make some links stronger. Less self-managed collector drift. Better scaling. Fewer forgotten agents. More unified querying with CloudWatch and PromQL. That is valuable.

They also concentrate trust. If CloudWatch becomes the place where more Prometheus data lands, collector configuration becomes a first-class platform artifact. It should live near the IaC, have owners, and have a rollback path that does not involve three people clicking in a console while the incident channel develops a personality.

what i would do first

If I were rolling this out, I would avoid the grand observability transformation slide deck and start smaller.

First, pick one environment and one workload type. ECS services through Cloud Map, for example. Prove that discovery, security groups, scrape intervals, labels, dashboards, and alarms make sense.

Second, put the collector config in version control. No mystery config. No "we changed it in the console because the graph looked lonely."

Third, define a label contract before onboarding the second team. That is where standards become folklore if nobody writes them down.

Fourth, set cost alarms early. Not after the first strange invoice. Early. Scrape volume is part of the rollout.

Fifth, document the trust path for incident responders. If an alert fires from a managed collector, people should know where the target is, which config scraped it, and who owns the metric.

That is not bureaucracy.

That is how you make "managed" mean less work instead of less visibility.

the punchline

CloudWatch managed Prometheus collectors are a good feature because they remove a chunk of observability plumbing nobody wants to babysit.

But plumbing is not meaningless just because a cloud provider operates part of it.

The scrape layer decides what your company can see. It shapes alerts, dashboards, SLOs, autoscaling, rollback confidence, and sometimes the emotional weather of an incident call.

So yes, let AWS run the collectors if that reduces toil. Let CloudWatch pull Prometheus-compatible metrics from EKS, ECS, EC2, MSK, and OpenSearch. Use PromQL where it helps. Delete some self-managed collector snowflakes.

Just do not confuse managed infrastructure with absent ownership.

The metrics pipeline is becoming a cloud control plane.

Treat the scrape config like production code, the labels like an API, and the bill like an alert signal.

Otherwise the graph will look calm right up until reality files a ticket.

references

To test my projects, I use Railway. If you want $20 USD to get started, use this link.

DEV Community