DEV Community: Andrew Eells

We've Spent 25 Years Shifting Left. The Next Shift Is Down.

Andrew Eells — Fri, 24 Jul 2026 07:28:29 +0000

I've been building software professionally for about 25 years.

Looking back, I realised that a surprising amount of that time has been spent doing one thing.

Shifting left.

Moving work earlier. Catching problems sooner. Automating the things we used to do by hand at the end of a project.

It's been one of the defining trends of modern software engineering.

But recently I've started wondering whether we've been optimising along the wrong axis.

Twenty-five years of shifting left

Around the turn of the millennium, continuously building and testing software instead of integrating everything just before release was still a relatively new idea.

Continuous integration emerged. CruiseControl popularised the approach, Hudson followed a few years later, and eventually Jenkins became the tool many teams knew best. "Does it build?" became a question answered automatically on every commit.

That was shift-left number one.

Testing and integration moved earlier.

Then continuous integration became continuous delivery. Martin Fowler and Jez Humble helped turn the idea into mainstream engineering practice. Building software wasn't enough; every commit produced something releasable.

Then the "D" quietly changed meaning.

Delivery became deployment.

Releasing software stopped being a scheduled event and became something that happened continuously as a consequence of merging code.

Then came DevOps.

Operations moved left too. Infrastructure, monitoring, deployments and incident response all became part of the same team's responsibility.

"You build it, you run it."

Then security took its turn. DevSecOps pushed security left as well - threat modelling, dependency scanning and policy checks moving out of a late-stage gate and into the pipeline itself.

Every few years another discipline moved closer to the developer.

Testing.

Delivery.

Deployment.

Operations.

Security.

It genuinely made us better engineers.

AI isn't really shifting left

Today it's fashionable to put AI into the same story.

I'm not convinced that's what's happening.

AI isn't primarily moving work earlier.

It's helping us perform the same work faster.

That's an important distinction.

Where AI has changed things is by lowering the barrier to entry. Someone with relatively little operational experience can now generate a CI pipeline, Terraform configuration or deployment workflow in minutes.

Useful?

Absolutely.

But if AI's biggest contribution is helping us recreate the same infrastructure more quickly, then perhaps recreating it was never the thing we should have been doing in the first place.

The pattern I couldn't ignore

Here's what those 25 years actually looked like.

I built CI pipelines.

Then I built them again somewhere else.

Then again.

I created infrastructure-as-code.

Then recreated it for the next company.

I wired up authentication.

Observability.

Deployment pipelines.

Audit logging.

Secrets management.

Monitoring.

Rollback procedures.

Different organisations.

Different technology stacks.

Different decades.

Almost exactly the same engineering disciplines.

None of that work was the product.

None of it differentiated the business.

It was simply the operational foundation required before the product could exist.

These aren't product problems. They're engineering fundamentals. Yet almost every team rebuilds them from scratch, every single time.

Shift-left made us do this work earlier.

It never asked why we were repeating it at all.

Every startup, every consultancy and every enterprise greenfield project still begins by rebuilding the same engineering capabilities before a single customer problem gets solved.

Forge is about shifting down

That's the shift I'm interested in now.

Not left.

Down.

Imagine rotating the software lifecycle ninety degrees.

Across the top sits your product: the services, APIs and user experience your customers actually pay for.

Underneath sits everything every serious SaaS product inevitably needs.

Authentication.

Identity.

CI/CD.

Observability.

Audit.

Notifications.

Document services.

Infrastructure.

Operational tooling.

These capabilities aren't unique to your business.

They're shared engineering concerns.

So why are they still treated as project work?

Shifting down means moving those undifferentiated engineering disciplines beneath your application into a reusable operational foundation instead of rebuilding them inside every new product.

That's what Forge Platform is.

Not another framework.

Not another starter template.

An opinionated operational backbone for modern SaaS products.

Authentication, users, audit, notifications, documents and a BFF orchestration layer. Infrastructure-as-code. CI/CD. Observability. Security. Everything deployed into your own AWS account, from your own GitHub repositories, remaining entirely under your control.

The disciplines I spent decades shifting left don't disappear.

They simply stop living inside every application.

Left was never the destination

Shift-left was absolutely the right idea.

Finding problems earlier has transformed our industry.

But somewhere along the way we confused the mechanism with the objective.

The objective was never simply to push more responsibility onto developers.

It was to stop repeatedly paying for the same engineering work in every company, on every project, for every new product.

Some engineering problems only need solving once.

The operational backbone beneath your SaaS product is one of them.

After 25 years of shifting left, that feels like the direction we've really been heading all along.

If that resonates, I'd genuinely like to hear from you - you can find out more at forgeplatform.software.

AWS Transform and Superwerker: same vocabulary, different problems

Andrew Eells — Sat, 11 Jul 2026 18:04:14 +0000

Most people will read AWS Transform's landing zone agent as Superwerker's smarter cousin. I don't think that's right.

Same destination. Different problems. Treating them as substitutes is how greenfield teams buy enterprise process they don't need — and how migration teams under-buy governance they do.

I saw the announcement via a LinkedIn post, and it stuck because the vocabulary overlaps heavily with a tool I've actually used and rate highly: Superwerker. I haven't run Transform myself — this is a read from the announcement and docs, not a hands-on review. I'd genuinely like to hear from anyone who has.

Superwerker: opinionated defaults, shipped

Superwerker is a free, MIT-licensed CloudFormation stack. You run it, and it stands up a multi-account baseline — Control Tower, SSO, and sensible guardrails — without a discovery workshop. We used it on a recent greenfield build and had SSO and baseline guardrails live in an afternoon. The entire decision tree is a public repo you can read end to end before you commit to anything.

I'm generally a fan of opinionated tooling. If there is a sensible default, I'd rather spend engineering time building product than debating OU structures.

If I were founding again tomorrow with a blank AWS org, I'd run Superwerker before I spent a day on account design. Greenfield teams don't fail because their landing zone wasn't migration-aware. They fail because they never got a baseline and spent months refactoring after the event.

Transform: a different problem

Transform's agent is a different shape of tool entirely. It's not a standalone landing zone builder — it's embedded in AWS Transform's end-to-end migration workflow. Transform gathers information about your organisation, migration waves, compliance requirements and account structure before proposing a landing zone.

That's a fundamentally different problem from "deploy me a sensible default."

Where Forge sits

Forge deliberately stops at the landing zone boundary. That's infrastructure governance, and every organisation has different requirements. Once that foundation exists — whether it came from Superwerker, AWS Transform or something else — that's where Forge starts.

A working rule of thumb

Greenfield startup? Use Superwerker.
Enterprise migration? Look seriously at AWS Transform.
Brownfield AWS estate? Use judgement. Every organisation carries different history and constraints.

I haven't used Transform yet, so I'd genuinely like to hear from teams that have. I'm much more interested in where this rule breaks down than where it holds.

How AI finally solved AWS architecture diagrams

Andrew Eells — Fri, 10 Jul 2026 05:47:49 +0000

After years of fighting draw.io, CloudFormation graphs and auto-generated spaghetti, architecture diagrams felt like the final boss of platform engineering - one I'd never quite beaten.

I've been building software professionally for over 25 years.

I have yet to be genuinely excited by AI's impact on large-scale software engineering. It's exceptionally good at bounded problems - a Bash script, a utility class, an isolated feature - and it's undeniably accelerating delivery. But building an entire production platform is a different challenge. That's still about architecture, trade-offs, operational thinking and years of accumulated engineering judgement.

I don't believe "vibe coding" is replacing experienced engineers any time soon. AI is an extraordinary force multiplier, but it doesn't replace engineering judgement. If you're only capable of building poor software today, AI mostly helps you build poor software faster.

But this...

This genuinely felt like the first time AI solved this category of problem I'd been wrestling with for years.

In under two hours I went from...

"A salesperson asked if I had an architecture diagram. I said, 'Yes... but it's rubbish.'"

...to publishing this.

You can see the live diagram on the Forge docs site, with more detail in the AWS platform infrastructure guide.

After literally years of trying different approaches, this is the first workflow that has produced an architecture diagram I'd happily put in front of customers.

The problem with architecture diagrams

Anyone building on AWS has probably been here.

The architecture changes.

The diagram doesn't.

Eventually someone opens draw.io and spends half a day dragging icons around.

Six months later it's wrong again.

I've spent years trying to solve this problem.

Manual diagrams

I've built diagrams using tools including:

draw.io AWS diagram shapes
Lucidchart
Microsoft Visio
Cloudcraft (live AWS/Azure scanning and visual designer)
Eraser AI AWS Diagram Generator (prompt-to-diagram)

The results can look fantastic - whether you drew them by hand, scanned live infrastructure, or generated them from a written description.

Until the infrastructure changes.

Then they immediately start drifting from reality.

CDK diagram generators

I also explored tools including:

cdk-dia
cdk2d2
AWS PDK CDK Graph (with its diagram plugin)

These are clever projects.

They faithfully represent your infrastructure.

Unfortunately, for customer-facing architecture communication, that's exactly the problem.

A single ECS service explodes into something like:

Task Definition
Execution Role
Task Role
Security Groups
Target Groups
Listener Rules
Auto Scaling
CloudWatch Alarms
Log Groups
IAM Policies
ENIs

Everything is technically correct.

Nothing is architecturally useful.

CloudFormation graphs answer:

"What AWS resources exist?"

Architects are usually trying to answer:

"How does this system work?"

Those are completely different questions.

The CDK diagram generators were right.

They just weren't publishable.

AWS Workload Discovery

I even evaluated AWS' own solution:

AWS Workload Discovery on AWS

It discovers infrastructure.

It inventories resources.

It builds dependency graphs.

For me?

It simply wasn't worth the cost - over $400/month just to deploy the CloudFormation stack and keep the discovery pipeline running, so you can draw a diagram of what you already deployed with CloudFormation. That is a genuinely terrible proposal for a documentation problem.

Again, it documents what AWS has deployed.

It doesn't communicate your architecture.

The breakthrough

The breakthrough came from this AWS blog:

Build AWS architecture diagrams using Kiro CLI and MCP

At first I thought:

"Interesting... but they're just feeding text into an LLM."

Then I realised something.

That isn't a weakness. It's the entire point.

An experienced solutions architect doesn't think in CloudFormation resources.

They mentally ignore:

IAM Roles
Security Groups
Parameters
Log Groups
Listener Rules
Auto Scaling Targets

and instead think:

Application Load Balancer
ECS Services
PostgreSQL
Redis
S3
Cognito

That's architecture.

Not implementation.

The LLM wasn't trying to replace an architect.

It was acting as an architectural abstraction layer - the same idea as a C4 container diagram: document intent at the level stakeholders actually care about.

That was the lightbulb moment.

Cursor did almost all of the work

Armed with that idea, I pointed Cursor at my AWS CDK repository.

I also gave it some context from a conversation I'd had with ChatGPT about why CloudFormation graphs are the wrong level of abstraction - useful for framing, but Cursor did the actual work.

Then I basically let it loose.

It analysed the repository.

It walked the stacks.

It understood the constructs.

It traced the networking.

It identified which AWS resources represented meaningful architectural components and which were simply implementation plumbing.

Then it produced:

a curated architecture model
rendering instructions
a repeatable workflow
automation scripts
documentation updates
and finally, the diagram itself.

I didn't manually reverse-engineer thousands of CloudFormation resources into a diagram. I reviewed and refined what the tooling produced.

That's a huge distinction.

The AI wasn't making architectural decisions.

It was reverse engineering architectural decisions I'd already made in code.

My role was limited - mostly review, a few rounds of feedback, and sanity-checking that the subnet story matched what we'd actually built.

That's a far harder problem than it sounds

The YAML wasn't the clever part.

The clever part was that Cursor understood:

an ECS Task Role isn't an architectural component
an Application Load Balancer definitely is
six Fargate services belong together
subnet boundaries matter
CloudFront belongs at the edge
PostgreSQL, Redis and Cognito deserve first-class representation

None of that existed in a single file.

It emerged from analysing thousands of lines of CDK spread across stacks, constructs and configuration - including constructs like ForgeStaticEdgeDeliveryConstruct, which encodes the CloudFront → S3 static UI + API path proxying story as a single architectural unit rather than a pile of distribution behaviours and origin policies.

That's genuinely impressive.

Was it perfect first time?

No.

I iterated a couple of times.

I told Cursor to:

collapse repeated Cognito connections and repeated metrics connections to AMP
show public/private subnet boundaries in the VPC

The second render was immediately publishable.

There was also a small tooling wrinkle: the AWS Diagram MCP server package has been yanked upstream, so we pinned the last published version (1.0.23) in a source-controlled mcp.json. That kind of friction is worth mentioning - this wasn't magic; we wrestled with it slightly to get it working.

The hybrid workflow

The workflow ended up looking like this.

CDK repository
        │
        ▼
Cursor analyses architectural intent
        │
        ▼
Curated intermediate model (YAML)
        │
        ▼
Kiro CLI
        │
        ▼
AWS Diagram MCP ([Model Context Protocol](https://modelcontextprotocol.io/))
        │
        ▼
Published AWS architecture diagram
        ([Python diagrams](https://diagrams.mingrammer.com/) + [Graphviz](https://graphviz.org/) + AWS icons)

The architecture model is version controlled.

The prompt is version controlled.

The rendering script is version controlled.

The generated diagram can be regenerated whenever the platform evolves.

For the first time ever, my architecture documentation actually feels maintainable.

Why this works

The key insight is surprisingly simple.

Architecture diagrams should document intent, not implementation.

My CDK constructs already encode that intent.

They have names like:

ForgeStaticEdgeDeliveryConstruct
ForgeEcsWorkloadConstruct
ForgeCognitoIdpStack

Those are architectural concepts.

CloudFormation is an implementation detail.

Trying to diagram CloudFormation is a bit like trying to explain a car by drawing every nut and bolt.

Sometimes abstraction is exactly what you want.

AI finally solved the right problem

This experience genuinely changed my opinion of where AI delivers value.

Not by replacing engineers.

Not by generating production systems from prompts.

Not by writing thousands of lines of code you'll spend next week debugging.

Instead, it behaved like a very competent technical collaborator.

It absorbed an entire repository.

It reconstructed the architecture.

It understood what mattered.

It ignored what didn't.

It accepted feedback.

It refined the output.

The AI wasn't designing my platform.

But it did an excellent job of documenting a platform that already exists - the same platform I write about when debugging AMP remote write failures or explaining what production-ready actually means.

Would I trust it blindly?

Absolutely not.

The architecture still belongs to the engineers who built the system.

The AI isn't deciding what the platform looks like.

It's modelling something that already exists.

The human remains accountable.

The AI removes an enormous amount of tedious work.

That's augmentation.

Not replacement.

What's next?

This has me thinking the future isn't another CDK diagram generator.

It's architecture-aware tooling.

Imagine a workflow that can:

analyse your CDK constructs
infer architectural components
maintain a version-controlled intermediate model
regenerate diagrams automatically
produce different views for developers, operations, security reviews and customer documentation

That feels considerably more exciting than another CloudFormation graph.

And I essentially have that version-controlled now - tied to a real ECS on Fargate platform, not a toy example.

Here's how Cursor did it

The interesting part isn't the PNG. It's the intermediate model - a curated, solutions-architect view derived from CDK constructs, not from the synthesized CloudFormation dependency graph.

The model header explains the intent:

---
# Forge Platform - curated architecture model
#
# Derived from CDK constructs under infra/src/lib (NOT from the synthesized
# CloudFormation dependency graph). Low-level noise (IAM roles/policies,
# security groups, log groups, alarms, KMS keys, SSM params, ENIs, target
# groups, listener rules) is deliberately omitted.
#
# Source of truth mapping:
#   - Edge / CloudFront .......... forge-static-edge-delivery-construct.ts
#   - Ingress (ALBs) ............. forge-ecs-ingress-construct.ts
#   - Compute (Fargate) .......... forge-ecs-workload-construct.ts
#   ...

Nested groups encode the VPC security boundary:

groups:
  - id: vpc
    label: VPC (ForgePlatformVpc)
  - id: public_subnets
    label: Public Subnets
    parent: vpc
  - id: private_subnets
    label: Private Subnets
    parent: vpc
  - id: compute
    label: ECS Fargate Cluster
    parent: private_subnets
  # ...

Aggregate connections collapse repeated edges - six identical "service auth" lines become one:

  - from: compute
    to: cognito_service
    label: service auth
    aggregate: true

The MCP server registration is trivial - no secrets, just a pinned package launch:

{
  "mcpServers": {
    "awslabsawsdiagrammcpserver": {
      "command": "uvx",
      "args": [
        "--from",
        "awslabs.aws-diagram-mcp-server==1.0.23",
        "awslabs.aws-diagram-mcp-server"
      ],
      "env": {}
    }
  }
}

And regeneration is a one-liner:

./infra/architecture/render-diagram.sh

Which syncs the MCP config, then drives Kiro non-interactively:

kiro-cli mcp import --file "${mcp_config}" --force global
kiro-cli chat --no-interactive --trust-all-tools "$(cat diagram-prompt.md)"

The prompt tells the MCP server how to map component types to diagrams nodes, nest subnet clusters, and honour aggregate edges - treating the YAML as the single source of truth.

That's the whole trick: deterministic extraction from code you already wrote, LLM-assisted rendering for polish.

Final thoughts

This is probably the first AI workflow I've adopted where I immediately thought:

"I'm never going back."

Not because the AI was smarter than me.

Because it could absorb an entire repository, retain all of that context, and perform a task that would have taken me several evenings of tedious work.

I've spent years looking for a workflow that made architecture diagrams feel like source code instead of PowerPoint.

This approach is by far the best I've seen.

Not because AI magically solved architecture.

Because it finally solved the tedious part.

AMP Remote Write HTTP 400: The CloudWatch Metric That Solved It

Andrew Eells — Sat, 04 Jul 2026 00:32:05 +0000

AMP said 400. CloudWatch said why.

We turned on Amazon Managed Prometheus (AMP) remote write from our Quarkus services on ECS — Micrometer snapshots, Snappy-compressed Prometheus remote write, SigV4 signing — and everything mostly worked.

Mostly.

Every service logged AMP remote write failed with status 400 on a ~60 second cadence. Not constantly. Not only at startup. Intermittently, across the fleet, continually.

HTTP 400 with an empty body is a frustrating place to start.

This post is how we narrowed it down, why the obvious suspects weren't the culprit, and the CloudWatch metric that finally made the failure mode obvious.

The setup

Forge pushes application metrics in-process:

Micrometer with the Prometheus v1 registry (/q/metrics for local scrape, MetricSnapshots for export)
Remote write to AMP over HTTPS with Content-Encoding: snappy and AWS SigV4 (aps signing name)
One AMP workspace shared by every ECS service in the environment (auth, actor, document, notification, audit, BFF, …)
GraalVM native images on Fargate, private subnets, no collector sidecar

The architecture is deliberate: fewer moving parts, right-sized tasks, metrics, and traces exported directly from the app.

When remote write misbehaves, the failure is somewhere in a short chain: scrape → encode → compress → sign → POST.

What we ruled out first

Wrong URL

A previous deploy had pointed at the wrong path suffix. AMP's real endpoint is:

https://aps-workspaces.<region>.amazonaws.com/workspaces/<ws-id>/api/v1/remote_write

Not .../remote_write alone (that returned 404).

We verified the CloudFormation export, the ECS task environment variable, and a SigV4-signed POST with an empty Snappy payload to the correct URL. That returned 200. So URL and IAM were fine.

SigV4 and transport

The same signing code path used in production (AwsSignedHttpTransport) worked from a JVM probe on a developer machine against the live workspace. Empty payload and full payload both succeeded when credentials and URL were correct.

"A bad metric family"

We scraped live exposition text from a running native task (more on that pattern below), loaded it into a JVM integration test, encoded it with the same remote-write encoder, and POSTed to AMP.

200. Full payload accepted.

So the metric content at scrape time was not inherently toxic. The encoder and Snappy path on JVM were fine.

That left something about the native periodic push path or fleet-wide ingestion behaviour — not a single broken histogram hiding in auth-service.

The pivot: CloudWatch `DiscardedSamples`

AMP exposes vended CloudWatch metrics under the AWS/Prometheus namespace. One of them is DiscardedSamples, with a Reason dimension explaining why samples were dropped.

We listed metrics for our workspace:

aws cloudwatch list-metrics \
  --namespace "AWS/Prometheus" \
  --metric-name "DiscardedSamples" \
  --region us-west-2

The only reason showing material volume:

Reason = new-value-for-timestamp

Querying that dimension:

aws cloudwatch get-metric-statistics \
  --namespace "AWS/Prometheus" \
  --metric-name "DiscardedSamples" \
  --dimensions \
    Name=Workspace,Value=ws-xxxxxxxx \
    Name=Reason,Value=new-value-for-timestamp \
  --start-time "$START" --end-time "$END" \
  --period 300 \
  --statistics Sum \
  --region us-west-2

We saw hundreds of discards per five-minute bucket, sustained, fleet-wide.

That is not a signing bug. It is not Snappy corruption.

In Prometheus remote write semantics, new-value-for-timestamp means: AMP already has a sample for this exact time series (same metric name and label set) at this exact timestamp, but the new sample has a different value.

In Prometheus, a series is defined by the metric name and its complete label set. If two writers produce identical labels, the backend treats them as one time series regardless of which process produced them.

Same series. Same millisecond. Different number. Rejected.

Once we saw that, the intermittent 400s and the ~95% push success rate (success counter climbing, failure counter occasionally ticking) both made sense: most pushes partially or fully conflicted with other writers or stale timestamps; some requests failed hard at the HTTP layer.

The actual bug: six services, one series

Each ECS service exports the same Micrometer binders: jvm_*, process_*, system_*, HTTP server metrics, and our own forge_observability_amp_push_* counters.

We had no common tags — no service, no instance, no pod.

So from AMP's perspective:

auth-service   → jvm_threads_live_threads = 21  @ T
actor-service  → jvm_threads_live_threads = 14  @ T
document-service → jvm_threads_live_threads = 19 @ T

That is one series (jvm_threads_live_threads with identical labels) receiving different values at the same timestamp from different tasks, every push interval, forever.

Classic multi-writer collision.

It is the same class of problem as running multiple Prometheus replicas remote-writing the same targets without external labels — except we did it with six application services and one workspace.

Why the JVM probe didn't catch it

The probe used a single scraped exposition file from one task. One writer. AMP happily ingested it.

The failure mode only appears when multiple tasks push overlapping label sets to one workspace on the same schedule.

Why `hasScrapeTimestamp()` wasn't the story

We also suspected stale per-datapoint scrape timestamps from the registry. On JVM Micrometer scrape, hasScrapeTimestamp() was false for every data point in our tests. The old encoder fell back to System.currentTimeMillis() per sample anyway.

Fleet collision fit the CloudWatch evidence better than stale scrape timestamps.

We still normalized to one batch push timestamp per remote-write request — correct semantics for periodic push export — but the fix that cleared discards was disambiguating series.

The fix

1. Common tags (the real fix)

In forge-kit forge-metrics, a CDI MeterFilter:

MeterFilter.commonTags(
    Tag.of("service", applicationName),   // quarkus.application.name
    Tag.of("instance", hostName)        // ECS task hostname; override via config
);

After deploy, AMP queries looked like:

forge_observability_amp_push_success_total
jvm_threads_live_threads{service="auth-service"}
count by (service) (jvm_memory_used_bytes)

Six services, six distinct label sets. Immediately after deploying the common tags, new DiscardedSamples{Reason="new-value-for-timestamp"} buckets dropped to zero across the workspace while historical buckets retained the earlier collisions. No other code changes were required.

2. Richer failure logging

AmpMetricsExporter had only logged status 400. We added Snappy body size, snapshot count, and AMP response body on failure so the next incident doesn't require guesswork.

3. Batch push timestamp

Remote-write encoding now stamps every sample in a WriteRequest with one wall-clock time at export. That is standard push-agent behaviour and avoids mixed timestamps within a single push. It did not replace the need for service / instance tags.

How we verified ingestion (not just "no errors")

Empty HTTP failures are necessary but not sufficient. We confirmed data was stored and queryable:

QUERY_ENDPOINT="https://aps-workspaces.us-west-2.amazonaws.com/workspaces/${WORKSPACE_ID}/api/v1/query"

awscurl --service aps --region us-west-2 \
  -X POST "${QUERY_ENDPOINT}" \
  -H 'Content-Type: application/x-www-form-urlencoded' \
  -d 'query=forge_observability_amp_push_success_total'

Success counters for all six workloads, each with service and instance labels. Failure metric absent (never incremented since deploy). CloudWatch discards at zero post-rollout.

That is the bar: PromQL returns your series with the labels you expect, and DiscardedSamples stays quiet.

Appendix: debugging when you can't reach `/q/metrics`

Our laptops can't hit task private IPs in a NAT-less VPC. To compare native exposition with the JVM encoder, we used a one-off Fargate task pattern:

Publish a tiny private ECR image (busybox + wget) — Docker Hub and public ECR aren't reachable from the VPC without NAT.
Run a task in the cluster that wgets http://<task-private-ip>:8080/q/metrics (same path the target group uses for health checks).
Pull logs into a local file and feed it to a JVM probe (FORGE_AMP_METRICS_FILE=...).

We also briefly added a same-SG ingress on 8080 so the scrape task could reach the service ENI — then revoked it after debugging. That rule is not part of normal infra; ALB → task rules stay as designed.

This tooling was valuable for one INT session. We don't plan to keep the scrape image or scripts in the application repo long term — the durable lessons are the CloudWatch reason dimension, the multi-writer tag discipline, and the PromQL verification queries.

Takeaways

AMP HTTP 400 with an empty body sends you down a long hallway. DiscardedSamples by Reason is often the door.
new-value-for-timestamp on a shared workspace screams label collision — multiple writers, identical series, aligned push intervals.
Micrometer MeterFilter.commonTags with low-cardinality service (and instance when you scale replicas) is not optional when many processes remote-write to one Prometheus-compatible backend.
Single-service probes prove encoding, not fleet behaviour. Reproduce collisions with multi-writer thinking.
Confirm consumption with PromQL and CloudWatch ingestion metrics — not only "the error log stopped."

If you're wiring Micrometer → AMP on ECS, check your tags before you tune Snappy JNI or second-guess your SigV4 implementation.

The platform was fine. We were just telling six services to sign the same name on the same line.

The software nobody plans to build - but every successful team eventually does...

Andrew Eells — Fri, 26 Jun 2026 19:59:56 +0000

Every software company starts with one product.

The product customers buy.

The product investors care about.

The product the roadmap revolves around.

But given enough time, almost every successful engineering team finds itself building something else.

Not because they planned to.

Because they have to.

The second product

It doesn't have a marketing website.

Customers never ask for it by name.

Nobody demos it to investors.

Yet it quietly grows alongside the business.

It's your engineering foundations.

The authentication services.

The deployment pipelines.

The infrastructure.

The observability stack.

The audit logging.

The notification system.

The release processes.

The operational tooling.

The architectural conventions.

Individually, none of these things are your product.

Collectively, they're what allow your product to scale.

It happens one decision at a time

Nobody sets out to spend months building engineering foundations.

Instead, they make perfectly reasonable decisions.

"We'll automate deployments later."

"This service just needs its own authentication for now."

"We'll improve the monitoring once we've got more customers."

"We'll standardise this after the next release."

Every one of those decisions is commercially rational.

Product delivery has to come first.

But each compromise adds another piece to a second codebase that the business never intended to own.

The hidden backlog

Earlier in my career, while CTO at a UK insurance startup, I estimated that around 60% of our engineering backlog wasn't product work at all.

It was engineering foundations.

Improving CI/CD.

Standardising infrastructure.

Strengthening security.

Adding observability.

Operational tooling.

Developer experience.

Performance and security testing.

None of those stories generated revenue directly.

But every one of them made future product delivery faster, safer and more predictable.

Eventually we realised something important:

We weren't just building an insurance platform anymore.

We were also building the engineering foundations that made the insurance platform possible.

The Groundhog Day problem

Over the last 20+ years I've worked across startups, consultancies and enterprise engineering teams.

Different industries.

Different products.

Different company sizes.

Yet I kept rebuilding remarkably similar engineering foundations.

Authentication.

Audit.

Notifications.

Infrastructure.

Deployment pipelines.

Observability.

Release management.

Security controls.

Developer tooling.

Different implementations.

The same problems.

After a while it started to feel like Groundhog Day.

Every organisation was independently solving problems that thousands of engineering teams had already solved before.

Engineering foundations are inevitable

This isn't an argument against building engineering foundations.

They're essential.

Every successful software company eventually needs them.

The question is simply when and how you build them.

Do you invest months (or years) constructing them incrementally while trying to deliver product?

Or do you begin with mature engineering foundations already in place and let your team focus on the capabilities that actually differentiate your business?

That's a very different starting position.

Final thoughts

I've come to believe that one of the biggest hidden costs in software engineering isn't technical debt.

It's repeatedly rebuilding the same engineering foundations.

Not because they're unique.

But because every organisation assumes it has to start from scratch.

That realisation is ultimately what led me to build Forge Platform.

After spending more than two decades repeatedly building the same operational capabilities across startups and enterprise programmes, I wanted to create something that lets engineering teams begin with mature foundations already in place - so more of their time is spent building the product they're actually in business to create.

If that resonates, you can learn more at https://forgeplatform.software.

What "production-ready" actually means - and why most teams discover it too late.

Andrew Eells — Sat, 06 Jun 2026 03:54:10 +0000

"Production-ready" is one of the most misused phrases in software engineering.

It usually means:

it runs
it deploys
it works in a happy path

But in real systems, production readiness is not about functionality.

It’s about behaviour under failure, change, and scale.

The difference between working and production-ready

A system is not production-ready when:

it can be deployed

It is production-ready when:

it can fail safely
it can be observed
it can be redeployed without service interruption
it behaves consistently under load
it can be operated by people who did not build it

Most early-stage systems do not meet this bar.

Not because teams are careless - but because these properties are usually added after the system exists.

The problem with "we'll add it later"

In practice, "later" becomes:

after customers arrive
after scale pressure begins
after incidents expose gaps
after engineering velocity slows

At that point, the system is no longer neutral.

It has opinions:

about structure
about deployment
about observability
about service boundaries

And those opinions are expensive to change.

Where teams actually spend their time

Across multiple environments I’ve worked in — from startups to large AWS-based enterprise systems — a consistent pattern appears:

Engineering effort splits into two categories:

domain / product requirements and features
engineering foundation and operational work

In many early systems, the work that goes into engineering foundations - such as deployments, versioning, build and test standards and optimisations, pipelines etc. - becomes a dominant and usually hidden cost.

At one startup, my estimate was that "technical" stories accounted for the majority of backlog creation over time, eclipsing feature development.

This is not an edge case.

This is how systems evolve.

Why this is so hard to avoid

Most teams don't consciously choose to neglect operational maturity.

The problem is that product work is always visible, while engineering foundations are largely invisible.

A new feature can be demonstrated to customers, investors, and stakeholders. It can be tied directly to revenue, growth, or market validation. Improvements to deployment pipelines, observability, security controls, or operational tooling rarely have that luxury. Their value is indirect, preventative, and often only becomes obvious when something goes wrong.

As a result, engineering teams are under constant pressure to prioritise business-driven outcomes over engineering excellence. Every sprint presents another feature request, customer commitment, sales opportunity, or roadmap deadline competing for attention.

Over time, small compromises accumulate:

Deployment processes remain partially manual because "we'll automate it later."
Monitoring exists, but not at the depth needed to diagnose production issues quickly.
Security controls are good enough for today's customers, but not tomorrow's.
Operational knowledge lives in people's heads rather than in systems and documentation.

None of these decisions are unreasonable in isolation. In fact, most are rational responses to commercial pressure.

The challenge is that operational maturity compounds in exactly the same way technical debt does. The cost of postponing it is often hidden until growth, scale, compliance requirements, or a production incident suddenly expose the gap.

By that point, fixing the foundations is competing with an even larger backlog, a larger customer base, and a business that has become increasingly dependent on systems that were never designed for the level of demand being placed on them.

The real definition of production-ready

A more accurate definition is:

A system is production-ready when its operational properties are designed, not discovered.

That includes:

observability as a first-class concern
consistent service structure and bounded contexts
predictable deployment behaviour
explicit failure handling patterns
security and access boundaries defined early

The uncomfortable truth

Most teams don’t lack capability.

They lack a reusable starting point.

So they rebuild production-readiness repeatedly, instead of inheriting it once.

The shift that matters

The real architectural question is not:

"How do we make this production-ready?"

It is:

"Why are we rebuilding production readiness every time?"

This is the problem space I’ve been focused on with Forge: creating a reusable foundation so teams don’t rediscover production-readiness under pressure.

The real startup killer isn't product — it's building platform foundations from scratch.

Andrew Eells — Fri, 08 May 2026 22:15:14 +0000

There's a pattern I've seen repeat a few times over 20 years building software systems in both startup and enterprise environments.

Most early-stage teams believe they are building a product.

In reality, they are building two things at once:

Their actual product
An internal platform they didn't intend to build

And the second one quietly consumes man-years of engineering time.

The hidden tax on early-stage teams

At some point, every startup hits the same phase:

The MVP works
The first customers arrive
Engineering velocity starts to slow
"Just one more service" becomes a platform discussion

And suddenly the backlog shifts.

Not because the product changed - but because the foundations underneath it needed to evolve (without the wheels falling off!).

Based on greenfield experience, I'd guesstimate that up to 60% of engineering effort can be focused on platform foundations, delivery lifecycle and operational processes, not domain features or core business differentiators.

That ratio is not unusual.

It is normal.

And it is also destructive.

The repeated mistake

Across organisations, I've seen the same systems rebuilt repeatedly:

CI/CD pipelines
Infrastructure-as-code structures
Observability setups
Security and access control patterns
Service templates and API conventions
Deployment and rollback strategies

Each time:

slightly different
slightly inconsistent
always re-learned under pressure
and generally whilst trying to keep production lights on

The irony is that none of these are domain-specific.

They are global engineering concerns.

Yet every team reinvents them.

Why this keeps happening

It's not incompetence.

It's timing.

Early-stage teams optimise for:

speed
product delivery
survival

So they defer platform thinking until it becomes unavoidable.

At which point:

the system is already live
constraints are already baked in
rewrites are expensive

So they rebuild under pressure instead of designing under intention.

The real cost

The cost isn't just engineering time.

It's:

delayed product delivery
inconsistent system behaviour
increased operational risk
premature senior hiring in platform roles
architectural fragmentation across services (and teams)

Most importantly:

It shifts engineering from building product value to managing internal complexity.

What changes when you design it once

When these concerns are treated as a reusable foundation instead of one-off decisions:

teams ship faster
systems stay consistent
operational overhead drops
architecture stops diverging across services
engineering focus returns to product domain logic

This is the problem I've been formalising into an opinionated bootstrap approach with Forge Platform.

It enables production-grade microservices from day one with enterprise foundations at startup speed.

Not because teams cannot build these things.

But because they keep building them repeatedly, under pressure, in slightly different ways, everywhere.

The question I keep coming back to

If every team ends up rebuilding the same foundations…

Why are we still rebuilding them every time?

If you have any questions or are interested to find out more, it would be great to hear from you in the comments or feel free to say hello.

DEV Community: Andrew Eells

We've Spent 25 Years Shifting Left. The Next Shift Is Down.

Twenty-five years of shifting left

AI isn't really shifting left

The pattern I couldn't ignore

Forge is about shifting down

Left was never the destination

AWS Transform and Superwerker: same vocabulary, different problems

Superwerker: opinionated defaults, shipped

Transform: a different problem

Where Forge sits

A working rule of thumb

How AI finally solved AWS architecture diagrams

The problem with architecture diagrams

Manual diagrams

CDK diagram generators

AWS Workload Discovery

The breakthrough

Cursor did almost all of the work

That's a far harder problem than it sounds

Was it perfect first time?

The hybrid workflow

Why this works

AI finally solved the right problem

Would I trust it blindly?

What's next?

Here's how Cursor did it

Final thoughts

AMP Remote Write HTTP 400: The CloudWatch Metric That Solved It

The setup

What we ruled out first

Wrong URL

SigV4 and transport

"A bad metric family"

The pivot: CloudWatch DiscardedSamples

The actual bug: six services, one series

Why the JVM probe didn't catch it

Why hasScrapeTimestamp() wasn't the story

The fix

1. Common tags (the real fix)

2. Richer failure logging

3. Batch push timestamp

How we verified ingestion (not just "no errors")

Appendix: debugging when you can't reach /q/metrics

Takeaways

The software nobody plans to build - but every successful team eventually does...

The second product

It happens one decision at a time

The hidden backlog

The Groundhog Day problem

Engineering foundations are inevitable

Final thoughts

What "production-ready" actually means - and why most teams discover it too late.

The difference between working and production-ready

The problem with "we'll add it later"

Where teams actually spend their time

Why this is so hard to avoid

The real definition of production-ready

The uncomfortable truth

The shift that matters

The real startup killer isn't product — it's building platform foundations from scratch.

The hidden tax on early-stage teams

The repeated mistake

Why this keeps happening

The real cost

What changes when you design it once

The question I keep coming back to

The pivot: CloudWatch `DiscardedSamples`

Why `hasScrapeTimestamp()` wasn't the story

Appendix: debugging when you can't reach `/q/metrics`