<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: Andrew Eells</title>
    <description>The latest articles on DEV Community by Andrew Eells (@aeells).</description>
    <link>https://dev.to/aeells</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F3920704%2Fc98d7902-1d6d-428b-a0c2-418f69b17e2d.jpg</url>
      <title>DEV Community: Andrew Eells</title>
      <link>https://dev.to/aeells</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/aeells"/>
    <language>en</language>
    <item>
      <title>Vibe Coding Is Reckless. Defined Architecture Patterns Fix It.</title>
      <dc:creator>Andrew Eells</dc:creator>
      <pubDate>Mon, 10 Aug 2026 06:17:34 +0000</pubDate>
      <link>https://dev.to/aeells/vibe-coding-is-recklessdefined-architecture-patterns-fix-it-2k27</link>
      <guid>https://dev.to/aeells/vibe-coding-is-recklessdefined-architecture-patterns-fix-it-2k27</guid>
      <description>&lt;p&gt;Andreas Horn leads AIOps work at BLP - real enterprise IT. A while back he &lt;a href="https://www.linkedin.com/posts/andreashorn1_unpopular-opinion-most-of-the-people-confidently-activity-7471821090412138496-QfVi" rel="noopener noreferrer"&gt;posted something&lt;/a&gt; that stuck with me: most people calling vibe coding the future have never sat through a 2 a.m. call after a misconfigured permission policy or firewall rule took production down.&lt;/p&gt;

&lt;p&gt;Demos look great. Small projects hold up. None of it survives a real enterprise, where system complexity and constraints are the default.&lt;/p&gt;

&lt;p&gt;He's right. He's also spent a decade watching AI initiatives succeed or fail inside real companies.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F8czg3lx8dg7hsur094pv.jpeg" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F8czg3lx8dg7hsur094pv.jpeg" alt="Shipping code is easy. Shipping into enterprise complexity is the work." width="800" height="1200"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Here's the sharper point: this isn't a talent problem, and it isn't a caution problem. It's a constraint problem. The &lt;a href="https://en.wikipedia.org/wiki/List_of_system_quality_attributes" rel="noopener noreferrer"&gt;architectural "-ilities"&lt;/a&gt; - scalability, security, maintainability, observability, and the rest - are exactly the properties teams reinvent under deadline when no convention exists.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://12factor.net/" rel="noopener noreferrer"&gt;The twelve-factor app&lt;/a&gt; made this same bet well over a decade ago: a fixed convention for how a SaaS app runs in production, not something every team invents from scratch. That bet gets sharper once the thing touching your system is an agent, not a person.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why vibe coding breaks
&lt;/h2&gt;

&lt;p&gt;A coding agent doesn't reason about your architecture. It reads what's there and extends it. Give it a codebase where the -ilities are already expressed the same way a dozen times, and it follows those patterns reliably. Give it a blank project - which is what vibe coding usually means - and it has nothing to follow. It invents the patterns instead: how identity is checked, what gets logged, how failures propagate, where rate limits live. The result might be plausible. It is not always production-grade.&lt;/p&gt;

&lt;p&gt;That gap shows up in public. &lt;a href="https://www.wired.com/story/thousands-of-vibe-coded-apps-expose-corporate-and-personal-data-on-the-open-web/" rel="noopener noreferrer"&gt;WIRED reported&lt;/a&gt; research from Red Access finding thousands of publicly accessible vibe-coded apps with effectively no authentication - and close to two thousand appearing to leak sensitive data. Security is only one -ility. The same blank-canvas pressure applies to scalability, observability, deployability, and maintainability. Agents optimise for something that works. The cross-cutting properties are what they have to guess.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why Backbone fixes it
&lt;/h2&gt;

&lt;p&gt;Two agents, same task: add a feature that touches identity, authorization, and an audit trail - the kind of change that forces several -ilities into play at once.&lt;/p&gt;

&lt;p&gt;On a blank project, the agent invents those concerns as it goes: where identity lives, how authorization is checked, what gets audited, how operational boundaries are drawn. Decisions about the architecture happen while the feature is being written.&lt;/p&gt;

&lt;p&gt;On Backbone, those decisions already exist. Authentication is wired to Cognito and OAuth2. Authorization runs through an existing identity model. Audit events already have a schema, cross-cutting wiring, and a destination. Deployment, configuration, health checks, observability, and rate limiting are already established. The agent isn't designing the -ilities. It's adhering to them - extending a system with known scalability, reliability, and security characteristics rather than inventing those characteristics feature by feature.&lt;/p&gt;

&lt;p&gt;That's the bet behind Backbone: agents are better at extending an established pattern than inventing production architecture from scratch. Service-to-service calls use explicit AWS IAM identities and policies, not credentials embedded in application code - one more decision that's already made, not one your agent has to guess at.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://docs.backbonehq.io/docs/adrs" rel="noopener noreferrer"&gt;Published ADRs&lt;/a&gt; capture the constraints behind those patterns - why Cognito, why IAM for service auth, why the observability stack looks the way it does. Not marketing. The trade-offs.&lt;/p&gt;

&lt;h2&gt;
  
  
  Where this stops working
&lt;/h2&gt;

&lt;p&gt;Convention only helps if it's right. A flawed pattern gets propagated faster and more confidently by an agent than by a human - it doesn't second-guess precedent, it follows it. That's an argument for getting the pattern right once, by people who've made the mistakes already, not for skipping review. An agent can follow a correct pattern into a situation it was never built for, and catching that is still your job.&lt;/p&gt;

&lt;p&gt;One more thing: "vibe coding" might not be the word anyone's using in two years. The shift underneath it - agents doing more of the typing, over bigger diffs, with less supervision - isn't going anywhere.&lt;/p&gt;

&lt;h2&gt;
  
  
  Architecture engineering, not prompt engineering
&lt;/h2&gt;

&lt;p&gt;The real distinction isn't whether AI wrote the code. It's whether the agent was working inside architecture that already solved the -ilities - identity, authorization, auditing, observability, configuration, deployment - or making those calls up as it went.&lt;/p&gt;

&lt;p&gt;If those decisions don't exist, the agent invents them while it builds your feature. If they do exist, it reinforces them.&lt;/p&gt;

&lt;p&gt;As coding agents become capable of producing larger and larger changesets, that's the bet that matters: not prompt engineering, but architecture engineering.&lt;/p&gt;

</description>
      <category>ai</category>
      <category>architecture</category>
      <category>softwareengineering</category>
      <category>devops</category>
    </item>
    <item>
      <title>Quarkus + GitHub Actions: Reducing CI Time by Half Without Larger Runners</title>
      <dc:creator>Andrew Eells</dc:creator>
      <pubDate>Fri, 07 Aug 2026 02:36:51 +0000</pubDate>
      <link>https://dev.to/aeells/quarkus-github-actions-reducing-ci-time-by-half-without-larger-runners-51ao</link>
      <guid>https://dev.to/aeells/quarkus-github-actions-reducing-ci-time-by-half-without-larger-runners-51ao</guid>
      <description>&lt;p&gt;Our Quarkus monorepo CI pipeline had plateaued at &lt;strong&gt;9–11 minutes&lt;/strong&gt; on GitHub Actions free-tier hosted runners.&lt;/p&gt;

&lt;p&gt;After a focused optimisation pass, it now completes in &lt;strong&gt;about 5 minutes 20 seconds&lt;/strong&gt;, &lt;strong&gt;without&lt;/strong&gt; self-hosted runners, larger GitHub runners, or paid caching products.&lt;/p&gt;

&lt;p&gt;The key discovery was that the remaining time was &lt;strong&gt;not&lt;/strong&gt; in dependency downloads or Maven compilation. It was in &lt;strong&gt;duplicate Quarkus work&lt;/strong&gt;: repeated augmentation, repeated application boots, and infrastructure startup that some test suites did not actually need.&lt;/p&gt;

&lt;p&gt;This post walks through the changes that produced the biggest gains, the ones that barely moved the needle, and the final workflow shape.&lt;/p&gt;

&lt;h2&gt;
  
  
  The baseline was already heavily optimised
&lt;/h2&gt;

&lt;p&gt;Before touching anything, the workflow already had:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Reactor parallelism via &lt;code&gt;.mvn/maven.config&lt;/code&gt; (&lt;code&gt;--threads=2C&lt;/code&gt;)&lt;/li&gt;
&lt;li&gt;Maven Build Cache extension for local incremental builds&lt;/li&gt;
&lt;li&gt;Surefire and Failsafe configured for &lt;code&gt;parallel=classes&lt;/code&gt; with an explicit thread count&lt;/li&gt;
&lt;li&gt;A split &lt;strong&gt;build → artifact restore → test&lt;/strong&gt; shape so test jobs did not rebuild the world&lt;/li&gt;
&lt;li&gt;Parallel composite setup where it was safe (JDK + tooling overlapping other preparation)&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;That matters, because it changes the question from &lt;em&gt;"how do we make Maven faster?"&lt;/em&gt; to &lt;em&gt;"what is still dominating wall clock after the obvious tuning is already in place?"&lt;/em&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  Where we started
&lt;/h2&gt;

&lt;p&gt;Backbone's platform repository is a Maven multi-module Quarkus monorepo: shared libraries, several application services, a BFF API, UI modules, and scaffold code. The module names below (&lt;code&gt;actor&lt;/code&gt;, &lt;code&gt;audit&lt;/code&gt;, &lt;code&gt;auth-service&lt;/code&gt;, etc.) are illustrative; the important point is the &lt;strong&gt;shape&lt;/strong&gt; of the reactor rather than the specific repo layout.&lt;/p&gt;

&lt;p&gt;A rough &lt;code&gt;cloc&lt;/code&gt; of the parts CI actually cares about (Java, YAML, shell; excluding &lt;code&gt;target&lt;/code&gt;, &lt;code&gt;node_modules&lt;/code&gt;, etc.) lands around &lt;strong&gt;~24k lines of code&lt;/strong&gt;, including &lt;strong&gt;~14k Java&lt;/strong&gt; across roughly &lt;strong&gt;300 source files&lt;/strong&gt;. Big enough that cold Quarkus boots matter; small enough that a vanity matrix of one job per module mostly pays setup tax.&lt;/p&gt;

&lt;p&gt;The critical workflow looked like this:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;build&lt;/strong&gt;: &lt;code&gt;mvn install -DskipTests&lt;/code&gt;, upload jars/classes/local &lt;code&gt;io.backbone&lt;/code&gt; artifacts&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;unit-test&lt;/strong&gt;: Surefire against the restored reactor&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;integration-test&lt;/strong&gt;: Failsafe &lt;code&gt;@QuarkusTest&lt;/code&gt; classes with local AWS emulation and/or Postgres&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;A representative run before the major changes looked like this:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Job&lt;/th&gt;
&lt;th&gt;Wall clock&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;build&lt;/td&gt;
&lt;td&gt;~2.5 min&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;int-test&lt;/td&gt;
&lt;td&gt;~8.5 min&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;workflow&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;~11.4 min&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;The optimisation target was the &lt;strong&gt;end-to-end workflow wall clock&lt;/strong&gt; (checkout through reporters), not a single Maven invocation in isolation.&lt;/p&gt;

&lt;p&gt;One integration-test &lt;code&gt;verify&lt;/code&gt; invocation reported:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;[INFO] Total time:  02:09 min (Wall Clock)
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;If Maven itself said &lt;strong&gt;2 minutes&lt;/strong&gt; while the workflow still took &lt;strong&gt;10–11 minutes&lt;/strong&gt;, the missing time was clearly &lt;strong&gt;job setup, emulator startup, artifact restore, and Quarkus application boots&lt;/strong&gt;. That observation ended up driving almost every useful optimisation that followed.&lt;/p&gt;

&lt;h2&gt;
  
  
  The wrong instinct: copy the ECS deploy matrix
&lt;/h2&gt;

&lt;p&gt;We already parallelise ECS deploys with a GitHub Actions matrix, so the obvious idea was &lt;em&gt;"one integration-test job per service"&lt;/em&gt;.&lt;/p&gt;

&lt;p&gt;That would have been a mistake.&lt;/p&gt;

&lt;p&gt;Each matrix cell would have re-paid:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;JDK and tool setup&lt;/li&gt;
&lt;li&gt;Artifact download&lt;/li&gt;
&lt;li&gt;AWS emulator startup&lt;/li&gt;
&lt;li&gt;Cognito / DynamoDB / S3 seeding where required&lt;/li&gt;
&lt;li&gt;Postgres startup where required&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;We only had a handful of integration-test classes. Fixed setup cost multiplied by N shards usually loses to &lt;strong&gt;one job that pays setup once and parallelises the Maven reactor internally&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;So we kept the job count small and focused on &lt;strong&gt;eliminating duplicated work inside the existing jobs&lt;/strong&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  Prerequisite: LocalStack → Floci
&lt;/h2&gt;

&lt;p&gt;Before this CI pass, we had already migrated from &lt;a href="https://localstack.cloud/" rel="noopener noreferrer"&gt;LocalStack&lt;/a&gt; to &lt;a href="https://floci.io/floci/" rel="noopener noreferrer"&gt;Floci&lt;/a&gt; as the local AWS emulator on &lt;code&gt;:4566&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;The original motivation was &lt;strong&gt;Cognito support&lt;/strong&gt;, not CI speed. LocalStack Community did not provide a credible Cognito story for our authentication flows, so auth integration tests previously required a &lt;strong&gt;real AWS-backed job&lt;/strong&gt; alongside a LocalStack-backed job for the other services.&lt;/p&gt;

&lt;p&gt;Floci supports enough of Cognito for our password, refresh-token, and JWT flows that we could collapse those two worlds into a &lt;strong&gt;single local emulator&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;It was also noticeably lighter in CI.&lt;/p&gt;

&lt;h3&gt;
  
  
  LocalStack startup (older run)
&lt;/h3&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Step&lt;/th&gt;
&lt;th&gt;Wall clock&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;./.github/actions/localstack&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;~49 s&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;h3&gt;
  
  
  Floci startup (current run)
&lt;/h3&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;floci start --detach --image floci/floci:1.5.34
floci wait --timeout 2m
...
Waiting... (119s remaining)   Floci AWS is ready (http://localhost:4566)
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The entire start-and-wait sequence completed in &lt;strong&gt;about 16 seconds&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;Smaller footprint, no auth-token tax for CI, and Cognito in the same process that already emulates DynamoDB, S3, and SES. That consolidation mattered more than the raw startup-time improvement.&lt;/p&gt;

&lt;p&gt;The migration was not entirely frictionless. Floci's &lt;code&gt;InitiateAuth&lt;/code&gt; + &lt;code&gt;REFRESH_TOKEN_AUTH&lt;/code&gt; path initially accepted invalid refresh tokens and minted new JWTs, which broke our negative-path authentication tests. We filed &lt;a href="https://github.com/floci-io/floci/issues/2113" rel="noopener noreferrer"&gt;floci-io/floci#2113&lt;/a&gt;, and it quickly attracted two fix PRs (&lt;a href="https://github.com/floci-io/floci/pull/2132" rel="noopener noreferrer"&gt;#2132&lt;/a&gt; and &lt;a href="https://github.com/floci-io/floci/pull/2135" rel="noopener noreferrer"&gt;#2135&lt;/a&gt;). That kind of maintainer response is exactly what makes an emulator migration feel safe rather than lonely.&lt;/p&gt;

&lt;h2&gt;
  
  
  Step 1: actually forward the Maven flags
&lt;/h2&gt;

&lt;p&gt;CI had been invoking:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;task &lt;span class="nb"&gt;test&lt;/span&gt;:integration &lt;span class="nt"&gt;--&lt;/span&gt; &lt;span class="nt"&gt;-Dtest&lt;/span&gt;.thread.count&lt;span class="o"&gt;=&lt;/span&gt;2
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;but the Task definition never forwarded &lt;code&gt;{{.CLI_ARGS}}&lt;/code&gt;, so Failsafe never saw the thread-count override.&lt;/p&gt;

&lt;p&gt;After fixing that, we enabled reactor parallelism for the integration-test invocation:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;task &lt;span class="nb"&gt;test&lt;/span&gt;:integration &lt;span class="nt"&gt;--&lt;/span&gt; &lt;span class="nt"&gt;-Dtest&lt;/span&gt;.thread.count&lt;span class="o"&gt;=&lt;/span&gt;2 &lt;span class="nt"&gt;-T&lt;/span&gt; 1C
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;This helped, but only modestly. The dominant cost was still &lt;strong&gt;Quarkus application startup&lt;/strong&gt;, not Maven's ability to schedule modules concurrently.&lt;/p&gt;

&lt;h2&gt;
  
  
  Step 2: treat Quarkus boots as a budget
&lt;/h2&gt;

&lt;p&gt;Every &lt;code&gt;@QuarkusTest&lt;/code&gt; class pays for an application start. Once we started looking at boots as a &lt;strong&gt;finite budget&lt;/strong&gt;, several easy wins appeared.&lt;/p&gt;

&lt;h3&gt;
  
  
  What changed
&lt;/h3&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Excluded scaffold integration tests from CI&lt;/strong&gt;: &lt;code&gt;template-service&lt;/code&gt; remains in-tree as a scaffold, but CI no longer boots Quarkus for a module that rarely changes.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Merged overlapping integration tests&lt;/strong&gt;: notification unsubscribe coverage moved into &lt;code&gt;NotificationResourceIT&lt;/code&gt;; Cognito user and service cases became nested tests under a single &lt;code&gt;CognitoAuthenticationProviderIT&lt;/code&gt; so they share one application boot.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Demoted thin HTTP integration tests to unit tests&lt;/strong&gt;: &lt;code&gt;DocumentResourceIT&lt;/code&gt; became focused unit tests around the Tika, S3, and DynamoDB behaviour we actually cared about.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The suite still provided meaningful coverage, but we were no longer starting Quarkus for every historical convenience test class.&lt;/p&gt;

&lt;p&gt;At this point the full workflow was down to &lt;strong&gt;roughly 9.5 minutes&lt;/strong&gt;: an improvement, but nowhere near the eventual 5-minute target.&lt;/p&gt;

&lt;h2&gt;
  
  
  Step 3: split integration tests by infrastructure
&lt;/h2&gt;

&lt;p&gt;The remaining integration tests naturally separated into two infrastructure groups:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Job&lt;/th&gt;
&lt;th&gt;Infrastructure&lt;/th&gt;
&lt;th&gt;Modules&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;int-auth-test&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Floci + Cognito seed&lt;/td&gt;
&lt;td&gt;&lt;code&gt;auth-service&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;int-rest-test&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Postgres&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;actor&lt;/code&gt;, &lt;code&gt;audit&lt;/code&gt;, &lt;code&gt;notification&lt;/code&gt;
&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;The important observation was that &lt;strong&gt;auth tests did not need Postgres&lt;/strong&gt;, and the REST-resource tests did not need DynamoDB template seeds or Cognito setup.&lt;/p&gt;

&lt;p&gt;So we stopped starting &lt;strong&gt;Floci and Postgres in the same job&lt;/strong&gt;, and we stopped seeding notification templates when nothing in the test suite actually read them.&lt;/p&gt;

&lt;p&gt;The critical-path shape changed from:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;build + (Floci + Postgres + all ITs serially)
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;to:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;build + max(unit, int-auth, int-rest)
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;That was the first optimisation that made the workflow feel &lt;strong&gt;dramatically&lt;/strong&gt; shorter, bringing it down to &lt;strong&gt;around 6.5 minutes&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;A useful side note: avoid parallelising &lt;code&gt;shell-tools&lt;/code&gt; (which uses &lt;code&gt;apt&lt;/code&gt;) with &lt;code&gt;postgres&lt;/code&gt; (which also uses &lt;code&gt;apt&lt;/code&gt;). That turned into a classic package-manager lock race. Floci was safe to overlap with Temurin and shell-tool setup because it does not compete for &lt;code&gt;apt&lt;/code&gt; locks.&lt;/p&gt;

&lt;h2&gt;
  
  
  Step 4: stop using Graal where you are not building native images
&lt;/h2&gt;

&lt;p&gt;Native-image builds still require GraalVM. Unit tests and &lt;code&gt;@QuarkusTest&lt;/code&gt; do not.&lt;/p&gt;

&lt;p&gt;We introduced a &lt;strong&gt;Temurin 25 composite action&lt;/strong&gt; for all non-native workflows and kept GraalVM only on the ECR native-image pipeline.&lt;/p&gt;

&lt;p&gt;This was not a massive wall-clock win (roughly &lt;strong&gt;20 seconds&lt;/strong&gt;), but it reduced runner setup overhead and removed a surprising amount of cognitive noise from the workflow definitions.&lt;/p&gt;

&lt;h2&gt;
  
  
  Step 5: make CI install skip &lt;code&gt;quarkus:build&lt;/code&gt;
&lt;/h2&gt;

&lt;p&gt;This was the cleanest Maven optimisation in the entire exercise.&lt;/p&gt;

&lt;p&gt;A normal Quarkus &lt;code&gt;mvn install&lt;/code&gt; performs &lt;strong&gt;package-time augmentation&lt;/strong&gt;. Our test jobs restore artifacts and then run &lt;code&gt;@QuarkusTest&lt;/code&gt;, which boots directly from the &lt;strong&gt;test classpath&lt;/strong&gt;. They do &lt;strong&gt;not&lt;/strong&gt; need the production runner jar produced by the build job.&lt;/p&gt;

&lt;p&gt;So the CI install step became:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;mvn &lt;span class="nb"&gt;install&lt;/span&gt; &lt;span class="nt"&gt;-Dmaven&lt;/span&gt;.test.skip &lt;span class="nt"&gt;-Dquarkus&lt;/span&gt;.build.skip&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="nb"&gt;true&lt;/span&gt; &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;-pl&lt;/span&gt; &lt;span class="s1"&gt;'!libs/reactor,!ui,!ui/web-actor,!services/template-service'&lt;/span&gt; &lt;span class="nt"&gt;-am&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The logs changed from multi-minute Quarkus augmentation phases to a stream of:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;[INFO] Skipping Quarkus build
[INFO] Skipping Quarkus build
...
[INFO] Total time:  31.650 s (Wall Clock)
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The &lt;strong&gt;build job&lt;/strong&gt; dropped from roughly &lt;strong&gt;2 minutes&lt;/strong&gt; to about &lt;strong&gt;80 seconds&lt;/strong&gt;, which removed a substantial chunk of the remaining critical path.&lt;/p&gt;

&lt;p&gt;Excluding &lt;code&gt;ui&lt;/code&gt;, &lt;code&gt;web-actor&lt;/code&gt;, and &lt;code&gt;template-service&lt;/code&gt; from the CI install reactor also mattered. Those modules were simply &lt;strong&gt;not what &lt;code&gt;02-build-test&lt;/code&gt; was trying to validate on every push&lt;/strong&gt;. One small Maven quirk worth noting: excluding &lt;code&gt;!ui&lt;/code&gt; alone is insufficient; Maven will still include the child module unless you also exclude &lt;code&gt;!ui/web-actor&lt;/code&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  Step 6: run Failsafe without packaging again
&lt;/h2&gt;

&lt;p&gt;After the install phase stopped producing runner jars, the integration-test jobs were still invoking &lt;code&gt;verify&lt;/code&gt;, which re-entered &lt;code&gt;package&lt;/code&gt; and &lt;code&gt;quarkus:build&lt;/code&gt; for the service modules.&lt;/p&gt;

&lt;p&gt;For &lt;code&gt;@QuarkusTest&lt;/code&gt;, that was pure wasted heat.&lt;/p&gt;

&lt;p&gt;The integration-test invocation became:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;mvn &lt;span class="nt"&gt;-pl&lt;/span&gt; services/auth-service &lt;span class="se"&gt;\&lt;/span&gt;
  test-compile &lt;span class="se"&gt;\&lt;/span&gt;
  failsafe:integration-test failsafe:verify &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;-DskipUnitTests&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;A recent auth-job run reported:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;[INFO] BUILD SUCCESS
[INFO] Total time:  01:21 min (Wall Clock)
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;We experimented with dropping &lt;code&gt;-am&lt;/code&gt; after artifact restore first. That cleaned up the reactor graph, but it barely moved wall clock because the upstream jars were relatively cheap.&lt;/p&gt;

&lt;p&gt;The &lt;strong&gt;real lever&lt;/strong&gt; was skipping package-time augmentation during &lt;code&gt;install&lt;/code&gt;; &lt;strong&gt;Failsafe-without-package&lt;/strong&gt; was the complementary optimisation on the test side.&lt;/p&gt;

&lt;h2&gt;
  
  
  The finish line
&lt;/h2&gt;

&lt;p&gt;A current green run of &lt;code&gt;02-build-test&lt;/code&gt; on &lt;code&gt;main&lt;/code&gt; looks roughly like this:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Job&lt;/th&gt;
&lt;th&gt;Wall clock&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;build&lt;/td&gt;
&lt;td&gt;~1.3 min&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;int-rest-test&lt;/td&gt;
&lt;td&gt;~2.8 min&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;int-auth-test&lt;/td&gt;
&lt;td&gt;~3.7 min&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;unit-test&lt;/td&gt;
&lt;td&gt;~3.1 min&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;workflow&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;~5.5 min&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;That is &lt;strong&gt;roughly half of where we started&lt;/strong&gt;, and the number developers actually experience day-to-day is &lt;strong&gt;"about five minutes twenty"&lt;/strong&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  What barely moved the needle
&lt;/h2&gt;

&lt;p&gt;This is the section I wish more CI optimisation posts included. These ideas sounded promising during planning, but they were &lt;strong&gt;not&lt;/strong&gt; the reason the pipeline got fast:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Maven &lt;code&gt;-T&lt;/code&gt; by itself&lt;/strong&gt;: useful, but not transformative once Quarkus boots dominate.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Dropping &lt;code&gt;-am&lt;/code&gt; after artifact restore&lt;/strong&gt;: cleaner reactor hygiene, only a few seconds of savings.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Failsafe-without-package after install already skipped &lt;code&gt;quarkus:build&lt;/code&gt;&lt;/strong&gt;: a real improvement, but smaller than the install-phase optimisation.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Persisting Maven Build Cache in GitHub Actions&lt;/strong&gt;: still on the shelf; &lt;code&gt;actions/setup-java&lt;/code&gt; caches &lt;code&gt;~/.m2/repository&lt;/code&gt;, not &lt;code&gt;~/.m2/build-cache&lt;/code&gt;, so the extension kept writing cache entries without ever reading them across workflow runs.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;We stopped when the next ideas started looking like &lt;strong&gt;"fewer services"&lt;/strong&gt; or &lt;strong&gt;"don't boot Quarkus at all"&lt;/strong&gt;, which would have traded away the confidence the pipeline exists to provide.&lt;/p&gt;

&lt;h2&gt;
  
  
  The patterns worth stealing
&lt;/h2&gt;

&lt;p&gt;If I had to compress the whole exercise into five transferable lessons:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Budget Quarkus boots.&lt;/strong&gt; Merge overlapping tests, demote HTTP-thin integration tests, and exclude scaffold modules from CI.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Split jobs by infrastructure requirements, not by module count.&lt;/strong&gt; Auth + Floci versus REST + Postgres beat six tiny jobs that each restarted the world.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Match the JDK to the work.&lt;/strong&gt; GraalVM for native-image builds; Temurin for everything else.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Do not package what &lt;code&gt;@QuarkusTest&lt;/code&gt; will rebuild anyway.&lt;/strong&gt; &lt;code&gt;-Dquarkus.build.skip=true&lt;/code&gt; during CI install is boring, reliable, and surprisingly effective.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Choose an emulator that covers the awkward service.&lt;/strong&gt; Floci's Cognito support let us delete an entire AWS-shaped CI job category, not merely shave a few seconds off container startup.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;The final result was less about discovering a clever Maven flag and more about &lt;strong&gt;refusing to pay for the same Quarkus application three times on the way to asserting a single HTTP status code&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;We'll happily take &lt;strong&gt;five minutes twenty&lt;/strong&gt;.&lt;/p&gt;

</description>
      <category>devops</category>
      <category>quarkus</category>
      <category>maven</category>
      <category>gha</category>
    </item>
    <item>
      <title>We've Spent 25 Years Shifting Left. The Next Shift Is Down.</title>
      <dc:creator>Andrew Eells</dc:creator>
      <pubDate>Fri, 24 Jul 2026 07:28:29 +0000</pubDate>
      <link>https://dev.to/aeells/weve-spent-25-years-shifting-left-the-next-shift-is-down-19mh</link>
      <guid>https://dev.to/aeells/weve-spent-25-years-shifting-left-the-next-shift-is-down-19mh</guid>
      <description>&lt;p&gt;I've been building software professionally for about 25 years.&lt;/p&gt;

&lt;p&gt;Looking back, I realised that a surprising amount of that time has been spent doing one thing.&lt;/p&gt;

&lt;p&gt;Shifting left.&lt;/p&gt;

&lt;p&gt;Moving work earlier. Catching problems sooner. Automating the things we used to do by hand at the end of a project.&lt;/p&gt;

&lt;p&gt;It's been one of the defining trends of modern software engineering.&lt;/p&gt;

&lt;p&gt;But recently I've started wondering whether we've been optimising along the wrong axis.&lt;/p&gt;

&lt;h2&gt;
  
  
  Twenty-five years of shifting left
&lt;/h2&gt;

&lt;p&gt;Around the turn of the millennium, continuously building and testing software instead of integrating everything just before release was still a relatively new idea.&lt;/p&gt;

&lt;p&gt;Continuous integration emerged. CruiseControl popularised the approach, Hudson followed a few years later, and eventually Jenkins became the tool many teams knew best. "Does it build?" became a question answered automatically on every commit.&lt;/p&gt;

&lt;p&gt;That was shift-left number one.&lt;/p&gt;

&lt;p&gt;Testing and integration moved earlier.&lt;/p&gt;

&lt;p&gt;Then continuous integration became continuous &lt;strong&gt;delivery&lt;/strong&gt;. Martin Fowler and Jez Humble helped turn the idea into mainstream engineering practice. Building software wasn't enough; every commit produced something releasable.&lt;/p&gt;

&lt;p&gt;Then the "D" quietly changed meaning.&lt;/p&gt;

&lt;p&gt;Delivery became &lt;strong&gt;deployment&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;Releasing software stopped being a scheduled event and became something that happened continuously as a consequence of merging code.&lt;/p&gt;

&lt;p&gt;Then came DevOps.&lt;/p&gt;

&lt;p&gt;Operations moved left too. Infrastructure, monitoring, deployments and incident response all became part of the same team's responsibility.&lt;/p&gt;

&lt;p&gt;"You build it, you run it."&lt;/p&gt;

&lt;p&gt;Then security took its turn. &lt;a href="https://medium.com/slalom-build/devsecops-in-effect-at-the-web-application-firewall-472481688f8d" rel="noopener noreferrer"&gt;DevSecOps&lt;/a&gt; pushed security left as well - threat modelling, dependency scanning and policy checks moving out of a late-stage gate and into the pipeline itself.&lt;/p&gt;

&lt;p&gt;Every few years another discipline moved closer to the developer.&lt;/p&gt;

&lt;p&gt;Testing.&lt;/p&gt;

&lt;p&gt;Delivery.&lt;/p&gt;

&lt;p&gt;Deployment.&lt;/p&gt;

&lt;p&gt;Operations.&lt;/p&gt;

&lt;p&gt;Security.&lt;/p&gt;

&lt;p&gt;It genuinely made us better engineers.&lt;/p&gt;

&lt;h2&gt;
  
  
  AI isn't really shifting left
&lt;/h2&gt;

&lt;p&gt;Today it's fashionable to put AI into the same story.&lt;/p&gt;

&lt;p&gt;I'm not convinced that's what's happening.&lt;/p&gt;

&lt;p&gt;AI isn't primarily moving work earlier.&lt;/p&gt;

&lt;p&gt;It's helping us perform the same work faster.&lt;/p&gt;

&lt;p&gt;That's an important distinction.&lt;/p&gt;

&lt;p&gt;Where AI has changed things is by lowering the barrier to entry. Someone with relatively little operational experience can now generate a CI pipeline, Terraform configuration or deployment workflow in minutes.&lt;/p&gt;

&lt;p&gt;Useful?&lt;/p&gt;

&lt;p&gt;Absolutely.&lt;/p&gt;

&lt;p&gt;But if AI's biggest contribution is helping us recreate the same infrastructure more quickly, then perhaps recreating it was never the thing we should have been doing in the first place.&lt;/p&gt;

&lt;h2&gt;
  
  
  The pattern I couldn't ignore
&lt;/h2&gt;

&lt;p&gt;Here's what those 25 years actually looked like.&lt;/p&gt;

&lt;p&gt;I built CI pipelines.&lt;/p&gt;

&lt;p&gt;Then I built them again somewhere else.&lt;/p&gt;

&lt;p&gt;Then again.&lt;/p&gt;

&lt;p&gt;I created infrastructure-as-code.&lt;/p&gt;

&lt;p&gt;Then recreated it for the next company.&lt;/p&gt;

&lt;p&gt;I wired up authentication.&lt;/p&gt;

&lt;p&gt;Observability.&lt;/p&gt;

&lt;p&gt;Deployment pipelines.&lt;/p&gt;

&lt;p&gt;Audit logging.&lt;/p&gt;

&lt;p&gt;Secrets management.&lt;/p&gt;

&lt;p&gt;Monitoring.&lt;/p&gt;

&lt;p&gt;Rollback procedures.&lt;/p&gt;

&lt;p&gt;Different organisations.&lt;/p&gt;

&lt;p&gt;Different technology stacks.&lt;/p&gt;

&lt;p&gt;Different decades.&lt;/p&gt;

&lt;p&gt;Almost exactly the same engineering disciplines.&lt;/p&gt;

&lt;p&gt;None of that work was the product.&lt;/p&gt;

&lt;p&gt;None of it differentiated the business.&lt;/p&gt;

&lt;p&gt;It was simply the operational foundation required before the product could exist.&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;These aren't product problems. They're engineering fundamentals. Yet almost every team rebuilds them from scratch, every single time.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;Shift-left made us do this work earlier.&lt;/p&gt;

&lt;p&gt;It never asked why we were repeating it at all.&lt;/p&gt;

&lt;p&gt;Every startup, every consultancy and every enterprise greenfield project still begins by rebuilding the same engineering capabilities before a single customer problem gets solved.&lt;/p&gt;

&lt;h2&gt;
  
  
  Forge is about shifting down
&lt;/h2&gt;

&lt;p&gt;That's the shift I'm interested in now.&lt;/p&gt;

&lt;p&gt;Not left.&lt;/p&gt;

&lt;p&gt;Down.&lt;/p&gt;

&lt;p&gt;Imagine rotating the software lifecycle ninety degrees.&lt;/p&gt;

&lt;p&gt;Across the top sits your product: the services, APIs and user experience your customers actually pay for.&lt;/p&gt;

&lt;p&gt;Underneath sits everything every serious SaaS product inevitably needs.&lt;/p&gt;

&lt;p&gt;Authentication.&lt;/p&gt;

&lt;p&gt;Identity.&lt;/p&gt;

&lt;p&gt;CI/CD.&lt;/p&gt;

&lt;p&gt;Observability.&lt;/p&gt;

&lt;p&gt;Audit.&lt;/p&gt;

&lt;p&gt;Notifications.&lt;/p&gt;

&lt;p&gt;Document services.&lt;/p&gt;

&lt;p&gt;Infrastructure.&lt;/p&gt;

&lt;p&gt;Operational tooling.&lt;/p&gt;

&lt;p&gt;These capabilities aren't unique to your business.&lt;/p&gt;

&lt;p&gt;They're shared engineering concerns.&lt;/p&gt;

&lt;p&gt;So why are they still treated as project work?&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Shifting down means moving those undifferentiated engineering disciplines beneath your application into a reusable operational foundation instead of rebuilding them inside every new product.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;That's what Forge Platform is.&lt;/p&gt;

&lt;p&gt;Not another framework.&lt;/p&gt;

&lt;p&gt;Not another starter template.&lt;/p&gt;

&lt;p&gt;An opinionated operational backbone for modern SaaS products.&lt;/p&gt;

&lt;p&gt;Authentication, users, audit, notifications, documents and a BFF orchestration layer. Infrastructure-as-code. CI/CD. Observability. Security. Everything deployed into your own AWS account, from your own GitHub repositories, remaining entirely under your control.&lt;/p&gt;

&lt;p&gt;The disciplines I spent decades shifting left don't disappear.&lt;/p&gt;

&lt;p&gt;They simply stop living inside every application.&lt;/p&gt;

&lt;h2&gt;
  
  
  Left was never the destination
&lt;/h2&gt;

&lt;p&gt;Shift-left was absolutely the right idea.&lt;/p&gt;

&lt;p&gt;Finding problems earlier has transformed our industry.&lt;/p&gt;

&lt;p&gt;But somewhere along the way we confused the mechanism with the objective.&lt;/p&gt;

&lt;p&gt;The objective was never simply to push more responsibility onto developers.&lt;/p&gt;

&lt;p&gt;It was to stop repeatedly paying for the same engineering work in every company, on every project, for every new product.&lt;/p&gt;

&lt;p&gt;Some engineering problems only need solving once.&lt;/p&gt;

&lt;p&gt;The operational backbone beneath your SaaS product is one of them.&lt;/p&gt;

&lt;p&gt;After 25 years of shifting left, that feels like the direction we've really been heading all along.&lt;/p&gt;

&lt;p&gt;If that resonates, I'd genuinely like to hear from you - you can find out more at &lt;a href="https://forgeplatform.software/" rel="noopener noreferrer"&gt;forgeplatform.software&lt;/a&gt;.&lt;/p&gt;

</description>
      <category>engineering</category>
      <category>architecture</category>
      <category>ai</category>
      <category>discuss</category>
    </item>
    <item>
      <title>AWS Transform and Superwerker: same vocabulary, different problems</title>
      <dc:creator>Andrew Eells</dc:creator>
      <pubDate>Sat, 11 Jul 2026 18:04:14 +0000</pubDate>
      <link>https://dev.to/aeells/aws-transform-and-superwerker-same-vocabulary-different-problems-2m8g</link>
      <guid>https://dev.to/aeells/aws-transform-and-superwerker-same-vocabulary-different-problems-2m8g</guid>
      <description>&lt;p&gt;Most people will read &lt;a href="https://aws.amazon.com/blogs/migration-and-modernization/automate-your-landing-zone-creation-with-aws-transform/" rel="noopener noreferrer"&gt;AWS Transform's landing zone agent&lt;/a&gt; as Superwerker's smarter cousin. I don't think that's right.&lt;/p&gt;

&lt;p&gt;Same destination. Different problems. Treating them as substitutes is how greenfield teams buy enterprise process they don't need — and how migration teams under-buy governance they do.&lt;/p&gt;

&lt;p&gt;I saw the announcement via a LinkedIn post, and it stuck because the vocabulary overlaps heavily with a tool I've actually used and rate highly: &lt;a href="https://github.com/superwerker/superwerker" rel="noopener noreferrer"&gt;Superwerker&lt;/a&gt;. I haven't run Transform myself — this is a read from the announcement and docs, not a hands-on review. I'd genuinely like to hear from anyone who has.&lt;/p&gt;

&lt;h2&gt;
  
  
  Superwerker: opinionated defaults, shipped
&lt;/h2&gt;

&lt;p&gt;Superwerker is a free, MIT-licensed CloudFormation stack. You run it, and it stands up a multi-account baseline — Control Tower, SSO, and sensible guardrails — without a discovery workshop. We used it on a recent greenfield build and had SSO and baseline guardrails live in an afternoon. The entire decision tree is a public repo you can read end to end before you commit to anything.&lt;/p&gt;

&lt;p&gt;I'm generally a fan of opinionated tooling. If there is a sensible default, I'd rather spend engineering time building product than debating OU structures.&lt;/p&gt;

&lt;p&gt;If I were founding again tomorrow with a blank AWS org, I'd run Superwerker before I spent a day on account design. Greenfield teams don't fail because their landing zone wasn't migration-aware. They fail because they never got a baseline and spent months refactoring after the event.&lt;/p&gt;

&lt;h2&gt;
  
  
  Transform: a different problem
&lt;/h2&gt;

&lt;p&gt;Transform's agent is a different shape of tool entirely. It's not a standalone landing zone builder — it's embedded in AWS Transform's end-to-end migration workflow. Transform gathers information about your organisation, migration waves, compliance requirements and account structure before proposing a landing zone.&lt;/p&gt;

&lt;p&gt;That's a fundamentally different problem from "deploy me a sensible default."&lt;/p&gt;

&lt;h2&gt;
  
  
  Where Forge sits
&lt;/h2&gt;

&lt;p&gt;&lt;a href="https://forgeplatform.software/" rel="noopener noreferrer"&gt;Forge&lt;/a&gt; deliberately stops at the landing zone boundary. That's infrastructure governance, and every organisation has different requirements. Once that foundation exists — whether it came from Superwerker, AWS Transform or something else — that's where Forge starts.&lt;/p&gt;

&lt;h2&gt;
  
  
  A working rule of thumb
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Greenfield startup?&lt;/strong&gt; Use Superwerker.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Enterprise migration?&lt;/strong&gt; Look seriously at AWS Transform.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Brownfield AWS estate?&lt;/strong&gt; Use judgement. Every organisation carries different history and constraints.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;I haven't used Transform yet, so I'd genuinely like to hear from teams that have. I'm much more interested in where this rule breaks down than where it holds.&lt;/p&gt;

</description>
      <category>aws</category>
      <category>architecture</category>
      <category>ai</category>
      <category>devops</category>
    </item>
    <item>
      <title>How AI finally solved AWS architecture diagrams</title>
      <dc:creator>Andrew Eells</dc:creator>
      <pubDate>Fri, 10 Jul 2026 05:47:49 +0000</pubDate>
      <link>https://dev.to/aeells/how-ai-finally-solved-aws-architecture-diagrams-4ilf</link>
      <guid>https://dev.to/aeells/how-ai-finally-solved-aws-architecture-diagrams-4ilf</guid>
      <description>&lt;p&gt;After years of fighting draw.io, CloudFormation graphs and auto-generated spaghetti, architecture diagrams felt like the final boss of platform engineering - one I'd never quite beaten.&lt;/p&gt;

&lt;p&gt;I've been building software professionally for over 25 years.&lt;/p&gt;

&lt;p&gt;I have yet to be genuinely excited by AI's impact on large-scale software engineering. It's exceptionally good at bounded problems - a Bash script, a utility class, an isolated feature - and it's undeniably accelerating delivery. But building an entire production platform is a different challenge. That's still about architecture, trade-offs, operational thinking and years of accumulated engineering judgement.&lt;/p&gt;

&lt;p&gt;I don't believe "vibe coding" is replacing experienced engineers any time soon. AI is an extraordinary force multiplier, but it doesn't replace engineering judgement. If you're only capable of building poor software today, AI mostly helps you build poor software faster.&lt;/p&gt;

&lt;p&gt;But this...&lt;/p&gt;

&lt;p&gt;This genuinely felt like the first time AI solved &lt;em&gt;this category of&lt;/em&gt; problem I'd been wrestling with for years.&lt;/p&gt;

&lt;p&gt;In under two hours I went from...&lt;/p&gt;

&lt;p&gt;"A salesperson asked if I had an architecture diagram. I said, 'Yes... but it's rubbish.'"&lt;/p&gt;

&lt;p&gt;...to publishing this.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F1q384arr1tj6mf1hm84f.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F1q384arr1tj6mf1hm84f.png" alt="Forge Platform AWS architecture diagram showing CloudFront edge delivery, VPC public and private subnet boundaries, ECS Fargate microservices, RDS PostgreSQL, ElastiCache Redis, Cognito identity, and observability services" width="800" height="1519"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;You can see the &lt;a href="https://docs.forgeplatform.software/docs/welcome#architecture-and-runtime-model" rel="noopener noreferrer"&gt;live diagram on the Forge docs site&lt;/a&gt;, with more detail in the &lt;a href="https://docs.forgeplatform.software/docs/infrastructure" rel="noopener noreferrer"&gt;AWS platform infrastructure guide&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;After literally &lt;strong&gt;years&lt;/strong&gt; of trying different approaches, this is the first workflow that has produced an architecture diagram I'd happily put in front of customers.&lt;/p&gt;

&lt;h2&gt;
  
  
  The problem with architecture diagrams
&lt;/h2&gt;

&lt;p&gt;Anyone building on &lt;a href="https://aws.amazon.com/" rel="noopener noreferrer"&gt;AWS&lt;/a&gt; has probably been here.&lt;/p&gt;

&lt;p&gt;The architecture changes.&lt;/p&gt;

&lt;p&gt;The diagram doesn't.&lt;/p&gt;

&lt;p&gt;Eventually someone opens draw.io and spends half a day dragging icons around.&lt;/p&gt;

&lt;p&gt;Six months later it's wrong again.&lt;/p&gt;

&lt;p&gt;I've spent years trying to solve this problem.&lt;/p&gt;

&lt;h3&gt;
  
  
  Manual diagrams
&lt;/h3&gt;

&lt;p&gt;I've built diagrams using tools including:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;&lt;a href="https://www.drawio.com/docs/diagram-types/aws-diagrams/" rel="noopener noreferrer"&gt;draw.io AWS diagram shapes&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://www.lucidchart.com/" rel="noopener noreferrer"&gt;Lucidchart&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://www.microsoft.com/en-us/microsoft-365/visio/flowchart-software" rel="noopener noreferrer"&gt;Microsoft Visio&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://www.cloudcraft.co/" rel="noopener noreferrer"&gt;Cloudcraft&lt;/a&gt; (live AWS/Azure scanning and visual designer)&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://www.eraser.io/ai/aws-diagram-generator" rel="noopener noreferrer"&gt;Eraser AI AWS Diagram Generator&lt;/a&gt; (prompt-to-diagram)&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The results can look fantastic - whether you drew them by hand, scanned live infrastructure, or generated them from a written description.&lt;/p&gt;

&lt;p&gt;Until the infrastructure changes.&lt;/p&gt;

&lt;p&gt;Then they immediately start drifting from reality.&lt;/p&gt;

&lt;h3&gt;
  
  
  CDK diagram generators
&lt;/h3&gt;

&lt;p&gt;I also explored tools including:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;&lt;a href="https://github.com/pistazie/cdk-dia" rel="noopener noreferrer"&gt;cdk-dia&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://github.com/duartealexf/cdk2d2" rel="noopener noreferrer"&gt;cdk2d2&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://aws.github.io/aws-pdk/developer_guides/cdk-graph/index.html" rel="noopener noreferrer"&gt;AWS PDK CDK Graph&lt;/a&gt; (with its &lt;a href="https://aws.github.io/aws-pdk/developer_guides/cdk-graph-plugin-diagram/index.html" rel="noopener noreferrer"&gt;diagram plugin&lt;/a&gt;)&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;These are clever projects.&lt;/p&gt;

&lt;p&gt;They faithfully represent your infrastructure.&lt;/p&gt;

&lt;p&gt;Unfortunately, for &lt;em&gt;customer-facing architecture communication&lt;/em&gt;, that's exactly the problem.&lt;/p&gt;

&lt;p&gt;A single ECS service explodes into something like:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Task Definition&lt;/li&gt;
&lt;li&gt;Execution Role&lt;/li&gt;
&lt;li&gt;Task Role&lt;/li&gt;
&lt;li&gt;Security Groups&lt;/li&gt;
&lt;li&gt;Target Groups&lt;/li&gt;
&lt;li&gt;Listener Rules&lt;/li&gt;
&lt;li&gt;Auto Scaling&lt;/li&gt;
&lt;li&gt;CloudWatch Alarms&lt;/li&gt;
&lt;li&gt;Log Groups&lt;/li&gt;
&lt;li&gt;IAM Policies&lt;/li&gt;
&lt;li&gt;ENIs&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Everything is technically correct.&lt;/p&gt;

&lt;p&gt;Nothing is architecturally useful.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-template-resource-type-ref.html" rel="noopener noreferrer"&gt;CloudFormation&lt;/a&gt; graphs answer:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;"What AWS resources exist?"&lt;/strong&gt;&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;Architects are usually trying to answer:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;"How does this system work?"&lt;/strong&gt;&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;Those are completely different questions.&lt;/p&gt;

&lt;p&gt;The CDK diagram generators were right.&lt;/p&gt;

&lt;p&gt;They just weren't publishable.&lt;/p&gt;

&lt;h3&gt;
  
  
  AWS Workload Discovery
&lt;/h3&gt;

&lt;p&gt;I even evaluated AWS' own solution:&lt;/p&gt;

&lt;p&gt;&lt;a href="https://docs.aws.amazon.com/solutions/latest/workload-discovery-on-aws/solution-overview.html" rel="noopener noreferrer"&gt;AWS Workload Discovery on AWS&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;It discovers infrastructure.&lt;/p&gt;

&lt;p&gt;It inventories resources.&lt;/p&gt;

&lt;p&gt;It builds dependency graphs.&lt;/p&gt;

&lt;p&gt;For me?&lt;/p&gt;

&lt;p&gt;It simply wasn't worth the cost - &lt;strong&gt;over $400/month&lt;/strong&gt; just to deploy the CloudFormation stack and keep the discovery pipeline running, so you can draw a diagram of what you already deployed with CloudFormation. That is a genuinely terrible proposal for a documentation problem.&lt;/p&gt;

&lt;p&gt;Again, it documents what AWS has deployed.&lt;/p&gt;

&lt;p&gt;It doesn't communicate your architecture.&lt;/p&gt;

&lt;h2&gt;
  
  
  The breakthrough
&lt;/h2&gt;

&lt;p&gt;The breakthrough came from this AWS blog:&lt;/p&gt;

&lt;p&gt;&lt;a href="https://aws.amazon.com/blogs/machine-learning/build-aws-architecture-diagrams-using-amazon-q-cli-and-mcp/" rel="noopener noreferrer"&gt;Build AWS architecture diagrams using Kiro CLI and MCP&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;At first I thought:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;"Interesting... but they're just feeding text into an LLM."&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;Then I realised something.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;That isn't a weakness. It's the entire point.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;An experienced solutions architect doesn't think in CloudFormation resources.&lt;/p&gt;

&lt;p&gt;They mentally ignore:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;IAM Roles&lt;/li&gt;
&lt;li&gt;Security Groups&lt;/li&gt;
&lt;li&gt;Parameters&lt;/li&gt;
&lt;li&gt;Log Groups&lt;/li&gt;
&lt;li&gt;Listener Rules&lt;/li&gt;
&lt;li&gt;Auto Scaling Targets&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;and instead think:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Application Load Balancer&lt;/li&gt;
&lt;li&gt;ECS Services&lt;/li&gt;
&lt;li&gt;PostgreSQL&lt;/li&gt;
&lt;li&gt;Redis&lt;/li&gt;
&lt;li&gt;S3&lt;/li&gt;
&lt;li&gt;Cognito&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;That's architecture.&lt;/p&gt;

&lt;p&gt;Not implementation.&lt;/p&gt;

&lt;p&gt;The LLM wasn't trying to replace an architect.&lt;/p&gt;

&lt;p&gt;It was acting as an &lt;strong&gt;architectural abstraction layer&lt;/strong&gt; - the same idea as a &lt;a href="https://c4model.com/" rel="noopener noreferrer"&gt;C4 container diagram&lt;/a&gt;: document intent at the level stakeholders actually care about.&lt;/p&gt;

&lt;p&gt;That was the lightbulb moment.&lt;/p&gt;

&lt;h2&gt;
  
  
  Cursor did almost all of the work
&lt;/h2&gt;

&lt;p&gt;Armed with that idea, I pointed &lt;a href="https://cursor.com/" rel="noopener noreferrer"&gt;Cursor&lt;/a&gt; at my &lt;a href="https://docs.aws.amazon.com/cdk/v2/guide/home.html" rel="noopener noreferrer"&gt;AWS CDK&lt;/a&gt; repository.&lt;/p&gt;

&lt;p&gt;I also gave it some context from a conversation I'd had with ChatGPT about why CloudFormation graphs are the wrong level of abstraction - useful for framing, but Cursor did the actual work.&lt;/p&gt;

&lt;p&gt;Then I basically let it loose.&lt;/p&gt;

&lt;p&gt;It analysed the repository.&lt;/p&gt;

&lt;p&gt;It walked the stacks.&lt;/p&gt;

&lt;p&gt;It understood the constructs.&lt;/p&gt;

&lt;p&gt;It traced the networking.&lt;/p&gt;

&lt;p&gt;It identified which AWS resources represented meaningful architectural components and which were simply implementation plumbing.&lt;/p&gt;

&lt;p&gt;Then it produced:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;a curated architecture model&lt;/li&gt;
&lt;li&gt;rendering instructions&lt;/li&gt;
&lt;li&gt;a repeatable workflow&lt;/li&gt;
&lt;li&gt;automation scripts&lt;/li&gt;
&lt;li&gt;documentation updates&lt;/li&gt;
&lt;li&gt;and finally, the diagram itself.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;I didn't manually reverse-engineer thousands of CloudFormation resources into a diagram. I reviewed and refined what the tooling produced.&lt;/p&gt;

&lt;p&gt;That's a huge distinction.&lt;/p&gt;

&lt;p&gt;The AI wasn't making architectural decisions.&lt;/p&gt;

&lt;p&gt;It was reverse engineering architectural decisions I'd already made in code.&lt;/p&gt;

&lt;p&gt;My role was limited - mostly review, a few rounds of feedback, and sanity-checking that the subnet story matched what we'd actually built.&lt;/p&gt;

&lt;h2&gt;
  
  
  That's a far harder problem than it sounds
&lt;/h2&gt;

&lt;p&gt;The YAML wasn't the clever part.&lt;/p&gt;

&lt;p&gt;The clever part was that Cursor understood:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;an ECS Task Role isn't an architectural component&lt;/li&gt;
&lt;li&gt;an Application Load Balancer definitely is&lt;/li&gt;
&lt;li&gt;six Fargate services belong together&lt;/li&gt;
&lt;li&gt;subnet boundaries matter&lt;/li&gt;
&lt;li&gt;CloudFront belongs at the edge&lt;/li&gt;
&lt;li&gt;PostgreSQL, Redis and Cognito deserve first-class representation&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;None of that existed in a single file.&lt;/p&gt;

&lt;p&gt;It emerged from analysing thousands of lines of CDK spread across stacks, constructs and configuration - including constructs like &lt;code&gt;ForgeStaticEdgeDeliveryConstruct&lt;/code&gt;, which encodes the CloudFront → S3 static UI + API path proxying story as a single architectural unit rather than a pile of distribution behaviours and origin policies.&lt;/p&gt;

&lt;p&gt;That's genuinely impressive.&lt;/p&gt;

&lt;h2&gt;
  
  
  Was it perfect first time?
&lt;/h2&gt;

&lt;p&gt;No.&lt;/p&gt;

&lt;p&gt;I iterated a couple of times.&lt;/p&gt;

&lt;p&gt;I told Cursor to:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;collapse repeated Cognito connections and repeated metrics connections to AMP&lt;/li&gt;
&lt;li&gt;show public/private subnet boundaries in the VPC&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The second render was immediately publishable.&lt;/p&gt;

&lt;p&gt;There was also a small tooling wrinkle: the &lt;a href="https://github.com/awslabs/mcp/tree/main/src/aws-diagram-mcp-server" rel="noopener noreferrer"&gt;AWS Diagram MCP server&lt;/a&gt; package has been yanked upstream, so we pinned the last published version (&lt;code&gt;1.0.23&lt;/code&gt;) in a source-controlled &lt;code&gt;mcp.json&lt;/code&gt;. That kind of friction is worth mentioning - this wasn't magic; we wrestled with it slightly to get it working.&lt;/p&gt;

&lt;h2&gt;
  
  
  The hybrid workflow
&lt;/h2&gt;

&lt;p&gt;The workflow ended up looking like this.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;CDK repository
        │
        ▼
Cursor analyses architectural intent
        │
        ▼
Curated intermediate model (YAML)
        │
        ▼
Kiro CLI
        │
        ▼
AWS Diagram MCP ([Model Context Protocol](https://modelcontextprotocol.io/))
        │
        ▼
Published AWS architecture diagram
        ([Python diagrams](https://diagrams.mingrammer.com/) + [Graphviz](https://graphviz.org/) + AWS icons)
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The architecture model is version controlled.&lt;/p&gt;

&lt;p&gt;The prompt is version controlled.&lt;/p&gt;

&lt;p&gt;The rendering script is version controlled.&lt;/p&gt;

&lt;p&gt;The generated diagram can be regenerated whenever the platform evolves.&lt;/p&gt;

&lt;p&gt;For the first time ever, my architecture documentation actually feels maintainable.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why this works
&lt;/h2&gt;

&lt;p&gt;The key insight is surprisingly simple.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Architecture diagrams should document intent, not implementation.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;My CDK constructs already encode that intent.&lt;/p&gt;

&lt;p&gt;They have names like:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;&lt;code&gt;ForgeStaticEdgeDeliveryConstruct&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;&lt;code&gt;ForgeEcsWorkloadConstruct&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;&lt;code&gt;ForgeCognitoIdpStack&lt;/code&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Those are architectural concepts.&lt;/p&gt;

&lt;p&gt;CloudFormation is an implementation detail.&lt;/p&gt;

&lt;p&gt;Trying to diagram CloudFormation is a bit like trying to explain a car by drawing every nut and bolt.&lt;/p&gt;

&lt;p&gt;Sometimes abstraction is exactly what you want.&lt;/p&gt;

&lt;h2&gt;
  
  
  AI finally solved the right problem
&lt;/h2&gt;

&lt;p&gt;This experience genuinely changed my opinion of where AI delivers value.&lt;/p&gt;

&lt;p&gt;Not by replacing engineers.&lt;/p&gt;

&lt;p&gt;Not by generating production systems from prompts.&lt;/p&gt;

&lt;p&gt;Not by writing thousands of lines of code you'll spend next week debugging.&lt;/p&gt;

&lt;p&gt;Instead, it behaved like a very competent technical collaborator.&lt;/p&gt;

&lt;p&gt;It absorbed an entire repository.&lt;/p&gt;

&lt;p&gt;It reconstructed the architecture.&lt;/p&gt;

&lt;p&gt;It understood what mattered.&lt;/p&gt;

&lt;p&gt;It ignored what didn't.&lt;/p&gt;

&lt;p&gt;It accepted feedback.&lt;/p&gt;

&lt;p&gt;It refined the output.&lt;/p&gt;

&lt;p&gt;The AI wasn't designing my platform.&lt;/p&gt;

&lt;p&gt;But it did an excellent job of documenting a platform that already exists - the same platform I write about when debugging &lt;a href="https://forgeplatform.software/blog/when-amp-returns-400-check-discarded-samples/" rel="noopener noreferrer"&gt;AMP remote write failures&lt;/a&gt; or explaining &lt;a href="https://forgeplatform.software/blog/what-production-ready-actually-means/" rel="noopener noreferrer"&gt;what production-ready actually means&lt;/a&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  Would I trust it blindly?
&lt;/h2&gt;

&lt;p&gt;Absolutely not.&lt;/p&gt;

&lt;p&gt;The architecture still belongs to the engineers who built the system.&lt;/p&gt;

&lt;p&gt;The AI isn't deciding what the platform looks like.&lt;/p&gt;

&lt;p&gt;It's modelling something that already exists.&lt;/p&gt;

&lt;p&gt;The human remains accountable.&lt;/p&gt;

&lt;p&gt;The AI removes an enormous amount of tedious work.&lt;/p&gt;

&lt;p&gt;That's augmentation.&lt;/p&gt;

&lt;p&gt;Not replacement.&lt;/p&gt;

&lt;h2&gt;
  
  
  What's next?
&lt;/h2&gt;

&lt;p&gt;This has me thinking the future isn't another CDK diagram generator.&lt;/p&gt;

&lt;p&gt;It's architecture-aware tooling.&lt;/p&gt;

&lt;p&gt;Imagine a workflow that can:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;analyse your CDK constructs&lt;/li&gt;
&lt;li&gt;infer architectural components&lt;/li&gt;
&lt;li&gt;maintain a version-controlled intermediate model&lt;/li&gt;
&lt;li&gt;regenerate diagrams automatically&lt;/li&gt;
&lt;li&gt;produce different views for developers, operations, security reviews and customer documentation&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;That feels considerably more exciting than another CloudFormation graph.&lt;/p&gt;

&lt;p&gt;And I essentially have that version-controlled now - tied to a real &lt;a href="https://docs.aws.amazon.com/AmazonECS/latest/bestpracticesguide/intro.html" rel="noopener noreferrer"&gt;ECS on Fargate&lt;/a&gt; platform, not a toy example.&lt;/p&gt;

&lt;h2&gt;
  
  
  Here's how Cursor did it
&lt;/h2&gt;

&lt;p&gt;The interesting part isn't the PNG. It's the intermediate model - a curated, solutions-architect view derived from CDK constructs, not from the synthesized CloudFormation dependency graph.&lt;/p&gt;

&lt;p&gt;The model header explains the intent:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight yaml"&gt;&lt;code&gt;&lt;span class="nn"&gt;---&lt;/span&gt;
&lt;span class="c1"&gt;# Forge Platform - curated architecture model&lt;/span&gt;
&lt;span class="c1"&gt;#&lt;/span&gt;
&lt;span class="c1"&gt;# Derived from CDK constructs under infra/src/lib (NOT from the synthesized&lt;/span&gt;
&lt;span class="c1"&gt;# CloudFormation dependency graph). Low-level noise (IAM roles/policies,&lt;/span&gt;
&lt;span class="c1"&gt;# security groups, log groups, alarms, KMS keys, SSM params, ENIs, target&lt;/span&gt;
&lt;span class="c1"&gt;# groups, listener rules) is deliberately omitted.&lt;/span&gt;
&lt;span class="c1"&gt;#&lt;/span&gt;
&lt;span class="c1"&gt;# Source of truth mapping:&lt;/span&gt;
&lt;span class="c1"&gt;#   - Edge / CloudFront .......... forge-static-edge-delivery-construct.ts&lt;/span&gt;
&lt;span class="c1"&gt;#   - Ingress (ALBs) ............. forge-ecs-ingress-construct.ts&lt;/span&gt;
&lt;span class="c1"&gt;#   - Compute (Fargate) .......... forge-ecs-workload-construct.ts&lt;/span&gt;
&lt;span class="c1"&gt;#   ...&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Nested groups encode the VPC security boundary:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight yaml"&gt;&lt;code&gt;&lt;span class="na"&gt;groups&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
  &lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="na"&gt;id&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;vpc&lt;/span&gt;
    &lt;span class="na"&gt;label&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;VPC (ForgePlatformVpc)&lt;/span&gt;
  &lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="na"&gt;id&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;public_subnets&lt;/span&gt;
    &lt;span class="na"&gt;label&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;Public Subnets&lt;/span&gt;
    &lt;span class="na"&gt;parent&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;vpc&lt;/span&gt;
  &lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="na"&gt;id&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;private_subnets&lt;/span&gt;
    &lt;span class="na"&gt;label&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;Private Subnets&lt;/span&gt;
    &lt;span class="na"&gt;parent&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;vpc&lt;/span&gt;
  &lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="na"&gt;id&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;compute&lt;/span&gt;
    &lt;span class="na"&gt;label&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;ECS Fargate Cluster&lt;/span&gt;
    &lt;span class="na"&gt;parent&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;private_subnets&lt;/span&gt;
  &lt;span class="c1"&gt;# ...&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Aggregate connections collapse repeated edges - six identical "service auth" lines become one:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight yaml"&gt;&lt;code&gt;  &lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="na"&gt;from&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;compute&lt;/span&gt;
    &lt;span class="na"&gt;to&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;cognito_service&lt;/span&gt;
    &lt;span class="na"&gt;label&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;service auth&lt;/span&gt;
    &lt;span class="na"&gt;aggregate&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;true&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The MCP server registration is trivial - no secrets, just a pinned package launch:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight json"&gt;&lt;code&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"mcpServers"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="nl"&gt;"awslabsawsdiagrammcpserver"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt;
      &lt;/span&gt;&lt;span class="nl"&gt;"command"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"uvx"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
      &lt;/span&gt;&lt;span class="nl"&gt;"args"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="w"&gt;
        &lt;/span&gt;&lt;span class="s2"&gt;"--from"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
        &lt;/span&gt;&lt;span class="s2"&gt;"awslabs.aws-diagram-mcp-server==1.0.23"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
        &lt;/span&gt;&lt;span class="s2"&gt;"awslabs.aws-diagram-mcp-server"&lt;/span&gt;&lt;span class="w"&gt;
      &lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt;&lt;span class="w"&gt;
      &lt;/span&gt;&lt;span class="nl"&gt;"env"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{}&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;And regeneration is a one-liner:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;./infra/architecture/render-diagram.sh
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Which syncs the MCP config, then drives Kiro non-interactively:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;kiro-cli mcp import &lt;span class="nt"&gt;--file&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="k"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;mcp_config&lt;/span&gt;&lt;span class="k"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt; &lt;span class="nt"&gt;--force&lt;/span&gt; global
kiro-cli chat &lt;span class="nt"&gt;--no-interactive&lt;/span&gt; &lt;span class="nt"&gt;--trust-all-tools&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="si"&gt;$(&lt;/span&gt;&lt;span class="nb"&gt;cat &lt;/span&gt;diagram-prompt.md&lt;span class="si"&gt;)&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The prompt tells the MCP server how to map component types to &lt;a href="https://diagrams.mingrammer.com/" rel="noopener noreferrer"&gt;diagrams&lt;/a&gt; nodes, nest subnet clusters, and honour aggregate edges - treating the YAML as the single source of truth.&lt;/p&gt;

&lt;p&gt;That's the whole trick: &lt;strong&gt;deterministic extraction from code you already wrote, LLM-assisted rendering for polish.&lt;/strong&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  Final thoughts
&lt;/h2&gt;

&lt;p&gt;This is probably the first AI workflow I've adopted where I immediately thought:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;"I'm never going back."&lt;/strong&gt;&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;Not because the AI was smarter than me.&lt;/p&gt;

&lt;p&gt;Because it could absorb an entire repository, retain all of that context, and perform a task that would have taken me several evenings of tedious work.&lt;/p&gt;

&lt;p&gt;I've spent years looking for a workflow that made architecture diagrams feel like source code instead of PowerPoint.&lt;/p&gt;

&lt;p&gt;This approach is by far the best I've seen.&lt;/p&gt;

&lt;p&gt;Not because AI magically solved architecture.&lt;/p&gt;

&lt;p&gt;Because it finally solved the tedious part.&lt;/p&gt;

</description>
      <category>aws</category>
      <category>architecture</category>
      <category>ai</category>
      <category>devops</category>
    </item>
    <item>
      <title>AMP Remote Write HTTP 400: The CloudWatch Metric That Solved It</title>
      <dc:creator>Andrew Eells</dc:creator>
      <pubDate>Sat, 04 Jul 2026 00:32:05 +0000</pubDate>
      <link>https://dev.to/aeells/amp-remote-write-http-400-the-cloudwatch-metric-that-solved-it-64i</link>
      <guid>https://dev.to/aeells/amp-remote-write-http-400-the-cloudwatch-metric-that-solved-it-64i</guid>
      <description>&lt;p&gt;AMP said 400. CloudWatch said why.&lt;/p&gt;

&lt;p&gt;We turned on Amazon Managed Prometheus (AMP) remote write from our Quarkus services on ECS — Micrometer snapshots, Snappy-compressed Prometheus remote write, SigV4 signing — and everything &lt;em&gt;mostly&lt;/em&gt; worked.&lt;/p&gt;

&lt;p&gt;Mostly.&lt;/p&gt;

&lt;p&gt;Every service logged &lt;code&gt;AMP remote write failed with status 400&lt;/code&gt; on a ~60 second cadence. Not constantly. Not only at startup. Intermittently, across the fleet, continually.&lt;/p&gt;

&lt;p&gt;HTTP 400 with an empty body is a frustrating place to start.&lt;/p&gt;

&lt;p&gt;This post is how we narrowed it down, why the obvious suspects weren't the culprit, and the CloudWatch metric that finally made the failure mode obvious.&lt;/p&gt;

&lt;h2&gt;
  
  
  The setup
&lt;/h2&gt;

&lt;p&gt;Forge pushes application metrics in-process:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Micrometer&lt;/strong&gt; with the Prometheus v1 registry (&lt;code&gt;/q/metrics&lt;/code&gt; for local scrape, &lt;code&gt;MetricSnapshots&lt;/code&gt; for export)&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Remote write&lt;/strong&gt; to AMP over HTTPS with &lt;code&gt;Content-Encoding: snappy&lt;/code&gt; and AWS SigV4 (&lt;code&gt;aps&lt;/code&gt; signing name)&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;One AMP workspace&lt;/strong&gt; shared by every ECS service in the environment (auth, actor, document, notification, audit, BFF, …)&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;GraalVM native&lt;/strong&gt; images on Fargate, private subnets, no collector sidecar&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The architecture is deliberate: fewer moving parts, right-sized tasks, metrics, and traces exported directly from the app.&lt;/p&gt;

&lt;p&gt;When remote write misbehaves, the failure is somewhere in a short chain: scrape → encode → compress → sign → POST.&lt;/p&gt;

&lt;h2&gt;
  
  
  What we ruled out first
&lt;/h2&gt;

&lt;h3&gt;
  
  
  Wrong URL
&lt;/h3&gt;

&lt;p&gt;A previous deploy had pointed at the wrong path suffix. AMP's real endpoint is:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;https://aps-workspaces.&amp;lt;region&amp;gt;.amazonaws.com/workspaces/&amp;lt;ws-id&amp;gt;/api/v1/remote_write
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Not &lt;code&gt;.../remote_write&lt;/code&gt; alone (that returned 404).&lt;/p&gt;

&lt;p&gt;We verified the CloudFormation export, the ECS task environment variable, and a &lt;strong&gt;SigV4-signed POST with an empty Snappy payload&lt;/strong&gt; to the correct URL. That returned &lt;strong&gt;200&lt;/strong&gt;. So URL and IAM were fine.&lt;/p&gt;

&lt;h3&gt;
  
  
  SigV4 and transport
&lt;/h3&gt;

&lt;p&gt;The same signing code path used in production (&lt;code&gt;AwsSignedHttpTransport&lt;/code&gt;) worked from a JVM probe on a developer machine against the live workspace. Empty payload and full payload both succeeded when credentials and URL were correct.&lt;/p&gt;

&lt;h3&gt;
  
  
  "A bad metric family"
&lt;/h3&gt;

&lt;p&gt;We scraped live exposition text from a running native task (more on that pattern below), loaded it into a JVM integration test, encoded it with the same remote-write encoder, and POSTed to AMP.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;200. Full payload accepted.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;So the metric &lt;em&gt;content&lt;/em&gt; at scrape time was not inherently toxic. The encoder and Snappy path on JVM were fine.&lt;/p&gt;

&lt;p&gt;That left something about the &lt;strong&gt;native periodic push path&lt;/strong&gt; or &lt;strong&gt;fleet-wide ingestion behaviour&lt;/strong&gt; — not a single broken histogram hiding in auth-service.&lt;/p&gt;

&lt;h2&gt;
  
  
  The pivot: CloudWatch &lt;code&gt;DiscardedSamples&lt;/code&gt;
&lt;/h2&gt;

&lt;p&gt;AMP exposes &lt;a href="https://docs.aws.amazon.com/prometheus/latest/userguide/AMP-CW-usage-metrics.html" rel="noopener noreferrer"&gt;vended CloudWatch metrics&lt;/a&gt; under the &lt;code&gt;AWS/Prometheus&lt;/code&gt; namespace. One of them is &lt;strong&gt;&lt;code&gt;DiscardedSamples&lt;/code&gt;&lt;/strong&gt;, with a &lt;strong&gt;&lt;code&gt;Reason&lt;/code&gt; dimension&lt;/strong&gt; explaining why samples were dropped.&lt;/p&gt;

&lt;p&gt;We listed metrics for our workspace:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;aws cloudwatch list-metrics &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;--namespace&lt;/span&gt; &lt;span class="s2"&gt;"AWS/Prometheus"&lt;/span&gt; &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;--metric-name&lt;/span&gt; &lt;span class="s2"&gt;"DiscardedSamples"&lt;/span&gt; &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;--region&lt;/span&gt; us-west-2
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The only reason showing material volume:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Reason = new-value-for-timestamp
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Querying that dimension:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;aws cloudwatch get-metric-statistics &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;--namespace&lt;/span&gt; &lt;span class="s2"&gt;"AWS/Prometheus"&lt;/span&gt; &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;--metric-name&lt;/span&gt; &lt;span class="s2"&gt;"DiscardedSamples"&lt;/span&gt; &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;--dimensions&lt;/span&gt; &lt;span class="se"&gt;\&lt;/span&gt;
    &lt;span class="nv"&gt;Name&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;Workspace,Value&lt;span class="o"&gt;=&lt;/span&gt;ws-xxxxxxxx &lt;span class="se"&gt;\&lt;/span&gt;
    &lt;span class="nv"&gt;Name&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;Reason,Value&lt;span class="o"&gt;=&lt;/span&gt;new-value-for-timestamp &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;--start-time&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$START&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt; &lt;span class="nt"&gt;--end-time&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$END&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt; &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;--period&lt;/span&gt; 300 &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;--statistics&lt;/span&gt; Sum &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;--region&lt;/span&gt; us-west-2
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;We saw &lt;strong&gt;hundreds of discards per five-minute bucket&lt;/strong&gt;, sustained, fleet-wide.&lt;/p&gt;

&lt;p&gt;That is not a signing bug. It is not Snappy corruption.&lt;/p&gt;

&lt;p&gt;In Prometheus remote write semantics, &lt;strong&gt;&lt;code&gt;new-value-for-timestamp&lt;/code&gt;&lt;/strong&gt; means: AMP already has a sample for this &lt;strong&gt;exact time series&lt;/strong&gt; (same metric name and label set) at this &lt;strong&gt;exact timestamp&lt;/strong&gt;, but the new sample has a &lt;strong&gt;different value&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;In Prometheus, a series is defined by the metric name and its complete label set. If two writers produce identical labels, the backend treats them as one time series regardless of which process produced them.&lt;/p&gt;

&lt;p&gt;Same series. Same millisecond. Different number. Rejected.&lt;/p&gt;

&lt;p&gt;Once we saw that, the intermittent 400s and the ~95% push success rate (success counter climbing, failure counter occasionally ticking) both made sense: most pushes partially or fully conflicted with other writers or stale timestamps; some requests failed hard at the HTTP layer.&lt;/p&gt;

&lt;h2&gt;
  
  
  The actual bug: six services, one series
&lt;/h2&gt;

&lt;p&gt;Each ECS service exports the same Micrometer binders: &lt;code&gt;jvm_*&lt;/code&gt;, &lt;code&gt;process_*&lt;/code&gt;, &lt;code&gt;system_*&lt;/code&gt;, HTTP server metrics, and our own &lt;code&gt;forge_observability_amp_push_*&lt;/code&gt; counters.&lt;/p&gt;

&lt;p&gt;We had &lt;strong&gt;no common tags&lt;/strong&gt; — no &lt;code&gt;service&lt;/code&gt;, no &lt;code&gt;instance&lt;/code&gt;, no &lt;code&gt;pod&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;So from AMP's perspective:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;auth-service   → jvm_threads_live_threads = 21  @ T
actor-service  → jvm_threads_live_threads = 14  @ T
document-service → jvm_threads_live_threads = 19 @ T
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;That is &lt;strong&gt;one series&lt;/strong&gt; (&lt;code&gt;jvm_threads_live_threads&lt;/code&gt; with identical labels) receiving &lt;strong&gt;different values at the same timestamp&lt;/strong&gt; from different tasks, every push interval, forever.&lt;/p&gt;

&lt;p&gt;Classic multi-writer collision.&lt;/p&gt;

&lt;p&gt;It is the same class of problem as running multiple Prometheus replicas remote-writing the same targets without external labels — except we did it with six application services and one workspace.&lt;/p&gt;

&lt;h3&gt;
  
  
  Why the JVM probe didn't catch it
&lt;/h3&gt;

&lt;p&gt;The probe used a &lt;strong&gt;single&lt;/strong&gt; scraped exposition file from &lt;strong&gt;one&lt;/strong&gt; task. One writer. AMP happily ingested it.&lt;/p&gt;

&lt;p&gt;The failure mode only appears when &lt;strong&gt;multiple tasks&lt;/strong&gt; push &lt;strong&gt;overlapping label sets&lt;/strong&gt; to &lt;strong&gt;one workspace&lt;/strong&gt; on the same schedule.&lt;/p&gt;

&lt;h3&gt;
  
  
  Why &lt;code&gt;hasScrapeTimestamp()&lt;/code&gt; wasn't the story
&lt;/h3&gt;

&lt;p&gt;We also suspected stale per-datapoint scrape timestamps from the registry. On JVM Micrometer scrape, &lt;strong&gt;&lt;code&gt;hasScrapeTimestamp()&lt;/code&gt; was false for every data point&lt;/strong&gt; in our tests. The old encoder fell back to &lt;code&gt;System.currentTimeMillis()&lt;/code&gt; per sample anyway.&lt;/p&gt;

&lt;p&gt;Fleet collision fit the CloudWatch evidence better than stale scrape timestamps.&lt;/p&gt;

&lt;p&gt;We still normalized to &lt;strong&gt;one batch push timestamp per remote-write request&lt;/strong&gt; — correct semantics for periodic push export — but the fix that cleared discards was &lt;strong&gt;disambiguating series&lt;/strong&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  The fix
&lt;/h2&gt;

&lt;h3&gt;
  
  
  1. Common tags (the real fix)
&lt;/h3&gt;

&lt;p&gt;In forge-kit &lt;code&gt;forge-metrics&lt;/code&gt;, a CDI &lt;code&gt;MeterFilter&lt;/code&gt;:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight java"&gt;&lt;code&gt;&lt;span class="nc"&gt;MeterFilter&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;commonTags&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;
    &lt;span class="nc"&gt;Tag&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;of&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="s"&gt;"service"&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="n"&gt;applicationName&lt;/span&gt;&lt;span class="o"&gt;),&lt;/span&gt;   &lt;span class="c1"&gt;// quarkus.application.name&lt;/span&gt;
    &lt;span class="nc"&gt;Tag&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;of&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="s"&gt;"instance"&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="n"&gt;hostName&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt;        &lt;span class="c1"&gt;// ECS task hostname; override via config&lt;/span&gt;
&lt;span class="o"&gt;);&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;After deploy, AMP queries looked like:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;forge_observability_amp_push_success_total
jvm_threads_live_threads{service="auth-service"}
count by (service) (jvm_memory_used_bytes)
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Six services, six distinct label sets. Immediately after deploying the common tags, new &lt;code&gt;DiscardedSamples{Reason="new-value-for-timestamp"}&lt;/code&gt; buckets dropped to zero across the workspace while historical buckets retained the earlier collisions. No other code changes were required.&lt;/p&gt;

&lt;h3&gt;
  
  
  2. Richer failure logging
&lt;/h3&gt;

&lt;p&gt;&lt;code&gt;AmpMetricsExporter&lt;/code&gt; had only logged &lt;code&gt;status 400&lt;/code&gt;. We added &lt;strong&gt;Snappy body size&lt;/strong&gt;, &lt;strong&gt;snapshot count&lt;/strong&gt;, and &lt;strong&gt;AMP response body&lt;/strong&gt; on failure so the next incident doesn't require guesswork.&lt;/p&gt;

&lt;h3&gt;
  
  
  3. Batch push timestamp
&lt;/h3&gt;

&lt;p&gt;Remote-write encoding now stamps every sample in a &lt;code&gt;WriteRequest&lt;/code&gt; with &lt;strong&gt;one wall-clock time&lt;/strong&gt; at export. That is standard push-agent behaviour and avoids mixed timestamps within a single push. It did not replace the need for &lt;code&gt;service&lt;/code&gt; / &lt;code&gt;instance&lt;/code&gt; tags.&lt;/p&gt;

&lt;h2&gt;
  
  
  How we verified ingestion (not just "no errors")
&lt;/h2&gt;

&lt;p&gt;Empty HTTP failures are necessary but not sufficient. We confirmed data was &lt;strong&gt;stored and queryable&lt;/strong&gt;:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nv"&gt;QUERY_ENDPOINT&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;"https://aps-workspaces.us-west-2.amazonaws.com/workspaces/&lt;/span&gt;&lt;span class="k"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;WORKSPACE_ID&lt;/span&gt;&lt;span class="k"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;/api/v1/query"&lt;/span&gt;

awscurl &lt;span class="nt"&gt;--service&lt;/span&gt; aps &lt;span class="nt"&gt;--region&lt;/span&gt; us-west-2 &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;-X&lt;/span&gt; POST &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="k"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;QUERY_ENDPOINT&lt;/span&gt;&lt;span class="k"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt; &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;-H&lt;/span&gt; &lt;span class="s1"&gt;'Content-Type: application/x-www-form-urlencoded'&lt;/span&gt; &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;-d&lt;/span&gt; &lt;span class="s1"&gt;'query=forge_observability_amp_push_success_total'&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Success counters for all six workloads, each with &lt;code&gt;service&lt;/code&gt; and &lt;code&gt;instance&lt;/code&gt; labels. Failure metric absent (never incremented since deploy). CloudWatch discards at zero post-rollout.&lt;/p&gt;

&lt;p&gt;That is the bar: &lt;strong&gt;PromQL returns your series with the labels you expect&lt;/strong&gt;, and &lt;strong&gt;DiscardedSamples stays quiet&lt;/strong&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  Appendix: debugging when you can't reach &lt;code&gt;/q/metrics&lt;/code&gt;
&lt;/h2&gt;

&lt;p&gt;Our laptops can't hit task private IPs in a NAT-less VPC. To compare native exposition with the JVM encoder, we used a one-off Fargate task pattern:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Publish a tiny &lt;strong&gt;private ECR&lt;/strong&gt; image (&lt;code&gt;busybox&lt;/code&gt; + &lt;code&gt;wget&lt;/code&gt;) — Docker Hub and public ECR aren't reachable from the VPC without NAT.&lt;/li&gt;
&lt;li&gt;Run a task in the cluster that &lt;code&gt;wget&lt;/code&gt;s &lt;code&gt;http://&amp;lt;task-private-ip&amp;gt;:8080/q/metrics&lt;/code&gt; (same path the target group uses for health checks).&lt;/li&gt;
&lt;li&gt;Pull logs into a local file and feed it to a JVM probe (&lt;code&gt;FORGE_AMP_METRICS_FILE=...&lt;/code&gt;).&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;We also briefly added a &lt;strong&gt;same-SG ingress on 8080&lt;/strong&gt; so the scrape task could reach the service ENI — then &lt;strong&gt;revoked&lt;/strong&gt; it after debugging. That rule is not part of normal infra; ALB → task rules stay as designed.&lt;/p&gt;

&lt;p&gt;This tooling was valuable for one INT session. We don't plan to keep the scrape image or scripts in the application repo long term — the durable lessons are the &lt;strong&gt;CloudWatch reason dimension&lt;/strong&gt;, the &lt;strong&gt;multi-writer tag discipline&lt;/strong&gt;, and the &lt;strong&gt;PromQL verification queries&lt;/strong&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  Takeaways
&lt;/h2&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;AMP HTTP 400 with an empty body&lt;/strong&gt; sends you down a long hallway. &lt;strong&gt;&lt;code&gt;DiscardedSamples&lt;/code&gt; by &lt;code&gt;Reason&lt;/code&gt;&lt;/strong&gt; is often the door.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;&lt;code&gt;new-value-for-timestamp&lt;/code&gt;&lt;/strong&gt; on a shared workspace screams &lt;strong&gt;label collision&lt;/strong&gt; — multiple writers, identical series, aligned push intervals.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Micrometer &lt;code&gt;MeterFilter.commonTags&lt;/code&gt;&lt;/strong&gt; with low-cardinality &lt;code&gt;service&lt;/code&gt; (and &lt;code&gt;instance&lt;/code&gt; when you scale replicas) is not optional when many processes remote-write to one Prometheus-compatible backend.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Single-service probes prove encoding&lt;/strong&gt;, not fleet behaviour. Reproduce collisions with multi-writer thinking.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Confirm consumption&lt;/strong&gt; with PromQL and CloudWatch ingestion metrics — not only "the error log stopped."&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;If you're wiring Micrometer → AMP on ECS, check your tags before you tune Snappy JNI or second-guess your SigV4 implementation.&lt;/p&gt;

&lt;p&gt;The platform was fine. We were just telling six services to sign the same name on the same line.&lt;/p&gt;

</description>
      <category>devops</category>
      <category>aws</category>
      <category>observability</category>
      <category>prometheus</category>
    </item>
    <item>
      <title>The software nobody plans to build - but every successful team eventually does...</title>
      <dc:creator>Andrew Eells</dc:creator>
      <pubDate>Fri, 26 Jun 2026 19:59:56 +0000</pubDate>
      <link>https://dev.to/aeells/the-software-nobody-plans-to-build-but-every-successful-team-eventually-does-1pbf</link>
      <guid>https://dev.to/aeells/the-software-nobody-plans-to-build-but-every-successful-team-eventually-does-1pbf</guid>
      <description>&lt;p&gt;Every software company starts with one product.&lt;/p&gt;

&lt;p&gt;The product customers buy.&lt;/p&gt;

&lt;p&gt;The product investors care about.&lt;/p&gt;

&lt;p&gt;The product the roadmap revolves around.&lt;/p&gt;

&lt;p&gt;But given enough time, almost every successful engineering team finds itself building something else.&lt;/p&gt;

&lt;p&gt;Not because they planned to.&lt;/p&gt;

&lt;p&gt;Because they have to.&lt;/p&gt;

&lt;h2&gt;
  
  
  The second product
&lt;/h2&gt;

&lt;p&gt;It doesn't have a marketing website.&lt;/p&gt;

&lt;p&gt;Customers never ask for it by name.&lt;/p&gt;

&lt;p&gt;Nobody demos it to investors.&lt;/p&gt;

&lt;p&gt;Yet it quietly grows alongside the business.&lt;/p&gt;

&lt;p&gt;It's your engineering foundations.&lt;/p&gt;

&lt;p&gt;The authentication services.&lt;/p&gt;

&lt;p&gt;The deployment pipelines.&lt;/p&gt;

&lt;p&gt;The infrastructure.&lt;/p&gt;

&lt;p&gt;The observability stack.&lt;/p&gt;

&lt;p&gt;The audit logging.&lt;/p&gt;

&lt;p&gt;The notification system.&lt;/p&gt;

&lt;p&gt;The release processes.&lt;/p&gt;

&lt;p&gt;The operational tooling.&lt;/p&gt;

&lt;p&gt;The architectural conventions.&lt;/p&gt;

&lt;p&gt;Individually, none of these things are your product.&lt;/p&gt;

&lt;p&gt;Collectively, they're what allow your product to scale.&lt;/p&gt;

&lt;h2&gt;
  
  
  It happens one decision at a time
&lt;/h2&gt;

&lt;p&gt;Nobody sets out to spend months building engineering foundations.&lt;/p&gt;

&lt;p&gt;Instead, they make perfectly reasonable decisions.&lt;/p&gt;

&lt;p&gt;"We'll automate deployments later."&lt;/p&gt;

&lt;p&gt;"This service just needs its own authentication for now."&lt;/p&gt;

&lt;p&gt;"We'll improve the monitoring once we've got more customers."&lt;/p&gt;

&lt;p&gt;"We'll standardise this after the next release."&lt;/p&gt;

&lt;p&gt;Every one of those decisions is commercially rational.&lt;/p&gt;

&lt;p&gt;Product delivery has to come first.&lt;/p&gt;

&lt;p&gt;But each compromise adds another piece to a second codebase that the business never intended to own.&lt;/p&gt;

&lt;h2&gt;
  
  
  The hidden backlog
&lt;/h2&gt;

&lt;p&gt;Earlier in my career, while CTO at a UK insurance startup, I estimated that around &lt;strong&gt;60% of our engineering backlog&lt;/strong&gt; wasn't product work at all.&lt;/p&gt;

&lt;p&gt;It was engineering foundations.&lt;/p&gt;

&lt;p&gt;Improving CI/CD.&lt;/p&gt;

&lt;p&gt;Standardising infrastructure.&lt;/p&gt;

&lt;p&gt;Strengthening security.&lt;/p&gt;

&lt;p&gt;Adding observability.&lt;/p&gt;

&lt;p&gt;Operational tooling.&lt;/p&gt;

&lt;p&gt;Developer experience.&lt;/p&gt;

&lt;p&gt;Performance and security testing.&lt;/p&gt;

&lt;p&gt;None of those stories generated revenue directly.&lt;/p&gt;

&lt;p&gt;But every one of them made future product delivery faster, safer and more predictable.&lt;/p&gt;

&lt;p&gt;Eventually we realised something important:&lt;/p&gt;

&lt;p&gt;We weren't just building an insurance platform anymore.&lt;/p&gt;

&lt;p&gt;We were also building the engineering foundations that made the insurance platform possible.&lt;/p&gt;

&lt;h2&gt;
  
  
  The Groundhog Day problem
&lt;/h2&gt;

&lt;p&gt;Over the last 20+ years I've worked across startups, consultancies and enterprise engineering teams.&lt;/p&gt;

&lt;p&gt;Different industries.&lt;/p&gt;

&lt;p&gt;Different products.&lt;/p&gt;

&lt;p&gt;Different company sizes.&lt;/p&gt;

&lt;p&gt;Yet I kept rebuilding remarkably similar engineering foundations.&lt;/p&gt;

&lt;p&gt;Authentication.&lt;/p&gt;

&lt;p&gt;Audit.&lt;/p&gt;

&lt;p&gt;Notifications.&lt;/p&gt;

&lt;p&gt;Infrastructure.&lt;/p&gt;

&lt;p&gt;Deployment pipelines.&lt;/p&gt;

&lt;p&gt;Observability.&lt;/p&gt;

&lt;p&gt;Release management.&lt;/p&gt;

&lt;p&gt;Security controls.&lt;/p&gt;

&lt;p&gt;Developer tooling.&lt;/p&gt;

&lt;p&gt;Different implementations.&lt;/p&gt;

&lt;p&gt;The same problems.&lt;/p&gt;

&lt;p&gt;After a while it started to feel like Groundhog Day.&lt;/p&gt;

&lt;p&gt;Every organisation was independently solving problems that thousands of engineering teams had already solved before.&lt;/p&gt;

&lt;h2&gt;
  
  
  Engineering foundations are inevitable
&lt;/h2&gt;

&lt;p&gt;This isn't an argument against building engineering foundations.&lt;/p&gt;

&lt;p&gt;They're essential.&lt;/p&gt;

&lt;p&gt;Every successful software company eventually needs them.&lt;/p&gt;

&lt;p&gt;The question is simply &lt;strong&gt;when&lt;/strong&gt; and &lt;strong&gt;how&lt;/strong&gt; you build them.&lt;/p&gt;

&lt;p&gt;Do you invest months (or years) constructing them incrementally while trying to deliver product?&lt;/p&gt;

&lt;p&gt;Or do you begin with mature engineering foundations already in place and let your team focus on the capabilities that actually differentiate your business?&lt;/p&gt;

&lt;p&gt;That's a very different starting position.&lt;/p&gt;

&lt;h2&gt;
  
  
  Final thoughts
&lt;/h2&gt;

&lt;p&gt;I've come to believe that one of the biggest hidden costs in software engineering isn't technical debt.&lt;/p&gt;

&lt;p&gt;It's repeatedly rebuilding the same engineering foundations.&lt;/p&gt;

&lt;p&gt;Not because they're unique.&lt;/p&gt;

&lt;p&gt;But because every organisation assumes it has to start from scratch.&lt;/p&gt;

&lt;p&gt;That realisation is ultimately what led me to build Forge Platform.&lt;/p&gt;

&lt;p&gt;After spending more than two decades repeatedly building the same operational capabilities across startups and enterprise programmes, I wanted to create something that lets engineering teams begin with mature foundations already in place - so more of their time is spent building the product they're actually in business to create.&lt;/p&gt;

&lt;p&gt;If that resonates, you can learn more at &lt;strong&gt;&lt;a href="https://forgeplatform.software" rel="noopener noreferrer"&gt;https://forgeplatform.software&lt;/a&gt;&lt;/strong&gt;.&lt;/p&gt;

</description>
      <category>saas</category>
      <category>aws</category>
      <category>softwareengineering</category>
      <category>discuss</category>
    </item>
    <item>
      <title>What "production-ready" actually means - and why most teams discover it too late.</title>
      <dc:creator>Andrew Eells</dc:creator>
      <pubDate>Sat, 06 Jun 2026 03:54:10 +0000</pubDate>
      <link>https://dev.to/aeells/what-production-ready-actually-means-and-why-most-teams-discover-it-too-late-20ha</link>
      <guid>https://dev.to/aeells/what-production-ready-actually-means-and-why-most-teams-discover-it-too-late-20ha</guid>
      <description>&lt;p&gt;"Production-ready" is one of the most misused phrases in software engineering.&lt;/p&gt;

&lt;p&gt;It usually means:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;it runs&lt;/li&gt;
&lt;li&gt;it deploys&lt;/li&gt;
&lt;li&gt;it works in a happy path&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;But in real systems, production readiness is not about functionality.&lt;/p&gt;

&lt;p&gt;It’s about behaviour under failure, change, and scale.&lt;/p&gt;

&lt;h2&gt;
  
  
  The difference between working and production-ready
&lt;/h2&gt;

&lt;p&gt;A system is not production-ready when:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;it can be deployed&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;It is production-ready when:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;it can fail safely&lt;/li&gt;
&lt;li&gt;it can be observed&lt;/li&gt;
&lt;li&gt;it can be redeployed without service interruption&lt;/li&gt;
&lt;li&gt;it behaves consistently under load&lt;/li&gt;
&lt;li&gt;it can be operated by people who did not build it&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Most early-stage systems do not meet this bar.&lt;/p&gt;

&lt;p&gt;Not because teams are careless - but because these properties are usually added after the system exists.&lt;/p&gt;

&lt;h2&gt;
  
  
  The problem with "we'll add it later"
&lt;/h2&gt;

&lt;p&gt;In practice, "later" becomes:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;after customers arrive&lt;/li&gt;
&lt;li&gt;after scale pressure begins&lt;/li&gt;
&lt;li&gt;after incidents expose gaps&lt;/li&gt;
&lt;li&gt;after engineering velocity slows&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;At that point, the system is no longer neutral.&lt;/p&gt;

&lt;p&gt;It has opinions:&lt;/p&gt;

&lt;p&gt;about structure&lt;br&gt;
about deployment&lt;br&gt;
about observability&lt;br&gt;
about service boundaries&lt;/p&gt;

&lt;p&gt;And those opinions are expensive to change.&lt;/p&gt;

&lt;h2&gt;
  
  
  &amp;nbsp;Where teams actually spend their time
&lt;/h2&gt;

&lt;p&gt;Across multiple environments I’ve worked in — from startups to large AWS-based enterprise systems — a consistent pattern appears:&lt;/p&gt;

&lt;p&gt;Engineering effort splits into two categories:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;domain / product requirements and features&lt;/li&gt;
&lt;li&gt;engineering foundation and operational work&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;In many early systems, the work that goes into engineering foundations - such as deployments, versioning, build and test standards and optimisations, pipelines etc. - becomes a dominant and usually hidden cost.&lt;/p&gt;

&lt;p&gt;At one startup, my estimate was that "technical" stories accounted for the majority of backlog creation over time, eclipsing feature development.&lt;/p&gt;

&lt;p&gt;This is not an edge case.&lt;/p&gt;

&lt;p&gt;This is how systems evolve.&lt;/p&gt;

&lt;h2&gt;
  
  
  &amp;nbsp;Why this is so hard to avoid
&lt;/h2&gt;

&lt;p&gt;Most teams don't consciously choose to neglect operational maturity.&lt;/p&gt;

&lt;p&gt;The problem is that product work is always visible, while engineering foundations are largely invisible.&lt;/p&gt;

&lt;p&gt;A new feature can be demonstrated to customers, investors, and stakeholders. It can be tied directly to revenue, growth, or market validation. Improvements to deployment pipelines, observability, security controls, or operational tooling rarely have that luxury. Their value is indirect, preventative, and often only becomes obvious when something goes wrong.&lt;/p&gt;

&lt;p&gt;As a result, engineering teams are under constant pressure to prioritise business-driven outcomes over engineering excellence. Every sprint presents another feature request, customer commitment, sales opportunity, or roadmap deadline competing for attention.&lt;/p&gt;

&lt;p&gt;Over time, small compromises accumulate:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Deployment processes remain partially manual because "we'll automate it later."&lt;/li&gt;
&lt;li&gt;Monitoring exists, but not at the depth needed to diagnose production issues quickly.&lt;/li&gt;
&lt;li&gt;Security controls are good enough for today's customers, but not tomorrow's.&lt;/li&gt;
&lt;li&gt;Operational knowledge lives in people's heads rather than in systems and documentation.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;None of these decisions are unreasonable in isolation. In fact, most are rational responses to commercial pressure.&lt;/p&gt;

&lt;p&gt;The challenge is that operational maturity compounds in exactly the same way technical debt does. The cost of postponing it is often hidden until growth, scale, compliance requirements, or a production incident suddenly expose the gap.&lt;/p&gt;

&lt;p&gt;By that point, fixing the foundations is competing with an even larger backlog, a larger customer base, and a business that has become increasingly dependent on systems that were never designed for the level of demand being placed on them.&lt;/p&gt;

&lt;h2&gt;
  
  
  &amp;nbsp;The real definition of production-ready
&lt;/h2&gt;

&lt;p&gt;A more accurate definition is:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;A system is production-ready when its operational properties are designed, not discovered.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;That includes:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;observability as a first-class concern&lt;/li&gt;
&lt;li&gt;consistent service structure and bounded contexts&lt;/li&gt;
&lt;li&gt;predictable deployment behaviour&lt;/li&gt;
&lt;li&gt;explicit failure handling patterns&lt;/li&gt;
&lt;li&gt;security and access boundaries defined early&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  The uncomfortable truth
&lt;/h2&gt;

&lt;p&gt;Most teams don’t lack capability.&lt;/p&gt;

&lt;p&gt;They lack a reusable starting point.&lt;/p&gt;

&lt;p&gt;So they rebuild production-readiness repeatedly, instead of inheriting it once.&lt;/p&gt;

&lt;h2&gt;
  
  
  The shift that matters
&lt;/h2&gt;

&lt;p&gt;The real architectural question is not:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;"How do we make this production-ready?"&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;It is:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;"Why are we rebuilding production readiness every time?"&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;This is the problem space I’ve been focused on with Forge: creating a reusable foundation so teams don’t rediscover production-readiness under pressure.&lt;/p&gt;

</description>
      <category>startup</category>
      <category>devops</category>
      <category>microservices</category>
      <category>aws</category>
    </item>
    <item>
      <title>The real startup killer isn't product — it's building platform foundations from scratch.</title>
      <dc:creator>Andrew Eells</dc:creator>
      <pubDate>Fri, 08 May 2026 22:15:14 +0000</pubDate>
      <link>https://dev.to/aeells/the-real-startup-killer-isnt-product-its-building-infrastructure-from-scratch-3olf</link>
      <guid>https://dev.to/aeells/the-real-startup-killer-isnt-product-its-building-infrastructure-from-scratch-3olf</guid>
      <description>&lt;p&gt;There's a pattern I've seen repeat a few times over 20 years building software systems in both startup and enterprise environments.&lt;/p&gt;

&lt;p&gt;Most early-stage teams believe they are building a product.&lt;/p&gt;

&lt;p&gt;In reality, they are building two things at once:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Their actual product&lt;/li&gt;
&lt;li&gt;An internal platform they didn't intend to build&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;And the second one quietly consumes man-years of engineering time.&lt;/p&gt;

&lt;h2&gt;
  
  
  The hidden tax on early-stage teams
&lt;/h2&gt;

&lt;p&gt;At some point, every startup hits the same phase:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;The MVP works&lt;/li&gt;
&lt;li&gt;The first customers arrive&lt;/li&gt;
&lt;li&gt;Engineering velocity starts to slow&lt;/li&gt;
&lt;li&gt;"Just one more service" becomes a platform discussion&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;And suddenly the backlog shifts.&lt;/p&gt;

&lt;p&gt;Not because the product changed - but because the foundations underneath it needed to evolve (without the wheels falling off!).&lt;/p&gt;

&lt;p&gt;Based on greenfield experience, I'd guesstimate that up to 60% of engineering effort can be focused on platform foundations, delivery lifecycle and operational processes, not domain features or core business differentiators.&lt;/p&gt;

&lt;p&gt;That ratio is not unusual.&lt;/p&gt;

&lt;p&gt;It is normal.&lt;/p&gt;

&lt;p&gt;And it is also destructive.&lt;/p&gt;

&lt;h2&gt;
  
  
  The repeated mistake
&lt;/h2&gt;

&lt;p&gt;Across organisations, I've seen the same systems rebuilt repeatedly:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;CI/CD pipelines&lt;/li&gt;
&lt;li&gt;Infrastructure-as-code structures&lt;/li&gt;
&lt;li&gt;Observability setups&lt;/li&gt;
&lt;li&gt;Security and access control patterns&lt;/li&gt;
&lt;li&gt;Service templates and API conventions&lt;/li&gt;
&lt;li&gt;Deployment and rollback strategies&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Each time:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;slightly different&lt;/li&gt;
&lt;li&gt;slightly inconsistent&lt;/li&gt;
&lt;li&gt;always re-learned under pressure&lt;/li&gt;
&lt;li&gt;and generally whilst trying to keep production lights on&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The irony is that none of these are domain-specific.&lt;/p&gt;

&lt;p&gt;They are global engineering concerns.&lt;/p&gt;

&lt;p&gt;Yet every team reinvents them.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2F1fl48d9xk1a44815ojhp.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2F1fl48d9xk1a44815ojhp.png" alt="Taskfile commands" width="800" height="281"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  Why this keeps happening
&lt;/h2&gt;

&lt;p&gt;It's not incompetence.&lt;/p&gt;

&lt;p&gt;It's timing.&lt;/p&gt;

&lt;p&gt;Early-stage teams optimise for:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;speed&lt;/li&gt;
&lt;li&gt;product delivery&lt;/li&gt;
&lt;li&gt;survival&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;So they defer platform thinking until it becomes unavoidable.&lt;/p&gt;

&lt;p&gt;At which point:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;the system is already live&lt;/li&gt;
&lt;li&gt;constraints are already baked in&lt;/li&gt;
&lt;li&gt;rewrites are expensive&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;So they rebuild under pressure instead of designing under intention.&lt;/p&gt;

&lt;h2&gt;
  
  
  The real cost
&lt;/h2&gt;

&lt;p&gt;The cost isn't just engineering time.&lt;/p&gt;

&lt;p&gt;It's:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;delayed product delivery&lt;/li&gt;
&lt;li&gt;inconsistent system behaviour&lt;/li&gt;
&lt;li&gt;increased operational risk&lt;/li&gt;
&lt;li&gt;premature senior hiring in platform roles&lt;/li&gt;
&lt;li&gt;architectural fragmentation across services (and teams)&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Most importantly:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;It shifts engineering from building product value to managing internal complexity.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;h2&gt;
  
  
  What changes when you design it once
&lt;/h2&gt;

&lt;p&gt;When these concerns are treated as a reusable foundation instead of one-off decisions:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;teams ship faster&lt;/li&gt;
&lt;li&gt;systems stay consistent&lt;/li&gt;
&lt;li&gt;operational overhead drops&lt;/li&gt;
&lt;li&gt;architecture stops diverging across services&lt;/li&gt;
&lt;li&gt;engineering focus returns to product domain logic&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;This is the problem I've been formalising into an opinionated bootstrap approach with &lt;a href="https://forgeplatform.software/" rel="noopener noreferrer"&gt;Forge Platform&lt;/a&gt;. &lt;/p&gt;

&lt;p&gt;It enables production-grade microservices from day one with enterprise foundations at startup speed.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fabdduz4lfi107g90ss8v.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fabdduz4lfi107g90ss8v.png" alt="Forge Platform GHA workflows" width="800" height="307"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Not because teams cannot build these things.&lt;/p&gt;

&lt;p&gt;But because they keep building them repeatedly, under pressure, in slightly different ways, everywhere.&lt;/p&gt;

&lt;h2&gt;
  
  
  The question I keep coming back to
&lt;/h2&gt;

&lt;p&gt;If every team ends up rebuilding the same foundations…&lt;/p&gt;

&lt;p&gt;Why are we still rebuilding them every time?&lt;/p&gt;

&lt;p&gt;If you have any questions or are interested to find out more, it would be great to hear from you in the comments or feel free to say &lt;a href="mailto:hello@forgeplatform.software"&gt;hello&lt;/a&gt;.&lt;/p&gt;

</description>
      <category>startup</category>
      <category>devops</category>
      <category>microservices</category>
      <category>discuss</category>
    </item>
  </channel>
</rss>
