Oluwasegun Adedigba for AWS Community Builders

Posted on Nov 11 Edited on Nov 11

Non-functional Application Requirements: Maintainability

#aws #architecture #ai #devops

On August 1, 2012, Knight Capital rolled out new code to its order router by hand and missed one of the eight servers. The new code also reused a flag that once switched on "Power Peg", a feature nobody had used since 2003. On the missed server, that flag woke the dead code up. In 45 minutes the router sent millions of orders and got more than 4 million executions while trying to fill 212 customer orders, and Knight lost more than $460 million. The SEC found that no second technician reviewed the deploy, and no written procedure required one (SEC order, 2013).

Nothing in that story is exotic. It took dead code, a flag with two meanings and a deploy that depended on nobody missing a step.

That's what maintainability failures usually look like. The system runs fine until somebody has to change it.

TL;DR

Maintainability is how cheaply and safely you can change a running system. Measure it with DORA's five delivery metrics.

Prompts and model IDs are code, model end-of-life is a scheduled outage, and MCP servers are dependencies. Version, test and inventory all three.

Gate every prompt or model change on an eval suite: a golden set, cheap checks, then a calibrated judge.

AI raises throughput but, per DORA 2025, still hurts stability. Small batches and fast rollback turn speed into delivery.

What: maintainability, and how to measure it

ISO/IEC 25010:2023 defines maintainability as the "capability of a product to be modified by the intended maintainers with effectiveness and efficiency." For a cloud system, I break that into five parts you can check:

Modifiability. You change one thing without touching five others.
Testability. A machine tells you in minutes whether the change broke anything.
Observability. You can follow one request end to end.
Deployability. You can ship and undo a change without a hero.
Operability. On-call can run it from a runbook, upgrades included.

Clean code helps, but a tidy codebase with a manual deploy and no rollback is still hard to maintain.

The measurable form is DORA's delivery metrics, five since 2024. Throughput is change lead time, deployment frequency and failed deployment recovery time. Instability is change fail rate and deployment rework rate, the share of deploys forced by a production incident. DORA says to measure one service at a time and never turn the numbers into targets. For AI features, add the eval pass rate per prompt version and the days until your nearest model end-of-life.

Why: neglect compounds

Unmaintainable code costs time. Adam Tornhill and Markus Borg found that low-quality code had 15 times more defects than high-quality code, and that fixing issues in it took 124% more development time (Code Red, 2022).

AI made the problem faster, not smaller. DORA's 2024 report estimated that a 25% rise in AI adoption went with a 7.2% drop in delivery stability. The 2025 edition, the latest at the time of writing, surveyed nearly 5,000 professionals. 90% use AI at work, and AI adoption now goes with higher throughput but still with lower stability (DORA, 2025).

Don't trust how fast it feels, either. In METR's 2025 trial, experienced open-source developers took 19% longer with AI tools while believing they'd been 20% faster (METR, 2025). METR called the data from its late-2025 follow-up, published in February 2026, "an unreliable signal", so treat the 19% as a warning about perception.

When: where maintainability enters the lifecycle

Design. Give every prompt, model ID and flag one owner.
Build. Everything deployable lives in Git, prompts and IaC included.
Test. Evals on every prompt or model change, a canary on every deploy.
Operate. Drift checks, a lifecycle inventory, traces you can follow, and a monthly look at the DORA metrics.

Where: two pitfalls that predate AI

Each pitfall follows the same arc: what you'll notice, why it happens, the fix, and the tool I'd reach for.

Pitfall 1: Deploys that live in someone's head

You'll notice it as a deploy that only works when one person runs it, and a rollback nobody wants to try.

It happens because the deploy started as a script for one server and grew. Nobody wrote every step down, and a missed step fails silently, as it did at Knight.

flowchart LR
  A["Manual deploy<br/>to 8 servers"] --> B["1 server missed"]
  C["New code reuses<br/>an old flag"] --> D["Flag switched on<br/>at market open"]
  B --> D
  D -->|"on the missed server"| E["Power Peg code<br/>unused since 2003 runs"]
  E --> F["4 million executions in 45 min<br/>loss over 460M USD"]

It took three ordinary gaps lining up: a manual deploy, dead code left in place, and a flag that meant two things.

The fix is a pipeline that is the only road to production, with rollback that needs no human. CodePipeline orchestrates, CodeBuild tests, and CodeDeploy shifts traffic. For Lambda, CodeDeployDefault.LambdaCanary10Percent5Minutes sends 10% of traffic to the new version, waits five minutes, then shifts the rest. Add a CloudWatch alarm to the deployment group with auto-rollback on DEPLOYMENT_STOP_ON_ALARM, and a bad canary redeploys the last good revision while you sleep. CDK Pipelines updates its own stages from the repo once deployed. GitHub Actions works too, and CodeCommit has been generally available again since November 2025.

Content needs staged rollouts too. CrowdStrike's July 2024 outage came from a content update, and its root-cause fixes include staged rings with bake time. Prompts and feature flags are content. Keep flags in AWS AppConfig, and never give an old flag a new meaning.

Pitfall 2: The console fix that never made it back

You'll notice it when the next deploy undoes last month's console hotfix, or fails on a resource that "shouldn't" look like that.

It happens because incidents reward speed, and nobody owns the follow-up.

The fix is to check for drift on a schedule and before every deploy, then fix it in code:

cdk drift MyStack --fail exits with code 1 on drift, so it works as a pipeline step.
The AWS Config rule cloudformation-stack-drift-detection-check runs periodically, within a 15-minute limit, so scope it with tags.
Drift-aware change sets (November 2025) with --deployment-mode REVERT_DRIFT put drifted resources back to the template.
In Terraform, terraform plan -refresh-only -detailed-exitcode returns 2 when something changed outside Terraform.

CloudFormation only checks properties you set explicitly, so write the ones that matter into the template.

Maintainability in the age of AI

AI changes maintainability from two sides. As a workload, it adds things that change without a code diff. As an engineering aid, it writes more code than your review process was built for.

AI as a workload

Pitfall 3: Prompts and model IDs outside version control

You'll notice it when answers got worse on Tuesday and the diff is empty. The prompt lives in a console text box or an environment variable, and the model ID is a string someone edited.

It happens because prompts feel like copy, so they get edited like copy. But a prompt plus a model ID is your feature's behavior, so changing either is a release.

The fix is to keep both in Git and ship them through the pipeline. Amazon Bedrock Prompt Management stores prompts as numbered snapshot versions, so the app calls version 7 while you edit the draft. When you pass a prompt version ARN to Converse as the modelId, the request can't include system, inferenceConfig, toolConfig or additionalModelRequestFields (Converse API reference). That's the point. The version owns the model and its settings, not the caller.

Here's the shift-assistant prompt for Harbor Works, the fictional plant in my companion app Luso, as CDK. The template text sits in a file beside the code.

Checked against the CDK CfnPrompt reference and AWS::Bedrock::PromptVersion on Oct 29, 2026. Type-checked and synthesized with aws-cdk-lib 2.275.0 and TypeScript 5.9.3. Not deployed — test in a sandbox account first.

import { createHash } from 'node:crypto';
import { readFileSync } from 'node:fs';
import { App, RemovalPolicy, Stack, StackProps } from 'aws-cdk-lib';
import * as bedrock from 'aws-cdk-lib/aws-bedrock';
import * as ssm from 'aws-cdk-lib/aws-ssm';
import { Construct } from 'constructs';

// Prompt text, model ID and settings live in Git, next to the code that calls them.
const variant: bedrock.CfnPrompt.PromptVariantProperty = {
  name: 'main',
  templateType: 'TEXT',
  modelId: 'eu.anthropic.claude-sonnet-4-6', // pinned EU geo inference profile
  inferenceConfiguration: { text: { temperature: 0.2, maxTokens: 512 } },
  templateConfiguration: {
    text: {
      text: readFileSync('prompts/shift-assistant.txt', 'utf8'),
      inputVariables: [{ name: 'line' }, { name: 'question' }],
    },
  },
};

export class PromptStack extends Stack {
  constructor(scope: Construct, id: string, props?: StackProps) {
    super(scope, id, props);

    const prompt = new bedrock.CfnPrompt(this, 'ShiftAssistant', {
      name: 'shift-assistant',
      defaultVariant: 'main',
      variants: [variant],
    });

    // Every PromptVersion property forces replacement, so a hash of the variant in
    // the description cuts a new numbered version whenever anything in it changes.
    const hash = createHash('sha256').update(JSON.stringify(variant)).digest('hex');
    const version = new bedrock.CfnPromptVersion(this, 'ShiftAssistantVersion', {
      promptArn: prompt.attrArn,
      description: `release ${hash.slice(0, 12)}`,
    });
    version.applyRemovalPolicy(RemovalPolicy.RETAIN); // old versions stay callable

    // The eval stage and the app read the ARN from here: one place to promote or roll back.
    new ssm.StringParameter(this, 'CandidateArn', {
      parameterName: '/harbor-works/shift-assistant/candidate-prompt-arn',
      stringValue: version.attrArn,
    });
  }
}

new PromptStack(new App(), 'HarborWorksPrompts', { env: { region: 'eu-west-1' } });

Rolling back is a git revert, which cuts a new version with the old content. Not on Bedrock? A prompts/ folder plus a versioned model ID in config does the same job.

Pitfall 4: Shipping LLM changes with no eval suite

You'll notice it when prompt changes are "tested" in a playground with three questions, and users find the four answers that broke.

It happens because LLM output isn't deterministic, so teams decide it can't be tested. It can. You test properties and judgments instead of exact strings, in three layers:

A golden set of real questions with reference answers, owned by someone who knows the domain. I'd start with 50 from production logs and add every reported bug.
Cheap checks first. Must include, must not include, valid JSON. They cost nothing and catch a lot.
A calibrated judge for the rest. In Zheng et al.'s 2023 study, strong LLM judges agreed with humans over 80% of the time, as often as humans agree with each other, but showed position, verbosity and self-enhancement bias. Use a judge from a different model family at temperature 0, and check it against human labels first.

On AWS, Bedrock evaluations run judge jobs with built-in metrics such as Builtin.Correctness, even on responses from other providers. AgentCore Evaluations (GA March 2026) adds on-demand agent regression tests for CI/CD. The open-source choice is promptfoo, still MIT-licensed after joining OpenAI in 2026.

Pitfall 5: Treating model end-of-life as a surprise

You'll notice it when a model ID that worked for a year starts returning ResourceNotFoundException: This model version has reached the end of its life.

It happens because nobody put model lifecycles on a calendar. Bedrock models go from Active to Legacy to EOL. AWS publishes the EOL date when a model enters Legacy, usually six months ahead and sometimes only 45 days. In Legacy, existing customers "may lose access after 15 days of inactivity". After EOL, requests fail and "migration will not happen automatically" (Bedrock model lifecycle). Claude Sonnet 4.5 is already Legacy, with EOL on April 8, 2027.

flowchart LR
  A["Active<br/>earliest EOL on the model card"] -->|"AWS publishes the EOL date"| L["Legacy<br/>6 months or 45 days of notice"]
  L --> E["End of life<br/>requests fail"]
  A -.-> Y1["You: pin the ID<br/>and record the dates"]
  L -.-> Y2["You: eval the successor<br/>and migrate"]
  E -.-> Y3["You: already moved"]

The EOL date only appears once a model enters Legacy, so that's when your migration clock starts.

The fix is an inventory with an alarm. This check fails a daily build 90 days before trouble.

Checked against the GetFoundationModel API and the model lifecycle guide on Oct 29, 2026. Unit-tested with botocore Stubber on boto3 1.43.121. Not deployed — test in a sandbox account first.

"""Fail (exit 1) if a model we depend on is Legacy or within 90 days of EOL."""

import datetime as dt
import sys

import boto3

# Base model IDs: strip the eu./us./global. inference-profile prefix.
IN_USE = ["anthropic.claude-sonnet-4-6", "amazon.nova-2-lite-v1:0"]
WARN = dt.timedelta(days=90)

bedrock = boto3.client("bedrock", region_name="eu-west-1")
now = dt.datetime.now(dt.timezone.utc)
at_risk = []
for model_id in IN_USE:
    life = bedrock.get_foundation_model(modelIdentifier=model_id)["modelDetails"].get(
        "modelLifecycle", {}
    )
    eol = life.get("endOfLifeTime")  # set once the model enters Legacy
    print(f"{model_id}: {life.get('status')} eol={eol}")
    if life.get("status") == "LEGACY" or (eol and eol - now < WARN):
        at_risk.append(model_id)

sys.exit(1 if at_risk else 0)

Run the check from a Region that lists every model you use. In Europe, Sonnet 4.6 runs In-Region only in London (eu-west-2) and Ireland reaches it through the EU profile, so if the call fails in eu-west-1, point the client at eu-west-2.

Your upgrade runbook is then the eval suite: point a branch at the new model, run the golden set, compare. Runtimes need the same calendar. Lambda deprecates Python 3.10 on October 31, 2026, and Amazon Inspector exports SBOMs in CycloneDX 1.4 or SPDX 2.3 for EC2, ECR and Lambda, so the inventory isn't a spreadsheet.

Pitfall 6: Tracing a non-deterministic request with a retiring SDK

You'll notice it when a user says the assistant "went in circles". Your logs show a 200 after 14 seconds, but not which prompt version ran or which tools it called.

It happens for two reasons. One question becomes a chain of model and tool calls that request logs can't show. And many teams use the X-Ray SDKs, which entered maintenance mode on February 25, 2026 and reach end of support on February 25, 2027 (AWS, 2025). The X-Ray service itself stays.

The fix is OpenTelemetry, through AWS Distro for OpenTelemetry (ADOT) or the upstream SDKs, exporting to CloudWatch's OTLP endpoint at https://xray.<region>.amazonaws.com/v1/traces with SigV4. Follow the GenAI semantic conventions: gen_ai.operation.name and gen_ai.provider.name on every inference span, plus gen_ai.request.model and the gen_ai.usage.* token counts. Then add one attribute of your own, the prompt version ARN. It turns "answers got worse" into a query.

sequenceDiagram
  participant U as Operator
  participant API as Assistant API
  participant BR as Bedrock
  participant MCP as Fault history MCP server
  participant CW as CloudWatch OTLP
  U->>API: Why did line 2 stop
  Note over API: invoke_agent span with the prompt version ARN
  API->>BR: chat span, gen_ai.request.model
  BR-->>API: tool call requested, token counts
  API->>MCP: execute_tool span, get_fault_history
  MCP-->>API: last 10 faults
  API->>BR: second chat span with the tool result
  BR-->>API: answer and finish reason
  API-->>U: answer
  API->>CW: export all spans over OTLP

One question becomes four spans, and the prompt version attribute links a bad answer to the change that caused it.

The conventions are young. Every gen_ai.* attribute is at Development stability, so pin your instrumentation library. Prompt and completion text sits in an opt-in event because it often holds personal data, so leave it off in production by default.

CloudWatch generative AI observability (GA October 2025) gives you dashboards for model invocations and AgentCore agents, and CloudWatch Omni (GA September 2026) adds traces and evaluators in the IDE. Prefer to own the backend? Jaeger takes the same OTLP spans.

Pitfall 7: MCP servers nobody reviewed

You'll notice it when an agent behaves differently and no code changed. Or worse, it behaves the same and quietly does something extra.

It happens because MCP servers skip the controls you put on libraries. They get installed with an unpinned npx command, they hold credentials, and their tool descriptions go straight into the model's context. The MCP specification itself says tool annotations "should be considered untrusted, unless obtained from a trusted server."

That's already been exploited. In September 2025, the unofficial postmark-mcp npm package shipped 15 clean versions, then version 1.0.16 quietly BCC'd every email it sent to an attacker (The Register, 2025).

flowchart LR
  APP["Shift assistant"] --> PV["Prompt version<br/>Git and Bedrock"]
  PV --> M["Model ID<br/>pinned, has an EOL date"]
  APP --> FW["Agent framework<br/>pinned library"]
  APP --> MCP1["Fault history MCP server<br/>internal, reviewed"]
  APP --> MCP2["Email MCP server<br/>third-party npm package"]
  MCP2 --> CRED["Mail API credentials"]
  MCP2 -.->|"tool descriptions enter the prompt"| M
  RT["Lambda runtime<br/>deprecation date"] --> APP

Each box changes on its own schedule, and the third-party server is the one holding credentials, so each needs a pinned version and an owner.

The fix is to treat each MCP server as a third-party package with network access. Pin exact versions, never @latest. Put servers in your SBOM, and give each one least-privilege credentials of its own. Commit a snapshot of the tool schemas with a test that fails when the live tools differ, so a changed description becomes a red build.

AI as an engineering aid

Pitfall 8: AI-written code outpacing review

You'll notice it when pull requests get bigger, review turns into skimming, and change fail rate creeps up while everyone feels faster.

It happens because generating code got cheap and reviewing it didn't. DORA's 2025 report calls AI an amplifier, and its AI Capabilities Model singles out two habits: working in small batches, and strong version control, where frequent commits and frequent rollbacks amplified the gains.

The fix is to let machines carry the checks so people review intent. Cap pull request size, require tests with generated changes, and enforce lint, types and coverage in CodeBuild. Keep code owners, so generated code still has a human one. If change fail rate rises while lead time falls, you're borrowing against the future.

The same rules make coding agents useful for upgrades. Kiro (GA November 2025) does spec-driven development with property-based tests. AWS Transform custom (GA December 2025) runs Java 8 to 17, Python and Node.js upgrades from one CLI command. Still on Amazon Q Developer in the IDE? Its plugins reach end of support on April 30, 2027, with Kiro as the named successor.

Whatever the agent, the gate is the same: it opens a pull request, the full test and eval suite runs, and a person approves. An agent migration without tests is just a very large unreviewed diff.

Still immature: judges need recalibrating, the GenAI conventions are unstable, and MCP has no common standard for signing servers.

Hands-on: an eval gate for prompt and model changes

At Harbor Works, the shift assistant answers operators' questions about line faults. A wrong answer could mean someone clears a jam without locking out the conveyor, so prompt changes ship like code.

flowchart LR
  PR["Pull request<br/>prompt, model ID or code"] --> SRC["CodePipeline<br/>source and trigger"]
  SRC --> STG["Deploy to staging<br/>new prompt version"]
  STG --> CB["CodeBuild<br/>lint, unit tests, evals"]
  CB -->|"pass rate at or above the gate"| PROD["Deploy to production<br/>CodeDeploy canary"]
  CB -->|"below the gate or any error"| STOP["Pipeline stops<br/>report lists failing cases"]
  PROD -->|"canary alarm fires"| RB["Automatic rollback<br/>last good revision"]

A prompt or model change can't reach production without passing the same eval gate and canary as code.

The staging deploy runs the pitfall 3 stack, which writes the new prompt version ARN to Parameter Store. CodeBuild reads it and runs lint, tests and evals.

Checked against the CodeBuild buildspec reference and available runtimes on Oct 29, 2026. YAML-parsed, not run in CodeBuild — test in a sandbox account first.

version: 0.2

env:
  variables:
    JUDGE_MODEL_ID: "eu.amazon.nova-2-lite-v1:0" # a different family from the generator
    EVAL_THRESHOLD: "0.9"
  parameter-store:
    # Written by the staging deploy: ARN of the prompt version under test
    PROMPT_VERSION_ARN: "/harbor-works/shift-assistant/candidate-prompt-arn"

phases:
  install:
    runtime-versions:
      python: 3.12
    commands:
      - pip install -r requirements-dev.txt # pinned boto3, pytest, ruff
  build:
    commands:
      # One phase on purpose: a failing command skips the rest of the phase,
      # so the paid eval step only runs after lint and unit tests pass.
      - ruff check .
      - ruff format --check .
      - pytest -q --junitxml=reports/unit.xml
      - python evals/run_evals.py evals/golden.jsonl reports/evals.xml

reports:
  unit-tests:
    files: ["unit.xml"]
    base-directory: reports
    file-format: JUNITXML
  llm-evals:
    files: ["evals.xml"]
    base-directory: reports
    file-format: JUNITXML

Each line of the golden set is one case:

{"id": "e17-conveyor-jam", "vars": {"line": "2", "question": "Conveyor stopped with fault E-17. What do I do first?"}, "reference": "E-17 is the jam sensor. Lock out the conveyor before clearing the jam, then reset from the HMI.", "must_include": ["lock"], "must_not_include": ["bypass the sensor"]}

The runner applies the string checks, asks the judge only when they pass, and writes JUnit XML for CodeBuild's test reports.

Checked against the Converse API reference on Oct 29, 2026. Unit-tested with botocore Stubber on boto3 1.43.121 and Python 3.13, ruff clean. Not run against Bedrock — test in a sandbox account first.

"""Eval gate: run the golden set against a prompt version, write JUnit XML."""

import json
import os
import re
import sys
import xml.etree.ElementTree as ET

import boto3
from botocore.config import Config

brt = boto3.client("bedrock-runtime", config=Config(retries={"mode": "standard"}))
PROMPT_ARN = os.environ.get("PROMPT_VERSION_ARN", "")  # ...:prompt/ID:VERSION
JUDGE = os.environ.get("JUDGE_MODEL_ID", "")
THRESHOLD = float(os.environ.get("EVAL_THRESHOLD", "0.9"))
RUBRIC = (
    "Grade a plant-floor assistant.\nQuestion: {q}\nReference: {ref}\n"
    "Candidate: {ans}\nScore 1 only if the candidate agrees with the reference "
    'and gives a safe next step. Reply with JSON only: {{"score": 0 or 1, "reason": "..."}}'
)


def text_of(r: dict) -> str:
    return next(b["text"] for b in r["output"]["message"]["content"] if "text" in b)


def generate(variables: dict) -> str:
    # The prompt version pins model, system prompt and inference config, so
    # Converse rejects system/inferenceConfig/toolConfig here. Variables only.
    r = brt.converse(
        modelId=PROMPT_ARN,
        promptVariables={k: {"text": v} for k, v in variables.items()},
    )
    return text_of(r)


def judge(case: dict, answer: str) -> tuple[int, str]:
    prompt = RUBRIC.format(
        q=case["vars"]["question"], ref=case["reference"], ans=answer
    )
    r = brt.converse(
        modelId=JUDGE,  # a different model family from the generator
        messages=[{"role": "user", "content": [{"text": prompt}]}],
        inferenceConfig={"maxTokens": 200, "temperature": 0},
    )
    raw = text_of(r)
    try:
        verdict = json.loads(re.search(r"\{.*\}", raw, re.DOTALL).group(0))
        return int(verdict["score"]), str(verdict.get("reason", ""))
    except (AttributeError, ValueError, KeyError):
        return 0, f"unparseable judge output: {raw[:120]}"  # fail closed


def check(case: dict) -> str | None:
    """None if the case passes, otherwise the reason it failed."""
    answer = generate(case["vars"])
    low = answer.lower()
    missing = [w for w in case.get("must_include", []) if w.lower() not in low]
    banned = [w for w in case.get("must_not_include", []) if w.lower() in low]
    if missing or banned:  # cheap string checks before paying for a judge call
        return f"missing={missing} banned={banned}"
    score, reason = judge(case, answer)
    return None if score == 1 else f"judge: {reason}"


def main(golden: str, junit: str) -> int:
    with open(golden, encoding="utf-8") as f:
        cases = [json.loads(line) for line in f if line.strip()]
    suite = ET.Element("testsuite", name="llm-evals", tests=str(len(cases)))
    failed = 0
    for case in cases:  # a Bedrock error raises and fails the build: fail closed
        tc = ET.SubElement(suite, "testcase", classname="golden", name=case["id"])
        if reason := check(case):
            failed += 1
            ET.SubElement(tc, "failure", message=reason[:500])
    suite.set("failures", str(failed))
    ET.ElementTree(suite).write(junit, encoding="utf-8", xml_declaration=True)
    rate = (len(cases) - failed) / len(cases)
    print(f"pass rate {rate:.1%} on {len(cases)} cases (gate {THRESHOLD:.0%})")
    return 0 if rate >= THRESHOLD else 1


if __name__ == "__main__":
    sys.exit(main(sys.argv[1], sys.argv[2]))

Before you run it:

Permissions. The build role needs ssm:GetParameters, bedrock:RenderPrompt, bedrock:InvokeModel on the inference profiles and the foundation models in every Region they route to, and the CodeBuild report actions.
Threshold. Run the suite on today's prompt, set the gate just below that score, and raise it as the golden set grows.
Cost. Each passing case costs two model calls, plus build minutes ($0.005 a minute for general1.small in AWS's pricing example).

Add the pitfall 5 lifecycle check as another command, and a model EOL date fails the build three months early.

Decide: which tools, and what it costs elsewhere

Decision	Choose the first when	Choose the second when
CDK Pipelines vs. GitHub Actions	Your infrastructure is CDK and you want a self-updating pipeline	You deploy Terraform or mixed tools, or already live in GitHub
CDK vs. Terraform	AWS-only, TypeScript team, you want `cdk drift` and drift-aware change sets	Multi-cloud or SaaS providers, or `plan` review is your change control
Bedrock Prompt Management vs. prompt files only	You're on Bedrock and want versions callers can't override	You use several model providers
Bedrock or AgentCore evaluations vs. promptfoo	You want managed jobs, built-in metrics or agent checks	You want evals beside unit tests, for any provider
CloudWatch GenAI observability vs. open-source OTLP backend	Mostly AWS, and you want AgentCore dashboards with no extra infrastructure	Multi-cloud, or you must own retention

Maintainability work pulls on the other eight requirements:

Other NFR	What maintainability work does to it
Security	Pipelines concentrate power, so guard the pipeline role
Compliance	Git history and prompt versions record who changed what
Accessibility	Checks such as axe-core run in the same build as lint
Performance	Canaries add minutes per deploy, tracing a little per request
Availability	Small, reversible changes lower change fail rate, and drift breaks failovers
Scalability	IaC stamps out new cells and Regions instead of hand-building them
Cost	Evals spend tokens on every change, and traces cost storage
Sustainability	Eval runs and long trace retention burn hidden compute

Who: owners and the shared responsibility split

Activity	Product owner	Tech lead	Developers	Platform / SRE
Pipeline, canaries, rollback alarms	I	A	C	R
IaC and drift checks	I	A	R	R
Golden set and eval threshold	A	C	R	I
Prompt and model versions	C	A	R	I
Model, runtime and MCP inventory	I	A	C	R
Review rules for AI-generated code	I	A	R	C

R = responsible, A = accountable, C = consulted, I = informed.

Who patches what depends on the compute type:

EC2. You patch the OS, agents and AMIs.
Containers. You own images and base-image updates. On Fargate, AWS patches the host.
Lambda. AWS patches the runtime. You move off it before its block dates.
Managed AI (Bedrock). AWS and the provider run the model and publish its lifecycle dates. You pin IDs, own the evals and migrate before EOL.

Monday checklist

[ ] Make the pipeline the only way to production, with alarm-based rollback on one service.
[ ] Run a drift check against production and fix what it finds in code.
[ ] Move every prompt and model ID into Git, then search for model ID strings anywhere else.
[ ] Write 20 golden cases from real questions and record today's pass rate.
[ ] Check your judge against 50 human-labelled answers before you gate on it.
[ ] Add the model lifecycle check as a daily build.
[ ] Put the prompt version ARN on your spans, and plan the move off the X-Ray SDKs before February 25, 2027.
[ ] Pin every MCP server to an exact version and commit its tool schemas.
[ ] Track change fail rate and deployment rework rate per service.

Conclusion

Knight Capital didn't lose $460 million to a hard problem. It lost it to a manual deploy, a dead feature and a flag with two meanings. Every pitfall here is a system nobody can fully see or safely change.

AI gives the old mistakes new places to hide. A prompt edited in a console is a manual deploy. A model past its EOL date is an unpatched runtime. An MCP server on @latest is an unpinned dependency holding credentials. The fixes are the ones you know: version it, test it, trace it, and keep a way back. The new part is the eval suite, because "does it still work?" now has a fuzzy answer.

Give every change one meaning, one owner and one way back.

Series: Non-functional Application Requirements

Non-functional Application Requirements: An Introduction
Non-functional Application Requirements: Security
Non-functional Application Requirements: Compliance
Accessibility
Performance
Availability
Scalability
Maintainability
Cost (coming soon)
Sustainability (coming soon)

Cover image: AWS re:Invent 2023 – Keynote with Dr. Werner Vogels

Next up, Cost: one real, sourced case study of a team cutting its AWS bill, and what per-token pricing does to unit economics.

DEV Community