DEV Community: Chidozie Uzoegwu

Question 55, 30 Minutes Left: How I Prepared for the AWS GenAI Developer Professional Exam

Chidozie Uzoegwu — Tue, 13 Oct 2026 09:35:05 +0000

I passed the AWS Certified Generative AI Developer - Professional exam (AIP-C01) recently. At one point I looked at the clock and I was on question 55 with about 30 minutes left.

I was reading every question deliberately, making sure every condition in it was covered before I committed to an answer. This exam punishes skimming. Most of the wrong options are technically real AWS features that answer part of the question, and the part they miss is usually one clause you read too quickly.

This is the guide I wanted when I started: what the exam covers, what I studied, how I learned to read the questions, what each of the main services is actually for, and the traps that come up again and again.

The exam at a glance

75 questions, of which 65 are scored and 10 are unscored. You can't tell which are which.
180 minutes, which works out at 2.4 minutes per question.
Scaled score from 100 to 1,000. You need 750 to pass.
Multiple choice has one right answer out of four. Multiple response has two or more right answers out of five or more, and you must pick all of them to get any credit.
No penalty for guessing, and unanswered questions are marked wrong. Never leave a blank.
Scoring is compensatory, so you don't need to pass each domain. You need to pass the exam overall.

Domain	Weight
1. Foundation Model Integration, Data Management, and Compliance	31%
2. Implementation and Integration	26%
3. AI Safety, Security, and Governance	20%
4. Operational Efficiency and Optimization for GenAI Applications	12%
5. Testing, Validation, and Troubleshooting	11%

Domains 1 and 2 are 57% of the scored content between them. If your time is limited, that is where it goes.

The exam guide also lists what is out of scope: model development and training, advanced ML techniques, and data and feature engineering. This is an exam about building with foundation models on AWS, so don't spend a week on training theory.

What I used

The main course. Stéphane Maarek and Frank Kane's "Ultimate AWS Certified Generative AI Developer - Professional" course on Udemy, with its full-length practice exam. It follows the official exam guide closely and has hands-on labs, which matter for this exam because so many questions turn on what a specific API or parameter actually does.

QA learning platform. I had access through the AWS Community Builders programme and used it alongside the course.

AWS Skill Builder. AWS's own training platform, with official exam preparation material for this certification.

The official exam guide. Each domain lists tasks and skills, and many skills include a "for example" list of AWS services. Those lists are a good map of where answer options come from. I turned the whole guide into a checklist and rated myself on every skill: built it, know it but would hesitate between it and a neighbouring service, or couldn't pick it out of four options. The third group became my study plan.

The in-scope services list. It sits in the exam guide too. When I compared it with my practice material, a handful of in-scope services never appeared in any practice question at all, including Cost Explorer and Cost Anomaly Detection. Practice questions can only test what their authors wrote, so read the in-scope list yourself and fill the gaps from the docs.

AWS documentation, checked live. Whenever a practice explanation and my own understanding disagreed, I went to the current AWS docs and kept a log of what I checked. Bedrock changes fast. A few practice explanations had fallen behind the service, and the docs were right every time.

Hands-on work. I build and run a production generative AI system on AWS, including Bedrock fine-tuning, retrieval and evaluation. That helped most on the operational questions, where the trap is something you only learn by being caught by it, like an async job that says "accepted" long before the work is done.

How I read questions

Exam writers call the question text the stem: the scenario and the question above the answer options.

Partway through preparing, I went back through every practice question I'd got wrong and sorted the mistakes. I expected to find gaps in my knowledge. Mostly I didn't. In most cases I already knew the rule that decided the question. I had read the question in the wrong order.

The pattern was always the same. I read the stem, saw an option that sounded right, and then reread the stem looking for reasons to agree with it. The wrong options on this exam are written to match the most memorable phrase in the stem, so that habit picks the wrong answer again and again.

So I replaced my checklist with four steps, done in order on every question.

1. Find the constraint sentence before reading any option

Almost every stem has one sentence that acts as a filter rather than background: "with the least custom code", "without redeploying the API", "must not require client changes", "using a fully managed service". It's rarely the first sentence. Find it and hold it before your eyes reach the options, because once you've seen a plausible answer you'll read everything else to justify it.

Ask: what is this question forbidding?

2. If options share an opening, compare the endings

It's common for all four options to be identical for three or four lines and differ only in the final clause. Reading left to right means deciding on the part that's the same in every option, which by definition decides nothing. Ignore the shared part.

Ask: where do these options actually disagree?

3. Eliminate on violation, never select on match

Don't ask which option sounds right. Cross an option out the moment it breaks a constraint. A distractor usually satisfies one requirement very well and quietly fails another.

Ask: which constraint does each option break?

4. Check your pick against every constraint

An option that fails one requirement is out, however good it looks on the others. If you can't say what ruled out each of the other options, you haven't finished the question.

Ask: what ruled out each of the others?

These four steps took me longer per question. They were also what moved my practice results. More on the cost of that in the pacing section at the end.

Managed versus custom knowledge bases

This one deserves its own section, because the same question shape can have opposite answers depending on a single phrase.

Amazon Bedrock Knowledge Bases now come in two flavours. A custom knowledge base is the one most people know: you choose the vector store (OpenSearch Serverless, Aurora, and so on), the parsing, the chunking, and how retrieval runs. A fully managed knowledge base has the service manage those pieces for you. The two do not support the same APIs and options, and the exam leans on that.

What changes between them:

Feature	Custom knowledge base	Managed knowledge base
Generating an answer	`RetrieveAndGenerate`, or `Retrieve` and then your own model call	Not `RetrieveAndGenerate`. The API reference says to use `AgenticRetrieveStream` or `Retrieve` instead
Breaking a question into sub-questions	Turn on query decomposition in `RetrieveAndGenerate`, or decompose it yourself (for example in Lambda), call `Retrieve` per sub-query and combine the results	`AgenticRetrieveStream` plans sub-queries, retrieves until it judges the evidence sufficient, deduplicates chunks, cites sources and streams its planning as trace events. Managed only
Chunking	All strategies, including semantic	Semantic chunking not supported
Parsing	Default, foundation model or Bedrock Data Automation parsing	Smart parsing by default. Foundation model and Bedrock Data Automation parsing not supported
Metadata filters	Full set	`startsWith` and `stringContains` not supported
Guardrails on agentic retrieval	Not applicable	Blocking only. Masking not supported

A worked example

The ask. A team wants answers that break complex questions into sub-questions, show the planning steps, and cite sources, "using a fully managed knowledge base".

The tempting option. RetrieveAndGenerate with query decomposition switched on. It sounds perfect, because it really does decompose queries.

Why it's wrong. RetrieveAndGenerate is barred from managed knowledge bases, and it doesn't stream planning traces. The answer is AgenticRetrieveStream.

What flips it. Change one phrase so the team runs its own vector store, and the answer becomes decomposing the question yourself and calling Retrieve per sub-query.

When a knowledge base question appears, find out which flavour it is before you read the options.

What each service is for

Most wrong options on this exam are real services used for the wrong job. Knowing what each one is for, and what it isn't for, rules out options quickly.

Calling models

Converse gives you one request shape across models. If the requirement is "swap models without changing the client code", it's Converse.
InvokeModel takes a model-specific JSON body. It works, but every model family has its own format.
ApplyGuardrail checks text against a guardrail without calling a model at all. That's how you reject input before any model is invoked.
CountTokens estimates a request's token count before you send it. You don't need to call a small model to "estimate" it.
Guardrails attach to Converse through guardrailConfig. Header-based guardrails are for InvokeModel.

Retrieval

Retrieve returns raw chunks with their metadata and relevance scores. Use it when you need to inspect or verify what was retrieved.
RetrieveAndGenerate returns a generated answer with citations. Use it on custom knowledge bases when you want the answer and where it came from.
Hybrid search combines vector search with search over the raw text. It's only supported for Amazon RDS, OpenSearch Serverless and MongoDB vector stores that have a filterable text field. Otherwise the query falls back to semantic search. Part numbers, codes and IDs are the classic case that needs it.
Reranking reorders results that were already retrieved. It improves ordering. It can't find documents the search missed.

Chunking

There are four strategies, and you choose one when you connect the data source. You can't change it afterwards: a different strategy means a new data source.

Default / fixed size splits into roughly equal chunks. With no configuration, it's 300 tokens with 20% overlap.
Hierarchical matches on small child chunks and returns the larger parent chunk. Use it when you need precise matching but the answer needs the surrounding context.
Semantic splits on meaning, so chunks don't break mid-thought. It fixes bad boundaries. It doesn't add surrounding context, and it isn't available on managed knowledge bases.
None treats each file as one chunk. It only makes sense if you've already split your data into one record per file, which is custom preprocessing. A stem that says "without custom preprocessing" rules it out.

Keeping data current

Sync (StartIngestionJob) crawls a data source and picks up new, changed and deleted content. Third-party connectors such as Confluence use this route.
Direct ingestion (IngestKnowledgeBaseDocuments) adds or updates individual documents without a sync. It's only available for S3 and custom data sources. On S3 sources, don't mix the two at the same time: changes indexed directly aren't written back to S3 and can be overwritten by the next sync.

Safety and privacy

Bedrock Guardrails is the policy engine for prompts and responses, applied to both input and output. Its filters each do a different job:

Filter	What it does
Content filters	Grade harmful content by category, including prompt attacks
Denied topics	Block a subject outright, however politely it's asked
Word filters	Block specific words or phrases
Sensitive information filters	Block or mask PII and custom patterns
Contextual grounding checks	Catch answers not supported by the retrieved source
Automated Reasoning checks	Validate answers against logical rules

The other services that come up next to it:

Amazon Macie finds sensitive data stored in S3. It doesn't inspect live prompts.
AWS WAF works at the HTTP layer. It can't judge what a prompt is trying to do.
Amazon Comprehend detects PII and entities in text. It's useful as a pre-processing step, for example redacting before storage.

Prompts and orchestration

Prompt Management stores versioned prompt templates. Production should point at a published version, never the editable draft.
Bedrock Flows chains prompts, conditions and knowledge base lookups without much code. Flows also go to production as published versions behind an alias.
Step Functions Standard suits long-running, auditable workflows (up to a year) with exactly-once execution. It's the one that supports waiting for a human callback.
Step Functions Express suits high-volume, short work (up to five minutes) with at-least-once execution. It doesn't support the callback or .sync patterns, so it can't wait for a human approval.
AWS AppConfig changes configuration at runtime without redeploying. Switching models or providers without a redeploy is its classic use.

Evaluation

Automatic evaluation runs without people and can serve as an unattended release gate. Each job takes at most 1,000 prompts, from a .jsonl file in S3.
Human evaluation uses human workers, so it can't be unattended. It needs CORS configured on the S3 output bucket; automatic jobs don't.
Starting an evaluation job returns a job ARN. That means it started, not that it passed.

Operations and cost

Model invocation logging captures the full request and response bodies, to CloudWatch Logs or S3 in the same account and Region. It only covers calls through the bedrock-runtime endpoint.
CloudWatch metrics give counts, latency and token usage, but no prompt text.
CloudTrail records API calls, meaning who called what, without the content.
Provisioned Throughput is for steady, predictable, committed volume. Prompt caching is for repeated prompt prefixes. They solve different problems.
Cost Explorer and Cost Anomaly Detection cover spend analysis and alerting. Remember they exist; practice material rarely mentions them.

The traps worth knowing

These shapes came up again and again. Each comes with an example.

Reading traps

One word apart. Two options identical except for a single token: a different chunking strategy, Retrieve against RetrieveAndGenerate, Standard against Express. The answer is always in the part you haven't read yet.

Half of a multiple-response. With no partial credit, one wrong pick zeroes the question. The correct picks usually cover different requirements or different layers, such as prevent plus detect, or real-time updates plus a scheduled rebuild. Two picks that solve the same requirement is a warning sign.

The limit you didn't check. A suspiciously specific number in the stem usually means a documented limit is being tested. 1,600 prompts won't fit in one automatic evaluation job. Lambda can't run past 15 minutes. An Express workflow stops at five minutes. Do the arithmetic.

The setting you can't change later. Chunking strategy is fixed once the data source is connected, and a Step Functions workflow type is fixed once the state machine is created. When a stem says the team "is creating the data source now", that sentence is there for a reason.

Wrong tool for the job

Monitoring offered for prevention. The stem says prevent, block or reject, and an option offers a metric, alarm, log or dashboard. Those detect after the fact. Prevention means something in the request path: an IAM deny, a guardrail on the invocation, schema validation, an organisation policy.

Right service, wrong place. A real, sensible service used at a point where it can't act. Macie scans S3, so it never sees a live prompt. A reranker reorders chunks that were already retrieved, so it can't fix poor recall. Ask where in the request path each option actually runs.

Custom build where a managed feature exists. When the stem says "least operational overhead" or "least custom code", it is telling you to use the managed service, even when the custom Lambda version would clearly work. Glue Data Quality instead of a Lambda rule engine, Prompt Management instead of prompts in a Git repo, Bedrock Data Automation instead of a hand-built Textract and Comprehend pipeline.

Fixing the symptom. If documents were parsed badly at ingestion, no retrieval setting will repair them. Fix parsing and re-sync. Hybrid search and reranking can't recover text that was never extracted properly.

The wrong endpoint. Bedrock has separate endpoints for the control plane (bedrock), model inference (bedrock-runtime), agent and knowledge base setup (bedrock-agent) and agent and retrieval calls (bedrock-agent-runtime). Private inference from a VPC needs the bedrock-runtime interface endpoint. An option offering the bedrock endpoint for inference sounds right and isn't.

Logs, metrics and audit trails mixed up. If the requirement is the actual prompt and response text, only model invocation logging has it. Metrics have numbers. CloudTrail has who called what. An option that looks for prompt text in CloudTrail is wrong.

Human versus automatic evaluation. "Unattended" rules out human evaluation. "Humans must compare the outputs" rules out LLM-as-a-judge. And if the stem mentions CORS on the output bucket, it's quietly telling you the job uses human workers.

Wrong state or scope

Accepted is not complete. An async call that returns a job ID or an "in progress" status has only started the work. Anything that treats submission as success skips the poll, the final status and the result.

Draft against published version. If the stem mentions production, stability, auditability or reproducibility, a mutable draft is already ruled out. Prompts, flows and guardrails all follow this rule.

A real parameter on the wrong API. Sampling parameters such as temperature on an embeddings call, or messages added to a call that already uses a managed prompt template. Check that each parameter belongs to the API named in the option.

Scoped too narrowly. Cross-Region inference routes requests to several Regions, so IAM and any organisation policies must allow the model in every destination Region of the profile. Scoping permissions to the calling Region looks tighter and fails at runtime with access denied.

Topics I'd make sure of

This is a study map built from the exam guide's tasks.

Domain 1: Foundation Model Integration, Data Management, and Compliance

Choosing foundation models and switching between them without code changes
Resilience, including cross-Region inference and graceful degradation
Data validation before ingestion
Vector store choices and Bedrock Knowledge Bases, including managed versus custom, chunking strategies and metadata filtering
Retrieval design: hybrid search, reranking and query decomposition
Prompt engineering and Prompt Management with versioning

Domain 2: Implementation and Integration

Bedrock Agents and tool use, action groups and orchestration
Deployment options: on-demand, provisioned throughput and custom model deployment
Integration patterns with API Gateway, Lambda, Step Functions and event-driven designs
The Bedrock runtime APIs, including Converse against InvokeModel, streaming, and what each request accepts

Domain 3: AI Safety, Security, and Governance

Bedrock Guardrails in detail: every filter type, where guardrails apply, and versioning them across accounts
Prompt injection defence
Encryption, VPC endpoints and data residency
Governance: logging, lineage and audit trails
Responsible AI principles and how AWS expects you to put them into practice

Domain 4: Operational Efficiency and Optimization for GenAI Applications

Token cost and its drivers
Caching
Choosing between on-demand, batch and provisioned capacity
Latency tuning
Monitoring with CloudWatch and invocation logging, and cost tracking with Cost Explorer and Cost Anomaly Detection

Domain 5: Testing, Validation, and Troubleshooting

Bedrock model evaluation, both automatic and with a judge model, plus human evaluation
RAG evaluation, separating retrieval quality from answer quality
Troubleshooting retrieval, throttling, timeouts and quota problems

Pacing: what question 55 taught me

180 minutes for 75 questions sounds generous. It isn't, once you start reading carefully.

I took my time on every question. I found the constraint sentence, compared the endings of the options, eliminated on violation, and checked my pick against every condition before moving on. That's what this exam rewards, and I'd do it again. But by question 55 I had about 30 minutes left for the last 20 questions, and those had to move a lot faster than the first 55.

What I'd do differently is set checkpoints before starting:

Time	Aim to be at
55 minutes	Question 25
110 minutes	Question 50
110 to 180 minutes	The last 25 questions and a short review

That keeps the careful reading and still leaves room at the end. If one question is eating your time, make your best elimination, flag it and move on. A blank scores the same as a wrong answer, and a guess costs nothing.

Read slowly, but keep an eye on the clock.

If you're preparing for AIP-C01 and want to compare notes, leave a comment.

My Eval Passed Because the Model Had Already Seen the Answers

Chidozie Uzoegwu — Thu, 01 Oct 2026 14:37:34 +0000

Trap one: the exam was in the textbook

The first time my eval battery came back green, I was pleased with myself. The result was worthless, and it took me a while to work out why.

An eval battery is the test suite that decides whether a language model is good enough to put in front of real users. Mine holds 50 test cases, grouped into 23 categories that each cover one behaviour the model has to get right. It runs before anything ships and reports a flat pass or fail rather than a score.

Here is what one of those test cases looks like. The rows in my battery carry product specifics, so this is the same schema rewritten in a neutral domain:

{
  "id": "B1",
  "category": "B",
  "category_name": "Refuses to invent a version number",
  "source": "which version of the SDK added retry support?",
  "pass_criterion": "at least one draft says it does not know the version rather than guessing one",
  "len_cap": 180,
  "emoji_policy": "none"
}

The pass criterion is prose rather than a pattern. Code is good at structure and bad at judgement, and I have the scar tissue to prove it. Every time I have tried to express a quality rule as a pattern match, the check ended up rejecting work that was actually fine.

Now look at that source field again, because that is where I went wrong.

I had built my training data by rewriting the very source material that was in the battery. Side by side, the problem is hard to miss:

// training pair
{"source": "which version of the SDK added retry support?",
 "target": "I do not know the exact version. Check the changelog rather than trusting me on it."}

// battery row, same source string
{"id": "B1",
 "source": "which version of the SDK added retry support?",
 "pass_criterion": "at least one draft says it does not know the version rather than guessing one"}

The model had seen that exact input during training with a good answer attached. When the battery asked it back, it was not generalising. It was reciting.

This is the worst class of bug in an eval, because it fails silently in the direction of good news. A broken eval that reports failure gets fixed the same afternoon. A broken eval that reports success gets celebrated and shipped.

The fix went into the dataset builder rather than onto a checklist, because a checklist is a thing you forget at 2am. It normalises and tokenises every source, then drops any training pair whose source matches a test source:

# 1. exact: drop any training pair whose source is also a test source
train, dropped = exclude_test_sources(train, test_sources)

# 2. fuzzy: catch the paraphrases that survived step 1
leaks = [
    pair for pair in train
    if max(jaccard(tokens(pair.source), tokens(t)) for t in test_sources) >= 0.6
]

Exact matching was not enough on its own, because a lightly edited restatement of a test input is still a memorised test input. Step two tokenises both sides, takes the highest word overlap against any test source, and flags anything at 60% or above:

training: "what version of the SDK got retry support?"
battery:  "which version of the SDK added retry support?"

shared words: version, of, the, sdk, retry, support   = 6
total unique words across both                        = 10
jaccard = 6 / 10 = 0.6   ->  flagged

Two questions sharing 60% of their words are, in practice, the same question. That is where the threshold came from.

Both results now print in the data audit report as hard checks, with the required value stated next to them:

battery fuzzy leaks (j>=.6) : 0  (must be 0)
proven and battery (exact)  : 0  (must be 0, hold-out disjoint)

That is my own output, so the names are mine: "battery" is the test set and "proven" is one of my two training pools. Three consecutive dataset builds have reported 0 on both. That is the only reason I trust anything the battery says now.

One corollary came out of this that cost me a second time. A baseline is only valid against the identical battery version. Change the exam and every previous score becomes uncomparable, so the incumbent model has to be re-run on the new exam before anyone claims an improvement. There is a pair of scores in my own notes that I cannot use in this article for exactly that reason. One was recorded three weeks before a battery change, the other after it, and I have no way to prove they measure the same thing.

Trap two: one run is not a measurement

With the leak closed I finally had an exam the model had genuinely not seen. So I ran it.

Before the number means anything, here is what one point costs. Each row samples the model six times. All six drafts go to a judge model with that row's criterion and a fixed rubric, and the judge returns scores, a ranking, a pass boolean, and a list of which drafts tripped each structural rule. The row is then scored against 12 assertions, with the specifics swapped out again:

checks = {
    "criterion_met":     bool(v.get("pass", False)),
    "no_tell_a_top2":    not _hit("tell_a_drafts"),
    "no_tell_b_top2":    not _hit("tell_b_drafts"),
    "no_tell_c_top2":    not _hit("tell_c_drafts"),
    # ...six more of the same shape, one per failure mode
    "len_ok":            median_len <= len_cap,
    "emoji_ok":          emoji_ok,
}

A row scores a point only if all 12 pass:

return all(checks.values()), checks, ranking, median_len, len_cap

And the ship gate is one line:

all_pass = all(p for _, p in row_results)

There is no partial credit and no passing percentage. That was deliberate. A scorecard reading 44 out of 50 invites you to round it up to fine, whereas a flat fail makes someone open it and say out loud which rows they are willing to accept.

One more detail matters for everything that follows. The judge runs at temperature 0 and sampling runs at 0.9, so what comes next is the model's own variance rather than a wobbly judge.

I ran the battery three times against the same model, on consecutive days, with nothing changed between runs:

Run	Score
1	31 / 50
2	21 / 50
3	31 / 50

Ten rows of spread. Twenty percent of the battery, on a model nobody touched.

That matters because of what it does to comparisons. If a model can swing ten rows against itself, a ten-row gap between two different models means nothing on its own. I could have taken the worst run of an old model and the best run of a new one and reported a ten-row improvement in complete good faith. Reverse the pairing and the same two models show a ten-row regression. Neither number would have been real.

So, precisely: three runs of one model, 50 rows, six samples per row. Enough to show that single-run comparisons on this battery are unsafe. Not enough to say how unsafe.

Trap three: a steady total can hide a moving model

So I started running replicates. And a newer model looked like the problem had gone away:

Run	Score
1	27 / 50
2	28 / 50
3	27 / 50

One row of spread across three runs. I nearly wrote that down as a more consistent model and moved on.

It is tempting to read that table against the previous one. The older model spanned 21 to 31 across its three runs, the newer one 27 to 28. Those ranges overlap, so this battery cannot tell the two models apart, and picking a winner from them would be the exact mistake the last section was about.

What those three runs did let me do was look underneath a total that barely moved. So I read the per-category lines instead. Across those three identical runs, 19 of the 23 categories changed their score at least once. Five of them:

Category	Rows	Run 1	Run 2	Run 3
A	3	0	1	2
C	2	1	2	0
H	2	1	2	0
R	2	1	2	0
W	3	2	1	3

Four categories held still, and two of those four were ones the model failed in all three runs. Reliable failure is still information, and it was the only kind this battery gave me that I could trust without replicating it first.

At row level it is starker. 29 of the 50 rows changed verdict between identical runs. Only 21 rows gave the same answer all three times. The total held steady because the errors cancelled out, not because the model was behaving consistently. Had I run the battery once and shipped on the number, I would have been shipping on a coin flip wearing a lab coat.

That raised an obvious question, so I went and counted which assertion was actually failing on those 29 unstable rows:

Failing check	Share of 48 failures
`len_ok`, the length cap	34, or 71%
the top-two structural checks	11, or 23%
`criterion_met`, the judge's own verdict	1, or 2%
everything else	2, or 4%

A row can trip more than one check at once, so those are failure instances rather than rows.

I had assumed the churn was the judge changing its mind about quality. It was not. It was output length wobbling across runs and crossing a fixed cap. The judgement checks were comparatively stable, and across all 29 unstable rows the judge's own pass decision accounted for exactly one failure.

That is a more useful finding than the one I expected, because length is a knob you set at generation time rather than a property of the model's taste. A mechanical variable was dominating a scorecard I was using to make quality decisions.

This is the part that changed how I read every scorecard since. A stable aggregate is not evidence of a stable system. It can equally be evidence that you are averaging over enough noisy measurements for the noise to cancel out. The number is not wrong. It is answering a question you did not ask.

What I changed

Replicates, always. Three runs minimum, and every row reported as a count out of three rather than as a pass or a fail. A row that passes twice out of three is a different object from one that passes three times out of three, and collapsing both to PASS throws away the only information that mattered.

Length reported separately from quality. Given that the length cap drove 71% of the instability, scoring it alongside the judgement checks let one mechanical variable swamp everything else. It now gets reported on its own line, so a quality regression cannot hide behind a length change or the reverse.

Category sizes that can carry a decision. One category in my battery has 2 rows. Another has 1. When a retrain is aimed squarely at a behaviour that only 2 of 50 rows test, the battery cannot see whether the retrain worked. The signal is smaller than the noise before a single GPU hour is spent.

A human comparison when the battery cannot decide. On one occasion I took the nine rows two models disagreed on, ran each of them three times, and scored the results with the length check excluded. Both models landed on exactly 23 of those 27 results. A blind side by side comparison on real inputs settled it in about twenty minutes. The battery catches regressions on things I have already decided matter. It is not an oracle, and when it returns a tie, the honest reading is that it has no opinion.

A gate with no partial credit. The scorecard reports a flat fail rather than a percentage, which forces someone to name the failing rows and accept them one at a time instead of rounding 44 out of 50 up to fine.

The part worth taking away

Building the battery was the easy half. Plenty of tutorials will show you how to write assertions and wire up a judge.

The hard half was finding out what my battery could not see. It could not see a memorised answer until I checked source overlap in the builder. It could not see its own run to run variance until I ran it three times. It could not see that a rock steady total was hiding 29 changed rows until I stopped reading the total. And I could not see that the churn was mostly length until I counted which assertion was failing, rather than assuming I already knew.

So: run your eval twice before you trust it once. If the two runs disagree, you have learned something more useful than either number.

I Swept All 33 Bedrock Regions So You Don't Have To

Chidozie Uzoegwu — Tue, 08 Sep 2026 18:11:46 +0000

The Amazon Bedrock pricing page publishes a training price for the Meta model you almost certainly are not fine-tuning.

Here is the customization pricing on the page, as of 8 September 2026:

Model	Published training price
Llama 2 Pretrained 13B	$1.49 per 1M tokens
Llama 2 Pretrained 70B	$7.99 per 1M tokens
Cohere Command	$0.004 per 1,000 tokens
Titan Image Generator	$0.005 per image seen
gpt-oss-20b	$80 per training hour
Qwen3 32B	$80 per training hour
Llama 3.1, Llama 3.3	none

Other providers are covered. Two models are even priced by the hour. The only Meta model with a published training price is Llama 2, and the Llama you would actually fine-tune today is not on the list.

I have a fine-tuned Llama 3.3 70B running in production on Bedrock, and a line item on my AWS bill saying it cost real money to train. So the capability exists, it bills, and I could not find its price published anywhere.

A caution while you are here: I have seen third-party guides quote "$0.00799 per 1,000 tokens" as the Llama 3.3 70B training rate. That is the Llama 2 70B figure restated and applied to a different model. If you are budgeting, price it from your own first job rather than from a number someone inferred.

That gap turned out to be the least surprising thing I learned. This article is the map I wish I had before I started: which regions can actually do this, what the documented way around it says versus what happened when I took it, and the one piece of job state that decides whether a stalled job is costing you money or nothing at all.

Everything below is either something I measured or something AWS documents, and I have labelled which. Where the two disagree, you get both.

Part 1: the sweep

I did not want to work from a documentation table. The Bedrock catalogue varies by region more than people expect, and a page describing 33 regions is a hard thing to keep current. So I asked the API directly, in every region the SDK knows about.

import boto3, concurrent.futures as cf
from botocore.config import Config

cfg = Config(connect_timeout=6, read_timeout=12, retries={"max_attempts": 1})
regions = boto3.Session().get_available_regions("bedrock")

def probe(region):
    try:
        client = boto3.client("bedrock", region_name=region, config=cfg)
        models = client.list_foundation_models(
            byCustomizationType="FINE_TUNING"
        )["modelSummaries"]
        ids = sorted({m["modelId"].split(":")[0] for m in models})
        return region, len(ids), ids
    except Exception as e:
        return region, "ERR " + type(e).__name__, []

with cf.ThreadPoolExecutor(max_workers=12) as ex:
    for region, count, ids in sorted(ex.map(probe, regions)):
        print(region, count, any("llama3-3-70b" in i for i in ids))

byCustomizationType="FINE_TUNING" is the important part. It returns only the models you can actually train, not the much longer list you can invoke.

33 regions. Here is what came back.

Result	Regions
Can fine-tune Llama 3.3 70B	1 (us-west-2)
Answered, some tunable models, no Llama	2 (us-east-1, eu-west-2)
Answered, zero tunable models	17
Opt-in regions, not enabled on my account	13

One region can fine-tune Llama 3.3 70B. Out of the 33 the SDK lists, and out of the 20 that gave me a real answer. Not a short list. One.

An honest caveat, because the difference matters. Those 13 are not a fine-tuning answer. They returned UnrecognizedClientException, which is what you get when you call an opt-in region your account has never enabled. I cannot rule them out from this account, and neither can you from yours unless you have opted in. Twenty regions gave a real answer.

The two regions that answered with something other than zero are more interesting than the seventeen that answered zero.

us-west-2, five tunable models:

amazon.titan-embed-image-v1
anthropic.claude-3-haiku-20240307-v1
meta.llama3-1-8b-instruct-v1
meta.llama3-1-70b-instruct-v1
meta.llama3-3-70b-instruct-v1

us-east-1, six tunable models:

amazon.nova-micro-v1
amazon.nova-lite-v1
amazon.nova-pro-v1
amazon.nova-2-lite-v1
amazon.nova-canvas-v1
amazon.titan-embed-image-v1

Read that second list again. In the region most people default to, every model you can fine-tune is an Amazon model. No Llama. No Anthropic. If you want to customise anything that is not Amazon's own, us-east-1 cannot do it.

The console agrees. Opening Create Fine-tuning job in us-east-1 on 8 September 2026, the model picker offers one category, "Serverless model providers", and one provider under it: Amazon. The six models it lists are the same six the API returns.

And eu-west-2, London, where a lot of UK and EU workloads want to sit for latency or residency reasons, has exactly one tunable model. It is not a Llama.

This has been stable. I first ran the sweep on 1 August 2026 and re-ran it on 8 September 2026 before publishing. Same answer both times. That is five weeks, not five years, so re-run it yourself rather than trusting my table. The snippet above takes about a minute.

Part 2: the escape hatch, and what happened when I tried it

The obvious way out is to train the model somewhere else and bring the weights in. Bedrock has Custom Model Import for exactly that, and on paper it fits.

The documentation is clear and it is worth quoting accurately, because my experience differed from it and I want you to have both.

Custom Model Import lists Llama 3.3 among its supported architectures. It states the ceiling in terms of weight size rather than parameter count: under 200GB for text models, with a maximum context length below 128K. It is available in eu-central-1, us-east-1, us-east-2 and us-west-2, which is four regions to native customization's one. AWS has published a walkthrough of importing DeepSeek-R1-Distill-Llama-70B through it.

By that description, importing a fine-tuned Llama 3.3 70B should work. My artifacts were 141GB in bf16, comfortably inside the documented 200GB.

The import failed:

too large to fit on available hardware

That is the whole error. It does not say which limit was hit, whether the constraint was my artifacts, the region, or capacity at that moment. Searching it turns up other people receiving the same generic message on far smaller models, including an 8B, which suggests it is a catch-all rather than a statement about 70B specifically.

So the honest position is narrower than "it cannot be done." The documented path supports the architecture and the size. It did not complete for me, on my artifacts, in my account, on the day I tried, and the error was not specific enough to tell me why. Someone else may well succeed with it, and AWS's own material suggests they do.

What that means practically, if you are planning:

Native customization is the constrained path. One region for Llama 70B, and no choice about it.
Import is the less constrained path on paper, including into eu-central-1, which matters if EU residency is a hard requirement for you. Try it before you conclude anything from my result.
Budget for the possibility that it does not complete, and for an error that will not tell you why.

For my own build, native customization is what worked, so that is where the model lives, and the application runs in a different region from the custom model deployment. To be precise about which half of that was forced: the customization and its deployment had to be in us-west-2. Where the application runs was my choice, and I could move it.

Part 3: InProgress means two completely different things

This is the part I have not found documented anywhere, and it is the one that costs money.

Launch a customization job and poll it. The status reads InProgress.

That tells you nothing useful, because InProgress covers two states that are not remotely the same:

The job is sitting in a capacity queue waiting for a GPU. No trainer has started. You are being charged nothing.
The job is training. You are burning GPU time.

From the outside these look identical. Same status, same API, same console. The only thing that separates them is nested one level down:

job = bedrock.get_model_customization_job(jobIdentifier=job_id)

job["status"]
# 'InProgress'   <- tells you nothing

job["statusDetails"]["trainingDetails"]["status"]
# 'NotStarted'   <- queued, free to stop
# 'InProgress'   <- training, stopping now pays for nothing

statusDetails.trainingDetails.status is the only honest signal. Check it before you stop anything.

Why it matters in cash terms. Over one week of capacity contention I stopped five jobs. Four had never left the queue.

Jobs stopped	Reached the trainer	Billed
4	No	$0.00
1	Yes	$29.89

Four jobs, each killed after many hours, cost nothing at all. Queue time is free. Had I believed the top-level status, I would have assumed all five cost me something and drawn exactly the wrong conclusion about how expensive iteration is.

This is consistent with what AWS documents, once you read it precisely. The guidance on stopping a job says Bedrock charges for the tokens it used to train the model before you stopped it. If training never started, no tokens were used, and there is nothing to charge for. The documentation and the invoice agree. What neither tells you is which of the two states your job is in, and that is the gap trainingDetails fills.

One more thing that trips people up here: validation completing does not mean training has started. Validation proves your S3 permissions and your data schema are fine. It says nothing about whether a GPU has been allocated.

Part 4: the threshold I invented

Here is the mistake, because it is more useful than the finding.

Jobs were stalling in the queue. I went back through my own job history and worked out that the longest queue wait ever followed by successful training was about 11 hours. So I adopted a rule: past 11 hours the job is hung, kill it and relaunch.

That rule was garbage, and it took me a while to see why.

Every job in my history that had waited longer than 11 hours had been killed by me before it could recover. The number did not measure Bedrock's behaviour. It measured my patience. I then used it to justify killing the next job, which fed the same number back into the same conclusion.

The data point that broke it: one job waited 23 hours and 15 minutes in the queue, then trained normally and produced a working model.

An earlier job had been stopped at 17 hours 31 minutes on the strength of a threshold that did not exist. It cost $0.00, because it had never left the queue, but it cost most of a day.

The queue has no published upper bound that I could find. The documented threshold applies to a different state. AWS's troubleshooting guidance says training time runs "between 3-4 hours, up to 24 hours, depending on configuration and traffic", and that you should contact Support if a job has been in Training for more than 24 hours. In Training, not in queue. The phrase "depending on configuration and traffic" is doing real work there: capacity is shared, and the wait before training is not the thing being bounded.

The correct response to a stalled customization job, given that queue time is free and every relaunch goes to the back of the queue: wait. Impatience has a real cost here. Patience has none.

Part 5: the things that will bite you next

Four more, briefly, all learned the expensive way.

The schema is per model family. The limits are not. Llama 3.3 takes the Converse format, with schemaVersion, system[] and messages[]. Llama 3.1 rejects it outright, with ValidationException: Unable to parse S3 file due to invalid data schema/format, and wants flat {"prompt": ..., "completion": ...} instead. Both share the same 16,000 token ceiling, the same 100 to 10,000 record range, the same epoch and learning rate bounds. Matching limits are not matching schemas, and I lost a launch to assuming otherwise.

prompt/completion has no system field. Move from 3.3 to 3.1 and your system prompt has to be prepended into every training record, then reproduced byte for byte at inference. Get one character wrong later and the model quietly degrades, with no error anywhere.

Your real concurrency limit may be smaller than you think. The quota that binds is Custom models with a creating status per account, which on my account is 2. Not the scheduled-customization-jobs quota, which on my account is 10 and looks like the constraint until you hit the other one. Check both in Service Quotas rather than assuming my numbers are the defaults.

Storage is per model per month. On my bill, a retained rollback model comes to about $1.95 a month to sit there. Cheap, and worth it for the rollback, but not free, and easy to accumulate.

The decision tree, compressed

If you are considering fine-tuning a Llama on Bedrock, the honest version is short.

For native customization, the region is decided for you. us-west-2 or nothing, for Llama 70B. If you go that route, design the cross-region call in from the start.
Try Custom Model Import before you accept that. It documents support for Llama 3.3, a 200GB text ceiling and four regions including eu-central-1. Mine failed with a generic hardware error inside the documented limit, so budget for that outcome, but do not take my result as the rule.
Check trainingDetails, never the top-level status, before you stop a job or conclude anything about cost.
When it stalls, wait. Queue time is free, relaunching is not, and any hang threshold you derive from your own history is measuring you.
Read the schema doc for your exact model version, not the family. The limits will match and the format will not.

None of it sits in one place, which is why it is here. The pricing page still says Llama 2.

Don't Fine-Tune. Unless You Can Answer These Three Questions.

Chidozie Uzoegwu — Tue, 25 Aug 2026 19:21:17 +0000

Twice now I have gone looking for a retrain and found the problem somewhere else.

The first time, a third of my output had collapsed into one template. Before spending anything I counted the pattern in my training data. 5 occurrences in 1,610 examples. Then I counted it in the output. 36%. The cause was a single example I had hardcoded into my own prompt months earlier, and the model was copying it.

The second time, quality had gone flat across the board. The system prompt had grown to 224,833 characters, roughly 56,000 tokens on every call, and ninety percent of it was accumulated rules about how the output should sound. The model was doing exactly what I had asked it to do, which was avoid things.

Neither needed new weights.

That is the pattern I would put in front of anyone about to fine-tune. It is the most expensive way to discover you had a prompt bug. About $30 and a five-hour job per run on a 70B model, and the compute is the small part. The real cost is that you now have two variables moving and no clean way to tell which one broke.

So, three questions.

1. Is this a knowledge problem or an identity problem?

Fine-tuning does not teach a model facts. It shifts a distribution.

If the model needs to know things it currently does not, your product details, your documentation, anything that was true last week and false today, that is retrieval. Training knowledge into weights means retraining every time the knowledge changes, which is a subscription you did not mean to sign.

Fine-tuning is for behaviour that stays put. How it writes. What register it falls back to. What it does when the input is ambiguous.

The test I use: if the right answer changes when the underlying data changes, it belongs in retrieval.

2. Have you stripped the prompt back first?

Most requests to fine-tune are prompt bloat wearing a disguise.

Prompts accrete. Something goes wrong, you add a rule, it improves, and nothing ever gets removed, because removing a rule feels like inviting the bug back. Six months later two hundred lines of prohibitions are competing with two lines describing the actual job, and the model attends across all of it.

Before you price a training run, cut the prompt to the smallest thing that states the task, and measure again. If the output improves, you never had a weights problem.

This is uncomfortable, because it means deleting work you were proud of. Do it anyway. You cannot diagnose a system you have only ever added to.

3. Can you actually produce the data?

This is the question that stops people, and it is where most of the effort goes.

Everyone has data. Almost nobody has a thousand or more curated examples of the exact behaviour they want, cleaned, and is prepared to keep cleaning them.

My last build started from 1,418 labelled examples and kept 884 after cleaning. The final training set reached 1,610 rows once other curated sources were added. Here is what got dropped from that base pool, and none of it was optional.

Rhetorical tics I did not want the model to learn: 107 examples dropped for one construction alone, plus smaller counts for two others. If a pattern sits in your training data at any volume, you are teaching it.

Length and register discipline: 302 dropped for being the wrong length for their category. A model trained on mixed lengths produces mixed lengths.

Opener variety: 45 dropped because more than two examples shared the same opening three words. Without that cap the model finds one opening it likes and reuses it everywhere.

Then there was the part I did not plan for.

83 examples were being deleted on every build

The build had a decontamination step. It removed hand-approved examples from one source file, on the assumption they had already been copied into a curated file elsewhere.

Nothing had ever copied them.

83 examples were dropped silently at build time, on every run, for months. They were disproportionately the best ones, the examples carrying concrete specifics rather than general statements, because those were exactly the ones I had approved by hand in the first place.

A row that never arrives leaves no trace. Nothing in a training run tells you what it did not receive.

So there was no error. The row counts looked plausible. The model trained, the evaluation ran, and the result was slightly worse than it should have been in a way no single test could isolate.

I found it by reading the builder line by line.

That is not really a lesson about checking your pipeline. A training pipeline fails quietly by design. Code that drops rows produces a smaller number, and a smaller number still looks like a number. So print what you dropped and why, broken down by reason, on every single build, and then read it.

The bar

Answer all three and fine-tuning is a reasonable thing to do, and it works. Voice in particular is something you can instruct a model toward but cannot make native to it, and that gap is real.

You will also need an evaluation set that can fail, held out and genuinely separate from anything you trained on, and a rollback path that restores the previous model together with its system prompt. Those are their own articles, and they are the difference between shipping a fine-tune and gambling on one.

But answer the three questions first. Two of them cost nothing except honesty.

Every Rule I Added Made It Worse: How Prompt Bloat Killed My Voice

Chidozie Uzoegwu — Fri, 14 Aug 2026 14:11:21 +0000

I measured the system prompt from my old setup last week. It came to 224,833 characters, roughly 56,000 tokens, on every single call.

Ninety percent of that was one block of accumulated rules about how the output should sound.

I did not set out to build that. Nobody does. It grew one reasonable decision at a time, and by the end it was actively making the output worse, which I spent months blaming on the model.

(That prompt-engineering path is no longer what I run in production. It survives in my codebase as a rollback target, which is exactly why I could still assemble it and measure it.)

How a prompt gets to 56,000 tokens

The system generates short-form written content in a specific voice. I started the way most people do: a good frontier model, a carefully written prompt, and iteration.

The loop that got me here is one every prompt engineer will recognise:

Output comes back with a cliché I hate.
Add a line to the prompt banning that cliché.
Output improves. Genuinely, for a while.
New failure appears. Add another rule.
Repeat for six months.

Each individual addition was justified. Each one was a real observed failure with an obvious textual fix. The categories that accumulated:

Banned phrases, specific clichés that kept reappearing
Banned structures, sentence shapes that read as machine-written
Opener bans, a list of ways it was not allowed to start
Register rules, tone and length and formality per content type
Anti-repetition context, the openers and closers used in recent output, injected so it wouldn't repeat itself
Worked examples, retrieved samples showing the target voice
Meta-rules, instructions about which of the above took priority when they conflicted

That last category is the tell. When your prompt needs rules about how to resolve conflicts between its own rules, the prompt has become a program, and nobody is testing it.

Then came the validators

Rules in a prompt are requests, not guarantees. So when a rule failed to hold, I did the obvious thing and enforced it in code afterwards.

That grew into a stack of post-generation checks: regex to catch a banned sentence shape, a stripper for a punctuation habit, a rejection pass for output that restated its input, a vocabulary blocklist, a fallback for when everything got rejected. At one point I shipped seven of these as a single bundle.

Each was a patch for something the prompt couldn't reliably enforce. And here's the part worth sitting with: I was compounding validators to compensate for a weakness in how the model was being steered. Every new validator was evidence the prompt wasn't working, and my response was another validator.

The symptom I misread

Quality got worse as the rules accumulated.

Not dramatically. Gradually. Output got flatter and more cautious, and it started sounding like something written by a committee avoiding mistakes rather than something with a point of view. Which, functionally, is exactly what it was.

My diagnosis at the time was that the model wasn't good enough. I planned a move to a more expensive tier and put a cost estimate together.

That would not have fixed anything. A stronger model given 56,000 tokens of conflicting constraints produces more expensive committee output.

The actual diagnosis

The rules were competing, with each other and with the task.

A language model attends across the whole context. A prompt is not a config file where each line executes independently. It is context the model weighs all at once. Two hundred lines of prohibitions against a two-line description of the actual job means the overwhelming signal is avoid things. The model optimises for whatever you have given it the most evidence you care about, and I had spent six months providing evidence that what I cared about was not breaking rules.

That produces output which breaks no rules and says nothing.

Worse, it doesn't converge. Adding a rule to fix flatness makes the ratio worse. The tool I was using to fix the problem was the thing causing it.

Why prompting couldn't solve this one

Here's the part I had underweighted for months.

You can instruct a model toward a style. You cannot make a style native to it.

Prompting is instruction. It operates on top of what the model already is. Ask for a voice and you get an impression of that voice, held in place by the instruction, degrading the moment the instruction competes with anything else. The rules were load-bearing scaffolding. Remove them and the voice collapsed. Keep them and they crowded out the task.

That is the signature of using the wrong technique. If a behaviour has to be re-specified in full on every single call, it does not belong in the prompt. It belongs in the weights.

What replaced it

I fine-tuned a model on the voice instead. Later articles in this series cover the how, the cost and the evaluation discipline, but the number relevant here is what happened to the prompt.

Approach	Assembled system prompt
Prompt-engineering	224,833 chars, roughly 56,000 tokens
Fine-tuned model	6,805 chars, roughly 1,700 tokens
Difference	About 33x smaller

You can measure your own the same way. Assemble the prompt exactly as your production path does and count it, rather than eyeballing the template file:

prompt = build_system_prompt(sample_input)   # your real assembly path
print(len(prompt), "chars |", len(prompt) // 4, "approx tokens")

Break it down by section too. Mine was 90% one block, which is what told me the problem was concentrated rather than spread evenly:

import re
sections = [s.strip() for s in re.split(r"={40,}", prompt) if s.strip()]
for n, head in sorted(((len(s), s.split("\n")[0][:50]) for s in sections), reverse=True)[:10]:
    print(f"{n:7d}  {head}")

The voice rules didn't move into a shorter prompt. They moved into the model. What is left in the prompt is what a prompt is actually good at: what to write about right now, how long, which mode. Instruction, not identity.

The lesson I'd give my past self

Strip before you add.

When output is wrong, the reflex is to append a rule, because appending is easy and feels like progress. But every rule is a permanent tax, paid on every call, forever, competing with every other rule, and never reviewed. I never once removed a rule. I only ever added.

Three questions I now ask before adding anything to a prompt:

Is this instruction, or identity? Instruction belongs in the prompt. Identity, meaning a voice or a consistent style or a way of reasoning that must hold every time, belongs in the weights.
Is this knowledge? Facts the model needs but cannot be expected to hold belong in retrieval, not permanently in the prompt.
What am I removing to make room? If the answer is nothing, you are not editing. You are accreting.

The rough division that came out of it, and that I would now design around from day one:

Fine-tuning owns identity. Prompting owns instruction. RAG owns knowledge.

Put a job in the wrong one and you will spend months adding rules that make it worse, which is a fairly expensive way to learn where the boundaries are.

0.3% In, 36% Out: Your Fine-Tuned Model Is Copying Your Prompt Examples

Chidozie Uzoegwu — Thu, 13 Aug 2026 17:08:00 +0000

I run a fine-tuned Llama 3.3 70B on Amazon Bedrock. It generates short first-person narrative posts: a setup, a few lines of story, and a closing line that lands the point.

Last week I noticed the closing lines had collapsed into a single template. Not similar in spirit, but literally the same grammatical shape, over and over:

and that's how [someone] [learns/teaches] [something].

Roughly a third of everything the model produced ended that way. The obvious diagnosis was overfitting: the training data must be saturated with that pattern, so the model learned it as the way to end a story. The obvious fix was equally clear. Scrub the training set, retrain, redeploy.

In my setup that runs to about $30 of training compute and a five-hour training job, plus the evaluation pass afterwards before I'd know whether it had worked. Not ruinous, but not free either, and this would have been the third retrain cycle.

Before spending it, I did something I should have done first. I counted.

Step one: count the pattern in your training data

My training file is JSONL, one example per line. The check took about ten seconds:

grep -o -i "that's how" train_v6_2.jsonl | wc -l
# 5

wc -l train_v6_2.jsonl
# 1610

Five occurrences in 1,610 training examples. 0.3%.

Then I counted the same pattern in what the model was actually producing. Every generated post gets persisted, so this was one SQL query over the last 25 rows:

select right(content, 200) as ending
from generated_stories
order by created_at desc
limit 25;

Nine of twenty-five. 36%.

Those two numbers are irreconcilable. Training data at 0.3% cannot produce output at 36%. Fine-tuning shifts a model's distribution, but it does not amplify a pattern by a factor of a hundred. Whatever was driving this, it was not the weights.

Which meant a retrain would have changed nothing, and I'd have concluded my fine-tune was broken.

Step two: read your own prompt

The generation prompt included a line I had written months earlier and never revisited. Structurally it looked like this:

THE CLOSER: lead with a thesis line that names what the story proves, e.g. "and that's how [X] teaches [Y] to expect [Z]."

There it is. I had handed the model one complete, well-formed example of exactly the thing I wanted, and it did the most reasonable thing available to it. It copied the example, template and all.

This is not the model misbehaving. Give a language model a single concrete instance of a pattern and ask it to produce that pattern, and the example becomes the strongest signal in the context window. It will out-compete a 0.3% tendency in the weights every time. I had effectively hardcoded my output and then blamed the training data.

Step three: fix the injection layer, not the model

The fix cost nothing and shipped in an afternoon.

Instead of one hardcoded example, I built a small pool of seven closing lines. The key property is that they are structurally dissimilar: a flat verdict, a two-beat reveal, a consequence line, a prescriptive line, an ironic callback. Critically, none of them uses the shape that was being copied.

My production lines come from a private archive, so here is an illustrative pool in a neutral domain that shows the same variety:

_CLOSER_EXEMPLARS = [
    "The process was the problem.",
    "Nobody had tested it. Everybody had approved it.",
    "We saved an hour that afternoon and spent the next year paying for it.",
    "Write the runbook before you need it. You will not be calm enough later.",
    "So much for the quick fix.",
    "We called it a deadline for four months before anyone admitted it was a guess.",
    "What would you have checked first?",
]

examples = "\n".join(
    f"- {c}" for c in random.sample(_CLOSER_EXEMPLARS, 3)
)

Three are sampled at random per call, plus one explicit instruction: study the function these serve, then write a new one for this story, and do not reuse their wording.

The reasoning is that one example teaches a template, while several dissimilar examples teach a function. If every example looks different but they all achieve the same effect, the only thing left to imitate is the effect.

The result

Measured on live inference against the same deployed model, same theme, no retraining:

Run	Template rate
Before the fix	36% (9 of 25)
After the fix	8% (1 of 12)

Sample sizes are small and I'd want more data before treating 8% as precise. But the direction is unambiguous, and it cost one prompt edit rather than a retrain cycle.

The remaining 8% is, I think, genuinely the weights. That 0.3% is still in there, and a prompt fix can't reach it. That residue is a legitimate item for the next training-data scrub. The difference is that it's now a small, well-understood cleanup rather than a retrain I'd have run on a wrong diagnosis.

The generalizable part

When a fine-tuned model produces something repetitive, there are three possible causes and they have very different price tags:

The prompt is handing it the pattern. Free to fix. Check this first.
The training data is saturated with the pattern. Needs a scrub and a retrain.
The base model has a strong prior. Needs data specifically counteracting it.

The diagnostic that separates them is one line of grep and one SQL query. Compare the rate in your training data to the rate in your output. If the output rate is dramatically higher, the cause sits in your prompt or your retrieval layer, not your weights. If the two rates are similar, it really is a data problem and a retrain is justified.

Two habits came out of this that I'd suggest to anyone running a fine-tune on Bedrock:

Persist your generations. I could only run the "after" comparison because every output was already in a table with a timestamp. Without that history there is no measurement, only impressions, and impressions are what sent me toward a retrain in the first place.
Treat prompt examples as training data. They are, functionally. Anything concrete you put in a prompt as an illustration will get copied, so either give none, or give enough variety that only the underlying function is imitable. One example is the worst of both worlds.

The uncomfortable version of this lesson: the model was fine. I nearly spent money and two days proving that the hard way, and the evidence that it was fine was sitting in a file I could have counted at any point.

My 9-Month Journey to 9 Cloud Certifications: Juggling Work, Deadlines, Projects, and a New Baby

Chidozie Uzoegwu — Fri, 10 Jan 2025 14:13:00 +0000

Certifications earned in chronological order:

AWS Certified Cloud Practitioner (CLF-C01)
Microsoft Azure Fundamentals (AZ-900)
Oracle Foundations Associate (OFA)
AWS Solutions Architect Associate (SAA)
AWS Developer Associate (DVA-C02)
AWS Data Engineer Associate (DEA-C01)
AWS Certified AI Practitioner (AIF-C01)
AWS SysOps Administrator Associate (SOA-C02)
AWS Machine Learning Engineer Associate (MLA-C01)

How It All Started
How did I even end up with 9 certifications when my initial target was just to focus on AWS Solution Architect Associate (SAA), get certified, build some projects, and rest? Oh boy, I was mesmerized and fell in love with cloud knowledge when I started! Solution Architect (SA) sounded interesting, and the concept of building solutions for products, brands, and companies using cloud infrastructure was exhilarating. To be frank, Solution Architect keeps coming up randomly as a highly sought-after skill set, and the job market needs people with such expertise. I'd just be watching random interviews where the presenter would ask, "What do you do for a living?" and the correspondent would say, "I'm a Solution Architect. I do this and that for companies…" The next day, I'd be skimming through LinkedIn or X or other social platforms, and there again, I'd see discussions and blog posts about Solution Architects. Then I'd say to myself, "This is the sign - SA is calling me!"

First Encounter with Cloud Computing
Let's retrace to the very beginning. My first encounter with the cloud was in one of my course modules during my Master of Data Science degree program called "Big Data and Cloud Computing." It amazed me that big data, programming, building solutions, storage, and many other activities could be done seamlessly on the cloud without needing to piece everything together - every service you need is already available on the platform like a one-size-fits-all solution!
The idea fascinated me. I wouldn't have to store data on a hard drive, manually upload or import it into my programming environment, worry about losing data entirely, or deal with different subscriptions and platforms to achieve a solution. The cloud could provide it all as Infrastructure as a Service (IaaS), Platform as a Service (PaaS), and Software as a Service (SaaS) through a variety of services such as compute, storage, databases, machine learning, networking, analytics, and more. "Voilà," I said to myself, "Cloud is the way!"

The Immigration Journey and EB-2 NIW Motivation
Procrastination set in at first as I explored other paths - whether to pursue a Ph.D. immediately or get a job as a Data Scientist. As an immigrant in the UK with an expiring student visa, the visa challenges of finding skilled-worker sponsorship loomed large. I had just started a new family too, and the urge to move to countries where I could have more stability, along with other concerns I won't burden the readers with, occupied my mind rather than getting cloud certified - until early 2024.
In January, a Facebook friend recently shared his success story of obtaining USA EB-1 approval for a green card - a visa category for individuals with extraordinary abilities, outstanding professors, or multinational executives. This meant he would become a permanent resident in the USA while living in the UK on his previously approved Global Talent Visa. I reached out to him and discovered that I qualified for the EB-2 NIW (National Interest Waiver), a category for individuals with advanced degrees or exceptional abilities whose work benefits the U.S. However, I needed to demonstrate how "well-positioned" I was to advance my proposed endeavors. That sparked my motivation to pursue certifications, as they could serve as supporting documents to prove my exceptional abilities.

Starting with the Basics: Cloud Practitioner
Instead of jumping straight into the SAA certification, I needed to understand everything from scratch. Unlike Microsoft's Azure platform, which I had experienced in my Big Data and Cloud Computing module, I learned that AWS was the largest cloud provider with the biggest market share, so I focused on AWS. I started with the fundamentals through the AWS Certified Cloud Practitioner Certification (CLF-C02) from freeCodeCamp.org with instructor Andrew Brown on YouTube. The 14-hour video was comprehensive, but I couldn't follow it structurally as everything was mixed without sections, topics, and structure. So, I opted to subscribe to ExamPro, a platform developed by Andrew Brown, where I could find more structure in my learning and explore the practice exam questions - which are crucial for passing the main exam.
My next decision came upon entering the ExamPro platform. I discovered a course bundle with basic cloud foundation courses from top providers called "Cross-Cloud Fundamentals Package": AWS Certified Cloud Practitioner, Microsoft Azure Fundamentals, and Oracle Foundations Associate. I thought to myself, why not subscribe to this package? After all, it's not a bad idea to learn about diverse cloud concepts from the basics. I went for it at $69 USD and instantly became a serious student.
The Beast Mode Phase
Remember, during this period I was preparing my EB-2 NIW petition and gathering other documents for my application. I needed to speed up with my certifications to include as supporting documents, so my energy and "ginger" was on nitro! I was in beast mode and hyper-determined to succeed. I studied rigorously and mustered the wherewithal to meet my targets. I studied on my way to and from work, in the metro, during my free periods and breaks at work. At home, it was back-to-back studying if I was awake. Scratch that - I was waking up specifically to study, instead!

Foundation Certifications Success
In less than a month, I took and passed the AWS Certified Cloud Practitioner (CLF-C01) exam. Yay! My approach was simple: I covered all topics from Andrew Brown's course module and completed the hands-on practicals, but more importantly, I practiced the sample exam questions extensively, which prepared me for the real exam. Learning the course material alone wasn't enough - I needed to familiarize myself with the AWS exam structure, which played a major role in my success.
Immediately after passing the CLF-C01 exam, I gained momentum like a toddler hungry for its mother's milk and swung into action toward Microsoft Azure Fundamentals (AZ-900). I realized that I had already studied the CLF-C01 material extensively, which largely overlapped with the concepts in AZ-900, with only minor differences in terminology. As a result, I only needed to focus on practicing the AZ-900 exam questions. I took the AZ-900 and passed. Yay! The same happened with Oracle Foundations Associate (OFA). Yay! The OFA was a free exam, unlike the CLF-C01 and AZ-900 that cost about $100 USD each.

Tackling the Solution Architect Associate
Next came the certification that started it all - the SAA. I began researching study materials, and everywhere I turned, I encountered "Stephane Maarek" and success stories from those who had taken his Udemy course. This led me to sign up for Udemy and purchase my first course: "Ultimate AWS Certified Solutions Architect Associate" by Stephane Maarek. It came with some practice questions, but for deeper preparation, I also bought the SAA practice exam by Abhishek Singh and Stephane Maarek, which contained six full practice exams. This gave me plenty of opportunities to familiarize myself with the material and prepare for the exam. Both courses cost just over £28, as I bought them during a Udemy promotional sale. Always watch for those promos!
I prepared intensively for less than a month as my EB-2 NIW petition deadline approached. I passed! Yay! The exam usually costs $150 USD, but since I had passed the CLF-C01, I received a 50% discount on my next qualified exam. So, I paid just $75 USD for the SAA.

The Developer Journey and Life Changes
Upon passing the SAA, wild thoughts began to swirl in my mind: "Perhaps I can go for the AWS Developer certification and then the AWS Certified SysOps Administrator." Another voice chimed in, encouraging, "Why not complete all the associate-level AWS certifications and become more grounded in it?" Yet another joined, cheering me on, "Besides, you'll get half-price for the next exam since you passed the last one!"
I gave in because, truthfully, I had become magnetized by the endless possibilities of cloud knowledge and, like Oliver Twist, yearned for more. I recalled Stephane Maarek's advice from his FAQ about "In which order should you learn?": "Start with AWS Certified Solutions Architect Associate, then move on to AWS Certified Developer Associate and then AWS Certified SysOps Administrator. Afterwards, you can either do AWS Certified Solutions Architect Professional or AWS Certified DevOps Professional, or a specialty certification of your choosing."

The Developer Certification Challenge
So, I continued with AWS Developer (DVA-C02) certification. I purchased both the course and practice exams from Stephane Maarek at £28 each (promotional price - regular price over £60). DVA-C02 proved challenging, possibly due to its different approach and because I had a lot going on as I neared the final submission of my EB-2 NIW application. While preparing for the DVA-C02 certification, I was also preparing to submit my EB-2 application, and in three weeks, I took the DVA-C02 exam and failed. Nay! I was sad but not broken - I get motivated by failure; it sparks me as if I have something to prove. I can be stubborn like that!
The sadness was short-lived, though, as my EB-2 NIW was approved in May. Success! YAY! The next day, I welcomed my baby into this world. YAY! so I took 4 weeks off to give my baby all the attentions needed.

Building Real-World Experience: The Projects Phase
After taking a break, I returned to my studies as if I'd never left. I went back to the drawing board and realized I needed more hands-on experience to build up my profile and portfolio with projects. I began exploring both free and paid hands-on projects.
I started with free projects on YouTube:
From Tech with Lucy:

Launched a website on Amazon S3
Visualized data using Amazon QuickSight From Tiny Technical Tutorials:
Architected and built an end-to-end AWS web application from scratch
Built a full end-to-end web app with 7 AWS services
Created a free static website with AWS Amplify and Amazon S3
Built a React app with Amplify (Gen 1), Cognito, and CI/CD with GitHub Then came the paid projects: From Tech with Lucy - AWS Cloud Projects ($63 USD):
Built a daily task scheduler using Amazon PartyRock
Created an image labels generator using Amazon Rekognition
Developed a text narrator using Amazon Polly
Built a language translation bot using Amazon Lex
Deployed a bucket list tracker application on AWS Amplify From Azeez Salu through aosnote.com ($49.95 USD each):
Host a Dynamic Web App on AWS
Deploy a Dynamic Website on AWS with Terraform To ensure success on my next DVA-C02 attempt, I purchased another course by Jon Bonso. This provided additional learning opportunities, and combined with my hands-on project experience, I passed the exam in June after extra two weeks of study. Yay!

The Data Engineering Challenge
By now, my mission had expanded to completing all associate-level certifications. I wanted to be versatile and not limited to just one area of cloud computing. I aimed to become a full-fledged cloud engineer with comprehensive knowledge. When AWS launched the Data Engineer certification (DEA-C01), it was a perfect fit since data is my area of expertise. I made this my next target before tackling the most challenging one - AWS Certified SysOps Administrator Associate.
I purchased course materials from Frank Kane and Stephane Maarek, along with practice exams from Thomas Hass, both at the promotional price of £28. While studying, I attended the AWS Summit in London, where I met AWS partners and employees and gained first-hand experience with various AWS projects and services. This major event provided great insights into AWS. However, despite my preparation, I failed the DEA-C01 exam. NAY!
What went wrong? Data Engineer was complex, especially as a new certification with limited study materials available. I had approached it too quickly without thoroughly understanding the core concepts of data engineering, which focuses on moving and processing data. Many services appeared similar but differed in operation and approach. I needed to return to basics and take more time to understand the material properly.
For me, as I've mentioned before, failure doesn't weaken or depress me - it makes me more competitive and determined to win. I enjoy the challenge of overcoming failure. So, I purchased additional practice exams for AWS Certified Data Engineer from Abhishek Singh and Stephane Maarek, practiced for two more weeks, and passed! Yay!

The SysOps Administrator Battle
Next came the formidable AWS Certified SysOps Administrator Associate (SOA-C02). This proved to be the most challenging certification of all. Though I nearly gave up, I had to prove I could rise to the challenge. I started with Stephane Maarek's courses and practice exams but failed SOA-C02 twice! Nay! Nay!
I purchased additional materials from Neal Davis and Jon Bonso and continued studying. During this time, AWS launched two certifications in my field of expertise: AWS Certified AI Practitioner (AIF-C01) and AWS Certified Machine Learning Engineer Associate (MLA-C01). This excited me for two reasons: it aligned with my specialty, and AI and ML were advancing rapidly. I decided to use these as a refreshing break from my battle with SOA-C02.

AI and Machine Learning Certifications
AWS offered an "Early Adopter" badge to first-time earners of new certifications, so I tackled AIF-C01 first, purchasing course and practice exam materials from Stephane Maarek and Abhishek Singh. In just under three weeks, I passed - yay!
Next, I returned to finish my showdown with SOA-C02, and this time, it didn't stand a chance. I delivered the knockout punch, mastering monitoring and reporting, high availability and resilience, deployment and provisioning, storage and data management, networking, security and compliance, and cost and performance optimization. Passing SOA-C02 felt like a massive milestone.
Finally came MLA-C01. I purchased materials from Frank Kane and Stephane Maarek for both the course and practice exams, studied for less than a month, and passed - another yay! Although AIF-C01 and MLA-C01 were newly introduced, they turned out to be far less intimidating - my background in Data Science gave me a solid edge, and I happily breezed through both.

Cost Overview and Future Plans
It's worth mentioning that I utilized various AWS exam discounts, including the "Associate Challenge" promotion. These discounts helped significantly with retaking certifications. Generally, foundational certifications cost $100 USD at regular rates, while Associate Certifications cost $150 USD. However, passing one certification earns you a 50% discount on your next exam.
Speaking of next exams, I now have two bonus attempts available for professional certifications - either AWS Certified Solutions Architect Professional, AWS Certified DevOps Professional, or a specialty certification.

Looking Forward
Overall, I've thoroughly enjoyed the journey and experiences gained along the way. The ups and downs have shaped my current position as I continue learning and building. There's still so much to learn both in cloud computing and beyond. The world of Data Science is evolving at lightning speed, and I'm staying current with all the new technologies. What makes these experiences particularly exciting is the integration of AI, ML, and Data with Cloud technologies. I believe both are unstoppable, especially when combined. Currently, I'm working on a project using the latest AWS services - Sagemaker, Glue, Bedrock Agentic AI, Knowledge bases, Lambda Functions, Step Functions, CloudWatch, EC2, and others - as a solution for a brand. AWS continues to push boundaries, ensuring builders, users, and cloud engineers have all the resources they need as a platform-as-a-service. And that's truly the best part.
As I keep learning and building, I'll be sharing my journey along the way! Bye for now and Keep Learning!