A Field Guide to Agentic Engineering
By Matthew Odumosu, Senior Software Engineer
Part One: The Story
For most of my career, I was the one typing. I had late nights, branches with names I still remember, and the quiet pride of a bug fixed at 2 a.m. Writing code was how I proved I belonged in the room.
Before I ever led an agent, I worked as a prompt evaluator for Google. My job was to read model responses, compare them against written criteria, and decide which was better and why. I saw the same patterns over and over: answers that sounded confident and were wrong, answers that missed one instruction in a long request, answers that were right only by accident. That work taught me to read AI output the way a careful reviewer reads a pull request. It also taught me that the quality of the answer usually starts with the quality of the question.
So when AI coding agents arrived, I didn't welcome them. I watched them the way you'd watch a new hire who claims to know the codebase. I read every line, rewrote half of it, and told myself I was being careful. Really, I was afraid. If I stopped typing, I wondered what I would be.
The first lesson came from a small failure. I asked an agent for a fix, and it returned a clean, confident change. The tests passed. It looked like senior work. But it had solved the symptom and broken the rule the code existed to protect. I went back to my request and saw what I had actually sent: one vague line. The agent had done exactly what I asked. I just hadn't asked for what I meant.
That was the moment I understood that an AI can only go as far as the instructions you send it. Prompt engineering stopped sounding like a trick to me. It became clear thinking, written down, with the same care I had learned to apply when evaluating other people's model outputs.
My thinking then changed shape. Before, I thought in functions. Now I think in systems. Which service talks to which? Over what protocol: REST, gRPC, a Kafka topic? Is the call synchronous, so the caller waits, or asynchronous, so the message is fire and forget? What happens when the same message arrives twice? What happens when two requests touch the same row at the same moment? What happens when the downstream service is down for ten seconds?
I started putting those answers into my prompts. I'd tell an agent that a payment callback could arrive more than once, so the handler must be idempotent. I'd tell it that two workers could update the same balance, so it needed a locking strategy, and I'd say which one and why. I'd tell it to avoid N+1 queries, and to make a multi-step money movement all-or-nothing. Those were the things I used to catch in code review. Now I hand them over at the start. The agents didn't invent these concerns. I did, and they executed them.
The work itself also changed. I used to start with a ticket and open an editor. Now I start with the requirement. I connect Claude Code to Azure through MCP, so it can read the work item, the acceptance criteria, and the tasks beneath it. Then I connect Figma through MCP, so the agent can compare those criteria against the design and flag where they disagree. Only after that do I move into plan mode. The agent proposes a plan, I review the architecture in it, and then it starts building.
When it builds, it's not one agent doing everything. It writes the code, then writes automated tests against the acceptance criteria and runs them. A separate agent reviews the pull request. When I need to understand the data, I ask Claude Code to query the database from my terminal, using a read-only account, so I can check my assumptions before I trust them.
Trust came slowly. The first time an agent's change passed review and held up in production, I didn't celebrate. I just noticed I hadn't opened the file. The change that made it real came when a pipeline failed overnight. The watching agent saw the failure, diagnosed it, pushed a fix on a new branch, and opened a pull request with a note explaining what happened. I read the change, checked it against the requirement, and approved it. The fix was ready before I was.
I also learned that the agents forget. Every new session starts with an empty memory. Early on, I'd spend the first twenty minutes re-explaining the project, and then the agent would make the same mistake I had corrected the week before. The fix was a file, which I explain in Part Four. Once the project's knowledge lived in that file, each session started where the last one ended.
Here is what I understood by then. I wasn't writing less engineering. I was doing a different kind of engineering. My job is to decide which systems should exist, how they should talk to each other, which failures are acceptable, and what is safe to ship. The agents handle much of the typing. The thinking they depend on is still mine.
I'm faster now, and more reliable, but speed is not the main point. The main point is that my years of experience now have leverage. Every hard lesson I learned about locks, retries, and bad queries is something I can apply across many tasks at once. AI didn't replace that experience. It multiplied it.
The engineers who will thrive are not the ones who cling to the keyboard, and not the ones who hand over their judgment. They're the ones who can design systems, state requirements clearly, evaluate output critically, and lead a team of tools while staying curious enough to keep learning as the tools change.
Part Two: The Field Guide
This section is for anyone, an intern or an experienced engineer, who wants to work this way. Follow the steps in order. Parts Three through Eight go deeper on the engineering principles, context and cost, handover, documentation, and monitoring.
Figure 1. The senior engineer directs the flow, the agent team executes, and systems of record supply the facts. Gold markers show where a human approves.
Step 0: Adopt the right mindset
- The agent's output reflects your input. Vague requests produce confident, wrong answers.
- You own the decisions: what to build, what is safe, and what to ship.
- Treat every agent result as a pull request from a capable but unfamiliar teammate.
- Evaluate before you accept. Ask what the answer gets wrong, not only whether it runs.
Step 1: Pull the requirement from the source
Connect your work tracker (for example, Azure DevOps or Jira) through MCP. Ask the agent to:
- Retrieve the work item, its acceptance criteria, and its child tasks.
- Restate each criterion as a testable statement.
- List anything ambiguous, missing, or contradictory. Do not proceed until these are resolved.
Step 2: Reconcile the criteria against the design
Connect Figma through MCP. Ask the agent to compare each screen and interaction with the acceptance criteria and report:
- Requirements with no matching design.
- Design elements with no matching requirement.
- Conflicts, such as a validation rule the design doesn't show.
Step 3: Design the system before the code
Before any code is written, make the architecture explicit in the prompt or the plan. Answer these questions:
- Which systems are involved, and who calls whom? Draw it, even as a simple list.
- What protocol and what timing? REST or gRPC for synchronous calls, Kafka or a queue for asynchronous events. Note the timeout for each call.
- What can be retried, and is it idempotent? Any operation that can run twice, such as a webhook, a payment callback, or a message consumer, needs an idempotency key.
-
Where can concurrent writes collide? Name the locking strategy: optimistic locking with a version column, or pessimistic locking such as
SELECT ... FOR UPDATE. Explain why. - What must be all-or-nothing? Name each unit of work that must commit or roll back as one.
- How will data be fetched? Name the N+1 risk. Require batching, joins, or eager loading, and ask the agent to report query counts on the main paths.
- What happens when things fail? Define retry policy, backoff, dead-letter handling, and what the user sees.
- How will we know it works? Name the logs, metrics, and alerts that prove it.
Step 4: Plan, then build
- Ask for a plan in plan mode. Review it against Step 3 before approving.
- Let the build agent work on a branch, never on the main line.
- Ask for small, reviewable commits, each message stating the requirement it serves.
Step 5: Test against the criteria
- Ask an agent to write automated tests that map each acceptance criterion to at least one test.
- Check that the tests assert behavior, not just that the code runs.
- Include tests for the risks from Step 3: duplicate delivery, concurrent updates, partial failure, and retry exhaustion.
Step 6: Check the data from the terminal
- Connect the database to Claude Code through a read-only account.
- Use it to verify assumptions, such as "does this table really contain duplicates?" or "how many queries does this endpoint run?"
- Never give an agent write access to production data. Schema and data changes go through migrations that a human reviews.
Step 7: Review, then merge
- A separate review agent checks the pull request against the requirement and the architecture notes, not only against compilation.
- A human approves the merge.
Step 8: Watch the pipeline
- When a pipeline fails, the agent diagnoses the cause, makes a fix on a new branch, opens a pull request, and explains the change.
- A human reviews and merges it. The agent never merges its own work.
Step 9: Watch the logs
- Ask the agent to monitor logs and surface anomalies: error spikes, slow queries, retry storms, duplicate-key errors.
- It proposes a diagnosis and a fix. A human approves any production change, and every change is recorded.
Step 10: Hand over and document
- Freeze the API contract and hand it to frontend (Part Six).
- Run the acceptance-criteria tests before QA receives the branch (Part Six).
- Generate the release document and architecture diagram, then route them through review (Part Seven).
Part Three: Engineering Principles for Agent-Built Systems
An agent writes what you specify. These are the principles worth specifying. Each one includes a prompt you can adapt and a code sketch you can check the agent's output against.
3.1 Idempotency: doing it twice must equal doing it once
Networks retry. Providers redeliver webhooks. Consumers restart mid-batch. If an operation is not idempotent, every retry is a potential double charge.
The rule: every externally triggered write carries a unique key, such as the provider's transaction reference or a client-generated request ID. The database enforces uniqueness, so the constraint is the real guarantee. Any application-level check is only a fast path.
Prompt fragment:
The payment webhook can be delivered more than once. Use the provider's transaction reference as an idempotency key, stored in a processed_events table with a unique constraint on that column. On a duplicate, return the stored result without applying the balance change again. Do not rely only on a pre-check, since two concurrent deliveries can both pass it.
Sketch (Java):
@Transactional(rollbackFor = Exception.class)
public Result handleWebhook(WebhookEvent event) {
Optional<ProcessedEvent> prior = processedRepo.findByRef(event.getRef());
if (prior.isPresent()) {
return prior.get().getResult(); // fast path: duplicate, no side effects
}
try {
processedRepo.saveAndFlush(new ProcessedEvent(event.getRef())); // unique constraint
} catch (DataIntegrityViolationException e) {
return processedRepo.findByRef(event.getRef()).orElseThrow().getResult(); // lost the race
}
Account acct = accountRepo.findById(event.getAccountId()).orElseThrow();
acct.credit(event.getAmount());
Result result = Result.applied();
processedRepo.updateResult(event.getRef(), result);
return result;
}
The unique insert happens inside the same transaction as the balance change. If the balance change fails, the marker rolls back too, so a retry can still succeed.
3.2 Locking: deciding who wins a collision
Two requests reading the same balance and both writing balance - amount is a lost update. Choose a strategy on purpose.
- Optimistic locking (a version column): best when conflicts are rare. The update fails if the version changed, and the caller retries.
-
Pessimistic locking (
SELECT ... FOR UPDATE): best when conflicts are common or the critical section is short and must not fail. - Lock ordering: when a transaction locks more than one row, always lock in a consistent order, such as ascending ID. Otherwise two transfers in opposite directions can deadlock.
Prompt fragment:
Transfers lock both accounts. Acquire the locks in ascending account-ID order to prevent deadlocks between opposite-direction transfers. Use pessimistic locking here because the balance check and debit must not interleave. Add a test that runs two opposite transfers concurrently and asserts no deadlock and no lost update.
3.3 Transactions: all or nothing
A transfer is not one operation. It is a debit, a credit, a ledger entry, and an event. If any of those can succeed while another fails, the books are wrong. Put every step that must agree inside one database transaction, and roll back on any exception.
@Transactional(rollbackFor = Exception.class)
public void transfer(TransferCommand cmd) {
List<Account> locked = accounts.findAllByIdForUpdate(
List.of(cmd.fromId(), cmd.toId()).stream().sorted().toList()); // consistent lock order
Account from = find(locked, cmd.fromId());
Account to = find(locked, cmd.toId());
from.debit(cmd.amount()); // throws InsufficientFunds: whole transfer rolls back
to.credit(cmd.amount());
ledger.save(new LedgerEntry(cmd.ref(), from.getId(), to.getId(), cmd.amount()));
outbox.save(new OutboxEvent("transfer.completed", cmd.ref())); // same transaction
}
The dual-write problem: writing to the database and then publishing to Kafka can fail between the two. The answer is the outbox pattern. The event is written to an outbox table in the same transaction as the business change, and a separate relay reads the table and publishes. Either both the data and the event exist, or neither does.
Across services: a single transaction cannot span two services. Use a saga: a sequence of local transactions, each with a compensating action. If step three fails, the saga runs the compensations for steps two and one, in reverse.
Prompt fragment:
Implement the transfer as a single database transaction covering debit, credit, ledger entry, and outbox event. Any exception must roll back all of them. For the call to the external settlement service, which cannot join our transaction, write the settlement request to the outbox and let the relay send it. Define a compensating action for each step if the saga is later split across services.
3.4 Retry policy and exponential backoff
Retries help with transient faults: timeouts, connection resets, HTTP 503, HTTP 429. They make permanent faults worse. A validation error retried five times is still a validation error, and it costs five times the load.
The rules:
- Retry only errors that are transient. Never retry a 4xx validation or authorization error.
- Cap the number of attempts.
- Increase the delay exponentially: 200 ms, 400 ms, 800 ms, and so on, up to a maximum.
- Add jitter, a random offset, so many clients don't retry in lockstep and create a thundering herd.
- Pair retries with a timeout on every call, so a hung dependency can't hold a thread forever.
- Make the retried operation idempotent (see 3.1). Retries are only safe when repeating them is harmless.
Sketch (Java):
public <T> T withRetry(Supplier<T> op, int maxAttempts) {
long baseDelayMs = 200;
long maxDelayMs = 10_000;
for (int attempt = 1; ; attempt++) {
try {
return op.get();
} catch (TransientException e) { // only transient failures
if (attempt >= maxAttempts) {
throw new RetriesExhaustedException(e); // surface it, don't swallow it
}
long exp = baseDelayMs * (1L << (attempt - 1));
long capped = Math.min(exp, maxDelayMs);
long jittered = ThreadLocalRandom.current().nextLong(capped / 2, capped + 1); // partial jitter
sleep(jittered);
}
}
}
In production, a mature library such as Resilience4j provides retry, backoff, and circuit breaking with tested implementations. Ask the agent to use the library your team already runs, rather than hand-rolling one.
Prompt fragment:
Wrap calls to the settlement API in a retry with exponential backoff: base delay 200 ms, doubling, capped at 10 seconds, with jitter, maximum 5 attempts. Retry only on timeouts, connection failures, HTTP 429, and HTTP 503. Do not retry on other 4xx responses. After the final attempt, record the failure to the dead-letter table with the request payload and the last error, and raise an alert. Each call must have a 3-second timeout.
3.5 Resilience: fail well, not just rarely
- Timeouts on every outbound call, with no exceptions.
- Circuit breaker: after repeated failures, stop calling the dependency for a short period and fail fast. This protects both sides.
- Bulkheads: separate thread pools or connection pools per dependency, so one slow service can't exhaust everything.
- Dead-letter queues: messages that fail after all retries are parked for inspection, not dropped and not retried forever.
- Graceful degradation: decide in advance what the user sees when a dependency is down, such as a "pending" status rather than a crash.
Prompt fragment:
If the fraud-check service fails three calls in a 30-second window, open a circuit breaker for 60 seconds and return "pending review" to the caller instead of blocking the payment. Log every state change. Add tests for closed, open, and half-open transitions.
3.6 N+1 queries: the silent cost
Loading 500 orders, then loading each order's customer in a loop, issues 501 queries. It works in development and collapses in production.
Prompt fragment:
Load the orders for the report with their customers and line items in a fixed number of queries, using a join or a batch fetch keyed on the order IDs. Add a test that asserts the query count stays constant as the number of orders grows from 10 to 1,000.
A query-count assertion is the most reliable guard. Ask the agent to write one for every endpoint on a hot path.
3.7 Explicit architecture notes
Ask the agent to keep a short architecture section in CLAUDE.md (see Part Four) that lists each service, its protocol, its owned data, and its idempotency and locking rules. When a future agent changes one of those services, it reads the rules first.
Part Four: Context, Memory, and Cost
Agents do not remember last week. Each session starts empty. Left alone, the agent re-learns your project every time, burns tokens rediscovering the same facts, and repeats mistakes you already corrected. The fix is a project file that holds what the agent needs to know.
4.1 The project memory file
Create a file named CLAUDE.md in the root of the repository. Claude Code reads it automatically at the start of each session. Keep it in version control so the whole team shares it, and so changes can be reviewed.
A good file is short and specific. Suggested sections:
# Project: Payments Reconciliation
## Purpose
One paragraph: what this system does and for whom.
## Architecture
- ledger-service (Java, Spring Boot): owns account balances. REST inbound. Kafka outbound (topic: transfer.completed).
- webhook-service: receives provider webhooks. Idempotent on provider transaction ref.
- settlement-client: calls settlement API. Retry: exponential backoff, 5 attempts, dead-letter on exhaustion.
## Invariants (never break these)
- Every balance change happens inside a transaction with the ledger entry and outbox event.
- Accounts are locked in ascending ID order.
- No endpoint on the hot path may issue N+1 queries. Query-count tests guard this.
## Commands
- Build: ./gradlew build
- Unit and integration tests: ./gradlew test
- Start locally: docker compose up
## Conventions
- Branch per task. Small commits. Message states the requirement it serves.
- Schema changes via Flyway migrations only.
## Current status
- [x] Idempotency on webhook handler (merged)
- [ ] Circuit breaker on fraud-check (in progress, branch feat/fraud-breaker)
- Open question: should "pending review" be visible to end users? Decision pending.
## Decision log
- 2026-03: Chose optimistic locking for profile updates, pessimistic for balances. Reason: balance checks must not interleave.
Three rules keep it useful:
- Facts, not history. Record decisions and their reasons, not every conversation.
- Never store secrets. No keys, tokens, or connection strings. Point to where they live.
- Update it at the end of each task. Ask the agent to update the status and decision log as part of every pull request. A stale memory file is worse than none.
4.2 Clearing the conversation without losing the work
A long conversation costs more and performs worse. Every turn re-sends the history, so the bill grows, and the model's attention spreads across old, irrelevant detail. The workflow is:
- Finish a unit of work, such as one pull request.
- Ask the agent to update CLAUDE.md with the status, decisions, and any new invariant.
- Run
/clearto start a fresh context. - Begin the next task. The agent reads CLAUDE.md and knows the project again.
You lose the chatter. You keep the knowledge. In a mid-task situation where you don't want to start over, /compact summarizes the history instead of clearing it. Use it deliberately, and check that the summary kept the constraints that matter.
4.3 Token and cost management
Costs come from how much text is sent on each turn, how many turns there are, and which model handles each step. Manage all three.
- Keep the always-loaded context small. CLAUDE.md is loaded every session, so every line costs tokens every time. Keep it to the facts the agent needs on most tasks. Move long material, such as a full API spec or a detailed runbook, into separate files and reference them by path, so an agent reads them only when the task needs them.
- One task per session. Start a fresh session for each unrelated piece of work. Mixing a database migration with a UI change in one conversation drags irrelevant context into both.
- Use subagents for research. When an agent must search a large codebase or read many files, delegate that to a subagent. The subagent does the reading in its own context and returns a short conclusion, so the main conversation stays lean.
- Scope the reading. Tell the agent which files matter. "Read ledger-service/src/main/java/.../TransferService.java and its test" costs far less than "explore the repo."
- Match the model to the task. Use the most capable model for architecture, plan review, and difficult debugging. Use a smaller, faster model for routine edits, formatting, and mechanical refactors. Check current model options and pricing in the product documentation, since both change.
- Watch usage. Check your usage regularly. If a session is growing without progress, stop and restart from CLAUDE.md.
- Avoid pasting large outputs. Logs and test output can be enormous. Ask the agent to filter first, for example by searching for errors, the last 200 lines, or the failing test name, and paste only that.
4.4 Why this matters beyond cost
A well-maintained CLAUDE.md is also documentation for humans. A new engineer or an intern can read it to learn the architecture, the invariants, and the reasons behind past decisions. The file serves the agent, the team, and the audit trail at once.
Part Five: Measuring the Impact
Do not publish numbers you have not measured. Before you start, capture a baseline for a few representative tasks, then compare after. Useful measures:
- Lead time: time from requirement to merged change. Baseline: [your number]. Now: [your number].
- Defect escape rate: bugs found after release, per release. Baseline: [your number]. Now: [your number].
- Pipeline recovery time: time from failure to green build. Baseline: [your number]. Now: [your number].
- Incident response time: time from alert to resolution. Baseline: [your number]. Now: [your number].
- Review turnaround: time a pull request waits for review. Baseline: [your number]. Now: [your number].
- Token cost per merged change: total usage cost divided by merged changes. Baseline: [your number]. Now: [your number].
- Contract changes after handover: breaking changes requested by frontend after the contract was frozen. Baseline: [your number]. Now: [your number].
- QA bounce rate: share of features returned from QA for failures an acceptance test should have caught. Baseline: [your number]. Now: [your number].
- Documentation lag: time from deployment to an approved release document. Baseline: [your number]. Now: [your number].
- Mean time to detect: time from the start of a production problem to the first alert. Baseline: [your number]. Now: [your number].
- Alerts acted on: share of alerts that led to a documented action. Baseline: [your number]. Now: [your number].
Pair each number with a short description of the work it covers. "Reduced recovery time for failed builds from [X] to [Y] across [Z] incidents" is credible. "10x faster" is not.
Part Six: Handover to Frontend and QA
Integration pain usually starts when the backend and frontend disagree about the contract, or when QA learns what the feature does after it is built. Both problems are cheaper to prevent at the start.
6.1 Contract first
Write the API contract before the implementation is finished, and treat it as the agreement between teams. An agent can draft it quickly from the requirement and the architecture notes from Part Three.
The contract should specify:
- Every endpoint, method, and path, with its version (for example,
/v1/transfers). - Request and response schemas, with types, required fields, and value constraints.
- Authentication and required headers, including
Idempotency-Keyon any operation that moves money. - Error responses: each error code, its HTTP status, and when it occurs.
- Pagination, rate limits, and timeouts.
- At least one realistic example for each success and error case.
Prompt fragment:
From the requirement and the architecture notes in CLAUDE.md, draft an OpenAPI 3.1 specification for the transfer endpoints. Include the request and response schemas, the Idempotency-Key header, and every error response with its status code and an example body. Mark fields that are required and state their validation rules. Do not write implementation code yet. Flag any requirement that the contract cannot express, so we can resolve it before the frontend builds against it.
Example fragment (YAML):
paths:
/v1/transfers:
post:
summary: Create a transfer. Safe to retry with the same Idempotency-Key.
parameters:
- name: Idempotency-Key
in: header
required: true
schema: { type: string, maxLength: 64 }
requestBody:
required: true
content:
application/json:
schema:
type: object
required: [fromAccountId, toAccountId, amount, currency]
properties:
fromAccountId: { type: string }
toAccountId: { type: string }
amount: { type: integer, minimum: 1, description: "Minor units (kobo)" }
currency: { type: string, enum: [NGN] }
responses:
"201": { description: Transfer completed }
"409": { description: Insufficient funds or account locked }
"422": { description: Validation failed }
"503": { description: Settlement unavailable, safe to retry }
Once the contract is reviewed and agreed, version it and treat changes as breaking changes. Ask the agent to add contract tests (for example, with Pact or Spring Cloud Contract) so that any implementation drift fails the build instead of reaching the frontend.
6.2 The frontend handover package
Once the contract is frozen, the frontend can start in parallel with the rest of the business logic. The handover package should contain:
- The OpenAPI file, committed to the repository.
- A sandbox or mock server that returns the documented examples, so the frontend can build before the real service is ready.
- A short error-handling guide: which errors the user should see, which ones are retryable, and what each means in plain language.
- The state rules: for example, a transfer may be "pending" for up to a few minutes, and the screen must handle that state.
Prompt fragment:
Generate a frontend handover document from the OpenAPI spec. Include a one-paragraph summary per endpoint, an error-handling table mapping each error code to a user-facing message and a retry rule, and the state transitions of a transfer. Keep it under three pages. Do not change the contract.
6.3 QA from acceptance criteria
QA should not discover the requirement after the feature is built. Acceptance criteria in the work tracker become the test cases, and every feature needs three kinds:
- Positive: the expected path works, such as a valid transfer completing.
- Negative: invalid input is rejected with the right error, such as an amount of zero or a missing currency.
- Edge: the boundaries and failures, such as a balance equal to the amount exactly, a duplicate webhook, two transfers racing for the same account, or settlement timing out.
Here is what that looks like for a transfer criterion:
| Type | Case | Expected result |
|---|---|---|
| Positive | Transfer equal to available balance | 201, balance becomes zero |
| Negative | Amount of 0 | 422, no ledger entry |
| Negative | Insufficient funds | 409, no balance change |
| Edge | Same Idempotency-Key sent twice | Same response, one debit |
| Edge | Two transfers from one account at once | One succeeds, one returns 409 |
| Edge | Settlement times out after three retries | Status "pending", dead-letter entry created |
I have Claude Code pull the test cases from Azure DevOps through MCP, map each case to an automated test, and run the suite before the work goes to QA. The agent reports which cases pass, which fail, and which have no test at all.
Prompt fragment:
Pull the test cases linked to work item [ID] from Azure DevOps. For each case, find or write an automated test that asserts the expected result. Run the full suite. Report a table of case, test name, and result, and list any case that has no test or cannot be automated. Do not mark a case as passing unless its assertion actually ran.
Running these tests before handover is what reduces back and forth. QA receives a branch where the known cases already pass, and their time goes to exploratory and risk-based testing.
6.4 Definition of done
A feature goes to handover only when:
- The contract is frozen and the frontend has its package.
- Every acceptance criterion has a mapped test, and those tests pass.
- The release documentation from Part Seven has been generated and is awaiting review.
- The pull request explains which requirement each change serves.
Part Seven: Documentation and Architecture After Each Deployment
Documentation written from memory after a release is usually late and incomplete. The better approach is to have the agent write it while the details are still in context, then have a human check it.
7.1 The release artifact
After each deployment, the agent produces a release document from the changes it made and the architecture it touched. It should contain:
- What changed: endpoints, data changes, and behavior changes, in plain language.
- Why: a link to the requirement and the decision log entries.
- Architecture: the services involved, the protocols between them, and any change to who calls whom.
- Invariants: rules that must hold, such as idempotency keys, locking order, and transaction boundaries.
- Operations: the metrics, logs, and alerts to watch, and the runbook steps for the known failure modes.
- Rollback: how to revert, including any migration that cannot be undone.
Prompt fragment:
The release for [feature] is deployed. Using the merged pull requests, the updated CLAUDE.md, and the OpenAPI spec, write a release document with these sections: What changed, Why, Architecture (include a Mermaid diagram of the services and protocols), Invariants, Operations, and Rollback. Note anything you could not confirm from the code, rather than guessing.
7.2 High-level architecture as a living diagram
Keep a Mermaid diagram of the system in the repository. Because it is text, it can be versioned and reviewed like code.
flowchart LR
Provider[Payment provider] -->|webhook| WH[webhook-service]
WH -->|idempotent handle| LEDGER[ledger-service]
FE[Web / mobile app] -->|REST /v1| LEDGER
LEDGER -->|outbox relay| KAFKA[(Kafka: transfer.completed)]
LEDGER --> DB[(Postgres)]
SETTLE[settlement-client] -->|retry + backoff| EXT[Settlement API]
KAFKA --> SETTLE
The agent updates the diagram in each release where the architecture changes, and a reviewer confirms it.
7.3 Review, approval, and the team library
Every release document goes through the same path:
- The agent generates the document and opens it as a pull request against the docs folder.
- A human engineer reviews it against the code, not only against the agent's summary.
- A reviewer approves it.
- The approved document is published to the team library (for example, the Azure DevOps wiki or Confluence), and the repository keeps the source.
- Each document is versioned by release, so the history shows what the system looked like at any point.
The library is organized by system, not by sprint. A new team member starts with the architecture overview, reads the invariants for the services they will touch, and then reads the release documents for recent changes. Onboarding becomes a reading path instead of a series of meetings.
Part Eight: Real-Time Monitoring with Grafana Cloud via MCP
Monitoring is most valuable when it catches a problem before customers feel it. Grafana Cloud stores logs (Loki), metrics (Prometheus or Mimir), and traces (Tempo). Connected through MCP, Claude Code can query all three in the same session where it can read the code.
8.1 Setup
- Create a read-only service account in Grafana Cloud with a scoped token. Never give monitoring access write permissions.
- Add the Grafana MCP server to the project configuration, and keep the token in the environment or a secrets manager, not in CLAUDE.md.
- Define the dashboards and alert rules yourself. The agent reads and explains them. It does not set thresholds on its own.
8.2 What to watch
- Error rate: the share of requests returning 5xx, per service and per endpoint.
- Spikes: a sudden rise compared with the same hour last week, not just a fixed threshold.
- Latency: p95 and p99, not averages, since averages hide the slow tail.
- Saturation: database connection pool usage, thread pool queues, and memory.
- Event health: Kafka consumer lag, retry counts, and dead-letter growth.
- Business signals: failed transfers per minute, reconciliation mismatches, and webhook duplicates.
Example queries the agent can run, adapted to your labels:
# 5xx error ratio, per service, over 5 minutes
sum by (app) (rate(http_server_requests_seconds_count{status=~"5.."}[5m]))
/
sum by (app) (rate(http_server_requests_seconds_count[5m]))
# p99 latency per endpoint
histogram_quantile(0.99,
sum by (le, uri) (rate(http_server_requests_seconds_bucket[5m])))
# ERROR lines from the ledger service in the last 5 minutes, grouped by message
sum by (msg) (
count_over_time({app="ledger-service"} |= "ERROR" | json | line_format "{{.msg}}" [5m])
)
8.3 Triage, not just alerting
Alerts tell you something is wrong. The agent helps decide what it is:
- It summarizes the alert, the affected service, and the time window.
- It checks whether a deployment happened shortly before the change.
- It pulls representative error logs and traces, and groups them by cause.
- It proposes a hypothesis and the runbook step that tests it.
- It drafts an incident timeline for the on-call engineer to correct and send.
A human decides the response and applies any production change.
Prompt fragment:
The alert "ledger-service 5xx ratio above 2% for 5 minutes" fired at 14:05. Using Grafana, pull the error ratio, p99 latency, and consumer lag for the ledger service and its Kafka consumers from 13:30 to now. Pull the ERROR logs grouped by message, and any traces for the top error. Check whether a deployment occurred in the 30 minutes before the alert. Report what changed, the likely cause with your confidence level, and the runbook step to test first. Do not change anything.
8.4 Being proactive
- Compare each deployment against its baseline in the first hour. Most regressions show up quickly.
- Track error budgets against your service level objectives, so the team sees a slow burn before it becomes an outage.
- Schedule a recurring health check that reviews the past day's error and latency trends and posts a summary to the team channel.
- Review alert noise each week. An alert that fires without action should be tuned or removed.
8.5 Guardrails
- Read-only tokens, scoped to the services each engineer needs.
- Redact personal data in logs before they reach the agent. Account numbers and customer names should never leave the monitoring platform.
- Human approval for every production change, including rollbacks and configuration changes.
- Every investigation is recorded in the incident timeline, so the next engineer can learn from it.
Part Nine: Guardrails That Keep This Safe
- Least privilege: read-only database accounts for exploration, read-only monitoring tokens, and narrow, scoped tokens for each MCP connection.
- No secrets in prompts or memory files: use the environment or a secrets manager.
- Human approval: for merges, schema changes, rollbacks, and anything that touches production.
- A decision log: record what was decided, by whom, and why. This is how the team learns and how audits work.
Closing Note
Your experience is the reason this works. Anyone can ask an agent to build something. The people who get real results know which lock to use, which retry is dangerous, which operation must be all-or-nothing, which contract the frontend depends on, and which error rate means the business is about to feel it. That judgment is what you bring, and this guide is a way to pass it on.
Before you publish, check these points:
- Evaluation experience: Check what your evaluator agreement lets you disclose, and adjust the wording of Part One to match.
- Production role: The guide says agents propose fixes and humans approve production changes. Confirm that matches your practice, since that line decides what you can claim.
- Story details: Replace the illustrative moments in Part One and the time comparisons with real incidents, numbers, and details.
- Code and query samples: The Java, YAML, PromQL, and LogQL examples are sketches. Check them against your stack, metric names, labels, and team libraries before readers rely on them.
- Numbers: Fill in Part Five only from measurements you have actually taken.
If you share one real incident (what failed, how Grafana surfaced it, what the agent found, and how long resolution took), I can rewrite the opening of Part One and the monitoring section around it.

Top comments (0)