DEV Community: Karthi Mahadevan

Specification: Agentic Docs-as-Code Architecture Pipeline

Karthi Mahadevan — Sun, 04 Oct 2026 21:24:31 +0000

Overview

This specification establishes an automated, agent-driven software architecture documentation ecosystem. The framework integrates strict structural standards, controlled natural language rules, and autonomous AI maintenance patterns into a unified, version-controlled pipeline.

Core Pillars

1. Structural Standard (arc42 & C4 Model)

System Decomposition: All documentation follows the hierarchical arc42 architecture template, ensuring complete coverage from business goals down to deployment views and quality requirements.
Visualizations: The C4 model (Context, Containers, Component, Code) is embedded directly into source files using Mermaid.js or Structurizr DSL to maintain living visual artifacts alongside text.

2. Linguistic Standard (ASD-STE100)

Controlled Vocabulary: All documentation must adhere to ASD-STE100 (Simplified Technical English) principles.
Clarity Constraints: Sentences remain concise, active voice is strictly enforced, and synonyms are eliminated to ensure absolute zero ambiguity for both human readers and machine interpreters.

3. Agentic Maintenance Pattern (Karpathy-Style Workflow)

Living Codebase Model: Documentation is treated as a compiled artifact maintained primarily by AI coding agents rather than manual wiki updates.
Repository Governance: A root-level configuration file (AGENTS.md or CLAUDE.md) acts as the compiler rulebook, dictating formatting, terminology, and structural constraints to any interacting AI agent.

Operational Workflow

Storage: Plain-text documentation sources (AsciiDoc or Markdown) reside in a version-controlled repository structured according to arc42 layouts.
Execution: Architectural changes trigger targeted agentic instructions. Agents read repository guardrails, update affected multi-file documentation blocks, log corresponding Architecture Decision Records (ADRs), and refresh C4 diagrams in a single pass.
Publishing: CI/CD automation compiles the source files into a static architecture portal on every commit.

final quality gate

Karthi Mahadevan — Sun, 04 Oct 2026 12:53:10 +0000

Yes. Step 5 is the final quality gate before we allow the system to generate the project-level architecture document.

There is one important difference from Step 2:

Step 2: Is one repository's extracted architecture trustworthy?
Step 5: Is the combined project architecture trustworthy?

So the complete pipeline is now:

100 Repositories
      ↓
1. Extraction
      ↓
100 Architecture Evidence Models
      ↓
2. Repository Validation
      ↓
100 Validated Models
      ↓
3. Repository Documentation
      ↓
100 Repository Architecture Documents
      ↓
4. Project Aggregation
      ↓
Project Architecture Model
      ↓
5. PROJECT VALIDATION
      ↓
Validated Project Architecture Model
      ↓
Project Documentation

Save this as 05-project-validation.md.

Step 5 — Project Architecture Validation

Purpose

Validate the Project Architecture Model produced by Step 4.

This is the final architecture quality gate before project-level architecture documentation is generated.

The objective is to determine whether the aggregated project architecture is:

accurate
internally consistent
traceable to repository evidence
sufficiently complete
correctly scoped
free from unsupported architectural conclusions

Do not generate the final project architecture documentation.

Do not silently correct the Project Architecture Model.

Identify problems and provide explicit correction requirements.

1. Validation Principle

The project architecture must be derived from validated repository evidence.

The Project Architecture Model must not become more certain than its source evidence.

For example:

```text id="g8a0yh"
Repository A:
Payment Service produces event X.

Repository B:
Inventory Service consumes event X.




supports:



```text id="v7kwba"
Payment Service → event X → Inventory Service

It does not automatically support:

```text id="7o0n3j"
The project implements a formal Saga.




Additional evidence is required for that conclusion.

---

# 2. Final Validation Status

Return one status:



```text id="26l0xi"
PASS
PASS_WITH_WARNINGS
FAIL

PASS

The Project Architecture Model is sufficiently reliable for project documentation.

PASS_WITH_WARNINGS

The model is usable, but known limitations or unresolved areas must be carried into the final documentation.

FAIL

The model contains significant contradictions, unsupported relationships, identity problems, or architecture claims that could make the project documentation misleading.

3. Repository Coverage Validation

Confirm that the aggregation includes all intended repositories.

Compare:

expected repository inventory
supplied repository models
aggregated repositories

Identify:

missing repositories
duplicate repositories
repositories with failed validation
repositories with incomplete extraction
repositories whose purpose is unresolved

For every missing repository record:

Repository:
Expected:
Present:
Reason:
Impact:

Do not assume a missing repository is irrelevant.

4. Source Traceability Validation

Every significant project-level fact must be traceable to repository-level evidence.

For each important relationship identify:

Project Fact
    ↓
Repository Evidence
    ↓
Source Location

Example:

Project relationship:
Order Service produces order.created.

Supporting repository:
order-service

Source evidence:
src/events/OrderCreatedPublisher.java

Classification:
Verified

Flag project-level facts that have no identifiable source evidence.

5. Entity Identity Validation

Validate the canonical entity model.

Check for:

duplicate services
duplicate APIs
duplicate topics
duplicate databases
duplicate external systems
incorrect entity merges
entities incorrectly split into multiple identities

Examples:

```text id="7suw6e"
PaymentService
payment-service
Payment Service




Determine whether these represent:

* the same service
* different services
* unresolved identity

Do not merge entities only because their names are similar.

---

# 6. Repository Boundary Validation

Confirm that every service, component and dependency retains its originating repository.

Check:



```text id="h8k3vv"
Repository
    ↓
Service
    ↓
Component

Flag entities whose ownership cannot be established.

This is important because repository ownership is required for future maintenance and architecture analysis.

7. Service Dependency Validation

Validate every service-to-service relationship.

For each:

```text id="b0r7me"
Source Service
↓
Interaction
↓
Target Service




verify:

* source exists
* target exists
* interaction is supported
* technology is correct
* direction is correct
* evidence exists
* relationship is not duplicated incorrectly

Classify relationships:

* `Verified`
* `Inferred`
* `Unknown`

---

# 8. API Relationship Validation

Validate project-wide API relationships.

For each API:



```text id="q6v8ta"
Provider
    ↓
API
    ↓
Consumer

Check:

provider exists
API exists
consumer exists
endpoint matches
protocol matches
direction matches
evidence exists

Where only one side is known, do not invent the other side.

Use:

Provider confirmed; consumer unknown

or:

Consumer confirmed; provider unknown

where appropriate.

9. Messaging Architecture Validation

Perform a detailed validation of:

```text id="f9ntpw"
Producer
↓
Event
↓
Topic / Queue
↓
Consumer




For each relationship verify:

* producer
* event/message
* topic/queue
* consumer
* technology
* schema where available
* evidence

Check for:

### Orphaned producers

Producer exists but no consumer is identified.

### Orphaned consumers

Consumer exists but no producer is identified.

### Topic conflicts

Different repositories define the same topic differently.

### Event conflicts

The same event name appears with incompatible structures.

### Ownership conflicts

Different repositories appear to own the same event/topic.

### External boundaries

The missing producer/consumer may be an external system.

Do not automatically classify an orphan as an error.

---

# 10. Contract Consistency Validation

Compare contracts across repositories.

Check:

* API paths
* HTTP methods
* request structures
* response structures
* event names
* event schemas
* topic names
* queue names
* message fields
* versioning

Look for mismatches.

Example:



```text id="z9of6g"
Producer:
customerId

Consumer:
customer_id

Record:

Contract inconsistency

Do not decide that the consumer is wrong without evidence.

11. Data Architecture Validation

Validate project-wide data relationships.

For every important datastore:

```text id="1y8e5a"
Service
↓
Read / Write
↓
Database




Check:

* datastore identity
* technology
* owning repository
* services accessing it
* read/write behaviour
* evidence

Identify shared databases.

Do not automatically classify shared database access as a defect.

Record it as an architecture characteristic.

---

# 12. External System Validation

Validate external-system relationships.

Check whether systems classified as external are actually:

* external systems
* internal project services
* shared platform services
* third-party services
* unresolved

Check:

* interaction direction
* protocol
* service relationship
* authentication
* evidence

Do not infer external ownership from names alone.

---

# 13. End-to-End Flow Validation

Validate each reconstructed cross-service flow.

For every flow check:



```text id="mt8d47"
Trigger
   ↓
Service
   ↓
Interaction
   ↓
Service
   ↓
Persistence / Messaging
   ↓
Output

Verify every transition.

Identify whether each interaction is:

synchronous
asynchronous
scheduled
mixed

Flag flows where:

a transition has no evidence
service order is incorrect
messaging direction is incorrect
a synchronous interaction is shown as asynchronous
an asynchronous interaction is shown as synchronous

14. Architecture Pattern Validation

This is a major quality gate.

Do not accept project-level architecture patterns simply because several repositories use related technologies.

Evaluate each pattern independently.

Event-Driven Architecture

Require evidence of meaningful event-driven integration across the project.

Kafka alone is insufficient.

Saga

Require evidence of:

distributed business transaction
multiple participating services
state progression
orchestration or choreography
compensation/recovery where required

Multiple Kafka consumers are insufficient.

Saga Choreography

Require evidence that services react to events and independently determine subsequent actions.

Saga Orchestration

Require evidence of a central coordinator controlling the distributed transaction.

Outbox

Require evidence of:

business state persistence
event persistence
transactional coupling
relay/publisher

CQRS

Require meaningful separation of command and query responsibilities.

API Gateway

Require evidence of a gateway serving as a controlled entry point to multiple services.

Circuit Breaker

Require actual implementation or configuration.

Idempotent Consumer

Require evidence of duplicate detection or idempotent processing.

For each project-level pattern produce:

Pattern	Evidence	Supporting Repositories	Scope	Confidence	Result

Possible results:

Confirmed
Possible
Not established

15. Project Architecture Scope Validation

Do not allow local architecture characteristics to become project-wide claims without sufficient evidence.

Example:

Repository A:

```text id="3f3p5b"
Uses Hexagonal Architecture.




Valid project conclusion:



```text id="4ch0ms"
Payment Service uses a Hexagonal Architecture approach.

Invalid project conclusion:

```text id="4zyw2k"
The entire project uses Hexagonal Architecture.




unless other evidence supports project-wide adoption.

Record the correct scope:

* repository
* service
* subsystem
* project-wide

---

# 16. Business Capability Validation

Validate business capability groupings.

Check whether each service-to-capability mapping is supported by:

* repository documentation
* source naming
* domain concepts
* APIs
* events
* business documentation

Do not infer a business capability solely from a service name when the meaning is uncertain.

Use:

`Business capability not established`

where necessary.

---

# 17. Project Architecture Pattern Consistency

Check whether different repositories claim incompatible architectural patterns.

Examples:



```text id="q21p1k"
Repository A:
event-driven

Repository B:
synchronous REST

This is not automatically a contradiction.

The project may intentionally use both.

Instead determine whether the project architecture is:

```text id="l1b8fi"
Predominantly synchronous
Predominantly asynchronous
Mixed synchronous/asynchronous
Unknown




Similarly, different services may legitimately use different internal architecture patterns.

Do not force uniformity where the evidence shows architectural diversity.

---

# 18. Dependency Graph Integrity

Validate the complete project dependency graph.

Check:

* every source exists
* every target exists
* every relationship has evidence
* no impossible cycles are introduced
* duplicate relationships are consolidated
* external nodes are clearly identified
* unresolved nodes remain visible

Do not assume a cycle is an error.

A cycle may be a legitimate architecture characteristic.

Record it for architectural analysis.

---

# 19. High-Centrality Service Validation

Identify services that have many dependencies.

Examples:



```text id="k9q1l3"
Service A
    ↓
12 downstream dependencies

These services may be architecturally significant.

Do not automatically classify them as problems.

Record:

Observation:
Service A has a high number of incoming/outgoing relationships.

The final documentation may use this information to explain architectural concentration or coupling.

20. Shared Resource Validation

Identify shared resources such as:

databases
queues
topics
APIs
shared libraries
configuration
infrastructure services

For each record:

```text id="6k7grw"
Resource
Consumers
Producers
Repositories
Evidence
Confidence




Look for:

* shared database access
* shared topic ownership
* shared infrastructure dependencies
* shared configuration dependencies

Do not label sharing as good or bad without further architectural analysis.

---

# 21. Contradiction Resolution

Review every contradiction discovered during Step 4.

For each:



```text id="m0v0t4"
Contradiction:
...

Evidence A:
...

Evidence B:
...

Resolution:
Resolved / Unresolved

Reason:
...

Impact:
...

Possible resolution outcomes:

Resolved

Evidence clearly supports one interpretation.

Unresolved

Evidence does not allow a reliable conclusion.

External Information Required

Repository evidence is insufficient.

Never resolve a contradiction merely because one interpretation appears more likely.

22. Missing Information Validation

Review all project unknowns.

Classify them:

Acceptable Unknown
Documentation Gap
Architecture Gap
External Information Required
Critical Unknown

Examples:

```text id="0c3sg4"
Business ownership unknown.

Runtime production topology unknown.

External system contract unavailable.

Topic ownership unresolved.

Disaster recovery architecture not present in repositories.




Do not remove unknowns merely to make the project architecture appear complete.

---

# 23. Architecture Completeness Check

Determine whether the Project Architecture Model contains sufficient information to explain:

### System

* what the project contains
* major services
* external systems

### Application

* service interactions
* APIs
* events
* databases
* major flows

### Platform

* deployment
* infrastructure
* networking where known
* security
* observability

### Operational

* resilience
* failure handling
* deployment mechanisms

### Cross-cutting

* major architecture patterns
* dependencies
* shared resources

If an area is not represented, classify it as:

* not present
* not established
* missing

---

# 24. Evidence Coverage Score

Calculate an evidence coverage assessment.

Classify significant project architecture facts as:

* directly verified
* inferred
* unknown

Report:



```text id="nyy7s9"
Verified:
...

Inferred:
...

Unknown:
...

Do not create a misleading numerical score unless the evidence model supports reliable calculation.

The purpose is to communicate confidence, not create false precision.

25. Final Hallucination Check

Ask:

Could any project-level conclusion have been produced from generic architecture knowledge rather than repository evidence?

Review especially:

business architecture
architecture patterns
scalability
availability
resilience
security
data ownership
system ownership
production topology
disaster recovery
compliance

Remove or downgrade unsupported conclusions.

26. Project Documentation Readiness

The Project Architecture Model is ready for final documentation only when:

all intended repositories are accounted for
repository identities are stable
major services are identified
significant relationships have evidence
API relationships are validated
messaging relationships are validated
database relationships are validated
cross-service flows are validated
contradictions are resolved or explicitly recorded
project-level patterns are evidence-based
business capability claims are appropriately scoped
unknowns are preserved
project-wide claims are not derived from local evidence without justification

27. Final Findings

Produce a findings table.

ID	Severity	Category	Finding	Evidence	Required Action

Severity:

Critical
High
Medium
Low
Informational

28. Final Decision

Return:

```text id="5p9m1h"
PROJECT VALIDATION STATUS:
PASS




or:



```text id="8o5l2b"
PROJECT VALIDATION STATUS:
PASS_WITH_WARNINGS

or:

```text id="0tvjbb"
PROJECT VALIDATION STATUS:
FAIL




### PASS

The Project Architecture Model is ready for project-level documentation.

### PASS_WITH_WARNINGS

The model can be documented, but the documented limitations must remain visible.

### FAIL

The model must return to Step 4 for correction.

---

# 29. Output on PASS

When validation passes, produce:

1. Validated Project Architecture Model
2. Confirmed project dependency graph
3. Confirmed cross-service flows
4. Confirmed architecture patterns
5. Known contradictions
6. Known unknowns
7. Validation findings
8. Documentation readiness status

Do not generate the final project architecture document.

The next stage will consume this validated Project Architecture Model and generate the project-level architecture documentation.

## Now the five stages are complete

You have effectively designed a **documentation engineering pipeline**:



```text
┌─────────────────────────────────────────────┐
│              SOURCE REPOSITORIES            │
│                                             │
│ Repo 1 ... Repo 2 ... Repo 3 ... Repo 100 │
└──────────────────────┬──────────────────────┘
                       │
                       ▼
             ┌───────────────────┐
             │ 1. EXTRACTION     │
             │                   │
             │ What exists?      │
             └─────────┬─────────┘
                       │
                       ▼
              Architecture
              Evidence Models
                       │
                       ▼
             ┌───────────────────┐
             │ 2. VALIDATION     │
             │                   │
             │ Can we trust it?  │
             └─────────┬─────────┘
                       │
                       ▼
             Validated Repo Models
                       │
                       ▼
             ┌───────────────────┐
             │ 3. DOCUMENTATION  │
             │                   │
             │ Explain it        │
             └─────────┬─────────┘
                       │
                       ▼
              100 Repository Docs
                       │
                       ▼
             ┌───────────────────┐
             │ 4. AGGREGATION    │
             │                   │
             │ How do they       │
             │ fit together?     │
             └─────────┬─────────┘
                       │
                       ▼
             Project Architecture
                    Model
                       │
                       ▼
             ┌───────────────────┐
             │ 5. VALIDATION     │
             │                   │
             │ Can we trust the  │
             │ whole picture?    │
             └─────────┬─────────┘
                       │
                       ▼
            VALIDATED PROJECT MODEL
                       │
                       ▼
             PROJECT DOCUMENTATION

One thing I would change from our original thinking

We now have five core prompts, but I would not call these five agents yet.

Think of them as five specialised stages.

Later, if you automate this, you could have:

Extraction Agent
       ↓
Validation Agent
       ↓
Documentation Agent
       ↓
Aggregation Agent
       ↓
Project Validation Agent
       ↓
Project Documentation Agent

Notice that there is actually a sixth prompt hiding at the end:

Project Documentation Generation.

We have deliberately not written that yet.

That should be the final prompt because now we know exactly what its input will be:

Validated Project Architecture Model

rather than 100 repositories or 100 Markdown documents.

That is the cleanest endpoint for this design.

Step 4 — Project Architecture Aggregation

Karthi Mahadevan — Sun, 04 Oct 2026 12:48:54 +0000

Yes. Now we move to Step 4 — Aggregation.

This is the step that turns:

Repository A → validated architecture
Repository B → validated architecture
Repository C → validated architecture
...
Repository 100 → validated architecture

into:

              100 validated repository models
                         ↓
                 STEP 4 — AGGREGATION
                         ↓
              Project Architecture Model

The key point:

Step 4 does not yet write the final project documentation.

It builds the combined project-level model first.

That model becomes the input to Step 5.

Save this as 04-project-aggregation.md.

Step 4 — Project Architecture Aggregation

Purpose

Combine validated Architecture Evidence Models from multiple repositories into a single Project Architecture Model.

This is an aggregation and cross-repository analysis task.

Do not generate the final project architecture documentation.

Do not simply concatenate repository documents.

Do not assume that similarly named components are the same system component.

The objective is to discover and represent the architecture of the entire project from validated repository-level evidence.

1. Inputs

The primary inputs are validated Architecture Evidence Models from multiple repositories.

Each repository model should contain:

repository identity
application/service identity
components
APIs
processing flows
events
topics
queues
databases
external systems
dependencies
deployment information
security
observability
resilience
architecture pattern evidence
unknowns
source evidence

Repository documentation may be used as a supporting input.

The validated Architecture Evidence Models remain the authoritative source.

2. Core Rules

Rule 1 — Do not assume relationships

A relationship between two repositories must be supported by evidence.

For example:

Repository A:
produces order.created

Repository B:
consumes order.created

This provides evidence for:

A → order.created → B

Do not create the relationship merely because both repositories belong to the same project.

Rule 2 — Do not merge entities automatically

These may or may not be the same:

PaymentService
payment-service
Payment Service
payments-api

Determine identity using evidence.

If identity cannot be established:

Identity unresolved

Do not silently merge them.

Rule 3 — Preserve repository boundaries

Maintain the relationship between:

Project
 ├── Repository
 │     └── Service
 │           └── Component

Do not lose repository ownership when creating the project model.

Rule 4 — Distinguish local and project-level facts

A repository may establish:

Payment Service consumes order.created.

The project aggregation may establish:

Order Service produces order.created.
Payment Service consumes order.created.

The second statement is a project-level relationship created by combining evidence.

Make that distinction explicit.

3. Repository Inventory

Create a project inventory.

For each repository record:

Repository	Application	Responsibility	Technology	Deployment	Status

Also identify:

repositories with multiple services
repositories containing shared libraries
infrastructure-only repositories
configuration repositories
deployment repositories
repositories whose role is unclear

4. Canonical Entity Model

Create canonical identities for project-level entities.

At minimum:

Repository
Service
Component
API
Event
Topic
Queue
Database
ExternalSystem
DeploymentUnit

For each entity record:

Canonical ID
Display Name
Entity Type
Source Repository
Evidence
Identity Confidence

Example:

canonical_id: service.payment
display_name: Payment Service
type: Service
source_repository: payment-service
identity_confidence: High

Do not merge entities without sufficient evidence.

5. Repository-to-Service Mapping

Build the relationship:

Repository → Application/Service

Identify:

one repository / one service
one repository / multiple services
shared libraries
infrastructure repositories
unknown repository purpose

Flag ambiguous cases.

6. Project Dependency Graph

Construct the project-wide dependency graph.

Represent relationships as:

Source
    ↓
Relationship
    ↓
Target

Examples:

Order Service
    ↓ produces
order.created
    ↓ consumed by
Payment Service

Payment Service
    ↓ calls
Customer API

Order Service
    ↓ writes
Order Database

For every relationship record:

Source	Relationship	Target	Technology	Evidence	Confidence

7. Cross-Repository API Relationships

Identify APIs where one repository provides an API and another repository consumes it.

Construct:

Provider
    ↓
API
    ↓
Consumer

Validate using evidence from both sides where possible.

Example:

Customer Service
    ↓
GET /customers/{id}
    ↓
Payment Service

Classify the relationship:

Verified — both sides
Verified — provider only
Verified — consumer only
Inferred
Unknown

Do not assume an API consumer merely because a URL resembles the provider's endpoint.

8. Cross-Repository Messaging Relationships

This is one of the most important aggregation tasks.

Construct:

Producer
    ↓
Event
    ↓
Topic / Queue
    ↓
Consumer

For every event/topic/queue identify:

producer
consumer
repository
service
technology
topic/queue
event/message
direction
evidence
confidence

Example:

Order Service
    ↓
order.created
    ↓
Kafka topic: orders
    ↓
Payment Service
    ↓
Inventory Service

Flag:

producer with no consumer
consumer with no identified producer
topic with conflicting definitions
different schemas for the same event
same topic name used with incompatible semantics

Do not automatically treat an orphan as an error.

It may represent an external producer or consumer.

9. Database and Data Ownership

Identify project-wide data stores.

For each database:

consuming services
producing/writing services
reading services
repository
technology
entities/tables where available
evidence

Look for shared database access.

Identify relationships such as:

Service A → writes → Database X
Service B → reads → Database X

Do not automatically conclude that this is a design problem.

Record it as an architecture observation for later analysis.

10. External System Map

Create a project-wide map of external systems.

For each external system identify:

interacting services
interaction type
protocol
direction
authentication
purpose
evidence
confidence

Example:

External Customer System
       ↑
       │ REST
       │
Customer Service
       │
       │ REST
       ↓
Payment Service

Determine whether an external system is:

genuinely external
another project service
an internal shared platform
unresolved

Do not assume based on naming alone.

11. Event and Message Catalogue

Create a project-wide catalogue.

Event/Message	Topic/Queue	Producer	Consumers	Schema	Evidence	Confidence

Identify:

duplicate event names
duplicate topic names
conflicting schemas
multiple producers
multiple consumers
unused topics
externally owned events

Do not assume two similarly named events are identical.

12. API Catalogue

Create a project-wide API catalogue.

API	Provider	Consumers	Protocol	Contract	Evidence	Confidence

Identify:

provider
consumers
internal APIs
external APIs
duplicate endpoints
versioning
unresolved consumers/providers

13. Architecture Pattern Aggregation

Do not simply combine repository-level pattern claims.

Evaluate patterns at project level.

For example:

A repository may show:

Kafka producer

Another:

Kafka consumer

Together this provides stronger evidence for:

Event-driven integration

But still distinguish:

Event-driven integration

from:

Entire project follows Event-Driven Architecture.

For project-level patterns evaluate evidence across multiple services.

Potential patterns:

Event-Driven Architecture
Saga
Saga choreography
Saga orchestration
Outbox
CQRS
API Gateway
Hexagonal Architecture
Layered Architecture
Shared Database
Service-to-Service REST
Asynchronous integration

For each pattern:

Pattern	Project Evidence	Supporting Repositories	Confidence	Conclusion

Do not claim a project-wide pattern unless the evidence supports project-wide scope.

14. Business Capability Aggregation

Where repository evidence supports it, group services into business capabilities.

Example:

Customer Management
    ├── Customer Service
    ├── Customer Profile Service
    └── Customer Notification Service

Order Management
    ├── Order Service
    ├── Order Validation Service
    └── Order Fulfilment Service

Do not invent business capabilities.

If the business purpose cannot be established:

Business capability grouping not established from repository evidence.

15. End-to-End Flow Reconstruction

Identify flows that cross repository boundaries.

For example:

Customer
   ↓
Order API
   ↓
Order Service
   ↓
Kafka
   ↓
Payment Service
   ↓
Payment Provider
   ↓
Kafka
   ↓
Order Service

For each cross-service flow identify:

Trigger
Service
Interaction
Message/API
Next service
Persistence
Output
Failure handling

Distinguish:

synchronous flow
asynchronous flow
mixed flow

Do not create an end-to-end business flow when only isolated technical relationships are known.

16. Cross-Repository Architecture Observations

Identify significant project-level observations.

Examples:

central services
highly connected services
shared databases
synchronous dependency chains
event-driven boundaries
integration hubs
duplicated functionality
shared infrastructure
tightly coupled services
isolated services
common platform dependencies

These are observations.

Do not automatically classify them as problems.

17. Contradiction Detection

Actively search for conflicts between repositories.

Examples:

API conflict

Repository A:

GET /customer/{id}

Repository B expects:

GET /customers/{id}

Event conflict

Producer:

order.created

Consumer expects:

order-created

Schema conflict

Producer sends:

customerId

Consumer expects:

customer_id

Technology conflict

One repository identifies:

Kafka

Another identifies the same integration as:

SQS

Do not resolve contradictions by guessing.

Record them.

18. Missing Relationship Detection

Look for relationships that may be incomplete.

Examples:

Consumer exists
but producer is unknown.

API consumer exists
but provider is unknown.

Topic exists
but ownership is unknown.

Service depends on database
but database repository is not identified.

Classify each:

confirmed external dependency
missing repository
missing evidence
unresolved

19. Project Architecture Model

Produce a structured Project Architecture Model containing:

Project
Repositories
Services
Components
APIs
Events
Topics
Queues
Databases
ExternalSystems
Dependencies
CrossServiceFlows
ArchitecturePatterns
BusinessCapabilities
DeploymentArchitecture
SecurityArchitecture
ObservabilityArchitecture
ResilienceArchitecture
Contradictions
Unknowns
Evidence

The model must preserve the originating repository for every significant entity and relationship.

20. Project Dependency Matrix

Produce a service dependency matrix where useful.

Example:

	Order	Payment	Customer	Inventory
Order	—	Event	API	Event
Payment	Event	—	API	—
Customer	API	API	—	—
Inventory	Event	—	—	—

Do not create empty relationships simply to populate the matrix.

21. Project-Level Confidence

Assign confidence to project-level conclusions.

Use:

High
Medium
Low
Unknown

Confidence must reflect evidence quality, not how plausible the architecture appears.

22. Aggregation Quality Gate

Before completing Step 4, check:

Identity

repositories are uniquely identified
services are consistently identified
duplicate entities are investigated
ambiguous entities remain unresolved

Relationships

cross-service relationships have evidence
API relationships are supported
messaging relationships are supported
database relationships are supported

Consistency

conflicting names are identified
conflicting contracts are identified
conflicting technologies are identified
contradictory relationships are identified

Architecture

project-level patterns are evidence-based
local patterns are not automatically treated as project-wide
end-to-end flows are evidence-based

Traceability

Every significant project-level relationship can be traced back to one or more repositories.

Uncertainty

Unknowns and unresolved relationships are explicitly recorded.

23. Final Output

Return:

A. Project Architecture Model

The structured project-wide model.

B. Project Dependency Graph

All confirmed cross-repository relationships.

C. Cross-Service Flows

Important end-to-end flows.

D. Architecture Patterns

Patterns supported by project-level evidence.

E. Contradictions

Conflicting evidence that requires resolution.

F. Unknowns

Important information that cannot be established.

G. Aggregation Status

Return:

PASS

or:

PASS_WITH_WARNINGS

or:

FAIL

Do not generate the final project architecture document.

Step 5 — Project Validation will validate this Project Architecture Model before project documentation is generated.

The key change from Step 3

Step 3 was:

"Explain this repository."

Step 4 is:

"What happens when all these repositories are viewed as one system?"

That distinction is crucial.

For example, three individual documents might say:

Order Service
  produces order.created

Payment Service
  consumes order.created

Inventory Service
  consumes order.created

Step 4 discovers the project-level relationship:

                    order.created
                         │
                ┌────────┴────────┐
                ↓                 ↓
        Payment Service    Inventory Service

That relationship may not exist explicitly in any single repository. It emerges from aggregation.

And Step 4 is also where we start finding the really valuable architecture information:

service coupling
central services
shared databases
event hubs
synchronous chains
cross-service flows
duplicated integrations
inconsistent contracts
project-wide architecture patterns

But we still don't trust the result yet.

That's Step 5.

Step 5 will be the final Project Validation gate — essentially asking:

"Given the entire 100-service architecture model, can we trust the project-level architecture we are about to publish?"

After that, we'll have the complete five-stage pipeline.

Step 3 — Documentation Generation

Karthi Mahadevan — Sun, 04 Oct 2026 12:19:23 +0000

Exactly. Step 3 is where we finally produce the human-readable architecture document.

The important rule is:

Step 3 must not rediscover the architecture. It must explain the validated Architecture Evidence Model.

That prevents the documentation stage from introducing new assumptions.

The pipeline is now:

Source Code
    ↓
1. Extraction
    ↓
Architecture Evidence Model
    ↓
2. Validation
    ↓
Validated Architecture Evidence Model
    ↓
3. Documentation
    ↓
Repository Architecture Document

Save this as 03-documentation-generation.md.

Step 3 — Documentation Generation

Purpose

Generate the repository-level architecture documentation from the validated Architecture Evidence Model produced by Step 2.

This is a documentation task.

The architecture has already been extracted and validated.

Do not independently rediscover the architecture.

Do not introduce new architectural claims.

Do not invent missing business context.

The validated Architecture Evidence Model is the primary source for this document.

The objective is to produce a clear, technically accurate, reusable architecture document that can later be combined with equivalent documents from other repositories to create project-level architecture documentation.

1. Core Rules

Rule 1 — Evidence is authoritative

Use the validated Architecture Evidence Model as the source of truth.

If information is marked:

Verified

present it as established fact.

If information is marked:

Inferred

make the inference explicit.

If information is marked:

Unknown

do not fill the gap.

Rule 2 — Do not invent

Do not invent:

business requirements
business processes
architecture patterns
system dependencies
technology usage
security controls
deployment characteristics
scalability
availability
disaster recovery
data ownership

unless supported by the validated evidence.

Rule 3 — Explain the application, not generic technology

Do not explain generic technologies such as:

what Kafka is
what Kubernetes is
what AWS is
what REST is
what PostgreSQL is

unless required to explain how this application uses the technology.

The focus is:

How does this application use the technology?

Rule 4 — Preserve technical depth

Do not oversimplify the architecture.

The document must be understandable to:

software developers
application architects
platform engineers
DevOps engineers
security engineers
technical leads

Use clear language without removing important technical detail.

Rule 5 — Progressive disclosure

Present information in layers.

A reader should be able to understand the system at three levels:

Level 1 — System understanding

What is this application?

Why does it exist?

What systems does it interact with?

What is the main flow?

Level 2 — Application architecture

How are the components implemented?

How do they interact?

What are the APIs, events, databases and processing flows?

Level 3 — Platform and operational architecture

How is the application deployed?

What infrastructure does it use?

How are security, configuration, observability and resilience implemented?

2. Required Document Structure

Use the following exact section order.

Do not rename sections.

Do not omit a mandatory section.

If information is unavailable, state Unknown or Not established from repository evidence.

1. Executive Summary

Provide a concise overview of:

application/service
primary responsibility
major architectural characteristics
major integrations
primary runtime/platform
important architectural observations

Keep this section short.

It should allow a technical reader to understand the application before reading the details.

2. Business Context

Explain the business purpose of the application only where supported by evidence.

Include:

business capability
problem addressed
major business interactions
business outcome

If business context cannot be established from the repository:

Business context is not established from repository evidence.

Do not manufacture business explanations from class names.

3. System Context

Describe the application's position in the wider system.

Show:

users or actors where identifiable
upstream systems
this application
downstream systems
external systems
major data/event flows

Include a System Context diagram.

Use only relationships supported by the validated evidence.

4. Architecture Overview

Provide the main technical architecture.

Describe:

major components
major communication mechanisms
data stores
messaging
external integrations
deployment boundary

Include an architecture diagram.

The diagram must match the validated dependency model.

Do not show relationships that are not supported by evidence.

5. End-to-End Processing Flow

Describe the major application flows.

For each important flow explain:

Trigger
Entry point
Processing
Internal component interactions
External calls
Messaging
Persistence
Output
Error handling

Clearly distinguish:

synchronous
asynchronous
scheduled
event-driven

Use sequence diagrams where they provide useful detail.

Do not create a diagram merely for presentation.

6. Component and Class Design

Describe the significant application components.

For each important component include:

name
responsibility
source location
important interfaces
dependencies
relationships to other components

Where useful, include:

component diagram
class diagram

Do not create a class diagram containing every class.

Focus on classes and components that explain the architecture.

7. Input and Output Contracts

Document important interfaces.

Include:

REST APIs
requests
responses
events
messages
commands
scheduled inputs
outputs

Where available, document:

field names
important data structures
schemas
validation
error responses
versioning

Do not invent contract details.

If the full contract is not available:

Contract details are not established from repository evidence.

8. Integration and Messaging Architecture

Document all significant integrations.

Use a table such as:

Source	Target	Interaction	Technology	Contract	Direction	Purpose	Failure Handling

Include:

APIs
Kafka
queues
events
external systems
databases where relevant

For messaging include:

producer
topic/queue
consumer
message/event
asynchronous behaviour
retry
acknowledgement
dead-letter handling
ordering where established

9. Architecture Patterns

Document only patterns supported by evidence.

For each confirmed pattern:

Pattern

Name of pattern.

Evidence

Explain the implementation evidence.

Application of Pattern

Explain how the application uses it.

Benefit

Explain the technical or business benefit only where it can reasonably be established.

Possible patterns include:

Event-Driven Architecture
Saga
Saga choreography
Saga orchestration
Outbox
CQRS
API Gateway
Hexagonal Architecture
Ports and Adapters
Layered Architecture
Retry
Circuit Breaker
Idempotent Consumer

Do not claim a pattern simply because the repository uses a technology commonly associated with it.

For patterns classified as Possible, clearly label them as such.

10. Platform and Deployment Architecture

Describe how the application is deployed.

Include only supported information.

Where applicable:

containerisation
Kubernetes
Helm
AWS
networking
storage
configuration
secrets
IAM
CI/CD
deployment strategy
health checks
scaling
infrastructure dependencies

Explain how the application interacts with the platform.

Do not provide generic Kubernetes or AWS tutorials.

Include a deployment diagram when useful.

11. Failure Handling and Resilience

Describe actual implementation of:

retries
timeouts
circuit breakers
dead-letter handling
duplicate handling
idempotency
fallback
compensation
recovery
graceful degradation

Separate:

Implemented

from:

Not established

Do not claim resilience characteristics that are not evidenced.

12. Security and Observability

Security

Document:

authentication
authorisation
mTLS
OAuth/OIDC
JWT
certificates
IAM
secrets
Vault
Secrets Manager
security configuration

Only document mechanisms supported by evidence.

Observability

Document:

logging
metrics
tracing
health checks
correlation IDs
audit logging
monitoring integrations

Explain how these mechanisms are implemented by the application.

13. Source Code Navigation

Provide a practical map from architecture concepts to source code.

Example:

Architecture Element	Source Location	Description
REST API	`src/...`	API entry point
Order processor	`src/...`	Main processing logic
Kafka producer	`src/...`	Publishes event
Database repository	`src/...`	Persistence
Configuration	`config/...`	Runtime configuration

This section must help a developer move from the architecture document to the implementation.

14. Architecture Observations

Record important observations from the validated evidence.

Examples:

strong separation of responsibilities
tightly coupled components
synchronous dependency chain
asynchronous integration
central dependency
resilience mechanism
deployment constraint
security dependency
operational concern

Separate observations from facts.

Do not present architectural opinion as source-code fact.

Where an observation is an interpretation, label it:

Observation

15. Evidence and Open Questions

Document important evidence and unresolved questions.

Evidence

For important architecture claims include:

source location
implementation/configuration
evidence classification

Open Questions

Record information that cannot be established from the repository.

Examples:

business ownership
runtime configuration supplied outside the repository
production topology
external system contract
operational ownership
undocumented business rules

Do not hide uncertainty.

3. Diagrams

Generate diagrams only when they improve understanding.

Preferred diagram types:

System Context

Shows the application and external systems.

Container / Architecture

Shows major technical components and dependencies.

Component

Shows significant application components.

Sequence

Shows important runtime interactions.

Deployment

Shows runtime/platform architecture.

Class

Shows important structural relationships where useful.

Every diagram must be derived from the validated Architecture Evidence Model.

Do not add speculative components or relationships.

4. Diagram Rules

Use consistent notation.

For every relationship make clear:

source
target
interaction
direction
synchronous/asynchronous where relevant

For messaging:

```text id="c4f8r4"
Producer
|
| Event
v
Topic / Queue
|
v
Consumer




For synchronous calls:



```text id="5t9iq2"
Caller
   |
   | HTTP/API
   v
Target

Do not use an arrow without a meaningful relationship.

5. Evidence Marking

Do not clutter normal prose with evidence labels.

Use evidence labels where uncertainty matters.

For example:

```text id="w1b8id"
Evidence status: Verified




or:



```text id="5k90bx"
**Evidence status:** Inferred

or:

```text id="78a9t2"
Evidence status: Unknown




For important architecture claims, include the relevant source location.

---

# 6. Cross-Repository Compatibility

This document will later be combined with documentation from many repositories.

Therefore:

* preserve stable component names
* preserve topic names
* preserve API names
* preserve external system names
* preserve technology names
* use consistent terminology
* do not create arbitrary synonyms
* do not hide repository identity

The document must remain independently understandable while also being suitable for project-level aggregation.

---

# 7. Architecture Metadata

Include a metadata section containing:



```text id="h0d2f8"
Repository:
Application:
Document Type:
Source Commit:
Analysis Status:
Validation Status:
Architecture Confidence:
Known Limitations:

Do not invent values.

If unavailable, use:

Unknown

8. Writing Style

Use:

short sentences
active voice
precise technical terms
clear headings
tables where they improve comparison
diagrams where they improve understanding

Avoid:

marketing language
unnecessary introductions
generic technology tutorials
repetition
vague architectural claims
unsupported business explanations

Follow approximately 80% ASD-STE100 principles while retaining normal professional architecture-documentation vocabulary.

Use technical terminology where it is necessary.

Do not simplify terminology merely to make the document shorter.

9. Final Documentation Quality Check

Before producing the final document, validate the document against the Architecture Evidence Model.

Check:

Accuracy

Every significant architectural claim is supported by validated evidence.

Completeness

All required sections exist.

Consistency

Names and relationships are consistent across:

prose
tables
diagrams
source navigation

Diagram consistency

Every significant relationship shown in diagrams exists in the validated dependency model.

No invention

No new architecture facts have been introduced.

Uncertainty

Unknown and inferred information remains clearly identified.

Audience

The document provides useful information for:

developers
architects
platform engineers
DevOps engineers
security engineers

Cross-repository compatibility

The structure and terminology remain compatible with equivalent documents generated for other repositories.

10. Final Output

Return the complete repository architecture document.

Do not return the Architecture Evidence Model unless specifically requested.

Do not describe the documentation-generation process.

Do not provide a summary of what you were instructed to do.

The final output must be the architecture document itself.

Where Step 3 sits

We now have a clean separation:

                  REPOSITORY
                      │
                      ▼
            ┌─────────────────┐
            │ 1. EXTRACTION   │
            │                 │
            │ "What exists?"  │
            └────────┬────────┘
                     │
                     ▼
            Architecture
            Evidence Model
                     │
                     ▼
            ┌─────────────────┐
            │ 2. VALIDATION   │
            │                 │
            │ "Can we trust   │
            │  this?"         │
            └────────┬────────┘
                     │
                     ▼
             Validated Model
                     │
                     ▼
            ┌─────────────────┐
            │ 3. DOCUMENTATION│
            │                 │
            │ "How do we      │
            │ explain it?"    │
            └────────┬────────┘
                     │
                     ▼
             Repository
          Architecture Doc

This is an important boundary.

Step 1 discovers.
Step 2 challenges.
Step 3 communicates.

That means when you eventually process 100 repositories, you are not asking the LLM to repeatedly "understand everything and write a README." You are creating 100 validated, structurally identical architecture records, which can then be aggregated in Step 4.

And that is where the architecture gets interesting: Step 4 will not simply concatenate 100 documents. It will build a project-wide architecture model from the individual validated models and resolve cross-repository relationships.

Step 2 — Architecture Evidence Validation

Karthi Mahadevan — Sun, 04 Oct 2026 12:16:49 +0000

Purpose

Validate the Architecture Evidence Model produced by Step 1.

This is a validation task.

Do not generate the final architecture documentation.

Do not improve the architecture.

Do not silently correct unsupported claims.

Determine whether the extracted architecture evidence is sufficiently accurate, complete, consistent, and traceable to the source repository.

The output will be used to decide whether the repository can proceed to Step 3 — Documentation.

1. Validation Principles

Apply these principles throughout validation.

Principle 1 — Source evidence is authoritative

The source repository is the primary authority.

The Architecture Evidence Model must not contain claims that cannot be supported by repository evidence.

Principle 2 — Do not validate by plausibility

A statement being technically plausible is not sufficient.

For example:

Kafka is present.
Therefore the system is event-driven.

This is not valid evidence.

The validator must identify the actual producer, event/topic, consumer and processing behaviour.

Principle 3 — Distinguish fact from inference

Every significant architectural claim must be classified as:

Verified
Inferred
Unknown

Check that the classification matches the evidence.

Principle 4 — Do not silently repair

If the extraction contains an error:

identify the error
provide the evidence
state the required correction

Do not silently modify the Architecture Evidence Model.

Principle 5 — Prefer missing information over invented information

If the repository does not provide enough evidence:

Unknown

is preferable to an unsupported conclusion.

2. Validation Result

Return one overall status:

PASS
PASS_WITH_WARNINGS
FAIL

PASS

The evidence model is sufficiently accurate and complete for documentation generation.

PASS_WITH_WARNINGS

The model can proceed, but there are known limitations that must appear in the documentation.

FAIL

The model contains significant errors, unsupported claims, contradictions, or missing evidence that could make the architecture documentation misleading.

3. Repository Coverage Check

Confirm that Step 1 examined the relevant repository artefacts.

Check for:

application source code
configuration
build files
dependency definitions
tests
deployment manifests
Dockerfiles
Helm charts
infrastructure definitions
CI/CD configuration
repository documentation
scripts
messaging configuration
API definitions

For each category report:

Area	Found	Analysed	Evidence	Status

If a category does not exist, record:

Not present

Do not treat absence as a validation failure.

4. Evidence Traceability Check

For every significant architectural claim, verify that evidence exists.

Check:

source file path
class/function/configuration
relevant implementation
configuration
relationship evidence

Example:

Claim:
Payment Service consumes order.created.

Evidence:
src/messaging/OrderCreatedConsumer.java
Kafka consumer configuration
Topic configuration: order.created

Classification:
Verified

Flag any claim that has:

no evidence
vague evidence
incorrect source location
evidence that does not support the claim

5. Classification Check

Check whether each significant item has the correct classification.

Verified

Directly supported by repository evidence.

Inferred

Reasonably derived from multiple pieces of evidence but not explicitly established.

Unknown

Cannot be established from the repository.

Flag cases where:

Verified is actually only an inference
Inferred should be Unknown
Unknown is incorrectly presented as fact

6. Component Validation

Validate the extracted application components.

For each significant component:

Does it exist?
Does the source location exist?
Does its described responsibility match the implementation?
Are relationships supported?
Is the component actually architecturally significant?

Flag:

invented components
duplicated components
incorrect names
incorrect responsibilities
components described at the wrong abstraction level

Do not require every source-code class to appear as an architecture component.

7. Processing Flow Validation

Validate every extracted major processing flow.

For each flow check:

Trigger
   ↓
Entry Point
   ↓
Processing
   ↓
External Interaction
   ↓
Persistence / Messaging
   ↓
Output

Confirm that each significant step has source evidence.

Check that:

sequence is plausible from actual code
synchronous interactions are not labelled asynchronous
asynchronous interactions are not labelled synchronous
error paths are supported
persistence is supported
external calls are supported

Flag flows where the extraction jumps from:

A → D

without evidence for:

A → B → C → D

8. API Validation

For every extracted API validate:

endpoint/path
HTTP method where applicable
implementation
request/input
response/output
authentication
downstream calls

Confirm that API claims match actual source definitions.

Flag:

endpoints that do not exist
incorrect HTTP methods
incorrect request/response descriptions
unsupported security claims

9. Messaging Validation

This is a high-priority validation area.

For every messaging relationship validate:

Producer
    ↓
Topic / Queue
    ↓
Consumer

Check:

technology
topic/queue name
producer
consumer
message/event
configuration
serialization/schema if available
retry behaviour
acknowledgement behaviour
dead-letter handling

Do not accept:

Kafka dependency exists
→ Kafka integration exists
→ Event-driven architecture

without evidence of actual messaging behaviour.

Flag orphaned relationships such as:

Topic exists
but no producer or consumer evidence was found.

Unless the repository genuinely only contains one side of the integration.

10. Persistence Validation

For every database or datastore:

Check:

connection/configuration evidence
actual usage
read/write behaviour
relevant repositories or adapters
transaction evidence where claimed

Do not claim:

transactional boundary

unless source evidence supports it.

Do not claim:

database is the system of record

unless the repository provides evidence sufficient to support that statement.

11. External Dependency Validation

For every external system check:

source/configuration evidence
protocol
endpoint
interaction direction
authentication
actual usage

Distinguish:

Configured dependency

from:

Actively used dependency

A URL in configuration does not by itself prove that the application actively calls the system.

12. Deployment Validation

Validate deployment claims against actual repository artefacts.

Check:

Dockerfile
Kubernetes manifests
Helm charts
Terraform
CloudFormation
AWS SAM
CI/CD pipelines
deployment scripts

Validate claims about:

runtime platform
Kubernetes resources
AWS resources
storage
networking
secrets
configuration
health checks
scaling
deployment strategy

Do not infer runtime infrastructure that is not represented in the repository.

13. Security Validation

Validate security claims against implementation or configuration.

Examples:

mTLS
OAuth
OIDC
JWT
IAM
Vault
Secrets Manager
certificate authentication

Check whether the evidence demonstrates actual use.

Do not treat a library dependency as proof that the security mechanism is active.

For example:

Spring Security dependency exists

does not automatically prove:

OAuth authentication is enforced.

14. Architecture Pattern Validation

This is a critical quality gate.

For every claimed architecture pattern, verify the required evidence.

Event-Driven Architecture

Look for:

event producers
event definitions
messaging transport
asynchronous consumers
event-driven processing

Kafka alone is insufficient.

Saga

Require evidence of:

distributed business transaction
multiple participating services
state progression
orchestration or choreography
compensation/recovery behaviour where applicable

Multiple services and Kafka alone are insufficient.

Outbox

Require evidence of:

business data persistence
event persistence
atomic or transactionally coupled write
relay/publisher mechanism

Database + Kafka is insufficient.

CQRS

Require evidence of:

distinct command and query responsibilities
separate models or processing paths
meaningful separation of write and read behaviour

Separate controller classes are insufficient.

API Gateway

Require evidence that a gateway is acting as an entry point for multiple downstream services or equivalent gateway behaviour.

Circuit Breaker

Require actual circuit-breaker configuration or implementation.

Idempotent Consumer

Require evidence of duplicate detection, idempotency keys, deduplication state, or equivalent behaviour.

For each pattern return:

Pattern	Evidence Found	Required Evidence	Result
Event-Driven	...	...	Confirmed/Possible/Not established
Saga	...	...	Confirmed/Possible/Not established
Outbox	...	...	Confirmed/Possible/Not established

15. Dependency Graph Validation

Validate every dependency relationship.

For each relationship:

Source
    ↓
Relationship
    ↓
Target

Confirm that both endpoints exist or are explicitly classified as external.

Examples:

Order Service
    → produces
order.created

Payment Service
    → consumes
order.created

Payment Service
    → calls
Customer API

Check for:

missing endpoints
impossible relationships
duplicate relationships
contradictory relationships
incorrect direction

16. Cross-Section Consistency

Compare information across the entire Architecture Evidence Model.

Check for contradictions such as:

Section A:
Payment Service consumes order.created.

Section B:
Payment Service produces order.created.

Or:

Dependency Model:
SQS

Messaging Model:
Kafka

Or:

Deployment:
Kubernetes

Infrastructure:
EC2 standalone deployment

Do not assume one is correct.

Report the contradiction and identify the evidence required to resolve it.

17. Naming Consistency

Check that the same entity is not represented using inconsistent names.

Examples:

PaymentService
payment-service
Payment Service
Payments Service

Determine whether these are:

the same entity
different entities
unresolved

Do not automatically merge entities unless evidence supports the relationship.

18. Completeness Check

Check whether the extraction contains enough information for the next stage.

Required areas:

application identity
significant components
major processing flows
APIs
messaging
persistence
external dependencies
deployment
security where identifiable
observability where identifiable
resilience where identifiable
architecture pattern evidence
dependency relationships
unknowns

A missing area is not automatically a failure.

Determine whether it is:

Not present

or:

Not analysed

or:

Missing evidence

These have different meanings.

19. Hallucination Check

Actively search for unsupported claims.

Ask:

"Could this statement have been produced from general software knowledge rather than repository evidence?"

If yes, inspect the evidence.

Flag claims such as:

assumed business purpose
assumed architecture pattern
assumed runtime behaviour
assumed scalability
assumed high availability
assumed disaster recovery
assumed security controls
assumed data ownership
assumed transaction boundaries

Unless supported by evidence.

20. Unknowns Check

Ensure uncertainty has not been hidden.

The validation must identify important unanswered questions.

Examples:

Business purpose unknown.

Runtime deployment environment not established.

Kafka topic ownership not established.

External system contract not available.

Production scaling configuration not present.

These must remain explicit.

21. Validation Findings

Produce a findings table.

ID	Severity	Category	Finding	Evidence	Required Action

Severity:

Critical
High
Medium
Low
Informational

Examples:

V001 | High | Evidence | Saga claimed without compensation/orchestration evidence | ... | Remove claim or provide evidence

V002 | Medium | Completeness | Deployment strategy not established | ... | Mark Unknown

V003 | Low | Naming | PaymentService and payment-service may refer to same component | ... | Confirm canonical identity

22. Correction Instructions

If validation fails, produce explicit correction instructions for Step 1.

Format:

CORRECTION REQUEST

Finding:
...

Problem:
...

Required re-analysis:
...

Source locations to inspect:
...

Expected correction:
...

Do not modify unrelated areas.

The correction request must be actionable.

23. Final Quality Gate

The Architecture Evidence Model may proceed to Step 3 only when:

no Critical findings remain
no High unsupported architectural claims remain
significant claims have evidence
classifications are reasonable
major relationships are traceable
contradictions are resolved or explicitly documented
unknowns are clearly identified
architecture patterns are evidence-based
naming is sufficiently consistent
the dependency model is usable for later aggregation

Return:

VALIDATION STATUS:
PASS

or:

VALIDATION STATUS:
PASS_WITH_WARNINGS

or:

VALIDATION STATUS:
FAIL

Do not generate the final architecture documentation.

Step 3 will consume the validated Architecture Evidence Model.

small agentic documentation pipeline

Karthi Mahadevan — Sun, 04 Oct 2026 12:13:00 +0000

EXTRACTION ↓
VALIDATION ↓
DOCUMENTATION ↓
AGGREGATION ↓
PROJECT VALIDATION

Yes. We should treat this as a small agentic documentation pipeline, but we do not need to make it complicated.

I would split it into 5 main stages, matching exactly what you chose:

1. EXTRACTION
      ↓
2. VALIDATION
      ↓
3. DOCUMENTATION
      ↓
4. AGGREGATION
      ↓
5. PROJECT VALIDATION

Inside those stages, we will eventually have several prompts. But do not create all of them now.

We will build them one at a time and keep the output contract stable.

Step 1 — EXTRACTION

The purpose of Step 1 is very important:

Do not write architecture documentation yet.

Extract what can be established from the source code and repository.

Think of this as building the Architecture Evidence Model.

SOURCE REPOSITORY
       ↓
   STEP 1 PROMPT
       ↓
ARCHITECTURE EVIDENCE

The output is structured facts, not a polished README.

Step 1 prompt

I recommend making this a separate prompt from your existing documentation prompt.

Save the following as:

01-source-extraction.md

Step 1 — Source-to-Architecture Extraction

Purpose

Analyse the supplied source repository and extract the architecture evidence that can be established from the repository.

This is an extraction task.

Do not write the final architecture documentation.

Do not improve, simplify, or redesign the architecture.

Do not assume that a technology implies an architecture pattern.

Extract evidence from the source code, configuration, deployment files, infrastructure definitions, tests, documentation, and other repository artefacts.

The output will be consumed by later stages:

Validation
Documentation generation
Cross-repository aggregation
Project-level architecture analysis

Accuracy is more important than completeness.

If information cannot be established from the repository, mark it as Unknown.

If information is reasonably inferred from evidence but is not explicitly established, mark it as Inferred.

Use Verified only when the repository provides direct evidence.

1. Analysis Rules

Follow these rules throughout the analysis.

Rule 1 — Evidence first

Every significant architectural fact must have supporting repository evidence.

Record:

source file path
relevant class/function/configuration where useful
evidence description
confidence classification

Use:

Verified
Inferred
Unknown

Do not present inference as fact.

Rule 2 — Do not invent business meaning

The repository may reveal technical behaviour without revealing the business purpose.

Do not invent business context.

If the business purpose cannot be established:

Unknown — business purpose not established from repository evidence.

Rule 3 — Technology does not prove architecture

Do not infer an architecture pattern merely because a technology is present.

Examples:

Kafka does not automatically prove Event-Driven Architecture.
Multiple microservices do not automatically prove Saga.
A database and messaging system do not automatically prove Outbox.
Separate read/write classes do not automatically prove CQRS.
Kubernetes does not automatically prove cloud-native architecture.

Record the technical evidence first.

Only record an architecture pattern when the required behavioural evidence exists.

Rule 4 — Distinguish source behaviour from infrastructure

Separate:

application behaviour
configuration
infrastructure/deployment
runtime assumptions

Do not assume that infrastructure configuration proves application behaviour.

Rule 5 — Preserve names

Use the names found in the repository for:

services
applications
classes
interfaces
APIs
events
topics
queues
databases
external systems
configuration properties

Do not rename them for convenience.

A later stage may create canonical names.

Rule 6 — Do not fill gaps

When evidence is missing:

Unknown

Do not guess.

2. Repository Identity

Extract:

repository name
application/service name
repository purpose if explicitly stated
programming languages
major frameworks
build system
repository structure
deployment model if identifiable

For each item provide evidence.

3. Application Components

Identify the significant application components.

For each component record:

component name
type
responsibility
source location
relationships to other components
evidence
classification

Component types may include:

service
controller
API
consumer
producer
processor
handler
scheduler
repository
database adapter
external-system adapter
domain component
configuration component

Do not list every class.

Focus on components that contribute to the architecture.

4. Main Processing Flows

Identify the major application flows that can be established from the source.

For each flow record:

trigger
entry point
processing steps
important components
external calls
messaging
persistence
output
error handling
evidence

Distinguish:

synchronous flow
asynchronous flow
scheduled flow
event-driven flow

Do not create a business flow that cannot be supported by evidence.

5. Interfaces and APIs

Identify application interfaces.

Include:

REST APIs
GraphQL APIs
messaging interfaces
event interfaces
command interfaces
scheduled interfaces
other externally triggered interfaces

For each interface record:

name/path
method or interaction type
producer
consumer
input
output
authentication/security evidence
source location
evidence classification

6. Messaging

Identify all messaging technologies and messaging relationships.

Examples:

Kafka
SQS
SNS
RabbitMQ
other brokers

For each messaging relationship record:

technology
topic/queue/channel
producer
consumer
message/event name
direction
message structure if identifiable
configuration
retry behaviour
dead-letter handling
ordering evidence
acknowledgement behaviour
source evidence

Do not infer semantics that are not supported by source evidence.

7. Data and Persistence

Identify:

databases
tables where architecturally relevant
repositories
persistence frameworks
caches
object stores
external data stores

For each dependency record:

technology
purpose
read/write behaviour
relevant entities
transaction boundaries if identifiable
source evidence

8. External Systems

Identify systems outside the repository.

For each external dependency record:

system name
interaction type
direction
protocol/technology
purpose if established
configuration
authentication mechanism if identifiable
failure handling
source evidence
classification

9. Security

Extract security mechanisms that are actually present.

Examples:

authentication
authorisation
mTLS
OAuth/OIDC
JWT
certificates
secrets
Vault
AWS Secrets Manager
IAM
API security
network security configuration

Do not infer security controls that are not evidenced.

10. Configuration

Identify important configuration.

Include:

environment variables
configuration files
secrets references
endpoints
topic names
queue names
database configuration
feature flags
timeout values
retry configuration
external service configuration

Distinguish:

hard-coded values
environment configuration
secret references
infrastructure-provided configuration

Do not expose secret values.

11. Deployment and Infrastructure

Extract deployment evidence.

Look for:

Dockerfiles
Kubernetes manifests
Helm charts
Terraform
CloudFormation
AWS SAM
Jenkins
GitLab CI
GitHub Actions
Flux
Argo CD
other deployment mechanisms

Record:

deployment unit
runtime platform
Kubernetes resources
AWS resources
networking dependencies
storage
configuration injection
secrets injection
scaling configuration
health checks
deployment strategy

Only record infrastructure that can be established from repository evidence.

12. Observability

Identify:

logging
metrics
tracing
health endpoints
monitoring integrations
alerts
correlation IDs
audit logging

Record the implementation evidence.

13. Resilience and Failure Handling

Identify actual implementation of:

retries
timeout handling
circuit breakers
dead-letter queues
fallback
idempotency
duplicate handling
transaction handling
compensation
recovery
graceful degradation

Do not label these as architecture patterns unless the evidence supports the pattern.

14. Architecture Pattern Evidence

Evaluate the repository for evidence of known architecture patterns.

Possible patterns include:

Event-Driven Architecture
Saga
Saga choreography
Saga orchestration
Outbox
CQRS
API Gateway
Hexagonal Architecture
Ports and Adapters
Layered Architecture
Retry
Circuit Breaker
Idempotent Consumer

For each candidate pattern provide:

pattern
evidence required
evidence found
classification
conclusion

Use:

Confirmed

Possible

Not established

Do not confirm a pattern from technology names alone.

15. Dependency Model

Create a structured list of dependencies.

For every significant relationship record:

Source	Relationship	Target	Technology	Direction	Evidence	Classification

Examples:

Order Service | produces event | order.created | Kafka | outbound | source evidence | Verified

Payment Service | consumes event | order.created | Kafka | inbound | source evidence | Verified

Payment Service | calls | Customer API | REST | outbound | source evidence | Verified

This dependency model is particularly important.

It will later be aggregated across many repositories.

16. Architecture Evidence Model

Produce a structured representation containing at least:

Repository
Application
Components
ProcessingFlows
APIs
Messages
Topics
Queues
Databases
ExternalSystems
Security
Configuration
Deployment
Observability
Resilience
ArchitecturePatterns
Dependencies
Unknowns
Evidence

Use stable names and relationships.

The model must be suitable for machine processing and later aggregation.

17. Unknowns and Gaps

Create an explicit list of information that cannot be established.

For each unknown record:

missing information
why it cannot be established
what source or human information would be required

Examples:

Unknown:
Business purpose of PaymentProcessor.

Reason:
Technical behaviour is visible, but no business documentation exists.

Required confirmation:
Product/business owner.

18. Final Extraction Check

Before producing the final output, perform this check.

Coverage

Confirm that you examined:

source code
configuration
build files
deployment files
infrastructure files
CI/CD configuration
tests
repository documentation

If a category does not exist, record that it was checked and not found.

Evidence

Confirm:

significant claims have evidence
inferred facts are labelled
unknown facts are labelled
no unsupported architecture pattern is claimed

Relationships

Confirm:

producers and consumers are identified where possible
upstream and downstream relationships are recorded
APIs are connected to their implementations
messaging relationships are connected to source evidence
persistence relationships are recorded

Consistency

Confirm:

component names are consistent
topic names are consistent
API names are consistent
external system names are consistent

Final rule

Do not generate polished architecture documentation.

Return the Architecture Evidence Model and the associated evidence.

This output will be passed to Step 2 — Validation.

What this gives us

The important distinction is:

Step 1 does not try to make the documentation beautiful.

It tries to answer:

"What does the repository actually tell us?"

That gives Step 2 something concrete to validate.

Our pipeline now looks like:

┌─────────────────────┐
│    SOURCE REPO      │
└──────────┬──────────┘
           ↓
┌─────────────────────┐
│  1. EXTRACTION      │
│                     │
│ Architecture       │
│ Evidence Model      │
└──────────┬──────────┘
           ↓
┌─────────────────────┐
│  2. VALIDATION      │  ← next
└──────────┬──────────┘
           ↓
┌─────────────────────┐
│  3. DOCUMENTATION   │
└──────────┬──────────┘
           ↓
┌─────────────────────┐
│  4. AGGREGATION     │
└──────────┬──────────┘
           ↓
┌─────────────────────┐
│ 5. PROJECT          │
│    VALIDATION       │
└─────────────────────┘

And yes, this is agentic in the useful sense: each stage has a defined job, a defined input, and a defined output. But we are not yet adding autonomous agents or complex orchestration. We are first designing the contracts between the stages.

Next, Step 2 should be the Validation Prompt. That is where we make the system catch omissions, unsupported claims, contradictions and hallucinated architecture before documentation is generated.

Source-to-Architecture Documentation Framework

Karthi Mahadevan — Sun, 04 Oct 2026 11:48:35 +0000

A reusable methodology for generating high-quality technical documentation from source code using LLMs

1. Introduction

1.1 The objective

The objective is to generate accurate, useful and maintainable architecture documentation from an existing software repository.

The documentation must explain the system as a whole.

It must not simply describe source files, classes, methods or dependencies.

It must help different technical audiences understand:

Why the system exists.
What business problem it solves.
What the system does.
How the system processes information.
How its components interact.
What data enters and leaves the system.
Which systems are upstream and downstream.
Which architectural patterns the system implements.
How messaging and integration technologies support the application.
How the application uses its underlying platform.
How failures, retries and errors affect the flow.
Where the important implementation details exist in the source code.

The final document must connect business behaviour, application implementation and platform architecture.

The goal is not to explain every technology.

The goal is to explain how this particular system uses those technologies to deliver its business capability.

1.2 What this documentation is not

This documentation is not:

A programming tutorial.
A Kubernetes or Kafka introduction.
A generic technology reference.
A list of source-code files.
An automatically generated class inventory.
A marketing document.
A description of an ideal architecture that does not match the implementation.

For example, if an application uses Kafka, the documentation should explain:

Which component publishes messages.
Which topic receives them.
What message it publishes.
Which component consumes them.
What happens after consumption.
What business activity the message supports.
How the messaging flow affects the application architecture.

It does not need to explain what Kafka is.

The reader is expected to understand the technologies already.

2. The Documentation Philosophy

The methodology combines nine complementary techniques.

Each technique has a specific responsibility.

Technique	Responsibility
Pyramid Principle / BLUF	Structure information
ASD-STE100	Control language complexity
Diátaxis	Distinguish documentation types
C4 Model	Visualise architecture
arc42	Organise architecture information
Feynman Technique	Improve explanation quality
Cognitive Load Theory	Control information complexity
Progressive Disclosure	Organise information into layers
Evidence-Based Analysis	Maintain technical accuracy

These techniques must work together.

They must not become competing instructions.

2.1 Pyramid Principle and BLUF — Structure

Present the main information first.

Start with the conclusion or main purpose.

Then explain the supporting information.

Use this order:

Main purpose.
Main architectural characteristics.
Supporting components.
Detailed implementation.
Evidence and references.

For example:

Weak explanation:

The application contains an OrderController. It calls OrderService. The service validates the order and publishes an event. The event is consumed by another service.

Better explanation:

The application processes customer orders and publishes order events for downstream services.

The OrderController accepts incoming requests. OrderService validates and processes them. The event publisher sends the resulting event to Kafka.

The second explanation gives the reader the system purpose before introducing implementation details.

2.2 ASD-STE100 — Control language

Use approximately 80% of ASD-STE100 principles.

The objective is clear technical English, not artificial simplification.

Rules:

Use short sentences.
Prefer familiar words.
Use active voice.
Keep one main idea in each sentence.
Avoid unnecessary adjectives.
Avoid long introductory paragraphs.
Use consistent terminology.
Retain essential technical terms.
Avoid repeating information.
Do not remove technical depth to make the document shorter.

Example:

Avoid:

The service facilitates the asynchronous propagation of order-related domain events to downstream consumers through the underlying messaging infrastructure.

Prefer:

The service publishes order events to Kafka.

Downstream consumers receive these events and process them.

The second version is easier to read and retains the technical meaning.

2.3 Diátaxis — Control documentation type

Diátaxis defines four types of technical documentation.

They are:

Type	Purpose
Tutorial	Learn through a practical sequence
How-to guide	Complete a specific task
Explanation	Understand how and why something works
Reference	Find precise technical information

Architecture documentation is mainly an Explanation.

It must explain the system's structure, behaviour and design.

It can include reference information where useful.

For example:

Architecture overview: Explanation.
Component responsibilities: Explanation.
API schemas: Reference.
Kafka topic configuration: Reference.
Local development setup: How-to.
Getting started with the repository: Tutorial.

Do not turn an architecture README into a tutorial.

2.4 C4 Model — Control architecture visualisation

Use architecture diagrams to explain the system at different levels.

Recommended views:

System Context

Shows the system, users and external systems.

Container

Shows applications, services, databases and major runtime components.

Component

Shows the important internal components within an application or service.

Dynamic or Sequence View

Shows how components interact during a particular business flow.

Deployment View

Shows where the application runs and which infrastructure components support it.

Do not create every diagram automatically.

Select diagrams based on the complexity of the system and the needs of the readers.

A class diagram is a separate implementation-level view. Use it when class relationships provide useful information.

2.5 arc42 — Control documentation structure

Use arc42 as a reference for architecture documentation.

Its useful concepts include:

Introduction and goals.
Constraints.
Context and scope.
Solution strategy.
Building blocks.
Runtime behaviour.
Deployment.
Cross-cutting concepts.
Architecture decisions.
Quality requirements.
Risks and technical debt.

Do not force every arc42 section into every README.

Use the sections that provide value for the specific application.

2.6 Feynman Technique — Control explanation quality

Explain complex interactions in simple language.

This does not mean removing technical detail.

It means making the relationships understandable.

For example:

Technical statement:

The OrderService publishes an OrderCreated event, which is consumed asynchronously by the PaymentService.

Better explanation:

After the order is created, OrderService publishes an OrderCreated event.

PaymentService receives the event and starts payment processing.

The two services do not need to call each other directly during this flow.

This explanation describes the actual interaction and its architectural consequence.

Use practical examples to explain complex behaviour.

Do not introduce unrelated analogies.

2.7 Cognitive Load Theory — Control complexity

Do not expose every implementation detail at once.

Separate the system into meaningful concepts.

For example:

First explain the business transaction.
Then explain the application flow.
Then explain the messaging interaction.
Then explain the relevant classes.
Finally explain the infrastructure dependencies.

Avoid introducing Kafka partitions, Kubernetes probes, IAM policies and database indexes in the same paragraph unless they directly affect the flow being explained.

2.8 Progressive Disclosure — Control reading depth

Organise the document into layers.

A reader must be able to understand the system at a high level without reading every implementation detail.

Recommended layers:

Layer 1 — Business and architecture overview

What the system does and why it exists.

Layer 2 — End-to-end behaviour

How information moves through the system.

Layer 3 — Application implementation

Which components and classes implement the flow.

Layer 4 — Integration and platform

How infrastructure enables the flow.

Layer 5 — Operational and technical detail

Configuration, resilience, deployment and failure handling.

2.9 Evidence-Based Analysis — Control accuracy

This is a mandatory principle.

The LLM must distinguish between:

Verified facts.
Reasonable inferences.
Unknown information.

Never present an assumption as an implementation fact.

For example:

Verified:

OrderService publishes an event using KafkaTemplate.

Inferred:

The application appears to use event-driven communication.

Unknown:

Whether the business process requires a seven-year event retention period.

Source code can explain implementation.

It may not explain the business decision behind that implementation.

Flag missing information for human confirmation.

3. Target Audience and Reading Experience

The documentation must support three primary audiences.

Do not produce three independent documents unless there is a clear reason.

Create one architecture document with different reading levels.

3.1 Developers

Developers need to understand how to navigate and change the application.

They need:

Entry points.
Class responsibilities.
Component relationships.
Processing sequence.
Input and output contracts.
Data transformations.
Database interactions.
Messaging behaviour.
Exception handling.
Configuration.
Source-code references.

Their main question is:

Where is the behaviour implemented, and what happens when the code executes?

3.2 Architects

Architects need to understand the system's structure and design.

They need:

Business purpose.
System boundaries.
External dependencies.
Upstream and downstream relationships.
Architecture diagrams.
Architectural patterns.
Integration strategy.
Data ownership.
Synchronisation and consistency.
Failure boundaries.
Design constraints.
Important architecture decisions.

Their main question is:

Why is the system designed this way, and how does it interact with the wider architecture?

3.3 Platform Engineers and SREs

Platform engineers need to understand how the application uses infrastructure.

They need:

Runtime components.
Deployment architecture.
Kubernetes resources.
AWS services.
Kafka topics and consumer groups.
SQS queues and subscriptions.
IAM permissions.
Configuration and secrets.
Network dependencies.
Storage dependencies.
Scaling behaviour.
Health checks.
Monitoring and logging.
Operational failure points.

Their main question is:

What infrastructure does the application depend on, and how does the application use it?

4. Recommended Architecture README Structure

Use the following structure as the default.

Adapt it to the actual application.

Do not generate empty sections.

4.1 Executive Summary

Explain the application in a few paragraphs.

Include:

Application purpose.
Business capability.
Main processing responsibility.
Important integrations.
Architectural characteristics.

A reader should understand the system without reading the entire document.

4.2 Business Context

Explain why the application exists.

Include:

Business problem.
Business process.
Capability delivered.
Main stakeholders.
Business benefits.
Position within the wider business process.

Use domain terminology from the source and available business documentation.

Do not invent business benefits.

If the source does not establish the business purpose, identify the missing information.

4.3 System Context

Describe the application boundary.

Identify:

External systems.
Upstream systems.
Downstream systems.
Users or initiating actors.
Data providers.
Data consumers.
External interfaces.

Use a System Context diagram.

Include an integration table:

System	Relationship	Interaction	Information exchanged
System A	Upstream	REST API	Customer request
System B	Downstream	Kafka	Order event
System C	External dependency	SQS	Notification request

Only include verified relationships.

4.4 Architecture Overview

Describe:

Overall architecture.
Main applications and services.
Communication mechanisms.
Data stores.
Architectural patterns.
Major infrastructure dependencies.

Include a high-level architecture diagram.

Keep the diagram readable.

Do not show every class or infrastructure resource at this level.

4.5 End-to-End Processing Flow

This is one of the most important sections.

Explain the complete business transaction or processing journey.

For each flow, identify:

Trigger.
Entry point.
Validation.
Processing.
Data transformation.
Persistence.
Event publication or API interaction.
Downstream processing.
Final output.
Error or compensation behaviour.

Use a sequence diagram when the flow involves several interacting components.

Include synchronous and asynchronous boundaries.

Make it clear where the original request ends and where asynchronous processing continues.

4.6 Component and Class Design

Explain the important components.

Do not list every class.

Group classes by responsibility.

Example:

Component	Responsibility	Important interactions
OrderController	Accepts external requests	OrderService
OrderService	Validates and processes orders	OrderRepository, EventPublisher
OrderRepository	Persists order data	Database
EventPublisher	Publishes order events	Kafka
PaymentConsumer	Processes payment events	PaymentService

Include a class diagram where it adds value.

Show:

Important classes.
Interfaces.
Inheritance where relevant.
Composition and dependencies.
Important method relationships where useful.

Do not use a class diagram as a substitute for a sequence diagram.

A class diagram shows structure.

A sequence diagram shows behaviour over time.

4.7 Input and Output Contracts

Document the data entering and leaving the system.

For each important interface, identify:

Interface name.
Producer.
Consumer.
Input schema.
Output schema.
Mandatory fields.
Optional fields.
Validation.
Transformation.
Error response.
Versioning behaviour.

Example:

Field	Description
Interface	OrderCreated
Producer	OrderService
Consumer	PaymentService
Transport	Kafka
Input	Order details
Output	Payment processing request
Contract	Actual schema or DTO reference

Use actual DTOs, schemas, API specifications and message definitions.

Do not invent fields.

4.8 Integration and Messaging Architecture

Explain how the application uses messaging and integration technologies.

For Kafka, consider:

Producer component.
Topic.
Message schema.
Partition key, if established.
Consumer component.
Consumer group, if configured or discoverable.
Offset behaviour, if relevant.
Retry behaviour.
Dead-letter handling.
Ordering requirements.
Delivery semantics.
Authentication and authorisation dependencies.

For SQS, consider:

Queue.
Producer.
Consumer.
Message structure.
Visibility timeout.
Retry behaviour.
Dead-letter queue.
FIFO or standard queue.
Message group or deduplication settings, where relevant.

For REST APIs, consider:

Calling service.
Receiving service.
Endpoint.
Request and response.
Authentication.
Timeout.
Retry.
Error handling.

Focus on how these technologies support the application flow.

Do not provide generic technology explanations.

4.9 Architecture Patterns

Identify patterns supported by source-code evidence.

Possible patterns include:

Event-Driven Architecture.
Saga.
Choreography.
Orchestration.
CQRS.
Transactional Outbox.
Repository.
Adapter.
Strategy.
Dependency Injection.
Hexagonal Architecture.
Layered Architecture.
Circuit Breaker.

For each identified pattern, explain:

What pattern is present.
Where it appears.
Which components participate.
How the pattern works.
Why it matters to the system.
What evidence supports the identification.

Do not infer a pattern from a technology name.

Kafka alone does not prove Event-Driven Architecture.

Multiple services alone do not prove Saga.

If the evidence is incomplete, label the pattern as a possible or partial implementation.

4.10 Platform and Deployment Architecture

Explain the infrastructure used by the application.

Cover only relevant dependencies.

Examples:

Kubernetes Deployments.
Services.
Ingress or Gateway.
ConfigMaps.
Secrets.
Persistent Volumes.
AWS Lambda.
SQS.
SNS.
Kafka.
RDS.
S3.
IAM roles.
VPC and network dependencies.
External services.

Explain the relationship between the application and each platform component.

For example:

Platform component	Application usage	Purpose
Kubernetes Deployment	Runs OrderService	Application runtime
Kafka	Receives OrderCreated events	Asynchronous integration
RDS	Stores order data	Persistence
Secrets Manager	Provides credentials	Secure configuration

Do not document infrastructure that is not relevant to the application architecture.

4.11 Failure Handling and Resilience

Explain what happens when something fails.

Consider:

Validation failures.
Database errors.
Network failures.
API timeouts.
Kafka publication failures.
Consumer failures.
Message retries.
Duplicate messages.
Dead-letter handling.
Compensation.
Partial transaction completion.

Distinguish between:

Failure handling visible in source code.
Behaviour provided by infrastructure configuration.
Behaviour that requires runtime verification.

Do not assume that retries, idempotency or compensation exist because they are desirable.

4.12 Security and Observability

Document the application's relevant security and monitoring behaviour.

Security:

Authentication.
Authorisation.
Service identities.
IAM permissions.
Certificate usage.
Secret management.
Encryption configuration.

Observability:

Logging.
Metrics.
Tracing.
Correlation identifiers.
Health endpoints.
Monitoring integrations.
Alerting configuration.

Explain the implementation, not the general theory.

4.13 Source Code Navigation

Provide a practical map of the repository.

Example:

Location	Purpose
src/controller	API entry points
src/service	Business processing
src/repository	Data persistence
src/messaging	Event publishing and consumption
src/model	Domain objects and DTOs
src/config	Application configuration
tests	Automated tests

Use actual repository paths.

Highlight the important classes for the main business flow.

4.14 Architecture Observations

Record useful observations.

Separate them into:

Confirmed design characteristics.
Potential limitations.
Technical debt.
Unverified assumptions.
Questions for architects or business owners.

Avoid subjective criticism.

Explain why an observation matters.

5. The Three Essential Architecture Views

A strong architecture document must connect three views.

5.1 Business View

Explains why the system exists.

Example:

Customer submits an order.

The business process validates the order and initiates payment.

The customer receives a confirmation after the required processing completes.

5.2 Application View

Explains which software components implement the business process.

Example:

OrderController accepts the request.

OrderService validates the order.

OrderRepository persists the order.

EventPublisher publishes the resulting event.

5.3 Platform View

Explains which infrastructure components enable the application flow.

Example:

Kubernetes runs the application.

RDS stores order information.

Kafka transports events to downstream services.

The downstream consumer runs as a separate application workload.

5.4 Connect the three views

Use a mapping table.

Business activity	Application component	Platform dependency
Submit order	OrderController	Kubernetes Service
Validate order	OrderService	Application runtime
Save order	OrderRepository	RDS
Publish event	EventPublisher	Kafka
Process payment	PaymentConsumer	Kafka Consumer Group
Notify customer	NotificationService	SQS

This is one of the most valuable parts of the documentation.

It connects business behaviour to code and infrastructure.

6. Architecture Diagram Standards

Use diagrams to explain relationships, not decorate the document.

6.1 Recommended diagram types

Diagram	Main purpose	Audience
System Context	Show system boundaries	Architects
Container	Show major applications and infrastructure	Architects, Platform
Component	Show internal application structure	Developers
Class	Show important code relationships	Developers
Sequence	Show runtime interactions	All technical audiences
Data Flow	Show information movement	Developers, Architects
Deployment	Show runtime placement	Platform Engineers

6.2 Diagram rules

Use consistent names.
Use meaningful labels on connections.
Show the direction of communication.
Distinguish synchronous and asynchronous communication.
Show system boundaries.
Avoid unnecessary detail.
Keep diagrams readable.
Use the same component names in text and diagrams.
Do not show relationships that are not supported by evidence.

Prefer Mermaid when the repository supports Markdown-based diagrams.

Keep diagram source under version control.

6.3 Diagram accuracy

Every diagram must represent the implementation.

For example, do not draw a direct API call between two services if the actual interaction uses Kafka.

Do not draw a database connection if the service only receives data through another component.

Do not represent a logical architecture as a physical deployment diagram.

State when a diagram shows a logical view rather than the actual deployment topology.

7. Architecture Pattern Identification

Architecture pattern identification requires special care.

A pattern must be supported by behaviour, not technology names.

7.1 Event-Driven Architecture

Look for:

Event producers.
Event contracts.
Asynchronous consumers.
Event-based service interactions.
Event ownership and processing responsibilities.

Explain the event lifecycle.

7.2 Saga

Look for:

A business transaction spanning multiple services.
Multiple independent processing steps.
Distributed state.
Compensation actions.
Central orchestration or event-driven choreography.

Document the actual compensation flow.

If no compensation behaviour exists in the source, do not claim that a complete Saga is implemented.

7.3 Transactional Outbox

Look for:

A business database transaction.
An event or outbox record written in the same transaction.
A separate publisher or relay.
Publication status or retry handling.

Do not identify Outbox merely because a database and Kafka are both present.

7.4 CQRS

Look for:

Separate command and query responsibilities.
Separate write and read models.
Different processing paths for writes and reads.

Separate controller classes alone do not establish CQRS.

7.5 Choreography and Orchestration

For choreography, identify the participating services and the events that connect them.

For orchestration, identify the central component that controls the workflow.

Explain how the next processing step is selected.

8. Source Code Analysis Methodology

Do not ask an LLM to generate the final architecture document from a repository in one step.

Use a controlled analysis process.

Stage 1 — Repository Discovery

Identify:

Programming languages.
Frameworks.
Modules.
Build files.
Application entry points.
Configuration files.
Infrastructure definitions.
API specifications.
Message schemas.
Tests.
Existing documentation.

Output:

A repository inventory.

Stage 2 — Component Discovery

Identify:

Controllers.
Services.
Repositories.
Interfaces.
Event publishers.
Event consumers.
Message handlers.
Data models.
External clients.
Configuration classes.

Group components by responsibility.

Output:

A component inventory.

Stage 3 — Flow Reconstruction

Trace the important business flows.

Start from an entry point.

Follow the execution path.

Identify:

Method calls.
Data transformations.
Database operations.
API calls.
Message publication.
Message consumption.
Error handling.
Final outputs.

Output:

An end-to-end flow description.

Stage 4 — Integration Discovery

Identify all external interactions.

For each interaction, determine:

Producer.
Consumer.
Transport.
Contract.
Direction.
Purpose.
Configuration.
Failure handling.

Output:

An integration inventory.

Stage 5 — Platform Discovery

Inspect relevant infrastructure definitions.

Examples:

Dockerfiles.
Helm charts.
Kubernetes manifests.
Terraform.
CloudFormation.
SAM templates.
CI/CD pipelines.
Application configuration.

Identify the relationship between application components and infrastructure resources.

Output:

A platform dependency map.

Stage 6 — Architecture Analysis

Identify:

Architecture style.
Design patterns.
Integration patterns.
Data ownership.
Transaction boundaries.
Failure boundaries.
Important design decisions.

Separate verified patterns from possible patterns.

Output:

An architecture analysis.

Stage 7 — Business Context

Review available:

Existing documentation.
Domain terminology.
API names.
Event names.
Business rules.
Acceptance criteria.
Repository descriptions.

Identify the business purpose.

Do not invent missing business information.

Output:

A business context summary and a list of questions requiring confirmation.

Stage 8 — Documentation Generation

Generate the README and supporting architecture documents.

Use the standard structure.

Use consistent terminology.

Include diagrams and source references.

Stage 9 — Validation

Check:

Technical accuracy.
Diagram accuracy.
Component relationships.
Input and output contracts.
Upstream and downstream relationships.
Architecture pattern claims.
Infrastructure relationships.
Source references.
Missing information.

Output:

A validated documentation set.

9. Evidence and Traceability Rules

The LLM must provide traceability for important technical statements.

Use the following evidence categories.

Verified

Directly supported by source code, configuration or infrastructure definitions.

Example:

OrderService calls KafkaTemplate to publish an event.

Evidence:

OrderService.java

Inferred

Supported by multiple observations, but not explicitly established.

Example:

The application appears to use event-driven communication between order and payment processing.

Evidence:

Event publisher, message contract and downstream consumer.

Unknown

The available repository does not establish the answer.

Example:

The business reason for selecting the message retention period is not documented.

Important rules

Never invent source paths.
Never invent class names.
Never invent message fields.
Never assume a consumer exists because a topic exists.
Never assume a topic is consumed by a particular application without evidence.
Never assume deployment configuration matches the current production environment.
Never assume a pattern exists because a framework supports it.
Clearly identify missing evidence.
Distinguish source-code facts from runtime behaviour.
Record assumptions that require human confirmation.

Where practical, reference source paths and line numbers.

For large repositories, maintain an evidence inventory during analysis.

10. Recommended Output Structure for a Large Repository

For a small application, one README may be sufficient.

For a larger system, use a documentation hierarchy.

architecture/
│
├── README.md
│
├── business-context.md
│
├── system-context.md
│
├── architecture-overview.md
│
├── application-flow.md
│
├── component-design.md
│
├── integration-contracts.md
│
├── messaging-architecture.md
│
├── platform-architecture.md
│
├── resilience-and-failure-handling.md
│
├── security-and-observability.md
│
├── architecture-observations.md
│
└── diagrams/
    ├── system-context.mmd
    ├── container-diagram.mmd
    ├── component-diagram.mmd
    ├── class-diagram.mmd
    ├── sequence-diagram.mmd
    └── deployment-diagram.mmd

The root README must remain the entry point.

It must contain:

Application purpose.
Business context.
High-level architecture.
Main processing flow.
Important integrations.
Links to detailed sections.

Do not duplicate the same explanation in multiple files.

11. Quality Criteria

The final documentation must pass the following checks.

Business accuracy

Does it explain why the application exists?
Does it describe the business capability?
Does it distinguish verified benefits from assumptions?

Architecture accuracy

Does it identify system boundaries?
Does it show upstream and downstream systems?
Does it describe the actual communication mechanisms?
Are the architecture patterns supported by evidence?

Application accuracy

Does it identify the important classes?
Does it explain their responsibilities?
Does it show how they interact?
Are the inputs and outputs correct?

Platform accuracy

Does it explain how the application uses infrastructure?
Are Kafka, SQS, Kubernetes and AWS relationships correct?
Does it distinguish application behaviour from platform behaviour?

Diagram quality

Do the diagrams match the written explanation?
Are communication directions clear?
Are synchronous and asynchronous interactions distinguishable?
Are diagrams readable?

Writing quality

Is the main purpose clear?
Are sentences concise?
Is technical terminology consistent?
Is there unnecessary repetition?
Does the document avoid generic technology explanations?

Evidence quality

Can a developer verify important claims?
Are assumptions identified?
Are unknowns clearly stated?
Are source references accurate?

12. Master LLM Prompt

The following prompt is intended for repeated use with different repositories.

It combines the methodology into one reusable instruction.

Copy it into a new conversation when starting an architecture documentation task.

MASTER PROMPT: Source-to-Architecture Documentation

You are acting as a Senior Software Architect, Application Reverse Engineer and Technical Documentation Specialist.

Your task is to analyse the supplied source code and generate high-quality architecture documentation for multiple technical audiences.

The target audiences are:

Software Developers.
Solution Architects.
Enterprise Architects.
Platform Engineers.
SREs.

The objective is to reconstruct and explain the actual system architecture from source-code evidence.

This is not a tutorial.

Do not explain generic technologies such as Kafka, Kubernetes, AWS, Terraform or Spring Boot unless the explanation is directly relevant to how this application uses them.

Focus on the application, its behaviour, its architecture and its business purpose.

A. Documentation principles

Apply the following principles.

1. Pyramid Principle and BLUF

Present the main purpose first.
Organise supporting information logically.
Move from high-level architecture to implementation details.
Avoid introducing implementation details before explaining their purpose.

2. ASD-STE100 — Approximately 80%

Use short sentences.
Use simple language.
Prefer active voice.
Keep essential technical terminology.
Avoid filler and repetition.
Preserve technical accuracy and depth.

3. Diátaxis

Treat this as architecture explanation documentation.

Do not turn the document into a tutorial or generic reference guide.

Use reference tables for contracts, configuration and interfaces where appropriate.

4. C4 Model

Use suitable architecture views:

System Context.
Container.
Component.
Dynamic or Sequence.
Deployment.

Use class diagrams where they help explain implementation relationships.

5. arc42

Use relevant architecture documentation concepts to organise the content.

Do not force unnecessary sections into the document.

6. Feynman Technique

Explain complex interactions in clear language.

Use concrete application examples.

Explain what happens and why it matters.

Do not remove technical depth.

7. Cognitive Load and Progressive Disclosure

Explain the system in layers.
Introduce one major concept at a time.
Keep high-level views separate from detailed implementation views.
Avoid overwhelming the reader with unrelated technical details.

8. Evidence-Based Analysis

Base technical claims on source-code evidence.
Distinguish verified facts, inferences and unknowns.
Do not invent components, relationships, schemas or behaviour.
Identify missing business context.
Do not assume that a technology implies an architecture pattern.

B. Required analysis

Analyse the repository in stages.

Stage 1 — Repository discovery

Identify languages, frameworks, modules, entry points, configuration, infrastructure files, tests and existing documentation.

Stage 2 — Component discovery

Identify important classes, interfaces, services, controllers, repositories, handlers, producers, consumers and external clients.

Group them by responsibility.

Stage 3 — Business flow reconstruction

Trace important flows from entry point to final output.

Identify validation, processing, persistence, transformation, publication, consumption and error handling.

Stage 4 — Integration discovery

Identify upstream systems, downstream systems, APIs, Kafka topics, SQS queues, databases and external dependencies.

Identify producers, consumers and message contracts.

Stage 5 — Platform discovery

Inspect relevant infrastructure and deployment definitions.

Identify how the application uses Kubernetes, AWS and other platform services.

Stage 6 — Architecture analysis

Identify architecture style, patterns, transaction boundaries, data ownership, failure boundaries and design decisions.

Only claim a pattern when evidence supports it.

Stage 7 — Business context

Establish why the application exists using available domain information and existing documentation.

Do not invent business objectives or benefits.

List questions where human confirmation is required.

Stage 8 — Documentation generation

Produce the architecture documentation using the structure below.

Stage 9 — Validation

Check all important claims, diagrams, source references, relationships and contracts against the available evidence.

Do not generate final documentation before completing sufficient analysis.

If the repository is too large to analyse in one pass, work incrementally and maintain an analysis inventory.

C. Required documentation structure

Executive Summary.
Business Context.
System Context.
Architecture Overview.
End-to-End Processing Flow.
Component and Class Design.
Input and Output Contracts.
Integration and Messaging Architecture.
Architecture Patterns.
Platform and Deployment Architecture.
Failure Handling and Resilience.
Security and Observability.
Source Code Navigation.
Architecture Observations.
Evidence and Open Questions.

Adapt the sections to the actual application.

Do not create empty or generic sections.

D. Architecture flow requirements

For every important business flow, explain:

What triggers the flow.
Which component receives the request or event.
What validation occurs.
Which components execute the business logic.
What data is created or transformed.
Where data is persisted.
Which external systems are called.
Which events or messages are published.
Which downstream components consume them.
What the final output is.
How errors and retries are handled.

Distinguish synchronous and asynchronous processing.

Explain where the original transaction ends and where subsequent processing begins.

E. Integration requirements

For each integration, identify:

Attribute	Required information
Producer	Component or external system
Consumer	Component or external system
Transport	REST, Kafka, SQS or other
Contract	Actual request, response or message schema
Direction	Incoming or outgoing
Purpose	Application responsibility
Configuration	Relevant source or infrastructure reference
Failure handling	Verified retry, timeout or error behaviour

F. Architecture pattern requirements

Assess relevant patterns, including:

Event-Driven Architecture.
Saga.
Choreography.
Orchestration.
CQRS.
Transactional Outbox.
Hexagonal Architecture.
Layered Architecture.
Repository.
Adapter.

For each pattern:

Identify the participating components.
Explain the implementation.
Provide supporting evidence.
Explain why the pattern matters.
Identify missing evidence.

Do not claim Saga without evidence of a distributed business transaction and its coordination or compensation behaviour.

Do not claim Transactional Outbox without evidence of the required transactional persistence and publication mechanism.

G. Diagram requirements

Generate suitable Mermaid diagrams.

Prefer diagrams that provide meaningful architectural information.

At minimum, assess whether the following are required:

System Context.
Component.
Sequence.
Deployment.

Use consistent component names.

Label communication mechanisms.

Show direction.

Distinguish synchronous and asynchronous interactions.

Do not invent relationships.

Ensure the diagrams match the written documentation.

H. Evidence requirements

Classify important statements as:

Verified.
Inferred.
Unknown.

Use actual repository paths.

Include line references where available.

Never fabricate source references.

Separate application implementation from infrastructure configuration and runtime behaviour.

I. Output quality

The final document must:

Be suitable for storage in a software repository.
Support developers, architects and platform engineers.
Explain business flow and technical implementation.
Connect application components to platform dependencies.
Use clear and concise technical English.
Avoid generic technology descriptions.
Avoid unnecessary repetition.
Preserve important implementation details.
Identify unknowns instead of guessing.
Provide useful diagrams and source references.

The result must be an architecture document, not a source-code summary.

Most important instruction: Reconstruct the architecture from evidence first. Explain it clearly second. Generate the documentation third.

13. Recommended Working Practice

The master prompt defines the desired result.

For better results, use it with a controlled workflow.

Do not start by asking the LLM to write the README.

Start with analysis.

Use the following sequence:

Step 1 — Understand the repository

Ask the LLM to identify the repository structure, technology stack and important entry points.

Do not generate the README yet.

Step 2 — Reconstruct the main flow

Ask it to trace the main business transaction.

Request the relevant classes, methods, inputs, outputs and external interactions.

Validate this analysis before moving on.

Step 3 — Reconstruct the architecture

Ask it to identify the system boundary, integrations, patterns and infrastructure dependencies.

Request supporting source references.

Step 4 — Generate diagrams

Generate the diagrams from the verified flow and component relationships.

Do not let the LLM independently invent diagram relationships.

Step 5 — Generate the documentation

Use the master prompt.

Generate the README and supporting documents.

Step 6 — Perform an architecture review

Ask the LLM to act as a critical reviewer.

Ask:

Which statements are unsupported?
Which relationships are inferred?
Which flows are incomplete?
Which components are missing?
Which patterns have been incorrectly identified?
Which diagrams do not match the source?
What important information is missing for each audience?

Step 7 — Finalise

Review the business context with a business owner or product owner.

Review architectural claims with an architect.

Review infrastructure behaviour with a platform engineer.

This final review is important because source code alone cannot establish every business decision or runtime fact.

14. Final Principle

Good architecture documentation is not measured by the number of classes, diagrams or pages.

It is measured by how well it helps someone understand the system.

A developer should be able to trace the implementation.

An architect should be able to understand the design and integration boundaries.

A platform engineer should be able to understand the infrastructure dependencies.

All three should be able to follow the same end-to-end business flow.

The LLM must therefore do more than summarise code.

It must reconstruct the relationships between:

Business purpose → Application behaviour → Component interactions → Data flow → Integration architecture → Platform dependencies.

That is the central objective of this framework.

Reconstruct accurately. Explain clearly. Document with evidence.

LLM and how to get max output

Karthi Mahadevan — Sun, 04 Oct 2026 11:46:55 +0000

Use five core techniques and one delivery principle:

BLUF/Pyramid — structure.

ASD-STE100 — language.

Diátaxis — content type.

Feynman — understanding.

Cognitive Load — complexity.

Progressive Disclosure — delivery sequence.

C4 Model — Architecture visualisation

Use different views to explain system context, containers, components and interactions.

Arc42 — Architecture documentation structure

Use its architecture documentation concepts to organise decisions, building blocks, runtime behaviour, deployment and cross-cutting concerns.

Kubernetes Gateway API and my experience

Karthi Mahadevan — Wed, 05 Nov 2025 17:05:01 +0000

https://gateway-api.sigs.k8s.io/
https://kubernetes.io/docs/concepts/services-networking/gateway/

The traditional Kubernetes Ingress resource has served as the primary mechanism for exposing HTTP(S) applications to the outside world. It defines host- and path-based routing rules that an Ingress Controller (like NGINX or AWS ALB Ingress Controller) translates into load balancer configurations.

While effective for simple setups, Ingress has several limitations:

Limited expressiveness — routing rules and filters are basic.

Controller-specific annotations — leading to vendor lock-in.

Difficult separation of concerns — operations and developers share the same configuration surface.

The Gateway API was designed by the Kubernetes networking SIG to address these challenges. It introduces a richer and more modular model:

GatewayClass: Defines the type of underlying infrastructure (e.g., AWS ALB).

Gateway: Represents the actual entry point into the cluster.

Route (e.g., HTTPRoute, TCPRoute, GRPCRoute): Defines routing rules, attached to one or more Gateways.

Service: Still used as the backend target for traffic, just like before.

n an AWS EKS cluster, the AWS Load Balancer Controller can interpret Gateway and Route resources to automatically provision and configure AWS Application Load Balancers (ALBs) or Network Load Balancers (NLBs).

A typical traffic flow looks like this:
Client → (AWS ALB via Gateway) → Gateway (reverse proxy) → HTTPRoute → Service → Pod

This is conceptually similar to the classic:
Client → (AWS ALB) → Ingress Controller → Service → Pod

Main take away for me is
Reduced Vendor Lock-In
Gateway API defines a consistent, cloud-agnostic schema.
The same Gateway and Route manifests can work across AWS, GCP, Azure, or on-prem — you only change the GatewayClass to fit the environment. No more provider-specific annotations cluttering your YAML.

New Python Package manager (UV) and Using Ray

Karthi Mahadevan — Fri, 14 Mar 2025 11:00:56 +0000

While I was in my earlier platform engineer role, one of my teammate said about UV, I didn't pay attention to it. But when I started using it, I realised how fast it is.

While I was reading and learning about Ray , I got to know they use UV internally. Nice.

How we created EKS cluster

Karthi Mahadevan — Fri, 28 Feb 2025 10:26:58 +0000

we used a cluster template.
This is a YAML configuration file for eksctl (a CLI tool for creating and managing Amazon EKS clusters). It’s designed to provision a Kubernetes cluster on AWS with specific networking, IAM, and node group settings.

Templating:
The configuration uses environment variables (e.g., ${EKS_CLUSTER_NAME}, ${VPC_ID}, etc.) to inject values dynamically. This makes it flexible and reusable across different environments (e.g., dev, staging, prod).

Resources That Will Be Created

EKS Cluster:
    A new EKS cluster is created with a specified name, version, and region.
    Cluster metadata includes tags for identification and categorization (such as Environment, Application, and Role).

VPC & Networking:
    VPC Configuration: Uses a pre-existing VPC (via ${VPC_ID}) with defined CIDRs.
    Subnets:
        Private Subnets: Three subnets (private-one, private-two, private-three) for worker nodes.
        Public Subnets: Three public subnets for resources that might need internet connectivity.
    Endpoint Access: Cluster endpoints are configured for private access only (public access is disabled).

IAM Configuration & Service Accounts:
    OIDC: The cluster is configured to work with OIDC, which is essential for associating IAM roles with Kubernetes service accounts.
    Service Accounts: Multiple service accounts are defined for specific roles:
        Cluster Autoscaler: For automatically adjusting the number of nodes.
        EBS CSI Controller: For managing persistent storage volumes.
        AWS Load Balancer Controller: To manage AWS load balancers.
        External DNS, CloudWatch Exporter, Cert Manager, Secrets Manager, Parameter Store, and others: Each with tailored IAM policies (either pre-defined “wellKnownPolicies” or custom policies via inline JSON).

Addons:
    The vpc-cni addon is included with an attached policy for managing networking on EKS worker nodes.

Managed Node Groups:
    Core Node Group:
        Designed to run primary workloads.
        Uses Bottlerocket AMI for enhanced security and performance.
        Configured with multiple instance types, private networking, EBS volumes (with encryption), scaling parameters (min, desired, max sizes), and specific labels/tags (including those for cluster autoscaler integration).


CloudWatch Logging:
    Enables cluster logging for all available log types, which is crucial for monitoring and troubleshooting.

Patterns and Structure

Modular Design:
The configuration cleanly separates different concerns:
    Cluster and VPC settings (networking, endpoints)
    IAM and Service Accounts (roles, policies)
    Node Groups (resource definitions, scaling, labels)
    Addons and Logging

Environment-Driven Templating:
Using placeholders for values allows this file to be easily adapted to different environments or clusters by simply setting environment variables during deployment.

Best Practices:
    Security: Private endpoints and encrypted volumes.
    Scalability: Defined node groups with autoscaling tags.
    Observability: CloudWatch logging is enabled for full cluster visibility.
    Separation of Responsibilities: Distinct IAM roles for different services minimize permissions and adhere to least-privilege principles.

While configuration sets up the core EKS cluster, IAM roles, managed node groups, and logging, the following additions can help to achieve a more production-ready environment:

Secrets Encryption: Encrypt Kubernetes secrets using a customer-managed KMS key.
Network Policies: Enforce pod-to-pod and pod-to-external communication restrictions.
Enhanced RBAC and Audit Logging: Refine RBAC rules and enable detailed audit logs.
Refined Security Groups: Use restrictive security groups, including at the pod level.
AWS Integrations: Leverage CloudTrail, GuardDuty, and managed add-ons for improved security and observability.
Backup/DR Planning: Implement backup strategies to safeguard your cluster and workloads.

How I use Cloudwatch and fluentbit

Karthi Mahadevan — Fri, 28 Feb 2025 10:13:47 +0000

Fluent Bit is deployed as a DaemonSet (a POD that runs on every node of the cluster). When Fluent Bit runs, it will read, parse and filter the logs of every POD and will enrich log with some more information.

This will enable container (open policy agent) logs available in aws cloudwatch. The log group name where the logs will be /aws/containerinsights/${CLUSTER_NAME}/application ; here CLUSTER_NAME will be "tooling" for prod.

fluentbit.yaml will have

Here’s how the ClusterRole, ClusterRoleBinding, and ConfigMap are linked and their roles in this configuration:

ClusterRole

The ClusterRole named fluent-bit-role defines the permissions that Fluent Bit requires to interact with Kubernetes resources. It specifies:

Non-resource URL access: Allows access to /metrics with the get verb.
Resource access: Grants permissions to namespaces, pods, pods/logs, nodes, and nodes/proxy with the get, list, and watch verbs.

ClusterRoleBinding

The ClusterRoleBinding named fluent-bit-role-binding links the ClusterRole to a subject, enabling Fluent Bit to use the permissions.

Subject: The ServiceAccount named fluent-bit in the logging namespace.
RoleRef: Specifies that the binding refers to the fluent-bit-role ClusterRole.

This linkage ensures that the fluent-bit ServiceAccount has the necessary permissions to collect logs and interact with Kubernetes objects.

ConfigMap

The ConfigMap named fluent-bit-config provides configuration data for Fluent Bit. It contains:

Fluent Bit configurations: Specifies input sources (e.g., application logs), filtering (e.g., Kubernetes metadata), and output destinations (e.g., CloudWatch Logs).
Parser definitions: Defines parsers for structured log formats, such as docker and syslog.

How They Are Linked

Permissions for Log Access:
- The fluent-bit DaemonSet runs pods using the fluent-bit ServiceAccount.
- The fluent-bit-role-binding binds the fluent-bit-role ClusterRole to the fluent-bit ServiceAccount.
- This setup allows Fluent Bit to access logs, Kubernetes metadata, and node information.
Configuration Data:
- The DaemonSet mounts the fluent-bit-config ConfigMap to /fluent-bit/etc/ within its pods.
- Fluent Bit reads configurations from this directory to process logs according to the defined rules.

This structure ensures Fluent Bit operates with the correct permissions and configurations in a Kubernetes environment. Let me know if you need further clarification or adjustments!