DEV Community: Parth Sarthi Sharma

Golden Paths: Why Platform Engineering Matters More Than Your Repo Strategy

Parth Sarthi Sharma — Sun, 04 Oct 2026 04:12:13 +0000

A repo strategy decides where your code lives. A golden path decides how fast your team can ship it.

A few days ago I wrote about monorepo vs polyrepo, and one line in that article kept coming back to me:

"The repository structure isn't necessarily the problem. Lack of platform engineering is."

This article is that sentence, expanded.

Because here's a pattern I keep seeing: a team spends weeks in architecture meetings debating repo boundaries, naming conventions, ownership models — and then a brand-new service still takes three weeks to reach production. Not because the code was hard. Because setting up CI, security scanning, secrets, logging, and a deployment pipeline from scratch was hard.

The repo was never the bottleneck. The paved road was missing.

1. The Question We Keep Asking Wrong

Most platform conversations start with:

"Should this be a shared library, a shared repo, a shared pipeline?"

The more useful question is:

"When an engineer wants to build and ship something new, what is the fastest, safest way for them to do it — without reinventing infrastructure every time?"

That question has a name: a golden path.

2. What a Golden Path Actually Is

A golden path (sometimes called a "paved road") is the opinionated, well-supported, default way of doing something inside an engineering organisation.

New service needed
        ↓
Golden path exists?
        ↓
   Yes ──────────────► Use the template
                        CI/CD pre-wired
                        Security scans included
                        Observability included
                        Deploy in hours

   No ───────────────► Build pipeline from scratch
                        Negotiate access
                        Copy-paste from another team
                        Deploy in weeks

It's not a mandate. It's not the only way. It's the way that requires the least thought, because the platform team has already made the hard decisions for you — and tested them.

3. A Familiar Scenario

Imagine two engineers, same company, same week, both asked to ship a new internal service.

Engineer A (no golden path):

Copies a Dockerfile from a service that "looked similar"
Manually requests CI runners
Asks three people how secrets are supposed to be managed
Finds out the logging format is inconsistent with the rest of the org
Ships in week 3, with gaps the security team finds a month later

Engineer B (golden path exists):

Runs platform create-service
Gets a repo with CI, linting, test scaffolding, a Dockerfile, and a deployment manifest already wired to the org's standards
Observability and secrets are configured by default
Ships by day 2

Same company. Same skill level. Different experience entirely — because of infrastructure, not ability.

This is the gap platform engineering exists to close.

4. Why "Just Pick Monorepo or Polyrepo" Doesn't Fix This

A monorepo makes code visible. A polyrepo makes ownership explicit. Neither one, by itself, gives you:

a pre-approved way to stand up a new service
consistent security scanning
consistent CI/CD behaviour
a known, supported way to deploy

You can have a perfectly organised monorepo and still have every team building CI pipelines from scratch inside it. You can have 80 polyrepos and still have them all following the exact same paved road, because a platform team maintains the template they were all created from.

Repo strategy is about where code lives.
Platform engineering is about how code gets from an engineer's laptop to production, safely, repeatedly, and without reinventing the wheel every time.

5. What This Looks Like in Practice

This isn't a new idea — several engineering orgs have publicly written about building exactly this kind of paved road:

Spotify popularised the term "golden path" itself, and open-sourced Backstage as a developer portal so engineers could discover services, templates, and documentation in one place instead of hunting through wikis.
Netflix has talked publicly about its "paved road" philosophy — teams are free to go off-road, but the paved road is deliberately the path of least resistance, kept effortless on purpose.
Uber and several other large engineering orgs built internal developer portals specifically because the cost of onboarding a new microservice had quietly become one of their biggest sources of engineering time loss.

The common thread isn't the tooling. It's the philosophy: make the right way the easy way.

6. The Anatomy of a Golden Path

A good golden path usually bundles together:

Golden Path
   │
   ├── Service template (language, folder structure, linting)
   ├── CI pipeline (build, test, security scan — pre-wired)
   ├── Deployment manifest (Kubernetes/Terraform defaults)
   ├── Observability (logging, metrics, tracing — on by default)
   ├── Secrets & access management (no manual requests)
   └── Documentation (how it works, how to deviate)

Notice the last item: how to deviate. A golden path that has no escape hatch isn't a golden path — it's a mandate wearing a nicer name. More on that below.

7. Golden Path vs Guardrail vs Mandate

These three get confused constantly, so it's worth separating them:

Concept	What it means	Example
Golden path	The easiest default. Optional but heavily incentivised.	A pre-built service template with CI/CD included
Guardrail	An automated check that prevents unsafe behaviour.	A pipeline that blocks deploys without passing security scans
Mandate	A rule enforced regardless of context.	"All services must be written in Go"

Healthy platform engineering leans on golden paths and guardrails. Mandates, used too liberally, create resentment and workarounds — engineers will always find a way around a rule they don't believe in.

8. The Hidden Cost of Not Having a Golden Path

When there's no paved road, every team quietly builds their own:

Team A → Jenkins, custom Dockerfile, manual secrets
Team B → GitHub Actions, different base image, different logging format
Team C → GitLab CI, yet another deployment pattern

Nobody did anything wrong. Each team solved their own problem reasonably. But now:

Security has to audit N different pipelines instead of one
Incident response is slower because every service logs differently
Onboarding a new engineer means re-learning infrastructure per team
Upgrading a shared dependency (say, a security patch) means N different manual rollouts

This is the same "hidden coupling" problem from the repo-strategy article — just moved from code to infrastructure.

9. Treat the Platform Like a Product

The single biggest mindset shift that makes platform engineering work: the platform team's users are other engineers, and the platform is a product they chose to use — not a policy they're forced to follow.

That means:

Golden paths should be genuinely easier than the alternative, not just officially sanctioned
Platform teams should gather feedback like a product team does — adoption metrics, friction points, feature requests
Deviating from the golden path should be possible, just slightly more effort — never blocked outright without a very good reason

If engineers are opting out of the golden path en masse, that's not a compliance problem. That's a product-market-fit problem.

10. A Practical Decision Framework

Instead of asking "do we need a platform team," ask:

Question	Signal
Are multiple teams solving the same infra problem independently?	Build a golden path
Does onboarding a new service take days instead of hours?	Build a golden path
Are security/compliance checks inconsistent across services?	Add guardrails
Is the platform team becoming a bottleneck for every deploy?	Loosen the mandate, strengthen self-service
Are engineers bypassing the golden path regularly?	Treat it as product feedback, not a discipline issue
Is the org under ~20 engineers?	A lightweight template may beat a full platform team

Platform engineering, like repo strategy, is a set of trade-offs — not a universal answer.

11. What I'd Avoid

Anti-pattern 1: Building the platform before anyone asks for it
Platform teams that build in isolation from product teams tend to build the wrong thing, beautifully.

Anti-pattern 2: No escape hatch
If the golden path is the only path, it becomes a bottleneck the moment it doesn't fit an edge case — and it always eventually doesn't.

Anti-pattern 3: Treating the platform as "done"
A golden path that isn't updated as the org's needs evolve becomes the exact legacy problem it was built to prevent.

Anti-pattern 4: Measuring success by mandate compliance
"90% of services use our template" is a vanity metric if engineers are using it because they have to, not because it's genuinely the fastest option.

12. How to Start, Practically

If there's no golden path today, you don't need a platform team of twenty to begin:

Pick the single most repeated piece of setup pain (usually CI/CD scaffolding or service bootstrapping)
Build one opinionated template for it
Use it yourself on the next real service — don't theorise it
Get one other team to adopt it and listen hard to their friction
Only then formalise it into a platform with documentation and ownership

Golden paths earn trust by being used, not by being announced.

13. The Takeaways

Repo strategy and platform engineering solve different problems. One is about code organisation, the other is about developer experience.
A golden path is the easiest default, not a mandate. It should be fast enough that engineers want to use it.
Guardrails, not rules, keep things safe at scale. Automate the check, don't just write the policy.
The platform is a product. Its users can, and will, opt out if it isn't good.
Start small. One good template beats a grand, unused platform roadmap.

Final Thought

The monorepo vs polyrepo debate gets attention because it's visible — you can see it in the folder structure. Platform engineering is invisible when it's working, which is exactly why it's underinvested.

But ask any engineer who has shipped a service in three hours on a well-paved road, and then shipped one in three weeks without it — they'll tell you which decision actually mattered more.

Choose your repo strategy deliberately. But build the road before you argue about where it should lead.

This is the second article in a series on engineering decisions that look architectural but are really about people and workflow. First one was on monorepo vs polyrepo — next one's on single-agent vs multi-agent systems, and it turns out the same boundary questions show up there too.

The Architecture of AI-Native Software Teams: What Changes When AI Writes the Code?

Parth Sarthi Sharma — Mon, 28 Sep 2026 09:53:31 +0000

AI is making code cheaper to produce. That doesn't make architecture less important. It makes good architecture more valuable.

A few years ago, the limiting factor for many engineering teams was simple:

How quickly can we write the code?

Today, that question is changing.

AI coding tools can already generate features, write tests, fix bugs, refactor code, update documentation and work through GitHub pull requests with increasing autonomy. GitHub's coding agent, for example, can take an issue, work in an isolated development environment, modify the code, run tests and open a pull request for human review.

Tools such as Claude Code are also moving beyond autocomplete toward agents that can navigate a codebase, edit multiple files, execute commands, test and debug software.

So here's the question I think engineering leaders should be asking:

If producing code is no longer the primary bottleneck, what becomes the bottleneck?

My answer:

Context, architecture, verification and decision-making.

And that changes how we should design software teams.

1. The Old Software Engineering Loop

The traditional development loop looks roughly like this:

Requirement
    ↓
Engineer understands requirement
    ↓
Design
    ↓
Write code
    ↓
Write tests
    ↓
Code review
    ↓
CI/CD
    ↓
Production

A significant amount of engineering time is spent on implementation.

Now introduce an AI coding agent:

Requirement
    ↓
Engineer + AI
    ↓
Design
    ↓
AI implements
    ↓
AI writes tests
    ↓
AI runs validation
    ↓
Engineer reviews
    ↓
CI/CD
    ↓
Production

The implementation step becomes dramatically cheaper.

But notice something interesting.

The human still needs to answer:

What are we actually building?
What should the architecture look like?
What constraints matter?
What should the agent not touch?
Is this API boundary correct?
Is this database model correct?
Is this security model acceptable?
Does this code belong here?
What happens when this fails?
What happens at 10x scale?
What happens during partial failure?

The human role is moving up the abstraction stack.

2. AI Doesn't Remove Engineering Work

This is where I think some AI discussions go wrong.

The assumption is:

AI writes code faster
        ↓
Software engineering becomes easier

The more realistic model is:

AI writes code faster
        ↓
More code gets produced
        ↓
More code needs validation
        ↓
More architectural decisions become important
        ↓
More system-level risks become possible

DORA's recent research describes AI as an amplifier of an organisation's existing strengths and weaknesses. Its 2026 analysis also found that the time saved during initial code creation is frequently reallocated to auditing and verification.

That's a really important signal.

The bottleneck may simply move.

3. The New Engineering Bottleneck

Imagine an engineer could previously produce:

100 lines/day

and an AI-assisted workflow can produce:

1,000 lines/day

That sounds like a 10x improvement.

But what if your team can only effectively review:

300 lines/day

You haven't created a 10x engineering organisation.

You've created a review queue.

                  AI
                   │
                   ↓
             ┌──────────┐
             │  Code    │
             │ generated│
             └────┬─────┘
                  ↓
        ┌───────────────────┐
        │ Human verification│
        └─────────┬─────────┘
                  ↓
              Production

This is one of the most important shifts I see coming:

Code generation is becoming cheaper. Code verification is becoming more valuable.

4. The New Division of Labour

I don't think the future is:

Humans → code
AI → code

It's closer to:

Humans own

Intent
Architecture
Boundaries
Business rules
Security decisions
Trade-offs
Risk
Quality bar
Production accountability

AI agents own more of

Implementation
Boilerplate
Tests
Refactoring
Documentation
Dependency upgrades
Migration scripts
Code exploration
Bug investigation

This doesn't mean AI should blindly own the second list.

It means those are areas where delegation can often provide high leverage.

The interesting engineering problem becomes:

How much autonomy should we give the agent for each type of change?

5. Not All Code Changes Should Have the Same Autonomy

This is one of the practical patterns I would introduce in an AI-native engineering team.

Classify changes by risk.

Level 1 — Low risk

Examples:

Documentation
Unit tests
Formatting
Simple refactoring
Boilerplate
Non-production tooling

AI can potentially operate with high autonomy.

Level 2 — Medium risk

Examples:

Business logic
API changes
Database queries
Performance changes
Dependency upgrades

AI can implement.

Human reviews carefully.

Level 3 — High risk

Examples:

Authentication
Authorisation
Payment logic
Data migrations
Encryption
Infrastructure
Production configuration
Security controls

AI can assist heavily.

But humans should retain strong approval and verification controls.

The workflow becomes:

                 AI autonomy
                     ↑
                     │
Low risk ────────────┤ High autonomy
                     │
Medium risk ─────────┤ Human review
                     │
High risk ───────────┤ Strong human control
                     │
                     ↓
                 Production

The goal isn't:

"Let AI do everything."

It's:

"Give AI as much autonomy as the risk model allows."

6. Architecture Becomes the Agent's Constraint System

Here's an interesting change.

Humans have always relied on architecture to constrain developers.

For example:

Controller
    ↓
Service
    ↓
Repository
    ↓
Database

Now that architecture also constrains AI agents.

This becomes extremely important.

If an agent sees:

services/
├── payments/
├── orders/
├── customers/
└── shared/

but there are no clear boundaries, it may make perfectly valid code changes that are architecturally wrong.

For example:

payments-service
        ↓
directly queries
        ↓
customers database

The code might compile.

Tests might pass.

The architecture is still wrong.

So architecture needs to become increasingly machine-readable.

7. Make Architecture Explicit

One practical pattern is to put architectural context directly into the repository.

For example:

repository/
│
├── README.md
├── ARCHITECTURE.md
├── ADR/
│   ├── 001-service-boundaries.md
│   ├── 002-database-choice.md
│   └── 003-event-driven-integration.md
│
├── services/
│
├── libraries/
│
├── tests/
│
└── docs/

Now the agent has something better than:

"Figure out how this system works."

It has explicit context.

For example:

Payments service

Owns:
- Payment lifecycle
- Payment state
- Payment reconciliation

Does not own:
- Customer identity
- Product catalogue

Integration:
- Customer information via Customer API
- Events published through Event Bus

Never:
- Access another service's database directly

This isn't documentation for humans only anymore.

It becomes part of the agent's operating context.

8. Your README Is Becoming Part of the Development Interface

We've historically treated documentation as something developers read.

With coding agents, documentation increasingly becomes something agents consume before making changes.

This changes what good documentation looks like.

Instead of:

This service handles payments.

we want:

# Payment Service

## Responsibilities

Owns payment lifecycle and reconciliation.

## Boundaries

DO:
- Create payment
- Authorise payment
- Capture payment
- Refund payment

DO NOT:
- Read customer database
- Modify order state directly
- Store authentication credentials

## Dependencies

Customer information:
Customer API

Order information:
Order API

## Testing

Run:
./gradlew test

Integration tests:
./gradlew integrationTest

This is useful to humans.

It is also much more useful to agents.

9. The Codebase Needs a "Constitution"

This is one of the patterns I think will become increasingly common.

Create a small set of explicit engineering rules.

For example:

ENGINEERING_RULES.md

Containing things like:

1. Services must not access another service's database.

2. Public APIs require backward compatibility.

3. Database migrations must be backward compatible.

4. All external calls require timeouts.

5. Retries must use bounded exponential backoff.

6. New APIs require contract tests.

7. Authentication logic must not be implemented inside
   individual business services.

8. Production infrastructure changes require human approval.

Now an AI agent doesn't need to infer all of these rules from thousands of lines of code.

You've made the constraints explicit.

The industry is already moving toward this idea of giving agents structured procedural and organisational context. Anthropic, for example, describes "Agent Skills" as a way to package instructions, scripts and resources so agents can perform specialised work more reliably.

10. Tests Become More Important Than Ever

There's another major shift.

Previously:

Tests protect us from developer mistakes.

In an AI-native workflow:

Tests become executable specifications for agents.

Consider:

Requirement:

A payment must never be captured twice.

A human can understand that.

But a much stronger contract is:

@Test
void shouldNotCapturePaymentTwice() {
    ...
}

Now the agent has a deterministic boundary.

This creates a powerful loop:

Requirement
     ↓
Architecture
     ↓
Tests
     ↓
AI implementation
     ↓
Run tests
     ↓
Fix
     ↓
Run tests again

The better your tests are, the more autonomy you can safely give the agent.

This is one reason coding agents tend to work better in well-tested codebases. GitHub explicitly highlights well-tested repositories as a strong environment for coding-agent tasks.

11. CI/CD Becomes an AI Safety Boundary

In traditional development:

Developer
   ↓
PR
   ↓
CI
   ↓
Review
   ↓
Merge

With agents:

Human
  ↓
Task
  ↓
AI agent
  ↓
Code
  ↓
Tests
  ↓
Security checks
  ↓
Policy checks
  ↓
PR
  ↓
Human approval

CI is no longer just:

"Does the code compile?"

It becomes an executable governance layer.

For example:

PR
 │
 ├── Compile
 ├── Unit tests
 ├── Integration tests
 ├── Contract tests
 ├── Static analysis
 ├── Dependency scan
 ├── Secret scan
 ├── Security tests
 ├── Architecture rules
 └── Policy checks

This is how you allow agents to move faster without removing engineering controls.

12. Don't Give Agents Production Access by Default

This sounds obvious.

But agentic workflows change the threat model.

An agent that can:

read code
write code
run commands
access network
modify infrastructure
deploy software

has a very different risk profile from autocomplete.

Modern agent tooling is therefore increasingly introducing sandboxing, scoped credentials and filesystem/network boundaries. Anthropic, for example, describes sandboxing Claude Code with filesystem and network isolation as part of making autonomous coding safer.

A practical model is:

Developer Agent
       │
       ├── Read repository
       ├── Modify branch
       ├── Run tests
       └── Build artifacts
              │
              ↓
         Human approval
              │
              ↓
          Deployment

Not:

AI
 ↓
Production

13. Small PRs Become Even More Important

Imagine an agent generates:

27 files
4,000 lines changed
3 new dependencies
2 database migrations

Technically impressive.

Practically horrible to review.

The human reviewer is now the bottleneck.

Instead:

Task
 ↓
Small change
 ↓
Tests
 ↓
PR
 ↓
Review
 ↓
Next change

Small changes make:

reviews easier
failures easier to diagnose
rollbacks safer
AI mistakes easier to detect
context easier to maintain

AI gives us the ability to generate huge changes.

Engineering discipline should prevent us from automatically accepting them.

14. Context Is Becoming an Engineering Resource

Here's another subtle change.

An AI agent is only as effective as the context it can access.

Consider:

Task:
"Add customer notifications."

That's almost useless.

Compare:

Goal:
Send an email when a payment succeeds.

Architecture:
Notifications are owned by notification-service.

Constraints:
- Payment service publishes PaymentCaptured event.
- Notification service consumes the event.
- Payment service must not call the notification service synchronously.
- Email delivery must be idempotent.

Acceptance criteria:
- Duplicate events must not send duplicate emails.
- Failed delivery must retry.
- Retry must be bounded.

The second task is dramatically easier for both humans and agents.

This leads to a new engineering skill:

Context engineering for software development.

Not prompt tricks.

Actual engineering context:

architecture
domain vocabulary
APIs
constraints
tests
examples
ADRs
operational knowledge
security rules

15. The New Staff Engineer Job

This is where I think the role of Staff and Principal Engineers changes significantly.

The question becomes less:

"How do I write this code?"

And more:

"How do I create a system where humans and agents can safely produce good software?"

That means investing in:

Architecture

Clear boundaries.

Developer experience

Easy local setup.

Documentation

Machine-readable context.

Testing

Strong executable specifications.

CI/CD

Automated verification.

Security

Least privilege and isolation.

Observability

Fast feedback from production.

Engineering standards

Explicit constraints.

Platform engineering

Reusable capabilities for every team.

In other words:

Staff engineering becomes increasingly about designing the environment in which software gets produced.

16. The Architecture of an AI-Native Team

Putting this together:

                    HUMAN
                      │
                      │
              Intent / Architecture
                      │
                      ↓
             ┌─────────────────┐
             │   AI Agent(s)   │
             └────────┬────────┘
                      │
        ┌─────────────┼──────────────┐
        ↓             ↓              ↓
     Code           Tests       Documentation
        │             │              │
        └─────────────┼──────────────┘
                      ↓
                 CI / Policy
                      │
        ┌─────────────┼─────────────┐
        ↓             ↓             ↓
     Security      Architecture   Quality
      checks          checks       checks
        │             │             │
        └─────────────┼─────────────┘
                      ↓
               Human Approval
                      │
                      ↓
                 Production
                      │
                      ↓
              Observability
                      │
                      ↓
                  Feedback

Notice what's missing?

A human manually writing every line.

The human is instead controlling:

intent → architecture → constraints → verification → risk → outcome

17. What Happens to Junior Engineers?

This is an uncomfortable question.

If AI can handle:

boilerplate
basic CRUD
straightforward tests
simple refactoring
documentation
basic debugging

then some traditional entry-level tasks become less valuable as learning mechanisms.

That doesn't mean junior engineers become unnecessary.

It means the learning path has to change.

A junior engineer shouldn't only learn:

How to write CRUD APIs

They should increasingly learn:

Why this API exists
Why this boundary exists
How failure happens
How data flows
How systems behave under load
How to test assumptions
How to review generated code
How to reason about security
How to operate software

The ability to judge code becomes more important than the ability to produce large amounts of code.

18. Measure the Right Things

One of the easiest mistakes is measuring AI adoption by:

Lines of code generated

or:

Number of AI prompts

Those aren't meaningful engineering outcomes.

I'd rather measure:

Lead time for change
Change failure rate
Review time
PR size
Defect rate
Rollback rate
Incident rate
Test coverage
Developer satisfaction
Time from idea → production

And perhaps one of the most interesting metrics:

How much human attention does each production change require?

Because that's potentially the new scarce resource.

19. A Practical AI-Native Development Workflow

If I were introducing agentic development into an engineering team tomorrow, I'd start relatively conservatively.

Step 1 — Give the agent context

README
ARCHITECTURE
ADRs
ENGINEERING_RULES
API contracts

Step 2 — Give it a bounded task

Bad:

"Improve the payment system."

Better:

"Add idempotency handling to POST /payments.

Do not change the API contract.

Use the existing IdempotencyStore.

Add unit and integration tests.

Do not modify database schema."

Step 3 — Let the agent implement

Agent
 ↓
Code
 ↓
Tests

Step 4 — Run automated verification

Build
Tests
Security
Architecture
Contracts

Step 5 — Human reviews the important things

Not:

"Did the AI write valid Java?"

But:

"Did it implement the correct behaviour?"

and:

"Did it preserve the architecture?"

Step 6 — Merge small changes

Keep the feedback loop tight.

20. The Biggest Anti-Pattern: AI + Bad Architecture

Here's the scenario I'm most concerned about.

You have:

Poor boundaries
+
Poor tests
+
Poor documentation
+
AI coding agent

What happens?

The AI doesn't magically fix the architecture.

It can actually make the problem worse faster.

Bad architecture
       ↓
AI generates more code
       ↓
More coupling
       ↓
More complexity
       ↓
Harder reviews
       ↓
More technical debt

This is exactly why the "AI makes developers 10x" narrative needs context.

AI can amplify good engineering.

It can also amplify bad engineering.

DORA's research makes essentially this broader point: AI tends to amplify the underlying capabilities and weaknesses of the organisation using it.

21. So What Actually Changes?

I don't think the fundamental principles of software engineering disappear.

They become more important.

We still need:

Good boundaries
Good APIs
Good data models
Good tests
Good observability
Good security
Good failure handling
Good operational practices

But now we also need:

Machine-readable architecture
Explicit engineering rules
Agent-friendly documentation
Risk-based autonomy
Automated verification
Strong CI/CD guardrails
Scoped agent permissions
Small change sets

That's what I would call an AI-native engineering environment.

22. My Rule of Thumb

If I had to summarise the whole article in one sentence:

When AI makes implementation cheap, invest more heavily in the things AI cannot safely decide for you: intent, architecture, constraints, verification and accountability.

Or, even more simply:

Before AI:

Code was expensive.
Review was cheaper.

With AI:

Code is cheaper.
Good judgment is expensive.

And that changes where engineering organisations should invest.

Final Thought

I don't think the future of software engineering is:

Humans stop coding and AI writes everything.

I think it looks more like:

Human
  ↓
Defines intent
  ↓
Designs boundaries
  ↓
Sets constraints
  ↓
AI implements
  ↓
AI validates
  ↓
Automated systems verify
  ↓
Human judges
  ↓
Production learns
  ↓
Architecture evolves

The interesting competitive advantage won't simply be who has access to the best coding model.

Most teams will have access to very capable models.

The advantage will increasingly come from:

Who has the best architecture, context, engineering constraints, feedback loops and ability to safely turn AI-generated code into production outcomes.

That's the real architecture of an AI-native software team.

Monorepo vs Polyrepo: What Actually Matters When You’re Building at Scale

Parth Sarthi Sharma — Sat, 26 Sep 2026 08:21:16 +0000

The repository strategy is not really a Git decision. It is an architecture, ownership, and team-boundary decision.

When a team starts building multiple services, one question usually appears sooner or later:

Should we keep everything in a monorepo, or should every service have its own repository?

The discussion often turns into a familiar list:

Monorepo = easier collaboration
Polyrepo = better isolation
Monorepo = easier refactoring
Polyrepo = better team autonomy

All of these can be true.

And yet, they don't answer the question that actually matters:

Which repository structure will make our engineers more effective as the system and organisation grow?

After working with distributed systems, microservices, shared libraries, CI/CD pipelines and multiple engineering teams, I've found that the repository itself is rarely the root problem.

The more important questions are:

Who owns the code?
How often do services change together?
How tightly coupled are the teams?
How are dependencies managed?
How expensive is CI/CD?
How easy is cross-service refactoring?
Where do security and access boundaries exist?
How much autonomy do teams actually need?

This article looks at monorepo vs polyrepo from that perspective.

1. First: What Are We Actually Deciding?

At first glance, the decision seems simple.

Monorepo

company/
├── services/
│   ├── orders/
│   ├── payments/
│   ├── customers/
│   └── notifications/
├── libraries/
│   ├── auth/
│   ├── logging/
│   └── common/
├── frontend/
└── infrastructure/

One repository contains multiple applications, services and shared components.

Polyrepo

orders-service       → repository
payments-service     → repository
customers-service    → repository
notifications        → repository
common-library       → repository
frontend             → repository
infrastructure       → repository

But Git repository boundaries don't necessarily equal architectural boundaries.

A system can have:

50 repositories
20 deployable services
5 teams
shared libraries
tightly coupled release processes

and still effectively behave like one system.

Likewise, a monorepo can contain completely independent applications owned by different teams.

So the real decision is not:

"How many Git repositories should we have?"

It is:

"Where should we place boundaries between teams, code, dependencies and delivery pipelines?"

That's a much more useful question.

2. Monorepo: What It Gives You

A monorepo provides one major capability that becomes extremely valuable as systems evolve:

You can see and change the system as one codebase.

Imagine we have:

orders-service
payments-service
customer-service

and we need to change an API contract.

In a monorepo:

Change API contract
       ↓
Update producer
       ↓
Update consumers
       ↓
Compile/test everything affected
       ↓
Review one change
       ↓
Merge

The compiler and CI system can help us identify the blast radius.

This becomes particularly valuable for:

API contract changes
shared libraries
security changes
large refactoring
framework upgrades
dependency upgrades
consistent engineering standards

For example:

Java 17 → Java 21

        ↓

orders
payments
customers
notifications
reporting

A monorepo makes it much easier to discover what needs to change.

3. The Hidden Superpower: Refactoring

This is one of the biggest practical advantages of a monorepo.

Suppose we have:

CustomerService.getCustomer()

used by:

Orders
Payments
Invoices
Reporting
Notifications

Now imagine changing:

getCustomer()

to:

getCustomer(CustomerId id)

In a monorepo, the IDE/compiler/test system can often identify the impact immediately.

With multiple repositories, you may need to:

Change the shared library
Publish a new version
Update repository A
Update repository B
Update repository C
Wait for their CI pipelines
Deal with version compatibility
Eventually remove the old API

The technical change may be tiny.

The organisational change can be much larger.

4. But Monorepos Are Not Free

A monorepo can become painful if we don't invest in engineering tooling.

Imagine this:

monorepo
│
├── 100 services
├── 20 libraries
├── 10 frontend applications
└── millions of lines of code

A developer changes one line in:

payments-service

and CI responds with:

Build everything
Test everything
Deploy everything

Now the repository becomes the bottleneck.

This is where people sometimes incorrectly conclude:

"Monorepos don't scale."

The more accurate conclusion is:

Naive monorepo tooling doesn't scale.

A serious monorepo needs:

dependency graphs
incremental builds
affected-test detection
remote build caching
parallel CI
clear ownership
CODEOWNERS
good developer tooling

The goal should be:

Developer changes payments
          ↓
Dependency graph
          ↓
Identify affected components
          ↓
Build/test only what matters

Not:

Developer changes payments
          ↓
Build the entire company

5. Polyrepo: Why Teams Choose It

Polyrepo usually starts with a very reasonable requirement:

"This team should be able to own and deploy its service independently."

For example:

Team A
 └── orders-service

Team B
 └── payments-service

Team C
 └── notifications-service

Each team gets:

its own repository
its own CI pipeline
its own deployment lifecycle
its own permissions
its own release cadence

This can work extremely well.

Especially when services have genuinely independent ownership.

6. The Hidden Cost of Polyrepo

The problem appears when repositories are technically separate but operationally coupled.

Imagine:

orders-service
       ↓
payments-service
       ↓
customer-service
       ↓
shared-library

Now a breaking change happens in the shared library.

The change becomes:

shared-library v2
       ↓
orders → update
payments → update
customer → update
reporting → update
notifications → update

Suddenly we have a distributed coordination problem.

The repositories are separate.

The architecture isn't.

This can lead to:

dependency version drift
duplicated code
inconsistent frameworks
inconsistent security fixes
incompatible API versions
difficult cross-repository refactoring
pull-request coordination across teams

7. One of the Biggest Lessons: Microservices ≠ Polyrepo

This is worth calling out explicitly.

A common architecture evolution looks like this:

Monolith
   ↓
Microservices
   ↓
One repository per microservice

The last step is not mandatory.

You can absolutely have:

                 Monorepo
                    │
       ┌────────────┼────────────┐
       ↓            ↓            ↓
    Orders       Payments     Customers
       │            │            │
       └────── independently ────┘
              deployable

The deployment boundary and repository boundary can be different.

That's an important architectural distinction.

8. Real Engineering Scenario: Shared Libraries

Let's look at a common example.

Suppose we have:

common-auth
common-logging
common-observability
common-http

In a polyrepo architecture, these often become separate repositories.

Then teams consume versions:

common-auth: 1.4.2
common-logging: 3.1.0
common-http: 2.7.1

Over time:

Service A → auth 1.4
Service B → auth 1.5
Service C → auth 1.3
Service D → auth 1.2

Now the organisation has effectively created multiple versions of the same engineering standards.

This isn't automatically wrong.

Sometimes independent versioning is exactly what we want.

But we should recognise the trade-off.

9. A Useful Rule for Shared Libraries

Ask:

Does this library represent a reusable capability or an organisational coupling mechanism?

For example:

Good candidate

Observability SDK

It may evolve independently.

Potentially problematic

CompanyBusinessLogicUtils

This can become a dumping ground for code that should actually belong to a domain.

A common anti-pattern is:

common/
├── StringUtils
├── DateUtils
├── CustomerUtils
├── PaymentUtils
├── OrderUtils
├── ValidationUtils
└── EverythingElseUtils

Eventually every service depends on common.

Congratulations.

You've created a distributed monolith with a shared library.

10. Repository Boundaries Should Follow Ownership

One of the strongest signals for repository boundaries is team ownership.

Imagine:

Team: Payments

Owns:
├── payments-api
├── payment-processing
├── reconciliation
└── payment-risk

These components may benefit from being in the same repository.

Because the same team:

changes them together
reviews them
deploys them
understands their dependencies

Now imagine:

Payments team
        ↓
Customer team
        ↓
Identity team

If every change requires three teams to coordinate, the problem may not be the repository.

It may be the architecture.

11. Repository Boundaries Should Follow Change Patterns

This is one of the questions I now ask when evaluating repository structure:

What code tends to change together?

If these always change together:

orders
orders-api
orders-domain
orders-tests

keeping them together makes sense.

If these change independently:

marketing-website
payment-processing
internal-data-platform

there is little value in forcing them into one repository purely for consistency.

A useful mental model is:

High change coupling
        ↓
Keep closer together

Low change coupling
        ↓
Separate when useful

12. CI/CD Changes the Equation

Repository strategy and CI/CD strategy are tightly connected.

A monorepo without good CI/CD becomes painful.

A polyrepo without standardised CI/CD also becomes painful.

Imagine 80 repositories.

If every repository has:

Jenkinsfile
Dockerfile
Terraform
security-scan.yml
deployment.yml

and every team maintains its own version, you eventually get:

Repo A → Java 17
Repo B → Java 21
Repo C → old security scanner
Repo D → custom deployment
Repo E → completely different pipeline

The repository structure isn't necessarily the problem.

Lack of platform engineering is.

A mature organisation should provide reusable delivery capabilities.

For example:

             Platform Engineering
                    │
        ┌───────────┼───────────┐
        ↓           ↓           ↓
      Build       Security    Deploy
        │           │           │
        └───────────┼───────────┘
                    ↓
              Product Teams

Teams should be able to own their applications without having to reinvent the delivery platform.

13. Developer Experience Matters More Than We Think

Ask a developer:

"How long does it take you to make a change across three services?"

That's often more useful than asking:

"Do we use a monorepo?"

Consider the workflow.

Monorepo

Clone
 ↓
Find services
 ↓
Make change
 ↓
Run affected tests
 ↓
Single PR

Polyrepo

Clone repo A
 ↓
Clone repo B
 ↓
Clone repo C
 ↓
Change shared contract
 ↓
PR A
 ↓
PR B
 ↓
PR C
 ↓
Version library
 ↓
Update dependencies
 ↓
Wait for pipelines

Neither is inherently better.

The important question is:

Which workflow reflects how your engineers actually work?

14. Security and Compliance Can Change the Decision

There are cases where separate repositories make practical sense.

For example:

Public product
        │
        ├── repository A

Highly restricted system
        │
        └── repository B

Different access requirements can be a legitimate reason for separation.

Other examples include:

regulated workloads
sensitive infrastructure
different organisational ownership
acquisition boundaries
external/open-source projects
different security classifications

This is where repository boundaries can become more than a developer-experience decision.

15. The Hybrid Model

In larger organisations, the answer doesn't have to be:

Everything → monorepo

or:

Everything → polyrepo

A hybrid approach can be very practical.

For example:

                    Engineering Organisation
                              │
          ┌───────────────────┼───────────────────┐
          ↓                   ↓                   ↓
     Product Monorepo     Platform Repo      Data Platform
          │                   │                   │
     ┌────┼────┐              │              ┌────┼────┐
     ↓    ↓    ↓              ↓              ↓    ↓    ↓
   API  Worker UI        Infrastructure     ETL  ML  Analytics

The key is that each boundary has a reason.

Not:

"We have always done it this way."

16. A Practical Decision Framework

Instead of asking:

"Which is better?"

I prefer asking these questions.

Question	Signal
Do multiple teams frequently modify the same code?	Monorepo
Do services have genuinely independent ownership?	Polyrepo
Are cross-service refactors common?	Monorepo
Do systems have different security/access boundaries?	Polyrepo
Do components share many libraries?	Monorepo can simplify this
Do teams have very different release lifecycles?	Polyrepo
Can CI efficiently build only affected components?	Makes monorepo more viable
Do teams require strong repository-level isolation?	Polyrepo
Are services actually independent despite being in one repo?	Monorepo can still work
Are many repos creating duplicated tooling?	Consider consolidation/platform tooling

The important thing is that these are signals, not absolute rules.

17. What I Would Avoid

Anti-pattern 1: "Microservices means one repo per service"

Not necessarily.

Repository boundaries and service boundaries solve different problems.

Anti-pattern 2: "Monorepo means one giant application"

Also not necessarily.

A monorepo can contain independently deployable services.

                 Monorepo
                    │
       ┌────────────┼────────────┐
       ↓            ↓            ↓
   Service A    Service B    Service C
       │            │            │
       ↓            ↓            ↓
   Deploy A     Deploy B     Deploy C

Anti-pattern 3: "Every shared class belongs in common"

This usually creates coupling.

Before creating a shared library, ask:

Is this genuinely reusable infrastructure, or are we hiding domain ownership problems?

Anti-pattern 4: "Every repository needs its own pipeline"

Standardise the platform.

Teams should own their delivery outcomes without necessarily owning every line of CI/CD implementation.

Anti-pattern 5: "Repository count is an architecture metric"

It isn't.

I'd rather know:

deployment frequency
lead time for change
change failure rate
dependency coupling
build time
test time
ownership clarity
developer productivity

than simply:

"We have 47 repositories."

18. What I'd Do on a New Project

If I were starting a new product today, I wouldn't begin with:

"Let's choose monorepo."

or:

"Let's choose polyrepo."

I'd start with:

Step 1 — Identify teams

Who owns what?

Step 2 — Identify deployment boundaries

What needs independent deployment?

Step 3 — Identify change coupling

What changes together?

Step 4 — Identify security boundaries

What requires separate access?

Step 5 — Understand developer workflows

How will engineers build and test changes locally?

Step 6 — Design CI/CD accordingly

Can we build/test only what changed?

Step 7 — Choose repository boundaries

Only now would I decide:

monorepo
        OR
polyrepo
        OR
hybrid

The repository strategy becomes an outcome of the architecture rather than the starting point.

19. My Rule of Thumb

If I had to reduce the whole discussion to one principle:

Put things together when they need to change together. Separate things when they need to be owned, secured, or released independently.

And there is an important second principle:

Don't use repository boundaries to solve problems that actually belong to architecture, ownership, or platform engineering.

A monorepo won't fix poor service boundaries.

A polyrepo won't create team autonomy by itself.

Microservices won't automatically give independent deployment.

And multiple Git repositories won't automatically create a scalable engineering organisation.

20. The Takeaways

If you're deciding between monorepo and polyrepo, I'd keep these points in mind:

1. Repository boundaries are architectural boundaries

Treat them deliberately.

2. Microservices do not require polyrepos

Services can be independently deployable from a monorepo.

3. Monorepos need good tooling

Incremental builds, dependency graphs, caching and affected-test execution become important at scale.

4. Polyrepos need platform engineering

Otherwise every team ends up maintaining its own CI/CD, security and developer tooling.

5. Shared libraries can create hidden coupling

Especially when they contain business logic.

6. Team ownership matters

Repository boundaries should make ownership clearer, not more complicated.

7. Look at change patterns

Ask what actually changes together.

8. Optimise for developer flow

The best architecture is often the one that reduces unnecessary coordination while preserving the boundaries the organisation genuinely needs.

Final Thought

The monorepo vs polyrepo debate is often presented as if there is a universally correct answer.

There isn't.

The better question is:

"What repository structure allows our teams to change the system safely, independently and efficiently?"

Once you start looking at the problem through:

ownership → coupling → change patterns → security → deployment → developer experience

the decision becomes much less ideological and much more engineering-driven.

And that's probably the most important lesson:

Choose repository boundaries based on how your organisation and software actually change — not based on what architecture is fashionable today.

Stop Mocking Your Test Dependencies— Get the Real Thing, Locally and in CI, in 3 Commands

Parth Sarthi Sharma — Tue, 25 Aug 2026 02:27:08 +0000

It's 2pm. You need to ship a fix before end of day. You run the integration suite and it fails — not because your code is wrong, but because staging is down. Someone else's bad migration took it out an hour ago, and now you're in the "is it up yet" channel with six other engineers, watching a thread instead of shipping.

Sound familiar? This is the actual, daily cost of how most teams do integration testing. And it usually comes down to one of two bad options.

The two options, and why both are bad

Mock everything. Fast, deterministic, runs anywhere — and proves almost nothing. Your mock says Postgres returns rows in this shape. Real Postgres doesn't care what your mock thinks. The bug that matters — the one your mock can't see by definition — ships to production and pages someone at 2am.

Share a staging environment. Real dependencies, finally — but now you're queued behind whoever's deploying, debugging, or has just quietly broken it for everyone. It's slow, it's flaky in ways that have nothing to do with your code, and "is staging even up right now" becomes a real question you ask multiple times a week.

Neither of these is a testing strategy. They're both just different ways of accepting risk.

There's a third option: real dependencies, but yours alone

Morarix is a CLI that reads a small, hand-authored dependency graph for your service and provisions the real thing — Postgres, Redis, Kafka, and eight AWS services via LocalStack — as disposable, isolated containers on your laptop or in CI. Not a shared environment you wait for. Not a mock you have to trust. The real dependency, health-checked, torn down when you're done, every time.

morarix init                # scaffold morarix.graph.json
morarix import compose      # or: convert an existing docker-compose.yml
morarix up                  # provision + validate — real Postgres, Redis, S3... on localhost

Then you run your existing test suite — completely unmodified — against those real dependencies. Same commands, locally and in CI.

Show, don't tell

Here's the whole morarix.graph.json for a FastAPI service backed by Postgres, Redis, and S3 (one of the reference examples):

{
  "schema_version": "1.0",
  "service": { "name": "python-fastapi", "repo_path": ".", "language": "python", "framework": "fastapi" },
  "dependencies": [
    { "id": "db", "type": "postgres", "fidelity": "real", "source": "manual",
      "required_env_vars": ["MORARIX_DB_CONNECTION_STRING"], "optional": false,
      "image": "postgres", "version": "16" },
    { "id": "cache", "type": "redis", "fidelity": "real", "source": "manual",
      "required_env_vars": ["MORARIX_CACHE_CONNECTION_STRING"], "optional": false,
      "image": "redis", "version": "7" },
    { "id": "assets", "type": "s3", "fidelity": "real", "source": "manual",
      "required_env_vars": ["MORARIX_ASSETS_HOST", "MORARIX_ASSETS_PORT"], "optional": false,
      "resource": { "bucket_name": "widget-assets" } }
  ]
}

That's it. No YAML you have to hand-sync with your CI config. fidelity: "real" means Morarix actually provisions a container and health-checks it — not "the process started," but "Postgres will accept a connection," "Redis responds to PING," "the LocalStack S3 endpoint is reachable."

Here's the actual output of running it — nothing hand-written for this post:

$ morarix up
morarix: loaded 3 dependencies for "python-fastapi"
  - db                   postgres       fidelity=real
  - cache                redis          fidelity=real
  - assets               s3             fidelity=real

morarix: environment ready (run_id=morarix-3f16480cfbc9-1786016512, mode=isolated) — morarix.connections.json and .env.morarix written

$ pytest -v
============================= test session starts =============================
collecting ... collected 9 items

tests/test_api.py::test_health_reports_ok PASSED                         [ 11%]
tests/test_api.py::test_create_widget_then_list_returns_it PASSED        [ 33%]
tests/test_api.py::test_get_widgets_is_cached_across_consecutive_calls PASSED [ 44%]
tests/test_data_access.py::test_insert_and_list_widgets_round_trips_through_postgres PASSED [ 77%]
tests/test_data_access.py::test_cache_set_and_get_round_trips_through_redis PASSED [ 88%]
tests/test_data_access.py::test_spec_upload_and_download_round_trips_through_s3 PASSED [100%]

======================= 9 passed, 3 warnings in 24.13s ========================

$ morarix down
morarix: environment torn down (was: isolated)

No compose file in the app repo. No mocks. No manual container wrangling. Just the real dependencies, up, tested against, and gone.

"Isn't this just docker-compose / Testcontainers / Tilt?"

Fair question — here's the actual difference:

vs. docker-compose by hand — compose has no fidelity concept (a service is either real or entirely absent), no per-type health checks (depends_on isn't "Postgres will actually accept a connection"), and nothing stops your compose file and your CI YAML from silently drifting apart over time.
vs. Testcontainers — Testcontainers is a library embedded in your test code, per language, with its own API to learn per dependency type. Morarix is a CLI: one graph, one morarix up, zero test-code changes, and the same graph works across every language in a polyglot org instead of being re-implemented per service.
vs. Tilt / Skaffold / Garden — those solve your whole live-reload dev loop, usually against Kubernetes, often building and deploying your app itself. Morarix deliberately doesn't touch your app — it only provisions what's around it, so your app keeps running exactly how you already run it.

The same three commands in CI

This is the actual payoff — not a separate CI-specific setup that quietly drifts from what you run locally, but the literal same commands:

- name: Install morarix
  run: go install github.com/morarixhq/morarix/cmd/morarix@latest
- run: morarix up
- run: pytest
- run: morarix down

ubuntu-latest already has Docker. That's the whole prerequisite.

Get it

go install github.com/morarixhq/morarix/cmd/morarix@latest

Prebuilt binaries for Linux, macOS, and Windows (amd64 + arm64) are on the v0.1.0 release — no Go toolchain required.

Currently working end-to-end for Go, Node.js, TypeScript, Java, .NET, and Python, with six runnable reference examples covering both onboarding paths (init from scratch and import compose from an existing docker-compose.yml), plus Postgres, MySQL, Redis, Kafka, REST, and LocalStack health checkers already built in.

Try it, then tell me where it breaks

This is genuinely early — v0.1.0, not a polished v3. If you've got a service with a Postgres/Redis/Kafka dependency sitting behind mocks or a shared staging environment you're tired of waiting on, that's exactly the case this is for. Point it at that service and see what breaks.

⭐ Repo: github.com/morarixhq/morarix
📖 Why Morarix (the full case): morarix-docs/concepts/why-morarix.md
🧭 Usage guide: morarix-docs/docs/usage-guide.md
🧪 Examples: morarix-examples

If it saves you a "is staging up" Slack thread, a star helps other people find it. If it breaks on your setup, open an issue — that's exactly the feedback this stage needs.

The Golden Pipeline for AI/ML Systems in Production

Parth Sarthi Sharma — Sun, 28 Jun 2026 09:45:37 +0000

Most AI/ML tutorials stop at training a model.

Real systems start after that.

In production, the hardest problems are not modeling — they are:

Data quality drift
Evaluation reliability
Deployment safety
Monitoring failure modes
Continuous improvement loops

This article breaks down a practical Golden Pipeline for AI/ML systems based on real production engineering practices.

1. The Golden Pipeline (End-to-End View)

A real production ML/AI system looks like this:

Data Ingestion → Validation → Feature Engineering → Training →
Evaluation → Model Registry → Deployment →
Shadow Testing → A/B Testing → Monitoring → Feedback Loop

Each stage is independently versioned and testable.

2. Data Ingestion Layer

Core Principle:

Never trust raw data.

Best Practices:

Use streaming ingestion (Kafka / Kinesis / PubSub)
Store raw + processed data separately
Enforce schema validation at ingestion time
Track full data lineage

Common Failure:

Most ML failures are NOT model failures — they are data pipeline failures.

3. Data Validation Layer

Before training:

Validate schema
Check missing values
Detect anomalies
Ensure type consistency

Recommended Tools:

Great Expectations
Pandera
Pydantic (very effective in Python systems)

Example:

from pydantic import BaseModel

class Transaction(BaseModel):
    amount: float
    currency: str
    timestamp: str

4. Feature Engineering Layer

=============================

Rule:

If a feature is not reproducible → it does not exist.

Best Practices:

Make feature pipelines deterministic
Avoid inline feature computation during training
Use feature stores when possible:
- Feast
- Tecton

5. Training Pipeline

=====================

Principles:

Training must be stateless
Every run must be reproducible
Log all hyperparameters
Version datasets

Tools:

MLflow
Weights & Biases
DVC
Hydra

6. Evaluation Layer (Critical)

===============================

This is where most systems fail.

Never rely on a single metric.

Use layered evaluation:

1. Exact Metrics

Accuracy
Precision / Recall
F1

2. Task-Specific Metrics

Exact match
Numeric tolerance (important in finance systems)
Structured output validation

3. LLM-Based Evaluation (if applicable)

Pairwise comparison
Rubric scoring
LLM-as-judge (carefully calibrated)

Key Insight:

Exact match is often WRONG in real-world systems.

Example:

Gold: -32%
Prediction: -32.82%

This should be considered correct under tolerance rules.

7. Model Registry

==================

Never deploy models directly.

Use a model registry:

MLflow Model Registry
SageMaker Registry
Custom versioned storage (S3/GCS)

Store:

Model version
Dataset version
Metrics
Git commit hash
Config snapshot

8. Deployment Strategies

=========================

Option 1: Direct deployment

Risky
No rollback

Option 2: Blue-Green Deployment

Two environments
Instant rollback possible

Option 3: Canary Deployment (Best Practice)

Deploy to small % of traffic
Gradually scale
Monitor closely

9. Shadow Mode (Highly Underrated)

===================================

Run new model in parallel with production:

No user impact
Compare outputs silently
Detect silent failures

Why it matters:

Prevents production incidents
Detects drift early
Validates behavior safely

10. A/B Testing

================

After shadow validation:

Split traffic between models
Measure real impact

Metrics:

Accuracy proxies
Latency
User engagement
Business KPIs

11. Monitoring

===============

If you don’t monitor, your model is broken already.

Monitor:

Data drift
Prediction drift
Latency
Error rates
Confidence distributions

Tools:

Prometheus
Grafana
Evidently AI
Arize

12. Feedback Loop

==================

This is where production ML systems improve.

Sources:

User corrections
Human labeling
Implicit feedback (clicks, edits)

This becomes future training data.

13. What Most Engineers Miss

=============================

Everything must be versioned
Assume models will fail
Always have rollback strategy
Data is more important than model
Evaluation is a first-class system

14. Final Takeaway

===================

A production AI system is NOT:

model training + deployment

It is:

a continuous loop of:data → training → evaluation → deployment → monitoring → feedback

The model is just one part of the system.

The pipeline is the product.

If You’re Building This

Start simple:

Add strict data validation first
Build evaluation before improving models
Introduce shadow mode early
Log everything from day one
Always design for failure

9 Practical Ways Senior ML Engineers Reduce Inference Latency

Parth Sarthi Sharma — Sat, 20 Jun 2026 07:37:31 +0000

Most teams blame the model when an AI application feels slow.

In reality, the model is often only one part of the latency budget.

A typical AI request may involve:

User Request
    ↓
Authentication
    ↓
Feature Retrieval
    ↓
Vector Search
    ↓
Agent Orchestration
    ↓
LLM Inference
    ↓
Guardrails
    ↓
Response Generation

By the time the user sees a response, latency has accumulated across multiple layers of the system.

After working on cloud-native systems, GenAI platforms, and distributed architectures, I've noticed that the best AI engineers focus on optimizing the entire pipeline—not just the model.

Here are 9 practical techniques commonly used in production AI systems.

1. Optimize Feature Retrieval Before Touching the Model

Many AI and ML systems spend more time fetching data than generating predictions.

Common examples:

Fraud detection systems fetching customer risk profiles
Recommendation systems retrieving user interaction history
Personalization engines loading customer attributes

A model that takes 50ms to infer becomes a 500ms system if feature retrieval takes 450ms.

Instead of:

Request
 ↓
Database Queries
 ↓
Model

Use:

Request
 ↓
Online Feature Store
 ↓
Model

Technologies commonly used:

Redis
DynamoDB
Feast Online Store
Tecton Online Store

The fastest prediction is often achieved by reducing feature lookup latency.

2. Separate Real-Time and Batch Features

Not every feature needs to be calculated at request time.

Bad:

Request
 ↓
Calculate 30-day spending history
 ↓
Model

Good:

Nightly Batch Pipeline
 ↓
Precompute Features
 ↓
Store in Feature Store

Request
 ↓
Feature Lookup
 ↓
Model

Examples of batch features:

Average spend last 30 days
Customer lifetime value
Product affinity scores

Examples of real-time features:

Transactions in last 5 minutes
Products viewed in current session
Failed login attempts

This reduces inference latency dramatically.

3. Cache Aggressively

One of the highest ROI optimizations.

Many requests are repetitive.

Examples:

Frequently asked support questions
Popular product recommendations
Repeated vector search results

Instead of:

Query
 ↓
RAG
 ↓
LLM

Use:

Query
 ↓
Cache Check
 ↓
Return Cached Response

Common technologies:

Redis
CloudFront
Application-level caches

A cache hit often reduces latency from seconds to milliseconds.

4. Reduce Retrieval Latency

In RAG systems, retrieval often becomes the bottleneck.

Typical latency contributors:

Large vector indexes
Excessive top-K retrieval
Poor filtering strategies

Instead of:

Search Entire Knowledge Base

Use:

Metadata Filters
 +
Vector Search

Examples:

Search only banking documents
Search only relevant departments
Search only customer-specific data

Reducing search space significantly improves response times.

5. Use Hybrid Retrieval Carefully

Many teams combine:

Vector Search
+
Keyword Search

which improves quality but increases latency.

Practical approach:

Keyword Search
 ↓
Candidate Set
 ↓
Vector Ranking

instead of searching the entire corpus twice.

Quality matters, but so does speed.

6. Parallelize Tool Calls and Agent Workflows

One of the most common mistakes in agentic systems is sequential execution.

Bad:

Agent
 ↓
Tool A
 ↓
Tool B
 ↓
Tool C

Total latency:

A + B + C

Better:

Agent
 ↓
Parallel Execution
 ↓
Tool A
Tool B
Tool C

Total latency:

max(A,B,C)

This can reduce response time by several seconds.

7. Use Smaller Models Where Possible

Not every task requires a large model.

Examples:

Task	Better Choice
Classification	Small Model
Intent Detection	Small Model
Routing	Small Model
Summarization	Medium Model
Complex Reasoning	Large Model

A common production pattern:

Small Model
 ↓
Route Request
 ↓
Large Model (only when needed)

This reduces both latency and cost.

8. Quantize Models

A technique heavily used in production ML systems.

Instead of:

FP32 Model

Use:

INT8
INT4

or similar quantized formats.

Benefits:

Smaller memory footprint
Faster inference
Lower infrastructure costs

Especially useful for:

Edge deployments
Real-time recommendation systems
High-throughput inference workloads

The trade-off is a small accuracy reduction.

9. Measure the Entire Latency Budget

This is where observability becomes critical.

Many teams optimize the model while ignoring everything else.

Track latency across:

Feature Retrieval
Vector Search
Agent Routing
Tool Calls
LLM Inference
Guardrails
Response Validation

A typical breakdown might look like:

Feature Retrieval      50ms
Vector Search         120ms
Tool Calls            300ms
LLM Inference        2200ms
Guardrails            150ms

Without tracing, teams often optimize the wrong component.

Platforms such as Langfuse, HoneyHive, Arize Phoenix, and OpenTelemetry-based observability stacks make these bottlenecks visible.

The Real Lesson

The fastest AI systems are rarely the ones with the fastest models.

They are the systems with:

Efficient feature retrieval
Smart caching
Optimized retrieval pipelines
Parallel execution
Right-sized models
Strong observability

Senior AI engineers optimize the entire system.

Because users don't care whether the delay comes from a vector database, a feature store, an agent, or an LLM.

They only notice one thing:

How long it takes to get an answer.

I Compared 7 AI Observability Platforms So You Don’t Have To (2026 Edition)

Parth Sarthi Sharma — Thu, 11 Jun 2026 05:50:50 +0000

The AI tooling ecosystem is exploding.

Every week there seems to be a new platform promising:

Better traces
Better evaluations
Better prompt debugging
Better monitoring
Better cost visibility

The challenge isn’t finding an AI observability tool anymore.

The challenge is choosing one.

If you’re building AI applications today, chances are you’ve come across names like Langfuse, LangSmith, HoneyHive, Helicone, Arize, Braintrust, or Phoenix.

After exploring these platforms, I noticed something interesting:

Most tools overlap in functionality, but each one is optimized for a very different workflow.

This article focuses on comparing the tools themselves—not explaining AI observability concepts.

Let’s dive in.

⸻

Evaluation Criteria

For this comparison I evaluated each platform across:

Tracing and debugging
Prompt monitoring
Evaluations (Evals)
Cost tracking
Dataset management
Self-hosting support
Enterprise readiness
Ease of adoption

⸻

Quick Comparison Table

Tool	Open Source	Tracing	Evaluations	Cost Monitoring	Self Host	Best For
Langfuse	✅ Yes	⭐⭐⭐⭐⭐	⭐⭐⭐⭐⭐	⭐⭐⭐⭐	✅ Yes	Most teams
HoneyHive	❌ No	⭐⭐⭐⭐⭐	⭐⭐⭐⭐⭐	⭐⭐⭐⭐	❌ No	Enterprise AI
LangSmith	❌ No	⭐⭐⭐⭐⭐	⭐⭐⭐⭐	⭐⭐⭐	❌ No	LangChain users
Helicone	⚠️ Partial	⭐⭐⭐	⭐⭐	⭐⭐⭐⭐⭐	✅ Yes	Cost visibility
Arize	⚠️ Partial	⭐⭐⭐⭐⭐	⭐⭐⭐⭐⭐	⭐⭐⭐⭐	⚠️ Limited	Large production systems
Braintrust	⚠️ Partial	⭐⭐⭐	⭐⭐⭐⭐⭐	⭐⭐	⚠️ Limited	Evaluation-first workflows
Phoenix	✅ Yes	⭐⭐⭐⭐	⭐⭐⭐⭐	⭐⭐	✅ Yes	Lightweight OSS setups

⸻

Langfuse

What Stands Out

Langfuse has become one of the most popular choices for AI engineering teams.

It combines:

Tracing
Prompt management
Evaluations
Dataset tracking
Cost analytics

in a single platform.

The biggest advantage is flexibility.

Unlike many commercial products, Langfuse does not lock you into a specific framework.

You can use:

OpenAI
Anthropic
Gemini
Bedrock
LangChain
LangGraph
Custom agents

without major friction.

Strengths

✅ Open source

✅ Self-hosting available

✅ Strong evaluation workflows

✅ Framework agnostic

✅ Excellent developer experience

Weaknesses

❌ More setup than fully managed platforms

❌ Enterprise features may require additional work

Best For

Teams wanting a long-term observability platform without vendor lock-in.

⸻

HoneyHive

What Stands Out

HoneyHive focuses heavily on enterprise AI quality and testing.

The platform goes beyond simple tracing and emphasizes:

Evaluation pipelines
Regression testing
Prompt experimentation
AI system quality measurement

This makes it particularly attractive for organizations deploying AI into production at scale.

Strengths

✅ Enterprise-grade workflows

✅ Strong evaluation capabilities

✅ Regression testing

✅ Production monitoring

Weaknesses

❌ Less attractive for hobby projects

❌ Commercial-first offering

Best For

Organizations that treat AI systems like mission-critical software.

⸻

LangSmith

What Stands Out

If your stack is already built around LangChain or LangGraph, LangSmith feels almost automatic.

The integration is excellent.

You get:

Agent traces
Execution paths
Prompt inspection
Chain debugging

with minimal effort.

Strengths

✅ Best LangChain integration

✅ Excellent trace visualization

✅ Fast setup

✅ Agent debugging experience

Weaknesses

❌ Less attractive outside LangChain ecosystems

❌ Limited self-hosting options

Best For

Teams deeply invested in LangChain or LangGraph.

⸻

Helicone

What Stands Out

Helicone is probably the easiest way to understand where your AI budget is going.

Its focus is much more operational than evaluation-centric.

You get visibility into:

Request volume
Token usage
Model consumption
Cost breakdowns

without significant complexity.

Strengths

✅ Excellent cost analytics

✅ Quick integration

✅ OpenAI proxy model

✅ Lightweight deployment

Weaknesses

❌ Evaluation capabilities lag competitors

❌ Less sophisticated tracing

Best For

Startups trying to control AI infrastructure costs.

⸻

Arize

What Stands Out

Arize comes from the machine learning observability world.

As a result, it brings strong production monitoring capabilities that many AI-native tools still lack.

The platform is particularly strong when organizations combine:

Traditional ML systems
Recommendation systems
LLM applications

inside the same environment.

Strengths

✅ Mature monitoring platform

✅ Strong evaluation tooling

✅ Enterprise scale

✅ ML + LLM support

Weaknesses

❌ Can feel overwhelming for small teams

❌ Higher operational complexity

Best For

Large-scale AI platforms operating in production.

⸻

Braintrust

What Stands Out

Braintrust takes a different approach.

Rather than starting with traces, it starts with evaluations.

The philosophy is simple:

“If you can’t measure quality, you can’t improve quality.”

This makes Braintrust especially useful for teams focused on:

Prompt optimization
Model comparisons
Benchmarking
Continuous evaluation

Strengths

✅ Excellent evaluation workflows

✅ Dataset management

✅ Benchmarking capabilities

✅ Model comparison workflows

Weaknesses

❌ Less focused on operational monitoring

❌ Tracing is not the primary strength

Best For

Teams building evaluation-driven AI development processes.

⸻

Phoenix

What Stands Out

Phoenix is one of the strongest open-source alternatives available.

It provides:

Tracing
Evaluation workflows
Debugging capabilities

without introducing significant operational overhead.

Many engineers adopt Phoenix because they want observability without committing to a larger commercial ecosystem.

Strengths

✅ Open source

✅ Lightweight deployment

✅ Good tracing

✅ Simple adoption

Weaknesses

❌ Smaller ecosystem

❌ Fewer enterprise features

Best For

Engineers wanting lightweight observability with minimal complexity.

⸻

My Recommendations

If I had to choose today:

My Recommendations

Scenario	Recommendation
Best Overall	Langfuse
Best Enterprise Choice	HoneyHive
Best LangChain Experience	LangSmith
Best Cost Tracking	Helicone
Best Evaluation Platform	Braintrust
Best Production Monitoring	Arize
Best Lightweight Open Source Option	Phoenix

⸻

Final Thoughts

The interesting thing about AI observability tools is that most of them solve similar problems.

The real difference is where they place their emphasis.

Langfuse focuses on flexibility.
HoneyHive focuses on enterprise quality.
LangSmith focuses on developer productivity.
Helicone focuses on costs.
Arize focuses on production monitoring.
Braintrust focuses on evaluations.
Phoenix focuses on lightweight open-source adoption.

There is no universally “best” platform.

The right choice depends on what bottleneck you’re trying to solve:

Debugging?
Evaluation?
Monitoring?
Cost optimization?
Enterprise governance?

Choose the tool that aligns with that bottleneck, and you’ll likely get far more value than chasing feature checklists.

What AI observability platform are you currently using, and what made you choose it?

How Senior Engineers Use AI Without Burning Through Token Limits - Reduce AI Token Usage by 60–90%

Parth Sarthi Sharma — Sat, 06 Jun 2026 12:16:25 +0000

Last month I watched a developer exhaust their Claude usage limit in less than a week.

They weren't generating massive applications.

They weren't building complex AI systems.

They were simply asking AI to repeatedly scan the same repository, read the same files, and explain the same architecture over and over again.

Sound familiar?

As AI-assisted development becomes mainstream, many teams are discovering a new engineering challenge:

Token efficiency.

Just as experienced engineers learned to optimize cloud spend, senior engineers are now learning to optimize AI context.

The difference between a developer who runs out of tokens every few days and one who comfortably works all month often isn't the AI model.

It's how they manage context.

Here's the toolkit and workflow I've seen work consistently.

The Hidden Cost of Vibe Coding

Imagine you ask:

Fix a bug in PaymentService.ts

Your AI assistant proceeds to:

Scan the entire repository
Read infrastructure code
Explore frontend folders
Traverse documentation
Load previous conversations
Inspect unrelated dependencies

You asked about one file.

The model consumed context from hundreds.

That's where your tokens disappear.

The goal isn't to reduce intelligence.

The goal is to reduce unnecessary context.

1. RTK: Stop Paying For Useless Command Output

One of the biggest hidden token sinks is terminal output.

Many AI coding agents automatically consume:

npm install logs
build outputs
test results
deployment logs
dependency resolution messages

Most of this information is irrelevant.

Tools like RTK solve this problem.

What RTK Does

RTK acts as a proxy layer between your development environment and the LLM.

Instead of forwarding everything:

npm install

RTK filters:

redundant messages
repeated warnings
progress indicators
noise

before they ever reach the model.

Benefits

Reported reductions:

60–90% reduction in token consumption for common development workflows
Faster agent reasoning
Cleaner context windows

The principle is simple:

If a human wouldn't read it, the model probably shouldn't either.

2. Lean-CTX: Compress Context Before It Reaches The Model

Most developers optimize prompts.

Few optimize files.

Large source files often contain:

generated code
comments
repetitive structures
boilerplate

Lean-CTX dynamically compresses and optimizes file content before it gets sent to the model.

Why It Matters

Instead of sending:

4,000 line file

you might send:

Relevant functions
Dependencies
Symbols
Interfaces

The AI receives the information it needs while consuming significantly fewer tokens.

Think of it as:

gzip for AI context.

3. AI Codex & Repository Indexers

One of the most expensive activities in AI coding is:

"Explore my codebase."

The model begins reading dozens of files trying to understand:

routes
APIs
schemas
services
components

This exploratory phase can easily burn tens of thousands of tokens.

Repository indexing tools solve this.

What They Generate

Instead of scanning everything:

Generate:

ROUTES.md
DATABASE_SCHEMA.md
COMPONENTS.md
SERVICES.md
DEPENDENCIES.md

Now the AI can understand the system from five small files instead of 500 source files.

Typical Savings

Many teams report avoiding:

30k–50k tokens

during initial codebase exploration.

This is one of the highest ROI improvements you can make.

The Caveman Rule: My Favorite Token Hack

This sounds ridiculous.

But it works.

When you need code, you don't need essays.

You don't need:

Certainly! Here's a detailed explanation...

You need:

Bug here.
Fix this.
Run test.
Done.

The Caveman Rule instructs the AI to:

skip conversational filler
avoid lengthy summaries
communicate with minimal words

Example:

Instead of:

I've identified several possible root causes...

You get:

Null value here.
Add guard clause.
Problem solved.

The technical accuracy remains.

The verbosity disappears.

Many developers report output token reductions approaching 75%.

Create A Project Brain

One of the biggest mistakes I see:

Developers repeatedly explaining their project.

Every new session starts with:

We're using:
- Node.js
- PostgreSQL
- Kubernetes
- OpenTelemetry
- GitHub Actions

Again.

And again.

Instead create:

CLAUDE.md
AGENTS.md
PROJECT_CONTEXT.md
ARCHITECTURE.md

Store:

architecture
conventions
coding standards
deployment patterns
repository structure

Now every session starts with shared understanding.

The AI spends less time learning.

You spend fewer tokens teaching.

The Fragmented Code Approach

Another expensive habit:

Rewrite the entire file.

The AI responds with:

2,000 lines

You pay for all of it.

Instead ask:

Modify only lines 120–150.
Return patch only.
No summary.

Benefits:

fewer output tokens
smaller future context
easier reviews
lower costs

The best AI engineers increasingly think in patches, not rewrites.

Native IDE Features Most Developers Ignore

Many modern AI IDEs already provide token optimization features.

Most people never use them.

Cost Caps

Set:

maximum tool calls
session budgets
usage limits

Treat tokens like cloud spend.

Because they are.

Compact Sessions

Claude and other tools support context compaction.

Example:

/compact

This removes:

redundant conversation history
obsolete decisions
resolved issues

while preserving important context.

Think:

garbage collection for conversations.

New Session, New Problem

One of the easiest wins:

Start fresh.

When:

a feature is complete
a bug is resolved
you're switching domains

create a new session.

Old conversations become baggage.

The model keeps re-reading:

mistakes
abandoned approaches
irrelevant context

Fresh context often produces better results.

My Personal Context Engineering Checklist

Before asking AI anything:

Repository

Exclude:

node_modules/
dist/
coverage/
build/
.next/
target/

Context

Maintain:

CLAUDE.md
AGENTS.md
ARCHITECTURE.md
PROJECT_CONTEXT.md

Tooling

Use:

RTK
Lean-CTX
AI Codex
Repository Indexers
Semantic Search
Code Graphs

Prompting

Prefer:

Patch only.
No summary.

instead of:

Explain everything.

Sessions

Compact regularly
Start fresh often
Keep contexts small

Final Thoughts

For years we optimized:

cloud costs
compute costs
storage costs
network costs

Now we need to optimize:

context costs

The next generation of high-performing AI engineers won't be the people with the biggest context windows.

They'll be the people who know exactly what context to send.

Prompt engineering helped us talk to AI.

Context engineering helps us scale AI.

And in the age of vibe coding, context is the new compute.

Reflection vs Reflexion Agents: The Next Leap in Agentic AI

Parth Sarthi Sharma — Sun, 22 Mar 2026 03:04:03 +0000

As generative AI systems evolve from simple prompt-response tools into autonomous agents, one capability is becoming increasingly critical:

The ability for AI systems to improve themselves during execution.

This is where two powerful concepts come into play:

Reflection
Reflexion

They sound similar. They are often confused.

But architecturally — and practically — they are very different.

Let’s break them down.

🚀 Why This Matters

If you're building:

AI copilots
Autonomous workflows
Multi-step reasoning systems
Or agentic architectures

Then how your system learns from mistakes will define:

Accuracy
Reliability
Cost efficiency
User trust

🧠 What is Reflection?

Reflection is when an AI system:

Reviews its own output and improves it within the same execution loop.

🔁 How it works

Generate response
Evaluate response (self-critique or evaluator model)
Refine response
Repeat until acceptable

🧩 Architecture Pattern

User Input
↓
LLM → Output
↓
Self-Evaluation (LLM or rule-based)
↓
Refinement Loop
↓
Final Output

✅ Key Characteristics

Happens within a single session
No memory across runs
Iterative improvement
Often uses:
- Self-critique prompts
- Evaluation models
- Chain-of-thought refinement

💡 Example

User asks:

"Summarize this legal document."

Reflection agent:

Generates summary
Checks:
- Missing clauses?
- Ambiguity?
Refines output

👍 Pros

Improves output quality instantly
No infrastructure complexity
Easy to implement

👎 Cons

No long-term learning
Repeats same mistakes across sessions
Increased latency (multiple LLM calls)

🔁 What is Reflexion?

Reflexion goes a step further.

It enables an AI system to learn from past mistakes and improve future performance.

This concept was popularized by research on self-improving agents with memory.

🔄 How it works

Perform task
Evaluate outcome
Store feedback in memory
Use memory to improve future decisions

🧩 Architecture Pattern

User Input
↓
Agent Execution
↓
Outcome Evaluation
↓
Memory Store (success/failure insights)
↓
Future Runs Use Memory

🧠 Key Difference

Reflection	Reflexion
Session-based	Cross-session
No memory	Persistent memory
Improves current output	Improves future outputs
Stateless	Stateful

💡 Example

AI agent writing grant applications:

Attempt 1: Rejected ❌
Stores feedback:
- "Too generic"
- "Lacks domain-specific references"

Next attempt:

Uses stored insights
Produces better output ✅

🔥 Why Reflexion is a Big Deal

Reflexion introduces something critical:

Learning without retraining the model

Instead of fine-tuning:

You store experiences
You adapt behavior dynamically

🏗️ Real-World Implementation

Reflection (simple)

Prompt chaining
Self-critique prompts
ReAct-style loops

Reflexion (advanced)

Requires:

Memory layer:
- Vector DB (e.g., embeddings)
- Key-value store
Feedback signals:
- Human feedback
- Automated scoring
Retrieval mechanism:
- Inject past learnings into prompts

⚙️ Example Stack

LLM: Claude / GPT / Nova
Memory: Vector DB (FAISS, OpenSearch)
Orchestration: LangChain / custom agents
Evaluation: Rule-based or LLM-as-judge

⚖️ When to Use What?

Use Reflection when:

You need better answers now
No need for memory
Simpler workflows

Use Reflexion when:

Tasks are repetitive and evolving
Feedback is available
Long-term improvement matters

🧠 Combining Both (Best Practice)

The most powerful systems use both:

Reflexion (long-term learning)
+
Reflection (short-term refinement)

👉 This creates:

Immediate quality improvement
Continuous learning over time

🧪 Real-World Use Cases

AI coding assistants
Customer support agents
Financial advisory copilots
Healthcare decision support
Autonomous research assistants

⚠️ Challenges

Reflection

Cost (multiple LLM calls)
Latency

Reflexion

Memory design complexity
Signal quality (bad feedback = bad learning)
Retrieval accuracy

🧭 Final Thoughts

We are moving from:

Prompt → Response

to:

Prompt → Reason → Reflect → Learn → Improve

🔥 Key Insight

Reflection makes AI smarter in the moment

Reflexion makes AI smarter over time

✍️ Closing

If you're building next-gen AI systems,

understanding this difference is not optional — it's foundational.

The future of AI is not just about better models.

It’s about better systems around those models.

💬 Curious how to implement Reflexion in production?

Happy to share a deep dive in the next post.

Prompt Engineering Is Not Enough: Enter Flow Engineering for Production LLM Systems

Parth Sarthi Sharma — Sat, 07 Mar 2026 03:20:46 +0000

Large Language Models have unlocked a new generation of applications — copilots, assistants, RAG systems, autonomous agents, and internal AI tools.

But many teams building with LLMs hit the same wall.

Their application works in demos… but becomes unreliable in production.

Why?

Because prompt engineering alone is not enough.

To build reliable AI systems, we need something more powerful:

Flow Engineering.

In this article, we'll explore:

Why prompt engineering alone fails in production
What Flow Engineering actually means
The architecture of real-world LLM systems
Practical examples engineers can implement today

The Era of Prompt Engineering

When GPT-style models first became popular, the focus was on prompt engineering.

Prompt engineering is the art of crafting instructions to guide the LLM to produce better responses.

Example:

You are a helpful assistant. 
Summarise the following meeting transcript in bullet points.
Focus only on action items.

Developers quickly discovered techniques like:

Few-shot prompting
Chain-of-thought prompts
Role prompting
Structured output prompts

These techniques improve individual LLM calls.

But they only solve part of the problem.

Prompt engineering optimises one interaction.

Real applications involve many interactions and system components.

The Problem with Prompt-Only Systems

Let's imagine we are building a simple customer support AI assistant.
A naive architecture might look like this:

User Question
      ↓
     LLM
      ↓
   Response

This works in simple demos.

But real systems quickly require more complexity.

For example:

Retrieve relevant documents
Use tools (APIs, databases)
Validate outputs
Retry on errors
Maintain conversation context
Apply guardrails
Log reasoning steps

Suddenly, our architecture looks more like this:

User Question
      ↓
Context Retrieval (RAG)
      ↓
Tool Selection
      ↓
LLM Reasoning
      ↓
Output Validation
      ↓
Response Generation

This multi-step pipeline is where Flow Engineering comes in.

What Is Flow Engineering?

Flow Engineering is the design of structured execution flows around LLMs.

Instead of focusing on a single prompt, engineers design end-to-end reasoning pipelines.

Think of it as:

Prompt Engineering = How the LLM thinks

Flow Engineering = How the system operates

Flow engineering involves designing:

Execution pipelines
Tool orchestration
State management
Error handling
Validation
Feedback loops

In other words:

Flow engineering treats LLM applications as distributed systems, not chatbots.

A Real Production Flow

Let's look at a simplified production AI flow.

User Question
   ↓
Input Guardrails
   ↓
Context Retrieval (Vector DB)
   ↓
Tool Routing
   ↓
LLM Reasoning
   ↓
Tool Execution
   ↓
Response Validation
   ↓
Final Answer

Each step solves a real engineering problem.

Guardrails

Prevent prompt injection or malicious input.

Context Retrieval

Fetch relevant documents using vector search.

Tool Routing

Determine which tools the AI should use.

Validation

Ensure output matches schema or safety rules.

Without this flow, AI systems become unpredictable.

Example: Prompt vs Flow

Let's compare two implementations.

Prompt Engineering Only

response = llm.invoke(
    "Summarise this transcript and extract action items."
)

This may work sometimes.

But what if:

transcript is too long
model hallucinate action items
output format changes
context is missing

Now let's see a flow-based approach.

Example: Flow Engineered System

def generate_meeting_summary(transcript):

    chunks = split_transcript(transcript)

    summaries = []

    for chunk in chunks:
        summary = llm.invoke(
            f"Summarise this transcript section:\n{chunk}"
        )
        summaries.append(summary)

    combined_summary = llm.invoke(
        f"Combine these summaries and extract action items:\n{summaries}"
    )

    validated_output = validate_schema(combined_summary)

    return validated_output

Now we have:

chunking
intermediate reasoning
structured validation

This dramatically improves reliability.

Key Components of Flow Engineering

Most production LLM flows include these components.

1. State Management
Flows maintain state across steps.

Example:

Conversation History
Retrieved Documents
Tool Results

Frameworks like LangGraph model this using state machines.

2. Tool Orchestration

LLMs often interact with tools.

Examples:

databases
APIs
search engines
internal systems

Flow engineering controls:

which tool to use
when to call it
how to merge results

3. Retry & Error Handling

LLMs are probabilistic.

Sometimes outputs are invalid.

A flow can automatically:

retry generation
correct formatting
request clarification

4. Guardrails & Validation

Before returning outputs, systems often validate:

JSON schema
safety policies
hallucinations

This prevents unreliable responses.

Flow Engineering Frameworks

Several frameworks help engineers implement LLM flows.

LangGraph

Models AI workflows as state machines.

Great for:

complex agent workflows
branching logic
memory management

Semantic Kernel

Popular in enterprise environments.

Supports:

planners
function calling
workflow orchestration

Custom Orchestration

Many teams implement flows directly using:

Python
Node.js
serverless pipelines

Because flows are essentially application logic.

Why Flow Engineering Matters

Companies deploying production AI systems quickly discover:

The challenge is not the model.

The challenge is system design around the model.

Flow engineering provides:

✔ reliability
✔ reproducibility
✔ observability
✔ safety
✔ scalability

Without it, LLM applications behave unpredictably.

The Shift AI Engineers Must Make

Early LLM development focused on prompts.

But the industry is moving toward AI systems engineering.

That means thinking in terms of:

pipelines
workflows
orchestration
tool ecosystems

In short:

AI applications are evolving from prompt-driven apps to flow-driven systems.

Final Thoughts

Prompt engineering is still important.

But in production systems, prompts are only one component.

The real power of modern AI systems comes from well-designed execution flows.

If you want reliable AI applications, start thinking like a systems engineer, not just a prompt writer.

What’s Next

In upcoming articles, we'll dive deeper into:

Reflection vs Reflexion agents
LangGraph state machines
Semantic Kernel orchestration
Model Context Protocol (MCP)

These concepts build on flow engineering to create more capable AI systems.

Secrets Management for LLM Tools: Don’t Let Your OpenAI Keys End Up on GitHub 🚨

Parth Sarthi Sharma — Sat, 14 Feb 2026 04:04:56 +0000

"A practical guide to securing LLM API keys, embeddings, vector

TL;DR: If you're building with LLMs and you're not treating secrets as first-class infrastructure, you're already at risk.

Every week, we see:

OpenAI keys pushed to GitHub
API keys logged in CloudWatch
Secrets hardcoded in Streamlit demos that later go to production

LLM systems multiply secrets quickly. If you don’t design for this early, things get messy fast.

This is a production-ready blueprint for securing LLM systems properly.

The Problem: LLM Secrets Multiply Fast 🐰

One LLM integration turns into dozens of credentials:

1 LLM API key (OpenAI / Anthropic)
→ 3 embedding endpoints
→ 5 vector store connections (Pinecone / Weaviate)
→ 2 RAG databases
→ 10 external tools (SerpAPI, Wolfram, etc.)
→ 50 microservices
= 70+ secrets

The bigger your AI system gets, the larger your attack surface becomes.

1️⃣ Never Hardcode Secrets

❌ Wrong (guaranteed leak eventually)

# NEVER DO THIS
from openai import OpenAI

client = OpenAI(api_key="sk-123...")

Hardcoded secrets:

End up in git history
Get copied into logs
Leak via screenshots or stack traces

✅ Right: Runtime Environment Injection

# config.py

import os
from openai import OpenAI

OPENAI_API_KEY = os.getenv("OPENAI_API_KEY")

client = OpenAI(api_key=OPENAI_API_KEY)

Principle:
Secrets should be injected at runtime, never committed to source code.

2️⃣ Use Cloud-Native Secrets Managers

If you're in production, use a managed secrets service.

AWS Secrets Manager + Lambda Example

# lambda_function.py
import json
import boto3
from openai import OpenAI

def get_secrets():
    client = boto3.client("secretsmanager")
    secret = client.get_secret_value(SecretId="llm-prod/openai")
    return json.loads(secret["SecretString"])

def lambda_handler(event, context):
    secrets = get_secrets()
    client = OpenAI(api_key=secrets["OPENAI_API_KEY"])
    # LLM logic here

Benefits:

Centralized storage
IAM-based access control
Audit logs
Automatic rotation support

Terraform for Secret Infrastructure

resource "aws_secretsmanager_secret" "llm_keys" {
  name = "llm-prod/openai"
  tags = {
    Environment = "Production"
    Team        = "AI"
  }
}

resource "aws_secretsmanager_secret_version" "llm_keys_version" {
  secret_id     = aws_secretsmanager_secret.llm_keys.id
  secret_string = jsonencode({
    OPENAI_API_KEY    = "sk-..."
    ANTHROPIC_API_KEY = "sk-ant-..."
    PINECONE_API_KEY  = "pxl-..."
  })
}

Infrastructure-as-Code ensures:

Repeatability
Auditability
No manual copy-paste secret management

3️⃣ Prefer Dynamic Credentials Over Static API Keys ⚡

Static API keys are long-lived and high risk.

Dynamic credentials reduce blast radius.

IAM Roles for Service Accounts (Kubernetes + AWS IRSA)

apiVersion: v1
kind: ServiceAccount
metadata:
  name: llm-worker
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::123456789012:role/llm-worker-role
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: llm-worker
spec:
  template:
    spec:
      serviceAccountName: llm-worker
      containers:
        - name: llm-worker
          env:
            - name: OPENAI_API_KEY
              valueFrom:
                secretKeyRef:
                  name: llm-secrets
                  key: openai-key

Even better: eliminate API keys entirely where possible and use workload identity federation.

4️⃣ Secure CI/CD with OIDC (No Long-Lived AWS Keys)

Never store AWS credentials in GitHub secrets if you can avoid it.

Use OIDC federation instead:

name: Deploy LLM Pipeline

on: [push]

jobs:
  deploy:
    runs-on: ubuntu-latest
    permissions:
      id-token: write
      contents: read
    steps:
      - uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: arn:aws:iam::123456789012:role/github-actions-llm-deploy
          aws-region: us-east-1

      - run: python deploy.py

This avoids:

Static AWS access keys
Manual credential rotation
CI secret sprawl

5️⃣ Agentic LLM Systems Need Scoped Secrets 🧠

When building multi-agent systems:

Each agent should have scoped credentials
Short-lived tokens preferred
No shared global API key across agents

Example pattern:

class LLMAgentSecrets:
    def __init__(self, sm_client):
        self.sm_client = sm_client

    def get_agent_secret(self, agent_id: str):
        secret_name = f"llm-agent-{agent_id}"
        secret = self.sm_client.get_secret_value(SecretId=secret_name)
        return secret["SecretString"]

Design for:

Isolation
Least privilege
Auditable access

✅ Production Security Checklist

☐ No hardcoded secrets (git grep -i "sk-")
☐ Cloud secrets manager in use
☐ IAM roles preferred over static keys
☐ OIDC for CI/CD
☐ Secrets scanning enabled (TruffleHog, GitGuardian)
☐ Log sanitization in place
☐ Rotation policy defined (≤ 90 days)
☐ Audit logging enabled
☐ Least privilege enforced

Common Leak Vectors 🚫

Leak Vector	Detection	Prevention
Git commits	`git log -p	grep sk-`	Pre-commit hooks
Logs	CloudWatch Insights	Log scrubbing
Docker images	Inspect image layers	Multi-stage builds
Memory dumps	`/proc/[pid]/environ`	Container hardening

Cost vs Risk 💰

Typical monthly cost for secure secrets management:

AWS Secrets Manager: ~$0.40 per secret
Secret scanning tools: modest monthly fee
OIDC: no additional cost
Compare that to:
Revoking leaked keys
Service outages
Customer trust damage

Security is cheaper than cleanup.

Key Takeaways 🎯

Dynamic > Static
Inject at runtime, never commit
Audit secret access
Rotate regularly
Scan continuously
Apply least privilege everywhere

LLMs are powerful.

But API keys are still just credentials — treat them like production infrastructure.

Have you ever dealt with an exposed LLM API key in production? What happened?
Let’s discuss 👇

Observability in AI Systems

Parth Sarthi Sharma — Tue, 27 Jan 2026 13:17:25 +0000

Why RAG Pipelines Fail Silently (and How to See It)

Traditional software taught us a hard lesson:

If you can’t observe it, you can’t operate it.

AI systems — especially RAG pipelines — are repeating the same mistakes we made with distributed systems a decade ago.

They look fine.
They respond fast.
They return answers.

And yet — they are quietly wrong.

This article explains:

Why observability is fundamentally harder in AI systems
What observability actually means for RAG pipelines
What signals matter (and which ones don’t)
How mature teams design observable AI systems

No dashboards for the sake of dashboards — only what helps you debug reality.

Why AI Observability Is Different From Traditional Observability

In classic systems, we observe:

CPU
memory
latency
error rates

In AI systems, the hardest failures are:

Semantic
Probabilistic
Contextual

A RAG pipeline can:

return HTTP 200
respond in 300ms
use the correct model

…and still give a wrong answer.

That’s why AI observability must go deeper.

The Core Problem With RAG Pipelines

A basic RAG flow looks like this:

User Query
↓
Embedding
↓
Vector Search
↓
Top-K Chunks
↓
Prompt Assembly
↓
LLM Generation

When the output is wrong, where did it fail?

Bad query?
Wrong chunks?
Missing chunks?
Prompt formatting?
Model hallucination?

Without observability, you’re guessing.

What “Observability” Means in the AI World

AI observability is the ability to answer:

Why did the system produce this answer for this input at this time?

That requires traceability, not just metrics.

The Four Pillars of RAG Observability

1️⃣ Query Observability

You must log:

Original user query
Rewritten / normalized query (if any)
Detected intent or routing decision

Why it matters

Many failures start with:

ambiguous questions
underspecified intent
bad query rewriting

If you can’t see the effective query, you can’t debug retrieval.

2️⃣ Retrieval Observability (Most Important)

This is where most RAG systems fail.

You should observe:

Retrieved chunk IDs
Source documents
Similarity scores
Chunk rank
Retrieval strategy used (vector, keyword, hybrid)

Example questions observability should answer:

Which chunks were retrieved?
Which chunk influenced the answer most?
Was relevant information missing?

If you don’t log retrieved chunks, you don’t have RAG observability.

3️⃣ Prompt Observability

Your prompt is your runtime program.

You must capture:

Final prompt sent to the LLM
Context size and token count
Chunk ordering
System instructions

Why?
Because subtle changes in:

ordering
truncation
formatting

can completely change answers.

4️⃣ Generation & Answer Observability

Beyond the final answer, log:

Model name & version
Temperature / decoding params
Token usage
Latency
Safety or refusal triggers

Advanced systems also track:

Answer confidence
Self-evaluation scores (Self-RAG)
Groundedness signals

The Most Common RAG Failure Modes (Seen in Production)

❌ “The model hallucinated”

Usually false.

More often:

Wrong chunk retrieved
Right chunk ranked too low
Context truncated
Outdated document used

Observability makes this visible.

❌ “Vector search is bad”

Often:

Chunking is wrong
Embedding mismatch
Query rewriting failed

Again — visible with the right signals.

Tracing a Single RAG Request (What Good Looks Like)

A single request trace should show:

Request ID: 9f23...

Query:
"Can I carry forward unused leave?"

Rewritten Query:
"Leave carry forward policy Australia"

Retrieved Chunks:

handbook.md#leave-carry-forward (score: 0.89)

policy.md#exceptions (score: 0.81)

Prompt Tokens:
3,214

Model:
gpt-4.1-mini

Answer:
"Yes, up to 10 days can be carried forward..."

Confidence:
High

If you can’t reconstruct this — you can’t debug.

Why Traditional Metrics Are Not Enough

Latency and cost are necessary — but insufficient.

AI systems need semantic metrics:

Groundedness
Faithfulness
Retrieval coverage
Answer stability over time

These are harder — but essential.

Observability Enables Advanced RAG Patterns

You cannot safely implement:

Adaptive RAG
Corrective RAG
Self-RAG

without observability.

Why?
Because all of them rely on feedback signals:

Was retrieval good?
Was the answer grounded?
Should we retry?

No signals → no control loop.

A Simple Observability Checklist

If you’re building RAG in production, you should be able to answer:

Which document influenced this answer?
Why was this chunk chosen over others?
What changed compared to yesterday?
Would a different retrieval strategy help?
Can I replay this request?

If the answer is “no” — observability is missing.

Final Thought

RAG pipelines don’t usually fail loudly.

They fail:

quietly
confidently
at scale

The future of AI systems isn’t just better models.

It’s systems that can explain themselves.

And observability is how that starts.

If you’ve debugged a RAG issue that turned out to be “invisible” at first, I’d love to hear what signal finally revealed it.