DEV Community: Ahmed Adawy

Limited-Time Leanpub Discounts: LLM, Semantic Search, and Python Engineering Bundles

Ahmed Adawy — Fri, 23 Oct 2026 05:56:08 +0000

If you are learning to build production-ready AI systems or improving your Python engineering workflow, I have two Leanpub bundles on discount until October 29, 2026.

LLM Foundations, Semantic Search & Production Engineering Bundle This bundle is aimed at developers who want to move beyond simple LLM demos and understand how to build practical systems involving:

LLM fundamentals

Semantic search

Production engineering practices

Discounted price: $12.15
Link:
https://leanpub.com/b/theproduction-readyllmautonomousaimasterybundle/c/LeanpubAuthorUpdate20261023

The Elite Python Optimization & Engineering Automation Bundle This bundle focuses on Python performance, clean engineering practices, and automation for real-world development work.

Discounted price: $9.11
Link:
https://leanpub.com/b/theelitepythonoptimizationengineeringautomationbundle/c/LeanpubAuthorUpdate20261023

Both coupon links are active from October 22 to October 29, 2026.

If you are deciding between them: choose the first if your focus is AI/LLM systems, and the second if your focus is Python engineering and automation.

Your LLM Is Slow. The GPU Might Not Be the Problem.

Ahmed Adawy — Thu, 22 Oct 2026 20:05:11 +0000

Your LLM Is Slow. The GPU Might Not Be the Problem.

A practical approach to tracing queueing, retrieval, prefill, and decoding in production AI systems.

A user asks a question.

Your application authenticates the request, searches a knowledge base, builds a prompt, calls an inference endpoint, and streams an answer.

The user waits.

Your dashboard says:

AI request duration: 11.4 seconds

That number tells you the experience was slow.

It does not tell you what to fix.

Should you change the model? Add replicas? Tune retrieval? Reduce the prompt? Investigate queueing? Fix a buffering proxy?

Before changing infrastructure, I would ask a more basic question:

Where did the time go?

One Request Contains Several Different Systems

Consider a retrieval-augmented application:

Client
│
API gateway
│
Authentication
│
Query embedding
│
Retrieval
│
Reranking
│
Prompt construction
│
Inference endpoint
│
Response streaming
│
Client rendering

This is an illustrative architecture, not a requirement that every application use all these components.

The important point is that the user experiences one operation while the application executes several.

A single request_duration measurement collapses those operations into one number.

That is useful for measuring the symptom. It is insufficient for choosing the intervention.

Google’s SRE guidance makes this distinction explicit: monitoring needs to answer both “what is broken?” and “why?” User-facing symptoms and internal causes are related, but they are not interchangeable. sre

Define the Measurement Boundary First

“Time to first token” sounds precise until two teams measure it differently.

An inference service might start its timer when it receives a request.

Your backend might start when it calls that service.

The browser might start when the user submits the question.

Those are different boundaries.

For an application, I would record at least these timestamps:

t0: Application receives request
t1: Retrieval starts
t2: Retrieval completes
t3: Model request is dispatched
t4: First content chunk arrives from model service
t5: First content chunk is forwarded to client
t6: Final response chunk is forwarded

If client instrumentation is available, add:

t7: First answer content is rendered

Name the measurements according to what they actually observe.

For example:

application_first_content_latency = t5 - t0
model_call_first_content_latency = t4 - t3
retrieval_duration = t2 - t1
stream_forwarding_delay = t5 - t4

A content chunk is not necessarily one model token. Avoid labeling a transport-level measurement as per-token latency unless the underlying data supports that interpretation.

Also, do not subtract timestamps from different machines as though their clocks were perfectly aligned. Prefer local elapsed-time measurements and a tracing system designed to represent distributed execution.

Separate Waiting From Working

Suppose the model call takes 2.3 seconds before returning its first content.

That observation alone does not reveal how much time was spent:

Traveling to the service.

Waiting for admission.

Waiting for execution resources.

Processing uncached prompt tokens.

Producing and returning the initial output.

If you operate the serving engine, expose those distinctions where possible.

If you use a hosted API and do not receive internal timing information, preserve that uncertainty.

Write:

Model-call first-content latency increased.
Internal queueing and execution times are not available.

Not:

Prefill became slower.

The second statement requires evidence that the first does not provide.

Instrument Request Paths, Not Just Components

A database dashboard can tell you the database is slow.

It cannot necessarily tell you whether the specific failed answer used that database path, a fallback path, or a cached result.

That is where distributed tracing becomes useful.

OpenTelemetry’s context-propagation model carries trace context across service boundaries. Logs can include trace and span IDs, making it possible to connect records to the execution that produced them. opentelemetry

For a proposed AI application, I would start with a span tree like this:

POST /answer
├── authenticate
├── embed_query
├── retrieve_documents
├── rerank_documents
├── construct_prompt
└── generate_answer
├── wait_for_first_content
└── consume_response_stream

The inference server may contribute additional internal spans if it is instrumented and propagates context appropriately.

Do not invent visibility that your deployment does not provide.

The goal is a navigable execution path, not an impressive-looking diagram.

Attach the Context That Makes Comparisons Useful

A trace becomes more informative when it records which version and workload it represents.

For the application design above, I would consider attributes such as:

service.version
deployment.environment
model.identifier
prompt.template_version
retrieval.index_version
input.token_count
output.token_count
retrieval.document_count

These are suggested fields; define naming and semantics consistently in your implementation.

The principle is more important than the exact names:

When performance changes, you need to identify what else changed.

Google’s SRE Workbook specifically recommends monitoring binary versions, feature flags, and dynamic configuration versions to help diagnose production regressions. sre

For an AI application, I would extend that discipline to prompt templates, retrieval configuration, and model routing.

Do not put secrets, authorization headers, or raw private prompts into telemetry by default. Decide what is necessary to collect, who can access it, and how long it should be retained.

An Illustrative Incident: Slow Answers After a Retrieval Change

Imagine this before-and-after comparison:

Measurement

Before

After

Retrieval duration

180 ms

190 ms

Documents selected

Input tokens sent to model

2,400

10,800

Model-call first-content latency

650 ms

2,300 ms

These are invented numbers for an engineering example, not a benchmark.

The database did not become dramatically slower.

The model did not change.

But the request sent to the model became much larger.

That gives us a hypothesis:

A retrieval-configuration change increased prompt size and contributed to higher first-content latency.

It does not yet prove the cause.

The next step is a controlled comparison:

Use the same evaluation questions.

Hold model selection and output limits constant.

Compare the old and new retrieval policies.

Control or record concurrency.

Record cache conditions where they are observable.

Measure answer quality as well as latency.

Reducing retrieved context is not automatically a successful optimization if it also removes the evidence needed to answer correctly.

The experiment needs both performance and quality criteria.

Compare Like With Like

An aggregate latency chart mixes workloads.

A short answer generated from a small prompt is not the same workload as a long answer generated from a large document collection.

Before concluding that a release caused a regression, I would break down results by a manageable set of dimensions:

Model or routing destination.

Input-length range.

Output-length range.

Application version.

Prompt-template version.

Deployment region.

Relevant request category.

Use request-specific details in logs and traces rather than turning every identifier into a metric label.

For example, a request ID is useful for finding one execution. It is not a useful grouping dimension for an application-wide latency dashboard.

The objective is to distinguish a genuine slowdown from a change in traffic mix.

“What Changed?” Is a Better Starting Point Than “What Looks Abnormal?”

When an incident begins after a release, the release is an obvious candidate.

But a temporal match is not proof.

The same caution applies to:

A model update.

A larger prompt.

A cache miss.

A traffic surge.

A dependency warning.

I would keep a short hypothesis table during an investigation:

Hypothesis

Supporting evidence

Evidence still needed

Retrieval change increased prompt-processing cost

Larger inputs after configuration change

Controlled old-versus-new comparison

Queueing increased under load

Higher first-content latency at higher concurrency

Serving-side admission timing

Streaming is buffered downstream

Backend receives content before client does

Proxy and client timing comparison

This makes uncertainty visible.

It also discourages the familiar incident pattern where a plausible explanation becomes the accepted explanation before anyone tests it.

An AI Investigation Assistant Should Have to Show Its Work

An LLM can help summarize traces, compare configurations, and generate candidate explanations.

That is a proposed role—not a claim that it can reliably identify root cause from arbitrary telemetry.

If I added an AI assistant to this workflow, I would require its output to separate:

Observation
Hypothesis
Supporting evidence
Missing evidence
Proposed test
Potential operational risk

A useful response might be:

Input-token counts increased after prompt-template version 12 was deployed. The sampled traces also show higher model-call first-content latency. This is consistent with increased prompt-processing work, but internal queue timing is unavailable. Compare versions 11 and 12 under matched load before attributing the regression to prefill.

That is more actionable than:

Root cause found: prompt too large. Roll back immediately.

The first answer explains what is known and proposes a test.

The second turns a hypothesis into an operational instruction.

A Minimal Investigation Checklist

Before buying more hardware or replacing the serving engine, I would check:

Does the latency measurement represent the user’s experience or only one backend call?

Can a failed request be followed across retrieval, generation, and streaming?

Are queueing and execution distinguishable where the infrastructure exposes them?

Did prompt size, output length, or traffic mix change?

Are deployments and configuration changes recorded?

Does the proposed fix preserve answer quality?

Can the suspected cause be tested without changing several variables at once?

This does not require collecting everything.

It requires collecting enough connected evidence to answer the next diagnostic question.

Where This Connects to My Book

I’m Ahmed Adawy, the author of AI Systems Engineering: From Prototype to Production.

The book sample explores the inference lifecycle, model-weight and KV-cache budgeting, prefill and decode, and serving metrics. It also includes an educational asynchronous serving simulator. ppl-ai-file-upload.s3.amazonaws

That foundation helps frame a useful distinction:

Understanding how an inference engine works is necessary for diagnosing its behavior—but application latency also includes the systems around it.

The simulator is an explanation tool, not a production benchmark. Likewise, the example in this article is a diagnostic scenario, not a claim of measured results from a customer deployment.

If you want to examine the book’s technical level and approach, start with the free sample:

Explore the book and read the sample on Leanpub

Disclosure: this is my book, and I earn from purchases.

The Question to Ask Before “Do We Need a Bigger GPU?”

A slow answer is an observation.

“Insufficient GPU capacity” is one possible explanation.

The engineering work is connecting the two with evidence.

Before changing hardware, establish where the request waited, what it processed, which dependencies it used, and what changed.

Otherwise, you may improve a component that was never the bootelneck.

# Your AI Assistant Remembered the Fact. It Still Got the Answer Wrong

Ahmed Adawy — Sun, 18 Oct 2026 21:13:12 +0000

Your AI Assistant Remembered the Fact. It Still Got the Answer Wrong.

Why reliable AI memory needs more than embeddings, longer context, or a better retrieval score.

An assistant remembers that Sarah works at Acme.

Then Sarah changes jobs.

A month later, you ask:

“Where was Sarah working when we met?”

The assistant confidently answers with her new employer.

It retrieved a real fact. It identified the right person. It may even have selected the most recently updated record.

And it still answered the wrong question.

This is a useful way to think about AI memory failures: sometimes the problem is not forgetting. It is remembering something without preserving the conditions under which it is relevant.

For an engineering design, I would treat validity as a first-class concern—not metadata added after retrieval.

One Sentence Can Contain Two Different Timelines

Consider this message:

“I moved to Pune in March.”

Suppose the user says it in June.

There are at least two distinct times to represent:

When the move occurred.
When the system learned about the move.

These timestamps answer different questions.

“Where did I live in April?” asks about the world.

“What did you know about my location in April?” asks about the system’s knowledge at that point.

Treating both as a single message timestamp loses that distinction.

This is the motivation behind a bitemporal representation: separate valid time—when a fact held in the world—from transaction or ingestion time—when the system recorded it. Graphiti’s architecture explicitly distinguishes these timelines and tracks validity and record lifecycle timestamps. arxiv

This is not a claim that every assistant needs a temporal graph database. It is a claim that the data model needs to distinguish questions the product promises to answer.

“Outdated” Does Not Mean “Useless”

Imagine these statements arriving over time:

“I work at Acme.”

“I’m joining Nova next month.”

“My start date moved back two weeks.”

“Actually, I decided to stay at Acme.”

A memory pipeline that simply overwrites current_employer risks collapsing several different concepts:

An existing state.
An intention.
A revised plan.
A canceled transition.

My proposed design rule is simple:

Do not turn every newly extracted sentence into an immediately valid replacement fact.

A plan is not an event. A correction is not always a new state. A later message can change how an earlier message should be interpreted.

Temporal memory systems such as Graphiti preserve superseded facts through invalidation rather than simply deleting them, allowing current and historical relationships to remain distinguishable. getzep

But preserving history is only part of the design. The reader must also choose the right historical state for the question.

Store Evidence Separately From Current Beliefs

I would separate three layers in a memory implementation.

1. Source evidence

What was actually said?

Keep the source message, speaker, timestamp, and enough surrounding context to interpret it.

2. Extracted assertions

What does the system think the source claims?

For example:

{
  "subject_id": "person_sarah_01",
  "predicate": "employed_by",
  "object_id": "company_acme_01",
  "assertion_type": "reported_state",
  "recorded_at": "2026-06-10",
  "valid_from": null,
  "valid_to": null,
  "source_message_id": "message_184"
}

This is an illustrative schema, not a complete temporal database design.

The null dates matter. If the source does not establish when employment started, the extractor should not invent a start date.

3. Derived views

What should the assistant currently believe?

A current-state view can select the assertions relevant to “Where does Sarah work now?”

A historical view can select evidence relevant to “Where did Sarah work when we met?”

These views should point back to the underlying evidence.

The architectural goal is to make interpretations revisable without losing the information used to produce them.

Identity Needs the Same Discipline

Now introduce two people named Sarah.

One is your colleague. The other is your sister.

A name alone cannot distinguish them in this example. But “name plus relationship” is not necessarily a permanent identifier either: colleagues become friends, managers change, and relationships can overlap.

I would use stable internal entity IDs, with names and relationships stored as evidence-backed attributes.

For ambiguous references, the write path should have more than two options.

Instead of only:

Create entity
or
Merge into existing entity

allow:

Create entity
Link with sufficient evidence
Hold as unresolved
Ask for clarification

A system that can preserve uncertainty does not have to turn every ambiguous mention into a confident merge.

Retrieval Should Select a State, Not Just a Sentence

Consider four questions:

Where does Sarah work?
Where did Sarah work last spring?
What did Sarah say about changing jobs?
What did we believe about Sarah’s employer before her correction?

They concern the same person and topic, but they ask for different evidence.

For this design, I would represent query intent explicitly enough to distinguish:

Current state.
State at a specified time.
Events or changes.
Previously recorded beliefs.

The retrieval pipeline could then look like this:

Question
   ↓
Entity resolution
   ↓
Temporal and question-intent interpretation
   ↓
Candidate retrieval
   ↓
Validity and provenance checks
   ↓
Evidence selection
   ↓
Answer or abstention

This is a proposed decomposition, not an argument that every box requires a separate model call.

The important part is that “most similar” and “appropriate evidence” are not treated as interchangeable.

Cross-References Should Not Become Conflicting Copies

Suppose a fact concerns both Sarah and Acme.

It should be discoverable through either entity.

For a structured implementation, I would prefer one canonical fact record referenced by both entity indexes rather than two independently editable copies:

Sarah index ──┐
              ├── fact_271
Acme index ───┘

That gives corrections one authoritative target.

Materialized copies may still be useful for performance, but then their consistency needs to be an explicit engineering responsibility.

A chapter metaphor can make navigation intuitive. It should not accidentally dictate an update model that is difficult to maintain.

A Good Retrieval Score Can Hide a Broken Write Path

Imagine a memory system with three stages:

Conversation → Extraction → Filing → Retrieval

A wrong answer could originate anywhere:

The extractor missed the relevant statement.
The entity resolver chose the wrong person.
The fact was assigned to the wrong attribute.
A correction was applied too broadly.
Retrieval selected the wrong time period.

Improving the retriever will not reveal which of these happened.

Before choosing the next model to train, I would run an ablation:

Condition	Extraction	Filing
A	Human-verified	Human-verified
B	Model-generated	Human-verified
C	Human-verified	Model-generated
D	Model-generated	Model-generated

Keep the retrieval procedure fixed.

Condition A is a reference condition for retrieval with verified inputs—not a theoretical upper bound.

The other conditions help investigate which upstream errors are responsible for the observed failures.

The evaluation should report those failures separately, rather than reducing every error to “memory recall.”

Test the Questions the System Should Refuse to Answer

A useful test set should contain more than questions with known, neatly stored answers.

I would include:

An unknown start date.
Two unresolved people with the same name.
A job offer that was never accepted.
A correction with no stated effective date.
Conflicting statements from different speakers.
A question whose answer was never provided.

For each case, define acceptable behavior in advance.

Sometimes the correct output is a fact.

Sometimes it is a qualified answer.

Sometimes it is:

“I know Sarah changed employers, but I don’t have enough information to determine where she worked on that date.”

An evaluation that always requires a factual answer cannot tell you whether the system handles missing evidence appropriately.

Measure the Assistant, Not Only the Memory Index

Retrieval evaluation is useful, but it does not automatically establish the performance of the complete assistant.

For a proposed system, I would track at least three groups of measurements.

Write-path measurements

Extracted-fact accuracy.
Entity assignment accuracy.
Attribute assignment accuracy.
Correction handling.
Update latency.

Read-path measurements

Relevant evidence retrieval.
Correct temporal selection.
Entity-confusion errors.
Unsupported-answer frequency.
Answerable versus unanswerable behavior.

End-to-end measurements

Final answer quality.
Tool calls per question.
Retrieved context tokens.
Response latency.
Recovery after incorrect writes.

A selective reader may reduce context size, but that remains an empirical question until the full reading loop is integrated and measured.

Likewise, an extra retrieval step might save prompt tokens while adding latency. The relevant trade-off belongs in the end-to-end results, not only in an architectural diagram.

Memory Is a Data-Lifecycle Problem

Semantic retrieval remains useful. It can help locate evidence when the user’s wording differs from the stored text.

But my design recommendation is to place it inside a broader lifecycle:

Capture
→ Interpret
→ Resolve identity
→ Record provenance
→ Establish validity
→ Retrieve
→ Verify
→ Correct

Existing systems already address parts of this lifecycle. Graphiti documents temporal invalidation and historical relationships; Mem0 exposes a memory-history interface for tracking how an individual memory has changed. Neither should be reduced to “just a pile of embeddings.” help.getzep

The useful engineering question is therefore not simply:

“How do we give an assistant memory?”

It is:

“What evidence does this assistant preserve, how does it revise that evidence, and under what conditions is it allowed to use it?”

An assistant that remembers yesterday’s fact is useful.

An assistant that can explain whether the fact is still true—and recognize when it cannot tell—is a more demanding system to build.

Your AI Prototype Is Not a Production System: The Engineering Behind Reliable AI Applications

Ahmed Adawy — Sun, 18 Oct 2026 03:48:47 +0000

Your AI Prototype Is Not a Production System: The Engineering Behind Reliable AI Applications

From containerization and deployment pipelines to observability, failure recovery, and scaling — what it actually takes to operate AI software in the real world.

By Ahmed Adawy

An AI application can look impressive during a demonstration and still be a long way from production-ready.

The model returns a useful answer. The API responds correctly. The retrieval pipeline finds relevant documents. Everything works when one developer runs the application locally.

Then the environment changes.

A model provider becomes slow. Several requests arrive simultaneously. A deployment introduces an incompatible dependency. The database reaches its connection limit. A background task fails after partially processing a document. An application restart interrupts work that was never persisted.

None of these problems necessarily means the underlying AI model is bad.

They reveal something more fundamental: the model is only one component of a software system.

The engineering challenge is to make the entire system predictable enough to deploy, diagnose, secure, scale, and maintain.

That distinction is the central idea behind production AI engineering.

My book, The AI Engineer — Volume II: Deploy. Scale. Operate., focuses on the engineering practices that become essential after an AI application moves beyond the prototype stage.

Explore the book on Leanpub

1. The Real Architecture Is Larger Than the Model

Consider a document intelligence service that accepts a file, extracts its text, generates embeddings, retrieves relevant information, and sends selected context to an LLM.

The visible feature might be summarized as:

Document → Embeddings → Retrieval → LLM → Answer

But a deployable application has a much larger execution path:

Client Request
      |
      v
API Gateway / Application API
      |
      v
Authentication and Input Validation
      |
      v
Application Service
      |
      +------> Database
      |
      +------> Object Storage
      |
      +------> Embedding Provider
      |
      +------> Vector Index
      |
      +------> LLM Provider
      |
      v
Response and Operational Telemetry

Every external dependency introduces a potential failure boundary. Every additional processing step can contribute latency. Every stored document introduces questions about access control, retention, and data integrity.

The system therefore needs to answer questions that a successful model call cannot answer:

What happens when an upstream service times out?
How are duplicate requests handled?
Which component is responsible for retrying a failed operation?
How do we know whether a deployment is healthy?
Can we identify the slowest stage of a request?
What happens to work that is interrupted by a restart?
How do we prevent logs from exposing sensitive information?

These are architectural questions, not merely operational details.

2. Containerization Is About Reproducibility, Not Just Packaging

A Python application depends on more than its source files. Its behavior can depend on the Python runtime, package versions, operating-system libraries, environment variables, filesystem assumptions, and startup configuration.

Docker provides a way to package an application and its dependencies into a container image. That image becomes a versioned artifact that can move through development, testing, and deployment environments.

A minimal FastAPI example illustrates the idea:

FROM python:3.12-slim

ENV PYTHONDONTWRITEBYTECODE=1
ENV PYTHONUNBUFFERED=1

WORKDIR /app

COPY requirements.txt .

RUN pip install --no-cache-dir -r requirements.txt

COPY . .

EXPOSE 8000

CMD ["uvicorn", "app.main:app", "--host", "0.0.0.0", "--port", "8000"]

This assumes the application entry point is app/main.py, the FastAPI instance is named app, and the dependency file contains the required packages.

Build and run it:

docker build -t ai-service .
docker run --rm -p 8000:8000 ai-service

The commands are straightforward. The engineering decisions around them matter more.

For example, a production image should use controlled dependency versions, avoid copying secrets into the image, exclude irrelevant files through .dockerignore, and consider running as a non-root user. The build process should also account for vulnerability scanning and the maintenance of base images.

Containerization does not make software automatically secure or reproducible. It gives the team a consistent unit to build, inspect, test, distribute, and deploy.

That consistency is what makes it valuable.

3. CI/CD Should Verify the Application, Not Merely Deploy It

A deployment pipeline is often described as a sequence of automation steps:

Commit
  ↓
Tests
  ↓
Build
  ↓
Publish Image
  ↓
Deploy

That is a useful starting point, but a successful deployment command is not proof that the application is healthy.

A stronger pipeline separates several concerns:

Code quality: static checks and formatting rules.
Behavior: unit and integration tests.
Artifact integrity: build and inspect the container image.
Security: scan dependencies and images where appropriate.
Deployment: release a known artifact to the target environment.
Verification: check application health and important behavior.
Recovery: define what happens if verification fails.

AI applications need particular care around testing.

A test suite that calls a live LLM for every assertion can become slow, expensive, and nondeterministic. Many application behaviors can instead be tested using controlled fake clients.

For example, a service should be testable independently of the external model provider:

from typing import Protocol


class TextGenerator(Protocol):
    def generate(self, prompt: str) -> str:
        ...


class SummaryService:
    def __init__(self, generator: TextGenerator) -> None:
        self.generator = generator

    def summarize(self, text: str) -> str:
        if not text.strip():
            raise ValueError("Text must not be empty")

        return self.generator.generate(
            f"Summarize the following text:\n{text}"
        )


class FakeGenerator:
    def generate(self, prompt: str) -> str:
        return "Deterministic test summary"


service = SummaryService(FakeGenerator())

result = service.summarize("A document about distributed systems.")

assert result == "Deterministic test summary"

This example tests application logic without making a network request or consuming model tokens.

It does not verify the real provider's behavior, so provider integration tests are still needed. The point is to test each boundary at the appropriate level.

A dependable pipeline combines fast deterministic tests with targeted integration tests and deployment checks.

4. Latency Must Be Measured Across the Entire Request

Suppose an AI endpoint takes 2.4 seconds to return a response.

That number alone does not tell us what needs to be optimized.

The request may spend time validating input, querying a database, generating an embedding, retrieving documents, waiting for an LLM, and serializing the result.

A simplified latency model is:

$$
T_{\text{total}} =
T_{\text{validation}} +
T_{\text{database}} +
T_{\text{embedding}} +
T_{\text{retrieval}} +
T_{\text{LLM}} +
T_{\text{response}}
$$

This is an approximation. Parallel operations, queueing, retries, and overlapping work can make real request timing more complicated.

Still, the model helps establish an important rule: measure individual stages before optimizing the system.

If the LLM accounts for most of the request time, optimizing a small Python function may have almost no effect on the user's experience.

If database connection acquisition dominates the delay, adding more application workers could make the problem worse by increasing pressure on the database.

If embedding generation is repeated for identical inputs, caching may help — provided the cache key, model version, and invalidation behavior are designed correctly.

Performance engineering begins with a hypothesis, a measurement, and a controlled change. It should not begin with adding infrastructure simply because the application feels slow.

5. Retries Can Recover From Failure — or Amplify It

Network calls fail. Providers throttle requests. Temporary errors occur.

Retries can help, but they are not automatically safe.

Imagine a service receiving 100 requests per second. If every request fails and each request retries three times, the upstream dependency may receive several times the original traffic precisely when it is already struggling.

A basic exponential backoff strategy increases the delay between attempts:

$$
d_n = \min(d_{\max}, d_0 \cdot 2^n)
$$

Here, (d_0) is the initial delay, (n) is the retry index, and (d_{\max}) is the maximum delay.

In practice, randomized jitter is often added to reduce synchronized retries from many clients.

Retries should also be bounded by an overall deadline. The application needs to distinguish transient failures from permanent errors, respect provider rate limits, and avoid retrying non-idempotent operations without a safe duplication strategy.

For operations that create records, charge money, enqueue jobs, or trigger external side effects, idempotency can be especially important.

The correct question is not simply, "Should we retry?"

It is: "Which failures are retryable, how many attempts are acceptable, what is the total time budget, and what prevents a retry from duplicating work?"

That is production engineering.

6. Observability Is How You Find Out What Actually Happened

A log message saying Request failed is rarely enough to diagnose a distributed application.

Useful observability combines three complementary signals.

Logs provide event-level context: which operation failed, what type of error occurred, and which request was involved.

Metrics reveal trends: request rate, error rate, latency percentiles, resource usage, queue depth, and dependency failures.

Traces help follow a request through multiple components and identify where time was spent.

For an AI service, additional measurements can include:

Model-provider latency and timeout rate.
Input and output token counts.
Embedding generation duration.
Retrieval duration and result counts.
Retry frequency and rate-limit responses.
Background-job completion and failure rates.
Estimated cost per request, where reliable usage and pricing data are available.

These measurements help distinguish different failure modes.

A rising response time with normal database latency may indicate a model-provider issue. Increasing queue depth may indicate that background workers cannot keep up with incoming work. A growing error rate after deployment may point to a configuration or dependency change.

Observability does not prevent every failure. It reduces the time required to understand and respond to failures.

One caution is essential: operational visibility must not become uncontrolled data collection. Avoid logging credentials, access tokens, full private documents, or sensitive prompts without a justified and carefully governed reason.

7. Kubernetes Is a Tool, Not a Production Strategy

Kubernetes can manage containerized workloads, schedule services, support health-based restarts, and provide mechanisms for scaling and service discovery.

But adopting Kubernetes does not automatically make an application reliable.

A service can still have poor health checks, unsafe database migrations, missing alerts, weak access controls, or no tested recovery procedure while running inside a sophisticated cluster.

The operational complexity also matters. Teams need to understand resource requests and limits, deployment configuration, service networking, secrets, storage, observability, and the consequences of scaling.

For a small application, a managed application platform may be the more sensible choice. For a system with multiple workloads, complex scheduling needs, established operational practices, and a clear requirement for orchestration, Kubernetes may be appropriate.

The decision should follow the problem.

A useful engineering principle is to adopt the simplest infrastructure that satisfies the application's actual reliability, security, and scaling requirements — and to revisit the decision when those requirements change.

8. Scaling Is About Bottlenecks and Trade-Offs

Adding more application instances increases available compute capacity only when the application and its dependencies can use that capacity effectively.

A stateless API may scale horizontally with relatively little difficulty. A database with a limited connection pool may not. An external LLM provider may enforce request or token limits. A vector database may become the dominant source of latency.

Scaling decisions must therefore consider the entire request path.

Common techniques include:

Caching repeated, valid computations.
Moving long-running tasks into background workers.
Improving database indexes and query plans.
Reusing connections safely.
Applying concurrency limits to expensive dependencies.
Batching operations when the workload supports it.
Reducing unnecessary payload sizes.
Measuring resource usage and queueing under representative load.

Each technique has costs.

Caching creates invalidation and consistency problems. More workers can increase database contention. Batching may improve throughput while increasing individual request latency. Aggressive concurrency can exhaust memory or trigger provider throttling.

The target is not maximum throughput at any cost. It is a system that meets its performance and reliability requirements at a sustainable operational cost.

9. Production Readiness Is a Set of Verifiable Conditions

Before calling an AI application production-ready, I would want clear answers to questions like these:

Deployment

Can the same artifact be promoted through environments?
Are configuration and secrets handled outside the application image?
Can a release be verified and, if necessary, rolled back?

Reliability

Are timeouts and retry policies bounded?
What happens when a dependency becomes unavailable?
Can interrupted background work be recovered safely?

Security

Are authentication and authorization enforced?
Are inputs validated and resource limits defined?
Are sensitive records and credentials protected?

Observability

Can the team investigate a failed request?
Are latency, errors, and resource saturation measured?
Do alerts identify actionable problems rather than generate noise?

Performance

Have representative workloads been measured?
Are the main bottlenecks understood?
Can the system handle expected concurrency without exhausting dependencies?

Maintenance

Are tests automated?
Are architecture and operational assumptions documented?
Can another engineer understand, deploy, and troubleshoot the system?

A checklist cannot guarantee reliability, but it can expose missing decisions before users discover them through failures.

Production readiness is not a label awarded because an application has Docker, Kubernetes, or a CI/CD pipeline. It is the result of engineering choices that can be explained, tested, and verified.

10. The Engineering Journey Behind This Book

These topics are connected, and they become much more useful when learned as parts of a complete system.

Containerization establishes a reproducible runtime. CI/CD turns changes into controlled releases. Cloud deployment introduces operational constraints. Observability helps diagnose behavior. Security protects the service and its data. Performance measurements guide scaling decisions. Architecture connects these practices into a maintainable application.

That progression is the focus of my book:

The AI Engineer — Volume II: Deploy. Scale. Operate.

The volume covers Docker and containerization, Linux and production operations, Git and GitHub workflows, CI/CD, cloud deployment, Kubernetes, observability, production security, scaling, architecture patterns, and production-readiness practices. It also includes a final engineering challenge and reference appendices.

It is intended for Python developers, backend engineers, AI engineers, and software developers who want to understand what it takes to operate AI applications beyond a local development environment.

The goal is not to collect tools or copy infrastructure configurations without understanding them. It is to build the engineering judgment needed to choose the right approach, recognize failure modes, and make systems easier to operate.

Read the book on Leanpub:

https://leanpub.com/theaiengineervolumeiideployscaleoperate

Final Thought

A model can produce an impressive answer in a few seconds.

A dependable AI product requires much more: controlled releases, secure data handling, measurable performance, observable failures, and recovery mechanisms that work when conditions are less than ideal.

That work is not separate from AI engineering.

It is part of AI engineering.

Build beyond the prototype. Deploy deliberately. Measure what matters. Design for failure.

Ahmed Adawy is the author of The AI Engineer — Volume II: Deploy. Scale. Operate.

# From AI Prototype to Production: A Practical Guide to Deploying, Scaling, and Operating AI Systems

Ahmed Adawy — Sat, 17 Oct 2026 04:44:41 +0000

By Ahmed Adawy

Your AI application works on your machine. Your API returns responses. Your model integration looks promising. Your tests pass.

Then production introduces a different set of problems.

Dependencies behave differently across environments. Deployments become fragile. API latency increases under load. Secrets end up in the wrong place. Logs reveal too little when something breaks. A service that worked perfectly during development becomes difficult to monitor, secure, and maintain.

This is where AI engineering becomes production engineering.

Building an AI-powered application is only one part of the job. Operating it reliably requires a broader understanding of software architecture, deployment automation, infrastructure, observability, security, and performance.

That is the engineering problem behind my book:

The AI Engineer — Volume II: Deploy. Scale. Operate.

Read the book on Leanpub:
https://leanpub.com/theaiengineervolumeiideployscaleoperate

The Missing Step Between Building AI and Operating AI

Modern AI development makes it relatively easy to connect a Python application to an LLM, expose an API, generate embeddings, or build a semantic search pipeline.

But a working demonstration is not the same as a dependable production system.

Consider an AI document-processing service. Its basic workflow might look like this:

Accept a document through an API.
Validate the input.
Extract and normalize the text.
Generate embeddings or call an AI model.
Store the results.
Return a response to the client.

That workflow describes functionality, but it does not answer the operational questions.

What happens if the model provider times out? How does the application recover from a failed deployment? Where are credentials stored? How do you detect rising latency? Can the service handle concurrent requests? How do you prevent a database connection pool from becoming a bottleneck? How can you reproduce a production failure locally?

These are not secondary concerns. They determine whether an AI application can be operated reliably.

1. Docker: Make the Runtime Reproducible

An application depends on more than its source code. It also depends on a Python version, system libraries, installed packages, configuration, and runtime assumptions.

Docker helps package an application and its dependencies into a reproducible container image.

For a Python API built with FastAPI, a basic Dockerfile might look like this:

FROM python:3.12-slim

ENV PYTHONDONTWRITEBYTECODE=1
ENV PYTHONUNBUFFERED=1

WORKDIR /app

COPY requirements.txt .

RUN pip install --no-cache-dir -r requirements.txt

COPY . .

EXPOSE 8000

CMD ["uvicorn", "app.main:app", "--host", "0.0.0.0", "--port", "8000"]

This example assumes that requirements.txt exists and that the application exposes a FastAPI instance named app in app/main.py.

Build the image:

docker build -t ai-document-service .

Run the container:

docker run --rm -p 8000:8000 ai-document-service

The API should now be reachable at http://localhost:8000, assuming the application starts successfully.

This is a starting point, not a complete production security configuration. A production image should also consider a non-root user, dependency pinning, image scanning, secret management, and an appropriate build context.

The objective is to reduce environmental surprises and make the application easier to build, test, and deploy consistently.

2. CI/CD: Turn Deployment Into a Repeatable Process

Manual deployment creates opportunities for inconsistent steps, forgotten checks, and configuration mistakes.

A continuous integration and continuous delivery pipeline can automate the path from a code change to a deployable artifact.

A typical workflow looks like this:

Developer Push
      |
      v
Automated Tests
      |
      v
Static Checks
      |
      v
Build Container Image
      |
      v
Security and Artifact Checks
      |
      v
Deploy to Target Environment
      |
      v
Health Verification

The precise stages depend on the project, but the principle remains the same: make important deployment steps explicit and repeatable.

For an AI application, automated checks can cover request validation, authentication, document processing, database behavior, model-client error handling, and API contracts.

Not every test should call a live LLM. Many application tests can use deterministic fake clients to avoid unnecessary cost, network dependencies, and unpredictable model output.

A reliable pipeline should tell you not only that a deployment command completed, but also whether the deployed application is healthy enough to serve requests.

3. Cloud Deployment: Design for the Actual Runtime

Deploying an application to the cloud is not simply a matter of uploading code.

You need to understand the runtime environment, network access, configuration, storage, process management, health checks, and failure behavior.

An AI application might depend on:

A FastAPI service for HTTP requests.
A relational database for application records.
Object storage for uploaded documents.
An embedding service or vector database for semantic retrieval.
An external LLM provider.
Background workers for long-running tasks.

Each dependency creates an operational boundary.

If the model provider becomes slow, should the API wait indefinitely? If document ingestion takes several minutes, should it run inside the original HTTP request? If a deployment restarts the application, what happens to queued work?

These questions influence architecture before the first production incident occurs.

4. Kubernetes: Understand Orchestration Before Adding Complexity

Kubernetes provides mechanisms for deploying and managing containerized workloads, including scheduling, service discovery, health-based restarts, and scaling.

However, Kubernetes is not automatically the right choice for every AI application.

A small service may be easier and cheaper to operate on a managed application platform. A larger system with multiple services, more demanding availability requirements, and established infrastructure practices may benefit from orchestration.

The important engineering skill is understanding the trade-offs.

Before adopting Kubernetes, identify the operational problem you are solving. Then evaluate deployment complexity, resource requirements, observability, scaling behavior, team experience, and ongoing maintenance.

Infrastructure should serve the application rather than become an objective in itself.

5. Observability: Make Failures Investigable

When an AI application fails, a single message saying “Internal Server Error” rarely provides enough information to identify the cause.

Useful observability combines several sources of evidence:

Logs help explain what happened during a request.

Metrics reveal trends such as latency, request volume, error rates, memory consumption, and resource saturation.

Traces help follow a request across services and identify where time is being spent.

AI applications also benefit from measuring model-specific behavior, including provider latency, token usage, timeout frequency, retry counts, and retrieval performance.

For example, a slow response could originate from database access, embedding generation, a remote model call, or a queue that is falling behind. Without useful measurements, those possibilities are difficult to distinguish.

Operational logs must be designed carefully. Avoid recording API keys, access tokens, private documents, or unnecessary personal information. Diagnostic detail should improve investigation without exposing sensitive data.

6. Production Security: Protect the Entire AI Workflow

AI security is not limited to the model itself.

A production system may expose API endpoints, process untrusted files, store confidential information, communicate with external providers, and execute background jobs.

Security therefore needs to be considered across the complete application.

Important areas include:

Authentication and authorization.
Secret and credential management.
Input validation and request-size limits.
Dependency and container-image security.
Database permissions and secure connections.
Rate limiting and abuse prevention.
Safe handling of uploaded documents.
Appropriate retention and protection of sensitive data.
Monitoring for suspicious access and unexpected usage.

An application can produce correct AI outputs and still be insecure if its API exposes data, its credentials are mishandled, or its resource limits are poorly designed.

Security is an architectural responsibility, not a final checklist added just before launch.

7. Scaling and Performance: Measure Before Optimizing

Scaling an AI application involves more than adding servers.

Performance may be limited by database queries, connection pools, synchronous operations, network latency, large request payloads, embedding generation, model-provider limits, or inefficient background processing.

Different bottlenecks require different solutions.

Caching can reduce repeated work when the cached result remains valid. Background jobs can move long-running operations outside the request-response path. Database indexing can improve query performance. Batching may improve throughput when the underlying service supports it. Concurrency controls can help protect limited resources.

But every optimization introduces trade-offs.

Caching creates invalidation questions. Retries can amplify traffic during an outage. Larger batches can increase memory use and response latency. More workers can overwhelm a database or external API.

A sound performance process starts with measurements, identifies the actual bottleneck, applies a targeted change, and measures the result again.

8. Architecture and Production Readiness

A maintainable AI system separates responsibilities instead of placing every operation in one large request handler.

For example:

API Layer
    |
    v
Application Services
    |
    +---- Document Processing
    |
    +---- Embedding and Retrieval
    |
    +---- Model Integration
    |
    +---- Persistence
    |
    +---- Background Jobs
    |
    v
Infrastructure and External Services

The boundaries do not have to correspond to separate microservices. They can begin as modules inside a single application.

The goal is to make responsibilities clear, dependencies explicit, and important behavior testable.

A production-readiness review should ask whether the system can be deployed repeatedly, configured safely, monitored effectively, recovered after failure, and maintained without relying on undocumented manual steps.

What This Book Covers

The AI Engineer — Volume II: Deploy. Scale. Operate. focuses on the engineering practices that take AI applications beyond the prototype stage.

The book covers:

Docker and containerization.
Linux and production operations.
Git, GitHub, and engineering workflows.
Continuous integration and continuous deployment.
Cloud deployment and infrastructure.
Kubernetes and cloud-native operations.
Observability, logging, and monitoring.
Production security.
Scaling and performance engineering.
AI engineering architecture patterns.
Production-readiness reviews.
A final engineering challenge and reference materials.

The material is organized around the progression from deployment fundamentals to operating and evaluating a more complete production system.

This volume is intended for Python developers, backend engineers, AI engineers, and software developers who want to understand what happens after an AI application works locally.

It is also useful for engineers who want a more structured way to think about deployment decisions, operational failure modes, and the trade-offs involved in running real-world AI systems.

The Goal Is Not Just to Make It Work

A successful prototype proves that an idea can work.

A production system must also be deployable, observable, secure, recoverable, and maintainable.

That difference is where much of the real engineering begins.

If you are building AI applications and want to strengthen the engineering practices around deployment, infrastructure, monitoring, security, and scaling, explore the book here:

The AI Engineer — Volume II: Deploy. Scale. Operate.

https://leanpub.com/theaiengineervolumeiideployscaleoperate

Author: Ahmed Adawy

Build beyond the prototype. Understand the system. Operate it with confidence.

Topics

AI Engineering, Production AI, Python, FastAPI, Docker, CI/CD, Kubernetes, Cloud Deployment, MLOps, Observability, AI Infrastructure, Distributed Systems, Software Architecture, DevOps, Performance Engineering

Transitioning to Production AI Engineering: A Technical Review of "The AI Engineer Bootcamp in a Book"

Ahmed Adawy — Thu, 08 Oct 2026 05:07:32 +0000

With the explosive growth of Generative AI, developers can easily write simple scripts that call Large Language Model (LLM) APIs. However, there is a massive gap between running a quick prototype in a Jupyter Notebook and building a production-grade AI system that is reliable, secure, and scalable.
This is where "The AI Engineer Bootcamp in a Book - Volume I: Build the Foundations" by Ahmed Adawy comes in. It serves as a comprehensive roadmap for developers looking to move past the amateur phase and enter the world of AI engineering with rigorous engineering standards.
Beyond the APIs: Why AI Engineering Matters
AI is often treated as a black box: you input text and get a response. But a true AI Engineer doesn't treat models in isolation from software infrastructure. A fully integrated production system requires:
Clean and robust software architecture to structure code and maintain scalability.
High-performance APIs built with FastAPI.
Reliable relational databases like PostgreSQL for efficient data persistence.
Production-grade authentication and authorization mechanisms.
Rigorous testing and debugging strategies to guarantee system stability.
Core Pillars of Volume I (Build the Foundations)
The book follows a project-based engineering journey, guiding readers step-by-step through the following technical pillars:
1. Software & Engineering Foundations
Shifting mindset to think like a production engineer.
Structuring professional Python applications.
Building production APIs with FastAPI and designing persistent data systems with PostgreSQL.
Implementing robust security, authentication, and testing practices.
2. AI Engineering Fundamentals
A deep dive into vector representations via Embeddings and vector similarity calculations.
Understanding Semantic Search and smart data retrieval mechanics.
Exploring Retrieval-Augmented Generation (RAG) architectures to connect models with private company data.
Managing context, handling multiple model providers, and achieving structured outputs.
3. Pipelines & System Performance
Handling heavy document ingestion and processing for technical pipelines.
Utilizing caching and background processing to optimize performance and reduce latency/costs.
Performance tuning to handle high traffic and concurrent requests.
Who This Book Is For
This book is designed specifically for:
Python developers transitioning into AI engineering.
Backend developers building AI-powered applications.
Software engineers working with LLMs and GenAI who want to move beyond prototypes.
Students and technical professionals seeking a production-oriented understanding of AI systems to lay the groundwork for cloud infrastructure, Kubernetes, and scalability covered in Volume II.

Conclusion & Call to Action
If you are looking for a practical reference that bridges the gap between AI theory and real-world production development, this book is a solid investment in leveling up your engineering skills.
You can explore the book details, read a free sample, and grab your copy via the link below:
The AI Engineer Bootcamp in a Book on Leanpub

Demystifying LLM Serving Infrastructure: How PagedAttention and Continuous Batching Scale Inference

Ahmed Adawy — Sun, 04 Oct 2026 20:40:57 +0000

Demystifying LLM Serving Infrastructure: How PagedAttention and Continuous Batching Scale Inference
Moving a Large Language Model (LLM) from a local prototype in a Jupyter notebook to a high-throughput, multi-tenant production environment is a brutal awakening. While data scientists spend months optimizing model weights, quantization, and fine-tuning, infrastructure engineers face a completely different set of demons: memory fragmentation, queue latency, and GPU starvation.
In a production serving environment, the bottleneck is rarely just the raw compute power (FLOPs) of the GPU; it is almost always memory bandwidth and capacity.
In this deep dive, we will unpack the two foundational pillars that revolutionized LLM serving engines like vLLM: PagedAttention and Continuous Batching.

The Silent Killer: Understanding the KV Cache Memory Explosion During autoregressive generation, an LLM generates tokens one by one. To avoid recalculating the Key and Value matrices for all previous tokens at every step, inference engines cache these tensors in GPU VRAM—collectively known as the KV Cache. The memory footprint of the KV cache scales linearly with sequence length and batch size: For a 70B parameter model with a 4K context window, the KV cache can easily consume tens of gigabytes of VRAM per request. The Flaw of Traditional Static Allocation Historically, serving frameworks allocated a contiguous chunk of VRAM upfront based on the model’s maximum possible sequence length (e.g., 4096 or 8192 tokens) to prevent out-of-memory errors during generation. This creates two massive inefficiencies:
- Internal Fragmentation: If a user only requests a 200-token response, the remaining pre-allocated memory for that slot sits idle and unusable.
- External Fragmentation: Varying request lengths make it nearly impossible to neatly pack memory blocks, leading to massive blocks of unused, stranded VRAM. As a result, traditional systems often waste 60% to 80% of their GPU VRAM, drastically choking concurrency.
PagedAttention: Virtual Memory for LLMs To solve memory fragmentation, researchers borrowed a concept that operating systems have used for decades to manage RAM: Virtual Memory and Paging. Introduced by vLLM, PagedAttention partitions the KV cache of each sequence into small, fixed-size blocks (e.g., 16 tokens per block). These blocks can be stored non-contiguously in physical GPU memory. [ Traditional Contiguous Allocation ] [ Request A: Used ][ Request A: Unused/Wasted (Max Len) ]

[ PagedAttention Block-Based Allocation ]
Block Table (Logical -> Physical Mapping):
Req 1 -> [Block #12] -> [Block #5] -> [Block #28]

How It Works Under the Hood:

The Block Table: Each request maintains a logical-to-physical block table managed by the inference engine.
On-Demand Allocation: Blocks are allocated on the fly as new tokens are generated, rather than reserving a massive block upfront.
Memory Sharing (Copy-on-Write): PagedAttention enables efficient memory sharing for advanced features like parallel sampling, beam search, and multi-turn chat branching, where multiple sequences share common prompt prefixes. The Impact: Memory waste drops from ~70% to under 4%. This allows the GPU to pack significantly more concurrent requests into VRAM, directly multiplying throughput.
1. Continuous Batching (Iteration-Level Scheduling) Memory management solves capacity, but what about scheduling efficiency? In traditional deep learning batching (static batching), a batch of requests is formed, sent to the GPU, and the entire batch must wait until every request finishes generating its final token. This leads to severe GPU starvation:
Short requests finish early, leaving their allocated slots empty while waiting for the longest request in the batch to complete.
Incoming requests must wait in an external queue until the current batch fully clears out.
Iteration-Level Scheduling to the Rescue
Continuous Batching (or iteration-level scheduling) changes the scheduling granularity from the request level down to the iteration (token) level.

Conceptual loop of Continuous Batching

while active_requests_queue or running_batch:

1. Complete one forward pass (generate 1 token for all active sequences)

logits = model.forward(running_batch)

2. Check for completed sequences

for req in running_batch:
if req.is_finished():
running_batch.remove(req)
# Immediately backfill with a new request from the queue
if active_requests_queue:
running_batch.add(active_requests_queue.pop())

By decoupling request lifecycles from batch lifecycles:

As soon as a request generates an end-of-sequence token, its slot is instantly freed.
A waiting request from the queue is slotted in on the very next forward iteration.
GPU compute units remain saturated near 100%, slashing average latency and increasing overall system throughput by up to 23x compared to static batching.
1. Bringing It All Together in Production When architecting an LLM serving stack today, building from scratch is rarely necessary thanks to production-grade engines built on these exact principles. Whether you deploy using vLLM, TensorRT-LLM, or TGI (Text Generation Inference), understanding these internals is critical for:
Right-sizing GPU instances: Knowing your KV cache block size helps calculate exact VRAM requirements per concurrent user.
Tuning max model len: Preventing over-allocation and maximizing concurrency limits.
Optimizing chunked prefill: Handling large context windows without stalling the generation queue. Key Takeaways for AI Engineers:
VRAM is your primary constraint: Optimize KV cache management before throwing more hardware at the problem.
Batching is dynamic: Never rely on static batch sizes for autoregressive text generation.
Infrastructure is code: Low-level memory scheduling decisions dictate your API's cost-per-token economics. What serving engine are you currently using in your production stack? Let’s discuss in the comments below! Artificial Intelligence, LLM, Machine Learning Infrastructure, و Python.

Code Is Not Cheap: Why Software Fundamentals Matter More Than Ever in the AI Era

Ahmed Adawy — Sat, 03 Oct 2026 20:43:10 +0000

An AI coding agent can inspect your repository, spin up a new endpoint, modify the database schema, wire the frontend, generate unit tests, handle errors, and produce a migration—all in under ten minutes. A task that once consumed an engineer’s entire afternoon can now be triggered with a single prompt.
The pull request sits there, green and ready. And this is precisely where the expensive part of software engineering begins.
The Illusion of Completion
Writing code has never been cheaper. Reading, verifying, operating, and living with the long-term consequences of that code, however, has never been more demanding.
When an AI agent generates a complete feature implementation, it operates in a vacuum of syntax and local context. It rarely accounts for the systemic ripples of its decisions. Consider the hidden questions that standard AI generation glosses over:

State Ownership: Does this new endpoint actually own this state, or is it creating a distributed monolith anti-pattern?
Idempotency: Can this request safely execute twice if a network timeout triggers an automatic retry?
Failure Modes: What happens if the database write succeeds, but the downstream event publishing fails?
Database Migrations: Can this schema migration roll back cleanly under heavy production load, or will it lock critical tables?
Compatibility Promises: Did the agent introduce a subtle breaking change in an internal API contract without anyone noticing? Answering these questions requires deep systems thinking—something a probabilistic token predictor cannot do for you. The Trap of Naive Abstractions When code generation is free, there is a strong temptation to let AI add layers of abstraction to make the implementation easier to generate. We see wrapper classes, overly complex async patterns, and generic layers that solve theoretical problems while introducing massive performance bottlenecks. In high-performance domains—such as scientific computing, low-level Python optimization, or LLM infrastructure engineering—naive abstractions destroy throughput. If you don't understand how Python’s asyncio event loops interact with memory allocation, or how GPU VRAM management and continuous batching function under heavy loads, your AI-generated system will crumble the moment it hits real production traffic. Operating What You Didn't Build The true cost of software is not creation; it is operation. When a production system throws a cascading error at 3:00 AM, the fact that the code was written in eight minutes by an AI agent provides zero comfort. You are the one who has to debug the stack trace, trace memory leaks, reason about concurrent state mutations, and explain to stakeholders why the system failed when reality violated the agent's implicit assumptions. AI is an incredible force multiplier, but it is a terrible architect. It accelerates implementation while amplifying the importance of fundamental engineering principles. Master the Foundation If you want to stay ahead in an industry flooded with automated code generation, your competitive edge is no longer how fast you type syntax. It is your mastery of system architecture, concurrency, data structures, and production-grade engineering. To help bridge the gap between rapid prototyping and bulletproof production systems, I’ve put together a comprehensive resource covering low-level software engineering, LLM infrastructure, and semantic search architecture: 👉 Explore The Generative AI & LLM Engineering Bundle on Leanpub Tags: Software Engineering Artificial Intelligence System Architecture Python DevOps

LLM Inference Engineering: Overcoming the KV-Cache Bottleneck and Maximizing Production Throughput

Ahmed Adawy — Fri, 02 Oct 2026 19:26:23 +0000

, you immediately hit the hard wall of AI systems engineering: memory management and GPU VRAM bandwidth.
The bottleneck is no longer just the static model weights; it’s the dynamic resources consumed by the model during text generation.

The Real Crisis: The Hidden Cost of KV-Cache During autoregressive generation, the model computes and stores the key and value states of the attention mechanism (the KV-Cache) for every previous token to avoid recalculating them with each new token.
- The Engineering Problem: The size of this cache scales linearly with the context length and the concurrent batch size.
- The Consequence: This leads to Memory Fragmentation. Traditional frameworks allocate contiguous, fixed-size memory blocks based on the maximum expected sequence length, wasting up to 60–80% of VRAM without actual utilization and frequently triggering the dreaded CUDA Out of Memory error.
The Architectural Solution: Virtual Memory and PagedAttention Just as operating systems (OS) solved limited RAM crises via Virtual Memory and paging, advanced LLM infrastructure relies on the same concept through algorithms like PagedAttention:
- How It Works: The KV-Cache is partitioned into small, fixed-size blocks or pages (e.g., 16 or 32 tokens).
- Memory Management: Pages do not need to be physically contiguous in GPU memory. A memory manager dynamically maps logical blocks to physical blocks.
- Direct Impact: Virtually eliminates memory fragmentation, enables block sharing in identical prompt caching scenarios, and multiplies VRAM utilization efficiency.
Throughput Optimization: Continuous Batching vs. Static Batching In traditional systems, batching relies on a static queuing mechanism (Static/Naive Batching): if one request finishes early, the rest of the batch must wait for the longest request to complete, causing severe GPU underutilization.
- The Engineering Fix (Continuous/Iteration-level Batching): Instead of waiting for an entire batch to finish, new requests are injected and completed requests are evicted at each generation step (iteration).
- Technical Result: Drastically reduces latency and exponentially increases throughput (tokens per second per compute dollar) in high-load production environments. Engineering Takeaway Building robust AI systems goes far beyond calling APIs; it demands rigorous control over low-level optimization, cache management, and resource allocation under heavy load. If you want to dive deeper into software architecture, advanced AI systems design, and high-performance inference tooling, check out the complete details and technical references in The Generative AI & LLM Engineering Bundle.

LLM Engineering AI Infrastructure Python GPU Optimization Performance Engineering Machine Learning Systems DevOps

Artificial Intelligence Software Engineering Programming Cloud Computing Deep Learning

Why "Working Code" Isn't Enough (A Journey into Computational Physics & High-Performance Engineering) We’ve all been there: you write so

Ahmed Adawy — Thu, 01 Oct 2026 21:33:00 +0000

Why "Working Code" Isn't Enough (A Journey into Computational Physics & High-Performance Engineering)
We’ve all been there: you write some code, it runs fine on your local machine or a test server, and you call it a day. But the moment it hits real production load, massive data, or edge cases, performance drops and things start breaking.
That’s usually where the real gap shows—the difference between code that just "works" and code built on rock-solid engineering foundations.
If you look at the heavy-hitting challenges in tech right now, real solutions don't come from just memorizing the latest framework. They come from a deep understanding of three core areas that rarely get combined properly:
Computational Physics: Learning how to translate real-world behaviors, simulations, and complex systems into precise mathematical and numerical models instead of guessing.
Cyber Defense: Moving beyond basic out-of-the-box configurations to genuinely understand how systems are secured right from the root level.
High-Performance Algorithms: The secret weapon that lets software squeeze every ounce of power out of hardware efficiently, rather than choking CPUs and wasting RAM.
Why Does This Matter?
Tackling problems this way completely shifts how you think. Instead of being a developer who just plugs in tools, you become an engineer who actually understands the magic happening under the hood.
If you're looking to level up your engineering mindset and dive deep into these fields without the fluff, I came across a focused, highly practical bundle that covers all of this in one place:
👉 The Computational Physics, Cyber Defense & High-Performance Algorithms Bundle
It’s built for anyone who wants to stop relying on quick fixes and start building robust, secure, and lightning-fast systems.
Have you ever had to optimize a system under heavy constraints? Let me know your thoughts or drop a comment below—let's chat about it!

PRODUCTION-GRADE AI SYSTEMS: 5 ARCHITECTURAL SHIFTS FROM PROTOTYPE TO SCALE

Ahmed Adawy — Sun, 27 Sep 2026 03:35:21 +0000

PRODUCTION-GRADE AI SYSTEMS: 5 ARCHITECTURAL SHIFTS FROM PROTOTYPE TO SCALE

Moving an LLM application or a Retrieval-Augmented Generation pipeline from a local Jupyter notebook to a production environment is where most engineering teams hit a brick wall. A prototype built with naive chunking, synchronous Python loops, and unoptimized inference endpoints looks great in a demo, but under real-world concurrency, it collapses under latency spikes, memory bloat, and compounding token costs.

Drawing from patterns in building scalable AI infrastructure and production systems, here are five crucial architectural shifts you need to make to transition your AI prototypes into robust, production-grade systems.

DITCH NAIVE CHUNKING FOR HIERARCHICAL AND SEMANTIC RETRIEVAL

Most RAG failures are not retrieval failures; they are chunking failures. Splitting documents strictly by fixed character lengths destroys semantic context, cutting sentences in half and breaking logical flow.

The Fix: Implement Hierarchical Semantic Chunking. Group text by structural boundaries like headers, section breaks, or paragraph semantic distance using embedding shifts rather than arbitrary lengths. Store parent-child relations: retrieve granular child chunks for precise vector matching, but feed the broader parent context to the language model.

UNCORK PURE PYTHON LOOPS IN DATA INGESTION PIPELINES

When processing millions of tokens, cleaning corpuses, or building vector embeddings, standard synchronous Python loops over records will bottleneck your CPU.

The Fix: Shift from procedural loops to vectorized operations using NumPy and Pandas, or leverage asynchronous processing and multiprocessing pools for Input/Output bound embedding API calls. Avoid heavy object creation inside hot loops to keep memory overhead predictable and prevent garbage collection pauses.

OPTIMIZE KV CACHING AND MEMORY MANAGEMENT FOR INFERENCE

As context windows expand to massive token limits, Key-Value cache memory consumption explodes. Standard serving setups often waste massive amounts of VRAM due to memory fragmentation and static allocation.

The Fix: Adopt memory-efficient attention mechanisms like PagedAttention, which is similar to virtual memory paging in operating systems, alongside continuous batching. This allows dynamic sharing and recycling of cache memory across requests, drastically increasing throughput and reducing GPU VRAM waste.

TREAT AGENTIC WORKFLOWS AS DISTRIBUTED STATE MACHINES

When building multi-agent systems or complex tool-use loops, letting models decide their own control flow indefinitely leads to infinite loops, runaway token bills, and cascading failures.

The Fix: Enforce strict deterministic boundaries. Define explicit state transitions, maximum step limits, and rigorous validation checks between sub-agents. Treat your agent orchestrators like distributed state machines where every tool execution has a clear success condition, timeout, and fallback policy.

DECOUPLE AND ORCHESTRATE AI WORKLOADS ON KUBERNETES

Running LLM inference servers, vector databases, worker queues, and API gateways on monolithic virtual machine instances creates single points of failure and resource contention.

The Fix: Containerize your AI services and deploy them on Kubernetes. Utilize GPU node affinity, horizontal pod autoscaling based on queue length and custom metrics, and robust liveness and readiness probes to handle failover gracefully under heavy traffic surges.

SUMMARY

Transitioning from prototype to production is not about writing more complex code; it is about stripping away naive assumptions. Build with clear boundaries, optimize your data pipelines at the hardware and memory level, and treat your infrastructure with the rigor of a senior systems architect.

Why Your Pure Python Loops Are Destroying Your AI Model’s Performance (And How to Fix Them)

Ahmed Adawy — Fri, 25 Sep 2026 17:29:04 +0000

Moving beyond standard scripts to build production-grade, high-throughput Python and AI pipelines.
We’ve all been there: you write a clean, elegant Python pipeline for preprocessing embeddings, tokenizing text, or handling data streams, only to watch it crawl at a snail's pace in production.
When building modern AI systems and heavy data pipelines, standard Python for loops, CPython bytecode overhead, dynamic typing, and the Global Interpreter Lock (GIL) can quickly turn a high-end GPU-accelerated workflow into an I/O and CPU bottleneck.
If you want to squeeze maximum performance out of your code, you need to shift from writing scripts that work to engineering high-performance systems. Let’s look at three core pillars to instantly accelerate your Python & AI codebases.
1. Vectorization Over Iteration
Stop writing manual loops over large arrays. By leveraging vectorized operations using NumPy and PyTorch tensors, you bypass CPython's per-element interpreter overhead and push computations down to optimized C/C++ backend kernels.
The Impact: This alone frequently yields 50x to 120x speedups compared to standard Python iteration. Instead of processing data item-by-item in Python space, let underlying hardware do the heavy lifting in contiguous memory blocks.
2. Respect Memory Layout and Cache Locality
Performance isn't just about CPU cycles; it's about memory architecture. Pointer chasing and fragmented memory allocations destroy CPU cache lines.
The Impact: Ensuring contiguous memory layouts (such as C-contiguous arrays in NumPy) drastically reduces cache misses, speeds up matrix multiplications, and prevents the dreaded "sawtooth" memory fragmentation graph under heavy concurrent workloads.
3. Eliminate Garbage Collection and Allocation Stalls
In high-throughput loops, frequent object creation triggers CPython's reference counting and garbage collector.
The Impact: These micro-stalls add up over millions of iterations. Pre-allocating buffers, reusing memory pools, and understanding internal memory allocation mechanics are absolute game-changers for real-time inference and data ingestion pipelines.
Want the Complete Engineering Blueprint?
I’ve codified these production-grade optimization patterns, CPython internals deep-dives, and actionable architectural checklists into a comprehensive engineering guide: "High-Performance Python for AI & Data Engineering".
It covers 4 core modules designed specifically to take your pipelines from sluggish prototypes to lightning-fast production systems:
Module 1: The Anatomy of Python Slowness & CPython Internals
Module 2: Vectorization over Iteration (NumPy & PyTorch Mastery)
Module 3: Memory Layout, Caching & Garbage Collection
Module 4: Production Benchmarks & Architectural Checklist

👉 Get your copy now:
Digital Edition on Payhip ($9)
Professional Publishing Edition on Leanpub
How are you currently handling performance bottlenecks and memory fragmentation in your high-throughput pipelines? Let’s discuss in the comments below!

DEV Community: Ahmed Adawy

Limited-Time Leanpub Discounts: LLM, Semantic Search, and Python Engineering Bundles

Your LLM Is Slow. The GPU Might Not Be the Problem.

# Your AI Assistant Remembered the Fact. It Still Got the Answer Wrong

Your AI Assistant Remembered the Fact. It Still Got the Answer Wrong.

Why reliable AI memory needs more than embeddings, longer context, or a better retrieval score.

One Sentence Can Contain Two Different Timelines

“Outdated” Does Not Mean “Useless”

Store Evidence Separately From Current Beliefs

1. Source evidence

2. Extracted assertions

3. Derived views

Identity Needs the Same Discipline

Retrieval Should Select a State, Not Just a Sentence

Cross-References Should Not Become Conflicting Copies

A Good Retrieval Score Can Hide a Broken Write Path

Test the Questions the System Should Refuse to Answer

Measure the Assistant, Not Only the Memory Index

Write-path measurements

Read-path measurements

End-to-end measurements

Memory Is a Data-Lifecycle Problem

Your AI Prototype Is Not a Production System: The Engineering Behind Reliable AI Applications

Your AI Prototype Is Not a Production System: The Engineering Behind Reliable AI Applications

1. The Real Architecture Is Larger Than the Model

2. Containerization Is About Reproducibility, Not Just Packaging

3. CI/CD Should Verify the Application, Not Merely Deploy It

4. Latency Must Be Measured Across the Entire Request

5. Retries Can Recover From Failure — or Amplify It

6. Observability Is How You Find Out What Actually Happened

7. Kubernetes Is a Tool, Not a Production Strategy

8. Scaling Is About Bottlenecks and Trade-Offs

9. Production Readiness Is a Set of Verifiable Conditions

10. The Engineering Journey Behind This Book

Final Thought

# From AI Prototype to Production: A Practical Guide to Deploying, Scaling, and Operating AI Systems

The Missing Step Between Building AI and Operating AI

1. Docker: Make the Runtime Reproducible

2. CI/CD: Turn Deployment Into a Repeatable Process

3. Cloud Deployment: Design for the Actual Runtime

4. Kubernetes: Understand Orchestration Before Adding Complexity

5. Observability: Make Failures Investigable

6. Production Security: Protect the Entire AI Workflow

7. Scaling and Performance: Measure Before Optimizing

8. Architecture and Production Readiness

What This Book Covers

The Goal Is Not Just to Make It Work

Topics

Transitioning to Production AI Engineering: A Technical Review of "The AI Engineer Bootcamp in a Book"

Demystifying LLM Serving Infrastructure: How PagedAttention and Continuous Batching Scale Inference

Conceptual loop of Continuous Batching

1. Complete one forward pass (generate 1 token for all active sequences)

2. Check for completed sequences

Code Is Not Cheap: Why Software Fundamentals Matter More Than Ever in the AI Era

LLM Inference Engineering: Overcoming the KV-Cache Bottleneck and Maximizing Production Throughput

Why "Working Code" Isn't Enough (A Journey into Computational Physics & High-Performance Engineering) We’ve all been there: you write so

PRODUCTION-GRADE AI SYSTEMS: 5 ARCHITECTURAL SHIFTS FROM PROTOTYPE TO SCALE

Why Your Pure Python Loops Are Destroying Your AI Model’s Performance (And How to Fix Them)