DEV Community: Bilgin Ibryam

Latency Patterns for Faster AI Applications

Bilgin Ibryam — Sat, 29 Aug 2026 20:16:40 +0000

A practical map for shortening the path from user intent to a useful AI response.

When you type a question into ChatGPT, latency is the time until the first useful response appears. The same applies when you click on a dashboard and wait for the first useful chart. That interval may include browser work, authentication, an API gateway, application logic, database queries, an LLM call, and the final rendering. The user waits for all the required work on this critical path, not only for one isolated component.

This is why a fast database query or LLM call does not prove that the application is fast. We need to measure the full path as a distribution and identify what controls the percentile we care about. Once we know what dominates the critical path, we can bring requests and data closer, reduce work, execute independent work concurrently, or anticipate predictable work. These are general distributed-system patterns that apply equally to AI applications. The map below adapts latency optimization ideas from Pekka Enberg’s Latency book into four practical pattern categories.

An agentic request may cross RAG retrieval, databases, a model gateway, one or more LLMs, MCP or other tool servers, and an orchestration loop. A large part of the latency often comes from the LLM calls, but the model is still only one segment of the path. Model-serving optimizations include quantization, decoding, and server scheduling. I cover those in this talk and this post. In this post, I explicitly focus on non-model latencies across the application path.

Each category explains the patterns through traditional application examples, then compares how they apply in an AI application. Taken together, I see 19 latency patterns across four categories.

Locality Patterns

The first thing to check is whether the request and the data it needs are far apart. That distance may be geographical, between services, between processes on one host, or even between a CPU core and memory. There are four common patterns for shortening this part of the path.

1. Colocation Pattern

Colocation shortens or removes a boundary between the source and the target. The scale can vary significantly. A CDN can serve the dashboard shell closer to the user. Application logic can run in the same region as its database instead of crossing a wide-area network. On one host, CPU affinity and NUMA-aware placement can keep hot work and memory near the cores that use them. The mechanism is the same even when the distances are very different.

When to use. Use colocation when the distance between components that frequently interact is a measurable part of the critical path.

The main trade-off. Tighter placement can reduce scheduling flexibility and make failover or scaling across locations more difficult.

2. Replication Pattern

Replication places a readable copy in more than one location. A nearby replica can shorten the read path, but only when the application is allowed to read from it. For example, a dashboard can read tenant data from the nearest regional replica instead of crossing regions for every query.

When to use. Use replication for read-heavy paths that can tolerate a clearly defined level of staleness.

The main trade-off. Synchronous replication may keep the write waiting for required acknowledgements, while asynchronous replication can respond earlier but allow lag. The application must decide which reads can tolerate older data, how they are routed, and what should happen during a failure.

3. Partitioning Pattern

Partitioning divides data or work into units that can be placed and processed independently. It does not make anything closer by itself, but a good partition key can keep a tenant’s query and data together. For example, a dashboard can partition metrics by tenant so that most queries touch one shard rather than several.

When to use. Use partitioning when requests and data share a stable key that can keep related work within one partition or location.

The main trade-off. A poor key can create a hot partition, while a request that spans several partitions brings routing and coordination back onto the critical path.

4. Caching Pattern

Caching creates a shortcut for repeated reads or computation. The hit ratio tells us how often the shortcut works, while the miss penalty tells us how painful the original path remains. For example, a dashboard can cache common aggregates close to the application instead of recalculating them for every page load.

When to use. Use caching when reads or computations repeat and their results can be safely reused.

The main trade-off. A high hit ratio can still produce a bad tail when misses are expensive. Freshness, invalidation, and ownership determine whether the cached result can be trusted.

Choosing What to Bring Closer

In a traditional application, start with the kind of distance that creates the delay. Use colocation when two components communicate frequently, replication when reads are far from the source, partitioning when related data can be kept together, and caching when the same data or computation is requested repeatedly. The patterns can be combined, but each addresses a different reason for travelling across the system.

In an AI application, apply the same test to the orchestrator, model gateway, retrieval store, state store, and tools. Colocate dependencies that communicate frequently, replicate read-heavy knowledge where residency rules allow it, partition state by tenant or session, and cache repeated retrieval, schema discovery, session summarization, policy lookup, or model prefill work. Cache keys must include every input that can change the meaning of a result, including tenant and authorization scope.

Work Reduction Patterns

If the latency trace points at computation as the bottleneck, the next move is to remove work. Work that no longer exists cannot delay the request, consume capacity, or create another failure point.

5. Algorithmic Work Reduction Pattern

An index over tenant and time can replace a full scan. A hash lookup can replace a linear search. Filtering before an expensive join or sort can reduce the dataset before the costly operation begins. These changes reduce the amount of work rather than making the same work slightly faster.

When to use. Use this pattern when query or computation cost dominates and a better algorithm, index, or data structure can reduce the work.

The main trade-off. A faster access path may require additional indexes, memory, preprocessing, or implementation complexity.

6. Selective Data Processing Pattern

A dashboard does not need the full object graph when the first chart uses five fields. Building large intermediate objects, copying them between layers, serializing them, and then discarding most of the result consumes time without changing what the user sees. A lean response should carry only what the next step needs.

When to use. Use selective data processing when the next step needs only a subset of the available data.

The main trade-off. Removing too much data can trigger another request or prevent a later step from completing.

7. Setup Reuse Pattern

Reusing an established connection, a parsed schema, or validated orchestration state avoids repeating the same setup on every request.

When to use. Use setup reuse when the same initialization cost is paid repeatedly and the resulting state can be retained safely.

The main trade-off. Reused state needs explicit lifecycle, freshness, isolation, and failure handling.

8. Request Coalescing Pattern

Combining several known reads into one request can remove repeated serialization, scheduling, and round trips. This is where request batching can help latency. For example, a dashboard backend can fetch the summary, alerts, and recent events in one call instead of making three sequential requests.

When to use. Use request coalescing when several predictable calls cross the same boundary and can be safely combined.

The main trade-off. A combined request can make one slow operation hold back all the others. It should be measured against the user-visible latency target, not assumed to be faster.

9. Runtime Tuning Pattern

Allocation, garbage collection, demand paging, scheduling, and context switches can all delay useful work. Profile the critical path first, then reduce only the runtime costs that measurably affect it. For example, a service may reduce short-lived allocations or tune its heap after profiles tie p99 spikes to garbage-collection pauses. These costs are measurable suspects, not default explanations for every slow service.

When to use. Use runtime tuning when profiles show that runtime overhead affects the percentile you are trying to improve.

The main trade-off. Runtime tuning can increase implementation complexity and may depend on a specific runtime, operating system, or hardware configuration.

Choosing What Work to Remove

In a traditional application, first identify what kind of work dominates the trace. Algorithmic work reduction changes how the result is found. Selective data processing reduces how much data moves through the path. Setup reuse avoids repeating initialization, while request coalescing removes repeated boundary crossings. Runtime tuning comes last because it is useful only when profiles point to the runtime itself.

In an AI application, remove model and tool calls before optimizing them. Use deterministic code or indexed search for direct lookup, filter and rank evidence before it enters the prompt, reuse model and tool connections where safe, and combine calls that always happen together. Route simple tasks to a smaller model when it meets the same quality target, stop generation when the useful answer is complete, and bound tool output before it becomes prompt input. Smaller prompts help only when they remain sufficient. If missing context causes another retrieval and model turn, the end-to-end path may become longer.

Once the remaining work is necessary, the next question is whether it really has to run in sequence.

Concurrent Execution Patterns

Some work is necessary but does not need to happen in sequence. Suppose authentication must finish first, but four dashboard panels are then independent. Running them serially makes the user wait for roughly the sum of their durations. Starting them together can bring the wait closer to the slowest required panel, plus scheduling and coordination overhead.

10. Synchronization Avoidance Pattern

A contended lock can make otherwise short operations line up behind one owner. Immutable state may remove the shared dependency, while a single-owner design may replace lock contention with an explicit queue. The important question is whether another operation must finish or release something before this one can continue.

When to use. Use synchronization avoidance when coordination over shared state is a measurable source of waiting and the state can be made immutable, partitioned, or owned by one component.

The main trade-off. Removing a lock may move the waiting elsewhere or require a different state-management model.

11. Independent Concurrency Pattern

Concurrency lets several tasks make progress, while parallelism executes work at the same time. For the dashboard, separate panel reads can run concurrently. In an agentic application, a research agent can query independent document stores and market-data sources in parallel, then join the results before synthesis. If one tool needs an identifier returned by another, that dependency remains serial.

When to use. Use independent concurrency when required tasks have no data dependency and the system has enough capacity to run them together.

The main trade-off. Concurrency does not help when one task genuinely needs the result of another, and it can make latency worse when the fan-out exhausts a shared dependency.

12. Progressive Response Pattern

An event loop can use resources more efficiently while external I/O is pending, but it does not make the database, model, or remote service intrinsically faster. The visible latency improves when an AI application streams useful model output or an independently useful tool result instead of holding everything until the full task completes.

When to use. Use a progressive response when a partial result is independently useful and time to first useful result matters to the user.

The main trade-off. This pattern shortens time to first useful result, not full completion, and the client must handle partial or provisional state.

13. Concurrency Budget Pattern

A queue stores waiting; it does not remove it. If arrivals exceed capacity, the queue grows and tail latency follows. Set a concurrency budget, propagate deadlines and cancellation, apply backpressure, and stop work when the user no longer needs it. For example, a dashboard can allow only four panel queries to run at once rather than sending an unbounded fan-out to the database.

When to use. Use a concurrency budget when one request can fan out across services, databases, models, or tools and overload a shared dependency.

The main trade-off. A budget that is too low leaves capacity unused, while one that is too high moves the queue into a downstream dependency.

14. Hedged Requests Pattern

Request hedging reduces exposure to a straggler by sending equivalent copies and accepting the first valid response. For example, after a short delay, a service can send the same read to a second replica and use whichever valid response arrives first.

When to use. Use hedged requests when rare stragglers dominate tail latency, spare capacity exists, and the operation is idempotent or side-effect-free.

The main trade-off. Hedging spends extra capacity and can worsen overload if it is used without a strict budget.

Choosing What Can Run Together

In a traditional application, synchronization avoidance removes unnecessary waiting, while independent concurrency overlaps work that has no dependency. A progressive response changes when useful output becomes visible, not how fast the full operation completes. A concurrency budget protects shared capacity, while hedged requests spend additional capacity to reduce exposure to rare stragglers.

In an AI application, use concurrency for independent model, tool, and retrieval calls, but keep dependent calls serial. A progressive response is a separate delivery decision: it changes when useful output becomes visible, not which tools can run in parallel. Apply one concurrency budget across models, subagents, tools, and retrieval, and hedge only side-effect-free or idempotent operations. The goal is a shorter dependency chain, not the largest possible fan-out.

Anticipation Patterns

If a remaining delay cannot be removed or safely overlapped after the request arrives, we can anticipate likely work and move it earlier.

15. Predictive Prefetching Pattern

Once authentication identifies the user, the application may fetch the default dashboard before the browser asks for its first panel. This is a prediction about a future read.

When to use. Use predictive prefetching when the next read is predictable enough to justify occasionally wasted work.

The main trade-off. Prefetching helps only when the data arrives in time and is still useful when the application needs it. A wrong prediction consumes resources without shortening the request.

16. Optimistic Update Pattern

A filter can appear selected while persistence continues, as long as the interface shows that the change is pending and can roll it back after a failure. This improves perceived latency after the user acts. It does not make durable completion happen earlier.

When to use. Use optimistic updates when success is common, failure is visible, and the operation can be safely reversed.

The main trade-off. A failed operation requires rollback, and provisional state must never be presented as completed work.

17. Speculative Execution Pattern

Speculative execution runs a likely branch before the actual choice arrives. For example, a dashboard can start calculating the most common time range while the user is still opening the time-range filter.

When to use. Use speculative execution when a small number of likely branches can run safely, independently, and be cancelled.

The main trade-off. The work must be isolated and cancellable because a wrong prediction consumes resources without helping the request.

18. Precomputation Pattern

Maintaining a tenant aggregate as events arrive can replace repeated foreground computation with a read. Precomputation moves known work from the foreground request to the point where the source data changes.

When to use. Use precomputation when an expensive result can be refreshed as its source changes rather than calculated for every request.

The main trade-off. Precomputed results consume storage and refresh capacity, and they may be stale when the source changes faster than they can be updated.

19. Prewarming Pattern

A small worker pool, connection pool, or execution environment can be initialized before traffic arrives. Prewarming moves predictable setup cost off the foreground critical path.

When to use. Use prewarming when cold-start or setup cost is predictable and keeping limited capacity ready is affordable.

The main trade-off. Warm resources consume capacity even when no request uses them.

Choosing What to Do Earlier

In a traditional application, predictive prefetching prepares a likely read, optimistic updates show likely success before durable completion, and speculative execution starts a likely branch. Precomputation prepares reusable results when source data changes, while prewarming prepares resources before requests arrive. The difference is what is moved earlier: data, feedback, a branch, a result, or capacity.

In an AI application, prefetch likely context once intent is known, run safe read-only searches for likely branches, precompute embeddings, indexes, summaries, and policy metadata, and prewarm agent workers or model and tool connections. An optimistic interface may show a tool action as pending before it commits, but never as completed.

For all of these patterns, track whether the early work was used, whether it completed in time, how much critical-path time it saved, how much work was discarded, and whether optimistic state had to roll back. Cancel early work when it is no longer useful. A prediction that delays normal traffic is not an optimization.

Find the Bottleneck, Choose the Pattern

Do not start by choosing a pattern. Start by tracing the full request and locating where delay enters the critical path: context retrieval, model access, tools, agent coordination, or verification and delivery.

Read the matrix in two moves. First, choose the column where the delay appears. Then move down that column and select the intervention that matches its cause:

Locality Patterns: The request, data, or dependency is too far away.
Work Reduction Patterns: The application performs unnecessary computation, data processing, or calls.
Concurrent Execution Patterns: Independent work is waiting in sequence.
Anticipation Patterns: Predictable work remains on the foreground path.

The cells are starting points, not a checklist. Pick the smallest pattern that changes the dominant part of the path. Before applying it, check whether it will amplify the tail or trade away quality, freshness, privacy, cost, or correctness. Then measure the same latency clock at the target percentile together with task success. If the bottleneck moves, return to the matrix.

Emerging Markdown Formats That Shape Coding Agent Behavior

Bilgin Ibryam — Sun, 16 Aug 2026 19:40:13 +0000

A practical map of the rules, procedures, plans, domain knowledge, and memory that guide agent-assisted software changes.

For years, the knowledge required to change software lived in many places. Non-functional requirements (NFRs), architectural constraints, and design decisions were captured in solution architecture documents, ADRs, security reviews, and wikis. Product intent lived in design documents and mockups. Work was broken down into tickets. Operational lessons survived in incident reports, Slack threads, playbooks, and the heads of experienced developers.

The repository contained the implementation: source code, tests, configuration, and dependency manifests. Developers read across the surrounding artifacts, reconciled the desired outcome with architectural judgment, and translated both into code.

As AI-assisted coding becomes normal, the repository is becoming the meeting point between that project knowledge and the agent changing the software. A monorepo brings codebases and their dependency graph together. An agent-ready repository also brings in the knowledge required to change them correctly.

Software has always been understood through different lenses: principles and NFRs, architecture and design, concrete specifications, implementation plans, domain rules, and lessons learned during operation. A growing family of Markdown files now captures these dimensions in forms that coding agents can discover and use. This article looks at the formats and tools that are emerging. Together, they let an agent act as an intent compiler, translating human intent and judgment into source code.

Agent onboarding starts with a README for agents

A human onboards to a project once. An agent effectively onboards every time it starts a task. It needs to orient itself in the repository, discover the relevant commands and conventions, and understand the rules that apply before it changes anything.

AGENTS.md is the closest thing to a neutral standard for this purpose. The project describes it as a README for agents. A root file can explain the repository structure, build commands, tests, coding conventions, and pull request expectations. Nested files can add instructions for a package or subsystem. The format deliberately has no required fields.

The ecosystem also has vendor-native alternatives that serve the same purpose:

AGENTS.md provides cross-tool project guidance for Codex, Cursor, Cline, Copilot, OpenCode, and other compatible agents.
CLAUDE.md and .claude/rules/ provide standing and path-scoped guidance for Claude Code.
GEMINI.md provides hierarchical context for Gemini CLI.
.github/copilot-instructions.md and path-specific instruction files guide GitHub Copilot.
.clinerules/ carries persistent and conditional rules for Cline.

Use pnpm, not npm. Do not edit generated clients. Run the billing integration tests after changing an invoice flow. Read the relevant ADR before moving a service boundary. These are standing instructions that should remain true across many tasks.

Once the agent understands the repository, the next question is how to perform a particular kind of work.

Skills package repeatable procedures

A skill is a playbook tailored to a particular kind of work.

Consider a database migration. AGENTS.md might state that every schema change needs a rollback path. A migration skill can describe the complete procedure: inspect the current schema, create the migration, update generated types, run compatibility checks, verify the rollback, and prepare the review summary.

The open Agent Skills specification gives that playbook a portable package. Each skill has a required SKILL.md containing metadata and instructions. It may also include scripts, references, templates, and other assets. Agents initially see the metadata, load the full instructions when the skill is relevant, then access supporting material as needed.

Agent Skills originated at Anthropic and were released as an open standard. They are separate from MCP, which Anthropic contributed to the Agentic AI Foundation. The skill format is now documented by Codex, Gemini CLI, GitHub Copilot, and other compatible clients.

The boundary is concrete:

AGENTS.md and native rule files describe standing context and constraints.
SKILL.md describes a repeatable procedure invoked for a specific task.

A procedure explains how to work. It still needs a reviewed description of what should be built.

Planning files make software design reviewable

Software design, planning, and task breakdown are also moving into versioned Markdown. When they exist only in chat, they disappear into session history. When they become repository artifacts, architects and developers can review them before an agent turns them into implementation.

GitHub Spec Kit is the clearest mainstream example. Its documentation reports more than 121,000 GitHub stars and 35 coding-agent integrations. Its default workflow uses a chain of files:

spec.md records the requirements and desired outcome.
plan.md explains the technical design and implementation approach.
tasks.md breaks the plan into executable units.
The agent implements and validates the change.

A project constitution can carry principles that should apply across many changes.

Spec Kit is not the only approach. OpenSpec, Kiro Specs, and project-specific PLANS.md use different files and workflows. They are nevertheless converging on a recognizable sequence: requirements, design, plan, tasks, implementation, and validation become separate artifacts that humans can review and agents can follow.

These files describe a generic software development lifecycle. Applications also need instructions for their particular domain.

Domain files give agents specialized context

A frontend, an identity platform, and a data pipeline do not need the same context. Domain files give agents specialized instructions for the parts of a system that generic onboarding, skills, and plans cannot fully describe.

ARCHITECTURE.md can describe system boundaries, components, dependencies, NFRs, and long-lived architectural decisions. It is an established human documentation convention that becomes operational for agents when project instructions tell them when to read it.
Google Labs' DESIGN.md describes visual identity, design tokens, components, and design rationale. The project currently labels the format alpha.
WorkOS' AUTH.md is an experimental, service-hosted recipe that teaches agents how to register and authenticate on behalf of users.
Kilo's REVIEW.md lets a repository define review priorities, severity, skipped files, verification expectations, and how a review agent should divide work.

These formats have different maturity levels and scopes. ARCHITECTURE.md is a long-standing documentation convention. DESIGN.md, AUTH.md, and REVIEW.md are emerging, domain-specific experiments. Their common direction is clear: more specialized engineering knowledge is becoming directly legible to agents.

Some of that knowledge is designed in advance. Other knowledge emerges only while the software is being built and operated.

Memory makes learned context durable

An unexpected build prerequisite, a debugging pattern, or the reason a deployment failed might previously have ended up in a ticket comment, commit message, Slack thread, incident report, or SRE playbook. Those records range from ephemeral to durable, but a coding agent may not find the relevant lesson when it begins the next task.

Agent memory keeps learned context associated with a repository and available across sessions. Claude Code makes the authorship distinction unusually clear:

Humans write CLAUDE.md to provide shared instructions, conventions, and project context.
Claude writes auto memory into a MEMORY.md index and optional topic files such as debugging.md and api-conventions.md.

The auto memory is repository-specific but machine-local. Claude normally stores it outside the Git repository, and it is not automatically shared with the team. Anthropic documents the two mechanisms separately.

Memory therefore does not replace tickets, commit history, or operational documentation. It reduces repeated rediscovery. When a learned fact becomes important to the whole team, it needs a promotion path:

The agent records a tentative observation in local memory.
A human reviews it when the observation recurs or affects shared work.
Durable knowledge moves beside the source code into a project rule, skill, ADR, or domain document.

Tentative knowledge stays local. Reviewed knowledge becomes shared. Kilo is experimenting with a similar loop for review: it can analyze how a team responds to review comments and propose changes to the repository's review guidance.

Putting the Markdown files together

Shared project files carry reviewed knowledge beside the source code. Agent-written memory remains machine-local until a durable lesson is promoted into the repository.

The practical distinction is the role each file plays, who writes it, and how broadly it is supported.

The formats range from cross-tool standards and established vendor conventions to emerging domain-specific experiments.

A project does not need every file. Use the smallest set that agents can reliably discover, keep reviewed knowledge beside the source code, and treat vendor-specific files as thin compatibility bridges when another file is canonical. AUTH.md remains service-hosted, while Claude's auto memory stays machine-local until a human promotes a durable lesson into the repository.

The takeaway

The repository is becoming the meeting point between source code and metacode: the Markdown that carries requirements, NFRs, architecture, design, rules, procedures, plans, and durable learning.

When agents can read both, they act as intent compilers. Source code becomes a byproduct: the executable result of human intent and judgment made legible to the agent.

The engineering task is to keep that metacode scoped, reviewed, and current. It does not replace testing or review. It gives the next code change a better source.

Originally published at The Generative Programmer.

10 Open-Source Projects for Securing AI Agent Skills

Bilgin Ibryam — Sun, 09 Aug 2026 12:42:15 +0000

In July, Hugging Face disclosed an intrusion driven end to end by an autonomous AI agent, starting with a malicious dataset and reaching credential harvesting and lateral movement across internal clusters. SkillScan researchers analyzed 31,132 public skills and flagged potentially dangerous patterns in 26.1%, while 5.2% showed high-severity patterns strongly suggesting malicious intent. Days later, NVIDIA and founding members launched the Open Secure AI Alliance to build and share open technologies for safeguarding software and agents; here are ten open-source projects already working across that emerging stack.

The emerging skill-security stack

The map shows each project's primary control point. Each project section below explains what it does, how it works, who it is for, and where its boundary ends.

1. NVIDIA SkillSpector: a pre-install scanner for skills

SkillSpector answers the first question most users have when they find a new skill: is this safe enough to install?

What it does. It scans a local file or directory, Git repository, URL, or ZIP and returns a risk score from 0 to 100, severity, recommendation, and findings. Its MCP wrapper also returns a safe_to_install signal. Reports can be written for humans or automation, including JSON, Markdown, and SARIF.

How it works. A source loader inventories the artifact, then deterministic checks inspect 68 patterns across 17 categories. Dependency findings use live OSV data when reachable and fall back to bundled data when it is not. An optional LLM pass adds semantic review before the findings are merged into an install recommendation. skillspector scan ./my-skill --no-llm needs no model key, although dependency coordinates may still be sent to OSV.

Best for. Individual developers, catalog maintainers, and CI pipelines that want a quick admission check with a clear decision at the end.

Boundary. SkillSpector does not execute the skill. Static and semantic inspection can identify suspicious intent and known patterns, but it cannot prove what a skill will do in a live environment.

2. Cisco AI Defense Skill Scanner: deeper multi-engine static analysis

Cisco AI Defense Skill Scanner is also a pre-install scanner, but it emphasizes a collection of specialized analysis engines and configurable security policy.

What it does. It scans a skill directory for malicious instructions, suspicious code, vulnerable execution paths, and behavioral risks, then produces console, JSON, Markdown, HTML, table, or SARIF output. A strict, balanced, or permissive scan policy tunes detection, while a separate severity threshold turns the result into a CI decision.

How it works. The default core combines YAML signatures and YARA rules, Python bytecode integrity checks, and shell-pipeline taint analysis. Optional local analyzers add Python AST and dataflow, trigger checks, and OSV lookups. Optional model and service analyzers add LLM review, meta-analysis, VirusTotal, or Cisco AI Defense. A strict high-severity CI gate is skill-scanner scan ./skill --policy strict --fail-on-severity high.

Best for. AppSec and platform teams that want explainable, CI-oriented checks and more language-aware analysis than a simple pattern scan.

Boundary. The core path is static: it inspects what could happen but does not run the skill. Some optional enrichments use external services and therefore do not belong in a strictly offline setup.

3. SkillWard: escalate suspicious skills into a sandbox

SkillWard treats skill analysis as an escalation pipeline. Cheap checks run first; uncertain cases can progress all the way to controlled execution.

What it does. It produces a behavioral verdict and evidence for a skill, including actions that are only visible when the skill is actually invoked.

How it works. Stage A uses YARA, regular expressions, and static inspection. Stage B asks a model to classify intent and escalates uncertain cases. Stage C launches the skill with an in-container agent inside Docker. A guard inspects tool calls and file content for evidence such as network access, sensitive writes, credential access, and exfiltration, while decoys make malicious behavior easier to expose. The semantic stage can use a locally hosted model.

Best for. Security researchers, reviewers, and higher-assurance catalogs investigating a suspicious or ambiguous skill after static scanning.

Boundary. Dynamic evidence is stronger, but it is also slower and depends on the test scenario reaching the malicious branch. “Nothing happened in this run” is not proof of safety.

4. Agent Audit: scan the whole agent repository

Agent Audit broadens the target beyond a packaged skill. It looks for security problems across the surrounding agent application.

What it does. It finds dangerous operations, untrusted-data flows into sensitive tools, exposed secrets, risky MCP configuration, skill and package risks, and excessive privileges. Its deepest AST and taint analysis is for Python, while the current scanner also covers TypeScript and JavaScript, Go, Solidity, SKILL.md, and configuration files. Results can fail a build and can be exported as SARIF.

How it works. Repository discovery feeds several analyzers: a Python AST scanner, intra-procedural source-to-sink taint tracker, tool-boundary analyzer, three-stage secret scanner, MCP configuration scanner, privilege scanner, and a shared rule engine. The current project documents 72 rules. A typical static gate is agent-audit scan . --fail-on high; an optional dynamic mode can connect to MCP servers for read-only inspection.

Best for. Developers and AppSec teams reviewing an agent as an application, especially Python-based projects with tools and MCP servers.

Boundary. The default scan is static and does not run the agent application. Optional MCP inspection expands discovery, but the project is not a runtime monitor or execution sandbox.

5. AgentShield: audit what is active on a developer machine

AgentShield shifts the target again, from a single downloaded artifact to the agent configuration already present in a project or user environment.

What it does. Its default mode discovers agent-related files and checks secrets, permissions, hooks, MCP configuration, and agent settings against 102 static rules in five categories. It can also execute hooks in a controlled temporary directory, run active injection tests, watch configuration changes, and install a PreToolUse policy hook.

How it works. The default invocation selects one root: a local .claude directory, otherwise the home .claude directory, otherwise the current directory. Discovery classifies findings by runtimeConfidence, separating active runtime and project-local material from examples, documentation, plugins, and hooks. npx ecc-agentshield scan runs the static checks; optional modes add model review, hook execution, continuous watch, or runtime allow and block decisions.

Best for. Developers, endpoint-security teams, and CI jobs that need to understand the effective agent attack surface of an environment.

Boundary. The default scan is a posture snapshot. Its optional watch and PreToolUse modes expand the boundary, but the hook sandbox is a host child process with a temporary working directory, not a hard operating-system or network boundary.

6. Microsoft Agent Package Manager: govern what gets installed

Microsoft Agent Package Manager, or APM, treats prompts, skills, MCP servers, and other agent assets as a dependency graph rather than files copied by hand.

What it does. It resolves declared agent dependencies, applies installation policy, deploys approved assets to a target harness, records exact versions and integrity data, and can export a CycloneDX or SPDX SBOM from the lockfile.

How it works. apm.yml declares the package graph and apm-policy.yml defines admission rules. During apm install, the resolver fetches dependencies into a cache, scans for hidden Unicode, requires explicit trust for transitive MCP server declarations, checks policy and integrity, deploys approved assets, and writes apm.lock.yaml. Later, apm audit verifies the declared, locked, and deployed state and replays a scratch install to detect drift. apm lock export produces the SBOM separately.

Best for. Platform teams that want repeatable, reviewable distribution of agent assets across developers and CI environments.

Boundary. APM governs what gets installed. It does not execute downloaded package code by default, but explicit lifecycle scripts and experimental canvas extensions are execution exceptions. The agent harness still governs what runs, so installation control must be paired with runtime policy or isolation.

7. NVIDIA Verified Agent Skills: signed provenance plus evaluation evidence

NVIDIA's verified agent-skills catalog is less a scanner than a trust and distribution pattern for published skills.

What it does. It gives consumers a curated catalog in which each accepted skill carries a skill specification, skill card, evaluation evidence, security review, and a detached signature that can be verified before installation.

How it works. An upstream release pipeline evaluates the skill, runs SkillSpector review, resolves findings, and signs the approved artifact. The catalog's hourly mirror checks required artifacts, signature state, and source or signature drift; it does not rerun the evaluation, and BENCHMARK.md is commonly published rather than required by the catalog gate. Consumers separately verify the directory against NVIDIA's root certificate, then install it.

Best for. Skill publishers, catalog operators, and organizations that need provenance and tamper detection, not just a point-in-time vulnerability scan.

Boundary. A valid signature proves origin and integrity, not harmlessness. The catalog’s review raises confidence, but runtime controls are still needed after a verified skill starts acting.

8. NVIDIA OpenShell: contain the running agent

NVIDIA OpenShell moves the control point from artifact inspection to execution. It assumes agent code and skills may be untrusted and gives them a constrained place to run.

What it does. It isolates an agent in a sandbox and enforces policy over filesystem access, processes, network connections, and model inference. Denied operations are stopped at the boundary rather than merely reported afterward.

How it works. The OpenShell Gateway stores desired state and asks a compute driver to provision the sandbox. Inside that sandbox, a Supervisor launches the restricted process. Filesystem and process controls are enforced locally, ordinary egress passes through a local policy proxy and OPA, and model traffic passes through an inference router. Docker and Podman are local options, the VM path is experimental, and Kubernetes is the cluster driver. On Kubernetes, that driver creates a Sandbox resource for the separate Agent Sandbox controller to reconcile into a Pod. The commands openshell sandbox create -- claude and openshell policy set demo --policy policy.yaml --wait are valid, but only network and inference policy are hot reloadable.

Best for. Platform and infrastructure teams running coding agents or other tool-using agents that need OS-level containment.

Boundary. OpenShell limits the blast radius of execution; it does not certify that the skill source is safe or decide whether the business intent of an allowed action is appropriate.

9. Kubernetes Agent Sandbox: provision isolated agent workspaces on Kubernetes

Kubernetes Agent Sandbox is the project that is easy to confuse with OpenShell. It is developed under Kubernetes SIG Apps rather than being a separate CNCF-hosted project. Kubernetes itself is a CNCF graduated project.

What it does. It provides a standardized Kubernetes API for isolated, stateful, singleton workloads, the shape needed by long-running coding agents and other autonomous runtimes. A sandbox can keep a stable identity and persistent storage, then be suspended, resumed, or replaced without treating it like a replicated web service.

How it works. The core Sandbox CRD and controller reconcile a Pod and its lifecycle, with a conditional headless Service and optional PVCs. Extension CRDs add reusable SandboxTemplate definitions, SandboxWarmPool capacity, and SandboxClaim allocation. A claim can request a warm pool, the pool references a template, and the controllers allocate or adopt a Sandbox. A template can add a managed NetworkPolicy and select a stronger RuntimeClass, including gVisor or Kata Containers. The Kubernetes introduction explains the broader model.

Google Cloud has also introduced Agent Substrate, an adjacent open-source project that pairs sandbox and snapshot capabilities with a specialized control plane for denser agent execution on Kubernetes. It maps many stateful actors onto fewer ready worker Pods, then suspends, resumes, and routes them on demand. The repository is still early and not production ready, and its threat model says security hardening is minimal, so treat it as a scalability substrate rather than an additional security control.

Best for. Platform teams building a multi-tenant Kubernetes substrate for agent runtimes, especially when they need stable workspaces, warm starts, persistent state, and a common API that several agent platforms can consume.

Boundary. Agent Sandbox provisions and manages the workload boundary; it is not an agent-aware security policy engine. The actual strength of isolation depends on the selected runtime, network policy, credentials, and cluster configuration. OpenShell can sit above it to add filesystem, process, egress, inference, and credential policy.

10. Microsoft Agent Governance Toolkit: policy-gate every action

Microsoft Agent Governance Toolkit, or AGT, focuses on the semantic action an agent is about to take: send an email, query data, call a tool, or delegate to another agent.

What it does. It wraps selected callables and configured framework intervention points, evaluates deterministic policy, can require human approval, and records an audit decision. A denied wrapped call never reaches the underlying tool.

How it works. A developer wraps a function with govern(my_tool, policy="policy.yaml"), which returns a GovernedCallable. Each wrapped invocation builds governance context, runs the YAML PolicyEngine with deny-overrides behavior, optionally requests approval, and writes to a Merkle-chained audit log. The broader toolkit separately offers Agent Control Specification support, OPA and Cedar backends, identity and trust, framework adapters, sandboxing, MCP security, compliance, and SRE modules.

Best for. Application, security, and compliance teams that need explainable authorization for every consequential agent action.

Boundary. The basic middleware and the agent share a process boundary, and the default audit log is in memory unless an external sink is configured. Microsoft's own guidance recommends container isolation for stronger protection; the policy gate should not be mistaken for the operating-system boundary provided by OpenShell.

Where the projects overlap and where they do not

The first five projects all find risk, but their inspection targets are different. SkillSpector and Cisco Skill Scanner focus most directly on a skill artifact. SkillWard adds controlled execution. Agent Audit follows dangerous flows through the wider agent codebase. AgentShield inventories the configuration that is active in an environment.

APM and NVIDIA Verified Agent Skills operate on the software-supply-chain problem. APM makes installation deterministic and policy-controlled; NVIDIA’s catalog adds publication requirements, evaluation evidence, and signed provenance. Neither replaces source analysis or runtime containment.

The final three control points intervene while an agent is operating, but at different boundaries. AGT authorizes the agent’s intended action. OpenShell constrains what the process can actually do at the operating-system and network boundary. Kubernetes Agent Sandbox provisions and manages the underlying stateful workload when that process runs on Kubernetes.

That produces a useful three-line mental model:

Intent: Is this requested action allowed for this agent? AGT answers that question.
Capability: Can this process reach the resource at all? OpenShell enforces that boundary.
Substrate: Where does the isolated stateful workload run? Agent Sandbox provides that lifecycle.

Comparison at a glance

The table condenses the primary target, mechanism, effect, and GitHub stars for all ten projects.

For an organization adopting third-party skills, I would start with four layers:

Scan before trust. Run SkillSpector or Cisco Skill Scanner on every incoming skill; escalate ambiguous, high-risk artifacts to SkillWard.
Control distribution. Use APM-style manifests, policies, lockfiles, and SBOMs; verify signatures when a curated catalog provides them.
Authorize actions. Put deterministic policy and audit around consequential tool calls with AGT or an equivalent middleware gate.
Contain execution. Run the agent inside an OpenShell-style sandbox, and use Agent Sandbox as the Kubernetes substrate when the workload needs cluster-native lifecycle and isolation.

No single project spans the entire lifecycle yet. That is not a weakness in the ecosystem; it is the shape of defense in depth. The important step is to stop treating a skill as “just a Markdown file.” It is a supply-chain artifact that can steer a privileged, autonomous runtime.

Originally published on The Generative Programmer.

Emerging Markdown Formats That Shape Coding Agent Behavior

Bilgin Ibryam — Mon, 03 Aug 2026 10:25:16 +0000

A practical map of the rules, procedures, plans, domain knowledge, and memory that guide agent-assisted software changes.

Agent onboarding starts with a README for agents

The ecosystem also has vendor-native alternatives that serve the same purpose:

AGENTS.md provides cross-tool project guidance for Codex, Cursor, Cline, Copilot, OpenCode, and other compatible agents.
CLAUDE.md and .claude/rules/ provide standing and path-scoped guidance for Claude Code.
GEMINI.md provides hierarchical context for Gemini CLI.
.github/copilot-instructions.md and path-specific instruction files guide GitHub Copilot.
.clinerules/ carries persistent and conditional rules for Cline.

Once the agent understands the repository, the next question is how to perform a particular kind of work.

Skills package repeatable procedures

A skill is a playbook tailored to a particular kind of work.

The boundary is concrete:

AGENTS.md and native rule files describe standing context and constraints.
SKILL.md describes a repeatable procedure invoked for a specific task.

A procedure explains how to work. It still needs a reviewed description of what should be built.

Planning files make software design reviewable

GitHub Spec Kit is the clearest mainstream example. Its documentation reports more than 121,000 GitHub stars and 35 coding-agent integrations. Its default workflow uses a chain of files:

spec.md records the requirements and desired outcome.
plan.md explains the technical design and implementation approach.
tasks.md breaks the plan into executable units.
The agent implements and validates the change.

A project constitution can carry principles that should apply across many changes.

These files describe a generic software development lifecycle. Applications also need instructions for their particular domain.

Domain files give agents specialized context

ARCHITECTURE.md can describe system boundaries, components, dependencies, NFRs, and long-lived architectural decisions. It is an established human documentation convention that becomes operational for agents when project instructions tell them when to read it.
Google Labs' DESIGN.md describes visual identity, design tokens, components, and design rationale. The project currently labels the format alpha.
WorkOS' AUTH.md is an experimental, service-hosted recipe that teaches agents how to register and authenticate on behalf of users.
Kilo's REVIEW.md lets a repository define review priorities, severity, skipped files, verification expectations, and how a review agent should divide work.

Some of that knowledge is designed in advance. Other knowledge emerges only while the software is being built and operated.

Memory makes learned context durable

Agent memory keeps learned context associated with a repository and available across sessions. Claude Code makes the authorship distinction unusually clear:

Humans write CLAUDE.md to provide shared instructions, conventions, and project context.
Claude writes auto memory into a MEMORY.md index and optional topic files such as debugging.md and api-conventions.md.

The agent records a tentative observation in local memory.
A human reviews it when the observation recurs or affects shared work.
Durable knowledge moves beside the source code into a project rule, skill, ADR, or domain document.

Putting the Markdown files together

Project knowledge moves from scattered systems into an agent-ready repository, where versioned intent can guide implementation.

The practical distinction is the role each file plays, who writes it, and how broadly it is supported.

The formats range from cross-tool standards and established vendor conventions to emerging domain-specific experiments.

The takeaway

The repository is becoming the meeting point between source code and metacode: the Markdown that carries requirements, NFRs, architecture, design, rules, procedures, plans, and durable learning.

When agents can read both, they act as intent compilers. Source code becomes a byproduct: the executable result of human intent and judgment made legible to the agent.

The engineering task is to keep that metacode scoped, reviewed, and current. It does not replace testing or review. It gives the next code change a better source.

Originally published at The Generative Programmer.

Stop Babysitting Your Coding Agent. Give It Backpressure.

Bilgin Ibryam — Sun, 07 Jun 2026 07:59:17 +0000

I keep seeing the same failure mode with coding agents:

The agent writes a diff.
The human notices a broken test.
The human explains the failure.
The agent retries.

That is not review. That is babysitting.

The fix is to move cheap, machine-readable feedback into the agent loop before the work reaches a human reviewer.

The problem

A coding agent is a fast producer. The human reviewer is a slow consumer.

That mismatch creates pressure. If the only meaningful feedback comes from the human, every trivial failure becomes review work. The human becomes the compiler, the test runner, the linter, the UI checker, and the reviewer.

The agent can generate code quickly, but generation without feedback is open loop. It produces a diff without knowing whether that diff is correct, useful, safe, or ready.

In Don’t waste your back pressure, Moss Banay describes this as wasted human backpressure: spending attention on mistakes the agent should have been able to detect itself, such as missing imports, broken builds, and visual errors.

Marc Brooker makes a related point in What’s Easy Now? What’s Hard Now?: agents are feedback loops around a useful but flawed model. The key shift is moving feedback from the human loop into the agent loop: build, test, inspect, repair, and iterate.

So the question is not only:

How good is the model?

The better question is:

What feedback can the agent use without asking me?

The pattern

Backpressure is feedback that reaches the agent before the agent reaches the human.

The useful loop looks like this:

generate → verify → repair → repeat

Backpressure can come from a type checker, a test, a linter, a build, a browser check, logs, traces, structural rules, or evals. The source matters less than the loop: the agent sees the failure, repairs the work, and tries again.

Intent: Move cheap correctness checks out of human review and into the agent loop.

Context: A coding agent can edit files, run commands, read failures, and retry.

Problem: Agents produce code faster than humans can validate it.

Solution: Expose fast, machine-readable feedback sensors to the agent: types, tests, linters, builds, browser checks, logs, traces, and structural rules.

Result: The agent fixes mechanical failures before review. The human focuses on intent, design, and trade-offs.

What counts as backpressure

Backpressure is any signal that can push back on bad agent output before a human spends attention on it.

Sensor	What it catches
Type checker	Invalid shapes, missing fields, bad contracts
Linter	Unsafe patterns, formatting noise, local conventions
Focused tests	Behavior regressions
Build	Integration failures
Browser check	Broken UI, missing elements, visual drift
Logs and traces	Runtime behavior
Structural rules	Architecture boundary violations
Evals	Meaning-level regressions

Thoughtworks calls these feedback sensors for coding agents: deterministic quality gates wired into agent workflows so failures trigger self-correction.

That distinction matters.

A CI failure after the agent is done is a gate.

A failure the agent sees while working is backpressure.

A green build is not enough

A green build is useful, but narrow.

It tells you the code compiled, the formatter ran, and the existing tests passed. It does not prove the product rule survived. It does not prove the generated test asserts meaningful behavior. It does not prove the refactor respected the architecture.

So the goal is not just to run CI earlier.

The goal is to make feedback fast, specific, and actionable enough for the agent to repair the work.

Bad feedback:

The code is wrong.

Better feedback:

auth/session.test.ts failed.

Expected expired sessions to redirect to /login.
Received 200 from /dashboard.

Fix the session expiration path.
Do not change the test.

The second version gives the agent something to do.

Effective feedback beats more feedback

More output is not the same as better backpressure.

A recent arXiv preprint, Scaling Laws for Agent Harnesses via Effective Feedback Compute, makes this point in research terms. The authors argue that agent harnesses scale less with raw tokens, tool calls, wall time, or cost, and more with how efficiently they convert raw budget into useful feedback.

They call this Effective Feedback Compute: feedback that is informative, valid, non-redundant, and retained for later decisions.

That maps directly to coding agents.

A test failure is useful only if the agent can understand it, trust it, avoid repeating it, and use it to change the next attempt.

Useful backpressure has four properties:

Property	Meaning
Informative	It tells the agent what failed and where
Valid	It comes from a trusted checker, test, tool, or observation
Non-redundant	It adds signal instead of repeating noise
Retained	It changes the agent’s next action

That is why a thousand lines of logs can be worse than a short, structured failure.

The goal is not more agent activity. The goal is more effective feedback.

How to implement it

Start with the boring checks.

Put the exact commands in AGENTS.md, CLAUDE.md, or the equivalent project instruction file:

After changing TypeScript:
- run npm run typecheck
- run npm run lint
- run npm test -- --changed
- fix failures before opening a PR

Then make the output efficient.

Do not dump hundreds of lines of passing test output into the agent context. Passing checks should be tiny. Failing checks should show the useful detail.

HumanLayer describes this as context-efficient backpressure: replace passing test, build, and lint output with a small success signal, but expose full output when a command fails.

Example:

✓ typecheck
✓ lint
✗ auth tests

FAIL auth/session.test.ts
Expected redirect to /login for expired session.
Received 200 from /dashboard.

That is enough signal for the agent to continue.

The rule:

Success should be compressed. Failure should be actionable.

The trade-off

Backpressure is not free.

Fast sensors improve flow. Slow sensors choke it.

Do not put every possible check inside the inner loop. Layer them.

In session:
typecheck, lint, focused tests, build, screenshots

Before PR:
full tests, integration tests, structural rules, security checks

Scheduled or explicit:
mutation testing, fuzzing, semantic evals, architecture drift review

Birgitta Böckeler’s Maintainability sensors for coding agents is useful here because it separates sensors that run during the coding session from sensors that belong in the pipeline, on a schedule, or in production.

That is the right mental model: not every check belongs in the same loop.

The inner loop should be fast enough that the agent can run it repeatedly. Slower checks belong outside the session unless the task is risky enough to justify the cost.

The rule

When you correct the same agent mistake twice, turn it into backpressure.

A test.
A type.
A linter rule.
A build check.
A browser assertion.
A better error message.
A small script the agent can run before it asks for review.

The future skill is not asking agents to try harder.

It is designing feedback loops that make bad output harder to accept.

I write about similar AI-assisted coding patterns in my newsletter.

Dapr as the Ultimate Microservices Patterns Framework

Bilgin Ibryam — Wed, 27 Sep 2023 09:06:20 +0000

In the world of software development, microservices architecture stands out as the industry benchmark. This architectural style segments applications into distinct services, each deployable independently and organized around specific business capabilities. Such a design ensures flexibility, scalability, and resilience, with each service typically overseen by a specialized team. However, while this approach offers numerous benefits, it also introduces complexities. To navigate these challenges, developers turn to patterns, drawing parallels to time-tested design patterns from traditional software development, providing solutions for building robust distributed systems. For those seeking a comprehensive guide on these patterns, Chris Richardson’s microservices.io stands as a trusted repository, rich with insights and best practices from industry experts.

A pattern language for microservices by Chris Richardson

However, understanding patterns is just one part of the equation. Implementing them in real-world scenarios requires tools and frameworks. Positioned as a premier microservices chassis, Dapr is crafted for creating distributed applications that are secure, resilient, scalable, and observable. It doesn't merely align with the microservices patterns; it amplifies their potential, refining and simplifying their real-world implementation.

In the subsequent sections, I'll go through the patterns outlined in microservices.io, shedding light on how Dapr helps implement each. While numerous frameworks aim to address the cross-cutting concerns inherent in microservices, Dapr distinguishes itself. Its polyglot nature, sidecar operational mode, and non-restrictive stance on application architecture make it a unique and invaluable asset in the microservices toolkit.

Microservice Chassis

When embarking on the development of an application, developers often find themselves investing significant time in addressing cross-cutting concerns such as security, externalized configuration, logging, health checks, metrics, and distributed tracing. While these elements might seem straightforward, defining a curated set of dependencies that will be used in tens or hundreds of services can be a complex endeavor. The challenge amplifies when adopting a microservices architecture, given the multitude of services and the frequent creation of new ones. The Microservice Chassis pattern offers a solution by proposing the creation of a framework that serves as the foundation for microservices development. This chassis provides reusable build logic and mechanisms to handle these cross-cutting concerns, streamlining the development process.

As a polyglot framework, Dapr seamlessly handles cross-cutting concerns, allowing developers to focus on core functionalities without getting entangled in complexities. It provides built-in mechanisms for security, configuration management, logging, and more. With features like the Access Control capabilities for security, Configuration API for externalized settings, and health API endpoints for monitoring, Dapr ensures that these foundational concerns are seamlessly integrated, allowing developers to focus on core business logic without getting mired in the intricacies of these concerns. Unlike other frameworks, Dapr doesn't impose constraints on the application, granting developers the freedom to choose their preferred language, runtime, and programming style. In essence, Dapr transforms the theoretical benefits of the Microservice Chassis pattern into tangible results for real-world applications.

Sidecar

In microservices, there's often a need to augment services with additional capabilities without modifying the core service logic. The Sidecar pattern addresses this requirement. It involves deploying components of an application into separate processes or containers to provide a modular and scalable architecture. The main service runs in one container, and the sidecar service, which extends or enhances the main service, runs in a separate container but in the same network namespace. This ensures that the main service and the sidecar can communicate as if they are in the same process while being isolated from each other. The primary advantage of this pattern is the ability to separate concerns, modularize your application, and ensure that each component is focused on a specific responsibility.

Dapr is among the most popular implementations of the Sidecar pattern. When integrated into a microservices environment, Dapr runs as a sidecar alongside your service, providing a plethora of additional capabilities without requiring any changes to the main service. This includes features like state management, service-to-service invocation, pub/sub messaging, and more. By leveraging Dapr's sidecar architecture, developers can augment their services with powerful features, ensuring a robust, scalable, and feature-rich microservices ecosystem.

Service Mesh

As the number of services grows, managing inter-service communication, security, and observability becomes increasingly complex. A Service Mesh is a dedicated infrastructure layer built to handle service-to-service communication in a transparent and technology-agnostic manner. It provides features like load balancing, service discovery, observability, and security without requiring changes to the application code. By offloading these concerns to the Service Mesh, developers can focus on building business logic, while the mesh ensures that services can securely and efficiently communicate with each other.

How Dapr and service meshes compare

Dapr operates as a lightweight service mesh, providing a network layer that facilitates service discovery and ensures secure service-to-service interactions. While there are overlapping capabilities between Dapr and traditional service meshes, Dapr distinguishes itself by being developer-centric, focusing on building blocks that simplify microservices development. Unlike service meshes which are primarily infrastructure-centric and deal with network concepts like IP and DNS addresses, Dapr offers service discovery and invocation via names, a more developer-friendly approach. Beyond the common features like mTLS encryption, metric collection, and distributed tracing, Dapr introduces application-level building blocks for state management, pub/sub messaging, actors, and more. This ensures that developers gain a comprehensive toolset, not just for networking, but for holistic microservices development.

Saga

In the world of microservices, ensuring data consistency across services can be a challenge, especially when each service has its own database. The Saga pattern provides a solution to this challenge. Instead of relying on traditional distributed transactions, the Saga pattern breaks the transaction into a series of local transactions, each executed within its own service and database. These local transactions are coordinated in a specific sequence to ensure overall data consistency. If one local transaction fails, compensating transactions are executed to revert the changes made by the previous transactions. This approach offers a way to maintain data consistency without the need for distributed transactions, which are often not feasible in microservices architectures.

Dapr workflow overview

Dapr offers a concrete solution to implement the Saga pattern through its Workflow API. This API allows developers to sequence local transactions, or implement other stateful workflow patterns ensuring that data remains consistent across services. Dapr Workflow API serves as a foundational tool in this regard, streamlining the process and ensuring reliability.

Transactional Outbox

In microservices architectures, a common challenge arises when a service command needs to update aggregates in the database and simultaneously send messages or events to a message broker. The goal is to ensure atomicity - if the database transaction commits, the messages must be sent; if the database rolls back, the messages must not be sent. Traditional distributed transactions (2PC) are often not feasible or desirable due to various constraints. The Transactional Outbox pattern addresses this issue. It suggests that the service stores the message in the database as part of the transaction that updates the business entities. A separate process then retrieves and sends these messages to the message broker. This ensures that messages are sent only if the database transaction commits, preserving data consistency and order of operations.

Dapr provides a robust solution to this challenge with its Outbox feature in StateStore API. This feature allows for atomic updates to the database while also sending messages to the designated broker. By utilizing the StateStore API, developers can seamlessly integrate the Transactional Outbox pattern into their microservices, ensuring data consistency and reliable message delivery across large number of databases and message brokers.

Messaging

In the realm of microservices, reliable asynchronous communication between services is paramount. Instead of services communicating directly with synchronous calls, they exchange messages via message channels. This asynchronous mode of communication decouples services, allowing them to operate independently. It ensures that even if one service is slow or unavailable, others can continue their operations without being directly affected. This approach enhances the system's resilience, scalability, and flexibility.

Dapr's PubSub API is tailored to harness the power of asynchronous messaging for inter-service communication. By leveraging this API, developers can easily implement messaging patterns in their microservices architecture. The PubSub API ensures reliable message delivery, supports multiple messaging brokers, and abstracts the complexities of direct broker interactions.

Request/Reply Interaction

Remote Procedure Invocation (RPI) is a communication style that enables services in a microservices architecture to communicate with each other by invoking methods in a remote service. The primary advantage of RPI is its straightforwardness, allowing for direct, point-to-point communication between services. However, it's essential to manage the associated challenges, such as service discovery, reliability, encryption, to ensure the system remains resilient and secure.

Dapr addresses the challenges with its Service Invocation API. This API provides an RPI-based protocol tailored for inter-service communication in a microservices setup. By abstracting the underlying complexities, Dapr ensures that services can communicate synchronously without getting entangled in the intricacies of direct service-to-service calls. Moreover, Dapr's Service Invocation API offers built-in features like retries, error handling, and traffic control, ensuring that communications are both reliable and secure.

Circuit Breaker

In a microservices architecture, services often collaborate to handle requests. However, when one service synchronously invokes another, there's a risk that the called service might be unavailable or might exhibit high latency, rendering it essentially unusable. Such scenarios can lead to resource exhaustion in the calling service, making it unable to handle other requests. This can further cascade the failure to other services throughout the application. The Circuit Breaker pattern addresses this challenge. It functions similarly to an electrical circuit breaker. When consecutive failures cross a certain threshold, the circuit breaker "trips." For a set timeout period, all attempts to invoke the problematic service fail immediately. After this period, the circuit breaker allows a few test requests. If these succeed, normal operation resumes; if not, the timeout period restarts.

Dapr offers a concrete solution to this challenge with its Resiliency policy. This policy ensures that when the failure rate of a call exceeds a certain threshold, the call fails immediately, preventing resource exhaustion and potential cascading failures. By leveraging Dapr's resiliency policy, developers can implement the Circuit Breaker pattern efficiently, ensuring that their microservices architecture remains robust and resilient against unexpected service failures or latencies.

Access Token

In the intricate landscape of microservices, ensuring secure communication and access between services is paramount. The Access Token pattern involves issuing tokens to clients, granting them limited access to a service. These tokens encapsulate the information required to determine whether a client is authorized to perform a given operation. The primary advantage of using access tokens is that they provide a way to ensure that only authenticated and authorized clients can access services or specific operations within those services.

Dapr secure communications architecture

Dapr provides a robust mechanism to implement this pattern through its Access Control capabilities based on SPIFFE Ids. With Dapr's Access Control, developers can define and enforce policies that restrict what operations calling applications can perform on the called app. This ensures a fine-grained control over service interactions, making the system more secure and resilient against potential threats.

Service Instance per Container

The Service Instance per Container pattern deployment strategy, places each service instance in its own container. Containers, being lightweight and isolated, provide an environment where the service can run with its dependencies, ensuring consistency across different stages of deployment. This approach offers several benefits: it ensures isolation, making each service instance independent of others; it provides scalability, as new instances can be spun up quickly; and it enhances portability, as each service containts its built-time dependencies.

Dapr aligns perfectly with this deployment model as it is designed to operate best in containerized environments. When a service is deployed with Dapr, a Dapr sidecar container runs alongside the service container, enhancing its capabilities without intruding into the service's operations. Dapr's deployment model, ensures that each service instance, along with its Dapr sidecar, remains isolated in its container, benefiting from the inherent advantages of the Service Instance per Container pattern such as scalable, and resilient microservices deployments.

Service Instance per VM

In certain deployment scenarios, especially when dealing with large-scale applications or when containers might not be the optimal choice, deploying each service instance on its own Virtual Machine (VM) becomes a viable strategy. The Service Instance per VM pattern emphasizes this approach. By allocating a dedicated VM for each service instance, you ensure that the service has a dedicated set of resources, leading to predictable performance. This isolation also means that failures or resource contention in one service won't directly impact others. Moreover, VMs provide a higher degree of isolation compared to containers, which can be crucial for certain security or compliance requirements.

Dapr is versatile and can be seamlessly deployed in a VM-based environment. Whether you're deploying Dapr on its own dedicated VM or using Dapr Ambient to share its capabilities among multiple Pods, Dapr ensures that microservices can communicate and operate efficiently. This flexibility means that developers aren't confined to containerized environments and can leverage Dapr's capabilities in VM-based deployments, ensuring that the benefits of Dapr, such as state management, service invocation, and pub/sub messaging, are available regardless of the deployment strategy. Dapr's deployment documentation provides insights into how it can be integrated into various environments, including VMs, or the real edge.

Service Discovery

In microservices architectures, the dynamic nature of service instances and their locations, especially in containerized environments, presents a challenge: how does a client of a service discover the location of a service instance? Two prevalent patterns address this challenge: Client-side Discovery and Server-side Discovery. The former involves clients querying a Service Registry to discover the current locations of service instances, ensuring they always communicate with available and healthy instances. On the other hand, the Server-side Discovery pattern simplifies client code by routing requests via a knowledgeable router, often a load balancer, which interacts with the service registry.

Dapr's sidecar architecture adeptly addresses both these patterns. While the sidecar operates alongside a service, akin to a client, it isn't embedded within the application. This unique positioning allows it to query a service registry, discovering other service instances' locations, and also act as a router for inter-service calls. By offloading the intricacies of service discovery to the Dapr Service Invocation API, developers can ensure reliable service-to-service communication, even in environments where service locations change dynamically.

Service Registry

In dynamic microservices environments, the locations and number of service instances can frequently change. This poses a challenge: how can clients or routers be aware of the current available instances of a service? The Service Registry pattern offers a solution. It proposes a centralized registry where service instances register themselves upon startup and deregister upon shutdown. This registry acts as a database of services, their instances, and their locations. Clients or routers can then query this registry to discover the current locations of service instances.

Dapr offers a seamless integration with the Service Registry concept, providing a unified interface to various service registry implementations. Dapr's pluggable name resolution components used in Service Invocation API cater to diverse hosting platforms, from Kubernetes, which utilizes its DNS service, to self-hosted machines using mDNS or even HashiCorp's Consul in varied environments.

Self Registration

The Self Registration pattern ensures that services can discover each other in inter-service communication. When a service instance starts up, instead of relying on an external agent or system to register it with a service registry, the service instance itself takes the responsibility of registering. This ensures that the service registry always has the most up-to-date information about available service instances. By automating the registration process, the Self Registration pattern reduces manual intervention, potential errors, and ensures that the service registry is always current.

When a service with a Dapr sidecar is deployed, the Dapr sidecar takes the initiative to register itself with the service registry. This automated process ensures that the service is immediately discoverable by other services in the ecosystem. This not only simplifies the deployment process but also enhances the reliability and efficiency of service-to-service communication in architectures built with Dapr.

3rd Party Registration

In some microservices architectures, not all services or endpoints are created or managed by the same team or entity. There might be third-party services or endpoints that need to be integrated into the system. These third-party services might not follow the same registration patterns as internal services. With the 3rd Party Registration pattern, instead of the service registering itself (as in Self Registration), an external agent or system is responsible for registering the service with the service registry. This ensures that third-party services, which might not have the capability or permission to register themselves, are still discoverable and can be integrated seamlessly into the system.

Dapr offers flexibility in this regard too. Even non-Dapr 3rd party endpoints can be registered within the Dapr runtime, ensuring they benefit from service discovery, resiliency, and observability features that Dapr provides. This means that developers can integrate third-party services into their Dapr-enabled microservices architecture without those services being Dapr-aware. This allows a more cohesive and resilient microservices ecosystem, irrespective of the origin of the services.

Externalized Configuration

In the world of microservices, applications often interact with various infrastructure and third-party services. Examples include service registries, message brokers, databases, payment processors, and more. A significant challenge arises when trying to ensure that a service can run across multiple environments (like dev, test, staging, production) without any modifications. The Externalized Configuration pattern recommends externalizing all application configurations. This ensures that the service remains environment-agnostic and can adapt to different setups without any code changes.

Dapr secrets stores overview

Dapr provides a robust solution to this challenge with its Secrets and Configuration APIs. These APIs allow developers to externalize configurations, including sensitive information like database credentials. Instead of hardcoding configurations or placing them in easily accessible files, Dapr ensures that they are securely stored and can be fetched dynamically when needed.

Health Check API

In a microservices architecture, ensuring the health, availability, and self-healing of service instances is paramount. The Health Check API pattern proposes that each service should expose an API endpoint (e.g., HTTP /health) that indicates the health status of the service. This endpoint performs various checks, such as the status of connections to infrastructure services, the health of the host (e.g., disk space), and any application-specific logic. By periodically querying this endpoint, monitoring systems, service registries, or load balancers can determine the health of a service instance. This ensures that alerts are generated for unhealthy instances, and requests are only routed to healthy service instances, enhancing the reliability and efficiency of the system.

Dapr elevates the health check pattern by conducting periodic health checks on your service, ensuring its optimal functioning. Once app health checks are enabled, the Dapr sidecar routinely polls the application. If a health issue is detected, Dapr takes proactive measures: it unsubscribes from all pub/sub subscriptions, halts all input bindings, and short-circuits service-invocation requests, ensuring they aren't forwarded to the application. This comprehensive approach ensures that any potential issues are swiftly identified and mitigated, fostering a robust and resilient system.

Distributed Tracing

In complex microservices architectures, understanding the flow of requests across multiple services can be challenging. The Distributed Tracing pattern involves instrumenting services with code that assigns each external request a unique identifier. This identifier is then passed to all services involved in handling the request. By doing so, it becomes possible to trace the journey of a request across various services, recording information such as start time, end time, and other relevant metrics providing invaluable insights into the behavior of the system.

Dapr distributed tracing overview

Dapr automatically takes care of creating trace headers and ensures they are captured and forwarded appropriately. With Dapr’s observability capabilities, developers don't need to manually instrument their code for tracing; Dapr handles it seamlessly. Moreover, Dapr integrates with popular tracing systems, ensuring that the traces can be visualized and analyzed in a centralized manner.

Application Metrics

Application Metrics are a set of quantitative data points that provide insights into the performance, behavior, and health of an application or service. By collecting and analyzing these metrics, developers and operations teams can identify bottlenecks, detect anomalies, and optimize the performance of their services, enabling proactive monitoring and ensuring optimal system health.

Dapr automatically gathers a wide range of networking metrics, capturing data related to request rates, error rates, and latency, among others. Dapr ensures that these metrics are delivered to a centralized metrics service, allowing for comprehensive monitoring and analysis. This means that developers and operations teams can have a unified view of the system's performance, irrespective of the number of services or their complexity.

Serverless Deployment

In the evolving landscape of software development, the need for scalable, and cost-effective deployment solutions is growing. Serverless Deployment refers to a deployment infrastructure that abstracts away any concept of servers, be it physical, virtual hosts, or containers. The primary advantage is that developers can focus solely on their code, without concerning themselves with the underlying infrastructure. The serverless platform automatically scales services based on the load, ensuring optimal resource utilization.

Dapr is gearing up for the serverless era too. Soon, developers will be able to access Dapr's rich capabilities as serverless APIs. This means that the vast array of features Dapr offers, from state management to messaging, will be available in a serverless context, ensuring developers get the best of both worlds. By integrating Dapr into serverless environments, developers can ensure more robust, scalable, and feature-rich applications without the complexities of managing the Dapr infrastructure. For the updates on this pattern, follow @diagridio on Twitter and you can be among the first to try it out.

Summary

Patterns play a pivotal role in software development, serving as a shared language to communicate common challenges and best practices. They encapsulate proven solutions to recurring problems, ensuring that developers don't have to reinvent the wheel with each new project. However, while patterns provide a conceptual blueprint, they remain abstract ideas. Historically, the challenges of building reliable applications and the implementations of these patterns were addressed using application servers, service buses, and microservices frameworks like Spring Cloud, among other language-specific solutions. But to truly bring these patterns to life in the modern era, we need cloud-native frameworks and chassis like Dapr. Dapr tackles these common engineering challenges by presenting patterns as polyglot APIs, aligning with the cloud-native philosophy and offering benefits to the whole organization. This ensures that developers can leverage best practices across multiple languages and platforms, streamlining the development process and enhancing application resilience and scalability.

Top 20 Must-Read Software Trends Reports for 2023

Bilgin Ibryam — Thu, 20 Jul 2023 09:14:09 +0000

In the rapidly evolving software industry, keeping up with new trends, tools, and best practices can be time-consuming. With so much information available, where do you start, and what sources can you trust? I've curated a list of reports that I follow to stay informed and ahead of the curve. These provide insights into everything from programming languages to DevOps, cloud strategy, and security. If you're interested in the latest trends and fascinating posts I come across, follow me or check out my latest writing on industry trends over at the Diagrid blog. I share anything I find insightful and worth reading in the world of cloud and distributed systems.

Here are the top 20 reports for 2023 (in no particular order) I came across so far:

While these reports offer valuable insights, it's important to keep in mind that they can be opinionated. The key to effectively leveraging these resources lies in cross-verifying trends from multiple sources and using them only as a guide for direction rather than absolute truths.

Are there any reports that should be on this list? Tag me on Twitter and I'll include them, subject to my checks I'm always keen to explore new sources! Found this list helpful? Go ahead, share it.

Call to action: Are you a Dapr user? Your experience is valuable! Contribute your insights and shape the State of Dapr Report.

DEV Community: Bilgin Ibryam

Latency Patterns for Faster AI Applications

Locality Patterns

1. Colocation Pattern

2. Replication Pattern

3. Partitioning Pattern

4. Caching Pattern

Choosing What to Bring Closer

Work Reduction Patterns

5. Algorithmic Work Reduction Pattern

6. Selective Data Processing Pattern

7. Setup Reuse Pattern

8. Request Coalescing Pattern

9. Runtime Tuning Pattern

Choosing What Work to Remove

Concurrent Execution Patterns

10. Synchronization Avoidance Pattern

11. Independent Concurrency Pattern

12. Progressive Response Pattern

13. Concurrency Budget Pattern

14. Hedged Requests Pattern

Choosing What Can Run Together

Anticipation Patterns

15. Predictive Prefetching Pattern

16. Optimistic Update Pattern

17. Speculative Execution Pattern

18. Precomputation Pattern

19. Prewarming Pattern

Choosing What to Do Earlier

Find the Bottleneck, Choose the Pattern

Further reading

Emerging Markdown Formats That Shape Coding Agent Behavior

Agent onboarding starts with a README for agents

Skills package repeatable procedures

Planning files make software design reviewable

Domain files give agents specialized context

Memory makes learned context durable

Putting the Markdown files together

The takeaway

10 Open-Source Projects for Securing AI Agent Skills

The emerging skill-security stack

1. NVIDIA SkillSpector: a pre-install scanner for skills

2. Cisco AI Defense Skill Scanner: deeper multi-engine static analysis

3. SkillWard: escalate suspicious skills into a sandbox

4. Agent Audit: scan the whole agent repository

5. AgentShield: audit what is active on a developer machine

6. Microsoft Agent Package Manager: govern what gets installed

7. NVIDIA Verified Agent Skills: signed provenance plus evaluation evidence

8. NVIDIA OpenShell: contain the running agent

9. Kubernetes Agent Sandbox: provision isolated agent workspaces on Kubernetes

10. Microsoft Agent Governance Toolkit: policy-gate every action

Where the projects overlap and where they do not

Comparison at a glance

Emerging Markdown Formats That Shape Coding Agent Behavior

Agent onboarding starts with a README for agents

Skills package repeatable procedures

Planning files make software design reviewable

Domain files give agents specialized context

Memory makes learned context durable

Putting the Markdown files together

The takeaway

Stop Babysitting Your Coding Agent. Give It Backpressure.

The problem

The pattern

What counts as backpressure

A green build is not enough

Effective feedback beats more feedback

How to implement it

The trade-off

The rule

Dapr as the Ultimate Microservices Patterns Framework

Top 20 Must-Read Software Trends Reports for 2023