DEV Community: Basavaraj SH

How to Give AI Agents a Safe Sandbox Using Docker

Basavaraj SH — Mon, 10 Aug 2026 15:13:26 +0000

AI agents that can execute code, browse files, or run shell commands are powerful - and risky. Docker sandboxes let you hand an agent real execution capability without letting it touch anything it shouldn't.

Disposable Containers as Agent Workspaces

When an AI agent needs to run code or interact with a filesystem, the naive approach is to let it run directly on the host. That works until it doesn't - a runaway loop, an unintended file deletion, or a dependency conflict can wreck your environment.

The better pattern is to spin up a fresh Docker container per agent task, give it only what it needs, and throw it away when the task is done. Each container provides an isolated workspace: it has its own filesystem, its own process namespace, and no access to host resources unless you explicitly grant them. The agent runs inside it, does its work, and the container is removed. If something goes wrong, the blast radius is one disposable container, not your machine or your production environment.

This pattern is especially useful for agents that use tool-calling (where the LLM triggers actual function execution) or code interpreter steps - any workflow where the model's output becomes a live command.

Real Example: Spin Up and Tear Down an Agent Sandbox

Here's a minimal pattern using the Docker SDK for Python to create a sandboxed execution environment for an agent task:

import docker

client = docker.from_env()

def run_in_sandbox(code: str) -> str:
 container = client.containers.run(
 image="python:3.12-slim",
 command=["python", "-c", code],
 mem_limit="128m",
 network_disabled=True,
 remove=True, # auto-delete after exit
 stdout=True,
 stderr=True,
 )
 return container.decode("utf-8")

output = run_in_sandbox("print(sum(range(100)))")
print(output) # 4950

A few details worth noting: network_disabled=True cuts off outbound calls from inside the container - critical if the agent-generated code might try to exfiltrate data or call external APIs unexpectedly. mem_limit prevents a runaway process from consuming host memory. remove=True means the container is deleted the moment it exits, so you're not accumulating stale containers.

For more complex agent setups (multi-step tasks, file I/O between steps), you can mount a temporary volume for the duration of the task and unmount it afterward, keeping the isolation intact while still allowing artifact passing between agent turns.

Key Takeaways

Giving AI agents a throwaway Docker container per task limits damage from unexpected or malicious code execution.
Disabling networking and capping memory are the two most important constraints to set from the start - not as an afterthought.
This pattern works with most agent frameworks (LangChain, LlamaIndex, custom tool-calling loops) since it wraps execution at the infrastructure layer, not inside the framework itself.

Are you currently running agent-generated code directly on the host, or do you already have an isolation layer in place - and if so, what does it look like?

Sources referenced: HackerNews - Docker Sandboxes discussion (403 points, 258 comments)

How OpenAI's Tiered Model Strategy Actually Works Under the Hood

Basavaraj SH — Fri, 07 Aug 2026 00:32:53 +0000

OpenAI is rolling out GPT-4o "Sol" improvements while expanding "Luna" access to free users - and the split reveals something worth understanding about how AI providers are structuring model tiers in 2025.

The Tiered Model Architecture

Modern AI providers don't ship one model - they ship a spectrum. A "Sol"-style tier typically targets reasoning-heavy tasks: long-context analysis, complex coding, multi-step planning. A "Luna"-style tier is a distilled or capacity-constrained variant optimized for speed and cost at scale.

The mechanism behind this involves a combination of model distillation (training a smaller model on outputs from a larger one), quantization (reducing numerical precision to shrink compute requirements), and sometimes speculative decoding (using the smaller model to draft tokens the larger one then validates). The result is a family of models that share a lineage but optimize for different latency and cost budgets. The Sol/Luna framing makes the tier explicit to users rather than hiding it behind vague "standard" vs. "advanced" labels - which creates clearer expectations and surfaces a real trade-off decision.

Real Example

If you're building on the API and need to decide which tier fits your use case, a simple benchmark loop can surface the difference quickly:

import openai, time

prompts = ["Summarize this 3000-word doc: ...", "Write a regex for nested JSON keys"]

for model in ["gpt-4o", "gpt-4o-mini"]:
 for prompt in prompts:
 start = time.time()
 r = openai.chat.completions.create(
 model=model,
 messages=[{"role": "user", "content": prompt}]
 )
 print(f"{model} | {round(time.time()-start,2)}s | {r.usage.total_tokens} tokens")

Run this against your actual workload - not synthetic benchmarks - before committing to a tier in production. Latency differences often matter more than quality differences for high-throughput pipelines, while quality gaps show up most on reasoning-dense tasks.

For non-API users: the practical signal is that if Luna is now free, it's worth testing it against whatever you were doing with the paid tier before assuming you need to upgrade. For many content-generation and summarization tasks, the delta is smaller than the pricing difference implies.

Key Takeaways

Tiered model families (Sol/Luna style) reflect real architectural differences - distillation, quantization, and compute budgets - not arbitrary feature locks.
For builders, the right tier decision depends on your task type: reasoning-heavy tasks favor the larger tier, throughput-heavy tasks often don't.
Explicit tier naming benefits users by making trade-offs visible - watch for this pattern to spread as providers compete on transparency, not just capability.

Have you found a task category where the mini/free tier consistently outperforms expectations - or one where it reliably falls apart?

Sources referenced: HackerNews discussion - "Improving GPT-4o Sol in ChatGPT, expanding GPT-4o Luna access for free users"

Building a Kubernetes Troubleshooting Agent With LangGraph

Basavaraj SH — Fri, 07 Aug 2026 00:25:18 +0000

Kubernetes incidents move fast - a pod crash loop at 2am doesn't wait for a human to read five dashboards. An autonomous SRE agent can triage, diagnose, and propose fixes while keeping a human in the approval loop for anything destructive.

The Core Pattern: Investigate First, Act With Permission

The safe design uses a two-phase loop. The agent gets read-only tools for observation - kubectl describe, log fetching, metrics queries - and a separate, gated set of write tools for remediation like scaling deployments or restarting pods. Every write action pauses and waits for explicit human approval before execution.

This maps cleanly onto LangGraph's interrupt mechanism, which lets a node in the agent graph pause execution, surface a structured decision to a human reviewer, and resume only on confirmation. The agent can reason over what it finds and generate a targeted fix rather than simply running a fixed runbook - but the blast radius is controlled by that approval gate.

Tracing is strongly recommended for this kind of agent. Without full visibility into which tool calls the agent made and what it decided at each step, debugging a bad remediation in production is nearly impossible. LangSmith (LangChain's observability layer) or any equivalent tracing setup captures the full execution trace so you can replay what happened.

Real Example

Here's a simplified LangGraph node that fetches pod status and flags for human review before restarting:

from langgraph.graph import StateGraph
from langgraph.checkpoint.memory import MemorySaver

def investigate(state):
 pod_status = kubectl_describe(state["pod_name"]) # read-only
 state["diagnosis"] = parse_crash_reason(pod_status)
 return state

def request_approval(state):
 # interrupt() pauses here; human sees diagnosis + proposed action
 human_decision = interrupt({
 "diagnosis": state["diagnosis"],
 "proposed_action": f"Restart pod {state['pod_name']}"
 })
 state["approved"] = human_decision["approved"]
 return state

def remediate(state):
 if state["approved"]:
 kubectl_rollout_restart(state["pod_name"]) # write action
 return state

graph = StateGraph(dict)
graph.add_node("investigate", investigate)
graph.add_node("request_approval", request_approval)
graph.add_node("remediate", remediate)
graph.set_entry_point("investigate")
graph.add_edge("investigate", "request_approval")
graph.add_edge("request_approval", "remediate")

The interrupt() call is what separates an autonomous agent from a fully automated one - it lets the agent do the cognitive heavy lifting while keeping a human accountable for state changes in production.

Key Takeaways

Separate read tools from write tools in your agent's toolkit - investigation should never require destructive permissions
Use an interrupt/approval gate before any action that modifies cluster state, not just the riskiest ones
Full execution tracing isn't optional for production agents; you need to reconstruct exactly what the agent reasoned and did

Have you found a reliable way to set the right granularity for approval gates - per action, per incident, or based on severity thresholds?

Sources referenced: LangChain Blog - "How we built an autonomous SRE agent for Kubernetes"

Qwen3 235B Tops the Agentic Benchmark - What That Test Actually Measures

Basavaraj SH — Fri, 07 Aug 2026 00:12:48 +0000

Agentic benchmarks rank models differently than chat benchmarks do, and the gap between the two scores is now wide enough to matter for real deployments.

The Agentic Index and Why It Diverges From Chat Leaderboards

Most familiar leaderboards (MMLU, HumanEval, MT-Bench) measure a model's single-turn reasoning or code correctness. The Agentic Index scores something harder: multi-step task completion, where the model must call tools, handle intermediate results, recover from errors, and reach a final goal across many turns - the same loop that runs inside frameworks like LangGraph, AutoGen, or CrewAI.

Qwen3 235B (a Mixture-of-Experts model, meaning only a fraction of its 235 billion parameters activate per token, keeping inference costs lower than the raw number suggests) recently topped this index. That's notable because the models that score highest on chat benchmarks don't consistently win on agentic tasks. In production workflows, models often excel at single-turn reasoning but struggle when required to plan, call a search tool, interpret results, and loop back across multiple steps.

The practical implication: if you're building an agent - a pipeline where the model drives tool use rather than just answering questions - benchmark selection should match your actual use case.

Real Example

Here's a minimal LangGraph agent setup that lets you swap the underlying model and observe how completion rate changes across a multi-step task:

from langgraph.prebuilt import create_react_agent
from langchain_openai import ChatOpenAI # swap for any compatible endpoint

model = ChatOpenAI(
 base_url="https://your-qwen-endpoint/v1",
 model="qwen3-235b-a22b",
 api_key="your-key"
)

agent = create_react_agent(model, tools=[search_tool, calculator_tool])
result = agent.invoke({"messages": [("user", "Research Q3 revenue trends and summarize top 3 drivers")]})
print(result["messages"][-1].content)

Swap base_url and model to point at any OpenAI-compatible endpoint - Ollama, Together AI, Fireworks - and run the same multi-step task. Agentic benchmark gaps become visible fast: a model that writes great prose may stall on tool chaining or repeat a failed call indefinitely.

Key Takeaways

Agentic benchmarks measure multi-turn, tool-using task completion - a fundamentally different capability than single-turn chat or code quality scores.
Qwen3 235B's MoE architecture keeps per-token compute lower than its parameter count implies, making it more viable for high-throughput agent loops.
If your workflow involves tool calls, planning steps, or autonomous loops, run your own task-specific eval rather than defaulting to the leaderboard most people quote.

Benchmark context shapes build decisions more than most teams realize - which agentic tasks are you actually running in production that a standard eval wouldn't cover?

Sources referenced: HackerNews discussion thread, Agentic Index leaderboard (referenced in discussion)

Choosing Between LangChain, LangGraph, and Deep Agents

Basavaraj SH — Thu, 06 Aug 2026 23:13:01 +0000

Picking the wrong agent framework early can mean a painful rewrite later. Here's how to read the decision fast.

The Core Distinction

These three frameworks operate at different levels of abstraction, and that's the real axis to choose on - not popularity or GitHub stars.

LangChain is a toolkit of composable components: chains, retrievers, prompt templates, memory, tool integrations. It's best for linear or lightly branching workflows - think a RAG pipeline (retrieve relevant documents, then generate an answer) where the steps are mostly predictable. You wire components together, it runs them in sequence.

LangGraph is built on top of LangChain but shifts the mental model from a chain to a stateful graph, where nodes are actions and edges are conditional transitions. You define explicit control flow: if the model requests a tool call, go to node B; if it hits a retry limit, go to node C. This matters the moment your agent needs to loop, branch on its own output, or recover from errors in a structured way. The state is persisted across steps, so you can inspect or interrupt mid-run - critical for human-in-the-loop workflows.

Deep Agents is a higher-abstraction layer designed for agents that run longer, more autonomous tasks without you hand-coding every state transition. If LangGraph gives you the graph, Deep Agents gives you sensible defaults on top of it - built-in memory, task decomposition, and persistence - so you spend less time on plumbing.

For fixed sequences of steps, use LangChain. For explicit branching logic and state management, use LangGraph. For long-horizon autonomy with less custom wiring, use Deep Agents.

Real Example

Here's a minimal LangGraph state graph in Python showing the branching pattern that LangChain alone can't express cleanly:

from langgraph.graph import StateGraph, END
from typing import TypedDict

class AgentState(TypedDict):
 messages: list
 needs_tool: bool

def router(state: AgentState):
 return "tool_node" if state["needs_tool"] else END

graph = StateGraph(AgentState)
graph.add_node("llm_node", call_llm)
graph.add_node("tool_node", call_tool)
graph.add_conditional_edges("llm_node", router)
graph.set_entry_point("llm_node")
app = graph.compile()

The add_conditional_edges call is what separates this from a plain LangChain chain - the agent decides its own next step based on state, not a fixed sequence you hard-coded.

Key Takeaways

Choose LangChain for predictable, linear pipelines where the steps don't need to loop back on themselves
Choose LangGraph when your agent needs conditional branching, retries, or human checkpoints - and you want explicit control over that logic
Choose Deep Agents when you want longer autonomous runs without writing the full state machine yourself

Are you currently managing branching logic in LangChain with workarounds, or have you already moved that work into LangGraph?

Sources referenced: LangChain Blog - Deep Agents vs LangChain vs LangGraph

Human Oversight of AI Agents Failed 33% of the Time in Testing

Basavaraj SH — Thu, 06 Aug 2026 22:11:45 +0000

When AI agents ask for permission to act, how often do humans actually catch the dangerous ones? A study on AI agent command approval accuracy across 40,000 simulated runs found the answer is: not nearly enough.

The Approval Gap in Agentic AI

Modern AI agents - systems that don't just answer questions but take sequences of actions (browsing, writing files, calling APIs, executing code) - typically include a "human-in-the-loop" checkpoint where a person approves or rejects a proposed command before it runs. The assumption baked into most agent frameworks is that this approval step catches harmful or unintended actions. The study broke that assumption: humans missed roughly one in three genuinely threatening commands when acting as approvers.

The failure mode isn't carelessness. It's cognitive load and interface design. Approval queues move fast. Commands often look benign in isolation - delete_temp_files() sounds harmless until you realize "temp" was redefined upstream in the agent's chain. The threat only makes sense in context, and reviewers rarely have that context surfaced to them at the moment of decision.

Real Example

Here's a simplified pattern from agentic pipelines where this goes wrong. An agent orchestrating a data cleanup task might generate a tool call like:

# Agent-generated action, presented to human approver
agent.run_tool("file_manager", {
 "action": "delete",
 "target": "processed/", # looks safe
 "recursive": True
})

To a reviewer approving dozens of these per session, processed/ sounds like scratch data. What's not shown inline: three steps earlier, the agent symlinked processed/ to a production directory. The approval UI showed the command. It didn't show the chain.

The fix isn't slower humans - it's better tooling. Agent frameworks like LangGraph and AutoGen support step-level trace logging; show the last N actions alongside the approval prompt, not buried in a separate log view, to give reviewers the context they need to actually evaluate risk. Some teams are also adding a lightweight secondary model (a "critic" or "red-teamer") that flags high-risk tool calls before they reach human review, so humans spend attention on the ones that actually need it.

Key Takeaways

Human-in-the-loop approval is not a reliable safety net by itself - 33% miss rate at scale is a significant risk surface
The core problem is missing context at decision time, not reviewer intent or effort
Pairing trace context surfacing with an automated pre-filter (critic model) makes human review meaningfully more effective
Agent frameworks already have the logging infrastructure to support this - it's largely a UI/UX and workflow design problem, not a new research challenge

The tooling to fix this exists today inside most major agent frameworks. The question is whether teams building production agents are actually wiring it up - are you surfacing full action traces to your human approvers, or just the single command?

Sources referenced: HackerNews discussion, 226 points, 178 comments - study referenced in thread on AI agent command approval accuracy across 40,000 simulated runs

How Open Embedding Models Can Match GPT-Class Retrieval Quality

Basavaraj SH — Thu, 06 Aug 2026 08:48:53 +0000

RAG (Retrieval-Augmented Generation - a pattern where you fetch relevant documents before generating an answer) lives or dies on retrieval quality. A cheaper retrieval stack isn't always a compromise.

The Retrieval Quality Gap Is Often in the Chunking, Not the Model

Most teams assume the embedding model is the bottleneck when retrieval underperforms. The real culprit is usually upstream: how documents are split before they're embedded. Overlapping chunks with a recursive character splitter, combined with a well-tuned open embedding model like bge-large-en-v1.5 or nomic-embed-text, can close a surprising amount of the gap against expensive proprietary embeddings - often without touching the LLM layer at all.

The pattern that tends to work: chunk at a meaningful boundary (paragraph or sentence, not fixed character count), add a small overlap window (10 - 15% of chunk size), then embed with a model that scores well on the MTEB benchmark (Massive Text Embedding Benchmark - a standardized leaderboard for comparing retrieval and semantic similarity models). Many open models in the 1B parameter range sit competitively on MTEB against models costing 100x more per token to call via API.

Real Example

Here's a minimal retrieval setup using LangChain with a local open embedding model via Ollama:

from langchain.text_splitter import RecursiveCharacterTextSplitter
from langchain_community.embeddings import OllamaEmbeddings
from langchain_community.vectorstores import Chroma

splitter = RecursiveCharacterTextSplitter(
 chunk_size=512, chunk_overlap=64, separators=["\n\n", "\n", ". "]
)
docs = splitter.create_documents([your_text])

embeddings = OllamaEmbeddings(model="nomic-embed-text")
vectorstore = Chroma.from_documents(docs, embeddings)

results = vectorstore.similarity_search("your query here", k=4)

Swapping nomic-embed-text for bge-large-en-v1.5 (via HuggingFace) takes one line change. Running both against your actual dataset on a held-out set of queries - not a synthetic benchmark - is the fastest way to know which one fits your domain.

The cost difference is real: embedding a million tokens locally via Ollama is effectively free after the initial model pull, versus measurable per-call costs at API scale.

Key Takeaways

Chunking strategy (boundary type + overlap) affects retrieval quality at least as much as the embedding model choice - fix this before swapping models.
Open embedding models in the 1B range score competitively on MTEB and run locally or on modest GPU instances.
Evaluate retrieval quality on your own queries, not just published benchmarks - domain-specific retrieval patterns differ from general benchmarks.

If you've swapped embedding models in a production RAG pipeline, which metric did you actually use to confirm the quality change - cosine similarity threshold, end-task accuracy, or something else?

Sources referenced: HackerNews discussion - "Beating GPT-5.6 Sol on retrieval with 100x cheaper open models", MTEB Leaderboard (Hugging Face)

How to Audit AI-Generated Content Before It Shapes Your Opinion

Basavaraj SH — Mon, 03 Aug 2026 10:01:47 +0000

The Detection Problem

AI-written articles, especially those produced at scale to push a particular narrative, can pass casual reading tests with ease. What gives them away is pattern, not grammar: suspiciously uniform sentence rhythm, missing primary sources, no named journalists with a searchable byline, and publication dates that cluster around news events in ways that suggest reactive content farming rather than original reporting.

The deeper issue is provenance opacity - you often can't tell who funded the content, what incentive sits behind it, or whether the "news site" has any editorial independence from the organization it covers (or attacks). This matters especially when the content targets specific individuals or companies in a competitive or regulatory space, where financial incentives to shape public perception are high.

Real Example

You don't need a specialized tool to run a basic credibility audit. A structured prompt against any article - fed into an LLM you trust - can surface red flags fast:

You are a media literacy analyst. Review the following article and output:
1. Named authors with verifiable professional history (yes/no + details)
2. Primary sources cited vs. secondary or no sources
3. Emotional or loaded language count (flag phrases)
4. Publication date relative to the news event it covers
5. Ownership/funding disclosure present (yes/no)

Article: [paste article text here]

Run this as a system prompt with the article as user input in any chat-based LLM interface or via API. The output won't be perfect, but it creates a forcing function - you're making the audit explicit rather than relying on gut feel. For teams processing a lot of external content (market research, competitive intelligence, policy tracking), this kind of lightweight pipeline can be automated or coded into a simple script that batches articles and logs flag counts.

Key Takeaways

Provenance and funding disclosure are the clearest signals of editorial independence; their absence is a flag worth taking seriously.
A structured LLM-based audit prompt is a practical, low-effort way to make credibility checks systematic rather than intuitive.

When you're doing competitive research or tracking industry commentary, how are you currently verifying whether a source has an undisclosed stake in the narrative it's publishing?

Sources referenced: HackerNews discussion thread, 179 points

How MCP Reconnected a Developer's Broken AI Workflow

Basavaraj SH — Sun, 02 Aug 2026 09:26:24 +0000

Model Context Protocol (MCP) - the open standard for letting AI models talk to external tools and data sources - has gone from "interesting spec" to "actually useful" faster than most people expected. The shift is visible in practice.

The Core Shift: From One-Shot Prompts to Connected Context

AI workflows often operate with manual handoffs: you paste text in, get text out, carry results from one tool to the next. MCP offers an alternative - a structured way for a model to call tools, read files, query APIs, and pass context across steps. The protocol defines a client-server model where your AI host (Claude Desktop, a custom agent, a VS Code extension) connects to MCP servers that expose specific capabilities. Each server declares what tools it offers, and the model decides when to invoke them. A workflow where "go check the database, then update the doc, then file the ticket" can happen in one session rather than three copy-paste cycles becomes possible.

The practical growth in MCP adoption has been enabled by the growing ecosystem of pre-built servers for common tools (GitHub, Postgres, Slack, filesystem access) that make the setup fast enough to actually reach for.

Real Example

Here's a minimal MCP server in Python using the official SDK that exposes one tool - a word count check a content editor or PM might actually want during a review workflow:

from mcp.server.fastmcp import FastMCP

mcp = FastMCP("content-tools")

@mcp.tool()
def word_count(text: str) -> dict:
 """Count words and flag if over 500."""
 count = len(text.split())
 return {"count": count, "over_limit": count > 500}

if __name__ == "__main__":
 mcp.run()

Run it with python server.py, point your MCP-compatible client at it, and the model can now call word_count mid-conversation - no plugin store, no custom API wrapper, no manual copy-paste. Add a second @mcp.tool() for a readability score, a Postgres query, or a Jira ticket creator, and the same pattern scales.

The key insight: you're not building an agent framework from scratch. You're writing small, testable functions and letting the protocol handle when and how the model calls them.

Key Takeaways

MCP standardizes tool-calling so AI models can pull live context and take actions without manual handoffs between steps
Pre-built MCP servers for GitHub, databases, and file systems mean you can connect an existing workflow in minutes rather than days
The FastMCP Python SDK reduces server setup to decorated functions - accessible for data scientists and engineers alike, not just AI infra teams

The pattern is simple enough that a solo freelancer and a five-person product team can both reach for it - what matters is knowing which tool connection would actually save you the most friction.

If you've tried wiring MCP into an existing workflow, which server or tool integration turned out to be genuinely useful versus more trouble than it was worth?

Sources referenced: Simon Willison's July 2026 Newsletter, MCP Python SDK documentation (modelcontextprotocol.io)

How the EU AI Act's Risk Tiers Actually Classify Your AI System

Basavaraj SH — Fri, 31 Jul 2026 13:55:24 +0000

The EU AI Act isn't just incoming regulation - it classifies AI systems by risk and sorts what you're required to build, document, and prove before you ship.

The Risk Tier System and What It Requires

The Act sorts AI systems into four risk tiers: unacceptable risk (banned outright), high risk, limited risk, and minimal risk. Most teams building internal tools, chatbots, or recommendation features land in the limited or minimal risk buckets - which carry light obligations, mostly around transparency (telling users they're interacting with AI).

The tier that demands real engineering attention is high risk. Systems in this category include AI used in hiring, credit scoring, education assessment, critical infrastructure, and healthcare triage. If your product touches any of those domains, you're looking at mandatory conformity assessments, human oversight mechanisms, detailed technical documentation, logging of system outputs, and data governance requirements before you can legally deploy in the EU.

High-risk systems must be built to support auditability from the start - not retrofitted. That means logging inputs and outputs with enough context to reconstruct decisions, building in a kill switch or human override, and maintaining a risk register that maps each failure mode to a mitigation.

Real Example

Here's a minimal logging pattern for a high-risk AI decision endpoint - the kind of audit trail the Act expects to exist:

import uuid, datetime, json

def log_ai_decision(input_data, model_output, model_id):
 record = {
 "decision_id": str(uuid.uuid4()),
 "timestamp": datetime.datetime.utcnow().isoformat(),
 "model_id": model_id,
 "input_hash": hash(json.dumps(input_data, sort_keys=True)),
 "output": model_output,
 "human_reviewed": False,
 }
 # Write to append-only store (e.g., S3, BigQuery, audit DB)
 audit_log.append(record)
 return record

This isn't compliance theatre - it's the minimum scaffold you need to answer "what did your system decide, and why, on date X?" - a question regulators can ask. The human_reviewed flag signals where oversight happened and where it didn't.

For non-technical roles: think of this as a receipt system for every AI decision your product makes. Without it, you can't demonstrate accountability after the fact.

Key Takeaways

The EU AI Act's impact on your team depends almost entirely on which risk tier your use case falls into - mapping that early saves rework later.
High-risk AI systems require audit logging, human override mechanisms, and pre-deployment conformity assessments - these aren't add-ons, they're architectural requirements.
Products in limited or minimal risk tiers mostly just need to disclose that they're AI-powered - a much lower bar, but still a legal one.

If you're currently building an AI feature for a regulated domain, what's your team's current approach to logging model decisions in a way that could survive a compliance review?

Sources referenced: OpenAI Blog - Advancing responsible AI across Europe; EU AI Act official text summary (European Parliament)

LangSmith LLM Gateway Adds Spend Limits and PII Redaction at Runtime

Basavaraj SH — Fri, 31 Jul 2026 10:49:48 +0000

AI agents that can call LLMs freely are a budget and compliance risk waiting to happen. LangSmith's new LLM Gateway addresses both without forcing you to rewrite your agent logic.

The Idea: Governance as Infrastructure, Not Afterthought

Most teams bolt on cost controls and data-privacy rules after something goes wrong - a runaway agent burns $400 in a weekend, or a support bot logs a customer's social security number into a trace. The LangSmith LLM Gateway moves those guardrails into the request path itself, sitting between your agent and the underlying model provider.

Spend limits are blocked at runtime - meaning an agent that exceeds its token budget stops before the API call goes out, not after you audit last month's invoice. PII redaction works the same way: sensitive patterns are stripped from the request and from the stored trace, so the compliance problem never reaches the model or the log in the first place. Crucially, trace continuity is preserved, so you still get full observability - you can see what the agent tried to do, just with the sensitive fields masked rather than missing.

Real Example: Wiring a Spend Limit Into Your Chain

If you're already using LangChain + LangSmith, the gateway integrates at the client initialization level. A minimal setup looks roughly like this:

from langsmith import Client

client = Client(
 api_key="ls-...",
 gateway_config={
 "spend_limit_usd": 10.0, # hard stop per session/agent run
 "pii_redaction": True, # strip emails, SSNs, phone numbers
 "fallback_on_limit": "error", # or "warn" to log and continue
 }
)

Once the gateway is in place, every LLM call your agent makes flows through it. The spend counter increments per token usage; hitting the ceiling raises a controlled exception your agent can catch and handle gracefully - returning a partial result or escalating to a human - instead of silently running up charges. PII redaction runs on both the outbound prompt and the inbound completion before either touches the LangSmith trace store.

This is particularly useful for Forward Deployed Engineers shipping agents to enterprise customers who have strict data-handling requirements baked into their contracts, and for product teams running multi-agent workflows where a single misconfigured node could drain a shared budget.

Key Takeaways

Runtime governance means the guardrail executes during the agent's API call, not in a post-hoc audit - which is the only layer that actually prevents harm.
PII redaction at the gateway level keeps sensitive data out of your trace store, which directly reduces compliance surface area without sacrificing observability.
Spend limits paired with a configurable fallback give agents a graceful degradation path instead of an uncontrolled failure or an unlimited bill.

If you're already tracking agent runs in LangSmith, what's your current approach to capping per-run LLM spend - token budget in the prompt, external middleware, or something else?

Sources referenced: LangChain Blog - LangSmith LLM Gateway announcement

How to Detect and Handle API Outages Gracefully in AI-Powered Apps

Basavaraj SH — Thu, 30 Jul 2026 13:58:50 +0000

When a major LLM provider goes down, your app shouldn't. Elevated error rates across model APIs are a when, not an if - so the resilience layer matters as much as the integration itself.

The Idea: Circuit Breakers and Fallback Chains

Most teams wire up an LLM API and handle errors with a basic try/except. That works fine until a real outage hits - then every request hangs or fails hard, and users see a broken product instead of a graceful degradation.

The pattern worth building is a fallback chain with a circuit breaker. Track consecutive failures, and once a threshold is crossed, stop hammering the primary provider and route to a backup - whether that's a different provider, a cached response, or a simplified deterministic answer. A circuit breaker (a pattern borrowed from distributed systems) sits in front of your API call and activates when failure rate spikes, protecting your app from compounding timeouts.

The two pieces together - fallback routing and circuit breaking - let you serve something useful even during a full provider outage.

Real Example

Here's a minimal Python pattern using tenacity for retries and a simple circuit breaker flag, with provider fallback:

import openai
import anthropic
from tenacity import retry, stop_after_attempt, wait_exponential

CIRCUIT_OPEN = False
FAILURE_COUNT = 0
THRESHOLD = 3

def call_primary(prompt):
 client = anthropic.Anthropic()
 return client.messages.create(
 model="claude-3-5-sonnet-20241022",
 max_tokens=256,
 messages=[{"role": "user", "content": prompt}]
 ).content[0].text

@retry(stop=stop_after_attempt(2), wait=wait_exponential(min=1, max=4))
def call_fallback(prompt):
 client = openai.OpenAI()
 return client.chat.completions.create(
 model="gpt-4o-mini",
 messages=[{"role": "user", "content": prompt}]
 ).choices[0].message.content

def get_response(prompt):
 global CIRCUIT_OPEN, FAILURE_COUNT
 if not CIRCUIT_OPEN:
 try:
 result = call_primary(prompt)
 FAILURE_COUNT = 0
 return result
 except Exception:
 FAILURE_COUNT += 1
 if FAILURE_COUNT >= THRESHOLD:
 CIRCUIT_OPEN = True
 return call_fallback(prompt)

In production you'd replace the global flags with something like pybreaker or a Redis-backed counter so the state persists across workers. The routing logic stays the same.

Key Takeaways

API outages from any LLM provider are recurring events - building for them upfront is cheaper than scrambling during one.
A circuit breaker prevents cascading timeouts by stopping requests to a failing endpoint once a failure threshold is hit.
A fallback chain (primary provider → secondary provider → cached/static response) keeps your product functional at degraded quality rather than fully broken.

Have you actually tested your LLM integration with the primary endpoint completely unreachable - what happened?

Sources referenced: HackerNews discussion on Claude elevated errors incident, Anthropic status page