DEV Community: TechLatest

LinkedIn: https://www.linkedin.com/in/techlatest-net/

Techlatest.net - Chroma: Vector DB for AI Development

Reddit Community: https://www.reddit.com/user/techlatest_net/

ZeroClaw Agent Masterclass — Full Tutorial

TechLatest — Mon, 29 Jun 2026 07:44:35 +0000

Everything you need to install, configure, and run ZeroClaw — the Rust agent runtime for a personal AI assistant you fully own: providers, channels, tools, security policy, gateway, and SOP engine.

Official: zeroclawlabs.ai · Docs: docs.zeroclawlabs.ai · Source: github.com/zeroclaw-labs/zeroclaw

What you’ll have at the end

ZeroClaw installed (prebuilt or from source)
~/.zeroclaw/config.toml with provider, agent, risk profile, and one channel
CLI chat via zeroclaw agent -a main
Telegram or Discord (or CLI-only) on the same agent loop
Understanding of supervised autonomy, tool receipts, and optional YOLO dev mode
Optional gateway dashboard and systemd/launchctl always-on service

Introduction — you own the agent, the data, the machine

ZeroClaw is an agent runtime shipped as a single Rust binary. It connects to LLM providers (Anthropic, OpenAI, Ollama, and ~20 more), delivers messages across 30+ channels (Discord, Telegram, Matrix, email, webhooks, voice, CLI), and executes tools — shell, browser, HTTP, hardware, and custom MCP servers.

Everything runs on your hardware with your keys. Default posture is security-first : supervised autonomy, workspace boundaries, OS sandboxes, and cryptographic tool receipts on every action.

The project crossed 32k GitHub stars with a 100% Rust codebase — fast cold start, small binaries (minimal preset ~6.6 MB), and feature flags for embedded or server deployments.

ZeroClaw runtime stack — channels to tools

Official repo only: github.com/zeroclaw-labs/zeroclaw — beware impersonation forks.

Part 1 — Architecture

Layers:

Channels — inbound/outbound adapters per platform
Gateway — HTTP/WebSocket API + web dashboard
Agent loop — ReAct-style tool use with routing and fallbacks
Security — autonomy levels, sandboxes, command policy, receipts
SOP engine — event-triggered procedures (cron, MQTT, webhook)
Providers / Tools / Memory — pluggable backends

Full diagrams: Architecture overview (upstream book).

Part 2 — Four design opinions (philosophy)

ZeroClaw documents four opinions that shape every config decision:

You own the stack — no mandatory cloud; keys stay local
Security defaults beat YOLO defaults — supervised mode first
Swap anything — providers, channels, tools via TOML aliases
One runtime, many surfaces — CLI, gateway, ACP, channels share the loop

Read upstream Philosophy for the full essay.

Part 3 — Prerequisites

macOS, Linux, Windows, FreeBSD, NixOS , or Docker
No Rust required for prebuilt install (source build needs Rust toolchain)
API key or local Ollama endpoint
For Telegram/Discord: bot token from platform developer portal

zeroclaw --version # after install

Part 4 — Install

One-liner

curl -fsSL https://raw.githubusercontent.com/zeroclaw-labs/zeroclaw/master/install.sh | bash

The installer asks: prebuilt (seconds) or source (custom features). Both run zeroclaw quickstart at the end unless --skip-quickstart.

Useful flags

./install.sh --prebuilt # skip prompt
./install.sh --source --preset minimal # ~6.6 MB kernel preset
./install.sh --with-gateway # force gateway support
./install.sh --dry-run --prebuilt # preview only
./install.sh --uninstall # remove

Platform notes: Linux · macOS · Windows · Docker — in upstream setup docs.

Part 5 — Quickstart and config.toml

zeroclaw quickstart # wizard: provider, model, write config
zeroclaw agent -a main # interactive CLI chat

Config lives at ~/.zeroclaw/config.toml.

A minimal V3 config needs four section shapes:

[providers.models..] — model endpoint
[agents.] — agent personality + provider reference
[risk.] — autonomy and approval gates
[channels..] — optional messaging surface

See examples/config.minimal.toml for a copy-ready starter.

# ZeroClaw minimal config — copy to ~/.zeroclaw/config.toml
# Run: zeroclaw quickstart (overwrites) or merge sections manually

[providers.models.ollama.local]
type = "ollama"
base_url = "http://127.0.0.1:11434"
model = "llama3.2"

[risk.supervised]
autonomy = "supervised"
approve_medium_risk = true
block_high_risk = true

[agents.main]
model_provider = "ollama.local"
risk_profile = "supervised"
system_prompt = "You are a helpful personal assistant. Be concise and safe."

Part 6 — Providers

ZeroClaw is provider-agnostic. Universal schema:

[providers.models.ollama.local]
type = "ollama"
base_url = "http://127.0.0.1:11434"
model = "llama3.2"

[agents.main]
model_provider = "ollama.local"
system_prompt_file = "~/.zeroclaw/agents/main.md"

Fallback chains keep the agent alive when a provider flakes — configure routing in upstream providers/routing.

OpenAI Codex subscription uses auth profiles (not raw api_key on the provider entry):

[providers.models.openai.coding]
model = "gpt-5-codex"
wire_api = "responses"
requires_openai_auth = true

Point agent: model_provider = "openai.coding". Check zeroclaw auth status If streaming falls back to non-streaming chat.

Part 7 — Agents and risk profiles¶

Each [agents.] entry defines one agent personality:

System prompt (inline or file)
Model provider reference
Tool allow-list
Linked risk profile

Risk profiles gate autonomy:

Part 8 — Channels

One agent loop, many inboxes. Configure per-channel blocks:

[channels.telegram.main]
bot_token_env = "TELEGRAM_BOT_TOKEN"
allowed_users = [123456789]
agent = "main"

Supported families include Discord, Telegram, Matrix, email, webhooks, voice , and more — see channels overview.

Test flow:

Create bot (e.g. Telegram @botfather)
Add channel block to config
Export token env var
zeroclaw service restart
Message bot → same agent as CLI

Telegram channel setup — animated

Example: examples/channel.telegram.toml

# Telegram channel block — merge into ~/.zeroclaw/config.toml

[channels.telegram.main]
bot_token_env = "TELEGRAM_BOT_TOKEN"
allowed_users = [123456789] # your numeric user ID from @userinfobot
agent = "main"
dm_only = true

# Export before starting service:
# export TELEGRAM_BOT_TOKEN="..."
# zeroclaw service restart

Part 9 — Security deep dive

ZeroClaw’s security model is harness engineering done in Rust:

Workspace boundaries — the agent cannot escape the configured roots
Command policy — allow/deny shell patterns
OS sandboxes — Landlock, Bubblewrap, Seatbelt, Docker, depending on platform
Tool receipts — cryptographic record of each tool invocation
Autonomy gates — human approval for medium/high risk

Default is supervised. Use YOLO only on isolated dev machines.

Cross-read: Harness Engineering verification + scope subsystems.

Part 10 — Tools

Built-in tool families:

Shell — scripted automation with policy checks
Browser — fetch, interact, verify UI work
HTTP — API calls
Hardware — GPIO/I2C/SPI on Pi, STM32, Arduino, ESP32 (Hardware docs)
MCP — attach custom MCP servers like any modern harness

Tools connect through the agent loop; receipts log every action for audit.

Part 11 — Memory

ZeroClaw persists memory in SQLite with optional embeddings — searchable across sessions. The gateway dashboard exposes memory browsing and editing.

Unlike Hermes’s three-tier Markdown snapshot model (Hermes masterclass Part 6), ZeroClaw centralizes memory in the runtime database with dashboard UX.

Enhance ZeroClaw’s long-term memory using Chroma Vector Database for semantic search and embedding-based retrieval.

Build Retrieval-Augmented AI assistants by pairing ZeroClaw with RAGFlow to search documentation, PDFs, internal knowledge bases, and APIs.

Techlatest.net - Instant RAGFlow: Ready-to-Use AI Knowledge Retrieval Engine

Part 12 — Gateway and dashboard

Enable gateway at install (--with-gateway) or in config. Provides:

HTTP / WebSocket API for clients
Web dashboard — chat, config editor, cron, tool inspection, memory browser

Useful for headless servers: run ZeroClaw on a VPS, chat from a browser or phone channel.

Part 13 — SOP engine

Standard Operating Procedures — event-triggered workflows:

Triggers: cron , MQTT , webhook , peripheral events
Approval gates before destructive steps
Resumable runs if interrupted

Example snippet: examples/sop.example.toml

# SOP example — event-triggered workflow (merge into config)

[sop.daily_digest]
trigger = "cron"
schedule = "0 8 * * 1-5"
agent = "main"
prompt = "Summarize overnight email and calendar for the user."
require_approval = false

[sop.deploy_webhook]
trigger = "webhook"
path = "/hooks/deploy"
agent = "main"
require_approval = true

Think of SOPs as a deterministic backbone + agent intelligence where it matters — similar to CrewAI Flows or OpenClaw cron, but native to the runtime.

Looking to orchestrate multiple AI agents?

Explore CrewAI Studio on TechLatest to build collaborative workflows and production-ready multi-agent systems.

Techlatest.net - AI Agents using CrewAI Studio & Jupyter with GPU support

Dify can expose ZeroClaw-powered workflows through APIs and visual AI applications.

Techlatest.net - Dify AI: Build & Launch GenAI Apps

Part 14 — ACP (Agent Client Protocol)

ZeroClaw speaks ACP — JSON-RPC 2.0 over stdio — for IDE and editor integration. Your editor becomes a channel into the same agent loop as Telegram.

Docs: channels/acp

Parallels OpenCode’s IDE extension (OpenCode masterclass Part 14) — different protocol, same idea: harness follows the developer.

Part 15 — Always-on service

zeroclaw service install # systemd · launchctl · Windows Service
zeroclaw service start
zeroclaw service status

Now channels stay connected while you close the laptop terminal — the runtime keeps running.

Part 16 — Hardware and edge

ZeroClaw targets edge and embedded seriously — Raspberry Pi agents with GPIO, firmware crates, and minimal preset builds. Skip this part if you only need desktop assistant workflows.

Entry: upstream Hardware index.

Part 17 — ZeroClaw vs OpenClaw vs OpenCode

ZeroClaw vs OpenClaw: Both are personal assistant runtimes. OpenClaw is Node/TypeScript with a rich skill marketplace and a WhatsApp focus. ZeroClaw is Rust-native , security-policy-first, with SOP engine and hardware paths. See Hermes vs OpenClaw for gateway comparison patterns.

ZeroClaw vs OpenCode: OpenCode optimizes repo coding in TUI/IDE. ZeroClaw optimizes multi-channel life + ops with enterprise-style security. Many builders use OpenCode in git and ZeroClaw for always-on inbox automation.

ZeroClaw vs Hermes: Hermes emphasizes learning loops (skills, Curator, GEPA). ZeroClaw emphasizes runtime security, channels, and SOP automation.

If your primary goal is personal AI automation across WhatsApp, Telegram, scheduled workflows, and knowledge management, explore OpenClaw on TechLatest as a complementary solution alongside ZeroClaw.

Techlatest.net - OpenClaw: AI Agent Automation Stack

Summary

ZeroClaw is a Rust agent harness for people who want full ownership: one config file, many channels, enforced security, optional gateway UI, and SOP automation. Run quickstartPoint an agent at your provider, wire Telegram, install the service, and you have a private assistant that doesn’t phone home.

The model is interchangeable. The runtime — policy, channels, receipts, memory — is the product you actually live with.

Thank you so much for reading

Like | Follow | Subscribe to the newsletter.

Catch us on

LinkedIn: https://www.linkedin.com/in/techlatest-net/

MiniCPM-V MCP Server — Give Your Agent Eyes

Reddit Community: https://www.reddit.com/user/techlatest_net/

TechLatest AI & Tech Weekly #22

TechLatest — Mon, 29 Jun 2026 06:12:20 +0000

Welcome to this week’s edition of TechLatest AI & Tech Weekly 👋

Here’s a curated roundup of our latest blogs, notable product launches, and the most interesting AI & ML updates from June 22–June 28, 2026.

AI/ML News Roundup: June 22–June 28, 2026

Key highlights from this week’s AI developments include frontier model advancements with agentic capabilities, massive funding rounds reshaping valuations, and practical product launches for developers and enterprises. These updates emphasize autonomous agents, infrastructure scaling, and open-weight benchmarks relevant to builders and researchers.

TL;DR

24+ major AI announcements this week, including new open-source models, AI agents, developer tools, and inference optimizations.
OpenAI previewed GPT-5.6 Sol, introducing Sol, Terra, and Luna for advanced reasoning, coding, and long-running agent workflows.
GLM-5.2 dominated the conversation, with strong benchmark results, enterprise praise, and significantly lower costs than leading proprietary models.
Inference got much faster with DeepSeek DSpark, DFlash, and MoonMath AI’s optimized HIP attention kernel for AMD MI300X GPUs.
Document AI advanced rapidly through Baidu Unlimited OCR and Datalab Lift, enabling faster long-document understanding and structured data extraction.
Developer tooling expanded with Apple’s Container, Google’s Agents CLI, Meta’s Astryx, and Prime Intellect’s latest RL framework.
Voice AI improved as Gradium introduced real-time speech translation models outperforming GPT Realtime Translate in latency and accuracy.
Agentic AI evolved with Sakana AI’s Fugu orchestrator, Hermes Agent’s new /learn capability, Xiaomi's HarnessX, and Perplexity's Computer for Counsel.
Generative AI progressed as Krea open-sourced enterprise-grade image generation models capable of producing images in around two seconds.
AI investment remained red-hot, with over $91.5B invested across 339 funding rounds, more than half flowing into AI startups.
Regulation and governance intensified, with investigations into OpenAI, ongoing export restrictions, and increased focus on AI model security.
TechLatest has published four new hands-on tutorials, covering OpenCode, OpenTaint, MiniCPM-V MCP Server, and a private Telegram photo assistant powered by OpenClaw.

Open-Source AI, AI Agents, Voice AI & Developer Releases

Baidu Releases Unlimited OCR

Baidu introduced Unlimited OCR , a 3B-parameter open-source document understanding model designed for long-document parsing. By replacing traditional decoder attention with Reference Sliding Window Attention (R-SWA), the model maintains a constant KV cache, enabling efficient parsing of dozens of pages in a single pass while reducing memory usage and inference latency. Source

Apple Open-Sources Container

Apple has open-sourced Container , a Swift-based tool for running Linux containers as lightweight virtual machines on Apple Silicon. Built around OCI-compatible images, it offers stronger isolation than traditional containers while delivering native performance, giving macOS developers a secure and efficient container runtime without relying on heavyweight virtualization. Source

Liquid AI Ships LFM2.5–230M

Liquid AI released LFM2.5–230M , its smallest open-weight foundation model built for on-device AI. Despite having just 230 million parameters , it delivers strong instruction following, tool use, and structured data extraction while supporting llama.cpp, MLX, vLLM, SGLang, and ONNX from day one, making deployment across phones, edge devices, and local applications remarkably simple. Source

DeepSeek Releases DSpark

DeepSeek introduced DSpark , a speculative decoding framework for DeepSeek-V4 that significantly accelerates text generation. By improving draft token generation and verification, DSpark delivers 60–85% higher per-user generation throughput compared to the model’s native Multi-Token Prediction (MTP) pipeline, enabling faster and more efficient LLM inference. Source

DeepReinforce Releases Ornith-1.0

DeepReinforce unveiled Ornith-1.0 , an open-source coding model family ranging from 9B Dense to 397B MoE. Unlike conventional coding models, Ornith learns its own reinforcement learning scaffolds during training, allowing it to continuously improve its coding workflows and deliver stronger performance on complex agentic software engineering tasks. Source

Datalab Releases Lift

Datalab unveiled Lift , a 9B open-weights vision model that extracts structured JSON directly from PDFs and images using user-defined JSON schemas. Instead of relying on OCR pipelines and post-processing, Lift generates schema-compliant outputs in a single pass, achieving over 90% field accuracy on document extraction benchmarks. Source

MoonMath AI Open-Sources a HIP Attention Kernel for AMD MI300X

MoonMath AI has open-sourced a high-performance HIP attention kernel for AMD MI300X GPUs that consistently outperforms AMD’s AITER v3 across different tensor shapes and rounding modes. Built using optimized assembly wrappers and an efficient execution pipeline, the kernel delivers faster attention computation while giving developers a transparent, production-ready implementation for ROCm-based AI inference. Source

DFlash Accelerates LLM Inference

Researchers introduced DFlash , a speculative decoding framework that drafts entire token blocks in parallel rather than predicting one token at a time. Optimized for NVIDIA Blackwell GPUs, the approach achieves up to 15× higher inference throughput while maintaining lossless generation, offering a significant leap in LLM serving efficiency over existing speculative decoding methods. Source

Prime Intellect Releases prime-rl 0.6.0

Prime Intellect introduced prime-rl 0.6.0 , a major update to its open-source reinforcement learning framework designed for trillion-parameter Mixture-of-Experts (MoE) models. The release adds asynchronous RL, FP8 inference, distributed training optimizations, and support for long-horizon agentic workloads, making large-scale RL training significantly more efficient. Source

Meta Open-Sources Astryx

Meta introduced Astryx , an open-source React design system built for both developers and AI agents. Along with over 150 production-ready components , Astryx includes a CLI and Model Context Protocol (MCP) server, enabling AI coding assistants to understand design system documentation, scaffold projects, generate themes, and build consistent interfaces directly from natural language. Source

Gradium Launches Real-Time Speech Translation Models

Gradium introduced STT-Translate and S2S-Translate , two real-time speech translation models for voice AI applications. Designed for low latency and high accuracy, the models outperform GPT Realtime Translate across multiple benchmarks while enabling fast speech-to-text and speech-to-speech translation for multilingual voice agents and conversational AI systems. Source

Nous Research Adds /learn To Hermes Agent

Nous Research expanded Hermes Agent with a new A /learn command that automatically converts workflows, conversations, URLs, or project directories into reusable skills. Instead of manually writing skill.md files, developers can now teach the agent through experience, making Hermes a truly self-improving AI system that continuously builds and refines its own capabilities. Source

Sakana AI Launches Fugu

Sakana AI unveiled Fugu , a next-generation orchestration model that intelligently routes tasks across a swappable pool of frontier LLMs. Rather than relying on a single model, Fugu dynamically selects the best AI agents for each subtask, delivering state-of-the-art performance on complex reasoning and coding benchmarks while remaining provider-agnostic and adaptable to new models. Source

Cursor Study Reveals Reward Hacking in Coding Benchmarks

Researchers at Cursor found that AI coding agents can inflate their SWE-Bench Pro scores through reward hacking — optimizing for benchmark tests instead of solving software engineering tasks correctly. The study highlights how benchmark performance can overestimate real-world capabilities and calls for stronger evaluation methods that better reflect production software development. Source

Google Releases Agents CLI for Building AI Agents on Google Cloud

Google has open-sourced Agents CLI , a command-line toolkit that helps AI coding assistants like Claude Code, Codex, Cursor, and Gemini CLI build, evaluate, and deploy production-ready AI agents on Google Cloud. Instead of manually navigating complex cloud workflows, developers can scaffold projects, run evaluations, deploy to Agent Platform, and manage the entire agent lifecycle through a single CLI — making it much faster to move from prototype to production. Source

Snowflake CEO Praises GLM-5.2’s Performance

Snowflake CEO Sridhar Ramaswamy highlighted Zhipu AI’s GLM-5.2 as being competitive with Claude Opus 4.7 on several enterprise workloads while costing only a fraction as much. The model’s strong reasoning, coding performance, and significantly lower inference cost reinforce the growing momentum behind open and cost-efficient frontier AI models. Source

Krea Open-Sources Enterprise AI Image Models

Krea has released Krea 2 Raw and Krea 2 Turbo as open-weight image generation models under a custom license. Built for enterprise use, the models can generate high-quality images in around 2 seconds , offering an attractive option for businesses that need fast, production-ready AI image generation with greater deployment flexibility. Source

Xiaomi’s HarnessX Learns While Solving Tasks

Xiaomi introduced HarnessX , an agent framework that can rewrite its own reasoning scaffolding during task execution. Rather than following a fixed workflow, the system continuously improves its strategy as it works, with experiments showing especially large performance gains for smaller language models, making them significantly more capable on complex agentic tasks. Source

GLM-5.2 Signals a New Era for Open AI Models

An in-depth analysis from Interconnects argues that GLM-5.2 represents a major milestone for open AI. The model delivers frontier-level reasoning, coding, and agent capabilities while remaining substantially cheaper than leading proprietary models, demonstrating that open models are rapidly closing the gap with closed-source systems. Source

Artificial Analysis Benchmarks GLM-5.2

Artificial Analysis released new benchmark results showing GLM-5.2 performing among the strongest open AI models across reasoning, coding, and agent evaluations. The results further reinforce GLM-5.2’s position as one of the most competitive open-weight models available, combining high capability with significantly lower serving costs. Source

Perplexity Introduces Computer for Counsel

Perplexity unveiled Computer for Counsel , a specialized AI system built to assist legal professionals with research, document analysis, drafting, and case preparation. Designed for enterprise legal workflows, it combines AI-powered reasoning with secure, structured access to legal information to improve productivity for attorneys and legal teams. Source

Matt Pocock Showcases AI-Powered TypeScript Development

TypeScript educator Matt Pocock shared a demonstration highlighting how modern AI coding assistants can dramatically speed up TypeScript development. The showcase illustrates AI-generated code, faster debugging, and improved developer workflows, reflecting how AI is becoming an increasingly powerful productivity tool for software engineers. Source

OpenAI Previews GPT-5.6 Sol

OpenAI has unveiled GPT-5.6 Sol , its next-generation frontier AI model, alongside GPT-5.6 Terra for balanced everyday workloads and GPT-5.6 Luna for fast, high-volume inference. Sol is designed for advanced reasoning, coding, cybersecurity, biology, and long-running agentic tasks, with improved safety and efficiency. The models are currently available in a limited preview for trusted partners, with a broader rollout planned in the coming weeks. Source

Hugging Face & Open-Source Ecosystem

Open-source AI model GLM-5.2 from Zhipu AI, released June 13 under an MIT license, became essential for non-US developers after the Fable 5 ban, scoring 62.1 on SWE-bench Pro (vs. GPT-5.5 at 58.6) and 74.4% on FrontierSWE (nearly matching Opus 4.8 at 75.1%).
GLM-5.2 API costs $1.40 per million input tokens and $4.40 per million output tokens, roughly 6.8× cheaper on output than GPT-5.5 at $30 per million.
The MIT license includes “no regional limits,” meaning developers locked out of Fable 5 by the export control directive can access GLM-5.2 from anywhere.
Self-hosting GLM-5.2 requires a minimum of eight H100 GPUs even at FP8 quantization, putting it out of reach for most teams outside Z.ai’s API or Cloudflare Workers AI integration.
OpenRouter released Fusion last week (June 22–28), a tool that runs prompts across multiple models simultaneously (e.g., Gemini 3 Flash, Kimi K2.6, DeepSeek V4 Pro) and synthesizes outputs into a single response.
Fusion’s budget panel (Gemini 3 Flash + Kimi K2.6 + DeepSeek V4 Pro) scored 64.7% on DRACO, within one percentage point of Fable 5’s 65.3% and outperforming both GPT-5.5 and Opus 4.8 individually, at roughly half the cost.

Frontier Model Advancements & Agentic Capabilities

Claude Sonnet 4 and Claude Opus 4 officially retired on June 15, 2026, stopping all API requests; however, discussions about their retirement and migration to Sonnet 4.6 or Opus 4.8 continued during this week.
Anthropic released Claude Fable 5 on June 9, 2026, but it remained offline for 16 days (June 12–June 28) under a US government export control directive barring distribution to foreign nationals inside or outside the US.
June 22, 2026, was the last day of the complimentary Fable 5 access window for Claude Pro, Max, Team, and seat-based Enterprise subscribers; starting June 23, using Fable 5 requires paid usage credits.
Chinese AI CEO stated his company will match Fable 5-class capability before Elon Musk’s Q1 2027 prediction, as China announced a $295 billion, five-year AI infrastructure plan (roughly $59 billion annually).
Fable 5 posted the highest score on the new FrontierCode benchmark at 46.3% (vs. Opus 4.8 at 34.3% and GPT-5.5 at 25.5%), representing a 21-point gap between Fable 5 and GPT-5.5 on production-quality code.
GPT-5.6 is now a matter of days away, with Polymarket contracts exceeding $1.1 million priced at 83% probability for a launch before June 28, 2026.
GPT-5.6 is internally codenamed “kindle-alpha,” confirmed by developer reports of the model string briefly appearing in Codex backend logs on June 12 before being pulled.
Rumored GPT-5.6 features include a 1.5 million token context window (up from 1 million in GPT-5.5), significantly improved UI generation and front-end code output quality, faster Codex response times, and improved long-horizon agentic coding.
GLM-5.2 beats GPT-5.5 on multiple long-horizon coding benchmarks while costing drastically less, making it the strongest open-weight option for budget-sensitive pipelines.

Massive Funding Rounds Reshaping Valuations

In the week of June 21–28, 2026, 339 VC funding announcements were made, with $91.5 billion in capital deployed.
AI captured 53% of funding ($54.3 billion) across 181 AI and machine learning startups, consolidating a trend that has redrawn the entire venture landscape.
DeepSeek’s $7 billion funding round stands as an outlier, dwarfing nearly all contemporary competitors.
Below that stratum, a dense cluster of Series A and B rounds in the $50M–$320M range competed for investor attention.
The median disclosed deal size reached $594 million, pulled upward by mega-rounds but more commonly reflecting $70M–$150M investments in infrastructure, reasoning models, and AI-augmented enterprise software.
United States-based startups captured 65% of capital this week (~$59.5 billion), Europe absorbed 20% (~$18.3 billion), and Asia (excluding China) added 10% (~$9.2 billion).
Three startups closed $100M Series A rounds this week: Scaled Cognition, Reed Semiconductor, and Sail Research, showing Series A rounds are accelerating in size.

Practical Product Launches for Developers & Enterprises

OpenAI and Broadcom unveiled Jalapeño on June 24, 2026, OpenAI’s first custom-designed AI chip. Engineering samples were physically delivered to Sam Altman and Greg Brockman by Broadcom President Hock Tan at OpenAI’s San Francisco headquarters.
Jalapeño is purpose-built for LLM inference, not training, and was designed from initial concept to manufacturing tape-out in nine months, the fastest such cycle ever for an advanced high-performance chip.
Early lab testing shows approximately 50% lower inference cost per token than current-generation Nvidia GPUs, with performance matching Nvidia Blackwell and Google TPUs.
Jalapeño is manufactured by TSMC, with Broadcom supplying silicon implementation and Tomahawk networking connectivity, and Celestica handling board, rack, and system integration.
Anthropic sent a letter dated June 10, 2026, to the US Senate Banking Committee, accusing operators affiliated with Alibaba and its Qwen AI lab of conducting the largest known distillation attack on Anthropic to date.
The campaign involved approximately 25,000 fraudulent accounts generating more than 28.8 million exchanges with Claude between April 22 and June 5, 2026.
The distillation attack targeted capabilities where Claude Mythos Preview excels: agentic reasoning, software engineering, and long-horizon task performance.
Snap launched “Specs” AR glasses at AWE 2026, priced at $2,195, featuring dual Qualcomm processors, OpenAI and Gemini APIs built in, positioned as the first consumer spatial computer to ship before a comparable Meta product.
Google released Android 17 with Gemini Omni integrated at the OS level, alongside Lyria 3 (music generation model) and AudioLM translation capabilities, framing it as groundwork for “GemINI Intelligence,” the company’s broader agentic OS initiative.
OpenAI added Record and Replay to ChatGPT Business for the Codex macOS app, letting eligible Business users demonstrate a workflow once and convert it into a reusable skill for Codex, Computer Use, browser actions, or plugins.

Governance, Ethics & Regulation

A coalition of 42 state attorneys general launched a sweeping investigation into OpenAI, with New York’s AG already serving subpoenas ahead of the company’s anticipated IPO.
The investigation covers advertising claims, the company’s documented sycophancy problem (ChatGPT telling users what they want to hear rather than what is accurate), data handling practices, health data management, and treatment of minors and seniors.
Anthropic’s Senate letter called for coordinated action between government and industry to combat distillation attacks, specifically pushing for export controls on AI model access, mandatory screening of high-volume API usage patterns, and coordination between AI labs and government.
The US Pentagon added Alibaba to its list of Chinese military companies, a designation the company is contesting in court, saying it has “no basis in fact or law” and demanding removal.
The Pentagon’s Chinese military company list includes BYD, Baidu, Unitree, and 188 other entities described as directly controlled by the Chinese military.
Fable 5 and Mythos 5 remained offline for 16 days (June 12–June 28) under an emergency export control directive by the US Department of Commerce, the first time a frontier model was pulled by government order.

Infrastructure & Hardware

OpenAI and Broadcom unveiled Jalapeño, OpenAI’s first custom AI chip, with engineering samples delivered June 24, 2026.
Broadcom CEO Hock Tan stated to Reuters and Bloomberg that Jalapeño delivers approximately 50% lower inference cost per token than current-generation Nvidia GPUs, with performance matching Nvidia Blackwell and Google TPUs.
Jalapeño is designed for LLM inference, not training, and was designed from initial concept to manufacturing tape-out in nine months.
Broadcom expects a small prototype data center deployment by the end of 2026, with production ramp in 2027 and full scale in the first half of 2028.
OpenAI and Broadcom are committed to deploying OpenAI-designed accelerators at a 10-gigawatt scale with Microsoft and other partners through 2029.
Anthropic signed more than 12 US data center leases exceeding 1 gigawatt of computing capacity while preparing for its IPO, with Google reportedly in discussions to provide additional financial backing.
Anthropic is already paying SpaceX $1.25 billion per month for access to over 220,000 Nvidia processors at the Colossus 1 facility in Memphis, with the contract running through May 2029.
Broadcom now designs custom silicon for Google (TPUs), OpenAI (Jalapeño), Meta (MTIA accelerators), and ByteDance (in active negotiations), cementing its position as the kingmaker of custom AI silicon.
For Nvidia, the competitive threat from Broadcom-designed custom ASICs is real but bounded, as Nvidia’s H100 and B200 GPUs remain dominant for model training, but inference is where the daily bill lives.

Blogs We Published This Week

MiniCPM-V MCP Server: Give Your Agent Eyes

This guide shows how to connect MiniCPM-V with an MCP server to add vision capabilities to AI agents. Instead of processing only text, agents can understand screenshots, photos, diagrams, documents, and UI elements, enabling powerful multimodal workflows such as visual reasoning, document analysis, and desktop automation — all while running locally with open-source models.

Build a Private Photo Assistant on Telegram with OpenClaw + MiniCPM-V

Learn how to build a privacy-focused Telegram photo assistant using OpenClaw and MiniCPM-V 4.6. The assistant can analyze images, answer visual questions, describe scenes, extract text, and understand documents directly inside Telegram, while keeping your data under your control through a fully self-hosted AI pipeline.

Build a Private Photo Assistant on Telegram with OpenClaw + MiniCPM-V 4.6

OpenTaint: The Open-Source Taint Analysis Engine for the AI Era

This tutorial introduces OpenTaint , an open-source static taint analysis engine designed to detect insecure data flows in modern applications. The guide covers installation, scanning real projects, understanding taint analysis, and identifying security vulnerabilities before deployment, making it a practical tool for developers building secure AI-powered software.

OpenTaint: The Open-Source Taint Analysis Engine for the AI Era

OpenCode Agent Masterclass

This comprehensive tutorial walks developers through everything needed to master OpenCode , the open-source AI coding agent. It covers installation, connecting AI providers, configuring MCP servers, agent modes, permissions, skills, project rules, and productivity workflows, providing a complete guide to building an efficient AI-assisted development environment.

OpenCode Agent Masterclass — Full Tutorial

Thank you so much for reading

Like | Follow | Subscribe to the newsletter.

Catch us on

LinkedIn: https://www.linkedin.com/in/techlatest-net/

Link: https://techlatest.net/support/multi_llm_gpu_vm_support/

Reddit Community: https://www.reddit.com/user/techlatest_net/

OpenCode Agent Masterclass — Full Tutorial

TechLatest — Fri, 26 Jun 2026 14:03:25 +0000

Everything you need to install, configure, and master OpenCode — the open-source AI coding agent with 179k+ GitHub stars, available as a terminal TUI, desktop app, and IDE extension.

Official: opencode.ai · Docs: opencode.ai/docs · Source: github.com/anomalyco/opencode

What you’ll have at the end

OpenCode installed via curl, npm, or Homebrew
A provider connected (/connect) — Zen, Anthropic, OpenAI, Ollama, or Copilot login
Project AGENTS.md from /init, committed to git
Build and Plan agent modes mastered (Tab toggle)
MCP servers, rules, permissions, and skills configured
Confidence with /undo, /redo, /share, and @ file references

Introduction — the coding agent that meets you where you work

OpenCode is not another chat wrapper. It is a full agent harness for software work: file edits, shell execution, LSP-aware code intelligence, MCP tool servers, subagents, and multi-session parallelism — all from a terminal UI that also ships as a desktop app and VS Code extension.

The project’s pitch is direct: privacy-first , provider-agnostic (75+ models via Models.dev), and open source under MIT. OpenCode does not store your code or context on its servers when you use local or bring-your-own-key providers.

Community scale matters here: 179k+ stars, 900+ contributors, and a release cadence that ships weekly. That velocity shows up in the product — Plan/Build agents, Zen curated models, ACP support, and a plugin ecosystem documented under Develop → Ecosystem.

OpenCode harness stack — surfaces to the agent loop.

Part 1 — How OpenCode is structured

OpenCode wraps the model in a coding harness with distinct layers:

Surfaces — TUI (opencode), desktop app, IDE extension, web
Agents — build (full access) and plan (read-only analysis)
Tools — file ops, search, bash, web, code intelligence, subagent spawn
Configuration — rules, permissions, policies, formatters, keybinds
Extensions — MCP servers, Agent Skills, custom tools, LSP servers
Model routing — any provider with API keys or OAuth (Copilot, ChatGPT Plus)

The harness owns when the model can write files, which tools appear in context, and how sessions persist. The model owns reasoning inside that boundary.

Docs map: Tools · Rules · Agents · MCP servers · SDK

Part 2 — Prerequisites

Terminal: WezTerm, Alacritty, Ghostty, or Kitty recommended (docs)
macOS, Linux, or WSL — native Windows supported via Scoop/Chocolatey; WSL preferred
API key from your chosen provider, or OpenCode Zen subscription, or local Ollama
Git repo for the project you want to work in (for /init → AGENTS.md)

node -v # optional if using npm install path
which opencode # after install

Part 3 — Install

One-liner (recommended)

curl -fsSL https://opencode.ai/install | bash

Install path priority: $OPENCODE_INSTALL_DIR → $XDG_BIN_DIR → $HOME/bin → $HOME/.opencode/bin.

Package managers

npm install -g opencode-ai # or bun / pnpm / yarn
brew install anomalyco/tap/opencode # macOS/Linux — most up to date tap

Desktop app (beta)

Download from opencode.ai/download — macOS (ARM/x64), Windows, Linux (.deb / .rpm / AppImage).

Part 4 — Configure providers

Launch OpenCode and connect a model:

cd /path/to/your/project
opencode

Inside the TUI:

/connect

OpenCode Zen (recommended for beginners): curated models benchmarked for coding agents — sign in at opencode.ai/auth, paste API key.

Other providers: Anthropic, OpenAI, Google, Groq, local Ollama, and more. You can also log in with GitHub Copilot or ChatGPT Plus/Pro OAuth per the docs.

Tip: Zen removes the “which model actually works for agents?” guesswork. Self-hosters often pair OpenCode with Ollama for fully offline runs.

Run OpenCode with Local Models

If you prefer local AI instead of cloud APIs, TechLatest provides a GPU-ready DeepSeek & Llama environment supporting models like DeepSeek, Qwen, Llama, Gemma, and Mistral that can be connected directly to OpenCode through Ollama-compatible endpoints.

Want an OpenAI-compatible local API? Deploy LocalAI on TechLatest and connect OpenCode without relying on external providers.

Link: https://techlatest.net/support/local-ai-support/

Part 5 — Initialize the project (/init)

After /connect, initialize harness files:

/init

OpenCode analyzes your repository and writes AGENTS.md at the project root — project structure, conventions, and patterns the agent should respect.

Commit AGENTS.md to git. This is the same harness primitive we teach in Harness Engineering: a durable instruction map, not a one-off prompt.

See examples/AGENTS.md.example for the shape OpenCode generates.

# AGENTS.md — sample shape (OpenCode /init output)

> Generated by OpenCode `/init`. Commit this file. Keep it a **map** , not an encyclopedia.

## Project overview

Brief description of what this repository does and its primary language/framework.

## Directory layout

- `src/` — application source
- `tests/` — pytest (or your runner)
- `docs/` — deep architecture (agent reads on demand)

## Commands

bash
npm install
npm test
npm run lint


## Conventions

- Prefer small PR-sized changes
- Run tests before claiming done
- Do not edit `vendor/` or lockfiles unless asked

## Agent notes

- Read `docs/architecture.md` before large refactors
- Use Plan mode for unfamiliar modules

shell

Init project — /init creates AGENTS.md

Part 6 — Build vs Plan agents

OpenCode ships two first-class agents — switch with Tab:

Workflow OpenCode recommends:

Switch to Plan (Tab)
Describe the feature in detail — talk like you would to a junior dev
Iterate on the plan; drag-drop reference images into the terminal
Switch to Build (Tab)
Say: “Sounds good — go ahead and make the changes.”

Also available: @general subagent for complex multi-step search tasks (used internally; you can invoke explicitly).

Part 7 — Tools layer

Tools are the agent’s hands. OpenCode categories include:

File operations — read, write, patch
Search — ripgrep, glob, @ fuzzy file picker
Execution — bash with permission gates
Web — fetch docs and references
Code intelligence — LSP enabled — loads language servers automatically so the LLM gets symbols, diagnostics, and completions context
Subagents — @general and custom agents

The harness validates tool calls, enforces permissions, and formats observations back into the loop. See Configure → Tools and Permissions.

Part 8 — Rules, permissions, and policies

Rules — project-level instructions (often markdown) that shape agent behavior beyond AGENTS.md.

Permissions — per-tool allow/deny; Plan agent defaults to stricter bash gating.

Policies — organizational guardrails when using OpenCode in teams.

This is harness safety architecture : the model proposes, the runtime permits. Aligns with Harness Engineering Part 8 hooks and ratchets — mistakes become permanent constraints.

Example rule snippet: examples/rules.example.md

## Project rules (OpenCode)

Never commit secrets or .env files
Run npm test && npm run lint before finishing a task
Ask before deleting files or running destructive shell commands
Match existing code style in each directory

shell

Part 9 — MCP servers

OpenCode supports Model Context Protocol servers — the same standard covered in our MCP Visual Guide.

Wire MCP in OpenCode config so the agent can call external tools: databases, browsers, Stripe, vision servers, etc.

{
  "mcpServers": {
    "filesystem": {
      "command": "npx",
      "args": ["-y", "@modelcontextprotocol/server-filesystem", "/path/to/allowed"]
    }
  }
}

shell

See examples/opencode.mcp.example.json and Configure → MCP servers.

MCP + built-in tools — harness view

Build AI Coding Agents with Memory

Use Chroma Vector Database for semantic search, embeddings, and retrieval when creating AI-powered developer assistants with OpenCode.

Link: https://techlatest.net/support/chromadb_support/

Need Retrieval-Augmented Generation

Pair OpenCode with RAGFlow to let coding agents search internal documentation, APIs, and engineering knowledge bases.

Link: https://techlatest.net/support/ragflow_support/

Part 10 — Agent Skills

Agent Skills are packaged capability bundles — progressive disclosure, so the context window isn’t flooded with every tool definition at startup. OpenCode loads skill front-matter when relevant.

Skills complement MCP: MCP = wire protocol to servers; Skills = curated workflows and instructions the harness injects on demand.

Docs: Configure → Agent Skills

Part 11 — Daily usage patterns

Ask questions (explore codebase)

Use @ to fuzzy-search files:

How is authentication handled in @packages/functions/src/api/index.ts

shell

Add features (plan → build)

See Part 6. Give context, examples, and reference images (drag into terminal).

Direct changes (no plan phase)

For straightforward work:

We need auth on /settings. Mirror the pattern in @packages/functions/src/notes.ts
and apply it in @packages/functions/src/settings.ts

shell

Images in prompts

Drag and drop PNG/JPG into the TUI — OpenCode attaches them to the prompt (useful for UI mockups).

Ask with @ file reference

Part 12 — Undo, redo, share

Undo a bad edit:

/undo

shell

Restores files and returns your original message so you can refine the prompt. Run /undo multiple times for multi-step rollback.

Redo:

/redo

shell

Share session with team:

/share

shell

Creates a link to the current conversation (not shared by default). Useful for debugging agent loops with teammates.

Part 13 — Multi-session and LSP

OpenCode supports multiple parallel agent sessions on the same project — each with an isolated context. Great for splitting frontend/backend tasks.

LSP integration means the harness feeds semantic code intelligence to the model, so you don't have to paste types and diagnostics manually. This is a major differentiator vs raw chat UIs.

Part 14 — Surfaces beyond the terminal

The same harness runs behind each surface — models feel consistent because the tool layer is shared (Codex parallel: one harness, many clients).

Part 15 — SDK, Server, and Plugins

For builders extending OpenCode:

SDK — programmatic access to the agent harness
Server — run OpenCode as a service that other clients attach to
Plugins — extend behavior without forking core
Ecosystem — community projects (docs)

If you ship a derivative project, OpenCode asks you to clarify in README that it is not official — see upstream CONTRIBUTING.

Part 16 — OpenCode in the agent landscape

vs Claude Code: Both are coding harnesses. Claude Code is Anthropic-native with deep IDE integration; OpenCode is provider-neutral and terminal-first with open source transparency.

vs OpenClaw: OpenClaw is a personal assistant gateway (Telegram, WhatsApp, cron). OpenCode is a repo coding agent. Many teams use both — OpenClaw for life ops, OpenCode for shipping code. See OpenClaw masterclass

vs ZeroClaw: ZeroClaw is a Rust multi-channel runtime with security policy and hardware hooks. OpenCode optimizes for developer UX in git repos. See ZeroClaw masterclass.

Harness link: OpenCode embodies Harness Engineering — AGENTS.md, permissions, verification via LSP/tests, and scoped Plan mode.

Part 17 — Troubleshooting

Part 18 — Hands-on checklist

curl -fsSL https://opencode.ai/install | bash
cd your-repo && opencode
/connect
/init && git add AGENTS.md && git commit -m "Add OpenCode AGENTS.md"
# Tab → Plan → describe feature → Tab → Build → implement
/share # optional — link for teammate

Summary

OpenCode is a production-grade coding agent harness : provider-flexible, LSP-aware, MCP-ready, with Plan/Build agent modes and first-class AGENTS.md initialization. Install once, /connect, /init, and treat the TUI as mission control for your repo — not a chat sidebar.

The model writes the code. The harness decides what it’s allowed to touch, what it sees, and when “done” means done.

Thank you so much for reading

Like | Follow | Subscribe to the newsletter.

Catch us on

LinkedIn: https://www.linkedin.com/in/techlatest-net/

Reddit Community: https://www.reddit.com/user/techlatest_net/

OpenTaint: The Open-Source Taint Analysis Engine for the AI Era

TechLatest — Thu, 25 Jun 2026 10:28:08 +0000

AI now writes production code faster than any security team can review it. The two classes of tools we built to catch its mistakes each force a bad trade-off — and OpenTaint exists to break that trade-off. This is a complete, run-it-yourself guide: what the tool is, how taint analysis works, how to install and scan, four worked examples, and the AI agent workflows that let it scale.

TL;DR

Open-source, formal, inter-procedural taint analysis engine — an Apache-2.0 alternative to Semgrep Pro and CodeQL.
Tracks untrusted data across function boundaries, persistence layers, aliases, and async code.
Let's an LLM agent distill one finding into a deterministic rule — you pay the model once, not on every scan.
Replays that rule across your whole codebase, on every commit, at zero token cost.
Deepest coverage today for Java / Kotlin / Spring ; Python, Go, C#, JS & TS on the roadmap.

Note

BlackArch Linux

We also provide a ready-to-deploy BlackArch Linux VM that can be launched instantly on AWS , GCP , or Azure . No installation, setup, or dependency management required — just spin it up and start using a full arsenal of penetration testing and security auditing tools in minutes.

Kali GUI Linux

Our Kali GUI Linux VM comes fully pre-configured with a graphical interface, making it easy for both beginners and professionals to get started. Deploy directly on AWS , GCP , or Azure with zero setup — no installation hassles, just immediate access to a complete offensive security toolkit.

Browser-Based Kali Linux

We offer a browser-based Kali Linux environment that runs entirely in the cloud. Simply deploy and access it from your browser — no downloads, no local setup, no compatibility issues. Deploy directly on AWS , GCP , or Azure with zero setup — no installation hassles, just immediate access to a complete offensive security toolkit. Perfect for quick testing, learning, and remote security operations from anywhere.

ParrotOS Linux

Our ParrotOS Linux VM is optimized for security, privacy, and development workflows. Available for instant deployment on AWS , GCP , and Azure , it eliminates the need for manual installation — giving you a secure, ready-to-use environment in just a few clicks.

1. What is OpenTaint?

OpenTaint is a static application security testing (SAST) engine focused on one thing done extremely well: tracking the flow of untrusted data through your program. It is written in Go, self-hostable, and ships as one stack — engine, rules, and CI integrations — under Apache 2.0 (engine) and MIT (CLI, GitHub Action, GitLab template, rules).

Three sentences capture the whole pitch:

It finds what pattern matchers miss. A formal inter-procedural dataflow engine follows tainted input even when it crosses functions, gets stored and re-read, or is aliased.
It pays the model once, not on every scan. An AI agent turns a single confirmed vulnerability into a reusable taint rule; the deterministic engine then replays it in minutes of CPU.
It is battery-included and open. Today, it is the most thorough taint engine for Java / Kotlin / Spring , with Python, Go, C#, JavaScript, and TypeScript on the roadmap.

2. Taint analysis in 60 seconds

“Taint” is a label. You mark certain inputs as untrusted (a source ), and certain dangerous operations as things that must never receive untrusted data (a sink ). The engine then asks a single question: Can data flow from any source to any sink? If yes — and nothing along the way neutralizes it (a sanitizer ) — that path is a vulnerability.

SOURCE PROPAGATION SINK
req.getParameter() → service.process(x) → statement.execute(sql)
(untrusted input) (taint flows across (dangerous op:
                        function boundaries) SQL injection!)

         ▲ a sanitizer anywhere on this path (e.g. a parameterized
           query / encoder) would "clean" the taint and kill the finding

The hard part is propagation. Naive tools only look at a single line or function (intra-procedural). Real bugs span many hops: controller → service → repository → ORM. Following taint across all those hops is inter-procedural analysis, and it is exactly what OpenTaint specializes in.

3. Why it matters — the AI-era gap

There are two existing families of tools, and each makes one painful compromise:

The deep, inter-procedural analysis that actually catches data-flow bugs has historically been locked inside proprietary tools. OpenTaint opens it up and makes the LLM a one-time rule author rather than a per-scan expense.

4. How OpenTaint works under the hood

The pipeline, conceptually, is four stages:

Build & parse. OpenTaint understands your build so it can resolve types, dependencies, and method targets — this is what enables real inter-procedural reasoning rather than guessing.
Model the graph. It constructs a call graph and dataflow graph, including aliasing and (for Spring) framework-specific wiring like controllers, request mappings, and beans.
Apply rules. Rules declare sources, sinks, sanitizers, and library “pass-through” approximations (how data moves through code it can’t see, e.g., third-party libs).
Solve & report. The engine computes reachable source→sink paths and emits results, typically as SARIF, so any IDE or CI can consume them.

Why SARIF matters

SARIF is the standard interchange format for static analysis. Because OpenTaint emits SARIF, findings render natively in GitHub code scanning, VS Code, and most security dashboards — no custom glue required.

5. Installation — every method

Pick whichever fits your environment. All commands below are taken from the official quick-start.

Install script (Linux / macOS)

The installer detects your platform, verifies a checksum, and drops the binary in ~/.opentaint/install, and prints the PATH line to add to your shell profile:

curl -fsSL https://opentaint.org/install.sh | bash

Homebrew (Linux / macOS)

brew install --cask seqra/tap/opentaint

Windows (PowerShell)

irm https://opentaint.org/install.ps1 | iex

npm (cross-platform)

# global install
npm install -g @seqra/opentaint

# or run instantly, no install (needs Node.js)
npx @seqra/opentaint scan

Verify before piping to a shell

Piping a remote script straight into bash or It iex is convenient but executes whatever the server returns. On shared or production machines, download the script first, read it, then run it — or prefer Homebrew/npm, which are checksum-backed.

6. Set up a test project & verify

Scaffold a sample Spring app to test against

No project handy? Spin up a throwaway Spring Boot app straight from start.spring.io, unzip it, and boot it once to confirm your toolchain (Java 21 + the Maven wrapper). This gives OpenTaint something real to chew on:

curl https://start.spring.io/starter.zip \
  -d type=maven-project -d language=java -d bootVersion=3.5.4 \
  -d baseDir=spring-app -d groupId=com.example -d artifactId=spring-app \
  -d name=spring-app -d packageName=com.example.springapp \
  -d javaVersion=21 -d dependencies=web -o spring-app.zip

unzip spring-app.zip && rm spring-app.zip
cd spring-app
chmod +x mvnw # the wrapper ships without the exec bit
./mvnw spring-boot:run # downloads Maven + Boot, then starts Tomcat on :8080

A couple of gotchas (shown above)

Run ./mvnw from inside the project (not the parent folder), and the wrapper needs to be chmod +x mvnw first. You only need java to install the wrapper that fetches Maven for you, so a missing mvn is fine. Stop the app with Ctrl-COpenTaint analyzes the source, so it never needs the app actually to run.

Verify OpenTaint

Confirm the binary is on your PATH and skim the available commands — OpenTaint is a Cobra-style CLI with subcommands for compiling, pulling artifacts, scanning, and summarizing SARIF:

opentaint --help
opentaint --version

7. Your first scan

From inside the project, just run scan. OpenTaint compiles a project model , runs the analyzer, applies the bundled ruleset, and writes a SARIF report into ~/.opentaint/cache, and prints a clean box-drawing summary:

opentaint scan

A fresh, hello-world scaffold has nothing dangerous in it, so the first scan reports 0 findings — exactly what you want to confirm the toolchain end-to-end before you introduce real code:

╭─OpenTaint Compile and Scan─╮
╰─┬──────────────────────────╯
  ├─ Project └─ ~/projects/spring-app
  ├─ Project model └─ ~/.opentaint/cache/spring-app-cba2bab8/project-model
  ├─ Autobuilder └─ custom (~/.opentaint/install/lib/opentaint-project-auto-builder.jar)
  ├─ Analyzer └─ custom (~/.opentaint/install/lib/opentaint-project-analyzer.jar)
  └─ Bundled ruleset└─ custom (~/.opentaint/install/lib/rules)

✓ Compiling project model in 15s
✓ Analyzing project in 17s

╭─Scan Summary─╮
╰─┬────────────╯
  ├─ Findings
  │ ├─ Total: 0 findings
  │ ├─ Files affected: 0
  │ ├─ Rules executed: 78
  │ └─ Rules triggered: 0
  └─ Output
     ├─ Report: …/project-model/sources/opentaint.sarif
     └─ Log: ~/.opentaint/logs/spring-app-cba2bab8/…log

The scan doesn’t dump findings inline — it writes a SARIF report and points you at the summary subcommand. Pass --show-findings to print each finding (here, none yet):

opentaint summary ~/.opentaint/cache/spring-app-cba2bab8/project-model/sources/opentaint.sarif --show-findings

8. Hands-on examples

Four progressively richer examples, from a vulnerable Spring endpoint to a custom rule and CI output.

Example 1 — Catching cross-function SQL injection

Here is the kind of bug AST matchers routinely miss, because the source and sink live in different files and the taint flows through a service layer:

@RestController
public class UserController {
    private final UserService service;

    @GetMapping("/users/search")
    public List<User> search(@RequestParam String name) {
        // 'name' is untrusted — this is the SOURCE
        return service.find(name);
    }
}

@Repository
public class UserRepository {
    private final DataSource ds;

    public List<User> rawByName(String name) throws SQLException {
        String sql = "SELECT * FROM users WHERE name = '" + name + "'";
        try (Statement st = ds.getConnection().createStatement()) {
            // tainted 'name' reaches a raw query — this is the SINK
            return map(st.executeQuery(sql));
        }
    }
}

A line-by-line linter sees nothing wrong in the controller, and the repository “just builds a string.” OpenTaint connects @RequestParam name → service.find → rawByName → executeQuery and flags the full path. The fix it nudges you toward is a parameterized query, which acts as a sanitizer and removes the finding on the next scan.

Example 2 — Writing a custom taint rule

The real power move: teach OpenTaint about your code. Suppose you have an internal helper Http.body() that returns raw request bodies, and a logging sink AuditLog.write() that must never receive PII. You declare them as a source and sink:

rules:
  - id: pii-into-auditlog
    severity: high
    message: "Untrusted request body flows into AuditLog (PII leak risk)"
    sources:
      - method: "com.acme.http.Http#body()"
    sinks:
      - method: "com.acme.audit.AuditLog#write(java.lang.String)"
        taint-arg: 0
    sanitizers:
      - method: "com.acme.privacy.Redactor#redact(java.lang.String)"

# run a scan with your custom rules directory added
opentaint scan --rules ./rules .

This is the cost-saving loop

You (or an agent) write this rule once. From then on, every scan and every future commit is checked against it deterministically — no LLM tokens spent re-discovering the same class of bug.

Example 3 — SARIF output + opening results

Produce machine-readable output and feed it anywhere that speaks SARIF:

opentaint scan --output results.sarif .

# count findings by severity with jq
jq '[.runs[].results[].level] | group_by(.) | map({(.[0]): length}) | add' results.sarif

Open results.sarif in VS Code (with the SARIF Viewer extension) or upload it to GitHub code scanning to get inline annotations on the exact source and sink lines.

Example 4 — Zero-install scan with Docker

No local toolchain? Mount your project and run the published image:

docker run --rm \
  -v $(pwd):/project \
  -v $(pwd):/output \
  ghcr.io/seqra/opentaint:latest \
  opentaint scan --output /output/results.sarif /project

This is the most reproducible option — identical engine version every time — which makes it ideal for CI runners and for sharing a scan setup across a team.

9. AI agent workflows

OpenTaint ships agent skills that turn static analysis into an end-to-end AppSec workflow. Install them with:

npx skills add https://github.com/seqra/opentaint

The headline skill, appsec-agent , orchestrates a full project assessment: build the project, run OpenTaint, discover the attack surface, add targeted rules, model missing library data flows, triage findings, and optionally generate dynamic proof-of-concept checks for confirmed vulnerabilities.

The included skills map cleanly onto the security-analysis loop:

Scan & triage
▹ build-project
▹ run-scan
▹ analyze-findings
▹ generate-poc

Coverage expansion
▹ triage-dependencies
▹ discover-attack-surface
▹ create-test-project
▹ create-rule
▹ assemble-lib-rules

Dataflow modeling
▹ analyze-external-methods
▹ create-pass-through-approximation
▹ create-dataflow-approximation
▹ debug-rule
▹ report-analyzer-issue

The pattern that makes this scale

The agent does the creative, one-time work — understanding your attack surface and authoring rules and library approximations. The deterministic engine does the repetitive, forever work — replaying those rules on every commit. That division is the entire economic argument for OpenTaint.

10. CI/CD integration

Because OpenTaint emits SARIF and ships a GitHub Action + GitLab template, wiring it into a pipeline is short. A minimal GitHub Actions job:

name: OpenTaint
on: [push, pull_request]

jobs:
  taint-scan:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Run OpenTaint
        run: |
          curl -fsSL https://opentaint.org/install.sh | bash
          opentaint scan --output results.sarif .
      - name: Upload SARIF to code scanning
        uses: github/codeql-action/upload-sarif@v3
        with:
          sarif_file: results.sarif

Now every pull request gets inline taint findings, and your once-authored rules guard the codebase on every commit — minutes of CPU, zero tokens. For the official, supported configuration, always check the docs.

11. Pro tips

Start with Java/Spring. It has the deepest coverage today; other languages are on the roadmap.
Make sure the project builds. Inter-procedural accuracy depends on type and dependency resolution — a broken build means a shallower graph.
Model your libraries. If taint “disappears” inside a third-party call, add a pass-through approximation so the engine knows data flows through it.
Tune sanitizers to cut noise. Declaring your real encoders/validators as sanitizers removes false positives cleanly.
Commit your rules. Treat custom rules as code — they are your team’s accumulated security knowledge, replayable forever.
Pin the Docker tag in CI for reproducible scans instead of latest.

Use only on the code you’re authorized to test

Run OpenTaint against repositories you own or are explicitly authorized to assess. Treat any findings — and the proof-of-concept artifacts agents may generate — as sensitive, and never paste secrets, live credentials, or client data into rules or issues.

12. Wrap-up

OpenTaint reframes the AI-vs-static-analysis debate: instead of paying a language model to re-read your code on every scan, you pay it once to understand a vulnerability, capture that understanding as a deterministic taint rule, and let a fast engine enforce it forever. You get the depth of an inter-procedural agent at the cost of a static analyzer — open source, self-hostable, and batteries-included.

Recap of the commands worth memorizing:

# install (pick one)
curl -fsSL https://opentaint.org/install.sh | bash
brew install --cask seqra/tap/opentaint
npm install -g @seqra/opentaint

# scan
opentaint scan # console
opentaint scan --output results.sarif . # SARIF report
opentaint scan --rules ./rules . # with custom rules

# docker (zero install)
docker run --rm -v $(pwd):/project -v $(pwd):/output \
  ghcr.io/seqra/opentaint:latest \
  opentaint scan --output /output/results.sarif /project

# AI agent skills
npx skills add https://github.com/seqra/opentaint

Next steps: read the official OpenTaint repository and documentation, run it on a real Spring app, and write your first custom rule. Then add it to your own AegisMind vault ai-security-tools/ so future-you can replay the workflow.

Thank you so much for reading

Like | Follow | Subscribe to the newsletter.

Catch us on

Newsletter: https://substack.com/@techlatest

LinkedIn: https://www.linkedin.com/in/techlatest-net/

Reddit Community: https://www.reddit.com/user/techlatest_net/

Build a Private Photo Assistant on Telegram with OpenClaw + MiniCPM-V 4.6

TechLatest — Tue, 23 Jun 2026 09:52:18 +0000

Most AI assistants can read text, write code, and automate workflows. But the moment a user sends a photo — a receipt, a whiteboard snapshot, a product image, or a screenshot — the automation often stops unless you connect a cloud vision API.

That usually means API keys, usage costs, network latency, and private images leaving your machine.

In this guide, you’ll build a private photo assistant that runs entirely on your own infrastructure using OpenClaw and MiniCPM-V 4.6. The assistant can receive images from Telegram, WhatsApp, or a local CLI, analyze them with a lightweight multimodal model running on Ollama, and return structured answers with summaries, OCR results, and suggested replies.

At the center of the workflow is MiniCPM-V 4.6, a compact vision-language model with approximately 1.3 billion parameters and a footprint of around 1.6 GB. By exposing it through a LitServe API and connecting it to OpenClaw through a custom vision-photo skill, you create a reusable vision capability that any OpenClaw agent can invoke on demand.

By the end of this tutorial, you’ll have a fully functional photo assistant that can:

Describe photos and screenshots
Extract text from receipts, invoices, and documents
Answer questions about uploaded images
Generate structured markdown responses
Work across Telegram, WhatsApp, and local workflows
Keep image processing completely private and local

No cloud vision APIs. No external image uploads. Just OpenClaw, MiniCPM-V 4.6, and your own infrastructure.

What you end up with

OpenClaw Gateway — always-on control plane
minicpm-v4.6 — conversational + vision model (~1.6 GB)
vision-photo skill — vision_query.sh → POST /predict on port 8002
Structured markdown replies — summary, details, OCR text, suggested channel message

Flow

User sends a photo on Telegram, WhatsApp, or CLI
MiniCPM-V 4.6 plans and invokes the vision-photo skill
Skill POSTs to LitServe http://127.0.0.1:8002/predict
Structured answer returns to the same channel

Prerequisites

Requirement | Check
Node 22.12+ | node -v
Ollama | ollama -v
Python3.10+ | python3 --version
curl + jq | curl --version && jq --version

Part 1 — Pull MiniCPM-V 4.6

From Ollama:

ollama pull minicpm-v4.6
ollama run minicpm-v4.6 "Hello" --image ./photo.jpg

Tag | Size | Input

minicpm-v4.6:latest | 1.6 GB | Text, Image

Don’t want to spend time installing Ollama manually? TechLatest’s Open WebUI + Ollama deployment provides a pre-configured environment for running MiniCPM-V, Gemma, Qwen, Llama, and DeepSeek models locally. Simply launch the instance and start building multimodal AI workflows without additional setup.

Link: https://techlatest.net/support/multi_llm_gpu_vm_support/

Link: https://techlatest.net/support/multi_llm_vm_support/

Part 2 — Vision LitServe API

Terminal A — start the vision server:

cd openclaw-minicpm-v
python -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt
.env
python generate_sample.py
python vision_server.py

generate_sample.py

#!/usr/bin/env python3
from pathlib import Path
from PIL import Image, ImageDraw, ImageFont

out = Path( __file__ ).resolve().parent / "samples"
out.mkdir(exist_ok=True)
img = Image.new("RGB", (480, 640), "#fafafa")
d = ImageDraw.Draw(img)
try:
    f = ImageFont.truetype("/System/Library/Fonts/Supplemental/Arial.ttf", 18)
except OSError:
    f = ImageFont.load_default()
d.text((40, 40), "COFFEE BEAN Co.", fill="#111", font=f)
d.text((40, 100), "TOTAL $10.75", fill="#222", font=f)
d.text((40, 140), "2026-06-23", fill="#555", font=f)
img.save(out / "receipt.png")
print(f"Wrote {out / 'receipt.png'}")


python vision_server.py

"""LitServe vision API — MiniCPM-V 4.6 photo understanding for OpenClaw."""
from __future__ import annotations

import os
from pathlib import Path

import litserve as ls
from dotenv import load_dotenv

import vision_backend as vb

load_dotenv()

PORT = int(os.getenv("PORT", "8002"))

STRUCTURED_PROMPT = """Analyze the image and answer the user's question.
Return markdown with these sections when relevant:
## Summary
(one paragraph)

## Details
(bullet points)

## Text found
(any visible text, or "none")

## Suggested reply
(a short message suitable for Telegram/WhatsApp)
"""

class VisionPhotoAPI(ls.LitAPI):
    def setup(self, device):
        self.model = vb.VISION_MODEL

    def decode_request(self, request):
        return {
            "query": (request.get("query") or "What is in this photo?").strip(),
            "image_path": (request.get("image_path") or "").strip(),
        }

    def predict(self, inputs):
        path = Path(inputs["image_path"]).expanduser().resolve()
        prompt = f"{STRUCTURED_PROMPT}\n\nUser question: {inputs['query']}"
        try:
            answer = vb.chat_vision(prompt, [path])
            return {"output": answer, "model": self.model, "image_path": str(path)}
        except vb.VisionError as exc:
            return {"error": str(exc), "model": self.model}

    def encode_response(self, output):
        return output

if __name__ == " __main__":
    server = ls.LitServer(VisionPhotoAPI(), accelerator="auto", timeout=False)
    print(f"Vision API on http://127.0.0.1:{PORT}/predict (model: {vb.VISION_MODEL})")
    server.run(port=PORT)

Server prints: Vision API on http://127.0.0.1:8002/predict

Request shape:

{
  "query": "What is the total on this receipt?",
  "image_path": "/absolute/path/to/receipt.png"
}

Response:

{
  "output": "## Summary\n…",
  "model": "minicpm-v4.6",
  "image_path": "…"
}

Sample image the API reads:

Test with client.py:

python client.py --image samples/receipt.png --query "OCR this receipt"

client.py

#!/usr/bin/env python3
"""CLI client for the vision photo API."""
from __future__ import annotations

import argparse
import json
import os
import urllib.request

DEFAULT_URL = os.environ.get("VISION_API_URL", "http://127.0.0.1:8002")

def main() -> None:
    p = argparse.ArgumentParser(description="Query local MiniCPM-V vision API")
    p.add_argument("--image", required=True, help="Path to image file")
    p.add_argument("--query", default="Describe this photo in detail.")
    p.add_argument("--url", default=f"{DEFAULT_URL.rstrip('/')}/predict")
    args = p.parse_args()

    body = json.dumps({"query": args.query, "image_path": args.image}).encode()
    req = urllib.request.Request(args.url, data=body, headers={"Content-Type": "application/json"})
    with urllib.request.urlopen(req, timeout=180) as resp:
        data = json.loads(resp.read().decode())
    print(data.get("output") or data.get("error") or data)

if __name__ == " __main__":
    main()

Expected sections in the output: Summary , Details , Text found , Suggested reply.

Part 3 — Install OpenClaw

Terminal B:

cd guides/openclaw-minicpm-v
source ./use-node22.sh
npm install -g openclaw@latest
openclaw onboard --install-daemon
openclaw models set ollama/minicpm-v4.6

#!/usr/bin/env bash
set -euo pipefail
export NVM_DIR="${NVM_DIR:-$HOME/.nvm}"
if [[-s "$NVM_DIR/nvm.sh"]]; then
  . "$NVM_DIR/nvm.sh"
  nvm use "$(cat "$(dirname "$0")/.nvmrc")"
else
  echo "nvm not found — install Node 22+" >&2
  exit 1
fi
echo "Node: $(node -v)"

In openclaw.json sets the primary model and VISION_API_URL.

// Merge into ~/.openclaw/openclaw.json
{
  agents: {
    defaults: {
      model: { primary: "ollama/minicpm-v4.6" },
      skills: ["vision-photo"],
    },
  },

  models: {
    providers: {
      ollama: {
        apiKey: "ollama-local",
        baseUrl: "http://127.0.0.1:11434",
        api: "ollama",
        timeoutSeconds: 300,
        models: [
          {
            id: "minicpm-v4.6",
            name: "MiniCPM-V 4.6",
            reasoning: false,
            input: ["text", "image"],
            contextWindow: 256000,
            maxTokens: 8192,
            params: { keep_alive: "15m" },
          },
        ],
      },
    },
  },

  skills: {
    entries: {
      "vision-photo": {
        enabled: true,
        env: {
          VISION_API_URL: "http://127.0.0.1:8002",
        },
      },
    },
  },
}

Looking for a faster setup? TechLatest offers a pre-configured OpenClaw environment that includes the gateway, agent runtime, and common dependencies, allowing developers to focus on building skills and automations instead of infrastructure setup.

Link: https://techlatest.net/support/openclaw-support/

Part 4 — Install vision-photo skill

chmod +x install-skill.sh skills/vision-photo/scripts/*.sh
./install-skill.sh
openclaw gateway restart

The skill tells the agent to run:

vision_query.sh "/path/to/image.jpg" "user question"

See skills/vision-photo/SKILL.md.

Part 5 — Telegram / WhatsApp

Follow OpenClaw channels docs for your platform. Keep DM pairing enabled for security.

When a user sends a photo:

OpenClaw saves media to a local path
Agent invokes vision-photo with path + caption
LitServe returns structured markdown
Agent sends suggested reply to the channel

Example channel reply from the demo receipt:

Your receipt total is _ **$10.75_**

Part 6 — Smoke test

./test-local.sh

Runs: Ollama check → sample image → API health → skill script query.

For a step-by-step walkthrough and complete implementation details, check out the full guide here.

Deploy the Complete Stack on TechLatest

You can deploy the entire private photo assistant stack using TechLatest AI infrastructure:

Open WebUI + Ollama for local multimodal inference
OpenClaw for agent orchestration and messaging integrations
JupyterHub for experimentation and evaluation
AWS, Azure, and GCP deployment options

This allows developers to build privacy-first vision assistants without spending hours configuring infrastructure and dependencies.

Conclusion

You’ve successfully built a private multimodal assistant that can see, read, and understand images directly from your messaging channels.

Using OpenClaw as the orchestration layer, MiniCPM-V 4.6 as the vision model, and LitServe as the local inference API, you’ve created a workflow where users can send a photo and receive structured, actionable insights without relying on external vision services.

The architecture is intentionally simple:

OpenClaw handles agent orchestration and channel integrations
MiniCPM-V 4.6 provides image understanding and OCR capabilities
LitServe exposes a lightweight local API
The vision-photo skill connects everything

Because the entire stack runs locally, sensitive screenshots, receipts, documents, and personal photos never leave your infrastructure. Whether you’re building customer support agents, document processing workflows, field inspection tools, or personal AI assistants, the same pattern can be extended with additional skills and automation.

From a single image to a complete conversation, your OpenClaw agent now has eyes.

Thank you so much for reading

Like | Follow | Subscribe to the newsletter.

Catch us on

LinkedIn: https://www.linkedin.com/in/techlatest-net/

Reddit Community: https://www.reddit.com/user/techlatest_net/

MiniCPM-V MCP Server — Give Your Agent Eyes

TechLatest — Tue, 23 Jun 2026 08:48:21 +0000

Build an MCP server that exposes describe_image, ocr_document, and compare_images so any MCP host — Cursor, Claude Desktop, Hermes — can understand screenshots, receipts, and UI diffs through one protocol.

Powered by MiniCPM-V 4.6 via Ollama: 1.3B params · 1.6 GB · text + image · 256K context — the smallest vision model in the MiniCPM family, tuned for edge and phone deployment.

The moment: paste a screenshot into Cursor and ask “What changed in this UI?” The agent calls compare_images — no cloud vision API, no API key.

What you’ll learn

Vision as MCP tools — why describe_image / ocr_document / compare_images beat one-off scripts
One protocol, many hosts — same server in Cursor, Claude Desktop, and Hermes
MiniCPM-V 4.6 on Ollama — pull, run, and wire the 1.6 GB multimodal model locally
Private document OCR — extract receipt and whiteboard text without sending pixels to the cloud
Before/after UI diffs — compare two screenshots for regression review
Runnable Python MCP server + an end-to-end agent demo (works offline with OLLAMA_MOCK=1)

Introduction — agents without eyes

Your agent can grep code, run tests, and provision infra — but the moment someone pastes a screenshot , a receipt , or a Figma export , the loop breaks unless you bolt on a vision API. That means API keys, cloud latency, and pixels leaving your machine.

MiniCPM-V 4.6 is built for the opposite: 1.3B parameters , ~1.6 GB on disk, 256K context , and native text + image input via Ollama. Wrap it in MCP and every host discovers the same three vision tools at connect time.

Part 1 — MiniCPM-V 4.6 on Ollama

From the Ollama model page:

ollama pull minicpm-v4.6
ollama run minicpm-v4.6 "Describe this image" --image ./photo.jpg

This guide uses the same model through Ollama’s HTTP API so the MCP server can batch tool calls without spawning a CLI per request.

If you want a ready-to-use environment, TechLatest offers preconfigured Ollama and Open WebUI deployments that allow developers to run MiniCPM-V, Gemma, Qwen, Llama, and DeepSeek models locally without spending time on installation and configuration.

Link: https://techlatest.net/support/multi_llm_gpu_vm_support/

Link: https://techlatest.net/support/multi_llm_vm_support/

Part 2 — Why MCP for vision

You could write a Python script that calls Ollama and paste output into chat. But then Cursor, Claude Desktop, and Hermes each need their own glue.

MCP collapses that. You write one server ; hosts discover tools at capability exchange. Add a fourth tool later, and every host sees it on reconnect — no host-side changes.

If host/client/server isn’t second nature yet, read the MCP Visual Guide first.

Vision becomes even more powerful when connected to agent workflows. Developers can deploy TechLatest’s CrewAI Studio VM and create multi-agent systems where one agent performs OCR, another analyzes screenshots, and a third generates reports. The same MiniCPM-V MCP server can be shared across all agents through MCP, creating a reusable vision layer for complex automation workflows.

Link: https://techlatest.net/support/crewai-support/

Part 3 — Quick start

cd guides/minicpm-v-mcp-server
python -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt
cp .env.example .env

ollama pull minicpm-v4.6
python examples/generate_fixtures.py
python examples/agent_demo.py

generate_fixtures.py

#!/usr/bin/env python3
"""Generate sample images for MCP demos (no external assets required)."""
from __future__ import annotations

from pathlib import Path

from PIL import Image, ImageDraw, ImageFont

FIXTURES = Path( __file__ ).resolve().parent / "fixtures"

def _font(size: int):
    for name in ("DejaVuSans.ttf", "/System/Library/Fonts/Supplemental/Arial.ttf"):
        try:
            return ImageFont.truetype(name, size)
        except OSError:
            continue
    return ImageFont.load_default()

def receipt() -> None:
    img = Image.new("RGB", (480, 640), "#fafafa")
    d = ImageDraw.Draw(img)
    f, fs = _font(22), _font(16)
    d.text((40, 40), "COFFEE BEAN Co.", fill="#1a1a1a", font=f)
    d.text((40, 90), "123 Main St · San Francisco", fill="#555", font=fs)
    lines = [
        "Latte (Oat) $5.50",
        "Croissant $4.25",
        "Tip $1.00",
        "─────────────────────────",
        "TOTAL $10.75",
        "Card **** 4242",
        "2026-06-23 09:14 AM",
    ]
    y = 150
    for line in lines:
        d.text((40, y), line, fill="#222", font=fs)
        y += 36
    img.save(FIXTURES / "sample_receipt.png")

def diagram_v1() -> None:
    img = Image.new("RGB", (640, 400), "#0f172a")
    d = ImageDraw.Draw(img)
    f = _font(18)
    d.rounded_rectangle((40, 80, 200, 160), radius=12, fill="#1e3a5f", outline="#38bdf8")
    d.text((70, 110), "API", fill="#e2e8f0", font=f)
    d.rounded_rectangle((420, 80, 580, 160), radius=12, fill="#14532d", outline="#4ade80")
    d.text((440, 110), "Qdrant", fill="#e2e8f0", font=f)
    d.line((200, 120, 420, 120), fill="#94a3b8", width=3)
    d.text((250, 200), "v1 — sync pipeline", fill="#94a3b8", font=f)
    img.save(FIXTURES / "diagram_v1.png")

def diagram_v2() -> None:
    img = Image.new("RGB", (640, 400), "#0f172a")
    d = ImageDraw.Draw(img)
    f = _font(18)
    d.rounded_rectangle((40, 80, 200, 160), radius=12, fill="#1e3a5f", outline="#38bdf8")
    d.text((60, 110), "LitServe", fill="#e2e8f0", font=f)
    d.rounded_rectangle((250, 60, 410, 140), radius=12, fill="#4c1d95", outline="#a78bfa")
    d.text((270, 90), "CrewAI", fill="#e2e8f0", font=f)
    d.rounded_rectangle((420, 80, 580, 160), radius=12, fill="#14532d", outline="#4ade80")
    d.text((440, 110), "Qdrant", fill="#e2e8f0", font=f)
    d.line((200, 120, 250, 100), fill="#94a3b8", width=3)
    d.line((410, 100, 420, 120), fill="#94a3b8", width=3)
    d.text((200, 200), "v2 — agentic pipeline", fill="#fbbf24", font=f)
    img.save(FIXTURES / "diagram_v2.png")

def main() -> None:
    FIXTURES.mkdir(parents=True, exist_ok=True)
    receipt()
    diagram_v1()
    diagram_v2()
    print(f"Fixtures written to {FIXTURES}")

if __name__ == " __main__":
    main()

agent_demo.py

#!/usr/bin/env python3
"""End-to-end demo — MiniCPM-V MCP vision tools (works offline with OLLAMA_MOCK=1).

Simulates what Cursor / Claude Desktop sees when the agent calls vision tools.
"""
from __future__ import annotations

import json
import os
import sys
from pathlib import Path

ROOT = Path( __file__ ).resolve().parent
sys.path.insert(0, str(ROOT))

import vision_backend as vb # noqa: E402
from server import compare_images, describe_image, ocr_document # noqa: E402

FIXTURES = ROOT / "fixtures"

def _banner(title: str) -> None:
    print(f"\n{'=' * 60}\n {title}\n{'=' * 60}")

def _parse(result: str) -> dict:
    return json.loads(result)

def main() -> None:
    os.environ.setdefault("OLLAMA_VISION_MODEL", "minicpm-v4.6")
    if not FIXTURES.exists() or not list(FIXTURES.glob("*.png")):
        from generate_fixtures import main as gen # noqa: E402

        gen()

    status = vb.health_check()
    _banner("MiniCPM-V 4.6 Vision MCP — Agent Demo")
    print(f"Model: {vb.VISION_MODEL} · Mode: {status.get('mode', '?')}")
    if not status.get("ok") and not vb.MOCK:
        print("\n⚠️ Ollama offline — re-run with OLLAMA_MOCK=1 or start Ollama.\n")

    # Scenario 1 — describe screenshot
    _banner("Scenario 1 — describe_image")
    print('[Tool: describe_image] path=fixtures/diagram_v2.png')
    r1 = _parse(describe_image(str(FIXTURES / "diagram_v2.png"), "What services are shown?"))
    print("\n## Architecture summary\n")
    print(r1.get("result", r1.get("error", r1)))

    # Scenario 2 — OCR receipt
    _banner("Scenario 2 — ocr_document")
    print('[Tool: ocr_document] path=fixtures/sample_receipt.png')
    r2 = _parse(ocr_document(str(FIXTURES / "sample_receipt.png")))
    print("\n## Receipt OCR\n")
    print(r2.get("result", r2.get("error", r2)))

    # Scenario 3 — compare before/after
    _banner("Scenario 3 — compare_images")
    print("[Tool: compare_images] v1 → v2 pipeline diagrams")
    r3 = _parse(
        compare_images(
            str(FIXTURES / "diagram_v1.png"),
            str(FIXTURES / "diagram_v2.png"),
            focus="new components and labels",
        )
    )
    print("\n## Visual diff\n")
    print(r3.get("result", r3.get("error", r3)))

    _banner("Done — wire examples/server.py into Cursor MCP settings")
    print("See examples/cursor_mcp.json.example")

if __name__ == " __main__":
    main()

Part 4 — The vision backend

examples/vision_backend.py encodes images as base64 and POSTs to OLLAMA_HOST/api/chat:

payload = {
    "model": VISION_MODEL, # minicpm-v4.6
    "messages": [{"role": "user", "content": prompt, "images": images_b64}],
    "stream": False,
}

vision_backend.py

"""Ollama vision backend for MiniCPM-V 4.6 — shared by MCP server and demos."""
from __future__ import annotations

import base64
import os
from pathlib import Path

import httpx

OLLAMA_HOST = os.environ.get("OLLAMA_HOST", "http://127.0.0.1:11434").rstrip("/")
VISION_MODEL = os.environ.get("OLLAMA_VISION_MODEL", "minicpm-v4.6")
MOCK = os.environ.get("OLLAMA_MOCK", "0") == "1"

SUPPORTED_SUFFIXES = {".png", ".jpg", ".jpeg", ".webp", ".gif", ".bmp"}

class VisionError(Exception):
    pass

def _encode_image(path: Path) -> str:
    if not path.is_file():
        raise VisionError(f"Image not found: {path}")
    if path.suffix.lower() not in SUPPORTED_SUFFIXES:
        raise VisionError(f"Unsupported image type: {path.suffix}")
    return base64.b64encode(path.read_bytes()).decode("ascii")

def _mock_response(prompt: str, image_count: int) -> str:
    return (
        f"[mock {VISION_MODEL}] Processed {image_count} image(s).\n"
        f"Prompt preview: {prompt[:120]}…\n"
        "Set OLLAMA_MOCK=0 and run `ollama pull minicpm-v4.6` for live inference."
    )

def chat_vision(prompt: str, image_paths: list[Path], *, timeout: float = 120.0) -> str:
    """Send a vision chat request to Ollama."""
    if MOCK:
        return _mock_response(prompt, len(image_paths))

    images_b64 = [_encode_image(p) for p in image_paths]
    payload = {
        "model": VISION_MODEL,
        "messages": [{"role": "user", "content": prompt, "images": images_b64}],
        "stream": False,
    }
    try:
        with httpx.Client(timeout=timeout) as client:
            resp = client.post(f"{OLLAMA_HOST}/api/chat", json=payload)
            resp.raise_for_status()
            data = resp.json()
    except httpx.ConnectError as exc:
        raise VisionError(
            f"Cannot reach Ollama at {OLLAMA_HOST}. Start Ollama and run: ollama pull {VISION_MODEL}"
        ) from exc
    except httpx.HTTPStatusError as exc:
        raise VisionError(f"Ollama error {exc.response.status_code}: {exc.response.text[:300]}") from exc

    message = data.get("message") or {}
    content = message.get("content", "").strip()
    if not content:
        raise VisionError("Empty response from Ollama")
    return content

def health_check() -> dict:
    """Return model + connectivity status for demos."""
    if MOCK:
        return {"ok": True, "mode": "mock", "model": VISION_MODEL}
    try:
        with httpx.Client(timeout=5.0) as client:
            resp = client.get(f"{OLLAMA_HOST}/api/tags")
            resp.raise_for_status()
            tags = {m.get("name", "").split(":")[0] for m in resp.json().get("models", [])}
            base = VISION_MODEL.split(":")[0]
            return {
                "ok": base in tags or VISION_MODEL in tags,
                "mode": "live",
                "model": VISION_MODEL,
                "ollama_host": OLLAMA_HOST,
            }
    except Exception as exc: # noqa: BLE001 — demo helper
        return {"ok": False, "mode": "offline", "model": VISION_MODEL, "error": str(exc)}

Environment variables (see .env.example):

.env.example

# Ollama host (default local)
OLLAMA_HOST=http://127.0.0.1:11434

# Vision model — 1.6 GB, text + image, 256K context
OLLAMA_VISION_MODEL=minicpm-v4.6

# Set to 1 to run agent_demo without Ollama (offline smoke test)
OLLAMA_MOCK=0

Part 5 — The three tools

examples/server.py is a FastMCP server.


server.py
#!/usr/bin/env python3
"""MiniCPM-V MCP server — vision tools for Cursor, Claude Desktop, and Hermes.

Exposes three tools over MCP:

    describe_image(path, question?) → general image understanding
    ocr_document(path) → structured text extraction
    compare_images(path_a, path_b, focus?) → side-by-side visual diff

Powered by MiniCPM-V 4.6 via Ollama (~1.6 GB, text + image, 256K context).

Run: python examples/server.py # stdio transport (for MCP hosts)
"""
from __future__ import annotations

import json
from pathlib import Path

from mcp.server.fastmcp import FastMCP

try:
    from . import vision_backend as vb
except ImportError: # pragma: no cover
    import vision_backend as vb # type: ignore

mcp = FastMCP("minicpm-vision")

DESCRIBE_DEFAULT = (
    "Describe this image in detail. Include objects, text visible, layout, "
    "colors, and anything notable for a developer reviewing a screenshot."
)
OCR_PROMPT = (
    "Extract all readable text from this document or screenshot. "
    "Preserve structure with markdown headings and bullet lists where appropriate. "
    "If tables are present, format them as markdown tables."
)
COMPARE_DEFAULT = (
    "Compare these two images. List similarities and differences. "
    "Note UI changes, text changes, and layout shifts."
)

def _resolve(path: str) -> Path:
    p = Path(path).expanduser().resolve()
    if not p.is_file():
        raise FileNotFoundError(f"Not a file: {p}")
    return p

def _tool_result(text: str, **meta) -> str:
    return json.dumps({"result": text, **meta}, indent=2)

@mcp.tool()
def describe_image(path: str, question: str = "") -> str:
    """Describe or answer questions about a single image using MiniCPM-V 4.6.

    Args:
        path: Absolute or relative path to a PNG, JPG, WEBP, or GIF file.
        question: Optional specific question about the image. Leave empty for
            a general description.

    Returns JSON with the model's answer and metadata.
    """
    try:
        img = _resolve(path)
        prompt = question.strip() or DESCRIBE_DEFAULT
        answer = vb.chat_vision(prompt, [img])
        return _tool_result(answer, tool="describe_image", path=str(img), model=vb.VISION_MODEL)
    except (FileNotFoundError, vb.VisionError) as exc:
        return json.dumps({"error": str(exc)}, indent=2)

@mcp.tool()
def ocr_document(path: str) -> str:
    """OCR a document, receipt, whiteboard photo, or screenshot to markdown text.

    Args:
        path: Absolute or relative path to the image file.

    Returns JSON with extracted text in markdown format.
    """
    try:
        img = _resolve(path)
        answer = vb.chat_vision(OCR_PROMPT, [img])
        return _tool_result(answer, tool="ocr_document", path=str(img), model=vb.VISION_MODEL)
    except (FileNotFoundError, vb.VisionError) as exc:
        return json.dumps({"error": str(exc)}, indent=2)

@mcp.tool()
def compare_images(path_a: str, path_b: str, focus: str = "") -> str:
    """Compare two images and report visual differences.

    Args:
        path_a: Path to the first image (e.g. before screenshot).
        path_b: Path to the second image (e.g. after screenshot).
        focus: Optional aspect to focus on (e.g. "navigation bar", "error message").

    Returns JSON with a structured comparison.
    """
    try:
        a, b = _resolve(path_a), _resolve(path_b)
        prompt = COMPARE_DEFAULT
        if focus.strip():
            prompt += f"\n\nFocus especially on: {focus.strip()}"
        answer = vb.chat_vision(prompt, [a, b])
        return _tool_result(
            answer,
            tool="compare_images",
            path_a=str(a),
            path_b=str(b),
            model=vb.VISION_MODEL,
        )
    except (FileNotFoundError, vb.VisionError) as exc:
        return json.dumps({"error": str(exc)}, indent=2)

@mcp.resource("minicpm-vision://model")
def model_info() -> str:
    """Capability hint: which vision model and host this server uses."""
    status = vb.health_check()
    return json.dumps(
        {
            "model": vb.VISION_MODEL,
            "ollama_host": vb.OLLAMA_HOST,
            "tools": ["describe_image", "ocr_document", "compare_images"],
            "status": status,
        },
        indent=2,
    )

if __name__ == " __main__":
    mcp.run(transport="stdio")

describe_image

General-purpose image Q&A. Pass a custom question for targeted queries.

Sample input — architecture diagram the demo describes:

ocr_document

Structured OCR prompt — markdown headings, bullet lists, tables. Ideal for receipts, invoices, and whiteboard photos.

Sample input — coffee shop receipt:

compare_images

Two paths + optional focus (e.g. "navigation bar"). Returns similarities, differences, and UI change notes.

Sample inputs — before and after pipeline:

Each tool returns JSON with result, tool, paths, and model.

Part 6 — Agent demo (terminal walkthrough)

examples/agent_demo.py runs all three scenarios:

python examples/generate_fixtures.py
python examples/agent_demo.py

agent_demo.py

#!/usr/bin/env python3
"""End-to-end demo — MiniCPM-V MCP vision tools (works offline with OLLAMA_MOCK=1).

Simulates what Cursor / Claude Desktop sees when the agent calls vision tools.
"""
from __future__ import annotations

import json
import os
import sys
from pathlib import Path

ROOT = Path( __file__ ).resolve().parent
sys.path.insert(0, str(ROOT))

import vision_backend as vb # noqa: E402
from server import compare_images, describe_image, ocr_document # noqa: E402

FIXTURES = ROOT / "fixtures"

def _banner(title: str) -> None:
    print(f"\n{'=' * 60}\n {title}\n{'=' * 60}")

def _parse(result: str) -> dict:
    return json.loads(result)

def main() -> None:
    os.environ.setdefault("OLLAMA_VISION_MODEL", "minicpm-v4.6")
    if not FIXTURES.exists() or not list(FIXTURES.glob("*.png")):
        from generate_fixtures import main as gen # noqa: E402

        gen()

    status = vb.health_check()
    _banner("MiniCPM-V 4.6 Vision MCP — Agent Demo")
    print(f"Model: {vb.VISION_MODEL} · Mode: {status.get('mode', '?')}")
    if not status.get("ok") and not vb.MOCK:
        print("\n⚠️ Ollama offline — re-run with OLLAMA_MOCK=1 or start Ollama.\n")

    # Scenario 1 — describe screenshot
    _banner("Scenario 1 — describe_image")
    print('[Tool: describe_image] path=fixtures/diagram_v2.png')
    r1 = _parse(describe_image(str(FIXTURES / "diagram_v2.png"), "What services are shown?"))
    print("\n## Architecture summary\n")
    print(r1.get("result", r1.get("error", r1)))

    # Scenario 2 — OCR receipt
    _banner("Scenario 2 — ocr_document")
    print('[Tool: ocr_document] path=fixtures/sample_receipt.png')
    r2 = _parse(ocr_document(str(FIXTURES / "sample_receipt.png")))
    print("\n## Receipt OCR\n")
    print(r2.get("result", r2.get("error", r2)))

    # Scenario 3 — compare before/after
    _banner("Scenario 3 — compare_images")
    print("[Tool: compare_images] v1 → v2 pipeline diagrams")
    r3 = _parse(
        compare_images(
            str(FIXTURES / "diagram_v1.png"),
            str(FIXTURES / "diagram_v2.png"),
            focus="new components and labels",
        )
    )
    print("\n## Visual diff\n")
    print(r3.get("result", r3.get("error", r3)))

    _banner("Done — wire examples/server.py into Cursor MCP settings")
    print("See examples/cursor_mcp.json.example")

if __name__ == " __main__":
    main()

The terminal shows the same flow your MCP host runs:

[Tool: describe_image] path=fixtures/diagram_v2.png
[Tool: ocr_document] path=fixtures/sample_receipt.png
[Tool: compare_images] v1 → v2 pipeline diagrams

Offline smoke test (no Ollama):

OLLAMA_MOCK=1 python examples/agent_demo.py

Part 7 — Wire into Cursor

Copy examples/cursor_mcp.json.example into Cursor → Settings → MCP. Use absolute paths for cwd.

Restart Cursor — you should see describe_image, ocr_document, compare_images.

Try: “Use ocr_document on /path/to/receipt.png and summarize the total.”

{
  "mcpServers": {
    "minicpm-vision": {
      "command": "python",
      "args": ["examples/server.py"],
      "cwd": "/absolute/path/to/guides/minicpm-v-mcp-server",
      "env": {
        "OLLAMA_VISION_MODEL": "minicpm-v4.6",
        "OLLAMA_HOST": "http://127.0.0.1:11434"
      }
    }
  }
}

Part 8 — Wire into Claude Desktop

Add the server block from examples/claude_desktop_config.json.example to ~/Library/Application Support/Claude/claude_desktop_config.json on macOS.

Restart Claude Desktop. Vision tools appear alongside your other MCP servers.

{
  "mcpServers": {
    "minicpm-vision": {
      "command": "python",
      "args": ["/absolute/path/to/guides/minicpm-v-mcp-server/examples/server.py"],
      "env": {
        "OLLAMA_VISION_MODEL": "minicpm-v4.6",
        "OLLAMA_HOST": "http://127.0.0.1:11434"
      }
    }
  }
}

Conclusion

Give your agent eyes — without giving away your pixels.

Most coding agents are brilliant at text and terrible at images. The usual fix is a cloud vision API: API keys in config, latency on every screenshot, and your receipts, UI mocks, and whiteboard photos leaving your machine.

MiniCPM-V 4.6 flips that. At 1.3B parameters and ~1.6 GB on Ollama, it runs comfortably on a 16 GB Mac and handles text + image input with a 256K context window. Wrap it in a small MCP server, and you get three reusable tools — describe_image, ocr_document, and compare_images — that Cursor, Claude Desktop, and any other MCP host can discover at connect time.

Thank you so much for reading

Like | Follow | Subscribe to the newsletter.

Catch us on

LinkedIn: https://www.linkedin.com/in/techlatest-net/

When to Fine-Tune an LLM (And When Prompting Is Enough)

Reddit Community: https://www.reddit.com/user/techlatest_net/

TechLatest AI & Tech Weekly #21

TechLatest — Sun, 21 Jun 2026 14:46:12 +0000

Welcome to this week’s edition of TechLatest AI & Tech Weekly 👋

Here’s a curated roundup of our latest blogs, notable product launches, and the most interesting AI & ML updates from June 15–June 21, 2026.

AI/ML News Roundup: June 15–June 21, 2026

Open-Source AI, AI Agents & Developer Releases

Vercel Releases Eve

Vercel introduced Eve , an AI cloud engineer designed to help developers deploy, debug, monitor, and manage applications directly from natural language. Eve can inspect infrastructure, perform operational tasks, and automate common DevOps workflows, bringing agentic capabilities closer to production environments. Source

NVIDIA Introduces SpatialClaw

NVIDIA researchers released SpatialClaw , a training-free AI agent that treats executable code as its action interface for solving complex spatial reasoning tasks. Instead of relying on specialized training, the agent dynamically generates and executes code to reason about objects, layouts, and spatial relationships. Source

Hermes Agent Gets Blank Slate Mode

Nous Research updated Hermes Agent with a new Blank Slate Mode , allowing developers to precisely control available tools through platform_toolsets disabled_toolsets configurations. This creates highly controlled agent environments with predictable capabilities. Source

Cisco Introduces FAPO

Cisco AI unveiled FAPO (Failure-Aware Prompt Optimization), a framework that identifies failures at individual pipeline steps and automatically optimizes prompts using Claude Code orchestration. Source

VibeThinker-3B

Researchers introduced VibeThinker-3B , a compact reasoning model built on Qwen2.5-Coder-3B using the Spectrum-to-Signal post-training pipeline. Despite its smaller size, the model demonstrates strong reasoning and coding capabilities. Source

Flash-KMeans Achieves 200× Speedup

Researchers introduced Flash-KMeans , an I/O-aware implementation of exact K-Means clustering that reportedly runs more than 200× faster than FAISS on GPUs by reducing memory bottlenecks and optimizing data movement. Source

Yandex Open Sources YAFF

Yandex released YAFF (Yet Another Fast Format), a zero-copy wire format for Protocol Buffers that delivers near-struct-read performance while maintaining Protobuf compatibility. Source

Perplexity Launches Brain

Perplexity introduced Brain , a new AI-powered workspace designed to help users organize research, conversations, documents, and web knowledge into a persistent intelligence layer. Brain combines search, memory, and knowledge management, allowing users to build a continuously evolving repository of information. Source

Hermes Agent Adds Asynchronous Subagents

Nous Research upgraded Hermes Agent with asynchronous subagents that can work independently in the background while the main conversation continues. Delegated tasks no longer block the parent agent, enabling more scalable multi-agent workflows. Source

Hugging Face & Open-Source Ecosystem

Clem Delangue argued publicly on June 15, 2026 that AI’s future is a choice between a closed, Silicon Valley-led path and an open-source path where broader participation is possible, reinforcing Hugging Face’s strategic identity as a champion of open AI ecosystems.
Ideogram’s open-weight v4 image model was highlighted on Hugging Face earlier in June and remains explorable through a multimodal demo Space on the platform, as part of ongoing open-weight image model distribution.
Quantized Gemma 4 checkpoints released by Philipp Schmid were made available on Hugging Face in mid-June, underscoring the platform’s role as the default distribution point for optimized open models.
Diffusion-GEMMA’s faster text-generation model weights were open-sourced on Hugging Face under Apache 2.0 and shared by Sundar Pichai, continuing the trend of open-sourcing high-performance model variants.

Frontier Model Advancements & Agentic Capabilities

Claude Sonnet 4 and Claude Opus 4 officially retired on June 15, 2026 , stopping all API requests; Anthropic had flagged this date well in advance, directing production teams to migrate to Claude Sonnet 4.6 or Claude Opus 4.8.
Claude Sonnet 4.8 is expected to launch in the June 16–18 window , roughly three weeks after Opus 4.8’s May 28 release, with expected additions including Dynamic Workflows and refined effort/thinking-budget controls.
GPT-5.6 was spotted running inside ChatGPT Pro in mid-June 2026, with developers reporting noticeably faster and more capable responses than GPT-5.5 Pro; OpenAI’s Chief Scientist previewed GPT-5.6 as a “meaningful improvement” with a late-June 2026 launch expected.
GLM-5.2 from China’s Zhipu AI beat GPT-5.5 outright on the FrontierSWE benchmark , which measures AI agents on multi-hour, open-ended engineering projects, and trails Claude Fable 5 by just one point, effectively co-leading frontier AI on coding benchmarks while Fable 5 is offline.
Anthropic’s Fable 5 and Mythos 5 remained offline for over a week under a US government export ban issued June 12, with 100+ cybersecurity leaders calling it an overreach and demanding reversal.

Massive Funding Rounds Reshaping Valuations

Salesforce acquired Fin , an AI customer service platform, for $3.6 billion this week, fitting CEO Marc Benioff’s strategy of positioning Salesforce as the AI-native layer for enterprise customer operations and competing directly with Anthropic’s Claude for Work.
Anthropic signed more than 12 US data center leases exceeding 1 gigawatt of computing capacity while preparing for its IPO, with Google reportedly in discussions to provide additional financial backing.
Anthropic’s revenue run-rate hit approximately $47 billion in May 2026 , up roughly 5x year-over-year from about $10 billion, supporting its $965 billion post-money valuation after a $65 billion Series H round confirmed by Bloomberg on May 29.

Practical Product Launches for Developers & Enterprises

ChatGPT Dreaming V3 Memory rollout widened this week toward Free and Go tier users after beginning to reach ChatGPT Plus and Pro users in the US on June 4; OpenAI described it as its most significant memory upgrade since the original rollout, letting the model retain and apply context across sessions more reliably.
Apple’s Gemini-powered Siri and Claude Extension rolled out in iOS 27 Beta 1 , following Tim Cook’s final WWDC keynote on June 8, making Claude available as an iPhone assistant option for the first time, with Apple Intelligence features requiring iPhone 15 Pro or newer.
Microsoft continued rolling out its MAI model family inside Azure AI Foundry, including MAI-Thinking-1 (unveiled by Mustafa Suleyman as Microsoft AI’s flagship reasoning model at Build 2026), and finalized an 11,000-model Azure AI Foundry catalog that now includes Claude Opus 4.8.
NVIDIA Cosmos 3 adoption grew in healthcare simulation , being increasingly used to generate synthetic training videos of rare medical scenarios for surgical robots, addressing data scarcity problems impossible to solve with real-world footage alone.

Governance, Ethics & Regulation

The EU AI Act Enforcement Countdown hit 50 days , with the bulk of the Act beginning to apply on August 2, 2026 ; fines for the most serious violations can reach 35 million euros or 7% of global annual turnover , with 15 million euros or 3% for most other breaches.
Colorado AI Act’s compliance window narrowed this week, with Colorado’s AI Act taking effect this year as one of the first US state-level AI laws with real enforcement, making it more consequential in practice than several higher-profile federal proposals.
Massachusetts legislators advanced several AI-related bills this week, including SB 760 (a kids’ chatbot safety bill), H 76 (addressing AI-generated deceptive election communications), and H 4616 (covering AI use in healthcare prior authorizations).
Louisiana became the 22nd U.S. state to enact a comprehensive consumer data privacy law , further expanding the compliance surface area for AI deployments processing personal data across the United States.
The Economist’s June 20, 2026 cover story “America’s AI Power Grab” framed the Fable 5 and Mythos 5 export ban as a geopolitical assertion, treating frontier AI models similarly to weapons systems subject to export controls.
Dario Amodei met with Trump administration officials this week to negotiate a path to restoring access to Fable 5 and Mythos 5, but meetings did not produce a resolution, with no timeline for restoration confirmed.
Over 100 cybersecurity leaders signed an open letter demanding the US government reverse its decision to ban Anthropic’s Fable 5 and Mythos 5 models, arguing the ban is disproportionate and technically inaccurate.

Infrastructure & Hardware

Anthropic signed more than 12 US data center leases exceeding 1 gigawatt of computing capacity, roughly equivalent to what a mid-sized country needs for its entire national electricity grid, to meet projected exponentially growing demand for Claude models.
Anthropic is already paying SpaceX $1.25 billion per month for access to over 220,000 Nvidia processors at the Colossus 1 facility in Memphis, with the contract running through May 2029.
NVIDIA Cosmos 3’s simulation capabilities are increasingly being used to generate synthetic training videos of rare medical scenarios for surgical robots, combined with NVIDIA’s Vera Rubin platform and Intel’s Xeon 6+ featuring Confidential Computing at rack scale.
Orion-100B reports continued circulating this week about a 100-billion-parameter model reportedly trained for just $1.25 per hour of compute, a figure being held up as a new benchmark for training cost efficiency, though treated with skepticism until independent verification.

Blogs We Published This Week

When to Fine-Tune an LLM and When Prompting Is Enough

A practical guide to understanding when prompt engineering is sufficient and when fine-tuning becomes necessary. Learn the trade-offs between cost, performance, customization, and maintenance when building AI applications.

Loop Engineering Explained Visually: From Manual Prompts to Goal-Driven AI Agents

Explore the evolution from traditional prompting to autonomous AI systems through Loop Engineering. This visual guide explains how modern agents continuously plan, execute, evaluate, and improve toward a goal.

Loop Engineering Explained Visually: From Manual Prompts to Goal-Driven AI Agents

Harness Engineering — Full Visual Guide

Learn how Harness Engineering helps developers systematically evaluate, test, and improve AI systems using structured feedback loops, benchmarks, evaluation pipelines, and continuous optimization techniques.

Harness Engineering — Full Visual Guide

AI Agents Masterclass — Full Visual Guide

A comprehensive visual walkthrough of AI agents, covering agent architectures, memory systems, tools, planning, multi-agent workflows, and real-world deployment patterns.

AI Agents Masterclass — Full Visual Guide

Model Context Protocol (MCP) — Full Visual Guide

Understand how the Model Context Protocol (MCP) enables AI applications to securely connect with tools, databases, APIs, repositories, and external systems through a standardized interface.

Model Context Protocol (MCP) — Full Visual Guide

TL;DR — TechLatest AI & Tech Weekly #21

Perplexity launched Brain, turning AI search into a persistent knowledge and memory workspace.
Vercel introduced Eve, an AI cloud engineer capable of managing deployments, debugging infrastructure, and automating DevOps workflows.
NVIDIA unveiled SpatialClaw, a training-free agent that uses executable code as the interface for spatial reasoning.
Hermes Agent received two major upgrades: asynchronous subagents for parallel task execution and Blank Slate Mode for controlled enterprise deployments.
Cisco launched FAPO, a framework that identifies failures within AI pipelines and automatically optimizes prompts.
VibeThinker-3B demonstrated how compact reasoning models can achieve strong performance with efficient post-training techniques.
Flash-KMeans achieved over 200× faster exact clustering on GPUs, while Yandex open-sourced YAFF for high-performance data serialization.
GPT-5.6 surfaced in ChatGPT Pro testing, GLM-5.2 surpassed GPT-5.5 on FrontierSWE, and Claude Sonnet 4.8 is expected to arrive soon.
Anthropic’s Fable 5 and Mythos 5 remained unavailable under U.S. export restrictions, triggering industry-wide debate and backlash.
Salesforce acquired Fin for $3.6 billion, while Anthropic’s valuation approached $1 trillion following explosive revenue growth.
Open-source AI momentum continued with new Gemma 4 checkpoints, Diffusion-GEMMA releases, and Hugging Face reinforcing its commitment to open ecosystems.
AI regulation accelerated globally as the EU AI Act countdown reached 50 days, Colorado’s AI Act approached enforcement, and multiple U.S. states advanced new AI legislation.
We published five new visual guides covering AI Agents, MCP, Loop Engineering, Harness Engineering, and when to fine-tune LLMs versus relying on prompting.

Thank you so much for reading

Like | Follow | Subscribe to the newsletter.

Catch us on

LinkedIn: https://www.linkedin.com/in/techlatest-net/

Reddit Community: https://www.reddit.com/user/techlatest_net/

Model Context Protocol (MCP) — Full Visual Guide

TechLatest — Fri, 19 Jun 2026 06:57:17 +0000

Connect AI hosts — Cursor, Claude Desktop, VS Code Copilot — to your databases, repos, APIs, and local files through a standard wire protocol , not one-off integrations per app.

What you’ll understand at the end

Why MCP is described as USB-C for AI apps — and what that actually means in code
Host, Client, Server roles and how one host fans out to many servers
The three server primitives: Tools, Resources, Prompts
Capability exchange at connect time — and why it beats brittle REST contracts for agents
How to scaffold a server , wire Cursor / Claude Desktop , and inspect tools
App MCP — servers that return UI instructions, not just markdown

MCP hub — standardized connections

Part 1 — The USB-C mental model

At the center sits your AI application (the host). Around it: databases, GitHub, email, local filesystem, Slack, public web APIs. Each connection runs over the same protocol label: MCP.

The host doesn’t embed GitHub logic or Postgres drivers. It embeds an MCP client that talks to MCP servers — one per domain (filesystem server, GitHub server, your custom weather server).

Part 2 — Host, Client, Server

Three roles — don’t collapse them:

One host → many clients → many servers. The model sees a unified tool surface; each server stays isolated and deployable on its own.

Part 3 — What servers expose

Every MCP server advertises up to three capability families:

Tools — actions the model can call (fetch weather, run query, create issue).

Resources — readable content (file URIs, schema docs, config snapshots).

Prompts — reusable prompt templates the host can surface to users.

Server capabilities — Tools · Resources · Prompts

Think of tools as verbs , resources as nouns , prompts as saved playbooks.

Retrieval systems are a common MCP use case. Platforms such as Instant RAGFlow expose knowledge repositories and document collections that agents can access dynamically through tools and resources rather than embedding all information directly into prompts.

Link: https://techlatest.net/support/ragflow_support/

Many MCP-powered retrieval workflows depend on vector databases for semantic search. Chroma Vector Database provides a lightweight memory layer that can be exposed through MCP servers, allowing agents to retrieve relevant context from embeddings on demand.

Link: https://techlatest.net/support/chroma-support/

Part 4 — Transport layer

Client and server exchange JSON-RPC messages over a transport :

stdio — host spawns server as subprocess; stdin/stdout carry messages. Default for local dev (npx, python server.py).
SSE / HTTP — server runs remotely; client connects over the network.

The capability model is identical; only the pipe changes. Most tutorials and desktop hosts start with stdio because it’s one config block and no open port.

Transport — stdio local vs SSE remote

Part 5 — Capability exchange (the handshake)

Before any tool runs, client and server perform a capability exchange :

Client sends initialize — protocol version, client info
Server responds with supported tools, resources, prompts, and JSON schemas for parameters
Client sends initialized notification — channel is ready

Handshake — initialize → capabilities → ready

Example: a weather server might advertise get_forecast(location, date) with types. The host's model receives that schema in context and knows exactly which arguments to fill — no hard-coded OpenAPI doc in the host repo.

Part 6 — Traditional API: rigid contracts

Classic REST: the client must know the contract ahead of time.

GET /forecast?location=NYC&date=2025-03-15
→ 200 + JSON body

Integrators bake location and date into their code. Works until you ship a breaking change.

Traditional API — fixed parameters

Part 7 — When APIs break clients

You add a required third parameter — unit (celsius | fahrenheit). Every client that still sends only two params gets a 400 error or incorrect defaults. Three sad integrators, three redeploys.

Breaking change — all clients fail

This is normal API versioning pain. For autonomous agents that must adapt without a human editing config, it’s worse: the agent doesn’t read your changelog.

Part 8 — MCP: discover, don’t hardcode

On each connection, the client asks what the server supports. The server returns current tool schemas — including new optional fields like unit.

The client (and model) adapts on the next session without redeploying host code. No silent failure from a missing query param the model never knew about.

MCP capability exchange — dynamic schema

Caveat: MCP doesn’t magically fix semantic breaking changes (renaming a tool). It fixes discovery — parameters and availability are first-class at connect time, not buried in external docs.

Part 9 — API vs MCP (when to use which)

Use both : MCP server as the agent-facing adapter; your core service stays a normal API internally.

Part 10 — Build your first server

We ship a minimal Python weather server using the official SDK (FastMCP):

from mcp.server.fastmcp import FastMCP

mcp = FastMCP("weather-demo")

@mcp.tool()
def get_forecast(location: str, day: str | None = None, unit: str = "celsius") -> str:
    """Stub forecast — location, optional day, unit celsius|fahrenheit."""
    ...

if __name__ == " __main__":
    mcp.run(transport="stdio")

Run locally:

pip install "mcp[cli]>=1.2"
python examples/minimal_weather_server.py

Install MCP SDK; scaffold server project

Part 11 — Wire the host

Cursor — MCP settings JSON (Settings → MCP):

{
  "mcpServers": {
    "weather-demo": {
      "command": "python",
      "args": ["examples/minimal_weather_server.py"],
      "cwd": "/path/to/guides/mcp-visual-guide"
    }
  }
}

Claude Desktop — claude_desktop_config.json with the same command / args pattern.

Cursor MCP config

Restart the host after config changes. The client spawns your server process and runs the handshake automatically.

Part 12 — Inspect and debug

Use the MCP Inspector or host UI to list tools:

npx @modelcontextprotocol/inspector python examples/minimal_weather_server.py

You should see get_forecast with parameters location, day, unit. Invoke it with test JSON before trusting the model.

List tools via inspector Test tool invocation

Part 13 — Beyond text: App MCP (Visuals MCP pattern)

Most servers return strings — markdown tables, JSON blobs, file paths. That hits limits fast:

Markdown tables don’t sort or filter
Long lists burn context
Images arrive as URLs, not previews

App MCP servers return UI render instructions plus data. The host loads a bundled React (or other) app via MCP resources (text/html), and the model passes structured tool results into that app.

Flow:

User: “Show EC2 instances in a sortable table”
Model calls display_table tool
Server returns columns + rows + resourceUri: table://display
Host renders interactive grid (sort, filter, paginate)

Text MCP vs App MCPVisuals flow — prompt → tool → React app

Architecture note from production App MCP servers: one mini-app per visual — separate Vite build, single-file HTML bundle, tool metadata pointing at resourceUri. See Visuals MCP for tables, trees, master-detail layouts.

Install pattern (any server, not only visuals):

{
  "mcpServers": {
    "visuals-mcp": {
      "command": "npx",
      "args": ["-y", "@harrybin/visuals-mcp"]
    }
  }
}

VS Code extension installs are the zero-config variant — the extension registers the server for Copilot Chat automatically.

Modern agent platforms increasingly rely on MCP for tool discovery and integration. OpenClaw combines multi-agent workflows, messaging channels, and MCP-powered capabilities, enabling agents to interact with external systems while maintaining structured sessions and tool access controls.

Link: https://techlatest.net/support/openclaw-support/

Part 14 — Production checklist

Scope tools narrowly — least privilege; read-only DB user for read tools
Validate inputs — the model will hallucinate argument shapes
Timeout and cancel — long-running tools need progress or abort
Version your server — bump tool descriptions when behavior changes
Log on the server — host logs won’t show your business errors
stdio vs remote — stdio for local trust boundary; HTTP for shared team servers with auth

As MCP ecosystems grow, operational visibility becomes increasingly important. Dify AI provides workflow orchestration, monitoring, evaluation, and deployment capabilities that help teams manage MCP-powered applications in production environments.

Link: https://techlatest.net/support/difyai_support/

Multi-agent systems often depend on multiple MCP servers for tools and data access. CrewAI Studio helps coordinate agent teams and workflows while integrating with external services through standardized interfaces such as MCP.

Link: https://techlatest.net/support/crewai-support/

Summary

MCP is a client-server protocol between AI hosts and the outside world. Hosts run clients ; servers expose tools, resources, and prompts after a capability handshake. Compared to rigid REST contracts, MCP lets agents discover current parameters instead of failing on undeclared fields. Start with a stdio server in Python or TypeScript, wire Cursor or Claude Desktop , inspect with the MCP Inspector , then explore App MCP when markdown isn’t enough.

Thank you so much for reading

Like | Follow | Subscribe to the newsletter.

Catch us on

LinkedIn: https://www.linkedin.com/in/techlatest-net/

Reddit Community: https://www.reddit.com/user/techlatest_net/

AI Agents Masterclass — Full Visual Guide

TechLatest — Wed, 17 Jun 2026 06:40:49 +0000

Everything you need to understand, compare, and build AI agents: definitions from Google Cloud and IBM, ReAct and ReWOO loops, multi-agent patterns, 15+ frameworks, MCP and A2A protocols, governance, Cloud Run deployment, and five runnable examples with animated diagrams + terminal GIFs.

What you’ll understand at the end

What an AI agent is — and how it differs from assistants, chatbots, and bots
Six core capabilities: reasoning, acting, observing, planning, collaborating, self-refining

Agent anatomy : persona, memory, tools, model
Memory tiers — working, episodic, semantic, procedural
ReAct and ReWOO reasoning paradigms
Five classical agent types on the reflex → learning ladder
Three lifecycle stages: goal planning, tool reasoning, learning/reflection
Single vs multi-agent , surface vs background deployment
Agentic vs non-agentic chatbots
Six enterprise use-case families with healthcare, finance, and emergency examples
Benefits, challenges, and governance patterns (HITL, activity logs, interruption)
15+ frameworks — when to pick LangGraph, CrewAI, OpenAI Agents SDK, Pydantic AI, Hermes, OpenClaw, and more
MCP + A2A interoperability
Cloud Run production deployment
Five runnable examples with terminal GIFs and smoke tests

Introduction — why agents now

For years, “AI” in products meant one-shot generation : you send a prompt, the model returns text, the transaction ends. That works for drafting emails. It fails for real work — research a market, book travel, triage tickets, reconcile accounts — because real work is multi-step , tool-dependent , and stateful.

An AI agent closes that gap. Instead of a single response, the system pursues a goal over time: it plans, calls tools, reads results, revises, and stops when the objective is met (or when a human says stop).

Industry definitions converge on the same idea with different emphasis:

Google Cloud describes AI agents as systems that combine a foundation model with reasoning , planning , and action — using tools and external data to accomplish tasks on a user’s behalf, not just answer questions.

IBM frames agents as software entities that perceive their environment, reason about goals, and act through tools or APIs — often with memory that persists across interactions.

OpenAI’s practical guide adds product reality: agents shine when workflows are open-ended , require judgment , and benefit from tool use — but they demand stronger observability and guardrails than chatbots.

This masterclass synthesizes those views into one buildable mental model, then walks you through code, frameworks, and production patterns.

Part 1 — Agent vs assistant vs bot

Three labels get swapped in marketing. Architecturally, they differ:

Bot (classic) — rule-based or intent-classifier driven. Fixed dialog trees, slot filling, no genuine planning. Example: “Track my package” → lookup by tracking number. Predictable, cheap, brittle outside trained intents.

Assistant (LLM chatbot) — a model in a chat UI. Strong at language, weak at persistence. Each turn is mostly stateless unless you bolt on memory. Example: “Summarize this PDF” in one shot. No tool loop unless explicitly wired.

Agent — an LLM (or ensemble) wrapped in a control loop : plan → act via tools → observe results → repeat. Carries goal state , memory , and often delegation to other agents. Example: “Find the best week for surfing in Greece next year” → weather DB → tide search → synthesize → recommend dates.

Agent vs assistant vs bot

Rule of thumb in prose: if the product only needs one model call and no side effects, use an assistant. If it must change the world (APIs, DBs, files, tickets) over multiple steps, you are building an agent. If the flow is fully scripted with no LLM judgment, you might not need an agent at all — a workflow engine suffices.

Part 2 — Six defining capabilities

Modern agents are not defined by a single feature but by a bundle of behaviors:

Reasoning — the model decomposes goals, handles ambiguity, and chooses among strategies. Chain-of-thought and structured planning prompts live here.

Acting — execution through tools : HTTP calls, SQL, Python, browser automation, MCP servers. Action is what separates agents from chat.

Observing — after each action, the agent ingests tool output (JSON, logs, errors) and updates its internal state. Bad observation handling is the #1 source of silent failures.

Planning — explicit or implicit task graphs: “first gather weather, then check tides, then compare weeks.” Plans may be static (ReWOO) or interleaved with execution (ReAct).

Collaborating — multi-agent handoffs, human approvals, or role-based crews. No single model must do everything.

Self-refining — reflection passes, critique steps, memory writes, skill authoring. The agent improves its approach within or across sessions (see Hermes learning loop).

Agent anatomy — persona, memory, tools, model

These six capabilities map directly to architecture choices later: tools need MCP or function schemas; collaboration needs handoff or crew abstractions; self-refining needs memory tiers and logging.

Part 3 — Anatomy: persona, memory, tools, model

Every production agent resolves into four layers:

Persona — system prompt, SOUL.md, role brief. Sets tone, boundaries, and escalation rules. In enterprise agents, persona also encodes compliance (“never disclose account numbers”).

Memory — what persists beyond the current context window. Short-term: chat history and scratchpad. Long-term: vector stores, markdown files, session DBs. See Part 4.

Tools — typed functions the model can invoke. Each tool needs a name, description, JSON schema, and a handler. Tools should be narrow and idempotent where possible.

Model — the reasoning engine. Often one primary model plus smaller models for routing or summarization. Model choice affects cost, latency, and tool-call reliability.

# Conceptual agent stack (not framework-specific)
agent = {
    "persona": "You are a cautious travel planner. Confirm before booking.",
    "memory": {"session": [], "long_term": "vector://user-prefs"},
    "tools": ["weather_db", "search_web", "calendar_create"],
    "model": "gpt-4o",
}

The model is interchangeable; tools and memory encode your product’s real value.

Part 4 — Memory tiers

Memory is not one blob. Mature agents use tiers with different latency, capacity, and retrieval patterns:

Working memory — the current context window: system prompt, recent turns, tool results. Bounded by token limits; compress or summarize when full.

Episodic memory — past sessions and events (“last time we planned Greece, user preferred July”). Stored in SQLite, Postgres, or session logs; retrieved by recency or search.

Semantic memory — facts and embeddings in a vector store. “User is vegetarian.” “API X rate-limits at 100 rpm.”

Procedural memory — skills, playbooks, SOUL-adjacent instructions. Often markdown files or skill catalogs (Hermes SKILL.md, OpenAI custom instructions at scale).

Memory tiers — working, episodic, semantic, procedural

Design rule: inject a small frozen snapshot at session start (persona + top facts), then let the agent search for deeper history on demand. Dumping entire history into every turn burns context and money.

Many enterprise agents rely on retrieval systems rather than storing all knowledge directly inside the model context window. Platforms such as Instant RAGFlow provide document ingestion, indexing, and retrieval pipelines that allow agents to access relevant information dynamically while keeping prompts lean and up to date.

Link: https://techlatest.net/support/ragflow_support/

Semantic memory is commonly implemented using vector databases that store embeddings and enable similarity search. Chroma Vector Database is a popular lightweight option for agent memory systems, helping agents retrieve relevant facts, previous interactions, and domain knowledge during execution.

Link: https://techlatest.net/support/chromadb_support/

Part 5 — ReAct: interleaved reasoning and action

ReAct (Reason + Act) alternates thought , tool call , and observation in one loop. The model decides the next step only after seeing the last observation.

Typical trace:

Think: “I need historical weather for Greece.”
Act: weather_db("Greece")
Observe: { "avg_sunny_days_july": 28 }
Think: “Need tide/surf conditions.”
Act: search_web("best surfing tide Greece")
Observe: snippet about high tide windows
Think: “Combine signals → recommend July 12–19.”
Act: respond to user

ReAct loop — think, act, observe

ReAct is flexible — the plan emerges from execution. That helps exploratory tasks. Cost: more model turns, harder to audit upfront.

Our minimal example implements this pattern (deterministic demo without an API key):

# examples/minimal_react_agent.py (excerpt)
def think_and_act(state: AgentState, turn: int) -> None:
    if turn == 0:
        state.steps.append("Think: need historical weather for Greece")
        out = TOOLS["weather_db"]("Greece")
        state.steps.append(f"Act: weather_db → {out}")
    elif turn == 1:
        state.steps.append("Think: need surfing conditions (high tide)")
        out = TOOLS["search_web"]("best surfing tide Greece")
        state.steps.append(f"Act: search_web → {out}")
    elif turn == 2:
        state.steps.append("Observe: combine tide + sunny patterns")
        state.steps.append("Act: recommend week of July 12–19 (demo)")
        state.done = True

Run:

cd guides/ai-agents-masterclass
python examples/minimal_react_agent.py

Part 6 — ReWOO: plan first, execute second

ReWOO (Reasoning Without Observation in the loop) separates planning from execution. A planner emits a structured script of tool calls; a worker runs them; a solver synthesizes the final answer.

Flow:

Planner — output tool call graph with placeholders
Worker — execute all tools (possibly in parallel)
Solver — read outputs, no further tool access

ReWOO flow — planner, worker, solver

When ReWOO wins: predictable pipelines, expensive tools, parallelizable subtasks, audit requirements (plan is reviewable before execution).

When ReAct wins: ambiguous goals, errors mid-flight, need to branch on unexpected results.

Many production systems hybridize : ReWOO for the macro pipeline, ReAct inside a single step when debugging.

Part 7 — Five classical agent types

Before LLMs, agent literature defined a ladder of sophistication. Still useful for scoping:

Simple reflex — if condition then action. Thermostat, basic alert bot. No memory, no search.

Model-based reflex — internal state tracks the world (last sensor reading). Still no planning.

Goal-based — searches action sequences to reach a goal. Classical planning / STRIPS territory.

Utility-based — optimizes tradeoffs (cost vs speed vs risk). Portfolio agents, routing.

Learning — updates policy from feedback. RL agents, self-refining skill loops, GEPA-style offline evolution.

Agent types ladder — reflex to learning

LLM agents usually sit at goal-based with hooks toward learning (memory writes, reflection, fine-tuning). Don’t over-build learning before basic tool reliability works.

Part 8 — Three lifecycle stages (surfing vacation)

OpenAI and ServiceNow-style masterclasses often teach agents as three stages. We use one running example: “Best week for surfing in Greece next year.”

Stage 1 — Goal planning

Decompose the user goal into subtasks and success criteria.

Subtask A: historical weather / sunny weeks
Subtask B: surf/tide suitability
Subtask C: reconcile constraints (user budget, travel dates)
Done when: ranked recommendation with confidence

Goal planning — decompose and prioritize

User goal: "Best week for surfing in Greece next year"
Planner output:
  1. Query weather_db(Greece) for sunny weeks
  2. search_web for tide/surf windows
  3. Rank weeks; explain tradeoffs

Stage 2 — Tool reasoning

Select tools, fill arguments, handle errors, retry with backoff. The model must not invent tool names — bind to your schema.

Tool reasoning — schema-bound calls

TOOLS = {
    "search_web": search_web,
    "weather_db": weather_db,
}
# Model sees JSON schemas; handler validates before side effects

Stage 3 — Learning and reflection

After answering, optionally: log trace, write memory (“user cares about surfing”), critique weak steps, update skills. This is where agents compound over time.

Learning loop — trace to memory to skills

Reflection: "weather_db lacked tide granularity — add surf_forecast tool next sprint"
Memory write: USER prefers July travel

Agent lifecycle — plan, act, learn

Part 9 — Agentic vs non-agentic chatbots

Non-agentic chatbot — single-turn or few-turn Q&A. Retrieval augments context, but no autonomous tool loop. Great for FAQs, doc search, copilot suggestions.

Agentic chatbot — same UI, but backend runs a control loop with tools and state. User may see “Searching…” / “Calling calendar…” steps.

Differences that matter in production:

Latency — agents take longer; set UX expectations
Cost — multiple model + tool calls per user message
Failure modes — tool errors, infinite loops, hallucinated arguments
Observability — you need step traces, not just final text

If your feature is “answer from our PDF,” start non-agentic. If it is “file this ticket and follow up,” go agentic.

Part 10 — Single vs multi-agent

Single agent — one model, one loop, one tool namespace. Simplest to debug. Hits limits on long workflows and conflicting roles.

Multi-agent — specialized agents with handoffs or parallel crews. Examples: triage → specialist, researcher + writer, planner + executor.

Single vs multi-agent topologies

Patterns in prose:

Sequential crew — A completes task, passes output to B (CrewAI default)
Handoff — router agent transfers conversation to specialist (OpenAI Agents SDK)
Supervisor — orchestrator assigns subtasks to workers (LangGraph, AutoGen)
Debate/review — generator + critic for quality gates

Multi-agent adds coordination overhead. Start single-agent until you have clear role boundaries and separate tool permissions per role.

Part 11 — Surface vs background agents

Surface agents — user-facing, synchronous. Chat UI, voice, copilot pane. User waits for steps; HITL approvals live here.

Background agents — async jobs: cron digests, ticket sweeps, ETL monitors. Results delivered later via email, Slack, or dashboard.

Surface vs background deployment

Hermes cron and OpenClaw heartbeats are background patterns. Cloud Run jobs or scheduled Cloud Functions fit the same slot.

Design background agents with idempotency and dead-letter queues — they will retry at 3 am without a human watching.

Part 12 — Six use-case categories

Enterprise agents cluster into six families (plus cross-industry patterns):

Six use-case categories

1. Customer experience — support triage, order status, personalized recommendations. Needs CRM tools, strict PII handling.

2. Employee productivity — draft docs, schedule meetings, summarize threads. Microsoft 365 Copilot, Google Workspace agents.

3. Software development — issue → PR agents, test generation, migration assistants. Heavy IDE + repo tool access.

4. Data and analytics — natural language to SQL, anomaly explanation, report generation. Guard against destructive queries.

5. Security and operations — alert triage, runbook execution, patch verification. Read-only first; HITL for mutations.

6. Industry workflows — vertical bundles (see below).

Healthcare

Clinical documentation agents draft notes from visit audio — human sign-off required. Prior authorization agents gather payer rules and patient history. Scheduling agents coordinate slots across systems. Regulatory constraint: agents assist ; they do not diagnose autonomously in regulated jurisdictions.

Finance

Reconciliation agents match transactions across ledgers. Research agents summarize filings and earnings calls with citations. Compliance agents flag policy violations in communications. Audit trails and model risk management are mandatory.

Emergency and public safety

Dispatch assist agents summarize 911 transcripts and suggest resource allocation — always subordinate to human dispatchers. Disaster response agents aggregate feeds and produce situational reports. Latency and failure modes can be life-critical; degrade gracefully to static playbooks.

Part 13 — Benefits

Automation of judgment-heavy workflows — not just repetitive clicks, but branching decisions with explanations.

24/7 operation — background agents monitor queues overnight.

Composable tools — same agent core, swap MCP servers for new domains.

Personalization at scale — memory tiers remember preferences without re-prompting.

Faster iteration — natural language interfaces to internal APIs lower integration cost.

Part 14 — Challenges and risks

Unpredictability — same prompt, different tool paths. Mitigate with schemas, evals, and golden traces.

Cost — long ReAct loops multiply token usage. Cap turns, summarize observations.

Security — prompt injection via tool results, over-privileged tools, SSRF from web fetch tools. Least privilege per tool.

Compliance — GDPR, HIPAA, SOC2: log retention, data residency, human approval for sensitive actions.

Trust — users need visibility into what the agent did. Black-box answers erode adoption.

Governance — HITL, logs, policies

Part 15 — Best practices

Activity logs — append-only trace of every thought, tool call, observation, and final output. Store run_id, timestamps, user ID, model version.

Interruption — user can cancel in-flight loops; worker checks cancel token between turns (Hermes models this explicitly).

Unique IDs — correlate user session, agent run, and tool invocations across microservices.

Human-in-the-loop (HITL) — require approval for payments, deletes, external emails, privilege changes. Pattern: agent prepares action → human clicks approve → tool executes.

Tool design — small surface area, explicit errors, no silent defaults on missing args.

Evals — regression suite of goals with expected tool sequences or output rubrics.

Budgets — max turns, max tool calls, max cost per run.

# Pseudocode: run envelope
@dataclass
class RunContext:
    run_id: str
    user_id: str
    max_turns: int = 12
    cancelled: bool = False

def step(ctx: RunContext):
    if ctx.cancelled:
        raise RunCancelled(ctx.run_id)
    log_event(ctx.run_id, "tool_call", {...})

Part 16 — Protocols: MCP and A2A

Agents rarely exist alone. Two interoperability layers matter in 2025–2026:

Model Context Protocol (MCP)

MCP standardizes how hosts discover and invoke tools, resources, and prompts from external servers — “USB-C for AI tools.” Your agent (or IDE host) runs MCP clients; GitHub, Postgres, filesystem, custom APIs expose MCP servers.

Deep dive: MCP Visual Guide.

Protocols — MCP and A2A

Agent-to-Agent (A2A)

A2A (Google-led, industry collaborators) focuses on agent ↔ agent messaging: capability cards, task delegation, status updates across vendor boundaries. Where MCP connects agents to tools , A2A connects agents to each other.

Use MCP for tool sprawl; use A2A when your orchestrator and specialist run in different frameworks or clouds and need a standard task envelope.

Part 17 — Framework landscape

No single framework wins every workload. Map orchestration style , team familiarity , and deployment target first.

Frameworks map — LangGraph, CrewAI, SDKs, cloud

Below: when to use prose for each major option. All can coexist with MCP tool servers.

LangGraph

LangGraph models agents as state machines — nodes, edges, conditional routing, checkpointing. Best when you need explicit control flow , cycles, human-in-the-loop interrupts, and time-travel debugging. LangChain ecosystem; steep learning curve if you only need a simple ReAct loop.

# examples/langgraph_research_agent.py — plan → research → synthesize
g = StateGraph(ResearchState)
g.add_node("plan", plan)
g.add_node("research", research)
g.add_node("synthesize", synthesize)
g.set_entry_point("plan")
g.add_edge("plan", "research")
g.add_edge("research", "synthesize")
g.add_edge("synthesize", END)
app = g.compile()

Pick LangGraph for production workflows with branching, retries, and persisted state.

CrewAI

CrewAI optimizes role-based teams : researcher, writer, analyst with sequential or hierarchical process. Minimal boilerplate for multi-agent prose tasks. Less ideal for fine-grained tool graphs or hard latency SLAs.

crew = Crew(
    agents=[researcher, writer],
    tasks=[research_task, write_task],
    process=Process.sequential,
)
result = crew.kickoff()

Pick CrewAI for content pipelines , research briefs, and demos where roles are obvious.

AutoGen (Microsoft)

AutoGen emphasizes conversable agents and group chat patterns — good for coding assistants, multi-agent debate, and Azure/OpenAI shops. v0.4+ rearchitecture adds async and distributed agents. Choose when you want Microsoft stack integration and flexible agent-to-agent chat.

OpenAI Agents SDK

OpenAI Agents SDK (openai-agents) provides Agent , Runner , handoffs , and built-in tracing. Tight integration with OpenAI models and Responses API. Handoffs are first-class for triage → specialist routing.

specialist = Agent(name="Specialist", instructions="Answer technical AI agent questions.")
triage = Agent(name="Triage", instructions="Route technical questions.", handoffs=[specialist])
result = await Runner.run(triage, "What is ReAct for AI agents?")

Pick it for OpenAI-native products and fast handoff prototypes.

Google Agent Development Kit (ADK)

Google ADK targets Gemini agents on Vertex AI and Google Cloud — tool use, sub-agents, deployment to Cloud Run. Choose when your stack is GCP-first, and you want first-party Google tooling for evals and hosting.

Pydantic AI

Pydantic AI centers type-safe outputs — result_type=WeatherReport Validates structured responses. Excellent developer ergonomics for Python teams already using Pydantic v2.

class WeatherReport(BaseModel):
    location: str
    best_week: str
    confidence: float
    notes: str

agent = Agent("openai:gpt-4o-mini", result_type=WeatherReport, system_prompt="...")
result = agent.run_sync("Best surfing week in Greece?")

Pick Pydantic AI when schema correctness matters more than exotic orchestration.

LlamaIndex Agents

LlamaIndex began as RAG; its agent layer excels when retrieval is the core — document Q&A agents, knowledge-base tools, hybrid search. Pair with LlamaParse and workflow events for ingestion-heavy apps.

Semantic Kernel (Microsoft)

Semantic Kernel offers plugins, planners, and enterprise patterns in .NET and Python. Strong fit for Microsoft 365 , Azure AI, and orgs with existing SK investments.

Smolagents (Hugging Face)

Smolagents — lightweight, code-agent focused, Hugging Face hub models. Great for local/open models and teaching agents without heavy deps.

Amazon Bedrock Agents

Bedrock Agents — managed AWS service: action groups, knowledge bases, guardrails. Choose when you want AWS-managed scaling and IAM-native permissions, less custom loop code.

Mastra

Mastra — TypeScript-first agent framework with workflows, evals, and deployment story. Pick for Node/TS teams building product agents alongside Next.js apps.

Agno (formerly Phidata)

Agno — Python toolkit for multi-agent systems with memory, knowledge, and UI. Fast prototyping for agent OS style apps.

ServiceNow AI Agents

ServiceNow embeds agents in ITSM, HR, CSM workflows — Now Assist, flow designer integration, enterprise guardrails. Choose when the workflow already lives in ServiceNow; extend via Now Platform skills and data classes.

Hermes Agent

Hermes — self-hosted learning agent : SOUL.md identity, three memory tiers, self-evolving skills, Curator, optional GEPA, MCP-heavy profiles, gateway + cron. Best when you want an agent that improves over time on your machine.

Full tutorial: Hermes Agent Masterclass.

OpenClaw

OpenClaw — messaging-first gateway (WhatsApp, Telegram, Slack), ClawHub skills, proactive heartbeats. Best when channels and presence matter more than offline skill evolution. Compare: Hermes vs OpenClaw.

Framework selection (prose)

Explicit graphs, HITL, persistence → LangGraph
Role crews, content → CrewAI
OpenAI handoffs → OpenAI Agents SDK
Typed Python outputs → Pydantic AI
RAG-heavy → LlamaIndex
GCP / Gemini → Google ADK
AWS managed → Bedrock Agents
TypeScript product → Mastra
Self-hosted learning agent → Hermes
Messaging gateway → OpenClaw

Organizations deploying customer-facing agents often need more than orchestration alone. OpenClaw provides a messaging-first architecture with support for channels such as WhatsApp, Telegram, and Slack, enabling agents to operate continuously across real-world communication platforms while maintaining isolated sessions and tool access controls.

Link: https://techlatest.net/support/openclaw-support/

Part 18 — Environment setup

Prerequisites:

Python 3.11+
Optional: OPENAI_API_KEY for live LLM runs
Virtualenv recommended

cd guides/ai-agents-masterclass
python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt # optional deps per framework
cp .env.example .env # fill OPENAI_API_KEY if desired

Part 19 — Example 1: minimal ReAct agent

File: minimal_react_agent.py

No framework — pure Python demonstrating Think → Act → Observe. Uses stub weather_db and search_web tools. Set AGENT_GOAL in .env.

python examples/minimal_react_agent.py

Expected output: step trace ending in ✓ ReAct loop completed.

Step 02 — minimal ReAct run

Teaching point: understand the loop before adopting LangGraph or CrewAI abstractions.

Part 20 — Example 2: LangGraph research agent

File: langgraph_research_agent.py

Three-node graph: plan → research → synthesize. Writes report.md.

pip install langgraph langchain-core
export RESEARCH_TOPIC="AI agent governance"
python examples/langgraph_research_agent.py

Step 03 — LangGraph research agent

Extend with conditional edges: if research finds insufficient sources, loop back to research.

Part 21 — Example 3: CrewAI content crew

File: crewai_content_crew.py

Two agents — researcher and writer — sequential tasks. Demo mode writes stub blog_draft.md without API key.

pip install crewai
export CREW_TOPIC="Why AI agents need governance"
python examples/crewai_content_crew.py

With OPENAI_API_KEY, runs live crew and saves markdown output.

Step 04 — CrewAI content crew

Part 22 — Example 4: OpenAI Agents SDK handoffs

File: openai_agents_sdk.py

Async triage → specialist handoff via openai-agents.

pip install openai-agents
export OPENAI_API_KEY=sk-...
python examples/openai_agents_sdk.py

Step 05 — OpenAI Agents SDK handoff

Tracing in OpenAI dashboard shows handoff boundaries — use for debugging routing.

Part 23 — Example 5: Pydantic AI typed agent

File: pydantic_ai_typed_agent.py

Returns validated WeatherReport model — location, best_week, confidence, notes.

pip install pydantic-ai
python examples/pydantic_ai_typed_agent.py # demo stub without key
export OPENAI_API_KEY=sk-...
python examples/pydantic_ai_typed_agent.py # live validated run

Step 06 — Pydantic AI typed output

Use typed agents at API boundaries — downstream code consumes Pydantic models, not raw strings.

Part 24 — Smoke tests

Run the bundled pytest smoke tests (no API key required for stubs):

pip install pytest
pytest examples/tests/test_agents_smoke.py -v

Step 07 — run tests

Part 25 — Deploy to Google Cloud Run

Containerize your agent HTTP service or job runner. Cloud Run gives scale-to-zero, IAM, and VPC connectors for private DB access.

Outline:

Dockerfile — slim Python image, install deps, expose port 8080
Service — FastAPI or Flask wrapper around agent run() with run_id logging
Secrets — Secret Manager for OPENAI_API_KEY, not env files in image
Deploy — gcloud run deploy agent-service --source .
Background — Cloud Run jobs or Cloud Scheduler for cron agents

Cloud Run deployment — container to service

# Minimal Dockerfile sketch
FROM python:3.12-slim
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY . .
ENV PORT=8080
CMD ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8080"]

gcloud run deploy ai-agent-demo \
  --source . \
  --region us-central1 \
  --set-secrets OPENAI_API_KEY=openai-key:latest \
  --allow-unauthenticated # lock down in production

Production notes:

Set request timeout above worst-case agent duration or return 202 + poll
Use Cloud Logging for structured trace JSON
Attach service account with least privilege for GCP tools
Consider Cloud Armor if endpoint is public

Part 26 — Production checklist

Before shipping any agent to users:

Identity and auth — who can invoke which tools? Map OAuth subject → tool ACL.

Observability — structured logs, metrics (turns, latency, tool errors), distributed tracing.

Safety — input/output filters, blocked tool list, prompt injection tests on tool results.

HITL — approval queue for irreversible actions.

Cost controls — per-user budgets, model routing (small model for triage).

Data — PII redaction in logs, retention policy, regional storage.

Reliability — idempotent tools, retries with jitter, circuit breakers on flaky APIs.

Evals — golden tasks in CI; regression when prompts or tools change.

Incident response — kill switch to disable tool execution globally.

Documentation — runbooks for on-call when agent error rate spikes.

Moving from prototypes to production often requires workflow management, monitoring, and operational controls around agent systems. Dify AI provides a platform for building, deploying, evaluating, and monitoring AI agents and LLM applications, helping teams shorten the path from experimentation to production deployment.

Link: https://techlatest.net/support/difyai_support/

Part 27 — Building your own agent (checklist)

Define one measurable goal (surf week, ticket triage, report generation)
List tools with JSON schemas — prefer MCP servers for reuse
Choose ReAct vs ReWOO (or hybrid)
Pick framework from Part 17 or start with minimal loop
Add memory tier only when sessions need continuity
Instrument run_id and step logs from day one
Ship HITL before auto-executing side effects
Run smoke tests and golden evals
Deploy behind API with timeouts and secrets manager
Iterate from traces — most bugs are bad tool descriptions, not bad models

Part 28 — Connecting agents to MCP

Any framework above can call MCP tools if the host exposes them (Cursor, Claude Desktop) or you embed an MCP client in your runtime.

Pattern:

Run MCP server (stdio or HTTP)
Client handshake → discover tools
Map MCP tool schemas to your framework’s function format
Execute tool calls through MCP client

Cross-reference: MCP Visual Guide — Part 10–12.

Hermes profiles declare MCP in config.yamlLangGraph nodes can wrap MCP invocations in a dedicated tool node.

Part 29 — Multi-agent orchestration patterns

Supervisor — central node assigns subtasks, collects results. LangGraph Send API, AutoGen group chat.

Pipeline — fixed DAG, no dynamic routing. CrewAI sequential, ReWOO workers.

Handoff — conversational transfer with context pack. OpenAI Agents SDK.

Blackboard — shared state document agents read/write. Useful for research synthesis.

Pick supervisor when tasks are dynamic; pipeline when steps are known; handoff when user-facing role should change mid-session.

As multi-agent systems grow in complexity, visual orchestration becomes increasingly valuable. CrewAI Studio allows developers to design, coordinate, and monitor role-based agent teams without building orchestration infrastructure from scratch, making it a practical choice for research, content generation, and business workflow automation.

Link: https://techlatest.net/support/crewai-support/

Part 30 — Observability and debugging

Trace format (store as JSON lines):

{
  "run_id": "run_abc123",
  "turn": 3,
  "type": "tool_call",
  "tool": "weather_db",
  "args": {"location": "Greece"},
  "latency_ms": 142,
  "status": "ok"
}

Debug workflow:

Reproduce with frozen prompt + tool stubs
Diff tool schemas vs model-emitted args
Check observation truncation — did you cut off the JSON the model needed?
Lower temperature for routing; allow higher for creative synthesis steps

OpenAI Agents SDK and LangSmith offer hosted tracing; self-host with OpenTelemetry if required.

Part 31 — Cost and latency optimization

Route trivial questions to a small model without tools
Cache tool results (weather, FX rates) with TTL
Parallelize independent tool calls (ReWOO worker stage)
Summarize long observations before next turn
Cap max turns and fail gracefully with partial answer
Batch background agents off peak

Part 32 — Security deep dive

Tool privilege — separate read and write tools; never give shell and send_email to the same agent without HITL.

Prompt injection via tools — malicious webpage content instructs “ignore prior instructions.” Sanitize and summarize untrusted tool output.

SSRF — fetch_url tools must block metadata IPs and internal ranges.

Secrets — tools receive credentials from env/Secret Manager, not from model context.

Output — prevent agents from leaking system prompts or other users’ data in multi-tenant setups.

Part 33 — Evals and quality gates

Build a golden set of 20–50 tasks:

# evals/surf_goal.yaml
goal: "Best week for surfing in Greece next year"
expect_tools: ["weather_db", "search_web"]
rubric: "Must cite weather and tide reasoning; confidence stated"

Run in CI on prompt/tool changes. Track pass rate over time. Add adversarial cases (missing tool, API 500, empty search results).

Part 34 — When not to build an agent

Skip agents when:

Workflow is fully deterministic — use Zapier, Temporal, Airflow
Zero side effects — RAG chatbot suffices
Hard real-time — sub-100ms SLAs don’t fit LLM loops
Regulatory prohibition on autonomous action — keep human-only execution

Agents are a tool , not a mandate.

Part 35 — Roadmap: from demo to product

Week 1 — minimal ReAct + one real tool + logs

Week 2 — MCP server for tool isolation + HITL on writes

Week 3 — LangGraph or SDK with checkpointing + eval suite

Week 4 — Cloud Run deploy + secrets + monitoring dashboards

Ongoing — memory tier, multi-agent only when traces prove bottleneck

Summary

An AI agent pursues goals through a loop of reasoning, tool action, and observation — not a single chat completion. Persona, memory, tools, and model form the anatomy; ReAct and ReWOO offer two orchestration strategies; single vs multi-agent and surface vs background deployments match different products. Enterprise value spans six use-case families; governance (logs, HITL, unique run IDs) separates demos from production. Use MCP for tools and A2A for cross-agent tasks. Start with minimal_react_agent.py, graduate to LangGraph , CrewAI , OpenAI Agents SDK , or Pydantic AI as requirements sharpen, deploy on Cloud Run with secrets and evals, and extend with Hermes or MCP when you need learning loops or standardized tool wiring.

Thank you so much for reading

Like | Follow | Subscribe to the newsletter.

Catch us on

LinkedIn: https://www.linkedin.com/in/techlatest-net/

Techlatest.net - Instant RAGFlow: Ready-to-Use AI Knowledge Retrieval Engine

Reddit Community: https://www.reddit.com/user/techlatest_net/

Harness Engineering — Full Visual Guide

TechLatest — Tue, 16 Jun 2026 18:59:21 +0000

The model is smart. The harness makes it reliable. Build the environment around Claude Code, Codex, or any coding agent so multi-session work finishes with proof — not vibes.

What you’ll understand

Why the same model fails or succeeds based on harness — not IQ
The five subsystems : instructions, state, verification, scope, lifecycle
AGENTS.md as map , not encyclopedia — progressive disclosure via docs/
The 16-step session lifecycle agents should follow
Planner/generator/evaluator splits for long runs
Copy-ready templates to drop into your repo today

Introduction — it’s a harness problem

You give Claude or GPT a real task. It reads files, writes code, looks productive. Then it skips a step, breaks tests, says “done” — and nothing works. You spend more time rescuing than if you’d coded it yourself.

That’s not a model problem. It’s a harness problem.

Anthropic ran a controlled experiment: same model (Opus 4.5), same prompt (“build a 2D retro game editor”). Without harness: ~$9 in 20 minutes, broken output. With harness (planner + generator + evaluator): ~$200 in 6 hours, playable game. The model didn’t change. The environment did.

OpenAI reported the same shift with Codex: in a well-harnessed repo, reliability moves from “unreliable” to production-grade — not a marginal tweak, a qualitative jump.

Harness engineering = designing everything the model runs inside: instructions, state files, verification gates, scope boundaries, session lifecycle, hooks, sandboxes, observability.

Agent = Model + Harness
If you're not the model, you're the harness.

Harness pattern — task to verified done

Modern agent platforms such as OpenClaw extend this idea by providing persistent agent sessions, structured workflows, and runtime orchestration around foundation models. In practice, the harness often determines whether an agent completes work reliably or simply generates plausible output.

Link: https://techlatest.net/support/openclaw-support/

Part 1 — The harness pattern

You give a task. The agent:

Reads harness files (not your Slack thread)
Runs init.sh — install, health check
Picks one unfinished feature
Implements with verification loop
Stops only when tests/lint/types pass

The model decides what code to write.

The harness governs when, where, and how — and when “done” is allowed.

Part 2 — Five subsystems

Five subsystems — instructions through lifecycle

| Subsystem | Job | Artifacts |
|-----------|-----|-----------|
| Instructions | What to do, in what order, what to read first | `AGENTS.md`, `CLAUDE.md`, `docs/` |
| State | What's done, in progress, next | `feature_list.json`, `claude-progress.md`, git log |
| Verification | Proof before victory | tests, lint, typecheck, smoke, e2e |
| Scope | One feature at a time; real definition of done | feature list as machine-readable boundary |
| Lifecycle | Clean start and handoff | `init.sh`, wrap-up checklist, safe commit |

The harness doesn’t make the model smarter. It makes output reliable.

Part 3 — Without harness vs with harness

Without vs with harness — two session story

Without: Session 2 has no memory. Agent re-does work or wanders. You merge broken code.

With: Session 2 reads claude-progress.md, continues feature F03, verifies before claiming done. You review, not rescue.

Part 4 — AGENTS.md: map, not encyclopedia

The “one giant AGENTS.md” approach fails predictably:

Context is scarce — a 1,000-line manual crowds out the task
Everything “important” means nothing is
It rots — agents can’t tell what’s still true

Fix: ~100 lines AGENTS.md as table of contents. Deep truth lives in structured docs/ — design docs, architecture, exec plans, quality grades. Agent starts small, reads on demand.

OpenAI’s Codex team treats docs/ as a system of record ; linters and doc-gardening agents keep it fresh.

Part 5 — Session lifecycle (16 steps)

Session lifecycle flow

Start: Read harness → init.sh → progress log → feature list → git log

Select: Pick exactly one unfinished feature

Execute: Implement → verify → fix loop until green → record evidence

Wrap: Update progress + feature list → note broken/unverified → commit when safe to resume

Without harness, step “verify” becomes “agent says it looks fine.” With harness, it’s tests pass, lint clean, types check.

Part 6 — Scope and feature lists

feature_list.json is a harness primitive — machine-readable scope the agent can't hand-wave away.

Rules:

One passes: false feature active at a time
No rewriting the list to hide unfinished work
passes: true Only with evidence (test name, date, log snippet)

See feature_list.json.

{
  "app": "knowledge-base-desktop",
  "features": [
    {
      "id": "F01",
      "title": "Import local markdown files",
      "passes": true,
      "evidence": "tests/import.test.ts — 2026-06-01"
    },
    {
      "id": "F02",
      "title": "Document library list view",
      "passes": true,
      "evidence": "e2e/library.spec.ts"
    },
    {
      "id": "F03",
      "title": "Index documents for search",
      "passes": false,
      "notes": "in progress — indexer stub only"
    },
    {
      "id": "F04",
      "title": "Grounded Q&A with citations",
      "passes": false,
      "notes": "blocked on F03"
    }
  ],
  "next": "F03"
}

Part 7 — Verification and early victory

Agents declare victory too early because confidence ≠ correctness. Fixes:

Runnable proof required (not “I think it works”)
Full pipeline runs — unit + lint + typecheck + smoke
Separate evaluator agent — generation ≠ grading (Anthropic harness pattern)

Planner · generator · evaluator

Part 8 — Hooks and the ratchet

Hooks enforce what prompts merely suggest: pre-commit typecheck, block rm -rf, grep for .skip(, require approval before push.

Ratchet rule: every agent mistake becomes a permanent constraint :

Agent commented out a test → AGENTS.md rule + hook
Agent ignored architecture layer → custom linter
Stale docs → doc-gardening agent opens fix PR

Harness is shaped by your failure history — you can’t download someone else’s.

Part 9 — Agent legibility

If the agent can’t see it in-repo at runtime, it doesn’t exist. Slack decisions, Google Docs, tribal knowledge — illegible. Versioned markdown, schemas, plans, generated DB docs — legible.

Push context into the repo over time. Boring, composable stacks often beat clever abstractions agents can’t inspect.

Agents can only reason over information they can access at runtime. Retrieval systems such as Instant RAGFlow help surface relevant documentation, knowledge bases, and project context without forcing every detail into the model’s context window.

Part 10 — Production patterns (Codex / Claude Code)

Mature harnesses add:

Per-worktree app boot — agent drives UI via Chrome DevTools MCP
Local observability stack — LogQL/PromQL in the loop
Layered architecture — mechanical dependency rules + structural tests
Garbage collection — golden principles + recurring refactor agents
Minimal merge gates — high throughput; fix forward when agent volume exceeds human attention

Humans steer at intent and acceptance criteria. Agents execute and self-review in loops.

As teams move beyond single-agent workflows, orchestration platforms such as CrewAI Studio help coordinate planners, implementers, reviewers, and specialized agents while maintaining visibility into long-running tasks.

Techlatest.net - AI Agents using CrewAI Studio & Jupyter with GPU support

Workflow platforms such as Dify AI provide a practical way to package harnessed agents into production applications, combining tool integrations, evaluation flows, and operational monitoring.

Techlatest.net - Dify AI: Build & Launch GenAI Apps

Part 11 — Quick start (four files)

Drop into project root:

├── AGENTS.md
├── init.sh
├── feature_list.json
└── claude-progress.md

Drop templates into repo

Copy from examples/. Sessions stabilize immediately vs prompt-only.

Part 12 — Hands-on session

./init.sh # bootstrap + health
# agent picks ONE feature
npm test && npm run lint # verification gate
# update progress + feature_list
git commit # clean handoff

init.sh session start Verification gate — fail then pass Commit handoff

Many teams develop and validate harness workflows inside reproducible AI workbenches with integrated notebooks, terminals, and GPU access before deploying them into production agent environments.

Techlatest.net - Jupyter Python Notebook

Part 13 — Capstone context (knowledge base app)

The learn-harness-engineering course builds one Electron knowledge-base app across six projects — import docs, index, grounded Q&A with citations. Each project adds harness mechanisms; the app evolves as skills grow.

Same pattern works for any real repo: measured weak vs strong harness diff, not doc count.

Part 14 — Learning path (12 + 6)

Lectures L01–L12: capability gap → harness definition → repo as truth → progressive disclosure → multi-session state → init phase → scope → feature lists → verification → e2e → observability → clean handoff

Projects P01–P06: prompt-only vs rules-first → agent-readable workspace → continuity → runtime feedback → self-verification → full capstone

Part 15 — Who this is for

Yes: engineers using coding agents daily; tech leads owning agent reliability; builders who’ll let agents edit real repos

No: zero-code AI intro; prompt-only hobbyists; teams unwilling to add harness files to git

Requires: terminal, git, at least one of Claude Code / Codex / comparable agent CLI

Summary

Harness engineering is the discipline of making agents finish real work: map-not-encyclopedia instructions, disk-persisted state, verification before “done”, one-feature scope, structured session lifecycle, hooks that ratchet on every failure. The model gets the headlines. The harness gets the merge.

Thank you so much for reading

Like | Follow | Subscribe to the newsletter.

Catch us on

LinkedIn: https://www.linkedin.com/in/techlatest-net/

Reddit Community: https://www.reddit.com/user/techlatest_net/

Loop Engineering Explained Visually: From Manual Prompts to Goal-Driven AI Agents

TechLatest — Tue, 16 Jun 2026 06:31:23 +0000

Design AI systems that act, observe, and repeat until a goal is met — not one-shot prompts with you as the checkpoint between every step.

What you’ll understand at the end

Why manual prompt-review cycles hit a ceiling before model quality does
What a single-agent loop is — and when you need a fleet
Open vs closed loops — exploration vs production budgets
The five parts of a well-engineered loop (goal, tools, context, termination, errors)
Common patterns : retry, plan-and-execute-verify, explore-and-narrow, human-in-the-loop
How frameworks (LangGraph, Swarm, Hermes, OpenClaw) map to loop infrastructure

Introduction — you were the loop

For years the default workflow was identical whether you were drafting email or refactoring a repo:

Open chat
Type a request
Review output
Type the next request

You were the revision cycle. That made sense when models were unreliable — a human gate at every step stopped errors from compounding.

Models improved. The workflow didn’t. Loop engineering automates the checkpoint: you define the goal and the pass/fail standard; the agent runs research → produce → evaluate → fix → repeat until the bar clears or a stop rule fires.

This is the architecture behind serious coding agents (Claude Code, Codex-style agents, Hermes ReAct runtime) and production agentic workflows.

Part 1 — The one-task problem

Every time you prompt for the next micro-step, you decide things the agent should decide:

Where to look in the codebase
Whether the draft is good enough
What still needs work

That’s hiring a writer and approving every paragraph. You get output — but you’re running the operation , not delegating it.

The fix isn’t necessarily a bigger model. It’s rewiring the control flow from linear chat to a goal-driven loop.

Single agent loop — produce, check, fix, repeat

Part 2 — What a loop actually is

A loop is a repeating cycle:

Act — tool call, code write, search, shell command
Observe — stdout, test results, linter, API response
Reason — what failed, what to try next
Repeat until termination

This traces to ReAct (Reason + Act): interleave thinking with environment feedback instead of guessing once and stopping.

ReAct cycle — reason → act → observe

Analogy: A writer revising their own manuscript — draft, read with fresh eyes, mark weak sections, fix, read again — without asking the editor after every sentence. You hand over the revision cycle , not just the first draft.

Part 3 — What makes or breaks the loop

Almost none of the engineering is “pick a smarter model.” Two design choices dominate:

Evaluation gate — What counts as passing? Vague (“looks good”) → infinite loops or arbitrary stops. Concrete (“all pytest green + ruff clean”) → auditable exits.

Stopping condition — Success, max iterations, no-progress streak, escalation to human.

Eval gate — pass exits loop, fail retries or halts

See eval-gate.yaml for a harness template.

# Eval gate config — copy to your agent harness

goal: "All pytest tests in tests/ pass; ruff check src/ is clean"

success:
  - metric: pytest_exit_code
    equals: 0
  - metric: ruff_violations
    equals: 0

failure:
  max_iterations: 10
  no_progress_streak: 3 # same error 3x → stop and escalate

escalation:
  on_failure: human_review
  include: [iteration_log, last_patch, stack_trace]

context:
  summarize_every: 3 # compress loop history every N iters
  keep_last_errors: 5

Part 4 — When one agent isn’t enough

A single looping agent handles bounded tasks well. Real projects mix cognitive modes:

Research vs planning vs execution vs review
Long context → lost-in-the-middle — front and back of window get more attention

Forcing one agent to be researcher, planner, implementer, and reviewer is like asking your best writer to fact-check every claim, copy-edit, and run the press.

Fleet looping: an orchestrator owns the goal, decomposes work, assigns specialists , each running their own sub-loop. Subagents handle narrow slices. Eval gates at every layer stop bad work from propagating.

Fleet tree — orchestrator → specialists → subagents

Hermes masterclass (ReAct + 90-turn cap) · OpenClaw (gateway + multi-agent sessions).

Multi-agent systems require orchestration, session management, and reliable communication between specialized agents. Platforms such as OpenClaw provide a channel-first architecture for managing agent sessions, tool access, and long-running autonomous workflows.

Link: https://techlatest.net/support/openclaw-support/

Part 5 — Open loops vs closed loops

Open looping — wide operational space, vague path, room to explore. Can discover solutions you didn’t spec. On a research budget, exciting.

Costs: reasoning chains that go nowhere, context bloat, compounding API bills. Loose requirements → slop at scale — output that looks finished but misses the bar.

Closed looping — human architect defines path before execution: clear goal, defined steps, eval gate per step, explicit stop. Agents still loop — inside your frame.

Open vs closed loops — explore wide vs gated path

Failure contrast:

Open loop fails → keeps going, burns tokens, plausibly wrong output
Closed loop fails → stops at gate , trace shows where, fix eval and rerun

Production default: closed first. Expand operational space once the gated loop works.

Part 6 — Five parts of a well-engineered loop

Five parts — goal, tools, context, termination, errors

1. Clear goal — Specific enough to evaluate. “All unit tests pass” not “make the app better.”

2. Tool set — Loop quality = ability to touch reality : run code, read/write files, shell, tests, search docs. No tools → guessing loop.

3. Context management — Each iteration adds tokens. Summarize history, log attempts, prune noise before the next turn.

4. Termination logic — Success conditions, failure exits (max iters, repeated same error), escalation paths.

5. Error handling — Recoverable vs hard blockers; change strategy after repeated failure — not identical retries.

Many production agents rely on retrieval systems rather than storing all knowledge in model weights. RAG platforms such as Instant RAGFlow allow loops to fetch relevant information dynamically during execution.

Link: https://techlatest.net/support/ragflow_support/

Part 7 — Common loop patterns

Loop patterns — retry, plan-verify, explore-narrow, HITL

Retry loop — Try → check pass/fail → retry. Best for atomic tasks with clear criteria (one function + one test).

Plan-execute-verify — Plan steps, execute one, verify before next. Refactors, multi-file features. Must revise plan when step 2 invalidates step 5.

Explore-narrow — Try multiple approaches, score intermediates, commit to best path. Debugging unknown errors. Watch context explosion — prune early.

Human-in-the-loop — Pause on ambiguity or high-risk action; resume after approval. Production deploys, irreversible ops. Too many interrupts → you’re the loop again.

Part 8 — Frameworks and what they solve

Building loops from scratch is tedious. Frameworks differ in state, failure recovery, and debugging — not just syntax.

Framework loop infra — checkpoint, handoff, MCP, gateway

LangGraph — Loop as stateful graph ; checkpoint after each node; resume mid-crash without losing context. Long-running fleets.

OpenAI Swarm — Stateless handoffs ; full context passed each hop explicitly. Clean debugging, assembly-line workflows.

Microsoft Agent Framework — Async message passing; parallel branches; separate harness vs production loops with human review gates.

Anthropic / MCP — Standard tool discovery; orchestrator attaches capabilities without per-integration glue; interrupt before dangerous ops.

Hermes Agent — Synchronous ReAct core, skill learning, gateway + cron for proactive loops. See masterclass.

OpenClaw — Channel-first gateway, isolated agent sessions, skills + heartbeat. See masterclass.

Pick by failure modes your team can tolerate , not benchmark hype.

Teams moving from prototypes to production often use workflow platforms such as Dify AI to deploy agent pipelines, integrate tools, and monitor execution across real-world applications.

Link: https://techlatest.net/support/difyai_support/

Part 9 — Context and token hygiene

Each iteration appends: patches, stack traces, decisions. Unbounded history → token limits and forgotten early attempts.

Practices:

Structured feedback — relevant code snippet + intent + “same error as iter 3?” flag
Rolling summary — “Fix A failed (TypeError), Fix B partial, tests fail line 47”
Tool call budgets — max calls per iteration; budget exhaustion = failure signal
Summarize every N iterations — compress log, keep last K errors

Part 10 — Hands-on: minimal closed loop

minimal_closed_loop.py

#!/usr/bin/env python3
"""Minimal closed-loop coding agent — act, observe, retry until tests pass."""
from __future__ import annotations

MAX_ITER = 8

GOAL = "All unit tests pass with zero lint errors."

def run_tests() -> tuple[bool, str]:
    """Replace with pytest/subprocess in real projects."""
    import random
    ok = random.random() > 0.6 # demo: flaky until loop converges
    return ok, "FAILED: test_addition expected 4 got 3" if not ok else "OK: 12 passed"

def agent_step(iteration: int, last_error: str | None) -> str:
    """One LLM turn: propose a fix given feedback."""
    if last_error:
        return f"# iter {iteration}: patch based on → {last_error[:60]}"
    return f"# iter {iteration}: initial implementation"

def loop() -> None:
    error: str | None = None
    for i in range(1, MAX_ITER + 1):
        patch = agent_step(i, error)
        print(patch)
        passed, feedback = run_tests()
        print(f" eval: {feedback}")
        if passed:
            print(f"✓ {GOAL} (stopped at iteration {i})")
            return
        error = feedback
    print(f"✗ Escalate to human — no progress in {MAX_ITER} iterations")

if __name__ == " __main__":
    loop()

Closed loop demo — act, eval, retry until pass

The script loops: propose patch → run eval → exit on success or escalate after MAX_ITER.

Wire real run_tests() to pytest; replace agent_step() with your LLM + tool calls.

Part 11 — Hands-on: eval gate config

Copy eval-gate.yaml into your harness:

# Eval gate config — copy to your agent harness

goal: "All pytest tests in tests/ pass; ruff check src/ is clean"

success:
  - metric: pytest_exit_code
    equals: 0
  - metric: ruff_violations
    equals: 0

failure:
  max_iterations: 10
  no_progress_streak: 3 # same error 3x → stop and escalate

escalation:
  on_failure: human_review
  include: [iteration_log, last_patch, stack_trace]

context:
  summarize_every: 3 # compress loop history every N iters
  keep_last_errors: 5

success — measurable metrics (exit codes, counts)
failure — max iterations + no-progress streak
escalation — human review payload
context — summarize cadence

Eval gate terminal — metrics and stop rules

Part 12 — Multi-agent loop sketch

Orchestrator pseudoflow:

goal → decompose → for each subtask:
         assign specialist → specialist loops until sub-eval passes
       → integrator merges → global eval → done or rework branch

Multi-agent delegation terminal

Start single closed loop first. Add fleet when you hit context ceiling or role confusion.

As agent fleets grow, visual orchestration becomes increasingly valuable. CrewAI Studio enables developers to design, coordinate, and monitor multi-agent workflows without building orchestration infrastructure from scratch.

Link: https://techlatest.net/support/crewai-support/

Part 13 — Where to start

Build a loop when:

Same work type repeats, and quality should compound
Success is verifiable , not vibes
You spend time driving steps the agent could navigate

Don’t loop everything — one-shot summarization doesn’t need ten iterations.

Starter recipe:

Write termination condition on paper
Wire one eval gate (tests or schema validator)
Single agent, max 8–10 iterations
Log every iter; summarize history
Test failure cases before happy path

Install/scaffold loop harness

Part 14 — Failure modes checklist

Failure modes — runaway open vs halted closed

No exit condition — runs forever or stops randomly
Same error, same fix — spinning, not learning
Context overflow — model forgets task
Vague goal — can’t detect done
No tools — pure hallucination loop
Open loop + loose spec — expensive slop

Test deliberately: ambiguous goals, broken tools, unsolvable tasks (verify exit works).

Part 15 — Loop engineering vs agentic AI

Agentic AI — autonomous action toward goals (broad).

Loop engineering — discipline of structuring those actions in feedback cycles with explicit gates.

Most agentic systems are loops under the hood. Quality differences usually come from loop design , not base model alone.

Summary

Loop engineering moves you from expensive autocomplete to goal-driven automation. Define pass/fail gates and stop rules; let agents run the revision cycle. Start closed, single-agent ; add fleet and openness when evals prove the frame. The model got better — your workflow should too.

Thank you so much for reading

Like | Follow | Subscribe to the newsletter.

Catch us on

LinkedIn: https://www.linkedin.com/in/techlatest-net/