DEV Community: ryan2run

T-Mem: Memory That Anticipates, Not Archives - Tencent's Breakthrough in AI Long-Term Memory

ryan2run — Tue, 15 Sep 2026 02:04:35 +0000

T-Mem: Memory That Antanticates, Not Archives

The Problem with Current AI Memory

Large language model agents are increasingly entering long-term companion chat scenarios — not just answering one question and leaving, but building lasting relationships with users. This creates a specific requirement for memory systems: users don't repeat the same things year after year in the same words. The memory system must retrieve past memories even when phrasing, topics, and contexts have completely changed.

But today, almost all long-term memory solutions share the same fundamental assumption: as long as the query is "similar enough" to the memory, it can be retrieved. This only works when memory and dialogue have literal or semantic similarity. In real conversations, however, phrasing changes, contexts shift, yet memories still need to be recalled.

Tencent's Solution: T-Mem

T-Mem flips this assumption upside down: instead of struggling to find similarity at retrieval time, save preset scenarios as bridges at the moment of memory write-in. This way, even when memory and dialogue have zero semantic similarity, key memories can still be recalled through contextual association.

Key Insight: A long-term memory system earns its adaptive value not by archiving the dialogue stream faithfully, but by anticipating, at write time, the future cues under which its contents will need to be reached.

How T-Mem Works

The 2x2 Quadrant Framework

T-Mem models memory across two orthogonal axes:

Retrieval Direction: Descriptive recall (surface similarity) vs. Associative recall (latent connection)
Memory Granularity: Single fact vs. Complete scene

This creates four quadrants (QI-QIV), each with a trigger family:

QI Entity Trigger: Enriches single facts with generalized labels
QII Bridge Trigger: Predicts "in what context will this fact be needed?"
QIII Forward Trigger: Thinks several steps ahead along the conversation
QIV Scene Trigger: Writes complete scenes as multi-dimensional archives

The Example That Explains Everything

Consider this scenario:

Months ago: "Xiao Zhang is allergic to seafood, went to the hospital last week, need to be careful."
Now: "Where are we going for team building next week?"

These two sentences share ZERO semantic similarity. But the memory about Xiao Zhang's seafood allergy is CRITICAL for choosing a restaurant that avoids seafood.

With T-Mem, when "Xiao Zhang is seafood allergic" is written to memory, the system asks: "In what future scenarios will this information be used?" It pre-saves triggers for team building restaurant selection, health checkups, allergy history collection, etc.

When "Where are we going for team building?" arrives, T-Mem lights up the pre-saved trigger, retrieves the seafood allergy memory, and suggests: "How about not going to the seaside? Xiao Zhang can't eat seafood."

Performance Results

T-Mem achieves dual SOTA on two memory benchmarks:

Benchmark	T-Mem Score	Improvement
LoCoMo	80.26%	New SOTA
LoCoMo-Plus	74.81%	New SOTA

The real test is LoCoMo-Plus, which deliberately removes all lexical and semantic similarity between clues and answers. Mainstream systems drop 28-50 percentage points; T-Mem only drops 5.45 percentage points.

Implementation

Code: github.com/Sherlockwz/T-Mem
Paper: arxiv.org/abs/2606.15405
Team: Tencent PCG (郭伟东, 王达凯, 汪子轩, 刘辉, 徐羽)
Published: EMNLP 2026 Main Conference
Deployed: QQ AI Partner project

Conclusion

T-Mem demonstrates that associative memory is not an optional bonus feature, but the missing half of the similarity-based approach. Memory systems should anticipate how their contents will be reached, not just archive what was said.

Tags: AI, Memory, LLM, Tencent, Agent, EMNLP

PhysBrain 1.5: Chinese Physical AI Breakthrough - Physical Foundation Model Tops Open Source Rankings

ryan2run — Tue, 15 Sep 2026 02:04:12 +0000

PhysBrain 1.5: Chinese Physical AI Breakthrough

The Physical AI Race Heats Up

Physical AI is becoming the most critical track in global AI for 2026. While large models have spent three years pushing "can talk, can think, can work" to the extreme, the industry is now asking: Can models enter the real world? Can they take action?

Recent experiments have put this question front and center:

Gemini Robotics 2 (Google DeepMind, July 2026): Based on Gemini 3.5 Flash, uses "embodied reasoning" brain ER 2 to make robots "watch videos and plan next steps"
GPT-6 Astra (OpenAI): Pushed the ceiling of general intelligence higher with breakthroughs in reasoning, coding, and multimodal understanding
Robocurve Experiment: Deployed GPT-6 Astra on a real robotic arm — completed 19/20 "put blocks in bowl" tasks, while Claude Fable 5.1 only completed 8

But top LLMs still have a gap in physical manipulation. On millimeter-level precision tasks like "blue puzzle piece alignment insertion," GPT-6 Astra's success rate drops from 95% to 10%.

Enter PhysBrain 1.5

On September 9, 2026, Chinese company DeepCybo (深度机智) released PhysBrain 1.5, a physical foundation model that achieves 72.5 average score across 28 public benchmarks — ranking #1 among open-source models, just 1 point behind top closed-source models GPT-6 Astra (73.3) and Gemini 3.6 Flash (73.0).

Key Achievements

Metric	Result
Open Source Rank	#1
Average Score (28 benchmarks)	72.5
Open Source #1	14 benchmarks
Open Source #2	10 benchmarks
vs. Hy-Embodied-VLM-1.0 (66.0)	+6.5 points
vs. RynnBrain 1.1 (63.1)	+9.4 points

Two Model Sizes

PhysBrain 1.5-2B: 66.6 average — exceeds all non-PhysBrain open models
PhysBrain 1.5-8B: 72.5 average — tops open source, nears closed-source leaders

Both models are fully open-source with Apache 2.0 license.

The Physical Loop Architecture

PhysBrain 1.5 is built around a unified architecture called Physical Loop:

Observe the world — perceive the environment
Understand space and tasks — comprehend spatial relationships
Judge action consequences — predict outcomes
Execute — take physical action
Correct based on feedback — self-correct from new observations

This is a continuous, stable closed-loop system that self-corrects based on feedback.

Three Core Capabilities

Embodied Understanding: Knows "what is happening now" — outputs structured spatial coordinates and trajectory targets
Action Generation: Knows "what to do next" — learns from human video, adapts to different robot types
Future State Prediction: Foresees "what happens after" — predicts 1-second physical state in RGB, depth, and robot mask

The Human Learning Approach

DeepCybo's key insight: Physical intelligence cannot be achieved by translating internet text into actions. Humans grow operational skills through repeated "see — try — get feedback — correct."

The team built Ego360, a human panoramic real-data system, using full-body pose, hand movement, and task-level voice from panoramic video. Physical pre-training supervision comes entirely from human interaction videos.

Open Source & Community

Project Page: deepcybo-physai.github.io/PhysBrain-1.5
Hugging Face: huggingface.co/collections/DeepCybo/physbrain-15
Eval Kit: github.com/DeepCybo-PhysAI/PhysBrainEvalKit
Demo: huggingface.co/spaces/hugging-apps/physbrain1-5-8b-demo

Within 3 days of launch, PhysBrain 1.5 received 3,000+ downloads on Hugging Face.

Conclusion

PhysBrain 1.5 demonstrates that physical foundation models are not a short-term trend but a long-term strategic investment. By focusing on human learning patterns and building a complete data-model-body loop, DeepCybo has positioned PhysBrain at the forefront of global physical AI.

Tags: AI, Robotics, PhysicalAI, OpenSource, DeepLearning

Iris Search Agent: Open Source Context Management Beats Parameter Scaling

ryan2run — Tue, 15 Sep 2026 02:03:50 +0000

Iris Search Agent: Open Source Context Management Beats Parameter Scaling

The Open Source Search Agent Revolution

AllSpark Research has released Iris, an open-source search agent that is changing the competitive landscape. Two models are available:

Model	Total Params	Active Params	BrowseComp
Iris-mini	35B	3B	82.2
Iris-pro	397B	17B	88.6

Both use Mixture-of-Experts architecture with 256K context windows. Weights are released under Apache 2.0.

Why Search Agents Are Different

Search agents differ from regular chat models because they must:

Decide what to search — autonomously choose queries
Evaluate results — determine if more searching is needed
Know when to stop — recognize when evidence is sufficient

For two years, this capability was dominated by closed-source products. Open source had little to compete with.

Benchmark Results

Four benchmarks, impressive results:

Benchmark	Iris-mini	Iris-pro
BrowseComp	82.2	88.6
BrowseComp-ZH	84.8	85.1
DeepSearchQA	86.9	92.9
HLE	52.3	56.4

The Surprising Chinese Result

On BrowseComp-ZH (Chinese), the 35B model scores 84.8 and the 397B model scores 85.1 — a difference of only 0.3 points. The same models on English BrowseComp show a 6.4 point gap.

Key insight: For Chinese tasks, parameter scale barely matters. Context management is the real differentiator.

Context Management Strategy

The paper compares three inference-time context strategies:

No management — baseline
Discard-all — clear tool call history when context exceeds threshold, restart from original question
Retry — summarize eliminated clues before continuing

Result: Both models improved with context management, and Iris-mini improved MORE than Iris-pro.

For small models, managing the ever-growing search history is more effective than adding parameters.

Training Data: Reverse-Engineered from Hyperlinks

Multi-hop search questions are hard to create. Writing a question that requires crossing 3-4 webpages is expensive, and models can cheat with string matching.

Iris takes a different approach:

Pick a seed page from web corpus hyperlink structure
Follow outgoing links to build an entity graph
Create multi-hop chains on the graph
Rewrite questions to ensure NO clue can be solved by literal matching

SFT-RL Climbing

Training uses a process called SFT-RL climbing: supervised fine-tuning and reinforcement learning alternate. The RL phase runs on real search, not offline snapshots. The reward model and observation summarization modules are deployed on the training cluster.

"Our policy is optimized by RL against live search."

This means every training step makes real network requests — significant engineering cost.

Why This Matters

Open source has caught up to closed source in general chat and code. Search has been the laggard.

The reason is clear: search agent capability is half model, half tool orchestration framework. Closed-source products tune both; open source often releases only the model.

Iris releases model weights, context management strategy, and data construction method — the complete package.

Community & Availability

Hugging Face: Models available for download
GitHub: Training pipeline code marked "coming soon"
License: Apache 2.0 — commercial use allowed
Reference: arXiv 2609.04304

For anyone wanting to run deep search, the 35B/3B spec is a sweet spot — runs on a single machine with the paper's context management.

Conclusion

Iris proves that open source finally has a serious competitor in search. The combination of model weights, context strategy, and data methodology closes a gap that has persisted for two years.

Tags: AI, SearchAgent, OpenSource, LLM, Agent

AI Agent Architecture Patterns: A Deep Dive into Modern Agent Design

ryan2run — Mon, 14 Sep 2026 00:30:44 +0000

AI Agent Architecture Patterns: A Deep Dive into Modern Agent Design

AI agents are transforming how we interact with technology. But behind every smart agent lies a carefully designed architecture. In this article, we explore the key patterns that power modern AI agents.

What is an AI Agent?

An AI agent is a system that can perceive its environment, make decisions, and take actions to achieve specific goals. Unlike traditional chatbots, agents can:

Plan multi-step tasks
Use external tools and APIs
Learn from feedback
Collaborate with other agents

Key Architecture Patterns

1. ReAct (Reasoning + Acting)

The ReAct pattern combines reasoning and acting in a loop:

Observe the current state
Reason about what to do next
Act using available tools
Observe the result
Repeat until the goal is achieved

This pattern is powerful because it allows agents to handle complex, multi-step tasks.

2. SOP (Standard Operating Procedure)

SOP agents follow predefined procedures for specific tasks. Think of it as a decision tree:

Define clear steps
Specify conditions for each branch
Allow tool usage at each step

This approach is great for tasks that require consistency and reliability.

3. Reflection

Reflection agents can self-correct by reviewing their own outputs:

Generate a solution
Critique the solution
Revise based on feedback
Repeat until satisfied

This self-improvement loop leads to higher quality outputs.

4. Multi-Agent Systems

The most powerful agents work in teams:

Planner: Breaks down complex tasks
Executor: Performs specific actions
Critic: Reviews and provides feedback
Coordinator: Manages communication

Each agent has a specialized role, leading to better outcomes.

Choosing the Right Architecture

Pattern	Best For	Complexity
ReAct	Complex reasoning tasks	Medium
SOP	Repetitive workflows	Low
Reflection	Quality-critical tasks	Medium
Multi-Agent	Large-scale projects	High

The Future of Agent Architecture

As AI advances, we expect to see:

More sophisticated planning capabilities
Better tool integration
Improved memory systems
Enhanced collaboration between agents

The key is choosing the right architecture for your use case.

Conclusion

AI agent architecture is a rapidly evolving field. By understanding these patterns, you can design more effective and reliable agents.

What architecture pattern do you find most interesting? Share your thoughts in the comments!

Tags: AI, Agents, Architecture, Machine Learning, AI Design

LLM Inference Optimization: Techniques for Faster and Cheaper AI

ryan2run — Mon, 14 Sep 2026 00:30:07 +0000

LLM Inference Optimization: Techniques for Faster and Cheaper AI

Large Language Models are powerful, but they can be slow and expensive. In this article, we explore practical techniques to optimize LLM inference.

Why Optimize LLM Inference?

As AI applications scale, inference costs and latency become critical bottlenecks. Optimization helps you:

Reduce response times
Lower computational costs
Scale to more users
Deploy on edge devices

Key Optimization Techniques

1. Quantization

Quantization reduces the precision of model weights:

INT8: 8-bit integers (4x speedup)
INT4: 4-bit integers (8x speedup)
FP8: 8-bit floating point

Trade-off: Slight accuracy loss for massive speed gains.

2. KV Cache Optimization

KV Cache stores attention computations:

PagedAttention: Memory-efficient caching
Sliding Window: Limited context windows
Compression: Reduce cache size

Result: Faster generation for long contexts.

3. Speculative Decoding

Use a smaller model to draft tokens:

Small model drafts multiple tokens
Large model verifies in parallel
Accept or reject drafts

Speedup: 2-3x without quality loss.

4. Prompt Optimization

Better prompts mean fewer tokens:

Compression: Remove redundancy
Structure: Clear formatting
Examples: Few-shot learning

5. Batch Processing

Process multiple requests together:

Dynamic batching
Padding optimization
Memory pooling

Performance Metrics

Technique	Speed	Cost	Quality
Quantization	4x	75% less	Minor loss
KV Cache	2x	50% less	None
Speculative	2.5x	60% less	None
Prompt Opt	1.5x	33% less	None

Implementation Tips

Start with KV Cache (easiest win)
Add quantization for edge deployment
Use speculative decoding for throughput
Optimize prompts for cost savings

The Future

Expect even more optimization techniques:

Hardware-specific kernels
Dynamic routing
Neural architecture search
Hybrid approaches

Conclusion

Optimization is not a one-size-fits-all solution. Choose techniques based on your priorities: speed, cost, or quality.

What optimization technique has worked best for you? Share your experience!

Tags: AI, LLM, Optimization, Machine Learning

AI Model Evaluation: Best Practices for Testing and Validation

ryan2run — Mon, 14 Sep 2026 00:29:37 +0000

AI Model Evaluation: Best Practices for Testing and Validation

Evaluating AI models is critical for ensuring quality, safety, and reliability. In this article, we explore best practices for model evaluation.

Why Evaluate AI Models?

AI models can make mistakes, show bias, or behave unexpectedly. Evaluation helps you:

Ensure model quality
Detect bias and fairness issues
Verify safety standards
Measure real-world performance

Evaluation Framework

1. Benchmarks

Standardized tests for model capabilities:

MMLU: Knowledge and reasoning
HumanEval: Code generation
GSM8K: Math problem solving
SuperGLUE: Language understanding

Benchmarks provide objective, comparable metrics.

2. Red Teaming

Adversarial testing to find weaknesses:

Prompt injection: Test for security
Jailbreak: Test for safety
Edge cases: Test for robustness
Bias detection: Test for fairness

Red teaming reveals vulnerabilities before deployment.

3. User Testing

Real-world usage feedback:

A/B testing: Compare model versions
User surveys: Gather subjective feedback
Usage analytics: Track real patterns
Error analysis: Study failure cases

User testing provides ground-truth insights.

Evaluation Metrics

Metric	What It Measures	Importance
Accuracy	Correct predictions	High
Latency	Response time	Medium
Fairness	Bias detection	High
Robustness	Error handling	High
Safety	Harm prevention	Critical

Best Practices

Multi-dimensional evaluation: Test across many dimensions
Continuous testing: Evaluate regularly, not just once
Human-in-the-loop: Combine automated and human review
Document results: Track improvements over time
Share findings: Learn from each other

Tools and Frameworks

MLflow: Experiment tracking
Weights & Biases: Model monitoring
DeepEval: Evaluation framework
LangSmith: LLM testing

The Future

Expect more sophisticated evaluation:

Automated red teaming
Real-time monitoring
Dynamic benchmarks
Community-driven evaluation

Conclusion

Model evaluation is not a one-time task. It's an ongoing process that requires multiple approaches.

What evaluation methods have you found most effective? Share your insights!

Tags: AI, Evaluation, Machine Learning, Testing

DeepSeek R1: The Open-Source Reasoning Revolution That Changes Everything

ryan2run — Sun, 13 Sep 2026 03:47:54 +0000

DeepSeek R1: The Open-Source Reasoning Revolution That Changes Everything

The Reasoning Problem

Traditional LLMs generate text token by token, left to right. This autoregressive approach works for simple tasks but struggles with complex reasoning, math, and multi-step logic.

The core problem: How do you get an LLM to think before answering?

The Solution: Mixture of Experts (MoE)

DeepSeek R1 uses a Mixture of Experts architecture combined with Reinforcement Learning from Reasoning Feedback (RLRF) to achieve:

Fast inference — Only activate relevant experts per query
Deep reasoning — Chain multiple reasoning steps internally
Open weights — Anyone can download and fine-tune

How MoE Works

Input arrives at the router
Router selects the top-k experts for this specific query
Experts process in parallel (math, code, logic, science)
Aggregator combines outputs into a coherent response

This is dramatically more efficient than activating all parameters for every query.

Performance Benchmarks

Benchmark	DeepSeek R1	GPT-4	Claude 3.5
Math (AIME)	79.4%	83.0%	81.0%
Coding (LiveCode)	61.2%	65.0%	63.0%
Reasoning (GPQA)	74.8%	78.0%	76.0%

Key insight: Open-source models are now competitive with and sometimes surpassing closed models on reasoning tasks.

Why This Matters

Accessibility — Anyone can download and run R1 locally
Transparency — Open weights mean open reasoning
Innovation — Researchers can fine-tune for specific domains
Cost — Open models reduce dependency on expensive APIs

The Road Ahead

With MoE plus RLRF, the gap between open and closed models continues to narrow. The next frontier? Multi-modal reasoning — combining text, vision, and audio into unified reasoning pipelines.

What reasoning benchmarks matter most to you? Share your thoughts below.

Multi-Modal AI: From Text to Vision and Beyond — The Unified Future

ryan2run — Sun, 13 Sep 2026 03:47:24 +0000

Multi-Modal AI: From Text to Vision and Beyond — The Unified Future

The Single-Modality Limit

For years, AI models were single-modality — text-only, image-only, or audio-only. This created silos:

A text model cannot see images
An image model cannot hear audio
Each modality required separate training

The problem: Real-world understanding is inherently multi-modal.

The Breakthrough: Unified Encoders

Modern multi-modal models use a shared latent space — a single representation that encodes text, images, audio, and video into a common format.

How It Works

Each modality has its own encoder (text tokenizer, image CNN, audio encoder)
Projections map each encoder output into the shared latent space
A unified transformer processes all modalities together
Task heads generate outputs in any modality

Why This Matters

Cross-modal retrieval: Search images with text queries
Visual question answering: Ask questions about images
Image captioning: Generate descriptions from visual input
Text-to-image generation: Create visuals from text prompts
Video understanding: Combine temporal plus visual plus audio signals

Real-World Applications

Domain	Application	Impact
Healthcare	Medical image plus report analysis	Better diagnostics
Education	Visual plus text learning	Personalized tutoring
Robotics	Vision plus language plus action	Autonomous navigation
Content Creation	Text-to-video plus audio	Creative automation

The Future: True Multimodal Intelligence

The next generation will feature:

Real-time multi-modal streaming — Process video, audio, and text simultaneously
Cross-modal generation — Generate video from text, audio from images
Embodied AI — Robots that see, hear, speak, and act
Human-level understanding — Context-aware across all sensory modalities

Which multi-modal application excites you most? Let us know in the comments.

AI Safety and Alignment: Building Trustworthy Agents That Do Not Fail You

ryan2run — Sun, 13 Sep 2026 03:46:58 +0000

AI Safety and Alignment: Building Trustworthy Agents That Do Not Fail You

The Trust Problem

As AI agents become more capable, trustworthiness becomes the critical differentiator. A model that is smart but unreliable is worse than useless — it is dangerous.

The Safety Pyramid

Building trustworthy AI requires layered defense:

Level 1: Technical Robustness

Error handling and edge case coverage
Input validation and sanitization
Graceful degradation under stress

Level 2: Interpretability

Model transparency and explainability
Activation visualization and probing
Mechanistic interpretability research

Level 3: Content Safety

Harmful output filtering
Toxicity detection and prevention
Bias mitigation and fairness

Level 4: Instruction Following

Accurate task completion
Refusal of harmful requests
Context-aware compliance

Level 5: Value Alignment

Human preference learning (RLHF)
Constitutional AI principles
Multi-stakeholder value balancing

Level 6: Robustness

Adversarial attack defense
Distribution shift handling
Out-of-distribution generalization

Why Each Layer Matters

Without Level 1, the system crashes on edge cases.
Without Level 2, you cannot debug failures.
Without Level 3, the system generates harmful content.
Without Level 4, the system ignores user intent.
Without Level 5, the system pursues wrong goals.
Without Level 6, the system fails in production.

Practical Safety Measures

Red teaming — Actively try to break your system
Evaluation benchmarks — Measure safety, not just accuracy
Human-in-the-loop — Keep humans in the decision loop
Monitoring — Track model behavior in production
Rollback plans — Have kill switches ready

The Bottom Line

Safety is not a feature — it is a foundation. Every AI system, regardless of capability, must be built on these layered principles.

What safety measures have you implemented? Share your experiences below.

AI Agent Tool Mastery: How Modern Agents Choose and Use Tools Effectively

ryan2run — Fri, 11 Sep 2026 02:17:32 +0000

AI Agent Tool Mastery: How Modern Agents Choose and Use Tools Effectively

The Tool Selection Problem

When an AI agent receives a complex request like "Find the weather, book a flight, and send a confirmation email," it must do more than just generate text. It needs to:

Understand intent — What does the user actually want?
Match tools — Which tools can fulfill each part of the request?
Chain operations — How do results from one tool feed into the next?
Handle failures — What happens when a tool fails or returns unexpected data?

The Architecture of Tool Selection

Modern AI agents use a multi-stage pipeline to select and execute tools:

Stage 1: Intent Analysis

The agent parses the user request to identify:

Primary goals (e.g., "get weather," "book flight")
Secondary goals (e.g., "send confirmation")
Constraints (e.g., "tomorrow morning," "economy class")

Stage 2: Tool Matching

Each identified goal is matched against available tools using:

Semantic similarity (does the tool description match the intent?)
Historical performance (has this tool succeeded for similar requests?)
Availability (is the tool currently accessible?)

Stage 3: Execution Chaining

Results from one tool may trigger additional tool calls:

Weather data → Flight search with date constraints
Flight results → Booking API with passenger details
Booking confirmation → Email tool with itinerary

Performance Metrics That Matter

Not all tools are created equal. Here's how different tool categories perform:

Metric	Search	Weather	Booking	Email
Accuracy	92%	88%	76%	95%
Speed	78	95	65	90
Reliability	88	92	70	96

Key Insight: High accuracy doesn't always mean high speed. Agents must balance these trade-offs based on context.

The Evolution of Agent Tools

The capability of agent tools has evolved dramatically:

2020: Basic functions — simple API calls with fixed parameters
2021: Static tools — predefined toolsets with limited flexibility
2022: Dynamic selection — agents could choose tools based on context
2023: Multi-tool chaining — agents could sequence multiple tool calls
2024: Self-improving tools — tools that learn from past failures
2025: Autonomous tool creation — agents that write their own tools
2026: Adaptive ecosystems — tools that evolve based on user behavior

Practical Example: A Day in the Life of an Agent

Let's walk through a real-world scenario:

User Request: "I need to travel to Tokyo next week for a conference. Find flights, book the cheapest option, and send the itinerary to my assistant."

Agent Actions:

Search API — Finds conference dates and location
Weather API — Checks Tokyo weather for next week
Flight Search API — Finds available flights
Booking API — Books the cheapest flight
Email API — Sends itinerary to assistant
Calendar API — Adds event to user's calendar

Result: A fully automated travel booking experience.

Common Pitfalls and How to Avoid Them

Pitfall 1: Over-Tooling

Using too many tools for simple tasks slows down responses.

Solution: Implement a complexity threshold — only chain tools when necessary.

Pitfall 2: Ignoring Tool Failures

When a tool fails, agents should have fallback strategies.

Solution: Implement retry logic, alternative tools, and user notifications.

Pitfall 3: No Performance Monitoring

Without tracking tool performance, agents can't optimize.

Solution: Log all tool calls, measure success rates, and adjust weights.

The Future of Agent Tools

As AI agents become more sophisticated, tool ecosystems will evolve:

Self-optimizing tools that adjust parameters based on usage patterns
Cross-platform tools that work across different services seamlessly
User-adaptive tools that learn individual preferences over time
Collaborative tools that work together across agent boundaries

Conclusion

AI agent tool mastery isn't just about having access to many tools — it's about selecting the right tools, chaining them effectively, and continuously improving based on performance data.

The agents that thrive will be those that treat tools not as static resources, but as dynamic components of an adaptive ecosystem.

What tool challenges have you encountered with AI agents? Share your experiences in the comments.

HarnessDev: Self-Evolving Agent Frameworks — How LLMs Build Their Own Infrastructure

ryan2run — Fri, 11 Sep 2026 02:07:56 +0000

HarnessDev: How LLMs Are Building Their Own Agent Frameworks

ByteDance's Breakthrough in Self-Evolving Agent Systems

Published: September 10, 2026 | Reading time: 12 minutes

The Revolutionary Research

Last week, ByteDance's Seed team, in collaboration with Singapore University of Technology and Design, Georgia Tech, and other institutions, released HarnessDev — a groundbreaking research project that answers a fundamental question in AI agent engineering:

Can LLMs create their own Agent Harnesses and continuously improve them based on task feedback?

The answer is a resounding yes.

What Is an Agent Harness?

An Agent Harness is the core control system that drives an AI agent. It includes:

Task execution loops
Tool selection and parameter constraints
Context management
State tracking
Result verification
Failure recovery mechanisms

Think of it as the "operating system" for an AI agent — without it, the agent is just a language model with no structure or direction.

The Two-Phase Process

HarnessDev divides the agent development process into two phases:

Phase 1: Creation

Starting from a Weak Seed Harness (a minimal framework with basic I/O capabilities), the LLM builds a complete agent harness by adding control logic for:

Execution — Task loops, planning, scheduling, and stopping conditions
Tools — Tool selection, parameter constraints, input/output handling, and error processing
Context — Organization of task information, code, history, and constraints
State — Current goals, progress tracking, attempt records, and failure information
Lifecycle — Timeout handling, recovery mechanisms, and task cleanup
Verification — Testing, result checking, completion determination, and logging

Phase 2: Evolution

Using the created harness as a starting point, the LLM continuously adjusts it based on downstream task feedback, evaluating performance on held-out tasks.

Key Findings

1. LLMs Can Build Effective Harnesses

18 Code Harnesses were created, adding a total of 17,111 lines of code
Gemini required the fewest changes (1,006 lines) but achieved the highest score on Terminal-Bench 2.1 (68.8)
All 18 harnesses implemented Execution Loops; Tools, Lifecycle, and Verification had high completion rates

2. Not All Code Is Used

Out of 108 component instances in Code Harnesses, only 72 were observed running in real tasks
18 components (all from State and Memory) never appeared in actual execution
26,679 task trajectories recorded zero checkpoint events, despite some harnesses implementing checkpoint logic

This reveals a critical insight: implementing a mechanism doesn't mean it's actually used.

3. The Verification Gap

In self-evaluation, Opus found that out of 100 runs, the harness reported success 99 times, but only 48 were actually correct. This led to the addition of a Completion Check mechanism.

Similarly, in Data tasks, 441 out of 2,325 executions produced degraded commits, but the harness failed to detect them.

4. Cross-Model Adaptation Challenges

When switching executors, performance varies significantly:

Opus's SWE-Pro Harness: 69.3 (Self-Eval) → 33.0 (Gemini executor)
Qwen Harness: Improved by 17.6 points on BrowseComp when using Gemini

This shows that harnesses become executor-specific over time, requiring re-tuning when switching models.

5. Evolution Limitations

5 evolution trajectories improved on the visible feedback set
However, improvements on held-out tasks were much smaller (average 3.11 points)
Only 53.1% of version changes showed consistent direction between feedback and held-out sets
Evaluation fluctuation is approximately ±4.75 points, making it hard to distinguish real improvements from noise

The Six Control Capabilities

HarnessDev categorizes agent control into six capabilities:

Capability	Description	Example
Execution	Task loops, planning, scheduling	When to stop, how to plan
Tools	Tool selection, parameters, error handling	Which API to call, how to handle errors
Context	Organization of task info and history	What context to provide the LLM
State	Goals, progress, failure records	Current state, attempt history
Lifecycle	Timeout, recovery, cleanup	Handle failures, recover from errors
Verification	Testing, result checking, logging	Verify results, log outcomes

Execution Cost Analysis

Different execution strategies significantly impact token consumption:

GPT-5.5 Harness: 29.3M tokens, medal rate 19.1
DeepSeek V4 Harness: 208.4M tokens, score 19.6

Same performance, 7x token difference!

Across the entire MLE-bench experiment, token overhead varied by 19x between different harnesses.

This highlights the importance of cost-aware design — a small performance improvement may not justify a large token increase.

Code Example: Creating a Simple Harness

# Weak Seed Harness (minimal framework)
class WeakSeed:
    def __init__(self):
        self.task = None
        self.state = "initial"
        self.history = []

    def execute(self, task):
        self.task = task
        self.state = "running"
        # Basic execution loop
        while self.state == "running":
            action = self.plan_action()
            result = self.call_tool(action)
            self.record_history(result)
            if self.is_complete():
                self.state = "completed"

    def plan_action(self):
        # LLM generates action
        pass

    def call_tool(self, action):
        # Execute tool
        pass

    def is_complete(self):
        # Check completion
        pass

# LLM enhances this with control logic
# (Execution, Tools, Context, State, Lifecycle, Verification)

Why This Matters

For AI Researchers

Shows LLMs can self-improve their own execution frameworks
Highlights the gap between implemented and actually used mechanisms
Reveals the importance of verification and cross-model adaptation

For AI Practitioners

Demonstrates the value of structured agent design over pure memory
Shows the importance of cost-aware optimization
Highlights the need for robust verification mechanisms

For the Industry

Represents a step toward self-evolving AI systems
Shows the potential for automated agent development
Highlights challenges in generalization and adaptation

Conclusion

HarnessDev represents a significant step toward self-evolving AI agents. However, several challenges remain:

Implementation vs. Usage Gap — Not all implemented mechanisms are actually used
Cross-Model Adaptation — Harnesses become executor-specific
Evolution Limitations — Improvements don't always generalize to new tasks
Cost Awareness — Performance gains may come with disproportionate token costs

The research provides valuable insights for building more robust, efficient, and self-improving AI agents.

This article is based on research published by ByteDance Seed team on September 8, 2026. Paper: arXiv:2609.01437 | Project: self-developing-agents.github.io

Procedural Graphs: Self-Improving LLM Agent Execution Structures

ryan2run — Fri, 11 Sep 2026 02:05:56 +0000

Procedural Graphs: Self-Evolving Execution Structures for LLM Agents

When AI Agents Start Writing Their Own "Brain Circuits"

Published: September 10, 2026 | Reading time: 12 minutes

The Revolutionary Research

On September 9, 2026, researchers Yuxing Lu, Yicheng Chen, and Shanchan Wu published a groundbreaking paper on Procedural Graphs — a self-evolving execution structure for LLM agents that can literally rewrite its own "brain circuits."

Paper: arXiv:2609.08593

The Problem with Current LLM Agents

Today's LLM agents (like AutoGPT, LangChain agents) work like this:

Maintain a growing memory of past thoughts, observations, actions
Generate next action based on this memory
No fixed structure, no predefined流程

This works for simple tasks, but breaks down for complex ones:

Lost goals: Forget what they're supposed to do in long interactions
Action mismatch: Call tools in wrong order (e.g., analyze before searching)
Repeated labor: Try same ineffective operations repeatedly
No planning: No global view, step-by-step navigation

Analogy: Like a chef without a recipe — overwhelmed by complex dishes.

The Solution: Procedural Graphs

Procedural Graphs organize procedural knowledge ("how to do") just like Knowledge Graphs organize factual knowledge ("what is"):

Type	Structure	Question
Knowledge Graph	(Entity, Relation, Entity)	What is?
Procedural Graph	(Procedure, Relation, Procedure)	How to?

Core Components

Nodes (程序步骤):

Each node represents a procedural step
Contains: description, expected I/O, success/failure conditions
Like a box in a flowchart

Edges (关系):

Represent relationships between steps
Types: Sequential ("then"), Conditional ("if...then"), Parallel ("simultaneously")
Like arrows in a flowchart

Attributes (属性):

Execution probability, average time, success rate, common error patterns
Update with execution experience

How It Works: A Concrete Example

Task: "Book a cheap flight from Beijing to Shanghai, departing tomorrow."

Procedural Graph:

[Start]
   ↓
[Search Flight Info]
   ↓
[Filter Low-Cost Options]
   ↓
[Check Seat Availability]
   ↓
[If Available: Fill Passenger Info]
   ↓
[Select Payment Method]
   ↓
[Confirm Order]
   ↓
[End]
   ↓
[If Unavailable: Return to Filter]
   ↓
[If No Satisfying Result: Expand Search]
   ↓
[Search Nearby Airports]
   ↓
[Return to Filter]

This is a directed graph with branches, loops, and conditionals — far more powerful than linear memory.

The Self-Evolution Mechanism

The most amazing capability is self-evolution:

Evolution Cycle

Collect trajectories: Record complete execution paths
Compare analysis: Contrast failed trajectories with successful ones
Identify differences: Find where things went wrong
Generate edits: LLM Refiner proposes modifications
Verify and retain: Test on validation set

Three Types of Edits

Topology Edits:
- Add new nodes (new steps)
- Remove redundant nodes
- Add/modify edges (change flow structure)
Attribute Edits:
- Update node success rate statistics
- Adjust condition thresholds
- Update execution probabilities
Content Edits:
- Modify node descriptions for accuracy
- Update guidance language for effectiveness

From Skeleton to Maturity

Researchers tested three initialization methods:

Initialization	Evolution Speed	Final Performance
Empty Graph (only "Start" node)	Slower	Close to others
Minimal Skeleton (basic manual nodes)	Fastest	Best
Expert Prior (human-designed)	N/A	Repairable if flawed

Amazing Discovery: Even starting from a flawed expert prior, the evolution mechanism can "repair" it to achieve good performance.

This shows robustness — doesn't require perfect initial design.

Why Graphs Beat Memory

Dimension	Pure Memory	Workflow Memory	Procedural Graph
Structure	None	Case-level abstract	Procedure-level
Generalization	Poor	Medium	Good
Explainability	Poor	Medium	Good
Evolution	None	Limited	Strong
Efficiency	Low	Medium	High

Key Advantage: Procedural Graphs abstract the general flow for a class of tasks, not just specific past cases.

Analogy:

Workflow Memory = Remember "Last time I made Mapo Tofu, I stir-fried meat first, then added bean paste"
Procedural Graph = Understand "General stir-fry flow: Heat pan → Add oil → Stir-fry main ingredient → Season → Serve"

The latter generalizes to any stir-fry; the latter can only repeat Mapo Tofu.

Experimental Results

WebShop (Web Shopping)

Task: Purchase items on e-commerce sites based on natural language instructions
Graph vs Memory: 15-25% success rate improvement
Evolved Graph vs Initial: 10-20% improvement

ALFWorld (Home Tasks)

Task: Execute daily tasks in simulated home environment
Graph helps remember complex object interaction sequences

HotPotQA (Multi-hop QA)

Task: Multi-step information retrieval and reasoning
Graph optimizes retrieval strategy and evidence integration

Tool Use Tasks

Task: Combine multiple APIs to complete complex goals
Graph ensures correct tool call order and parameter settings

Key Findings

Finding 1: Cross-LLM Generalization

Graph evolved on one LLM (e.g., GPT-4) can transfer to another (e.g., Claude or Llama)
Shows graphs capture task structure, not model-specific traits

Finding 2: Few-Shot Advantage

Effective even with few examples (<10)
Pure memory baseline drops sharply with few samples

Finding 3: Expert Prior Repairability

Can repair flawed human-designed starting points
Lowers deployment threshold — no perfect initial design needed

Deeper Insights

From Connectionism to Symbolism

Procedural Graphs represent an important trend: neural-network + symbolic-structure fusion.

Deep Learning (connectionism): Good at learning patterns from data, lacks explicit reasoning structure
Symbolic AI: Good at logical reasoning and structured knowledge, lacks learning from data

Graph combines both:

LLM provides semantic understanding
Graph structure provides procedural constraints

This is a Neuro-Symbolic architecture — possibly a key path to more reliable AI.

Biological Intelligence Analogy

Human brain similarly combines two systems:

System 1 (fast, intuitive, pattern-matching): Like LLM generation
System 2 (slow, logical, rule-based): Like Procedural Graph execution

The graph is like giving LLMs a System 2 — an explicit, checkable, fixable execution controller.

Code Example: Creating a Procedural Graph

from procedural_graph import ProceduralGraph, Node, Edge

# Create a simple graph for a coding task
graph = ProceduralGraph()

# Add nodes
start = Node("Start", description="Begin task")
search = Node("Search Info", description="Search for relevant information")
analyze = Node("Analyze Data", description="Analyze collected data")
code = Node("Write Code", description="Generate code solution")
verify = Node("Verify Result", description="Test and verify output")
end = Node("End", description="Task complete")

# Add edges
graph.add_edge(start, search, type="sequential")
graph.add_edge(search, analyze, type="conditional", condition="info_found")
graph.add_edge(analyze, code, type="sequential")
graph.add_edge(code, verify, type="sequential")
graph.add_edge(verify, end, type="conditional", condition="passed")
graph.add_edge(verify, code, type="conditional", condition="failed")  # Loop back

# Navigate the graph
current = start
while current != end:
    context = graph.get_local_subgraph(current)
    action = llm.generate_action(context)
    result = execute(action)
    current = graph.navigate(current, result)

Architecture Overview

Key Components

Graph Store: Stores nodes, edges, and attributes
Navigator: Determines next node based on current state
Refiner: Proposes graph edits based on trajectory feedback
Verifier: Tests graph modifications on validation set

Evolution Process

Collect execution trajectories
Compare successful vs. failed paths
Identify differences and generate edits
Verify edits on validation set
Apply successful edits to production graph

Software Engineering Perspective

Procedural Graphs introduce key concepts:

1. Separation of Concerns

"What to do" (task understanding): LLM
"How to do" (execution flow): Graph
"How to improve" (flow optimization): Refiner

2. Version Control

Every graph edit is recorded
Can rollback to previous versions
Can compare performance across versions

3. Testability

Graph can be independently tested on validation sets
Edit effects can be quantified
Avoids "black-box optimization" uncertainty

Conclusion

The story of Procedural Graphs is essentially a story about organization.

Information alone has no value. Only when organized into useful structures — recipes, flowcharts, algorithms, organizational charts — can it guide action, produce results, and continuously improve.

LLM agents have massive knowledge and powerful generation capabilities, but lack structured execution frameworks. Procedural Graphs fill this gap:

Give agents a "skeleton" — clear execution flow
Give agents "learning ability" — self-evolve from failures
Give agents "explainability" — humans can understand and modify its "thinking"

When that recipe starts rewriting itself, it's no longer just a book. It becomes a living thing — constantly adapting, learning, and improving.

This is the ultimate vision of Procedural Graphs: Not giving agents a fixed program, but giving them a brain that can write its own programs.

This article is based on research published by Yuxing Lu, Yicheng Chen, and Shanchan Wu on September 9, 2026. Paper: arXiv:2609.08593