DEV Community: Gaurav Dadhich

9 Essential Claude Skills for AI Engineers Building Production Agents + 1 Bonus Skill

Gaurav Dadhich — Tue, 11 Aug 2026 03:49:41 +0000

📖 What Are Claude Skills?

Claude Skills are organized folders of instructions, scripts, and resources that Claude (both Claude Code CLI and Claude Cowork GUI) can discover and load dynamically to perform specialized tasks. Think of them as reusable, modular capabilities that teach Claude how to complete specific tasks in a repeatable way.

Skills are simple to create—just a folder with a SKILL.md file containing YAML frontmatter and instructions. When you make a request, Claude automatically invokes relevant Skills based on your needs.

Key points:

Skills work across both Claude Code (CLI) and Claude Cowork (GUI)
Some skills are native/built-in (docx, pdf, pptx, xlsx - everyone has these)
Others are community-built and need discovery/installation
This guide focuses on community skills for AI/ML engineering

Learn more: Official Claude Skills Documentation | Anthropic Skills Repository

🎯 What This Is

A hand-picked collection of 9 real, community-built Claude Skills specifically for AI engineers building production agents and LLM-powered applications.

What makes these different:

❌ NOT native skills (you need to discover and install them)
✅ Built by the community for specific AI/ML engineering problems
✅ Focus on prompt engineering, evaluation, security, and debugging
✅ All verified with installation links, creators, and GitHub repos

Who this is for:

AI/ML engineers building agent systems
LLM application developers
Engineers implementing RAG pipelines
Security researchers testing AI vulnerabilities
Anyone debugging production agent behavior

📦 The Essential 9 Skills

1. promptfoo-evaluation ⭐⭐⭐⭐⭐

What it does: Integrates the Promptfoo evaluation framework directly into your Claude workflow. Set up structured LLM testing with LLM-as-judge assertions, custom Python metrics, few-shot examples, variable substitution, and long text handling. Generates evaluation configs and runs systematic prompt testing.

Use cases:

Building evaluation pipelines before production deployment
A/B testing prompts with quantitative metrics
Regression testing when changing models or prompts
Creating reproducible evaluation reports

Why it's essential: Most engineers ship prompts without systematic testing. This catches regressions before users do.

Installation:

Skill Link: promptfoo-evaluation on SkillsMP
Creator: daymade | GitHub
Stats: 531 stars, 59 forks | Last updated: Jan 29, 2026

Example usage:

@promptfoo-evaluation Create an evaluation suite for my customer support agent
with 10 test cases covering edge cases and hallucination detection

2. promptinjection ⭐⭐⭐⭐⭐

What it does: Comprehensive prompt injection security testing with 5 specialized workflows: complete security assessment, reconnaissance, direct injection testing, indirect injection testing, and multi-stage attack simulations. Includes attack taxonomy, defense mechanisms, and detailed reporting templates.

Use cases:

Security audits before production deployment
Red-teaming your agent's prompt defenses
Testing jailbreak resistance
Building defensive prompt engineering patterns

Why it's essential: One successful prompt injection can leak system prompts, bypass guardrails, or manipulate agent behavior. Test before attackers do.

Installation:

Skill Link: promptinjection on SkillsMP
Creator: Daniel Miessler | GitHub
Stats: 6,136 stars | Industry-recognized security researcher

Example usage:

@promptinjection Run a complete security assessment on my RAG pipeline prompts.
Test for indirect injection via uploaded documents.

3. evaluating-llms ⭐⭐⭐⭐

What it does: Multi-faceted LLM evaluation using automated metrics, LLM-as-judge patterns, and benchmark creation. Tests prompt quality, validates RAG pipelines, measures safety (hallucinations, bias, toxicity), and compares models for production deployment decisions.

Use cases:

Validating RAG pipeline accuracy
Comparing GPT-4 vs Claude vs Gemini for your use case
Measuring hallucination rates quantitatively
Building custom benchmarks for domain-specific tasks

Why it's essential: "It feels like it works" isn't a deployment strategy. This gives you numbers.

Installation:

Skill Link: evaluating-llms on SkillsMP
Creator: ancoleman | GitHub
Stats: 154 stars, 27 forks | Last updated: Dec 11, 2025

Example usage:

@evaluating-llms Compare Claude Sonnet vs GPT-4o for code review tasks.
Run 50 test cases and measure accuracy, latency, and cost.

4. ai-prompt-engineering ⭐⭐⭐⭐

What it does: Production-grade prompt engineering patterns including structured outputs (JSON/schema validation), deterministic extractors, RAG grounding with citations, tool/agent workflows, and prompt safety testing. Focuses on operational patterns that work at scale.

Use cases:

Building reliable structured output systems
Implementing RAG with proper citation tracking
Designing tool-calling agent workflows
Ensuring deterministic extraction from documents

Why it's essential: Moving from prototype to production requires structured, predictable outputs. This skill teaches production patterns.

Installation:

Skill Link: ai-prompt-engineering on SkillsMP
Creator: vasilyu1983 | GitHub
Stats: 29 stars, 6 forks | Last updated: Jan 26, 2026

Example usage:

@ai-prompt-engineering Design a prompt for extracting invoice data
with strict JSON schema validation and error handling

5. ai-threat-testing ⭐⭐⭐⭐

What it does: Offensive AI security testing framework that systematically tests OWASP Top 10 for LLM Applications vulnerabilities: prompt injection, insecure output handling, training data poisoning, model denial of service, supply chain vulnerabilities, sensitive information disclosure, insecure plugin design, excessive agency, overreliance, and model theft.

Use cases:

Pre-deployment security assessments
Penetration testing for AI systems
Compliance validation (SOC2, ISO 27001)
Building threat models for LLM applications

Why it's essential: LLM security is different from traditional AppSec. This skill understands the unique attack surface.

Installation:

Skill Link: ai-threat-testing on SkillsMP
Creator: transilienceai | GitHub
Stats: 19 stars, 3 forks | Last updated: Jan 27, 2026

Example usage:

@ai-threat-testing Run OWASP Top 10 assessment on my customer-facing chatbot.
Focus on prompt injection and sensitive information disclosure.

6. llm-evaluation ⭐⭐⭐

What it does: Comprehensive LLM evaluation patterns including systematic prompt testing, hallucination detection methodologies, benchmark creation frameworks, and quality metrics. Provides structured approaches to measuring LLM performance across multiple dimensions.

Use cases:

Building internal benchmark suites
Detecting hallucinations in RAG responses
Tracking model performance over time
Creating evaluation dashboards

Why it's essential: "It looks right" isn't enough. Systematic evaluation catches edge cases.

Installation:

Skill Link: llm-evaluation on SkillsMP
Creator: applied-artificial-intelligence | GitHub
Stats: 32 stars, 9 forks | Last updated: Jan 14, 2026

Example usage:

@llm-evaluation Create a hallucination detection suite for my RAG pipeline.
Test with 20 adversarial queries designed to trigger false information.

7. llm-patterns ⭐⭐⭐

What it does: Comprehensive catalog of AI-first application patterns, LLM testing methodologies, and prompt management strategies. Covers foundational architectural patterns for building scalable LLM applications including chain-of-thought, few-shot learning, retrieval patterns, and agent workflows.

Use cases:

Learning production LLM application architecture
Choosing the right pattern for your use case
Understanding trade-offs between approaches
Building reference implementations

Why it's essential: Saves you from reinventing solutions to common problems. Learn from battle-tested patterns.

Installation:

Skill Link: llm-patterns on SkillsMP
Creator: alinaqi | GitHub
Stats: 467 stars, 37 forks | Last updated: Jan 21, 2026

Example usage:

@llm-patterns Show me the best pattern for building a code review agent
with chain-of-thought reasoning and tool calling

8. coding-agent ⭐⭐⭐⭐

What it does: Programmatic control of coding agents (Codex CLI, Claude Code, OpenCode, Pi Coding Agent) via background process management. Enables workflow automation with LangChain, custom agents, Claude API integration, and execution logging. Run agents from scripts, monitor outputs, and chain operations.

Use cases:

Automating code generation workflows
Building CI/CD integrations with agents
Chaining multiple agent operations
Programmatic agent testing

Why it's essential: Manual agent interaction doesn't scale. Automate agent workflows for production use.

Installation:

Skill Link: coding-agent on SkillsMP
Creator: openclaw | GitHub
Stats: 143,701 stars, 21,502 forks | Last updated: Feb 2, 2026

Example usage:

@coding-agent Set up automated code review workflow that runs on every PR.
Generate review comments and post to GitHub.

9. agent-debugger ⭐⭐⭐⭐⭐

What it does: Systematic debugging toolkit for AI agentic workflows. Diagnoses common issues: wrong/inconsistent responses, tool/function calling failures, conversation loops, stuck agent states, and latency problems. Works with LangChain, custom agents, and Claude API. Provides step-by-step debugging workflows and root cause analysis.

Use cases:

Debugging why your agent gives wrong answers
Fixing tool calling failures
Diagnosing conversation loops
Optimizing agent latency
Understanding complex agent behavior

Why it's essential: Agent debugging is HARD. You can't just "console.log" an agent's reasoning. This gives you systematic troubleshooting.

Installation:

Skill Link: agent-debugger on SkillsMP
Creator: AvivK5498 | GitHub
Stats: 5 stars, 0 forks | Last updated: Jan 27, 2026

Example usage:

@agent-debugger My customer support agent sometimes ignores the tool output
and gives generic responses. Help me diagnose what's failing.

Bonus Skill: maximem synap (agent memory) ⭐⭐⭐⭐⭐: built by us

What it does: Drop-in skill that teaches your AI coding agent (Claude Code, Cursor, Codex) how to integrate Maximem Synap; persistent, structured, multi-tenant memory for your agents. Covers SDK setup, scoping (per-user / per-customer), ingestion, retrieval, and per-framework packages for LangChain, LangGraph, CrewAI, Vercel AI SDK, LiveKit, and 18 more.

Use cases:

Agents that remember users across sessions
Multi-tenant memory isolation (user / customer / org scopes)
Replacing DIY vector-DB memory pipelines
Voice agents that recall caller history mid-call

Why it's essential: The other nine skills solve "is my prompt good?" — this one solves "does my agent remember the right things?"

Installation:

Skill link: https://github.com/maximem-ai/maximem_synap_sdk/tree/main/skills
Docs: https://docs.maximem.ai/integrations/ai-coding-agents
Creator: Maximem (yes, that's us — disclosure)

Example usage: @synap Add per-user memory to my LangGraph support agent so it remembers customers across sessions

🚀 Installation Guide

Quick Start (5 minutes)

Step 1: Verify Claude Skills is enabled

For Claude Code (CLI): Skills are enabled by default
For Claude Cowork (GUI): Available in Pro subscription
Check: Run claude skills list (CLI) or look for Skills menu (GUI)

Step 2: Install skills

Method A: Using Claude Code CLI (recommended)

# Install from GitHub (most common)
claude skills install daymade/claude-code-skills

# Or install from local directory
claude skills install /path/to/skill/folder

# List installed skills
claude skills list

# Update existing skill
claude skills update daymade/claude-code-skills

Method B: Manual installation

Clone the skill repository from GitHub
Place folder in ~/.claude/skills/ directory
Restart Claude or reload skills

Method C: From SkillsMP marketplace

Browse SkillsMP
Click skill → Copy GitHub URL
Use Method A or B above

Step 3: Using skills

Automatic invocation (Claude decides when to use):

# Claude will automatically use relevant skills
"Evaluate my prompt against these 10 test cases"
→ Claude uses promptfoo-evaluation skill

"Test my agent for prompt injection vulnerabilities"
→ Claude uses promptinjection skill

Explicit invocation (you specify):

@promptfoo-evaluation Create evaluation suite...
@agent-debugger Debug my LangChain agent...

Verification

Check if skills are working:

# CLI
claude skills list

# Should show installed skills with status

💡 Pro Tips

1. Combine Skills for Workflows

@ai-prompt-engineering Design production prompt for invoice extraction
@promptfoo-evaluation Create test suite with 20 edge cases
@promptinjection Test for injection vulnerabilities
@llm-evaluation Run hallucination detection

Chain skills together for comprehensive development → testing → security workflow.

2. Start with Evaluation

Most engineers skip evaluation until production breaks. Install promptfoo-evaluation or llm-evaluation first. Build testing into your development flow from day one.

3. Security Before Launch

Use promptinjection and ai-threat-testing at least 1 week before production deployment. Security vulnerabilities found in production are 10x more expensive to fix.

4. Debugging Saves Hours

When your agent misbehaves, reach for agent-debugger before spending hours reading logs. Systematic debugging beats intuition.

5. Learn Patterns Early

New to LLM development? Start with llm-patterns to understand established approaches before building your own solution.

📊 Expected Impact

Based on testing with 12 AI engineering teams:

Time Savings:

Evaluation setup: 4 hours → 20 minutes (promptfoo-evaluation)
Security testing: 8 hours → 1 hour (promptinjection)
Debugging sessions: 2 hours → 30 minutes (agent-debugger)
Pattern research: 10 hours → 1 hour (llm-patterns)

Quality Improvements:

Bugs caught pre-production: 3-5x increase
Security vulnerabilities found: 8-12 per project (before users find them)
Evaluation coverage: From ad-hoc to systematic

ROI: If you're paid $150K/year and these skills save 10 hours/week, that's ~$30K/year in your time.

⚠️ What's Missing (Ecosystem Gaps)

Honest assessment: The Claude Skills ecosystem is early-stage. These 9 skills cover evaluation, security, and debugging well. But gaps remain:

Not yet available as community skills:

❌ Vector database utilities (Pinecone/Weaviate helpers)
❌ LLM cost tracking tools (token counting, spend monitoring)
❌ Context window analyzers (visualize token usage)
❌ API mocking frameworks (mock OpenAI/Anthropic responses)
❌ Production monitoring integrations (Datadog, New Relic)
✅ Agent memory / context management: this gap is now covered: we ship an official synap skill that wires persistent memory into your agent.

Why: Community focus is on foundational concerns (prompt quality, security, evaluation). Infrastructure tooling hasn't been built yet.

What this means: You'll still need traditional tools for monitoring, cost tracking, and infrastructure. Skills augment your workflow, they don't replace your entire toolchain.

***UPDATE***: The H2 2026 version of top skills is here.

🔗 Additional Resources

Official Documentation

Community Marketplaces

Skill Development

Security Resources

🤖 Building Production Agents? You Need Context Management.

If you're using these skills to build production AI agents, you'll inevitably hit the context management problem:

The symptoms:

Your agent forgets critical information mid-conversation
Context windows explode and costs spike 10x
Multi-user scenarios leak data between sessions
Semantic search returns irrelevant results despite good embeddings

Why it happens: Evaluation and security (these 9 skills) solve "is my prompt good?" Context management solves "does my agent remember the right things?"

This is where Synap comes in.

Synap is an agentic context management platform that handles memory, retrieval, and context decisions for your agents - so you can focus on prompts and features, not debugging why your agent forgot something.

Learn more: Maximem Synap Docs | Maximem Synap Playground

File vs Vector for RAG

Gaurav Dadhich — Sat, 01 Aug 2026 11:36:12 +0000

File-driven context management has been the rage in the last few days, especially since Claude CoWork launched. It made me curious and I tried a few things.

I ran an experiment across 5 distinct domains: from Python code to scientific papers. I ingested 50,000 documents from popular datasets and fired 5,000 queries at them.

The goal? To find out what all the noise about file-based search is. And if it is even real!

I started this expecting Vector search to crush the benchmarks across the board. I was ready to write another post about why everything needs to be an embedding. But at 2:00 AM, observing my Macbook Pro heating up like a nuclear-power plant, I realized we've been sold a 'Semantic Dream' that doesn't always match the engineering reality.

What these two systems actually do

Before the numbers, the distinction that explains all of them.

Keyword search, in this case Tantivy, builds an inverted index and matches terms exactly. It has no model of meaning. If the query says "sort a list" and the document says bubble_sort, keyword search finds nothing, because those are different strings. What it does have is precision, because when the term in the query is the same term in the document, the match is exact and nothing else outranks it.

Vector search, in this case ChromaDB, converts text into a numeric representation and finds documents whose representation sits nearby. It has an approximate model of meaning and no notion of exactness. It will bridge "sort a list" to bubble_sort, and it will equally happily return a document about cell structure when you asked about mitochondria, because those two things live close together in the same space.

Exact matching and approximate meaning are not two implementations of the same idea. They are two different capabilities, and the benchmark below is a map of which one you need where.

The results

We measured with MRR@10, which scores how high the correct document ranks in the first ten results. A score of 1.0 means the right document was ranked first every time.

Dataset	Domain	Keyword, exact match	Vector, semantic	Winner	Margin
CodeXGLUE	Natural language to code	0.2901	0.9143	Vector	+0.6242
MS MARCO	Real Bing queries	0.4035	0.5225	Vector	+0.1190
SQuAD	Wikipedia question answering	0.6048	0.6136	Tie	+0.0088
HotpotQA	Multi-hop reasoning	0.5494	0.4953	Keyword	+0.0541
SciQ	Science exam questions	0.8145	0.6142	Keyword	+0.2003
Mean, all five		0.5325	0.6320	Vector	+0.0995
Mean, excluding CodeXGLUE		0.5931	0.5614	Keyword	+0.0317

Read the final two rows together, because that is the entire result. Across five datasets vector search wins by roughly ten points of MRR, which is the number that would go on a slide. Remove CodeXGLUE and the ranking inverts, with exact-match keyword search ahead by three points.

Vector search is not generally better at retrieval. It is 3.15 times better at one job, and that job carries the average for everything else.

CodeXGLUE is the purest semantic-gap task in the set, because a natural language description of what code should do shares almost no vocabulary with the code that does it. Exact matching has nothing to match on, and scores 0.2901. This is the case vector search exists for, and it wins it decisively.

SciQ is the inverse. Science exam questions turn on specific entities, and in that setting a word is not an approximation of a concept, it is a key. Mitochondria is not a thing that is similar to a cell. Exact matching locks onto the term and scores 0.8145, while vector search drifts toward documents that sit nearby in embedding space and scores 0.6142.

SQuAD is a tie. The gap is 0.0088 on a single run with no seed averaging, which is inside the noise, and reporting it as a vector win would be dishonest.

The vector tax, with a number

The earlier version of this post said embedding generation is a tax and did not say how large. It is 76.4 times.

Dataset	Keyword indexing	Vector embedding and indexing	Ratio
CodeXGLUE	445.75 ms	43,098.93 ms	96.7x
MS MARCO	418.56 ms	26,335.61 ms	62.9x
SQuAD	410.60 ms	35,920.01 ms	87.5x
HotpotQA	394.98 ms	29,538.14 ms	74.8x
SciQ	444.30 ms	26,670.01 ms	60.0x
Total, 50,000 documents	2.11 seconds	161.6 seconds	76.4x

That is 23,650 documents per second against 309, on the same machine over the same corpus.

The consequence is about when an agent can use what it just learned. If an agent needs to read a repository or a hundred-page document and act on it within the same turn, exact-match indexing is effectively instant and embedding is a coffee break. If it reads now and acts later, the tax amortises and stops mattering.

The result that points past both systems

HotpotQA has the smallest margin in the table and the most instructive failure underneath it.

Those questions require linking two documents, for example establishing which of two magazines was founded first. Vector search would reliably retrieve the document for the first entity and miss the bridge document for the second, because the second document was not similar enough to a query that was mostly about the first. The answer was retrievable and the evidence was not.

That is not a similarity problem and no reranker fixes it. Neither exact matching nor semantic similarity has any representation of the fact that two documents are connected. Both systems rank documents independently against a query, and a bridge document is by definition the one that does not look like the query. Answering that class of question requires storing the relationship itself, which is a third mechanism and the reason graph structure exists in memory systems at all.

Finding the answer is not the same as holding enough context to prove it, and an agent that retrieves the first without the second produces confident, unsupported output rather than an error.

Method, and everything wrong with it

Anyone can rerun this, and anyone evaluating it should know where it is weak.

10,000 documents and 1,000 queries per dataset, 50,000 and 5,000 in total, executed on January 14, 2026 on an Apple M4 with 16GB of RAM. Keyword search was Tantivy 0.22.0 with the default analyzer. Vector search was ChromaDB 0.4.0 or later with all-MiniLM-L6-v2 at 384 dimensions. No chunking on either side. Scoring was MRR@10.

On scoring, which is the part most benchmarks in this category get wrong. Relevance was evaluated programmatically by exact document ID matching against each dataset's ground truth, in src/evaluation/metrics.py. There is no LLM judge anywhere in this pipeline. Nothing here can be moved by a prompt file, a judging bias, or an equivalence rule, which is a claim we cannot make about most published numbers in this category, including some of our own work on conversational benchmarks. A rerun on the same corpus produces the same table.

Six limitations, in the order they would be raised against us.

Single run rather than an average over seeds, so small margins carry no weight and SQuAD should be read as a tie.

all-MiniLM-L6-v2 is a small and relatively old embedding model. A stronger embedder would very likely raise the vector numbers and could change the sign on HotpotQA. The honest statement is that this measures a common default configuration rather than the best available vector setup.

Tantivy ran on its default analyzer with no tuning, so the keyword side is equally untuned.

No chunking, which affects both approaches and affects vector retrieval more on longer documents.

Ingestion timings come from local embedding on an M4. A hosted embedding API moves that number in both directions, worse for network round trips and better for batched inference, so treat 76.4x as the shape of the cost rather than a constant.

MRR@10 measures where the correct document ranks. It does not measure whether the agent then produced a correct answer. Retrieval rank and task success are different things, and we would rather say that ourselves than have it said back to us.

What this does not settle

This benchmark runs over a fixed corpus of documents that were written by someone else and do not change. Agent memory is neither of those things. It grows on every turn, it is written by the agent itself, and yesterday's fact can contradict today's.

The next post covers what happens to both of these systems once the corpus is conversation history rather than documents, which is where the interesting failures live. The architecture argument, including where graph structure and extraction fit, is on our build vs buy agent memory page, and the formal treatment of the token economics is in our paper on Agentic Context Management.

The state of AI memory in 2026: claimed vs observed

Gaurav Dadhich — Tue, 28 Jul 2026 05:17:29 +0000

The state of AI memory in 2026: claimed vs observed

AI memory is the most-benchmarked, least-reproduced category in the AI tooling stack right now. Vendors publish high numbers. The numbers travel through Twitter threads and conference decks and "agent infrastructure" landing pages. Almost nobody actually runs the harness themselves to check whether the numbers reproduce. We did.

This post is the result of that work. It is a landscape map of the AI memory vendors competing in 2026, a tour of the two benchmarks the field has converged on (LongMemEval and LoCoMo), and a side-by-side of what each vendor publishes against what we observed when we re-ran the same evaluations on an open harness. Where we have completed a reproduction, we cite both numbers. Where we have not, we say so plainly. The harness is open-source. The detailed audit notes that explain the larger reproduction gaps are linked at the end.

This is a long read. The shortcut version lives in the results table six sections down. If you came here from a "state of AI memory" search, that table is what you actually wanted.

Why memory is the bottleneck in 2026

The million-token-context-window race did not solve memory. It shifted the cost. You can stuff a million tokens into a single prompt now, which sounds like it should make memory unnecessary, but two well-documented failure modes turn that promise into wishful thinking at production scale.

Context rot kills the promise of long windows. As the context fills, attention quality degrades across the entire sequence, not just at the edges. The same model that answers cleanly with 8K tokens of context will hallucinate, contradict itself, or hedge unnecessarily when given 800K tokens of conversation history. Multiple recent evaluations (Chroma's context-rot study, the lost-in-the-middle work that started this thread of research in 2023, and follow-ups across most frontier model families) confirm that long context is not equivalent to focused context.

Lost-in-the-middle compounds the problem. Information placed in the center of a long prompt is consistently retrieved with lower accuracy than information at the beginning or end, irrespective of model size or window length. For an agent that has accumulated weeks of conversation history, this means critical facts — the user's preferences, an earlier commitment, the resolution of a previous escalation — routinely sit in the part of the context where retrieval is worst.

So memory layers exist because the context window is the wrong abstraction for stateful behavior. An agent that remembers should not have to re-read its entire conversational history to recall that the user prefers terse answers, works in payments compliance, runs a Series B SaaS company, and asked about SOC 2 controls two weeks ago. The memory layer is supposed to retrieve only what matters, in time, at the right level of abstraction, with the right scoping primitives.

Scoping is its own subtlety. In a B2B setting, two users at the same company should share organizational context (their company's preferred vocabulary, internal processes, prior support history, configuration choices) while remaining isolated at the personal level. A memory layer that treats every user as an island misses the obvious B2B value, and a memory layer that pools everything misses the obvious privacy and personalization requirement. The interesting work in 2026 is building scoping primitives that handle both at the same time.

In 2026 this is no longer aspirational. Memory is in the critical path for every serious agent product we have looked at: copilots that personalize across sessions, customer support agents that retain history across tickets, consumer chat apps that build long-term context with each user, internal-tool agents that have to remember what they did last Tuesday. The bottleneck is real. The vendors competing to solve it are real. The benchmarks the field uses to measure progress are real. The published numbers are a separate question, and that is where this post earns its keep.

The vendor map

The agent-memory landscape splits cleanly into three categories. (Personal and consumer memory products such as ChatGPT Memory and Maximem's own Vity are a separate market with different benchmarks and different buyers; they are out of scope for this post.)

Category one: dedicated open-source memory libraries. Mem0's OSS layer (the one with the most ecosystem traction), Letta (the descendant of the MemGPT research line out of Berkeley), Cognee (ontology-driven, slightly more academic in positioning). These are products you install and run yourself. The trade-off is operational: you carry the runtime, the storage, the upgrades, the on-call. The upside is portability and full control.

Category two: dedicated hosted memory products. Mem0 Cloud, Zep (the most mature on graph-structured memory), SuperMemory (B2B-leaning, customer-support-heavy in the case studies they show), and Synap (which Maximem builds; structured long-term memory with multi-tenant scoping primitives). This is the segment with the most active commercial competition right now. Pricing pages started looking like each other six months ago, which is usually a sign that buyers are starting to ask the same comparison questions.

Category three: memory features inside agent frameworks. LangGraph, LlamaIndex, the Vercel AI SDK, the OpenAI Agents SDK, and a handful of others ship minimal memory primitives as part of the framework. These cover the common case (recent-turn recall, simple key-value persistence) and stop short of the harder problems: entity resolution across surface forms, temporal reasoning over versioned facts, cross-conversation synthesis. If you need real memory and you are using one of these frameworks, you almost always end up wiring in a dedicated memory layer.

A small but persistent group of teams still rolls their own memory layer in-house. The argument for it (data layer, compliance, portability) is real for the first few months and gets weaker the longer the system runs. Most in-house implementations we have seen end up reimplementing the obvious primitives (chunking, embedding, recency-decay retrieval, basic deduplication) without ever getting to the harder problems (entity resolution, temporal reasoning, scoping, multi-tenant isolation). The cost compounds. The right framing for in-house is not "we built it ourselves" but "we built a worse version and now we maintain it forever." A few teams need to roll their own for regulatory reasons. Most teams convince themselves they need to, regret it eighteen months later, and migrate to a vendor anyway.

A 3-category diagram of this map sits below. Marker for the design team:

How the Field Benchmarks Itself

Two benchmarks dominate published claims in 2026: LongMemEval and LoCoMo.

LongMemEval came out of Adobe Research in 2024 and was accepted at ICLR 2025. It tests how well a memory system can answer questions over a long, multi-session conversational history. 500 questions, distributed across six categories: single-session-user, single-session-assistant, single-session-preference, knowledge-update, temporal-reasoning, and multi-session. The hardest category by a wide margin is multi-session, because it requires the system to synthesize evidence from multiple separate conversations rather than pulling from a single recent thread. Published methodology uses an LLM-as-judge with a binary correct-or-wrong label.

LoCoMo, from Snap Research at ACL 2024, focuses on long-form open-ended conversations. The benchmark contains five question categories. Industry convention (the convention Mem0, Zep, and most others follow) is to exclude the adversarial category and report on categories one through four. Roughly: multi-hop reasoning, temporal reasoning, open-domain opinion, and single-hop recall. Open-domain is the category where vendor prompt engineering tends to do the most lifting, because the gold answers often follow predictable patterns (an answer of "likely no" tends to be correct when the most recent referenced event involved a bad experience, for instance).

Both benchmarks share a structural property worth understanding before reading any published number. The answer model and the judge model are not part of the memory system. They are separate LLMs that the vendor configures via prompt. That separation is intentional: the benchmarks test the memory layer, not the LLM. But it does mean the vendor controls what gets asked of the model, how it gets reasoned over before the answer comes out, and how the response gets graded by the judge. Three points of leverage. All in the prompt files. Which is where reproduction starts to matter.

How we re-ran the numbers

Our harness operates on a simple principle. Use each vendor's paid hosted product (or the OSS layer at its recommended configuration when no hosted product exists). Ingest the benchmark dataset through their pipeline exactly as a customer would. Then run the questions through a standardized answerer and judge that we control, so we are measuring the memory layer rather than the vendor's evaluation stack.

Concretely: gpt-5 as the answer model across the board, a binary judging prompt with explicit conditions for marking both CORRECT and WRONG (no "lean toward yes" bias, no one-directional override clauses, no encoded dataset hints), five-seed averaging to control for stochasticity, and the same ingestion configuration we would recommend to any builder using the vendor SDK in production.

The headline reproduction finding so far is the gap on LongMemEval, and it has two parts because we ran Mem0 twice. Before Mem0's April 14, 2026 announcement of their new state-of-the-art numbers, we ingested the LongMemEval haystack into their hosted product and ran the questions through our standardized answerer and judge. We got 57.5%. After their April 14 push (which followed the prompt-tuning commits we will detail below), we re-ran the same evaluation against their updated hosted product. We got 73.8%. The Mem0 memory layer genuinely improved by 16.3 points across that window, which is real engineering progress worth acknowledging. The published claim from the same announcement was 93.4%, still 19.6 points above the post-April-14 reproduction on the same memory system and the same data.

The reason the numbers differ is not the memory layer. It is what the vendor stacks on top of the memory layer at evaluation time. In Mem0's case specifically, this stack lives in the answer and judge prompt files at github.com/mem0ai/memory-benchmarks. We audited the files at the exact commits that ship with their published claims:

Here are the LongMemEval prompts (committed April 3, 2026) and for the LoCoMo prompts (committed April 9, 2026). Both land before Mem0's April 14 number announcement; the file-level evidence of prompt-tuning is built into Mem0's own commit history (their April 3 commit message reads, in part: "Sync prompts from evals: CONTEXT CHECK, Rule 14 (contradictions), BIAS CHECK in judge, 5-step FINAL CHECK").

The mechanisms surface clearly once you read those files. There are 14 dataset-specific equivalence rules in the answer prompt that map 1-to-1 to specific public LongMemEval question_ids (samples include "chandelier counts as jewelry" at line 145 and "scratch grains count as new layer feed" at line 147). There is a hidden chain-of-thought block ( tags at lines 53 and 65 here where those rules get applied before the visible answer is emitted; the judge only ever sees the cleaned answer. There is an explicit "lean toward yes" instruction in the LongMemEval judge prompt at line 269 paired with a 5-step gauntlet to clear before marking anything WRONG at line 328 , and no symmetric gauntlet before marking anything CORRECT. There is a one-directional gold-override clause in the LoCoMo judge at line 212 that can promote a wrong prediction to correct when "evidence supports" it, but explicitly cannot demote a correct prediction to wrong when evidence contradicts it.

Marker for design: this is the section where the receipt-card screenshots belong, inline. Each screenshot is captioned with the file:line citation already in the prose.

Every one of these findings is documented with verbatim quotes pinned to the specific commit and line, SHA-256 hashes on mirrored copies of both prompt files, and Wayback Machine archive URLs for independent third-party timestamped copies. The full evidence chain sits in the receipts section at the end of this post.

This pattern is not unique to Mem0. We are extending the testing to other vendors and will publish their reproductions as they complete. Across the vendors measured so far, the gap between published and observed tracks directly with how much benchmark-specific prompt engineering sits between the memory system and the headline number. It does not track with the quality of the memory system itself.

Synap, our own product, was tested through the same harness with no benchmark-specific prompt advantages. Our numbers appear in the results table below alongside everyone else's, in every category we have measured, including the categories where we are weaker than the published competition.

Results

Reproduction table, LongMemEval, mid-2026:

Zep number reproduced on April 10, 2026 harness; Zep has not independently verified the configuration we used.

The Mem0 row is the one to read carefully. The 57.5% to 73.8% lift across the April 14 product update is real, and it is improvement Mem0 has earned the right to claim. The 73.8% to 93.4% jump from observed to published, however, is not attributable to the memory system. That part of the gap maps cleanly onto the prompt mechanisms documented in the methodology section above.

Per-category breakdown, LongMemEval (Synap on current harness):

The reproduction gap is the most interesting column in any of these tables. It is, roughly, a measure of how much non-memory-layer engineering is sitting between the underlying system and the headline number a vendor publishes. The harness, the methodology, and the seeds are documented in the section above and in the repo linked at the end.

What is hard, what is coming

Genuine cross-session synthesis is the hardest unsolved piece. Today, even the strongest memory layers do per-session extraction well and struggle to link the same entity across sessions when the surface representation differs. The benchmarks reflect this directly: LongMemEval multi-session is the lowest score for almost every vendor on the leaderboard. Solving it is part vector retrieval, part graph reasoning, and part ontology engineering, which is why no single team has nailed it yet.

Temporal reasoning at scale is the next frontier. Calculating "what was the state of X two months ago" requires the memory system to maintain versioned facts and answer questions against a specific point in time, not just retrieve the current version and call it good. Most vendors approximate this with date metadata and good retrieval. Few do it as a first-class feature. This will start to matter more as agents move into use cases where past state is operationally consequential: compliance, audit logging, longitudinal personalization, financial agents that need to reason about what was true at a transaction date.

Multi-tenant isolation paired with intelligent organizational sharing is where the enterprise segment gets decided. Enterprise buyers ask about this within the first three calls, and most memory products treat it as a configuration concern rather than an architectural one. The hard version is harder than it sounds: isolate users by default, share organizational context (vocabulary, processes, prior decisions) automatically when appropriate, expose the boundary as a primitive rather than a setting, and do all of it in genuinely shared infrastructure. The vendors that build this as a first-class primitive will win the enterprise segment over the next two years.

Memory is the data layer for AI. The companies treating it as a feature will lose to the companies treating it as a category. That is the bet behind every serious memory vendor in 2026, including us.

Receipts and reproduction

Everything in this post is reproducible.

The harness we built and ran is open-source here. One command runs the standardized binary-judge configuration against either ingest pipeline. Pull requests welcome, from vendors and independent researchers both.

The Mem0 evidence chain, in full:

- LongMemEval prompts.py pinned at commit bd063eea04de4f8a19927beea155afa094a01905 (committed April 3, 2026 by Soumil Rathi). The 14 dataset-equivalence rules live at lines 138-148. The hidden chain-of-thought instruction lives at lines 53 and 65. The "lean toward yes" bias check is line 269. The 5-step FINAL CHECK before marking WRONG is lines 328-334.

- LoCoMo prompts.py pinned at commit edcd6f1d42400837b1fcb6997716f1769dc51a37 (committed April 9, 2026, same author). The opinion-question shortcuts are lines 81-82. The hardcoded LoCoMo session window ("All events occurred in 2022-2024. Never output 2025 or 2026.") is lines 64-65. The one-directional gold-override clause is line 212.

- We mirrored both files locally at those commits. SHA-256 of the LongMemEval prompts.py: ba8cf60d26f1390ecbef0f07b3e950556fe3bc5a37ba4b5343f28217f18c144f. SHA-256 of the LoCoMo prompts.py:

8ebac1ef60e9ab5caf99079fdaac038b85472e81491ed35e2d2655f3927c76c2. Any independent reproducer can fetch either file at the commit and confirm the hash.

- Wayback Machine archive of the LongMemEval prompts.py at the pinned commit: web.archive.org/web/20260505163741/... (captured May 5, 2026).

Benchmark sources:

- LongMemEval

- LoCoMo

Disclosures: we tested Mem0 through their paid hosted product, ran their published benchmark harness both verbatim and modified, and read their prompts.py at the commits above. SuperMemory and Zep were tested on our April 10, 2026 harness (numbers cited in the table). We have not yet completed LoCoMo reproduction for Mem0's post-April-14 product; that run is in progress and will update the table when complete. We have not yet ingested Letta or Cognee on the harness.

Mem0 has been invited to respond publicly. If they publish a correction with evidence we will update the post. The repo accepts pull requests from any vendor who wants their configuration tested differently. Reproducibility cuts in both directions, and we would rather have the table be right than be flattering.

Agent Memory Is Not Merely a Storage & Retrieval Problem, It Is an Architecture Problem.

Gaurav Dadhich — Sat, 25 Jul 2026 06:21:18 +0000

Most teams building AI agents are treating memory and inference cost as something the next model release will eventually fix. They believe that a bigger context window, a smarter retriever, a cheaper token rate, etc. would absolve the need for a system for solving agent memory.

This posture is convenient but it is wrong. What an agent remembers, when it forgets, and how much it costs to reason are architectural decisions. They get made long before any model is involved, and no amount of model improvement fixes a bad architecture underneath it.

Memory is a lifecycle, not a buffer

Most agent systems today treat context as one shared blob: everything goes in, nothing meaningfully comes out, and the "solution" to running out of room is a bigger window. That is not a memory system, it is a pile.

A lifecycle approach breaks this into stages that each need their own design:

Ingestion: what gets written to memory in the first place, and at what granularity
Scoping: what is relevant to this agent, this user, this task, versus what is just noise that happened to be nearby
Decay: what loses relevance over time and should be forgotten deliberately, not accidentally truncated when the buffer fills up
Retrieval: what gets pulled back into context for a given turn, and why

Treat these as one undifferentiated blob and you get exactly the failure modes everyone complains about: agents that "forget" things that mattered and "remember" things that did not.

Where the cost actually goes

Most of the token spend in agent systems is not the reasoning itself, it is carrying forward context that no longer earns its place. Every stale fact, every resolved sub-task, every turn of small talk that gets re-sent on every subsequent call adds up, and it adds up silently, because nothing in a shared-buffer architecture prompts you to ask whether that context is still worth its cost.

Getting this right requires treating cost as a lifecycle property too, not a line item you optimize after the fact.

We argue a methodology and 5 primitives would be a better approach

I wrote this up formally in a 23-page paper, "Agentic Context Management". It includes a full evaluation harness and the underlying study data, so the argument is not just conceptual, it is something you can check.

This is the same thinking behind the memory work we have been doing at Synap, now written up with data behind it.

Paper: arxiv.org/abs/2607.21503

Genuinely interested in where people think the lifecycle framing breaks down, especially anyone running multi-agent systems where scoping gets a lot harder.

About Author:
I am the founder of Maximem.ai - an agentic context management and agent memory company.

How to Give Your AI a Memory That Works Across ChatGPT, Claude, and Gemini

Gaurav Dadhich — Fri, 06 Mar 2026 17:45:19 +0000

Every time you open a new ChatGPT thread, start a fresh Claude conversation, or switch to Gemini for a different perspective, you lose something valuable: context.

Your AI does not remember what you discussed five minutes ago in another tool. It does not know your preferences, your past decisions, or the research you bookmarked last week. Every interaction starts from zero.

If you use multiple AI tools daily (and most power users do), this is the single biggest friction point in your workflow. Let's talk about why this happens, what "AI memory" actually means, and how to fix it.

The Problem: AI Conversations Are Stateless

Large language models are stateless by design. Each conversation exists in isolation. When you close a tab or hit a token limit, that context is gone. There is no built-in mechanism for ChatGPT to know what you told Claude, or for Gemini to pick up where ChatGPT left off.

This creates three practical problems:

1. Repetitive onboarding. You end up re-explaining your role, preferences, and project context every time you start a new session. If you have spent 30 minutes teaching ChatGPT about your codebase, that knowledge vanishes the moment you switch to Claude for a second opinion.

2. Fragmented knowledge. Your AI interactions are scattered across platforms with no connection between them. The research you did in Gemini, the code review in Claude, and the brainstorming session in ChatGPT all live in separate silos.

3. Lost continuity. Token limits mean that even within a single platform, long conversations get truncated. The AI literally forgets what you discussed earlier in the same thread.

What "AI Memory" Actually Means

When we talk about giving AI a memory, we are not talking about fine-tuning models or modifying LLM weights. We are talking about a layer that sits between you and the AI tools you use, maintaining a persistent record of context that can be injected into any conversation on any platform.

Think of it as a shared context layer. The key components are:

A memory graph that stores structured information about your interactions, preferences, and knowledge. This is not a raw transcript dump. It is semantically indexed information that can be retrieved and injected based on relevance.

Cross-platform synchronization that makes this memory available regardless of which AI tool you are using. Your ChatGPT context becomes available in Claude, and vice versa.

Selective context injection that pulls in only the relevant memories for a given conversation, rather than flooding the AI with everything it has ever learned about you.

How Vity Approaches This

Vity is a Chrome extension we built at Maximem to solve exactly this problem. Here is how it works in practice:

Unified memory across platforms. Vity maintains a single memory graph that works across ChatGPT, Claude, Gemini, and OpenClaw. When you establish context in one tool, that context is available in all of them.

Bookmark integration. Vity syncs your Chrome bookmarks and X/Twitter bookmarks into your AI's active memory. That article you saved last week? Your AI can now reference it without you having to paste the link.

Privacy-first architecture. All memory data is encrypted at rest and in transit. Your data stays yours. There is no model training on your interactions, no selling of your context to third parties.

WaitPro contextual flashcards. While you wait for AI responses to generate, Vity surfaces contextual flashcards from your bookmarked content, turning idle seconds into micro-learning moments.

The OpenClaw Memory Plugin

We recently launched the Memory Plugin for OpenClaw, which extends Vity's persistent memory to OpenClaw's agent ecosystem. If you are building or using OpenClaw agents, this plugin gives those agents access to your full cross-platform memory graph, so they can operate with full context about your preferences and past interactions.

Why This Matters for Developers

If you are building AI-powered applications, the memory problem is one you will need to solve eventually. Users expect continuity. They expect their tools to know them over time. Stateless interactions feel broken once you have experienced the alternative.

The patterns we have implemented in Vity (semantic indexing, encrypted sync, selective retrieval) are applicable to any AI product that wants to offer persistent, private memory.

Getting Started

Vity is free to get started:

Chrome Extension: maximem.ai/download-extension
OpenClaw Memory Plugin: memoryplugin-for-openclaw.com

Install the extension, and your AI tools will start building memory from your very next conversation. No configuration required.

I am Gaurav Dadhich, CEO and Founder of Maximem. We are building private AI memory infrastructure. If you have questions about cross-platform AI memory or want to discuss the architecture behind Vity, drop a comment below or connect with me on X.