DEV Community: Arindam Majumder

I built a local dashboard to inspect Claude Code sessions, tokens, and costs

Arindam Majumder — Thu, 02 Apr 2026 07:58:55 +0000

I’ve been using Claude Code heavily over the last few weeks and started wondering where my tokens were actually going.

Claude stores everything locally in ~/.claude/, which is great, but the data mostly sits in JSON logs. If you want to understand session usage, token costs, tool calls, or activity patterns, you basically end up digging through raw files.

So I built a small tool called cc-lens.

I built a local dashboard to inspect Claude Code sessions, tokens, and costs
It’s a local-first dashboard that reads your Claude Code session files and turns them into something you can actually explore.

It runs entirely on your machine. It doesn't have any cloud sync, sign-ups, or telemetry.

Some things it shows:

• Usage overview: sessions, messages, tokens, estimated cost
• Per-project breakdown: see which repos are burning the most tokens
• Full session replay: inspect conversations turn-by-turn with token counts and tool calls
• Cost & cache analytics: stacked charts by model and cache usage
• Activity heatmap: GitHub-style view of when you’re using Claude the most
• Memory & plan explorer: browse/edit Claude memory files and saved plans
• Export/import: move dashboards across machines

You can run it instantly with:

npx cc-lens
(or clone the repo if you prefer).

Here's the Github Repo, if you want to try it out!

[Boost]

Arindam Majumder — Sat, 28 Mar 2026 13:37:31 +0000

Arindam Majumder for Studio1

Mar 17

Running LLM Applications Across Providers with Bifrost

#ai #llm #proxy #litellm

Comments

5 min read

Build a Semantic Movie Discovery App with Claude Code and Weaviate Agent Skills

Arindam Majumder — Fri, 27 Mar 2026 20:45:45 +0000

Introduction

Versatility in agentic coding is increasing as new tools such as Model Context Protocol (MCP) servers and Agent Skills become more common. At the same time, many developers ask the same question when building AI applications: should they use MCP servers or Agent Skills? The important thing is understanding what each approach does well and choosing the one that fits your use case.

In this post, we’ll explain what MCP servers and Agent Skills are and how they differ, including architecture diagrams and technical details. In the later sections, we’ll also walk through how to use Weaviate Agent Skills with Claude Code to build a “Semantic Movie Discovery” application with several useful features.

Let’s get started!

Understanding MCP

The Model Context Protocol (MCP) is an open standard introduced by Anthropic that enables Large Language Models (LLMs) to interact with external systems such as data sources, APIs and services. MCP provides a structured way for an AI agent to connect to compliant tools through a single interface instead of requiring custom integrations for each service.

MCP Architecture

The MCP system operates on a client–server model and consists of three main components.

Host: the application that runs the AI model and provides the environment where the agent operates.
Client: the protocol connector inside the host that handles communication between the model and MCP servers.
Server: an external service that exposes tools, resources, or prompts that the agent can access.

MCP and Agentic Coding

Before MCP, each AI tool required custom integrations for every external service it wanted to connect to. MCP simplifies this process by introducing a shared protocol that multiple agents and tools can use.

Developers can now expose capabilities through an MCP server once and allow any compatible agent to access them without building separate integrations for each system.

Understanding Agent Skills

Agent Skills, also introduced by Anthropic, provide developers with a simple way to extend AI coding agents without running MCP servers. An Agent Skill is a structured configuration file, usually written as markdown files with YAML metadata that defines capabilities, parameter schemas and natural-language instructions describing how the agent should use those capabilities.

AI tools such as Claude Code read these files at session start and load the skills directly into the agent's working context without requiring an additional runtime.

How Agent Skills Work

When Claude Code detects a skill file in the project directory (typically under .claude/skills/), it loads the manifest into the agent's context at the beginning of the session.
The skill definition describes available capabilities, how to invoke them correctly and when to prefer one approach over another. Because the instructions are written in natural language alongside parameter schemas, the agent can reason about how to use the skill.
Skills are portable across repositories. If a developer commits a skill file to a repository, any collaborator who clones the project and opens it in Claude Code automatically gains access to the same capabilities without additional setup.

MCP and Agent Skills solve different problems in agent systems. MCP provides a standardized way for AI agents to connect to external tools, APIs, databases and services through a client–server architecture with structured schemas. Agent Skills extend the agent’s capabilities through configuration files that define workflows, instructions and parameter schemas without requiring a running server.

In simple terms, MCP enables agents to access external systems, while Agent Skills define how agents perform tasks or workflows within their environment.

Weaviate Agent Skills

Weaviate has released an official set of Agent Skills designed for use with Claude Code and other compatible agent-based development environments like Cursor, Antigravity, Windsurf and more. These skills provide structured access to Weaviate vector databases, allowing agents to perform common operations such as search, querying, schema inspection, data exploration and collection management.

The repository includes ready-to-use skill definitions for tasks like semantic, hybrid and keyword search, along with natural language querying through the Query Agent. It also supports workflows such as creating collections, importing data and fetching filtered results, and cookbooks. This enables agents to interact/build with Weaviate and perform multi-step retrieval and agentic tasks more effectively.

Agent Skills and Vector Databases

AI coding agents face difficulties when working with vector databases. Vector database APIs provide extensive capabilities, including basic “key–value” retrieval, single-vector near-text searches, multimodal near-image searches, hybrid BM25-plus-vector search, generative modules and multi-tenant system support. Without structured guidance, even a capable coding agent may produce suboptimal queries: correct syntax but the wrong search strategy, missing parameters or failure to use powerful features like the Weaviate Query Agent.

Weaviate Agent Skills address this by providing correct usage patterns, parameter recommendations and decision logic, enabling coding agents to generate production-ready code from their initial attempts.

The Weaviate Agent Skills repository is organized into two main parts:

Weaviate 𝗦𝗸𝗶𝗹𝗹 (skills/weaviate): Focused scripts for tasks such as schema inspection, data ingestion and vector search. Agents use these while writing application logic or backend code.
𝗖𝗼𝗼𝗸𝗯𝗼𝗼𝗸𝘀 Skill (skills/weaviate-cookbooks): End-to-end project examples that combine tools such as FastAPI, Next.js and Weaviate to demonstrate full application workflows.

Weaviate Agent Skills work with several development environments, including Claude Code, Cursor, GitHub Copilot, VS Code and Gemini CLI. When connected to a Weaviate Cloud instance, agents can directly interact with database modules and perform search, data management and retrieval tasks.

To evaluate how effective Weaviate Agent Skills really are, let’s build a small project and see how they accelerate RAG and agentic application development with Claude Code.

Building the Semantic Movie Discovery Application

We will build a Movie Discovery App that takes a natural-language description and returns the most semantically similar movies from a Weaviate collection. In the process, we will explore Weaviate capabilities such as multimodal storage, named vector search, generative AI (RAG) and the Query Agent in action with Claude Code, showing how these Agentic tools help you build applications faster.

Prerequisites

Python 3.10 or higher
Weaviate Cloud – Create a free cluster and obtain an API key.
TMDB API key – Used to fetch movie metadata
OpenAI API key – Required for RAG features.
Access to Claude Code
Node.js 18+ and npm – Required to run the Next.js frontend

Step 1: Project Setup

Create a movie-discovery-app folder

mkdir movie-discovery-app

Create and activate a Python virtual environment in the folder

cd movie-discovery-app py -m venv venv && source venv\Scripts\activate.bat

Install Python dependencies

pip install weaviate-client==4.20.1 fastapi uvicorn[standard] openai weaviate-agents>=1.3.0 requests python-dotenv

Install Node.js dependencies for the frontend

cd frontend && npm install

Now, create a .env file at the project root. Add the following parameters to configure Weaviate Agent Skills with Claude Code, along with your OpenAI API key and TMDB API key.

WEAVIATE_URL=your-cluster-host-without-https
WEAVIATE_API_KEY=your-api-key
OPENAI_API_KEY=your-openai-key
TMDB API key=your-tmdb-api-key

After signing up for Weaviate, click the Create Cluster button to start a new cluster for your use.

Click “How to Connect” to view the required Weaviate connection parameters.

Now that everything is set up, we can connect Weaviate Cloud with Claude Code by running claude in your project terminal:

Use the following prompt in your Claude terminal:

Write and run `check_modules.py` that connects using `weaviate.connect_to_weaviate_cloud`with `skip_init_checks=True`, loads credentials from `.env` with `python-dotenv`,
and prints the full JSON list of enabled Weaviate modules.
Run it with `venv/Scripts/python check_modules.py`."

Step 2: Create A Weaviate Collection and Import Sample Movie Data

In this step, we create a Weaviate collection and import the movie dataset into Weaviate. The dataset contains movie metadata sourced from the TMDB API. Each entry includes: title, overview, release_date, poster_url, popularity, and other important movie fields. You can import a JSON or CSV dataset directly into Weaviate.

Run this prompt to retrieve the dataset from the TMDB API and save it to a file named movies.json.

Create a TMDB dataset JSON file, movies.json, that contains 100 movie metadata and poster URLs directly from the TMDB API.

Afterwards, Weaviate Import Skills creates a Weaviate collection and imports the data from movies.json into the Weaviate database. Claude code activates Weaviate to perform this action when prompted with:

Import `movie.json` into a new Weaviate collection called Movie

Then the data is imported

Step 3: Building the FastAPI Backend and Next.js Frontend with Weaviate Cookbooks

Weaviate cookbooks enable the app to use a two-layer architecture: a FastAPI backend that exposes REST endpoints and a Next.js frontend that renders the UI. The backend connects directly to Weaviate Cloud and the Weaviate Query Agent. Weaviate cookbooks also include some frontend guidelines to communicate with the Weaviate backend over HTTP.

The app is organized into two views accessed via a collapsible sidebar:

Search view: performs semantic search and RAG using Weaviate named vectors.
Chat view: handles multi-turn conversations through the Weaviate Query Agent.

Our app includes the following features:

Layer	Component	Role
Backend	backend.py (FastAPI) - REST API on port 8000/docs	Routes: GET /health, GET /search, POST /ai/explain, POST /ai/plan, POST /chat
Frontend	Next.js + TypeScript (port 3000)	Single-page app with sidebar navigation
	SearchView.tsx	Semantic search (near_text), AI explanations (single_prompt), Movie Night Planner (grouped_task)
	MovieCard.tsx	Renders base64 poster inline, watchlist add/remove button
	ChatView.tsx	Multi-turn Query AI Agent chat
	AppSidebar.tsx	Navigation (Search/Chat), Weaviate logo + feature summary, watchlist manager with ‘.txt’ export

Use the following prompts with Claude Code to generate the backend and frontend:

Backend Prompt:

/weaviate cookbooks 

Create `backend.py`: a FastAPI app with CORS enabled for localhost:3000.
Connect to Weaviate Cloud using credentials from .env with skip_init_checks=True.
The /search endpoint should return genre and vote_average alongside title, description, release_year, and poster.
Implement these routes:  

- GET  /health                  → {"status": "ok"}  
- GET  /search?q=...&limit=3    → near_text on text_vector, return title/description/release_year/poster  
- POST /ai/explain              → generate.near_text with single_prompt  
- POST /ai/plan                 → generate.near_text with grouped_task  
- POST /chat                    → QueryAgent.ask() with full message history

Frontend Prompt:

Using Weaviate cookbooks frontend reference, create a Next.js TypeScript app in the frontend/ folder.
MovieCard.tsx should display a star rating (vote_average) and genre tag beneath the movie title. 

Components needed:  

- page.tsx        — SidebarProvider layout, view state (search | chat)  
- SearchView.tsx  — search input, MovieCard grid, AI explain and plan buttons  
- MovieCard.tsx   — poster image, title, year, description, watchlist button  
- ChatView.tsx    — message bubbles, source citations, clear chat  
- AppSidebar.tsx  — navigation, Weaviate logo + feature list, watchlist + exportBackend base URL from NEXT_PUBLIC_BACKEND_HOST env var (default localhost:8000)

Run backend and frontend servers with: uvicorn backend:app --reload --port 800 and npm run dev

After this, Claude Code will automatically build the app by adding relevant files and start both servers. You can start using the application immediately.

The FastAPI backend runs at http://localhost:8000/docswhile the frontend app is available at http://localhost:3000

You can also manually start both processes in separate terminals:

# Terminal 1 — Backend 
uvicorn backend:app --reload --port 8000
# Terminal 2 — Frontend
cd frontend && npm run dev

Congratulations! You’ve completed the project without needing to do much manual configuration or coding. 🔥

Demo

So far, we have used Weaviate Agent Skills with Claude Code to build a Semantic Movie Discovery Application powered by an OpenAI API key, a TMDB API key, and Weaviate.

The Movie Discovery app we built includes the following features:

Semantic search: Describe a mood or theme and retrieve matching movies using vector-based search (near_text).
AI explanations: Generate per-movie summaries using RAG with single_prompt.
Movie Night Planner: Create a viewing order, snack pairings and a theme summary using grouped_task.
Conversational chat: Ask questions about the movie collection through a chat interface powered by the Weaviate Query Agent, with source citations.
Watchlist: Save movies during your session and export the list as a .txt file.

What’s Next?

You could add image-based search to find similar movies and better meet your movie choices. You could also include a hybrid search feature that incorporates keyword-heavy queries and image search.

You can take your app even further by getting up to speed with Weaviate’s latest releases and becoming familiar with features such as server-side batching, async replication improvements, Object TTL and many more.

To explore further, check out the latest Weaviate releases and join the discussion on the community forum.

Weaviant Agent Skills in Action

Weaviate modules were used in the application:

Text2vec-weaviate: Responsible for text embeddings.
Multi2multivec-weaviate: Responsible for embedding images.
Generative-openai: Integrates GPT directly into the query workflow.
Weaviate Skill: Creates a collection and imports data.
Weaviate Cookbooks Skill: For defining the app’s logic.
Weaviate Query Agent: A higher-level abstraction that accepts natural language queries, decides the best query method, executes queries, synthesizes results and returns answers.

Weaviate Agent Skills help in shipping faster and more accurate RAG applications. Backend development tasks such as schema inspection, data ingestion and search operations are automated and optimized. Ultimately, this helps developers save valuable development time.

Conclusion

Both MCP servers and Agent Skills provide useful patterns for building AI-powered applications. MCP servers are well-suited for exposing external tools and services through a standardized interface, while Agent Skills focus on guiding coding agents with structured workflows and best practices.

In this tutorial, we demonstrated how Weaviate Agent Skills can simplify development by helping Claude Code generate correct database queries, ingestion pipelines and search logic. By combining vector search, multimodal storage and generative capabilities, we built a semantic movie discovery application with minimal manual setup.

As agentic development environments continue to evolve, tools like MCP servers and Agent Skills will likely be used together. The key is understanding where each approach fits and selecting the one that best supports your application architecture.

Happy building.

Resources

Model Context Protocol
Weaviate Agent Skills
Claude Code
GitHub Repository for the Movie Discovery App

We Cut Our MCP Token Spend in Half. Here's the Architecture

Arindam Majumder — Wed, 25 Mar 2026 19:04:52 +0000

When we started scaling our MCP workflows, token usage was something we barely tracked. The system worked well, responses were accurate, and adding more tools felt like the right next step. Over time, the cost began rising in ways that did not align with how much the system was actually used.

At first, we assumed this was due to higher usage or more complex queries. The data showed something else. Even simple requests were using more tokens than expected. This led us to ask a basic question. What exactly are we sending to the LLM on every call?

A closer look made things clearer. The issue came from how the system was built. We handled context, tool definitions, and execution flow by adding extra tokens at every step.

This article explains how we found the root cause and redesigned the architecture to fix it. The changes cut our MCP token usage by nearly half and gave us better control over how the system behaves.

Understanding Token Usage in MCP Systems

Once we started examining token usage, a clear pattern showed up. The LLM was receiving far more context than most requests actually needed. A large part of this came from tool definitions being sent repeatedly on every call.

Each request included the full list of tools, even when only one or two were needed. On top of that, earlier outputs and intermediate results were passed back into the model. The context kept growing, even for simple queries.

The execution flow added to the problem. The LLM would choose a tool, call it, process the result, and then repeat the same cycle if another step was needed. Each step added more tokens, and the same data often appeared many times across calls.

This setup worked at a small scale. As the number of tools increased, the cost grew quickly. More tools meant more context. More steps meant repeated processing. The system was doing extra work without adding real value. At this point, the cause was clear. Token usage came from how the system handled context and execution. The design itself was driving the overhead.

Introducing Bifrost

We started looking for a way to change how the system handled tool execution. The goal was simple. Reduce the amount of context sent to the LLM and avoid repeated processing across steps.

During this process, we came across Bifrost, an open source MCP gateway. It works between the application, the model, and the tools. It brings structure for how tools are discovered and executed, so the LLM receives only what is needed on each call.

This changed how we thought about the system. Tool access became more controlled. Context stayed limited to what was required for each request. The overall flow of execution became easier to follow and reason about.

These changes directly addressed the issues we were seeing. Tool definitions were sent only when required. Repeated decision loops were reduced. The system handled execution in a more controlled and predictable way.

From here, the focus moved away from adjusting prompts and toward changing how the system runs end-to-end.

Architectural Changes with Bifrost Code Mode

The main change came from how execution was handled inside Bifrost. Code Mode is a Bifrost feature that changes how the LLM interacts with MCP tools. Earlier, the LLM handled both planning and step-by-step tool interaction. Each step required another call, and each call carried a growing context.

Code Mode separates these responsibilities. The LLM focuses on planning. It generates executable code that defines the full workflow for a task.

Code Mode works best when multiple MCP servers are involved, workflows have several steps, or tools need to share data. For simpler setups with one or two tools, Classic MCP works well.

A mixed setup also works. Use Code Mode for heavier workflows like search or databases, and keep simple tools as direct calls.

This includes:

Selecting the right tools
Passing data between tools
Defining how the final output is produced

The system exposes a minimal interface to the LLM. It can list available tools, read tool details, and, when required, understand how each tool works. Tool definitions are accessed on demand, which keeps the initial context small.

Once the plan is generated, execution moves to a runtime environment. The code runs in a sandbox and interacts directly with tools. All intermediate steps, tool responses, and data transformations stay within this layer.

This removes the need for repeated LLM calls during execution. The workflow runs in one pass, guided by the generated code. The LLM is involved mainly at the planning stage and for producing the final response if required.

The flow becomes more structured. A request comes in, relevant tools are identified, code is generated, and execution happens in a controlled environment. The system handles state and intermediate data outside the LLM.

This approach improves clarity in how tasks are executed. The generated code can be inspected, debugged, and understood directly. Each request follows a defined path, which makes behavior easier to track and reason about.

Using Bifrost CLI in Our Workflow

Getting started required two commands. First, start the gateway:

npx -y @maximhq/bifrost

Then launch the CLI from a separate terminal:

npx -y @maximhq/bifrost-cli

MCP servers are registered once through the API. The key flag is is_code_mode_client, which tells Bifrost to handle that server through Code Mode instead of sending its tool definitions on every request:

curl -X POST http://localhost:8080/api/mcp/client \
  -H "Content-Type: application/json" \
  -d '{
    "name": "youtube",
    "connection_type": "http",
    "connection_string": "http://localhost:3001/mcp",
    "tools_to_execute": ["*"],
    "is_code_mode_client": true
  }'

Once registered, the LLM discovers tools on demand using listToolFiles and readToolFile, then submits a full execution plan through executeToolCode. A workflow that previously took six LLM turns now completes in three to four.

Bifrost organizes tool definitions using two binding levels. Server-level (default) groups all tools from a server into one .pyi file. Tool-level gives each tool its own file — better for servers with 30+ tools. Set it once in config.json:

{
  "mcp": {
    "tool_manager_config": {
      "code_mode_binding_level": "server"
    }
  }
}

Debugging became simpler because the generated code is the execution plan. When something went wrong, the issue was visible directly in the code rather than buried in prompt chains. This setup also made execution easier to inspect.

results = youtube.search(query="AI infrastructure", maxResults=5)
titles = [item["snippet"]["title"] for item in results["items"]]
result = {"titles": titles, "count": len(titles)}

The execution runs in a Starlark interpreter, a restricted subset of Python. A few constraints to keep in mind:

No import statements, file I/O, or network access
Classes are not supported, use dictionaries
Tool calls run synchronously; async handling is not required
Each tool call has a default timeout of 30 seconds

Code Mode also works with Agent Mode for automated workflows. The listToolFiles and readToolFile tools are always auto-executable since they are read-only.

The executeToolCode tool only auto-executes if every tool call within the generated code is on the approved list. If any call falls outside that list, Bifrost returns it to the user for approval before running.

Impact on Token Usage and System Efficiency

The reduction in token usage came from four specific changes:

Tool schemas were sent only when required
Intermediate outputs stayed within the execution layer
Repeated context across steps was removed
Fewer LLM calls were needed, since execution moved to a sandbox and ran in a single flow

These changes had a clear effect. Token usage dropped by nearly half. Latency reduced along with it. Execution became more predictable, since each request followed a defined path with fewer moving parts.

The broader takeaway is clear. Token cost comes from system design. Small changes in prompts or outputs help at the edges. The main overhead comes from the system's structure.

LLMs work best when they focus on planning. Managing execution through repeated loops adds cost and introduces variability. A separate execution layer keeps the flow stable and easier to understand. Context also needs careful control. It should be built for each request with only the required information. Letting it grow across steps results in unnecessary overhead and increased token usage.

Conclusion

Token inefficiency in MCP workflows comes from system design. Bifrost and Code Mode introduced a clear separation between planning and execution. The LLM handles planning, and the runtime handles execution. This brought immediate and measurable improvements in both cost and system behavior.

If you are working with MCP workflows at scale, Bifrost is worth exploring. The documentation provides a good starting point to set up the gateway, connect servers, and run workflows using Code Mode.

Composer 2 is controversial, but my actual experience was solid

Arindam Majumder — Sat, 21 Mar 2026 06:55:09 +0000

I tried Composer 2 properly today, and honestly, if you put all the controversy aside for a second, the model itself is not bad at all.

In fact, my first impression is that it’s a real upgrade over Composer 1 and 1.5. I gave it a pretty solid test. I asked it to build a full-stack Reddit clone and deploy it too.

On the first go, it handled most of the work surprisingly well. The deployment also worked, which was a good sign. The main thing that broke was authentication.

Then on the second prompt, I asked it to fix that, and it actually fixed the auth issue and redeployed the app.

That said, it was not perfect. There were still some backend issues left that it could not fully solve. So I would not say it is at the level of Claude Opus 4.6 or GPT-5.4 for coding quality.

But speed-wise, it felt much faster. For me, it was around 5 to 7x faster than Opus 4.6 / GPT-5.4 in actual workflow, and it also feels much more cost-effective.

That combination matters a lot.

Because even if the raw coding quality is still below Opus 4.6 / GPT-5.4, the overall experience was smoother than I expected. It gets you from idea to working product much faster, and for a lot of people that tradeoff will be worth it.

My current take is:

Better than Composer 1 / 1.5 by a clear margin
Fast enough to change how often I’d use it
Good at getting most of the app done quickly
Still weak enough in backend reliability that I would not fully trust it yet for complex production work
Not as strong as Opus 4.6 / GPT-5.4 in coding depth, but still very usable

So yeah, I agree with the criticism that it is not on the same level as Opus 4.6 / GPT-5.4 for hard-coding tasks. ( may be because the base model is Kimi K2.5)

But I also think some people are dismissing it too quickly. If you judge it as a fast, cheaper, improved Composer, it is genuinely solid.

I shared a longer breakdown here with the exact build flow, where it got things right, and where it still fell short, in case anyone wants more context

Building an AI-Powered Content Moderation API with InsForge Edge Functions

Arindam Majumder — Fri, 20 Mar 2026 09:55:13 +0000

Introduction

Modern applications rely on user-generated content such as comments, reviews, and messages. Platforms must moderate this content to enforce safety policies and maintain compliance. Manual moderation does not scale, so production systems typically rely on automated moderation pipelines powered by AI.

Traditional implementations require multiple backend services. Developers often provision servers, integrate AI APIs, manage databases, and configure storage separately. This fragmented setup increases operational overhead and slows development.

InsForge simplifies this architecture by combining Edge Functions, PostgreSQL Database, Storage, and Model Gateway in a single platform. Benchmarks also show that it can deliver ~1.6× faster responses and 2.4x lower token usage compared to fragmented integrations.

In this tutorial, we will build a production-ready AI moderation API that runs entirely within InsForge.

What We Are Building

Here are the tools that we will be using to build a simple backend moderation workflow using InsForge core services:

AI Moderation API Endpoint: We will create an API endpoint using Edge Functions that accepts user-submitted text content and processes moderation requests.
AI-Powered Content Evaluation: The API will use Model Gateway to access an AI model that classifies submitted content as SAFE or UNSAFE.
Database Storage for Approved Content: Approved comments will be stored in a PostgreSQL Database managed by InsForge.
Attachment Handling with Storage: Optional user attachments will be uploaded and stored using Storage Buckets.
Automated Moderation Response: Unsafe content will be rejected immediately, and the API will return a structured moderation response.
Production-Ready Backend Workflow: The moderation pipeline will run entirely within InsForge using Database, Edge Functions, Model Gateway, and Storage, without external servers or additional infrastructure.

Project Setup and Repository Structure

Before configuring the backend resources, clone the project repository and review the project structure.

Clone the repository:

git clone https://github.com/Studio1HQ/Content-moderation-Insforge
cd content-moderation-app

Install dependencies:

npm install

The repository contains both the Next.js frontend and the InsForge Edge Function used for moderation.

Repository Structure

Folder	Purpose
`src/app`	Next.js application pages and layouts
`src/components`	UI components such as the moderation form
`src/lib`	Client utilities for connecting to InsForge APIs
`insforge-functions/moderate-comment`	Edge Function implementation for moderation
`handler.ts`	Serverless function that processes moderation requests

This structure keeps the frontend and backend logic organized within the same project while allowing the Edge Function to be deployed independently.

After cloning the repository, proceed with configuring the backend resources in InsForge.

Note: You can set up this backend in two ways. Follow the manual steps in this tutorial to create the database, storage bucket, and Edge Function using the dashboard and CLI. Alternatively, you can use InsForge MCP with your AI coding agent to provision the same resources using a single prompt. See the MCP section at the end of the article for the prompt template and instructions.

Step 1: Setting Up the Database

InsForge provides a managed PostgreSQL Database that you can configure directly from the dashboard.

Open the Tables Section

Open your project in the InsForge Dashboard.
In the left sidebar, select Tables.
Click the + icon next to Tables.

Create the following columns.

Column	Type	Description
`id`	`uuid`	Primary key for each comment
`content`	`string`	User submitted comment text
`attachment_url`	`string`	URL for uploaded file (optional)
`status`	`string`	Moderation result (`approved` or `rejected`)
`created_at`	`timestamp`	Time when the comment was created

Save the Table

Click Create Table to apply the schema.
The comments table will appear in the Tables panel.

Step 2: Creating the Edge Function

Next, create the serverless API that will process moderation requests.

InsForge Edge Functions allow you to run backend logic without managing servers. In this tutorial, the function receives user content, evaluates it using AI, and stores approved results in the database.

Navigate to the Edge Function directory in the repository:

insforge-functions/moderate-comment/

Inside this folder, there will be a file named:

handler.ts

This file will contain the moderation logic executed by the Edge Function.

The Edge Function performs the following tasks:

Accept a POST request containing user content.
Send the content to the AI model through Model Gateway.
Classify the content as SAFE or UNSAFE.
Upload attachments to Storage if present.
Insert approved content into the comments table.
Return a structured moderation response.

All moderation logic runs inside the Edge Function, keeping the backend workflow centralized within InsForge.

Deploy the function using the InsForge CLI:

insforge functions deploy moderate-comment--file ./insforge-functions/moderate-comment/handler.ts

Once deployed, the function becomes available as a backend API endpoint that the frontend application can call.

Step 3: AI Integration Inside the Function

The moderation logic inside the Edge Function uses Model Gateway, which provides unified access to multiple AI models directly within InsForge.

Model Gateway allows Edge Functions to call AI models without configuring external API clients or managing provider-specific integrations.

Open the Model Gateway section in the InsForge dashboard and enable a model for the project.

For this tutorial, enable:

openai/gpt-4o-mini

This model will be used to classify incoming content during moderation.

Use the CLI to send a test request to the moderation API.

insforge functions invoke moderate-comment--data"{\"content\":\"This community platform is very helpful.\"}"

This command sends a JSON payload containing the content field to the Edge Function.

The Edge Function also inserts the approved comment into the comments table in the database.

Step 4: Configuring Insforge Storage

The moderation workflow also supports optional file uploads using InsForge Storage. Storage provides an S3-compatible object storage system that integrates directly with Edge Functions and the database.

When a user submits a comment with an attachment, the Edge Function uploads the file to a storage bucket before inserting the comment into PostgreSQL.

Create a Storage Bucket

Open the Storage section in the InsForge dashboard.

Navigate to Storage in the sidebar.
Click Create Bucket.
Name the bucket: attachments

This bucket will store files uploaded with moderated comments.

The upload operation returns a public file URL, which is stored in the attachment_url column of the comments table.

The moderation function processes attachments as follows:

The user submits content with an optional file.
The Edge Function evaluates the text using AI moderation.
If the content is classified as SAFE, the file is uploaded to the attachments bucket.
The returned file URL is stored in the comments table.
If the content is UNSAFE, the function rejects the request and no file is uploaded.

This ensures that only approved content and attachments are stored, keeping the storage system aligned with the moderation rules.

Step 5: Building the Next.js UI

The repository already includes a Next.js application that provides a simple interface for interacting with the moderation API.

Navigate to the frontend code inside the src directory.

Key UI Files

File / Folder	Purpose
`src/app/page.tsx`	Main page that renders the moderation interface
`src/components`	Reusable UI components for the moderation workflow
`src/lib/insforge.ts`	Utility for connecting the frontend to the InsForge backend

The UI includes a form where users submit content for moderation.

The form collects:

Text content entered by the user
Optional file attachment
Submit an action that triggers the moderation request

When the user submits the form, the application sends a POST request to the Edge Function endpoint.

The UI handles the API response and updates the interface accordingly.

Approved comments appear in the moderation results section.
Rejected content displays an error message.
Approved entries are also visible in the comments database table.

This setup creates a complete workflow where the Next.js UI communicates with the InsForge Edge Function to perform moderation in real time.

Using an AI Agent to Build the UI

You can also accelerate this step using an AI coding agent (such as Cursor, Claude Code, or other agent-based tools). Instead of manually writing the UI components, the agent can generate the form, API calls, and component structure based on a prompt.

Example prompt:

Create a Next.js page for a content moderation demo.

Requirements:
- A form with a textarea for user comments
- An optional file upload input
- A submit button
- Send a POST request to the InsForge Edge Function endpoint for moderation
- Display the moderation result (approved or rejected) in the UI
- Use React state to handle form submission and responses

Step 6: Testing the API Endpoint

After deploying the Edge Function and setting up the UI, test the moderation workflow to verify that the API behaves correctly.

Submit Safe Content

Enter a comment through the UI and submit the form.

Expected behavior:

The Edge Function sends the content to the AI moderation model.
The model classifies the text as SAFE.
The function inserts the comment into the comments table in PostgreSQL.
If an attachment is included, the file is uploaded to the attachments storage bucket.
The API returns an approved response to the frontend.

Next, test a rejection case.

Expected behavior:

The Edge Function sends the text to the AI moderation model.
The model classifies the content as UNSAFE.
The function immediately returns a rejection response.
No entry is inserted into the comments table.
No file is uploaded to Storage.

The table in your Insforge dashboard also reflects the results:

Step 6: Deployment Using InsForge

Once the function and UI are ready, deploy the backend using the InsForge CLI. This publishes the Edge Function and connects it to the project environment.

Refer to the deployment guide here

Authenticate the CLI with your InsForge account.

insforge auth login

Complete the authentication process in the browser. Link the local project directory to your InsForge backend.

insforge link

Select the project created earlier in the InsForge dashboard. This connects the CLI to the correct backend workspace.

Deploy the Next.js application while passing the required environment variable.

insforge deployments deploy .--env"{\"NEXT_PUBLIC_INSFORGE_BASE_URL\":\"https://your-project.insforge.app\"}"

This environment variable allows the frontend to communicate with the deployed Edge Function.

Verify the Deployment

After deployment, the application becomes accessible via the InsForge-hosted domain.

Access the live demo here

Using MCP to Accelerate Development

Instead of manually creating tables, storage buckets, and Edge Functions, you can also configure the backend using Remote MCP (Model Context Protocol).

MCP exposes InsForge backend capabilities as tools that an AI coding agent can call to provision resources automatically. With a single prompt, the agent can generate the database schema, configure storage, and deploy the moderation function.

Example prompt used to create this backend workflow:

Create backend resources for a content moderation application using InsForge.

Requirements:

1. Create a PostgreSQL table named "comments" with fields:
   id (UUID primary key)
   content (text)
   attachment_url (text, nullable)
   status (text)
   created_at (timestamp)

2. Create a storage bucket named "attachments" for storing uploaded files.

3. Create an Edge Function named "moderate-comment" that:
   - accepts POST requests with comment text
   - sends the text to an AI model
   - classifies the content as SAFE or UNSAFE
   - uploads attachments to storage if present
   - inserts approved content into the database

Using MCP, developers can provision backend resources and deploy functions directly from prompts, significantly accelerating backend setup while keeping the same architecture described in this tutorial.

Refer to the quick demo here.

Conclusion

In this tutorial, we built a content moderation API using InsForge Edge Functions, integrated AI-powered classification through Model Gateway, stored approved results in PostgreSQL, and handled optional file uploads with Storage. The entire workflow runs inside InsForge, without external servers or fragmented infrastructure.

This approach demonstrates how developers can combine Edge Functions, AI integration, database services, and storage to implement production-ready backend APIs with minimal operational overhead.

If your application relies on user-generated content, moderation pipelines, or AI-assisted workflows, this architecture provides a straightforward and scalable foundation.

Ready to simplify your backend stack? Explore InsForge’s Edge Functions, Model Gateway, PostgreSQL database, and Storage services to build intelligent APIs without managing infrastructure.

Try InsForge
Quickstart guide here

Cursor Composer 2: Features, Pricing, Benchmarks, and Initial Impressions

Arindam Majumder — Thu, 19 Mar 2026 20:25:28 +0000

Introduction

Cursor has released Composer 2, the latest version of its in-house coding model.

The announcement is focused and fairly easy to summarize. Cursor is making three main claims:

Composer 2 is frontier-level at coding
it is materially better than previous Composer versions on Cursor’s published benchmarks
it is priced aggressively enough to be practical for everyday use

That combination makes the release worth paying attention to. In this post, I’ll walk through what Composer 2 is, what Cursor says improved, how the benchmark results look, what the pricing means, and my initial take on the release.

What Is Composer 2?

Composer 2 is Cursor’s latest in-house coding model.

Cursor describes it as frontier-level at coding and positions it as a better cost-performance option for agentic software work. The model is now available in Cursor, and the announcement puts most of the emphasis on three areas:

stronger coding performance
improved long-horizon task handling
lower cost than many competing fast models

Unlike some model launches that bundle a large number of product features together, this one is mostly about the model itself. Cursor is not presenting Composer 2 as a general platform shift. It is presenting it as a more capable and more economical coding model.

Composer 2 Key Features

The Composer 2 announcement is short, but there are still a few important takeaways.

Better coding performance

Cursor says Composer 2 delivers large improvements on all of the benchmarks it tracks, including Terminal-Bench 2.0 and SWE-bench Multilingual.

That matters because it suggests the gains are not limited to one internal evaluation. Cursor is showing improvement across several coding-oriented benchmarks rather than relying on a single headline number.

Continued pretraining

One of the most notable details in the post is that these improvements come from Cursor’s first continued pretraining run.

This is important because continued pretraining is often what gives a model a stronger base before more specialized post-training methods are applied. Cursor is explicitly saying that Composer 2 starts from a better foundation than earlier Composer versions.

Reinforcement learning for long-horizon tasks

Cursor also says it trains Composer 2 on long-horizon coding tasks using reinforcement learning.

This is probably the most interesting technical claim in the announcement. Cursor says Composer 2 can solve challenging tasks requiring hundreds of actions. That implies the model is being optimized for sustained multi-step software tasks, not just short code completions or simple edits.

A fast variant with the same intelligence

Cursor also introduces a faster Composer 2 variant and says it has the same intelligence.

That is a useful product choice. Instead of forcing users to pick between a “smart” model and a “fast” model family, Cursor is presenting speed as a deployment option on top of the same underlying capability level.

Composer 2 Benchmarks

Cursor publishes three benchmark comparisons in the announcement:

Model	CursorBench	Terminal-Bench 2.0	SWE-bench Multilingual
Composer 2	61.3	61.7	73.7
Composer 1.5	44.2	47.9	65.9
Composer 1	38.0	40.0	56.9

These gains are large enough to be meaningful.

The biggest point here is not just that Composer 2 is ahead of Composer 1 and 1.5, but that the improvements show up consistently across all three benchmarks. That gives the release more credibility than a single isolated result would.

Terminal-Bench 2.0 is especially relevant because Cursor frames it as an evaluation for agentic terminal use. If Composer 2 is genuinely stronger there, that supports Cursor’s claim that the model is getting better at longer, more interactive coding tasks.

SWE-bench Multilingual is also worth noting because it suggests broader coding competence beyond narrow English-only setups.

Still, these are vendor-published numbers, so the right takeaway is measured optimism rather than certainty.

How Composer 2 Is Priced

Cursor says Composer 2 is priced at:

$0.50 per million input tokens
$2.50 per million output tokens

The faster variant is priced at:

$1.50 per million input tokens
$7.50 per million output tokens

Cursor also says the fast variant has lower cost than other fast models and that fast will be the default option.

This part of the announcement is more important than it looks. Model releases are usually judged on benchmark quality first, but pricing determines whether a model becomes part of normal daily use or gets reserved for occasional high-value tasks. Cursor is clearly trying to push Composer 2 into the first category.

On individual plans, Composer usage is part of a standalone usage pool with generous usage included.

Composer 2 vs Earlier Composer Versions

Based on Cursor’s published table, Composer 2 is a clear step up from Composer 1.5 and Composer 1.

The improvement is visible across all the benchmarks included in the post, and Cursor attributes that jump to a combination of:

a stronger base model from continued pretraining
reinforcement learning on long-horizon coding tasks

That is a sensible recipe for a coding model. Better base training improves general capability, while long-horizon RL helps the model stay coherent over extended multi-step tasks.

From the announcement alone, Composer 2 looks like a real model upgrade rather than a minor iteration.

Initial Impressions

My first impression is that this is a disciplined release.

Cursor is not trying to claim that Composer 2 changes everything. The message is narrower and more believable: the model is better, it handles long-horizon coding tasks more effectively, and it is priced aggressively enough to be useful in regular workflows.

The long-horizon point is the one I would pay most attention to. A lot of coding models can produce a good patch in one pass. Fewer models stay reliable across a task that unfolds over many actions. If Composer 2 is genuinely stronger there, that is a meaningful improvement.

The pricing is the other major strength. A coding model can be strong on benchmarks and still be awkward in practice if the economics are wrong. Cursor seems to understand that and is making cost a central part of the launch rather than an afterthought.

At the same time, this is still an announcement built around Cursor’s own evaluation framing. The benchmark gains look strong, but the real test will be whether Composer 2 feels materially better in day-to-day software work.

Final Thoughts

Composer 2 looks like a meaningful upgrade to Cursor’s coding model stack.

The release is compelling for three reasons:

the benchmark gains are substantial
the training story is technically coherent
the pricing is practical

If you already use Cursor, Composer 2 is worth trying.

If you evaluate coding models more broadly, this release is notable because it tries to improve both capability and economics at the same time. That is the right combination to optimize for.

Running LLM Applications Across Providers with Bifrost

Arindam Majumder — Tue, 17 Mar 2026 16:15:23 +0000

Many modern applications include AI features that rely on large language models accessed through APIs. When an application sends a prompt to a model and receives a response, that request usually goes through an external service.

Getting access to different LLM models is easier today. Providers such as OpenAI and Anthropic provide model APIs, and platforms like Amazon Bedrock and Google Vertex AI give access to several models from one place. Because of this, many applications connect to more than one provider to compare models, manage cost, or keep a backup option if one service fails.

But each provider works a little differently. Authentication methods, rate limits, and request formats are not the same. Managing these differences inside an application can slowly add complexity to the system. In this article, let us explore Bifrost, an open-source LLM gateway that provides a single layer to route requests and manage interactions with multiple model providers.

The Hidden Cost of Provider Integrations

Connecting to several LLM providers may look simple at the start. Adding another provider can feel like just integrating one more API.

That situation changes once the application runs in production. Requests may need to go to different models based on cost, response quality, or latency. If a provider slows down or becomes unavailable, the system must redirect requests to another provider and keep the service running.

Handling these situations introduces additional logic into the codebase. The application needs to manage how requests are routed between models. It must also include retry logic for failed calls, fallback providers during outages, and tracking for how requests are distributed across models.

Each of these responsibilities adds extra work to the system. Over time, operational logic becomes part of the application and increases maintenance effort. This overhead becomes the hidden cost of working directly with multiple model providers.

Introducing Bifrost: A Gateway for LLM Infrastructure

Bifrost is an open-source LLM and MCP gateway designed to manage interactions between applications and model providers. It sits between the application and the LLM services and acts as a central layer that controls how requests move between systems.

Applications often connect directly to each provider they use. Bifrost adds a gateway layer between the application and the providers, so requests pass through a single entry point before reaching the model services.

This structure separates provider management from the application. The application sends requests to one endpoint, and the gateway manages communication with different model providers. Provider configuration and request handling stay inside the gateway layer, reducing provider-specific logic in the application code.

Core Infrastructure Capabilities

Bifrost provides several infrastructure capabilities for managing LLM interactions across providers. These capabilities move provider-specific handling out of the application and into the gateway layer.

Multi-provider routing: Bifrost supports multiple AI providers through a single API interface. Applications send requests to one endpoint, and the gateway routes each request to the configured provider or model.
Load balancing: When multiple providers or API keys are configured, Bifrost distributes requests across them based on defined rules. Traffic spreads across providers and reduces the chance of hitting rate limits on a single service.
Automatic fallback: When a provider returns an error or becomes unavailable, Bifrost sends the request to another configured provider.
Semantic caching: Bifrost stores responses and returns them for similar prompts. Prompt comparison uses semantic similarity. This reduces repeated API calls and improves response time.

Platform Support and Integrations

Bifrost fits environments where applications use multiple models and providers. The gateway exposes an OpenAI-compatible API, so applications that already use OpenAI SDKs can connect with minimal changes and send requests through a single endpoint.

Bifrost works with several LLM providers, such as OpenAI, Anthropic, Amazon Bedrock, Google Vertex AI, Cohere, and Mistral. Applications can reach these providers through the same gateway interface.

The gateway also supports the Model Context Protocol (MCP). Systems that use MCP can connect tools and external services through the same layer used for model requests. Bifrost also includes a plugin system for adding custom behavior such as request validation, logging, or request transformation.

Bifrost can run using tools such as NPX or Docker and can operate in local setups or production environments. The project is open source under the MIT license and can run across different infrastructure environments.

Gateway Performance and Benchmark

A gateway processes every request sent to a model provider. The performance of this layer becomes important in systems that handle a large number of AI requests.

Bifrost is written in Go, a language often used for backend services that process many requests simultaneously. The system focuses on keeping the extra processing time very small.

Benchmark tests show that Bifrost adds about 11 microseconds of latency at 5,000 requests per second. One microsecond equals 0.001 milliseconds, so 11 microseconds equals 0.011 milliseconds, which means the delay introduced by the gateway remains extremely small.

The published benchmarks were executed on AWS EC2 t3.medium and t3.large instances. These are cloud virtual machines with moderate CPU and memory resources that are commonly used to run backend services and APIs.

Bifrost also provides a public benchmarking repository with the scripts and setup used in the tests. Anyone can run the same tests or perform custom benchmarking based on their own infrastructure, traffic patterns, or model providers.

Getting Started with Bifrost

Bifrost is designed for quick setup and can run locally or in a server environment. The gateway can start in a few steps and begin routing LLM requests through a single endpoint.

One way to start Bifrost is by using NPX:

npx -y @maximhq/bifrost

Bifrost can also run using Docker, which allows the gateway to start inside a container environment:

docker run -p 8080:8080 maximhq/bifrost

After the gateway starts, applications can send LLM requests to the Bifrost endpoint. The gateway then routes the requests to the configured model providers.

Configuration options allow the gateway to define providers, API keys, routing rules, caching behavior, and fallback settings. These configurations control how requests move between different LLM providers.

Closing

Managing several LLM providers inside an application can introduce extra operational logic and maintenance effort. A gateway layer offers a cleaner structure for handling these interactions.

Bifrost provides this layer by placing a gateway between applications and model providers. Requests go through one endpoint, and the gateway manages routing and provider communication.

This approach keeps provider integrations outside the core application code and places request management in a separate infrastructure layer.

To explore configuration options, deployment steps, and additional features, refer to the official Bifrost documentation.

5 OpenClaw Plugins That Actually Make It Production-Ready

Arindam Majumder — Fri, 13 Mar 2026 15:19:20 +0000

Introduction

There is a certain point every serious OpenClaw user reaches. The agent is running, the setup works, and then slowly, almost without noticing, the cracks show up. A workflow that should take seconds starts requiring three follow-up prompts. The context window fills up with things the agent should already know. The API bill at the end of the month is higher than expected, and there is no clear answer for why.

Most people at this point start tweaking their skills, adjusting prompts, or switching models, but the problem is usually none of those things.

OpenClaw's default configuration is designed to get you started, not to match how you actually use it. The real power that makes it suitable for daily professional use lies in the plugin layer, yet most OpenClaw users have never explored it.

In this post, we are covering five OpenClaw Plugins, and each solves a different problem, each adding a layer that the default setup simply does not have. But before getting into the plugins themselves, it is worth understanding what separates a plugin from a skill.

What Are OpenClaw Plugins (And Why They're Different from Skills)

If you have spent any time in the OpenClaw community, you have probably seen both terms used interchangeably. They are not the same thing, and the distinction matters more than it seems.

A skill is a markdown file, specifically a SKILL.md that gets injected into the agent's context at inference time. It shapes how the agent thinks, what tone it uses, and what steps it follows. Every time the agent runs, that file loads into the prompt. Skills are useful for behavior, but they come at a cost: they consume tokens on every single request, whether or not they are relevant to what you asked.

A plugin is fundamentally different. It is a standalone executable that runs as a separate process alongside OpenClaw. Instead of loading into context, it exposes a set of tools through a defined interface that the agent can call when it actually needs them. OpenClaw loads plugins once at startup and calls into them only when a task requires it. No tokens consumed just by existing.

How to Install a Plugin

Installing any plugin follows the same pattern:

openclaw plugins install <plugin-name>

That command downloads the plugin, registers it in your OpenClaw configuration at ~/.openclaw/openclaw.json, and makes its tools available the next time the agent starts. You can open that file at any time to see which plugins are currently registered and adjust their individual configurations.

To confirm a plugin is active after installation, run:

openclaw plugins list

This returns all registered plugins and their current status. If something is not showing up, a full restart of the OpenClaw daemon is usually all it takes.

With that covered, here are the five plugins worth adding to your setup.

1. Manifest

When you configure OpenClaw, you pick a default model. Claude Opus, GPT-4, whatever you prefer. From that point on, every request, regardless of its type, goes to that model. Asking the agent to list files in a directory costs the same as asking it to debug a race condition across three services. The model does not know the difference, and OpenClaw does not try to make one.

This is where most API bills quietly spiral. Not from one expensive task, but from hundreds of simple ones hitting a premium model they never needed.

Manifest sits between OpenClaw and your LLM providers. Every request passes through it before reaching a model. It reads the request, classifies the task complexity, and routes it to the cheapest model capable of handling it. Simple lookups go to lighter models. Reasoning-heavy tasks escalate to whatever model can actually handle them. Routing occurs in milliseconds and is invisible to the agent; it only sees a response.

The cost difference compounds fast. Users running OpenClaw through Manifest have reported up to 70% reduction in monthly API spend, not by doing less, but by stopping the habit of paying Opus prices for Haiku-level work. The Manifest dashboard makes this visible: you can see cost broken down per session, per tool call, and per model, so you know exactly where your spend is going and whether the routing decisions are working as expected.

Installing Manifest:

openclaw plugins install manifest

Once installed, Manifest registers itself as the default routing layer. You can configure routing thresholds and model preferences in ~/.openclaw/openclaw.json under the manifest plugin entry.

Manifest makes the biggest difference in setups where the agent runs long sessions, handles multi-step tasks, or operates overnight without supervision. The more requests flow through OpenClaw, the more the routing logic saves, because the inefficiency it fixes is not a one-time cost; it is per-request.

2. Composio

Out of the box, OpenClaw cannot reach your Gmail, Slack, GitHub, or Notion. Not because the agent is incapable, but because every external service requires OAuth authentication, token management, and refresh handling, none of which OpenClaw sets up for you. Most people work around this by manually generating API keys, pasting them into configuration files, and hoping the tokens don't expire mid-session. It works until it does not.

Composio solves this at the authentication layer. It runs as an MCP server that sits between OpenClaw and every external app you want the agent to reach. You connect your accounts once through the Composio dashboard, and from that point on, OpenClaw talks to Composio, and it handles everything else. Token refresh, OAuth flows, rate limits, API versioning. None of that touches your OpenClaw config directly.

Each app connection runs in an isolated MCP session. If one integration fails or a token expires, it does not affect the others. The agent continues operating normally while Composio handles the reconnection in the background.

Installing Composio:

openclaw plugins install composio

After installation, connect your apps through the Composio dashboard and add the plugin entry to ~/.openclaw/openclaw.json:

{
  "plugins": {
    "composio": {
      "apiKey": "your-composio-api-key"
    }
  }
}

In practice, what this unlocks is straightforward. A single prompt like "summarize my unread emails, open a GitHub issue for anything that needs follow-up, and post a summary to the team Slack channel" now executes end to end, no switching tabs, no copying API keys, no manual auth setup. The agent has the required access, and Composio ensures it remains valid.

With 850+ supported apps, Composio covers most of what a professional OpenClaw setup would ever need to reach.

3. Hyperspell

OpenClaw's default memory is a MEMORY.md file. It grows with every session, gets compacted when it reaches a limit, loses information in the process, and reloads entirely on every turn, whether the content is relevant or not. For occasional use, this is fine, but for anyone relying on OpenClaw daily, it becomes a real problem fast.

Hyperspell replaces this layer entirely. It indexes your connected data sources, emails, documents, and past conversations into a knowledge graph, then injects only the relevant slice of that graph before each agent turn. The agent gets what it needs, not everything it has ever seen.

Memory also becomes sharper over time. Every query refines how the knowledge graph is indexed, so context recall improves the more you use it. An agent running Hyperspell can reference a decision you made three weeks ago without you having to bring it up.

Installing Hyperspell:

openclaw plugins install @hyperspell/openclaw-hyperspell

Connect your data sources through the Hyperspell dashboard, then add your API key under the hyperspell entry in ~/.openclaw/openclaw.json. Context injection is automatic from there.

4. OpenClaw Foundry

Most workflows repeat. You run the same sequence of tasks every morning, follow the same steps every time a PR needs review, and ask the agent the same three things before a meeting. OpenClaw handles all of these, but it handles them the same way every time, waiting for you to prompt it from scratch. It does not recognize the pattern. It does not try to make things easier on its own.

Foundry fixes this. It sits in the background during your sessions, watches what you ask for, and, when it detects a recurring pattern, writes a new tool definition into itself. That tool becomes part of the agent's available toolkit the next time you start a session, no manual configuration required.

What makes this different from writing a skill is the output. A skill adds behavioral instructions to the agent's context. Foundry creates an executable tool that the agent can call, with its own inputs and outputs, registered in the tool registry and available on demand.

Installing Foundry:

openclaw plugins install @getfoundry/foundry-openclaw

This downloads the plugin from npm, extracts it to ~/.openclaw/extensions/foundry/, enables it automatically, and restarts the gateway. After that, add the following to ~/.openclaw/openclaw.json:

{
  "plugins": {
    "entries": {
      "foundry": {
        "enabled": true,
        "config": {
          "autoLearn": true,
          "sources": {
            "docs": true,
            "experience": true,
            "arxiv": false,
            "github": false
          },
          "marketplace": {
            "autoPublish": false
          }
        }
      }
    }
  }
}

autoLearn: true is the key setting, it tells Foundry to continuously learn from your sessions without requiring you to trigger it manually. The sources block controls where Foundry pulls additional context when writing new tools: OpenClaw's own documentation, your past session experience, arXiv papers, and public GitHub repos. For most setups, keeping docs and experience enabled is enough to start.

5. Opik

Multi-step agent runs fail in non-obvious ways. A tool call returns incorrect output, a sub-agent silently errors out, and a model call takes 12 seconds on a task that should take 2 seconds. Without structured tracing, you are left reading raw logs and guessing. That gets old fast.

Opik is an open-source LLM and agent observability platform built by Comet ML. The OpenClaw plugin hooks into the gateway process and exports a structured trace for every run, LLM request and response spans, tool call inputs and outputs, sub-agent lifecycle events, latency at each step, and token usage with cost. Every event that matters has a corresponding span in the Opik dashboard.

Image reference: https://github.com/comet-ml/opik-openclaw

This is a different layer from what Manifest covers. Manifest tells you how much a request costs and which model handled it. Opik tells you what the agent actually did inside that request, which tools it called, in what order, what each one returned, and where the run slowed down or failed. Both answer different questions and neither replaces the other.

Installing Opik:

Requirements: OpenClaw >=2026.3.2, Node.js >=22.12.0

openclaw plugins install @opik/opik-openclaw

After installation, restart the gateway, then run the setup wizard:

openclaw opik configure

This validates your endpoint and API key and automatically writes the config. To verify everything is connected:

openclaw opik status

The recommended config in ~/.openclaw/openclaw.json looks like this:

{
  "plugins": {
    "entries": {
      "opik-openclaw": {
        "enabled": true,
        "config": {
          "apiKey": "your-api-key",
          "apiUrl": "https://www.comet.com/opik/api",
          "projectName": "openclaw",
          "workspaceName": "default"
        }
      }
    }
  }
}

For teams that cannot send trace data to a third party, Opik is fully self-hostable. Replace apiUrl with your own instance endpoint, and nothing else changes.

Where to Go From Here

Each plugin owns a distinct layer. Hyperspell handles context before the request starts. Manifest handles model routing during it. Composio handles external reach when the agent needs to act. Foundry watches for patterns across sessions and builds tools from them. Opik traces everything after the fact, so you know exactly what happened and why.

None of them overlap, and none of them are doing the same job twice. You can start with just one, whichever layer is causing the most friction in your current setup, and layer in the rest as your workflow grows.

Each plugin has its own documentation to read before configuring anything: Manifest, Composio, Hyperspell, OpenClaw Foundry, and Opik. The OpenClaw plugin docs cover the installation system in full if you want to go deeper on how plugins interact with the gateway.

[Boost]

Arindam Majumder — Sat, 07 Mar 2026 08:35:19 +0000

What is LLM Observability? The Complete Guide (2026)

Arindam Majumder ・ Mar 6

#ai #llm #observability #programming

What is LLM Observability? The Complete Guide (2026)

Arindam Majumder — Fri, 06 Mar 2026 07:59:12 +0000

TL;DR: LLM observability is your ability to understand what your language models are doing in production, not just whether they're up, but whether they're good.
This guide covers everything: what it is, how it differs from traditional monitoring, the four pillars, key metrics, RAG and agent observability, enterprise challenges, the current tools landscape, and how to implement it from scratch.

Shipping an LLM feature is no longer the hard part. Keeping it reliable, fast, and cheap in production is.

Most engineering teams hit the same wall about three months after their first production deployment:

responses quietly degrade,
costs balloon unexpectedly,
a customer escalation surfaces a class of failures nobody noticed.

And the team realizes they have almost no visibility into what the model is actually doing.

That's where we need LLM observability.

In this article, we'll understand what LLM observability is, why traditional monitoring isn't enough, the four pillars you need to instrument, and how to implement it, for simple deployments, RAG pipelines, and agentic systems alike.

Let's start with the problem that makes all of this necessary.

The Problem Nobody Talks About

You can have a perfect HTTP 200 response from an LLM API and still be on the edge of a production disaster.

The model responded. No error was thrown. The latency was acceptable. Your uptime check is green.

And somewhere in that response, the model hallucinated a fact, cited a policy that doesn't exist, or gave a customer the wrong refund amount, with complete confidence, in fluent prose.

This is the core problem that LLM observability exists to solve. Traditional software monitoring tells you whether your system worked. LLM observability tells you whether it worked well.

The gap between those two questions is where most enterprise AI deployments silently fail.

The LLM observability platform market was valued at $1.44 billion in 2024 and is projected to reach $6.80 billion by 2029, with a 36.3% CAGR. But market growth doesn't tell you what to do on Monday morning.

What is LLM Observability?

LLM observability is the practice of capturing, analyzing, and acting on telemetry data from large language model applications in production. It gives you the capacity to understand the internal states of an LLM system through its outputs, with which your team can ensure that the models function accurately, reliably, and safely at scale.

More precisely, LLM observability means capturing inference-level data, token usage, prompt content, response quality, error rates, latency breakdowns, and cost, and correlating it with user interactions to provide a complete, queryable picture of system behavior.

Before going further, three terms are worth distinguishing:

Term	Scope
LLM Monitoring	Continuously tracks performance metrics (latency, token usage, error rates) as a health signal
LLM Observability	Goes beyond monitoring & provides in-depth insight into how and why an LLM behaves the way it does
LLM Tracing	Captures request/response flow through an LLM pipeline, tracking inputs, intermediate stages, and outputs

Observability includes monitoring but adds root-cause analysis. Monitoring tells you something is wrong. Observability tells you why.

Why LLM Observability is Different from Traditional APM

Your platform team already runs Datadog, Grafana, New Relic, or some combination. You have RED metrics (Rate, Errors, Duration) on every service. Your infrastructure is well-monitored.

That's great but none of these are sufficient for LLMs.

Traditional APM was designed for deterministic systems. A successful API call means the function completed correctly. An HTTP 200 means the request was handled. These assumptions break completely the moment you introduce a language model.

They are non-deterministic in nature; they can give 5 different answers in 5 calls.

Here's the fundamental incompatibility:

Dimension	Traditional APM	LLM Observability
Primary question	Did it work?	Was it good?
Success signal	HTTP 200, no exception	Output quality score
Input space	Finite, structured	Infinite, natural language
Output behavior	Deterministic	Non-deterministic
Failure modes	Errors, timeouts	Hallucinations, drift, unsafe content
Cost model	Fixed compute	Variable (token-based)
Debugging unit	Stack trace	Prompt + context + response chain
Analysis type	Threshold alerts	Semantic search over interactions

The most dangerous LLM failures are silent. The model doesn't throw an exception when it makes something up. Degradations happen without a single error being raised.

A customer service bot starts giving inaccurate answers. A RAG pipeline silently returns less relevant chunks after an index update. A fine-tuned model's quality drifts after a base model version change by the API provider. None of these appear in your existing dashboards.

LLM observability doesn't replace traditional APM; it supplements it. You still need infrastructure health monitoring. You just need an additional layer that understands what's happening at the content level, not just the infrastructure level.

The Four Pillars of LLM Observability

Traditional observability has three pillars: logs, metrics, and traces. LLM observability keeps all three and adds a fourth that doesn't exist in traditional software monitoring at all.

Pillar 1: Metrics

Metrics in LLM observability cover two distinct layers, infrastructure performance and business quality, and you need both.

Performance metrics:

Time to First Token (TTFT) is the elapsed time between when a request is sent and when the first response token arrives. It's the primary latency signal for streaming interfaces because it's what users feel as "waiting." TTFT has two components: the prefill phase (the model processes all input tokens and builds its KV-cache) plus any scheduling or queue time. The longer the prompt, the higher the TTFT. This is why prompt size distribution is a leading indicator for latency degradation.
Token throughput (TPS) is the number of output tokens generated per second after the first token. This is your compute load metric. It's not the same as requests per second; a single request can be 50 tokens or 50,000. Token throughput tells you what the model is actually doing; RPS tells you how many requests arrived. Both matter, for different reasons.
End-to-end latency is what most teams track. But the unit that matters is percentiles, not averages. LLM response time distributions are heavily skewed by prompt length, context size, and load. p50 is your median user experience. p95 is your bad-day experience. p99 is where your SLA commitments should live. Average latency in a skewed distribution is a lie; it will always look better than what your worst users actually experience.

Error metrics:

Not all LLM errors are equal, and treating them as a single "error rate" metric is one of the most common mistakes in inference monitoring.

4xx errors: malformed requests, invalid parameters, context length violations. Client problem. The application team owns it.
429s: rate limit exhaustion. Capacity problem. Needs a quota increase or traffic shaping, not a bug fix.
5xx errors: infrastructure failures, model unavailability. Infra team incident. Pages differently from a 4xx.

When these are merged into a single error rate, a 429 storm looks identical to an infrastructure outage. You escalate to the wrong team, debug the wrong layer, and waste hours that your users are already feeling.

Nebius Token Factory separates these three error classes natively each tracked as its own signal, filterable by endpoint, region, and API key. This is how error monitoring should work by default.

Quality metrics:

These are unique to LLM observability and have no equivalent in traditional APM:

Groundedness score: Alignment between the response and source documents (critical for RAG)
Relevance score: How well the response addresses the actual user query
Faithfulness: Whether the response is supported by the retrieved context
Hallucination rate: Percentage of responses containing fabricated information
Refusal rate: Proportion of queries declined by safety filters

Cost metrics:

Token consumption drives your AI budget, and it can grow invisibly fast.

Track cost per request broken down by model, endpoint, team, and API key. Track input vs. output token ratios a single bloated system prompt, replicated across millions of requests, can materially change your monthly bill. Track cache hit ratio.

Most teams that audit their token usage find that a 30–50% reduction is achievable without any quality loss.

Nebius Token Factory tracks token throughput broken down by project, endpoint, and API key out of the box giving you the per-consumer cost visibility that most teams have to build custom tooling for.

Pillar 2: Traces

Traces are the backbone of LLM observability. A trace captures the complete lifecycle of a request as it moves through every component of your system, from the user's input through retrieval, re-ranking, the LLM call, post-processing, and back to the user.

The trace hierarchy for LLM applications:

Level	Description
Session	A multi-turn user conversation (groups related traces)
Trace	Complete end-to-end request lifecycle
Span	A discrete unit of work within the trace
Generation	A single LLM call, prompt in, completion out
Retrieval	A RAG document fetch operation
Tool Call	An external API call made by an agent
Event	A state milestone within a span

What each generation span should capture:

The exact prompt content (or a hash, if privacy-constrained)
Model name and version
Temperature and sampling parameters
Input token count and output token count
TTFT, total latency, and per-token latency
Cost in USD
Evaluation scores
Tool arguments and returns (for agents)
Retrieved document chunks and relevance scores (for RAG)

The critical discipline: propagate a single trace_id through every layer of the system. Application → retriever → guardrails → model call → post-processing. Without this thread, distributed traces are incoherent. Debugging becomes guessing.

Pillar 3: Logs

LLM logs differ from traditional application logs in one fundamental way: the payload is unstructured natural language, not structured error codes.

A JSON log of a successful API call tells you almost nothing about whether that call was valuable. You need a different approach.

Recommended log structure for every LLM call:

trace_id              # Propagated from parent span
timestamp
model_id
model_provider
input_tokens
output_tokens
latency_ms
ttft_ms
cost_usd
user_id               # If applicable
session_id
environment           # dev / staging / prod
application_name
error_type            # If applicable
evaluation_scores     # groundedness, relevance, etc.

Privacy-first default: Log metadata, not content. Token counts, model names, latency, and trace IDs by default, not raw prompt and response content. Prompts frequently contain PII. Enable full content capture only for authenticated sessions with explicit data governance controls.

A semantic search capability over stored prompts is eventually necessary at scale. Traditional log indexing is insufficient for natural language; you can't grep your way to the root cause in a production LLM incident when the relevant signal is "prompts that started producing low groundedness scores after Tuesday."

Pillar 4: Evaluation

This is the pillar that has no equivalent in traditional software observability. It is unique to LLM systems, and it is the pillar most teams skip.

Evaluation is the practice of systematically assessing the quality of LLM outputs. Not just whether requests were completed, but whether the responses were good by the standards your application requires.

LLM-as-Judge: A separate LLM evaluates the outputs of your primary LLM. Scores across dimensions: relevance, accuracy, coherence, safety, tone. Scales to production traffic volumes without human bottlenecks.
Online evaluation: Running evals on live traffic in real-time. Flags outputs that fall below quality thresholds as they happen.
Offline evaluation: Running evals on captured traces as a batch job. Slower but more thorough, uses more expensive evaluators on a sample of historical traffic.
Human annotation: The gold standard for precision. Human labelers reviewing flagged outputs and feeding corrections back into training datasets.

The practical implementation: define 3–5 evaluation metrics you will actually act on. Tracking 20 metrics and acting on none is worse than tracking 3 rigorously. For most applications, a good starting set is: groundedness (for RAG), response relevance, format compliance, and safety.

LLM Observability for RAG Pipelines

RAG (Retrieval-Augmented Generation) is now one of the most common LLM deployment patterns in the enterprise. It also introduces multiple independent observable stages, each of which can fail or degrade silently.

The observable stages in a RAG pipeline:

Query processing: User query embedding, query rewriting, or expansion
Retrieval: Vector database lookup, hybrid search, metadata filtering
Reranking: Scoring and ordering retrieved chunks by relevance
Context assembly: Stuffing ranked chunks into the prompt within context window limits
Generation: The LLM call
Post-processing: Citation extraction, formatting, safety filtering

Each stage has its own failure mode. Slow retrieval. Poor reranking quality. Context window overflow. Hallucination despite good retrieval. Good retrieval that the model ignores.

RAG-specific metrics:

Context relevance: Are the retrieved chunks actually relevant to the query?
Faithfulness/groundedness: Does the generated answer stay within what the retrieved documents support? (This is your RAG-specific hallucination detector.)
Answer relevance: Does the response address the original question?
Recall@k: Of all truly relevant documents in the corpus, what fraction was retrieved?
Chunk hit rate: How often at least one retrieved chunk was genuinely useful
Context window utilization: Truncation rates and overflow events

The key insight for RAG observability: evaluate components independently from the full pipeline. Good retrieval does not guarantee good answers. Good answers can sometimes emerge from poor retrieval.

You need both component-level evaluation (retrieval quality) and end-to-end evaluation (answer quality), and you need to track them separately, because they can disagree.

Wrap your retrieval function in a span that captures: which documents were fetched, their relevance scores, the query that produced them, and the latency. Link this span to the parent trace. When a user gets a bad answer, you need to be able to answer: "Was the retrieval bad, or was the model bad with good data?"

LLM Observability for Agentic Systems

Agents are the hardest observability problem in the LLM space.

In a simple LLM call, you have one input and one output. In an agentic system, a single user request might execute 15 LLM calls across multiple models, trigger 8 tool calls, read from memory, spawn sub-agents, and make decisions at branching points, all before returning a response.

None of this is visible in standard API logs.

Why agents are uniquely hard:

Execution path variability: The same input can produce completely different execution paths across runs. Slight phrasing changes, ambiguous instructions, or slight differences in retrieved memory can produce different tool call sequences.
Depth: Multi-hop reasoning chains mean errors can originate deep in the chain, far from where symptoms appear.
Temporal span: Agent tasks can run for minutes to hours. The request/response tracing model breaks down.
Silent failure: An agent might silently terminate early, return a partial result, or use the wrong tool, without any error being raised.

According to LangChain's State of AI Agents report, 89% of organizations have implemented some form of observability for their AI agents. But quality issues remain the primary production barrier, cited by 32% of organizations because monitoring that the agent ran is not the same as monitoring that the agent worked correctly.

Required trace data for agents:

Every LLM call: prompt, completion, token usage, latency, cost
Every tool call: function name, arguments passed, return value, latency
Every memory read/write
Every branching decision with the reasoning that led to it
Session context across multiple turns

The single most important capability: Step-level trace reconstruction. For any failed or low-quality agent run, you must be able to reconstruct the exact sequence of decisions the agent made. Not just "the agent returned a bad result", but "at step 7, the agent called the search tool with query X, received result Y, and then decided Z, which led to the failure."

Without that capability, debugging agentic failures is not debugging. It's guessing.

Enterprise LLM Observability Challenges

For enterprises, observability is not just a developer experience concern. It's a compliance, security, and governance concern.

Data Privacy

LLMs in enterprise settings operate on sensitive data, such as customer records, contracts, source code, financial data, and medical records. Standard observability that logs raw prompts can inadvertently create a regulated data lake nobody intended to build.

The practical implications:

PII in prompts: User inputs frequently contain names, account numbers, addresses, and medical information. Your observability pipeline needs PII scrubbing middleware before data is written to any storage system.
Prompt content as IP: Even without PII, system prompts may encode proprietary business logic, product roadmaps, or trade secrets. Who has access to that data in your observability tool?
Data residency: GDPR, HIPAA, and financial services regulations may prohibit transmitting prompt data to US-based SaaS platforms for EU-based or regulated-industry teams.

The default should be: log metadata, not content. Enable full prompt/response logging only where it's been explicitly reviewed and approved.

For teams running under GDPR or strict data residency requirements, Nebius Token Factory stores observability metrics in the EU-North region regardless of where inference runs and supports Zero Data Retention mode, where requests and outputs are never stored or reused. Both options exist because the right choice depends on your regulatory context.

Compliance Requirements

EU AI Act: Requires transparency, risk documentation, and human oversight for high-risk AI systems. Observability is the technical foundation for compliance; you cannot document risk you cannot measure.

GDPR: Data protection, right to explanation, right to erasure. Every AI-assisted decision that affects a user must be traceable. "Right to erasure" is particularly complex for LLMs; models don't support selective forgetting.

HIPAA: Any LLM processing patient information requires audit trails of every interaction, Business Associate Agreements (BAAs) with every vendor in the data path (including your observability tool), and strict access controls.

This is not theoretical. These requirements apply to production LLM deployments today, and the cost of non-compliance dwarfs the cost of proper observability infrastructure.

Multi-Tenancy

Enterprises serving multiple internal teams or external customers from shared LLM infrastructure face a set of observability challenges that single-tenant deployments don't.

Cost attribution requires per-request metadata tagging, team ID, product, and customer ID, so you can do showback and chargeback across cost centers. Without this, your AI costs are opaque at the organizational level, even if your infrastructure team can see the aggregate bill.

Nebius Token Factory's per-API-key and per-project filtering makes this practical without additional instrumentation each consumer's token usage, latency, and error rates are filterable independently.

Data isolation means Tenant A's prompts and conversation history must never appear in Tenant B's context. Your retrieval layer and vector stores must support tenant-scoped filtering, and your observability data access must be role-gated by the same tenant boundaries.

Differential SLAs require per-tenant performance tracking. If you've committed to different latency or availability targets for different customers or teams, you need to monitor against those targets independently.

Non-Determinism at Scale

A production LLM can return a different response to the same prompt across runs. This isn't a bug, it's inherent to how these systems work. But it fundamentally changes what monitoring means.

You cannot monitor LLM quality as binary pass/fail. You have to monitor distributions, trends, and statistical anomalies. A model behaving "correctly" on 97% of requests is different from one behaving correctly on 83%, but neither is 0% failure rate. You need statistical significance thresholds, not simple error rate alerts.

Model provider updates compound this. When OpenAI or Anthropic updates their base model, your fine-tuned model or prompt template may produce different outputs without any change on your end. Observability is the only way to catch this before users do.

The Current Tool Landscape

The LLM observability tool market has matured into three categories. Choosing the wrong one for your situation creates lock-in, privacy exposure, or observability blind spots.

Purpose-Built LLM Observability Platforms

Langfuse: Open source (MIT license), self-hostable, framework-agnostic. Best for teams with data residency requirements or those who want full control without per-request SaaS pricing at scale.
LangSmith: Commercial, deep LangChain/LangGraph integration. Best for teams already invested in the LangChain ecosystem who want native debugging tools.
Helicone: Proxy-based integration (change one URL, logging starts immediately). Best for fast time-to-value when using vanilla API calls.
Arize Phoenix: Open source, OpenTelemetry-native, strong for LLM and RAG evaluation. Best for teams wanting vendor-neutral instrumentation.
Maxim AI: Ultra-low latency gateway (<11 microseconds at 5,000 RPS) plus evaluation. Best for teams needing gateway + eval + observability in one platform.

Enterprise APM Platforms Adding LLM Support

Datadog LLM Observability: Native OTel GenAI Semantic Convention support (v1.37+) and "AI Guard" for real-time security guardrails. Best for enterprises already standardized on Datadog who want LLM observability in the same pane of glass as infrastructure monitoring.
Grafana + OpenLIT: For teams already running Prometheus/Grafana, OpenLIT (pip install openlit, two lines of setup) exports LLM metrics via OTLP to Grafana Cloud. Best for teams who want to stay in their existing observability stack.
Splunk: Hallucination detection, drift management, compliance audit trails. Best for enterprise security/compliance teams already standardized on Splunk.

Inference Platforms with Built-in Observability

An often-overlooked category: inference platforms that instrument your models at the infrastructure level, without requiring application-layer SDKs.

Nebius Token Factory is the clearest example of this approach. TTFT, token throughput, error categorization (4xx/429/5xx separately), active replica counts, and prompt size distributions are all tracked natively per endpoint, per API key, per region with near-real-time updates. Metrics export via Prometheus for integration with your existing Grafana dashboards.

The advantage of this approach: you get inference-layer observability without any instrumentation overhead in your application code. The disadvantage: it doesn't give you the application-layer context (session IDs, user IDs, evaluation scores) that purpose-built platforms provide. The right answer for most teams is both infrastructure observability at the inference layer and application observability at the SDK layer, unified in one Grafana dashboard.

How to Choose

Criteria	Recommendation
Data residency required	Self-hosted: Langfuse, Grafana + OpenLIT
Already on LangChain	LangSmith
Fastest time-to-value	Helicone (proxy)
Already on Datadog	Datadog LLM Observability
Evaluation-first approach	Arize Phoenix, Braintrust
Inference-layer visibility	Nebius Token Factory
Enterprise scale + compliance	Combination of the above

Final Thought

The teams that operate AI reliably are not necessarily the ones with the best models. They are the ones who can see what their models are actually doing in production and respond before their users notice.

LLM observability is not an ML concern or an infra concern. It is a product quality concern. Every blind spot in your observability stack is a failure mode that will eventually become an incident. The only variable is whether your team finds it first or your users do.

The four pillars metrics, traces, logs, and evaluation are your minimum. Start with the metrics and traces. Add evaluation before your team thinks you need it. Close the loop between observation and improvement.

Instrument everything. Ship nothing you can't see.

[Boost]

Arindam Majumder — Wed, 11 Feb 2026 20:15:31 +0000

Build a Multi-Agent Telecom Support System with CopilotKit & LangGraph JS

Arindam Majumder for CopilotKit ・ Feb 9

#webdev #ai #programming #opensource