DEV Community: Kumar Rohit

A beginner-friendly deep dive into LangSmith- traces, runs, observability, and why every LLM app needs this.

Kumar Rohit — Thu, 02 Jul 2026 07:05:51 +0000

Kumar Rohit

Jul 1

Stop Stalking Your Crush, Stalk Your Agents Instead: A LangSmith Deep Dive: Part - 1

#ai #monitoring #tooling #agents

6 min read

Stop Stalking Your Crush, Stalk Your Agents Instead: A LangSmith Deep Dive: Part - 1

Kumar Rohit — Wed, 01 Jul 2026 17:09:49 +0000

LangSmith is a monitoring and observability platform built by the creators of LangChain and LangGraph for tracing AI applications.

But before diving into LangSmith, let's first understand what observability and monitoring actually mean.

Observability is essentially keeping a close eye on your AI applications while they run — tracking exactly what input goes into each step and what output comes out of it. Take a RAG (Retrieval-Augmented Generation) application as an example. It's made up of several moving parts: vector stores, retrievers, documents, embeddings, and the LLM itself. Observability here means logging every single input and output as it flows between these components. This way, when something breaks (and something always breaks), you know exactly where to look instead of guessing.

Monitoring, although it sounds similar to observability, is actually a different concept. Monitoring is the process of tracking your system or application's metrics as a whole — things like latency across different runs, the cost of one end-to-end execution, and so on.

In short: Observability tells you what happened and why, while monitoring tells you how well things are performing overall. You need both — monitoring flags that something's wrong (say, latency spiked at 3 PM), while observability helps you drill down and find out exactly which component caused it.

Why do LLM apps specifically need this?

Traditional software is predictable. If you call a function with the same input, you get the same output — every single time. When something breaks, you add a print statement, check the logs, find the line, fix it. Done.
LLM applications don't work like that.The output is non-deterministic. The same prompt can produce a different response on every single run. So when a user complains "the answer was wrong," you can't just reproduce it and debug it. That exact run is gone — unless you logged it.
The pipeline has multiple steps, each of which can silently fail. Take a RAG application. When your app gives a wrong answer, where did it go wrong?

Did the vector store retrieve the wrong documents?
Did the retriever rank them poorly?
Did the prompt template stuff too much context in?
Did the LLM just hallucinate despite having the right context?
Or did the change in prompt caused it?

Without observability you can just guess and play catch up but not get the exact root cause of the problem. You'd have to manually test each component in isolation, which is slow and doesn't reflect what actually happened during that specific run. And if the workflow or application is complex containing lot of components, finding the issue would become a nightmare.

Every LLM call costs money — and in a multi-step pipeline, you might be making 5 to 10 LLM calls per user request without realizing it. Without monitoring, you have no idea which step is burning your budget. Is it the query rewriter? The summarizer? The final answer generator? You won't know until your API bill arrives.

This is exactly why LangSmith exists.

Let's learn about some core concepts of LangSmith -

Projects - A Project in LangSmith is simply a container for one of your AI applications. Every trace and run gets logged under a project so your data stays organized and separated.
Trace - A Trace represents one complete end-to-end execution of your application — from the moment a user sends an input to the moment your app returns a final response.

For example, a user asks: "What is the return policy?"
That single question triggers your entire RAG pipeline — retrieval, reranking, prompt construction, LLM call, response generation. All of that together, from start to finish, is one trace.

Runs - If a Trace is the full journey, a run is each individual step along the way.

Inside that one trace of "What is the return policy?", LangSmith breaks it down into runs:

Trace: "What is the return policy?"

Run 1: Embed the user query
Run 2: Retrieve documents
Run 3: Rerank documents
Run 4: Construct prompt
Run 5: LLM call

Setting up LangSmith

Step 1: Create a LangSmith Account
Head over to smith.langchain.com and sign up. Once you're in, navigate to Settings → API Keys and generate a new API key. Copy it somewhere safe.

Step 2: Create a Project
Once you're inside the LangSmith dashboard, create a new project. Give it a meaningful name that matches your application.

Step 3: Install the Package
pip install langsmith

Step 4: Make sure to have these environment variables

.env
LANGCHAIN_TRACING_V2=true  -- Turns tracing on/off — set to true to enable
LANGCHAIN_API_KEY=your-langsmith-api-key
LANGCHAIN_PROJECT=your-project-name -- Which project to send traces to

That's literally it. These three environment variables are all LangSmith needs to start capturing traces.

Let's see a simple langchain workflow in action tracked by LangSmith -

from dotenv import load_dotenv
from langchain_openai import ChatOpenAI
from langchain_core.prompts import PromptTemplate
from langchain_core.output_parsers import StrOutputParser

# Load environment variables
load_dotenv()

# Prompt to generate a detailed report
prompt1 = PromptTemplate(
    template="Generate a detailed report on {topic}",
    input_variables=["topic"],
)

# Prompt to summarize the report
prompt2 = PromptTemplate(
    template="Generate a 5 pointer summary from the following text:\n\n{text}",
    input_variables=["text"],
)

# Initialize models
model1 = ChatOpenAI(model="gpt-4o-mini")
model2 = ChatOpenAI(model="gpt-4o")

# Output parser
parser = StrOutputParser()

# Create sequential chain
chain = (
    prompt1| model1| parser| prompt2| model2| parser
)

# Configuration for tracing
config = {
    "tags": ["llm_app", "report generation", "summarization"]
}

# Invoke chain
result = chain.invoke(
    {"topic": "Artificial Intelligence"},
    config=config,
)

print(result)

The screenshot above shows a real LangSmith trace for a simple Sequential LLM application.

The middle panel breaks down the six runs inside this trace — PromptTemplate formatted the input, gpt-4o-mini made the first LLM call (13.68s, 1.1K tokens), StrOutputParser cleaned the output, then the chain continued with another PromptTemplate, a second LLM call via gpt-4o (4.85s, 1.4K tokens), and a final StrOutputParser. On the right, you can see the exact input (topic: AI Opportunity in India) and the structured output the LLM returned.

This is exactly what makes LangSmith powerful. When something goes wrong, you don't guess — you just open the trace and see precisely where it broke.

LangSmith works out of the box with LangChain and LangGraph — no extra setup needed. However, if your pipeline includes components that aren't natively part of these frameworks, LangSmith won't trace them automatically. For those cases, you can wrap the function with the @traceable decorator and LangSmith will capture it just like any other run.

from langsmith import traceable
from openai import OpenAI

client = OpenAI()

@traceable  # LangSmith will trace this function
def call_llm(question: str) -> str:
    response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": question}]
    )
    return response.choices[0].message.content

result = call_llm("What is RAG in AI?")
print(result)

The @traceable decorator tells LangSmith — "treat this function as a run, log its input and output." Works with any Python function, any framework.

Alternatives of LangSmith

Langfuse
Langfuse is open source and self-hostable, meaning your data never leaves your own infrastructure. It works with any LLM framework — not just LangChain — and comes with prompt versioning and evaluation built in. Best choice if data privacy is a concern.

Helicone
Helicone works as a proxy between your app and the LLM provider. You change one base URL and it automatically starts logging every request — tokens, cost, latency. No code changes needed. Best for teams who just want clean cost and usage visibility without a full observability setup.

Arize Phoenix
Phoenix runs completely locally — no cloud, no data sharing. It goes beyond just tracing, offering embeddings visualization and dataset analysis. Best suited for ML teams doing serious evaluation or fine-tuning work alongside production monitoring.

Conclusion

Let's zoom out and look at what we covered -

We started with a simple truth — LLM applications are fundamentally different from traditional software. They're non-deterministic, multi-step, and fail silently. A wrong answer doesn't throw an error. It just quietly erodes your user's trust until they stop using your product.

Observability and monitoring are your defense against that. Observability tells you what happened and why at every step. Monitoring tells you how well your system is performing over time. You need both. LangSmith gives you both — wrapped in a clean UI that integrates natively with LangChain and LangGraph with almost zero setup effort.

The best time to add observability to your LLM app is before you need it — because by the time something breaks in production and a user is complaining, you'll wish you had the trace from that exact run.

So stop stalking your crush and start stalking your agents. Every trace tells a story, and the better you understand those stories, the easier it becomes to build reliable AI applications.

In Part 2, we'll go beyond tracing and explore how LangSmith helps you evaluate prompts, create datasets, run experiments, collect user feedback, and continuously improve your LLM applications.

How I Built an AI-Powered Personal Chatbot That Represents Me Professionally (And How You Can Too)

Kumar Rohit — Fri, 03 Oct 2025 15:34:11 +0000

The Problem: Lost Opportunities in Professional Networking

Have you ever visited someone's professional website and wished you could just ask them directly about their experience? Or worse, have you missed potential opportunities because people couldn't easily learn about your background and expertise?

I was facing this exact problem. Despite having a solid professional background, my static website wasn't engaging visitors or converting them into meaningful connections. That's when I decided to build something revolutionary: an AI chatbot that actually represents me professionally.

The Solution: "The AI You" - Your Digital Professional Twin

I created an intelligent chatbot that:

Knows your professional background inside and out
Engages visitors in natural conversations about your expertise
Automatically captures leads when people show interest
Notifies you instantly when someone wants to connect
Runs 24/7 without requiring manual intervention

Think of it as having a digital version of yourself that never sleeps, always knows your resume, and can have meaningful conversations with potential clients, employers, or collaborators.

What Makes This Different (And Why It Works)

AI That Actually Understands Context

Unlike generic chatbots, this system is trained specifically on your professional summary. It doesn't just give scripted responses - it understands your unique background and can discuss your experience authentically.

Instant Notifications

The moment someone expresses interest or leaves their contact information, you get notified via Pushover (works on any device). No more checking your website daily for leads.

Lead Generation Built-In

The chatbot is designed to naturally guide conversations toward getting in touch, making it a powerful tool for networking and business development.

Cost-Effective

Using GPT-4o-mini, the operating costs are minimal - typically under $5/month even with heavy usage.

Technical Deep Dive: How It Actually Works

Core Architecture

The system is built on three main components:

Streamlit Frontend for a clean, professional chat interface
OpenAI GPT-4o-mini as the AI brain that understands your context
Custom Tools that handle lead capture and notifications

The Magic Behind the Scenes

The system prompt that makes the AI "be" you:

def generate_system_prompt():
    system_prompt = f"""You are acting as {name}. You are answering questions on {name}'s website,
    particularly questions related to {name}'s career, background, skills and experience.
    Your responsibility is to represent {name} for interactions on the website as faithfully as possible."""

    system_prompt += f"\n\n## Summary:\n{summary}\n\n##"
    system_prompt += f"With this context, please chat with the user, always staying in character as {name}."

    return system_prompt

Smart Lead Capture:

The chatbot includes intelligent tools that:

Record user details when someone wants to connect
Track unanswered questions to help you improve your professional summary
Send instant notifications so you never miss an opportunity

def record_user_details(email, name="not provided", notes="not_provided"):
    push(f"{name} with email {email} tried reaching out and these are the notes {notes}")
    return {"recorded": "ok"}

Step-by-Step: Build Your Own AI Professional Assistant

Prerequisites

You'll need:

Basic Python knowledge
An OpenAI API key
A Pushover account for notifications
Your professional summary in PDF format

1. Set Up the Environment

pip install streamlit openai PyPDF2 requests

2. Configure Your Professional Context

Place your professional summary in summary.pdf. This should include:

Your career background
Key skills and technologies
Notable projects and achievements
Areas of expertise

3. Set Up API Keys

Create a .streamlit/secrets.toml file:

OPENAI_API_KEY = "your_openai_api_key"
PUSHOVER_USER = "your_pushover_user_key"
PUSHOVER_TOKEN = "your_pushover_app_token"

4. Customize for Yourself

Update the name variable in chatbot.py:

name = "Your Name"  # Replace with your actual name

5. Deploy

For local testing:

streamlit run chatbot.py

For production, deploy to Streamlit Cloud or your preferred hosting platform.

Ready to Build Your AI Professional Twin?

The code is open-source and ready to customise. Whether you're a developer looking to showcase your skills, a consultant wanting to generate more leads, or a job seeker wanting to stand out, this AI assistant can transform how people interact with your professional brand.

Have questions about the implementation? Drop them in the comments below, and I'll help you get your AI professional assistant up and running.

Repo Link: https://github.com/k-Rohit/Agentic-AI

Live Demo: https://theaiagentme.streamlit.app/