DEV Community: Dmitrii

Anthropic Skills. The Landscape for New Models and Architecture

Dmitrii — Mon, 15 Dec 2025 08:50:11 +0000

Skills are modular, on-demand data that transform general-purpose LLMs into specialized agents. It's not about MCP or fancy protocols - it's about context engineering: loading the right information at the right time.

A skill is a memory, instruction, fact, or code snippet loaded on-demand into your LLM's context window.

A skill is also RAG + System Instructions + Domain Expertise.

Everything you can achieve with skills, you can technically achieve without them. Just load 50k tokens of tools, instructions, and examples into context, use the biggest reasoning model with enough thinking time, and you'll get decent results.

But the new skills-based approach is: simpler, faster, cheaper, and so scalable.

It's Not About MCP

I've been writing code for 14 years. I'm a tech geek following most AI updates, and a vibe-coder. Almost a year ago I shared my thoughts about MCP from a developer's perspective.

My core argument: MCP as an approach and programming pattern may not be the best solution.

Here's my personal experience so far.

I don't use MCP servers - because I don't need to, because they don't work well for my use cases
Coding agents already have enough - terminal commands, file system, web search handle most scenarios
When agents must call external tools (e.g., creating a Google Calendar event), I need more robust custom code, not an MCP wrapper
My debugging workflow - manually add 3-4 files + 1-2 documentation links. Works better than any automated context retrieval

Those 3-4 files and 1-2 links could be found automatically. That's what skills promise to deliver: to make agents smarter.

Context Engineering

I feel like architectural shift is coming:

Before: Big generalized pretrained model OR fine-tuned model
After: Small reasoning model with new architecture to learn + Skills for task-specific problems.

"Context engineering is the delicate art and science of filling the context window with just the right information for the next step." — Andrej Karpathy

The key insight: it's not about having the biggest model - it's about having the right context at the right moment.

The Path Forward

It's still R&D - a branch from mainstream LLM development, not a replacement. Anthropic has been working on this for 6+ months, and we're now just early adopters - discovering a better direction before the mainstream. My prediction, is that in 6 months, everyone will rush into skills-based agent architectures.

What we need:

Automatic skill discovery
Composable skill libraries - combine skills for complex multi-step workflows like n8n
Domain-specific skill packs - pre-built expertise for common developer tasks, e.g. Angular skills, or Github Runner skills

Conclusion

Skills represent this philosophy: modular, reusable, on-demand expertise that transforms any LLM into a specialized agent for your specific workflow. I believe this is the right path, and I'm planning closely follow its progress and start coding some cool new things for skills or using skills.

Building Reliable Pricing for AI Chatbots

Dmitrii — Thu, 23 Oct 2025 09:15:47 +0000

🚀 New Open-Source Project: We're building QuotyAI from the ground up with our backend engine and API open-sourced at QuotyAI/QuotyAI-Engine.

QuotyAI helps businesses create reliable pricing systems for chatbots and apps. We use AI to turn natural language business rules into working code that always gives consistent results.

🤖 The Big Problems We Solve

Building reliable pricing systems for chatbots and apps shouldn't be this hard. Yet most businesses struggle with the same frustrating issues.

Customers often get different quotes for identical requests, eroding trust and creating confusion. Manual coding of complex pricing rules takes weeks of developer time, and even then, subtle bugs can slip through. Testing becomes an endless cycle of trying to catch every possible scenario, while standard AI solutions deliver inconsistent results that are too slow for real-time customer interactions.

QuotyAI changes all of this by combining the best of AI automation with rock-solid reliability.

✨ What Makes QuotyAI Special

Instead of wrestling with code for months, you simply describe your pricing rules in plain English. Tell us "General cleaning costs $100 for 3 hours, deep cleaning is $1.50 per square meter," and our AI generates production-ready code automatically.

Every calculation follows your exact business rules with perfect consistency - no more "why did they get a different price?" questions. We automatically test millions of scenarios to catch problems before they affect customers, and our system delivers instant results that work reliably at scale.

Plus, QuotyAI integrates seamlessly with your existing tools - whether it's chatbots, CRMs, booking systems, or automation platforms like Zapier and n8n.

🚀 Key Features

QuotyAI combines AI-powered code generation with rock-solid reliability. You describe your pricing rules in plain English, and our system automatically creates production-ready code that handles complex calculations instantly.

The platform includes comprehensive testing that validates thousands of pricing scenarios automatically, ensuring accuracy before anything goes live. For businesses with multiple locations or brands, we provide secure multi-tenant support that keeps data completely isolated between companies.

Integration is seamless with RESTful APIs, full audit trails for compliance, and built-in testing environments. You can even upload pricing tables from images, and connect with popular automation tools like Zapier and n8n.

💡 Perfect For

Whether you're running a service business with complex pricing tiers, building an e-commerce platform that needs dynamic pricing, or developing chatbots that must provide reliable quotes - QuotyAI adapts to your needs.

Enterprises get the audit trails and compliance features they require, while startups benefit from fast setup without extensive coding. Anyone who needs consistent, automated pricing will find QuotyAI makes their life much easier.

🔮 Future Integrations

We're building connections to make QuotyAI work everywhere:

🤖 Chatbots: Dialogflow, Microsoft Bot Framework, Amazon Lex
🌐 Customer Service: Chatwoot, Intercom, Zendesk
⚡ Automation: Zapier, n8n, Microsoft Power Automate
📱 Apps: Mobile SDKs and API libraries

Our backend engine and API are open-sourced at QuotyAI/QuotyAI-Engine. Check out the full project at WitcherD/QuotyAI and our documentation to get started.

Choosing the Right AI Model for Stock Prediction

Dmitrii — Sun, 05 Oct 2025 04:28:26 +0000

Hey everyone! Following up on my previous post about building StocketAI, I wanted to dive deeper into how I'm picking AI models for stock prediction.

This research journey began with Google Deep Research, which provided the initial analysis and comparison of different AI models for stock prediction. This comprehensive AI-powered research served as my starting point for understanding the landscape of available models and their capabilities.

I'm not a finance expert or a machine learning expert. I'm a solution architect who's learning as I go, relying heavily on AI tools and research to figure this out. So let me break down what I've learned in simple terms.

Important Disclaimer ⚠️

This analysis represents my current understanding based on the research and experimentation I've done so far. I might be wrong, and I fully expect to change my approach as I learn more, experiment with real data, and discover new techniques. Consider this a snapshot of an ongoing journey rather than definitive conclusions.

The Big Challenge: Markets Keep Changing

Predicting stock prices 6 months ahead is really hard because markets are constantly changing. What worked last year might not work this year. The fancy term for this is "concept drift" - basically, the rules of the game keep changing.

Most AI models assume the future will look like the past, but that's not how stock markets work. Economic conditions change, new trends emerge, and what drove stock prices before might not matter anymore.

After researching different AI models in Qlib (a quantitative finance platform), here's what I learned:

The Winner: A Hybrid Approach

The best approach seems to be combining two things:

DDG-DA (Data Distribution Generation for Predictable Concept Drift Adaptation) - A meta-learning technique that helps models adapt to changing market conditions by predicting future data distribution changes¹
TFT (Temporal Fusion Transformer) - A state-of-the-art attention-based model for multi-horizon forecasting²

Think of DDG-DA as a "market change detector" and TFT as a "pattern finder." Together, they create a system that can handle the messy, ever-changing world of stock markets.

Why This Combo Works (In Simple Terms)

DDG-DA: The Market Change Detector

DDG-DA helps by:

Predicting how market conditions might change in the future using meta-learning³
Adjusting the training data so the model learns from "future-like" scenarios through data distribution generation
Basically preparing the model for surprises before they happen by proactively adapting to concept drift

It's like having a weather forecaster who not only tells you today's weather but also predicts how the climate might be changing over the next few months.

TFT: The Pattern Finder

TFT is great because:

It can look at long time periods (like 6 months of data) and find meaningful patterns using attention mechanisms⁴
It considers different types of information (like company basics, market trends, and economic indicators) through multi-modal input processing
It doesn't just look at stock prices - it tries to understand the bigger picture using temporal fusion of different data sources

Imagine trying to predict someone's behavior not just by looking at their recent actions, but by understanding their personality, their environment, and the broader context of their life.

Other Models I Considered

I also looked at other options available in Qlib like:

HIST (Heterogeneous Information Stock Transformer) - Uses concept stocks and relationship mining to find connections between different stocks and market sectors⁵
ADARNN (Adaptive RNN) - Another model that adapts to changing conditions using transfer learning, but more reactive than proactive⁶
Sandwich - A CNN-KRNN architecture designed for stock prediction⁷

The DDG-DA + TFT combo is the most reliable for long-term predictions.

What This Means for StocketAI

For my VN30 stock prediction project, this means:

I'll use the hybrid approach - DDG-DA to handle market changes + TFT for the actual predictions
Focus on 6-month predictions - This approach works best for longer time horizons
Keep it practical - I want models that work well in the real world, not just in theory

Building Confidence Through Multiple Models

What makes this even better is the idea of training multiple models and using meta-analysis to validate predictions. Instead of relying on just one model, we can train several different approaches and compare their results. The real confidence comes when multiple models - whether it's DDG-DA combined with TFT, or other approaches like HIST and ADARNN - all point to similar predictions. Only when we see that level of agreement across different modeling techniques do we really trust the forecast. This approach helps filter out the noise and gives us more reliable insights for making investment decisions.

Next Steps in My Journey

I'm currently setting up experiments to test this hybrid approach with real VN30 data. I'll be using AI tools to help me configure everything properly and understand what the results mean.

The goal is still the same: help regular people like me make better investment decisions without needing a finance degree or years of trading experience.

Come Join the Fun! 🎉

What do you think? Have you tried predicting stock prices with AI? What's been your experience? I'd love to hear from other non-experts who are figuring this out as they go!

Check out the StocketAI project on GitHub if you want to follow along with my experiments.

References

Built with ❤️ using AI assistance and some of coffee ☕

Original DDG-DA paper: "DDG-DA: Data Distribution Generation for Predictable Concept Drift Adaptation" (https://arxiv.org/abs/2201.04038) ↩
Original TFT paper: "Temporal Fusion Transformers for Interpretable Multi-horizon Time Series Forecasting" (https://arxiv.org/abs/1912.09363) ↩
Implemented in Qlib as a meta-model that works in four steps: (1) Calculate meta-information, (2) Train DDG-DA, (3) Inference to get guide information, (4) Apply guidance to forecasting models⁸ ↩
Implemented in Qlib as a benchmark model with full TensorFlow implementation supporting multi-horizon forecasting and quantile regression⁹ ↩
HIST model in Qlib uses concept stocks to capture market sector relationships and improve prediction accuracy ↩
ADARNN model in Qlib uses domain adaptation techniques to handle changing market conditions reactively ↩
Sandwich model in Qlib combines CNN and KRNN (Kernel Recurrent Neural Network) for spatiotemporal feature extraction ↩
Qlib Meta Controller Documentation: https://github.com/microsoft/qlib/blob/main/docs/component/meta.rst ↩
Qlib TFT Benchmark Documentation: https://github.com/microsoft/qlib/tree/main/examples/benchmarks/TFT ↩

I'm Building an AI to Predict Stocks!

Dmitrii — Fri, 03 Oct 2025 06:03:01 +0000

Just Starting Out 💡

Hey everyone! I'm a solution architect with no finance background, but I had this idea: what if I could build an AI that helps answer Should I buy this stock or sell it?

It's called StocketAI. I have little experience with Python and finance, and rely on AI and coding agents while building models for VN30 companies (Vietnam's top 30). The goal is to predict stock price movements in 1, 3, or 6 months, with low risk.

Check out the project on GitHub

I want to help regular people make better investment choices without being finance experts. Built with Python 3.12+ and qlib for quantitative finance modeling.

What Makes It Different ⚡

Most stock prediction tools are complicated. StocketAI is different - I'm not inventing new algorithms, but building a complete product by integrating existing powerful tools for data sources, technical analysis, and AI models.

Technically speaking, StocketAI uses a modular pipeline architecture with:

Multi-provider data acquisition from VCI, TCBS, MSN, and FMARKET with automatic fallback
Extensible provider interface for adding new data sources beyond vnstock
Qlib format conversion for processing and feature engineering
Advanced AI models including LightGBM, XGBoost, LSTM, GRU, and Transformer-based architectures

It's designed for regular developers like me, focusing on human-AI collaboration. It analyzes longer time periods (months, not days) and builds individual models for each company rather than relying on generic market approaches.

What You Actually Get

We'll be utilizing a lot of data to build good models: financial reports, trading data, news, company information, and market analysis from multiple sources to create comprehensive predictions.

You get to spot investment opportunities that show up over months of data instead of just reacting to today's market noise, plus you learn when to trust the predictions versus when market conditions make it hard to forecast accurately.

It also helps you learn faster by letting you try different AI models and see how they perform in various market situations. And it helps you make smarter choices by showing you how uncertain predictions are, so you don't get overconfident when markets are crazy.

What's Coming Next

StocketAI isn't just a tool - it's a growing project that will keep getting better! Here's what I'm excited to add:

Smarter AI Models:

Even better prediction accuracy with advanced machine learning techniques
Models that adapt and learn from new market conditions automatically
Easy-to-understand explanations of why the AI makes certain predictions

Research Tools:

Track and compare different model experiments to see what works best
Tools to understand which data points matter most for each company
Statistical testing to validate prediction quality

The goal is to make stock investing more accessible, understandable, and successful for regular people like us!

Come Join the Fun! 🎉

Check out the project on GitHub: github.com/WitcherD/StocketAI

Vibe code your next CV, for free

Dmitrii — Wed, 11 Jun 2025 16:03:28 +0000

Every once in a while, you need a CV - clean, well-formatted, easy to read, and professional-looking. But:

LinkedIn’s AI suggestions are still pretty weak.
LinkedIn's PDF export... looks a bit outdated.
Most online resume builders are behind a paywall and you don’t want to pay for a one-time thing.

Here's a free, fast workaround:

🛠 Step 1: For Everyone

Export your LinkedIn profile to PDF. (Go to your profile → More → Save to PDF)
Upload that PDF to Gemini (or ChatGPT).
Optional: Attach an image of a CV you liked from the internet. This gives the AI a visual reference for layout and style.

Prompt it like this:

Convert this PDF into a modern, professional CV.
- Generate clean HTML + CSS, suitable for printing as PDF
- Use a modern, clean and responsive layout and good typography and good visual hierarchy
- Highlight skills, roles, and achievements
- Proofread and improve the language
- Make it ATS-friendly and human-readable

Save the resulting HTML file, open it in your browser, and Print to PDF.

That’s it. You get a polished resume, fast — no signups, no watermarks, no recurring fees.

🖨 Step 2: Export as a PDF (Advanced, Browser-Agnostic)

If your browser's "Print to PDF" doesn’t render the layout properly, use Puppeteer - a Node.js library that gives you full control over headless Chrome.

Installation:

npm install puppeteer

Script: `export-cv.js`

const puppeteer = require('puppeteer');
const path = require('path');

(async () => {
    const browser = await puppeteer.launch({ headless: true });
    const page = await browser.newPage();

    const htmlFilePath = path.resolve(__dirname, 'cv.html');
    await page.goto(`file://${htmlFilePath}`, { waitUntil: 'networkidle0' });

    await page.pdf({
        path: 'cv.pdf',
        format: 'A4',
        printBackground: false,
        margin: {
            top: '8mm',
            right: '8mm',
            bottom: '8mm',
            left: '8mm'
        }
    });

    await browser.close();
    console.log('cv.pdf generated successfully!');
})();

Then run:

node export-cv.js

Distributed C# AI Framework for Enterprise: Orleans (Part 1)

Dmitrii — Sun, 11 May 2025 10:32:33 +0000

I'm a .NET solution architect, AI enthusiast, and... yes, a vibe coder.

It feels like just 10 years ago the big topic was breaking monoliths into microservices. Now, it's all about multi-agent frameworks, and .NET is definitely far behind Python and TS here.

The Good

.NET is a strong choice for enterprise development, enabling the creation of big, scalable, fault-tolerant, and distributed real-time applications.

The Bad

The most popular in .NET Semantic Kernel focuses on enabling AI capabilities in new applications, specifically designed for building AI-first experiences rather than serving as the foundation for traditional enterprise systems like ERP or broker platforms.

The Ugly

I want something similar to LangGraph: a combination of traditional bytecode infrastructure and AI integrations.

By "traditional bytecode," I mean the foundational elements like routing, messaging, recovery, observability, concurrency, isolation, ACID, timers and scheduling, state management, and all the workflow logic required around an application.

In our case, an Actor is an AI agent encapsulated within this bytecode layer. It's distributed and runs across multiple processes on a network.

Tech Stack

Why Microsoft Orleans

Microsoft Orleans is a cloud-native framework based on the virtual actor model. In Orleans, each actor (called a grain) is identified by a stable key and is always "virtually" available. Grains are activated on-demand and automatically garbage-collected when idle. This means developers write code as if all actors are in-memory, while the Orleans runtime transparently handles activation, placement, and recovery.

Grains encapsulate their own state and behavior, enabling intuitive modeling of business entities (customers, accounts, orders, etc.) as long-lived objects.

Orleans was designed for massive scale. By default grains automatically partition application state and logic, letting the system scale out simply by adding silos (server hosts).

Simply saying "refund agent for user 12345" in support chat is our grain, and we can have millions of them, with no engineering overhead.

public class RefundGrain: Grain, IRefundGrain
{
    public async Task Refund(decimal amount, string currency)
    {
        // The state is loaded; all you need to do is call an LLM.
    }
}

// no db calls, no api calls, no routes, simply like that.
var refundGrain = client.GetGrain<IRefundGrain>(12345);
await refundGrain.Refund(100, "USD");

Microsoft.Extensions.Ai

Microsoft.Extensions.AI libraries provide with a unified and consistent way to integrate and interact with various generative AI services, offering core abstractions like IChatClient and IEmbeddingGenerator to simplify the process, promote portability, and enable the easy addition of features such as telemetry and caching through dependency injection and middleware patterns.

Similar to LangChain, abstracts away OpenAI or Ollama from implementation details.

var response = await _chatClient.GetResponseAsync<ResponseModelType>(prompt);

Show me the code!

Full source code on Github

The primary goal of this application is to empower sales development representatives (SDRs) to reach out to prospects at the most opportune moment with highly relevant and personalized messaging, increasing the chances of engagement and conversion. Instead of cold outreach, it enables "warm" outreach based on real-time triggers.

Running the cluster is super simple.

.UseOrleans(siloBuilder =>
{
    siloBuilder
        .UseLocalhostClustering()
        .AddMemoryGrainStorageAsDefault()
        .AddActivityPropagation();
})

Then, moving to k8s is also quite straightforward

What I love about .NET is that it's perfect for day-two operations. The debugging, troubleshooting, and observability tools are seamlessly integrated for large enterprise products. Traces, metrics, logs to different sinks, everything is just couple of lines of configuration!

Github repo to see the code

Next Steps

I hope you now have an idea of what Orleans is and how it can be useful for building distributed applications.

Next time, we'll dive deep into AI implementation and explore using Orleans to build a chat application where each grain manages its own memory.

Hiring Best AI Talents: Interview Questions in 2025

Dmitrii — Fri, 27 Dec 2024 11:41:08 +0000

The first part of the article focuses on the characteristics and personality traits of developers (soft skills).
The second part covers topics to discuss during an interview.

Disclaimer

This article is focused on providing practical questions for companies aiming to integrate AI into their traditional products and businesses with minimal effort and high-quality outcomes. Examples include using AI chatbots for retail, analyzing patient data in healthcare, and delivering personalized experiences in educational platforms. It is not intended for research-oriented firms (e.g., Mistral, Anthropic, ElevenLabs) or enterprises (e.g., Google, Amazon, Microsoft).

The Value of Soft Skills

A trustworthy developer who is eager to learn and experiment can adapt more effectively to the rapidly evolving AI landscape. The ability to learn and pivot quickly is more valuable than proficiency in a specific framework or programming language.

If you know such a developer, you're in luck — they can solve problems without relying on guides like this.

The Changing AI Landscape: From NLP to LLMs and Multimodal models.

Previously, solving specific NLP problems required specialized tools and deep expertise in areas such as: Sentiment Analysis, Spam Detection, Topic Classification, Named Entity Recognition, Text Summarization, Translation, Duplicate Detection, Recommendation Systems, Intent Detection, Grammar Correction, Audio/Image Recognition.

Today, with the advent of large language models (LLMs) and multimodal models, many of these tasks can be addressed more efficiently and comprehensively. The focus has shifted from building custom models/pipelines to applying pre-trained models to real-world use cases.

Avoid Overengineering

When building your first product or MVP, simplicity should be the priority. If you're unsure where to start, begin with straightforward solutions.

For instance, if you're building an MVP with limited usage—say, 10 requests per day for customer support — and someone suggests training a BERT or ModernBERT model, hosting it locally, and managing the entire setup, that's likely overengineering. You'll invest significantly more time, but without the scale to handle 1,000 RPS or a dedicated tech team to maintain the system, it’s not a practical approach.

Getting started with conversational chatbots no longer requires an in-depth understanding of NLP concepts like encoder-only vs. decoder models. An analogy: You don't need to write assembly code to develop most business applications. Instead, nowadays you can rely on high-level languages, frameworks, or even no-code/low-code solutions to efficiently solve business problems.

Topics to Discuss During an Interview

From the business problem to implementation.
Each topic also includes links to specialized resources for deeper exploration

Dataset Management

Working with data is often the most challenging aspect of AI development. It's critical for fine-tuning and evaluations. Key topics include:

Preprocessing: Cleaning and preparing raw data for training. This includes transformations, combining data from multiple sources, and reducing dimensionality while preserving relevant information. Developers should also understand concepts like training, validation, and testing sets:
- Training set: Your textbook and practice problems (you actively learn from these).
- Validation set: Practice tests that are different from the practice problems (you use these to gauge your understanding and identify areas needing improvement, adjusting your study methods accordingly).
- Testing set: The actual exam (this is the final, unseen evaluation of your knowledge).
Labeling and Annotation: Labeling entities (e.g., person, organization, location) in text, annotating sentiment (positive, negative, neutral) in reviews, and other critical tasks for fine-tuning models.
Tooling: Tools for exploration, visualization, and annotation.
Data Pipelines and Workflows: Automation, data sources, responsibilities in the team.
Security and Privacy: Encryption, anonymization, pseudonymization, and access control to ensure data security and compliance with regulations.
Versioning

AI Agents Architecture

Consider the level of control and custom logic required for your application: Do you want the LLM to make decisions autonomously, or will it work in tandem with traditional bytecode programming logic? Then choose a framework or approach based on your specific requirements:

Low-code agent builders (e.g., n8n, Langflow)
Multi-agent systems (e.g., CrewAI, Autogen).
Multi-actor systems (e.g., LangGraph).
Custom architectures tailored to specific use cases (e.g. Semantic Kernel).

Streaming provides a more dynamic and responsive user experience but requires careful implementation to ensure that all system components support streaming capabilities. Messaging, on the other hand, is easier to debug and troubleshoot but may lead to a less seamless user experience compared to streaming.

Memory and State Management

Effective chatbot memory management involves balancing precision and recall with considerations of accuracy, latency, and cost. The principle is simple: "Garbage In, Garbage Out." The ultimate goal is to equip the agent with exactly what is needed—no more, no less.

Short-Term Memory (Conversation Thread)

Message Buffering: Retain the last N messages or a specific time window to maintain conversational context.
Summarization: Condense previous interactions to preserve relevance without overwhelming the system.
Session Timeout: Define when a session should expire (e.g., after 30 minutes of inactivity).
Tools History: Determine whether interactions with external tools should be included in the conversation history.
State Passing: Ensure seamless state transfer between agents or modules.
Entity Storage: Capture and update entities, facts, and IDs relevant to the conversation.
Update Timing: Decide when and how memory updates should occur.

Long-Term Memory

Diverse Storage: Leverage various storage solutions to retrieve relevant information efficiently.
Update Mechanisms: Implement robust processes for updating long-term memory with new data.
Few-Shot Prompting: Use stored conversations as context for dynamic prompting.
Data Masking: Ensure sensitive information is appropriately masked or anonymized.
Context-Dependent Instructions: Tailor memory behavior to the specific use case or scenario.

Storage Solutions

SQL Databases: Best suited for structured data and simple, predefined queries.
Vector Databases: Optimal for storing embeddings and performing similarity searches.
Document Databases: Ideal for unstructured data, such as conversation history, and flexible schemas.
Graph Databases: Perfect for representing and querying intricate relationships within data.

RAG, Embeddings, and Vector Stores

Deep technical topic to discuss.

Embedding Dimensions: Balance between detail capture and computational efficiency.
Sparse vs. Dense: Sparse embeddings for discrete features; dense embeddings for semantic relationships.
Model Selection: Pre-trained models (e.g., BERT, GPT) for general tasks; fine-tuned models for specialized domains.
Language Support: Coverage of all target languages; additional training data for less common ones.
Input Type: Embedding of text, non-text data, or both based on chatbot needs.
Vector Stores: Scalable, efficient databases with metadata integration for enhanced retrieval.
Data Retrieval: Use of varied agents, approaches, and workflows for effective data access.
Reranking and Filtering: Reranking, filtering, and scoping techniques to refine results and improve relevance.
Ingest Workflows: Seamless data ingestion and transformation for embedding and storage preparation.
Quality Assurance: Regular fine-tuning of retrieval processes to maintain accuracy.
Large Datasets: Document chunking and relevance ranking for extensive data handling.
Embedding Updates: Periodic refreshing of embeddings to ensure relevance.

Integrations

Yes, we aim to replace bytecode with tokens in most cases. However, integration with external systems remains one of the most time-consuming tasks, so it needs to be discussed as well.

APIs: REST, gRPC, GraphQL for standardized input/output interactions with AI models.
Webhooks: Real-time, event-driven communication between systems.
Two-Way Integrations: AI sending and retrieving relevant data in real time (e.g., chatbots accessing CRM data).
Data Synchronization: Consistent, up-to-date data through queues or pipelines.
Retries and Fallbacks: Failure management with retry mechanisms and default responses.
Error Handling: Input validation, error logging, and debugging alerts.
Performance Optimization: Batched API calls and caching for reduced latency.
Low-Code Platforms: like Zapier, Make for workflow automation.
Data Integration Tools: Airbyte, Apache Kafka for streaming and event processing.
Authentication: Api keys, OAuth2, 2-factor.
Encryption: Vaults, protocols, keys.

Models

Before diving into prompt engineering, it's crucial to choose the right model for your needs:

Open Source: Is the model proprietary or open source?
Modality: Supported input/output types (e.g., text, images, audio).
Batching: Cost-efficient processing in the background (e.g., hours).
Caching: Support for response reuse and optimization.
Fine-Tuning: Support and ease.
Cost: Pricing structure and affordability.
Context Length: Maximum tokens supported per input/output.
SDK Frameworks: Availability of developer tools and APIs.
Ecosystem: Compatibility with libraries, plugins, or platforms.
Scaling & Throughput: Limits and quota.
Latency: Average response time (ms, sec, or min: o1 vs gemini flash).
Built-in Tools: Features like reasoning, code interpretation, or search.

Every week, a new model emerges that surpasses all previous ones. So, just open Twitter (x.com) and follow OpenAI, Gemini, Anthropic, Mistral, Hugging Face, LLaMA, DeepSeek, Qwen, Gemma, and Phi.

Prompts

It all comes down to personal experience and applying tips. LLMs are non-deterministic, meaning that similar prompts can produce different results.

Prioritize Longform Data: Place detailed context at the start, instructions and examples at the bottom.
Prompt Chaining: Break tasks into steps (e.g., Extract → Transform → Analyze → Visualize).
- Accuracy: Each step gets full attention.
- Clarity: Simple tasks = clear outputs.
- Traceability: Spot and fix issues easily.
Chain of Thought: Encourage step-by-step reasoning.
Multishot Prompting: Provide multiple examples for better learning.
Adopt a Persona: Specify the model’s role for focused responses.
Use Delimiters: Separate distinct input parts clearly.
Prompt Caching: Reuse prompts for efficiency.
Structured Outputs: Request organized formats like JSON or tables.
Directional Cues: Add hints or keywords or formatting like JSON to focus LLM on required problem.
ReAct Approach: Combine reasoning and action in problem-solving.

Security / PII

Basic hygiene to protect customer's data:

Minimizing PII by removing or masking sensitive data.
Trying pseudonymization techniques.
Monitoring access and usage logs for unauthorized activity.
Implementing real-time security breach alerts.

LLM Evaluations

Huge topic to discuss.

Performing regression testing and testing across different models.
AI-Judge.
Evaluating model performance in live environments (e.g., helpfulness) and offline using established gold-standard datasets.
What specific metrics would you use to measure response accuracy in different contexts (e.g., question answering, summarization, dialogue)?
How do you balance competing evaluation objectives (e.g., accuracy vs. fluency, helpfulness vs. harmlessness)?
What are the advantages/disadvantages of different evaluation methods (human evaluation, automated metrics, adversarial testing)?
How would you detect context loss or contradictory statements across turns?
Efficiency and Performance: What specific metrics would you use to measure LLM efficiency, and how would you optimize for them in production? Consider latency, throughput, and memory usage.
Hallucination Detection: What specific techniques/tools would you use to detect hallucinations in LLMs? How would you distinguish between factual errors, creative interpretations, and genuine hallucinations?
Human Evaluation: What specific criteria would guide human evaluators assessing LLM output quality? How would you ensure inter-rater reliability and minimize subjective bias?

Observability

The first step is recognizing that an issue or hallucination exists. Then, you need to find the root cause, troubleshoot, and ensure it's resolved.

Your application/framework should send all necessary information to an observability platform. This includes:

Metrics: Performance and cost data.
Alerting: Automated alerts for performance issues or downtime.
Logs and Traces: To help identify hallucinations and analyze prompt and response variance.

Compare observability platforms, their SDKs, integrations, additional features, and cost.

Final Thoughts

Chatbot development and AI Agent development is not just about math and NLP anymore. Today's best AI developers are:

Versatile Problem-Solvers: They combine business sense with technical skills and learn quickly.
Good Communicators: They work well with others and think critically.
Practical Technologists: They know how to use existing models and tools efficiently.

Use the provided topics to determine whether your candidate is a good fit. Good luck!

AI Agents Architecture, Actors and Microservices: Let's Try LangGraph Command

Dmitrii — Mon, 23 Dec 2024 05:02:29 +0000

In enterprise software development, distributed systems have been essential for the last 15 years. We've embraced SOA, microservices, actor models like Akka (Akka.NET), Microsoft Orleans, Erlang process, and countless message brokers, frameworks, and architectures.

But two years ago, we started fresh.

With AI/LLM models, there were no established frameworks, no observability tools, and no automated testing solutions. We were starting from zero.

Brief Recup

AI Agent is a software entity powered by artificial intelligence designed to perform tasks autonomously or semi-autonomously in a specific environment to achieve particular goals. It processes inputs, makes decisions, and takes actions based on predefined rules, learned patterns, or dynamic interactions.

Actor is finer-grained and lightweight, isolated entity that encapsulates state and behavior, communicates via message passing, and processes one message at a time. Thousands or millions of actors can exist within a single system, often within the same process.

Microservice is independently deployable, highly maintainable, and typically communicates over a network using protocols like HTTP/REST or gRPC. Coarser-grained and heavier compare to actors. Each microservice is typically a standalone application or process.

Actors can be used within microservices to manage internal state and concurrency, combining the strengths of both paradigms. For example, a microservice can implement the actor model for event processing while exposing an API to other microservices.

The Evolving Role of AI Agents

The naming of AI agents depends on context, marketing, and sometimes misunderstandings.

At the product level (e.g. chatbot in your company) AI agent is an actor.
At the company level (e.g. Google Mariner) AI agent is a service.

Over time, the community may establish more precise terminology, such as micro-agent, AI actor, or AI service, to distinguish these concepts.

Aspect	Actor	Service / Microservice
Granularity	Fine-grained	Coarse-grained
State	Internal, encapsulated	External, often stateless
Communication	Messages	APIs over network
Concurrency	Built-in, per actor	Depends on service design
Scaling	Within system or distributed	Horizontal, per service
Fault Tolerance	Supervision hierarchies	External patterns/mechanisms
Use Cases	Real-time, event-driven	Enterprise, modular

If you compare tools like CrewAI Agent, Autogen Agent, or LangChain Agent to this table, you’ll see they function more like actors.

As for an AI service or AI microservice I haven’t fully defined this for myself yet. It might be something we don’t need, or it’s a concept still waiting to be built. I had hopes for Anthropic MCP to fill this role, but it’s not quite there yet. (I wrote more about this here.)

LangGraph: Moving Toward Multi-Actor Applications

Recently, LangGraph introduced commands and redefined itself from a "multi-agent" to a "multi-actor" framework. It now focuses on stateful, multi-actor applications with LLMs for building workflows.

I believe this is absolutely correct. Let's take a look closer and build some example.

Source code on GitHub

import { Annotation, START } from "@langchain/langgraph";
import { ChatOpenAI } from "@langchain/openai";
import { Command } from "@langchain/langgraph";
import { HumanMessage, SystemMessage } from "@langchain/core/messages";
import { StateGraph } from "@langchain/langgraph";
import dotenv from 'dotenv';

dotenv.config();

const StateAnnotation = Annotation.Root({
    customerInquiry: Annotation<string>({
        value: (_prev, newValue) => newValue,
        default: () => "",
    }),
    route: Annotation<string>({
        value: (_prev, newValue) => newValue,
        default: () => "",
    })
});

const model = new ChatOpenAI({
    modelName: "gpt-4o-mini"
});

const routeUserRequest = async (state: typeof StateAnnotation.State) => {
    const response = await model.withStructuredOutput<{ route: "quotation" | "refund" }>({
        schema: {
            type: "object",
            properties: {
                route: { type: "string", enum: ["quotation", "refund"] }
            },
            required: ["route"]
        }
    }).invoke([
        new SystemMessage('Please categorize the user request'),        
        new HumanMessage(state.customerInquiry)
    ]);

    const routeToFunctionName = {
        "quotation": "quotationAgent",
        "refund": "refundAgent"
    };

    return new Command({
        update: {
            route: response.route
        },
        goto: routeToFunctionName[response.route],
    });
};

const quotationAgent = (state) => {
    return {};
};

const refundAgent = (state) => {
    return {};
};

const graph = new StateGraph(StateAnnotation)
    .addNode("routeUserRequest", routeUserRequest, { ends: ["quotationAgent", "refundAgent"] })
    .addNode("quotationAgent", quotationAgent)
    .addNode("refundAgent", refundAgent)
    .addEdge(START, "routeUserRequest")
    .compile();


async function main() {
  try {
    await graph.invoke({ customerInquiry: 'Hi, I need refund' });
    console.log("Done");
  } catch (error) {
    console.error("Error in main function:", error);
  }
}

main();

This approach removes explicit edge declarations, leaving only nodes (actors).

In the future, LangGraph might go beyond its graph-based structure. By adding a message broker, actor addresses, and autodiscovery, it could evolve into something like Microsoft Orleans.

The Future of AI Service Communication

Tools like LangChain/LangGraph are still evolving. Right now, they focus on in-service design and lack of inter-service communication features, but they’re starting to add features for broader integration. For example, LangChain recently added OpenTelemetry support, which is critical for distributed systems.

The next big step for community will be enabling seamless AI-to-AI service communication. Whether it’s through Anthropic MCP, LangChain, or other innovations, this will define the future of AI in distributed systems.

A Practical Guide to Reducing LLM Hallucinations with Sandboxed Code Interpreter

Dmitrii — Sat, 21 Dec 2024 02:06:30 +0000

Most LLMs and SMLs are not designed for calulations (not talking about OpenAI o1 or o3 models). Just imagine the following dialogue:

Company: Today is Wednesday; you can return the delivery parcel within 24 hours.
Client: Okay, let's do it on Tuesday.

Are you sure the next AI response will be correct? As a human, you can understand that next Tuesday is six days ahead, while 24 hours is just one day. However, most LLMs cannot reliably handle such logic. Their responses are non-deterministic.

This issue worsens as the context grows. If you have 30 rules and a conversation history of 30 messages, the AI loses focus and makes mistakes easily.

Common Use-Case

You're developing an AI scheduling chatbot or AI agent for your company.
The company has scheduling rules that are frequently updated.
Before scheduling, the chatbot must validate customer input parameters.
If validation fails, the chatbot must inform the customer.

What Can We Do?

Combine traditional code execution with LLMs. This idea is not new but remains underutilized:

OpenAI integrates this feature into its Assistant API, but not in Complitions API.
Google recently introduced code interpreter capabilities in Gemini 2.0 Flash.

Our Solution Tech Stack

Docker (Podman)
LangGraph.js
Piston

Code Interpreter Sandbox

To securely run generated code, the most popular cloud code interpreters are e2b, Google, and OpenAI as I mentioned before.

However, I was looking for an open-source, self-hosted solution for flexibility and cost-effectiveness. So, 2 good options:

Piston
Jupyter

I chose Piston for its ease of deployment.

Piston Installation

It took me a while to understand how to add python execution environment to Piston.

0. Enable cgroup v2

For Windows WSL, this article was helpful.

1. Run a Container

docker run --privileged -p 2000:2000 -v d:\piston:'/piston' --name piston_api ghcr.io/engineer-man/piston

2. Checkout the Piston Repository

git clone https://github.com/engineer-man/piston

3. Add Python Support

Run the following command:

node cli/index.js ppman install python

By default, this command uses your container API running on localhost:2000 to install Python.

Example Code Execution

Using the Piston Node.js Client:

import piston from "piston-client";

const codeInterpreter = piston({ server: "http://localhost:2000" });

const result = await codeInterpreter.execute('python', 'print("Hello World!")');

console.log(result);

AI Agents Implementation

Source code on GitHub

We're going to use some advanced techniques:

Graph and subgraph architecture
Parallel node execution
Qdrant for storage
Observability via LangSmith
GPT-4o-mini, a cost-efficient LLM

Refer to the LangSmith trace for a detailed overview of the flow:
https://smith.langchain.com/public/b3a64491-b4e1-423d-9802-06fcf79339d2/r

Step 1: Extract datetime-related scheduling parameters from user input

Example: "Tomorrow, last Friday, in 2 hours, at noon time."
We use code interpreter to ensure reliable extraction, as LLMs can fail even with current date-time contextual information.

Example Prompt for Python Code Generation:

Your task is to transform natural language text into Python code that extracts datetime-related scheduling parameters from user input.  

## Instructions:  
- You are allowed to use only the "datetime" and "calendar" libraries.  
- You can define additional private helper methods to improve code readability and modularize validation logic.  
- Do not include any import statements in the output.  
- Assume all input timestamps are provided in the GMT+8 timezone. Adjust calculations accordingly.  
- The output should be a single method definition with the following characteristics:  
  - Method name: \`getCustomerSchedulingParameters\`  
  - Arguments: None  
  - Return: A JSON object with the keys:  
    - \`appointment_date\`: The day of the month (integer or \`None\`).  
    - \`appointment_month\`: The month of the year (integer or \`None\`).  
    - \`appointment_year\`: The year (integer or \`None\`).  
    - \`appointment_time_hour\`: The hour of the day in 24-hour format (integer or \`None\`).  
    - \`appointment_time_minute\`: The minute of the hour (integer or \`None\`).  
    - \`duration_hours\`: The duration of the appointment in hours (float or \`None\`).  
    - \`frequency\`: The recurrence of the appointment. Can be \`"Adhoc"\`, \`"Daily"\`, \`"Weekly"\`, or \`"Monthly"\` (string or \`None\`).  

- If a specific value is not found in the text, return \`None\` for that field.  
- Focus only on extracting values explicitly mentioned in the input text; do not make assumptions.  
- Do not include print statements or logging in the output.  

## Example:  

### Input:  
"I want to book an appointment for next Monday at 2pm for 2.5 hours."  

### Output:  
def getCustomerSchedulingParameters():  
    """Extracts and returns scheduling parameters from user input in GMT+8 timezone.  

    Returns:  
        A JSON object with the required scheduling parameters.  
    """  
    def _get_next_monday():  
        """Helper function to calculate the date of the next Monday."""  
        current_time = datetime.utcnow() + timedelta(hours=8)  # Adjust to GMT+8  
        today = current_time.date()  
        days_until_monday = (7 - today.weekday() + 0) % 7  # Monday is 0  
        return today + timedelta(days=days_until_monday)  

    next_monday = _get_next_monday()  
    return {  
        "appointment_date": next_monday.day,  
        "appointment_month": next_monday.month,  
        "appointment_year": next_monday.year,  
        "appointment_time_hour": 14,  
        "appointment_time_minute": 0,  
        "duration_hours": 2.5,  
        "frequency": "Adhoc"  
    }

### Notes:
Ensure the output is plain Python code without any formatting or additional explanations.

Step 2: Fetch Rules from Storage

And then transform them into Python code for validation.

Step 3: Run Generated Code in Sandbox:

const pythonCodeToInvoke = `
import sys
import datetime
import calendar
import json

${state.pythonValidationMethod}

${state.pythonParametersExtractionMethod}

parameters = getCustomerSchedulingParameters()

valiation_errors = validateCustomerSchedulingParameters(parameters["appointment_year"], parameters["appointment_month"], parameters["appointment_date"], parameters["appointment_time_hour"], parameters["appointment_time_minute"], parameters["duration_hours"], parameters["frequency"])

print(json.dumps({"validation_errors": valiation_errors}))`;

    const traceableCodeInterpreterFunction = await traceable((pythonCodeToInvoke: string) => codeInterpreter.execute('python', pythonCodeToInvoke, { args: [] }));
    const result = await traceableCodeInterpreterFunction(pythonCodeToInvoke);

Source code on GitHub

Potential Improvements

Implement an iterative loop for LLMs to debug and refine Python code execution dynamically.
Human in the loop for validation method code generation.
Caching generated code.

Final Thoughts

Bytecode execution and token-based LLMs are highly complementary technologies, unlocking a new level of flexibility. This synergistic approach has a bright future, for example AWS's recent "Bedrock Automated Reasoning", which appears to offer a similar solution within their enterprise ecosystem. Google and Microsoft also will show us something similar very soon.

Anthropic MCP: Developer's Thoughts

Dmitrii — Fri, 13 Dec 2024 15:08:35 +0000

Anthropic wants access to your computer and data. The more data they can use, the more useful their service becomes. Recently, Anthropic launched computer use and subsequently introduced Model Context Protocol (MCP), which is an effort to create a standardized framework for AI interactions with local and remote resources.

Example of MCP Components

The app can retrieve information from MCP servers (e.g., files or text, called resources) or invoke server functions (tools).

It's very simple. To encourage developers to create servers, Anthropic provided TypeScript and Python SDKs and made everything open-source.

Technical Overview

MCP Server: An HTTP Listener that uses JSON-RPC for requests and Server-Sent Events (SSE) for asynchronous communication (it's WebSockets, allows servers to send events back to the app)
MCP Schema: A JSON schema defining the structure of resources and interactions.
MCP Client: A TypeScript or Python library within the host app that connects to the server, manages server connections, processes queries, and handles responses.

While technically not groundbreaking, MCP marks the beginning of a standardization process. It addresses common technical needs, such as:

🔀 Connection lifecycle management.
⚡ Error handling.
🔬 Logging and monitoring capabilities.
✅ Schema/input validation.
⏳ Timeouts.
📃 Unified message formats.
📚 Testing.
and more.

That's nice, but what's next?

🔍 Google has introduced Project Mariner.
💻 Microsoft has Copilot.
🤖 OpenAI is incorporating tools into ChatGPT, with rumors swirling about their own browser.

But here’s the catch: none of these companies are discussing MCP. If you believe you can simply add MCP to your AI agent and expect seamless integration with all AI products, you’re likely mistaken 😥.

Need or force?

Standards like OpenAPI and OpenTelemetry gained popularity because they addressed pain points shared by millions of developers working with distributed systems and complex infrastructures.

The OpenAI API succeeded because they offered something so big that everyone wanted to use, and not because they open-sourced their schema and SDK.

Another example. Take cloud providers like AWS, GCP, Azure, Alibaba, and Yandex. Even though they offer similar products, their APIs remain non-standardized. If you’re building a product like Pinecone, you must integrate with each cloud individually. Maybe MCP falls into this category—useful only for niche scenarios?

Need something bigger?

As I mentioned earlier, Anthropic is solving their own commercial problem: they want access to your data. But why should I, as an AI developer building agents, care to adopt MCP?

What we really need is an AI-to-AI or AI-Agent-to-AI-Agent communication protocol, not just another wrapper around HTTP using opinionated protocols like JSON-RPC and SSE.

MCP flow

Dream flow

I'm thinking about a future where:

🤝 Claude Desktop uses a built-in code interpreter capable of interacting seamlessly with the external world and asking other agents for help using an internal language of LLMs. In this case, SDKs like MCP become unnecessary.
🔗 AI systems communicate directly with each other without relying on us as intermediaries.
🖼️ Protocols support realtime multimodality (speech, vision, text).
🧠 Context sharing is effortless, including system prompts, conversation history, and active prompts.
🏗️ Hierarchical problem-solving allows tasks to be delegated to knowledge subgraphs.
📊 Features like conversation summarization, compositional function calls, and conditional execution of functions.

It's kind of Globally distributed CrewAI or Autogen

Final thoughts

I believe the engineers at OpenAI and Anthropic are some of the smartest minds in AI, and they surely understand these challenges.

That said, MCP feels overhyped and not a true "game changer"—at least not yet. You might be better off waiting for further breakthroughs in local AI assistants before investing significant effort in adopting and maintaining this solution.

Open Source LLMOps LangSmith Alternatives: LangFuse vs. Lunary.ai

Dmitrii — Wed, 11 Dec 2024 03:55:54 +0000

LangSmith is a powerful LLMOps platform, but its cost and cloud reliance can be drawbacks. Open-source options like LangFuse and Lunary.ai offer open source self-hostable alternatives. This guide compares their features to help you choose the best fit for your needs.

⚠️ Note: This information is accurate as of December 2024. The landscape of LLMOps evolves rapidly, so updates within the next three months are likely. Also I'm focusing on TypeScript and NodeJs integrations and tooling, not Python.

For testing, I’ve integrated these tools into my LangGraph.js demo project, which mirrors common production tasks:

Nested execution flow (subgraphs).
Gemini and OpenAI LLM calls.
Input parameter handling.
Data retrievals from Qdrant.
Tagging for trace organization.
Conditional map-reduce branching.

Let’s explore how these platforms stack up against LangSmith.

TL;DR for the Busy Reader

Observability Only:
- Free and self-hosted: Choose LangFuse.
- Cloud-based: Opt for Lunary, which is more cost-effective.
Full-Feature LLMOps Suite:
- Using LangChain/LangGraph? Stick with LangSmith.
- Exploring other frameworks? Go with LangFuse.
Special Conditions: Non-profits, educational institutions, and open-source projects can negotiate favorable terms with LangFuse and LangSmith.

Traces

Traces are vital for effective LLM observability. All three platforms excel in:

Tree structure for trace visualization.
Metadata tagging for better trace organization.
Role-based conversation history (e.g., assistant, user, system).
Visual clarity with a polished UI.
Duration tracking for operations.
Pricing details for calls.
Error tracking for debugging.
Session-based data organization.
Support for Base64 images.

Platform-Specific Limitations

Image/Attachment Display: Neither LangFuse nor Lunary supports displaying images or attachments via URLs.
Lunary:
- Automatic PII masking requires an Enterprise license, with no manual masking options.
LangFuse:
- Automatic masking is unavailable, but basic manual masking is supported.

Overall, all three products are quite similar in a positive way when it comes to observability features

LangSmith UI

Lunary UI

LangFuse UI

Metadata and Search

All platforms offer robust search capabilities by date, metadata, IDs, status, and duration. No notable differences here.

Monitoring

Monitoring features are strong across all three platforms:

LangSmith: Offers custom charts for deeper insights.
LangFuse & Lunary: Provide built-in dashboards with filtering options.
- Lunary: Advanced monitoring available in the Team Tier.

LangFuse Monitoring Example

Datasets

LangSmith is fully featured.

Key Differences

Lunary: Doesn’t support adding traces to datasets directly.
Exporting:
- LangFuse lacks export options, such as those needed for OpenAI fine-tuning.
- Lunary requires a Team license for exports.

Playground

One of the must-have feature for debugging and improving agent prompts.

Lunary: Offers limited usage in the basic tier.
LangFuse: Requires a $100/user Pro license for self-hosted deployments.

LangFuse Playground Configuration

Lunary Playground Configuration

Hard to compete with Langsmith here.

Deployment

Lunary: Docker/Kubernetes deployment requires an Enterprise license. You have to install and maintain the product on your own. Don't remember when the last time run anything not in containers.
LangFuse: The free self-hosted docker version lacks a playground and LLM-as-judge evaluators.

Prompt Experiments

All platforms perform well in this area, offering robust tools for testing and refining prompts.

Integrations

LangFuse & Lunary: Compatible with LangChain, LangGraph, LlamaIndex, DSPy, and more.
LangSmith: Limited to LangChain and LangGraph.

Evaluators

Evaluators are essential for scoring traces and running tests on datasets.

LangSmith: Full evaluator suite.
LangFuse: Full evaluator suite. Requires a paid license for LLM-as-judge. Not limited with langchain.
Lunary: Lacks built-in evaluators.

LangFuse Evaluation

Both LangSmith and LangFuse have advanced scoring and evaluation features, extremely useful toolset for any LLM application.

Documentation

All 3 platforms provide perfect complete documentation with tons of examples.

Pricing

For a small team of 3 users:

Platform	Self-Hosted	Cloud
Lunary	Free	$20/user/month
LangSmith	N/A	$39/user/month (50% for startups)
LangFuse	Free (Observability), $100/user/month (LLMOps)	$60/user/month (50% for startups)

⚠️ Note: This information is accurate as of December 2024

More platforms

OpenLLMetry sdk + Traceloop. It's a great platform, but I was seaching for self-hosted product.
Phoenix - Arize AI one of the most powerful LLMOps tools with observability and evaluation features. But has poor langchain.js and other TypeScript framework integrations and focused on ML in general, not just LLM, so it's making UI less intuitive when develop chatbots or LLM agents.

Summary: Choosing an LLMOps Platform

Best for Observability:
- LangFuse: Free self-hosted option.
- Lunary: Affordable cloud-based solution.
Best for Full Features:
- LangSmith: Comprehensive LLMOps suite, ideal for LangChain/LangGraph users.
- LangFuse: Good for other frameworks.

Observability with Grafana Cloud and OpenTelemetry in .net microservices

Dmitrii — Thu, 06 Oct 2022 09:07:41 +0000

People say, that application development lifecycle consists of 3 steps

make it work
make it right ← we’re here
make it fast

Suppose you’re developing a microservice. It can be over rest/grpc/kafka or whatever, You’ve completed all functional and non-functional requirements incl. authentication/authorization, validation, your application is secure, scalable and solving a business problem.

In this article, you’ll find out how to make your app production-ready whether it is cloud native and hosted in Kubernetes or more traditional without using containers.

We will cover popular tools and frameworks aiming to solve common needs for every application without reinventing the wheel:

Grafana Cloud (Prometheus, Grafana, Loki, Tempo),
OpenTelemetry,
Serilog

If you haven’t heard about them, no worries, the article applies to any experience level. You can find a fully working demo project on Github.

Observability/Monitoring

Observability means some collected data explaining the state of your application. For production environments it is critical to know how your application behaves. Nginx post simplified it into 3 simple questions:

Metrics – “Is there a problem?”
Traces – “Where is the problem?”
Logs – “What is the problem?”

Plenty of cool tools capable of doing full monitoring of your system can answer all the questions above. The problem we face nowadays isn’t knowing the answer but having too many answers.

This article can help you make the right decision and save tons of time and money for your business. Described approach works best for start-ups and small companies.

From a high-level perspective, there are 2 popular groups:

SaaS: Dynatrace, Logz.io, Datadog, Honeycomb, New Relic
Open source, self-hosted: Prometheus + Grafana and ElasticSearch + Kibana.

Today we’re talking about Grafana Cloud (Prometheus for metrics, Loki for logs, Tempo for traces), a SaaS product. Its free plan includes:

10,000 series for Prometheus or Graphite metrics
50 GB of logs
50 GB of traces

It's very generous compared to competitors!

When exceeding these limits, you can freely choose to continue on SaaS or switch to open-source self-hosted distribution.

If you decide to give it a try, Sign Up here before moving to the next step.

Before we start, these 2 portals shortcut will be helpful:

https://grafana.com/orgs/{YouOrganizationName} - Account Management
https://{YouOrganizationName}.grafana.net - Grafana UI

Grafana Agent

Grafana Agent is responsible for delivering your metrics/traces from your application to the cloud. We use grafana agent for **metrics **and **traces **only. The approach for **logging **will be different.

The data flow is illustrated below:

Grafana Agent

Open https://{YouOrganizationName}.grafana.net

=> go to the integrations tab,
=> choose grafana agent
=> and then follow the instructions.

Grafana Agent can work on Windows/MacOS/Debian/RedHat

Grafana Agent Integration Tab

After installation, we need to configure the agent:

If you’re using Windows with the default installation, go to C:\Program Files\Grafana Agent and edit agent-config.yaml For other cases check the documentation

metrics:
  configs:
  - name: integrations
    remote_write:
    - basic_auth:
        password: {replaceit}
        username: {replaceit}
      url: {replaceit}
    scrape_configs:
      - job_name: dogs-service
        scrape_interval: 30s
        metrics_path: /metrics/prometheus
        static_configs:
          - targets: ['localhost:5000']
      - job_name: dogs-service-healthchecks
        scrape_interval: 30s
        metrics_path: /health/prometheus
        static_configs:
          - targets: ['localhost:5000']
  global:
    scrape_interval: 60s
  wal_directory: /tmp/grafana-agent-wal

traces:
  configs:
  - name: default
    remote_write:
      - endpoint: {replaceit}
        basic_auth:
          username: {replaceit}
          password: {replaceit}
    receivers:
      otlp:
        protocols:
          grpc:

To get the username, password and URLs, go to https://grafana.com/orgs/{YouOrganizationName} and hit ‘Details’ in Tempo and Prometheus.

Organization settings

Save the file and restart Grafana Agent service.

Restarting Grafana Agent

With this, Grafana Agent configuration is complete. Now, let’s send some data to Grafana Cloud.

Traces

To understand the traces it’s easier to take a look at the picture.

Grafana Tempo

In short, you can trace an activity through your services in a distributed system.

To demonstrate it, in our simple demo project we’ll be using 3 components:

.net 6 web api
sqlite database
external dogs api

To send traces to the monitoring system, we need to use some framework. OpenTelemetry is a standardized and recommended approach to implementing tracing in the application nowadays. It gets support from all popular tools so the integration will be seamless.

OpenTelemetry is a collection of tools, APIs, and SDKs. Use it to instrument, generate, collect, and export telemetry data (metrics, logs, and traces) to help you analyze your software’s performance and behavior.

We're going to use OpenTelemetry .NET SDK. Add following nuget dependencies to the project:

<PackageReference Include="OpenTelemetry.Exporter.OpenTelemetryProtocol" Version="1.4.0-alpha.2" />
<PackageReference Include="OpenTelemetry.Exporter.Prometheus.AspNetCore" Version="1.4.0-alpha.2" />
<PackageReference Include="OpenTelemetry.Extensions.Hosting" Version="1.0.0-rc9.6" />
<PackageReference Include="OpenTelemetry.Instrumentation.AspNetCore" Version="1.0.0-rc9.6" />
<PackageReference Include="OpenTelemetry.Instrumentation.EntityFrameworkCore" Version="1.0.0-beta.3" />
<PackageReference Include="OpenTelemetry.Instrumentation.EventCounters" Version="0.1.0-alpha.1" />
<PackageReference Include="OpenTelemetry.Instrumentation.Http" Version="1.0.0-rc9.6" />
<PackageReference Include="OpenTelemetry.Instrumentation.SqlClient" Version="1.0.0-rc9.6" />

Then, in Program.cs configure:

var builder = WebApplication.CreateBuilder(args);
builder.Services.AddOpenTelemetryTracing(options =>
{
    options.ConfigureResource(resourceBuilder =>
    {
        resourceBuilder.AddService(
            builder.Environment.ApplicationName,
            builder.Environment.EnvironmentName,
            builder.Configuration["OpenTelemetry:ApplicationVersion"],
            false,
            Environment.MachineName);
    })
    .AddHttpClientInstrumentation(instrumentationOptions =>
    {
        instrumentationOptions.RecordException = true;
    })
    .AddAspNetCoreInstrumentation(instrumentationOptions =>
    {
        instrumentationOptions.RecordException = true;
    })
    .AddSqlClientInstrumentation(instrumentationOptions =>
    {
        instrumentationOptions.RecordException = true;
        instrumentationOptions.SetDbStatementForText = true;
    })
    .AddEntityFrameworkCoreInstrumentation(instrumentationOptions =>
    {
        instrumentationOptions.SetDbStatementForText = true;
    })
    .AddOtlpExporter(opt =>
    {
        opt.Protocol = OtlpExportProtocol.Grpc;
        opt.Endpoint = new Uri(builder.Configuration["OpenTelemetry:Exporter:Otlp:Endpoint"]);
    });
});

‘OpenTelemetry:Exporter:Otlp:Endpoint’ comes from appsettings.json


  "OpenTelemetry": {
    "ApplicationVersion": "1.0.0", 
    "Exporter": {
      "Otlp": {
        "Endpoint": "http://localhost:4317"
      }
    }
  }

where http://localhost:4317 is an endpoint of the Grafana Agent we installed in the previous step.

By using OTLP protocol, our application will send traces to grafana agent which will take care of the rest. In our case, the agent will resend it to grafana cloud. If required, you can always easily switch from grafana cloud to your self-hosted tempo just by configuring the agent. There's no need to modify the source code,

That’s it.

Let’s run the app and hit the test endpoint.

GET {{host}}/api/v1/dogs/new

Testing traces

The parent span belongs to our API request, and 2 child spans for calling external API and saving data to the database. There’s a correlation between the duration, operation results, and metadata. That is basically everything we need to trace and debug activity from top to bottom.

Let’s hit another endpoint to see what we get in case of error:

GET {{host}}/api/v1/fail500

Going back to search panel, and searching for our request:

Traces Search Panel

Trace Tags

It looks perfect, we have all the needed information here to backtrace the issue: External API returned HTTP 404, Refit threw an exception, and our API returned HTTP 500 to the client.

Metrics

Metrics - aggregated real-time data to measure your application performance.

For example, it can be the latency of your API endpoints, number of http 5XX errors, and the free space on the hard drive.

There are many frameworks to collect metrics in .net core service:

All of them are working just fine, but we’re going to use OpenTelemetry sdk again, and then expose prometheus endpoint. Grafana Agent will fetch it and send to Grafana Cloud, similarly to traces.

var builder = WebApplication.CreateBuilder(args);
builder.Services.AddOpenTelemetryMetrics(options =>
{
    options.ConfigureResource(resourceBuilder =>
    {
        resourceBuilder.AddService(
            builder.Environment.ApplicationName,
            builder.Environment.EnvironmentName,
            builder.Configuration["OpenTelemetry:ApplicationVersion"],
            false,
            Environment.MachineName);
        resourceBuilder.AddTelemetrySdk();
    })
    .AddHttpClientInstrumentation()
    .AddAspNetCoreInstrumentation()
    .AddEventCounterMetrics()
    .AddPrometheusExporter();
});

var app = builder.Build();
app.UseHealthChecksPrometheusExporter("/health/prometheus", options =>
{
    options.ResultStatusCodes[HealthStatus.Unhealthy] = 200;
});

That’s it, very simple. Go to Grafana Cloud, change data source to prometheus and try to visualize some metrics of your choice. E.g. queries per second:

Setting up metrics visualization

Queries per second

You might already be familiar with Prometheus and Grafana metrics. It’s the most loved tool among DevOps experts all around the globe to monitor VMs, networks, databases, and whatever metrics you can imagine.

Healthchecks

Healthchecks are the edge case of app metrics to automate operations (e.g. when your app should automatically restart when it's running out of memory, route traffic, a new instance in your cluster becomes available, and so on).

Microsoft provided extensive documentation for us, explaining everything about health checks in depth.

To keep the article short and simple, I won't add any more detail in this subject. So we just simply try it out.

First step is Installing required packages:

    <PackageReference Include="AspNetCore.HealthChecks.Network" Version="6.0.4" />
    <PackageReference Include="AspNetCore.HealthChecks.Prometheus.Metrics" Version="6.0.2" />
    <PackageReference Include="AspNetCore.HealthChecks.Publisher.Prometheus" Version="6.0.2" />
    <PackageReference Include="AspNetCore.HealthChecks.System" Version="6.0.5" />
    <PackageReference Include="Microsoft.Diagnostics.NETCore.Client" Version="0.2.328102" />
    <PackageReference Include="Microsoft.Extensions.Diagnostics.HealthChecks.EntityFrameworkCore" Version="6.0.8" />

Configure program.cs

var builder = WebApplication.CreateBuilder(args);
builder.Services.AddHealthChecks()
    .AddDiskStorageHealthCheck(_ => { }, tags: new[] { "live", "ready" })
    .AddPingHealthCheck(_ => { }, tags: new[] { "live", "ready" })
    .AddPrivateMemoryHealthCheck(512 * 1024 * 1024, tags: new[] { "live", "ready" })
    .AddDnsResolveHealthCheck(_ => { }, tags: new[] { "live", "ready" })
    .AddDbContextCheck&lt;DogsDbContext>(tags: new[] { "ready" });

var app = builder.Build();
app.MapHealthChecks("health", new HealthCheckOptions
{
    Predicate = check => check.Tags.Contains("ready"),
    ResponseWriter = HealthChecksLogWriters.WriteResponseAsync
});

app.MapHealthChecks("health/ready", new HealthCheckOptions
{
    Predicate = check => check.Tags.Contains("ready"),
    ResponseWriter = HealthChecksLogWriters.WriteResponseAsync
});

app.MapHealthChecks("health/live", new HealthCheckOptions
{
    Predicate = check => check.Tags.Contains("live"),
    ResponseWriter = HealthChecksLogWriters.WriteResponseAsync
});

app.UseHealthChecksPrometheusExporter("/health/prometheus", options =>
{
    options.ResultStatusCodes[HealthStatus.Unhealthy] = 200;
});

In this example, we collect some crucial health check metrics and expose them in 3 different ways:

For kubernetes probes
For prometheus collector
For people

Kubernetes Probes

You can skip it if you’re not using Kubernetes.

Kubernetes pings our application. Depending on the result, it can decide to be up and running, restart ,or maybe wait a bit longer until the app loads all required dependencies. Only then it will be ready to handle incoming traffic.

Detailed documentation you can find here.

We’re exposing 2 endpoints:

/health/live - for **startupProbe and livenessProbe if the app signalized that is not live (some of healthchecks fauld) Kubernetes has to restart the service.
/health/ready - readinessProbe When the app is ready Kubernetes will route the traffic to this instance.

Prometheus healthcheck

Additional endpoint for exporting health checks in prometheus format. And similar to app metrics we can use them in grafana:

Prometheus exporter endpoint

Memory consumption

Human readable format

And the last endpoint just to simplify testing and operations. Just open it in your browser:

Human-readable health checks

Logs

Logging for development and production environments will be different.

On production best practice is to write logs to stdout, then use log collector (e.g. promtail) to deliver it to the storage. And for development purposes, it is much easier to have a sink and write logs directly to Grafana Cloud.

I believe that in 2022, OpenTelemetry logs are not ready for production use. It doesn't give you any advantages, and tooling is quite poor compared to well-known tools.

So I still recommend using Serilog for logging. It has an OpenTelemetry sink, so again, you can switch to OTLP anytime when needed without modifying your source code.

As usual we start with installing packages:

<PackageReference Include="Serilog.AspNetCore" Version="6.0.1" />
<PackageReference Include="Serilog.Enrichers.Demystifier" Version="1.0.2" />
<PackageReference Include="Serilog.Enrichers.Span" Version="2.3.0" />
<PackageReference Include="Serilog.Exceptions" Version="8.4.0" />
<PackageReference Include="Serilog.Exceptions.EntityFrameworkCore" Version="8.4.0" />
<PackageReference Include="Serilog.Exceptions.Refit" Version="8.4.0" />
<PackageReference Include="Serilog.Formatting.Compact" Version="1.1.0" />
<PackageReference Include="Serilog.Sinks.Grafana.Loki" Version="8.0.0" />

Program.cs:

builder.Host.UseSerilog((_, configuration) => configuration
    .ReadFrom.Configuration(builder.Configuration)
    .Enrich.WithSpan()
    .Enrich.WithExceptionDetails(new DestructuringOptionsBuilder()
        .WithDefaultDestructurers()
        .WithDestructurers(new IExceptionDestructurer[]
        {
            new DbUpdateExceptionDestructurer(),
            new ApiExceptionDestructurer()
        }))
    .Enrich.WithDemystifiedStackTraces();

builder.Services.AddHttpLogging(logging =>
{
    logging.LoggingFields = HttpLoggingFields.All;
});

var app = builder.Build();
app.UseHttpLogging();

appsettings.json:

"Serilog": {
    "Using": [
      "Serilog.Sinks.Grafana.Loki"
    ],
    "MinimumLevel": {
      "Default": "Debug"
    },
    "WriteTo": [
      {
        "Name": "Console",
        "Args": {
          "formatter": "Serilog.Formatting.Compact.CompactJsonFormatter, Serilog.Formatting.Compact"
        }
      },
      {
        "Name": "GrafanaLoki",
        "Args": {
          "uri": "https://logs-prod3.grafana.net",
          "credentials": {
            "login": "",
            "password": ""
          },
          "labels": [
            {
              "key": "service",
              "value": "demo-services-dogs"
            }
          ],
          "propertiesAsLabels": [
            "app"
          ]
        }
      }
    ]
  }

Don’t forget to update your grafana credentials in your secrets

In our demo project we’re going to log http requests/responses. Mirosoft provides 2 middleware to log http messages (body, headers etc).

http-logging <- we’re using this one
w3c-logger

Serilog is the most popular logging framework and has tons of extensions, for example:

WithSpan - adding information from OpenTelemetry traces.
WithExceptionDetails - log the exceptions in convenient and human-readable format.

Okay, after running our service go to Grafana Cloud -> Explore -> Loki Logs Datasource

Loki Logs Datasource

Here are our logs. Thanks to deep integration between Loki and Tempo, Grafana allows us to quickly jump from logs to according to traces.

Loki-tempo integration

Conclusion

Source code is on Github

In the article we explored Grafana Cloud and tried 3 observability tools:

Prometheus for metrics
Tempo for traces
Loki for logs

For every data store we use Grafana UI only, which is super convenient for analytics and troubleshooting.

We also got our hands dirty with OpenTelemetry, which aims to standardize observability tools and protocols to make distributed applications maintenance much easier.

In the next article we’ll cover more topics need to be done for production ready apps, such as:

Errors handling;
Retries, Jitter, Circuit Breaker patterns.

DEV Community: Dmitrii

Anthropic Skills. The Landscape for New Models and Architecture

It's Not About MCP

Context Engineering

The Path Forward

Conclusion

Building Reliable Pricing for AI Chatbots

🤖 The Big Problems We Solve

✨ What Makes QuotyAI Special

🚀 Key Features

💡 Perfect For

🔮 Future Integrations

Choosing the Right AI Model for Stock Prediction

The Big Challenge: Markets Keep Changing

The Winner: A Hybrid Approach

Why This Combo Works (In Simple Terms)

DDG-DA: The Market Change Detector

TFT: The Pattern Finder

Other Models I Considered

What This Means for StocketAI

Building Confidence Through Multiple Models

Next Steps in My Journey

Come Join the Fun! 🎉

References

I'm Building an AI to Predict Stocks!

Just Starting Out 💡

What Makes It Different ⚡

What You Actually Get

What's Coming Next

Come Join the Fun! 🎉

Vibe code your next CV, for free

🛠 Step 1: For Everyone

🖨 Step 2: Export as a PDF (Advanced, Browser-Agnostic)

Installation:

Script: export-cv.js

Distributed C# AI Framework for Enterprise: Orleans (Part 1)

The Good

The Bad

The Ugly

Tech Stack

Why Microsoft Orleans

Microsoft.Extensions.Ai

Show me the code!

Next Steps

Hiring Best AI Talents: Interview Questions in 2025

Disclaimer

The Value of Soft Skills

The Changing AI Landscape: From NLP to LLMs and Multimodal models.

Avoid Overengineering

Topics to Discuss During an Interview

Dataset Management

AI Agents Architecture

Memory and State Management

Short-Term Memory (Conversation Thread)

Long-Term Memory

Storage Solutions

RAG, Embeddings, and Vector Stores

Integrations

Models

Prompts

Security / PII

LLM Evaluations

Observability

Final Thoughts

AI Agents Architecture, Actors and Microservices: Let's Try LangGraph Command

Brief Recup

The Evolving Role of AI Agents

LangGraph: Moving Toward Multi-Actor Applications

The Future of AI Service Communication

A Practical Guide to Reducing LLM Hallucinations with Sandboxed Code Interpreter

Common Use-Case

What Can We Do?

Our Solution Tech Stack

Code Interpreter Sandbox

Piston Installation

0. Enable cgroup v2

1. Run a Container

2. Checkout the Piston Repository

3. Add Python Support

Example Code Execution

Script: `export-cv.js`