DEV Community: Madhesh T

What Drives User Success?

Madhesh T — Sun, 15 Mar 2026 10:53:39 +0000

I Built an ML Dashboard on Zerve to Find Out

A complete walkthrough of the idea, the models, the dashboard, and the finding that changed how I think about user retention.

The Problem No One Talks About Honestly

Every SaaS platform has a churn problem. But most of the time, the conversation around churn is reactive — someone cancels, someone stops logging in, a renewal gets missed. By the time the signal arrives, the user is already gone.

The real question is earlier than that. Not why did they leave — but who is about to leave, and what do we do right now?

That was the challenge Zerve put in front of me. They provided a dataset of user event data — every action a user takes on the platform generates an event — and asked a single, open-ended question.

Which user behaviours are most predictive of long-term success?

No predefined answer. No suggested approach. Just the data and the question.

Here is exactly what I built, how I built it, and what the data actually said.

What Zerve Is and Why It Made This Possible

Zerve is a notebook-based data platform with built-in AI capabilities. Users create canvases, write and run code, call AI agents, use generative AI tools, and collaborate with teammates — all inside one workspace.

What makes Zerve genuinely different as a development environment is the variable() function. Any output computed in a Zerve notebook — a dataframe, a model result, a summary table — can be called directly into a Streamlit application with a single line of code.

archetype_df = variable("kmeans_user_archetypes", "archetype_cluster_df")

That means the model and the dashboard are one connected system. No exporting CSVs. No manual uploads. No stale data. Every time the dashboard loads, it pulls live results directly from the notebook. That is not a minor convenience — it is a fundamental architectural advantage that made this entire project possible as a single coherent pipeline.

The Dataset

Each row in the dataset is one user. Across each user there are 43 behavioural features — signals describing what that user has done on the platform. The most important ones:

events_first_7d — Total activity in the first seven days. The single most intuitive early signal.

days_active_first_7d — How many of those seven days did they return? Frequency matters as much as volume.

agent_tool_calls — How many times did they call an AI agent? This is the platform's core capability. Usage here is the strongest predictor of everything that follows.

genai_events — Broader generative AI engagement. Prompting, AI-assisted code, model interactions.

credits_used_total — Credits are the platform's primary resource. Zero credit usage means zero real output.

feature_adoption_breadth — Did they explore multiple features or stay in one corner of the platform?

collaboration_index — How much did they work with others? Collaborative users embed themselves in team workflows, which dramatically increases retention.

execution_success_rate — When they ran code, did it work? A proxy for whether they are getting real value or hitting frustration.

target_success — The outcome variable. One or zero. Succeeded or did not. This is what both models are built around.

The Approach — Two Models, One System

I built two completely separate models. Understanding why both are necessary is the core of the whole project.

Model One — XGBoost Gradient Boosting Machine

A supervised learning model trained on all 43 features against the target_success label. The model learns which combinations of behaviour predict success and produces a probability for every user — which I scale into a zero to one hundred success likelihood score.

I chose XGBoost because user behaviour data is non-linear and messy. The relationship between agent calls and success is not a straight line — it interacts with credit usage, with feature breadth, with early engagement patterns. GBM handles these interactions naturally.

I used SHAP values — SHapley Additive exPlanations — to make the model interpretable. SHAP tells you not just what the model predicts, but which features drove each individual prediction up or down. This is how I can tell you with confidence what the top predictive signals are, rather than just reporting an accuracy score.

Model Two — KMeans Clustering, k=4

An unsupervised model — no success label involved. It looks purely at behavioural similarity and finds natural groups in the data. The number four was chosen because it produces four genuinely distinct archetypes without over-fragmenting the user base.

Why run clustering separately from prediction?

Because a score alone is not actionable. If a customer success manager sees a user scored 24, the immediate question is — what do I do? The score tells you how urgent the situation is. The cluster tells you what action to take. A Casual Explorer scoring 24 needs a re-engagement tutorial. An At-Risk User scoring 24 needs free credits to try AI features for the first time. Same score. Completely different response. You need both.

The Four Archetypes

Power Coders
Heavy agent usage, high credit consumption, scripting and deploying to production. These users found the platform's core value immediately. They have the highest success rate of any group and respond to capability-led upsells — more compute, advanced APIs, dedicated support.

Collaborators
Socially active builders working in shared canvases and team workflows. Their viral potential is enormous — every collaborator is a gateway to additional team seats. The right intervention is a frictionless team upgrade prompt when their collaboration score crosses a threshold.

Casual Explorers
They signed up, explored, created some things — but never formed a consistent habit. There is genuine intent here. It just has not converted. The thirty-day window is critical. A well-timed tutorial showing them a relevant use case reactivates more than twelve percent of this group.

At-Risk Users
The largest group. They browse canvases, perform basic actions, appear active — but they have never touched the AI features that drive real retention. Without a catalyst, they churn silently. The highest-leverage intervention is a time-sensitive free credit grant that removes the cost barrier to first AI use.

The Dashboard

Built in Streamlit, connected directly to Zerve notebook variables, and designed to serve five completely different business audiences from one tool.

Platform Overview — The executive layer. Success rate, score distribution across tiers, cluster breakdown, archetype success rate comparison, and score distribution by cluster. The full picture in one screen.

User Lookup — For the customer success team. Paste any user ID and get their score, cluster, raw probability, and a plain-English recommended action. A comparison chart shows their behavioural profile against the dataset median — you can see exactly which dimensions they are underperforming on and why they scored the way they did.

Cluster Recommendations — For the growth team. Each archetype has a specific recommended intervention with a trigger condition written as an executable rule — ready to drop directly into a CRM or marketing automation workflow. No manual review required.

ROI Table — For leadership and finance. Cluster size multiplied by expected lift gives estimated users impacted. Multiply by average contract value and you have a business case, not just a data science result.

Cluster Narratives — For everyone else. Plain-English portraits of each archetype written for product managers, marketers, and executives who need the story rather than the statistics. This is the layer that makes the intelligence portable — it travels into strategy documents, board decks, and onboarding playbooks.

The Finding

After all of it — the modelling, the clustering, the SHAP analysis — the answer to Zerve's question is this.

Early, deep engagement with AI features is the single strongest predictor of long-term user success.

Not tenure. Not the number of canvases created. Not account age. Whether a user called an agent, consumed credits, and engaged with generative AI in their first seven days is what separates the users who succeed from the ones who quietly disappear.

The users who stay on the surface — active but never crossing into the AI core - are not disengaged. They are one well-timed intervention away from becoming Power Coders. The platform's job is to make that crossing happen before day thirty.

That is what this dashboard is built to do.

Why Zerve Made This the Right Tool for the Job

I could have built this in any notebook environment. But Zerve's variable() system meant the model outputs flowed directly into the dashboard without any intermediate steps. The clustering results, the feature importances, the scored dataframe — all live in the notebook, all accessible in the app with one line.

That is not a workflow convenience. It is the reason the system works as a single coherent product rather than a collection of disconnected scripts. For anyone building production-grade data applications, that kind of tight integration between computation and presentation is genuinely rare.

#MachineLearning #DataScience #Streamlit #Python #XGBoost #KMeans #SHAP #ProductAnalytics #UserSuccess #ChurnPrediction #Zerve #BuildInPublic #MLEngineering #HackathonProject #DataDriven

RAG — Building Reliable AI Pipelines

Madhesh T — Sat, 28 Feb 2026 17:46:53 +0000

AUTHOR INTRO

I am Madhesh, a passionate developer with a strong interest in Agentic AI and DevOps. I enjoy learning new things, and I have always wanted to start writing blogs to connect with people. I chose to work on RAG because large language models (LLMs) are everywhere, and RAG adds significant power to them by providing proper context for user queries.

ABSTRACT

LLMs often hallucinate on domain-specific or recent data because they don’t have the proper context for user queries. Traditional LLM outputs rely solely on trained data, which may not contain up-to-date or domain-specific information. RAG overcomes these problems with strong retrieval pipelines. In this blog, I walk through designing and implementing a complete RAG pipeline using Elastic as the vector database. From ingesting documents to semantic retrieval and LLM augmentation, discover how Elastic’s vector capabilities deliver accurate, hallucination-resistant AI applications.

NAIVE SEARCH (KEYWORD SEARCH)

The naive way to search for relevant content in a document or database is by using a basic keyword search.

Example - search in a file:

grep "keyword" file.txt

Example - SQL keyword search in a database:

SELECT * FROM table_name WHERE column_name LIKE '%keyword%';

Keyword search works by finding exact matches. But if the user uses different words with the same meaning, keyword search fails. That is where semantic search and vector embeddings become useful.

TF-IDF

TF-IDF is a classic method to score how important a term is in a document relative to a corpus.

TF (Term Frequency) looks at how many times a word appears in a specific document.
DF (Document Frequency) is the number of documents where the word appears.
IDF (Inverse Document Frequency) measures the importance of the word across the entire document set.

DF(t) = number of documents containing term t

IDF(t) = log(N / DF(t)),   where N = total number of documents

TF-IDF weights terms that are frequent in a document but rare in the corpus, giving more relevant ranking than pure keyword counts.

BM25

BM25 is a ranking algorithm used in retrieval systems to determine the relevance of documents to a given user query. It is the default ranking algorithm used in systems like Elasticsearch and Whoosh. BM25 improves over TF-IDF by

Normalizing for document length
Saturating term frequency (more occurrences do not increase importance linearly)
Producing better relevance scoring in practice

Compute BM25 in Python:

from rank_bm25 import BM25Okapi

docs = [
    "machine learning is powerful",
    "deep learning uses neural networks",
    "machine learning and AI"
]

tokenized = [doc.split() for doc in docs]
bm25 = BM25Okapi(tokenized)

query = "machine learning".split()
scores = bm25.get_scores(query)
print(scores)

BM25 produces a score for each document based on the query and ranks them by relevance.

VECTOR EMBEDDINGS

When a user query uses a different word but similar meaning, keyword methods fail. This is where vector embeddings solve the problem.

Embeddings transform text into numerical vectors that capture semantic meaning. Similar texts have vectors close to each other in vector space.

Generate embeddings:

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("all-MiniLM-L6-v2")
texts = ["machine learning", "deep learning"]
vectors = model.encode(texts)
print(vectors.shape)   # (2, 384)

INTRO TO RAG PIPELINES

A RAG pipeline consists of several stages. The process of document ingestion occurs, and when an online query comes in, the retrieval of relevant documents and the generation of a response occur. Then, with the context it has, it augments and starts to generate an accurate response.

RELEVANT CONTEXT AND PREPROCESSING

First, ingest raw data into the RAG system. To make it effective, choose proper preprocessing techniques:

Chunking

Chunking breaks large documents into smaller pieces that are easier to index and retrieve. Good chunking balances context with retrieval efficiency.

VECTOR DATABASE

Once text is chunked and embedded into vectors, store it in a vector database (e.g., Elasticsearch). The vector DB stores embeddings and performs similarity search to match user queries with relevant chunks.

ELASTICSEARCH – SETUP & CODE

1. Create index with vector field

curl -X PUT "localhost:9200/docs" -H "Content-Type: application/json" -d '
{
  "mappings": {
    "properties": {
      "text":       { "type": "text" },
      "vector":     { "type": "dense_vector", "dims": 384 }
    }
  }
}'

2. Insert document with embedding

curl -X POST "localhost:9200/docs/_doc" -H "Content-Type: application/json" -d '
{
  "text":   "machine learning is powerful",
  "vector": [0.12, -0.93, ...]   # real embedding vector
}'

3. Query using BM25 (keyword search)

curl -X GET "localhost:9200/docs/_search" -H "Content-Type: application/json" -d '
{
  "query": {
    "match": {
      "text": "machine learning"
    }
  }
}'

4. Query using Vector Similarity

curl -X GET "localhost:9200/docs/_search" -H "Content-Type: application/json" -d '
{
  "knn": {
    "field":        "vector",
    "query_vector": [0.12, -0.93, ...],
    "k":            3,
    "num_candidates": 10
  }
}'

5. Hybrid Search (BM25 + Vector)

curl -X GET "localhost:9200/docs/_search" -H "Content-Type: application/json" -d '
{
  "query": {
    "bool": {
      "should": [
        { "match": { "text": "machine learning" }},
        {
          "knn": {
            "field":        "vector",
            "query_vector": [0.12, -0.93, ...],
            "k": 3,
            "num_candidates": 10
          }
        }
      ]
    }
  }
}'

Hybrid search combines keyword ranking (BM25) and semantic ranking (vector similarity).

RERANKING

Reranking is a post-processing step that improves result relevance by applying stronger scoring methods. It considers semantic relevance and similarity to reorder results for better quality. Reranking is more computationally expensive and is usually applied only to top results.

INTEGRATING ELASTIC WITH LLMS

Elastic can serve as the retrieval backend for a RAG system. When a user query arrives:

The query is embedded(converted to vector embeddings).
Elastic retrieves the most similar chunks (vector search).
The retrieved chunks are passed to the LLM.
The LLM generates an answer grounded in retrieved context.

This integration reduces hallucination and increases response accuracy.

PRODUCTION INSIGHT

When building a RAG pipeline, most developers focus heavily on the LLM and ignore the retrieval layer. In practice, retrieval quality matters more than model size. If the retriever returns irrelevant chunks, even the best LLM will confidently generate incorrect answers. I realized this while experimenting with chunk sizes and indexing strategies and small changes in chunking and overlap significantly changed answer quality.

Another important point is that hybrid search often performs better than pure vector search. Vector similarity is powerful for semantic understanding, but keyword signals still matter in production. In many cases, combining BM25 with vector search improved precision and reduced noise. Reranking also made a visible difference, especially when the initial retrieval returned loosely related results.

Latency is another real-world factor that is often underestimated. Running embeddings, querying vectors, reranking, and then calling an LLM adds up quickly. In production systems, you must balance accuracy with response time. Tuning the top-K retrieval size, embedding model selection, and reranking depth directly impacts both performance and cost.

Finally, data freshness matters. RAG systems must support continuous indexing. If documents are not updated properly, the system becomes stale and starts returning outdated context. In production, retrieval pipelines must be monitored just like any other backend service.

DEPLOY RAG MODELS ON CLOUD

Elastic Cloud provides a fully managed Elasticsearch environment with built-in scaling, security, and monitoring. Instead of managing nodes, shard allocation, replication, and cluster health manually, Elastic Cloud handles infrastructure operations. This allows developers to focus on indexing documents, embedding pipelines, hybrid retrieval, and LLM integration rather than maintaining search infrastructure.

For a RAG pipeline, Elastic Cloud supports:

Dense vector fields for storing embeddings
kNN vector search for semantic retrieval
BM25-based keyword search
Hybrid search combining lexical and semantic signals
Secure deployment with role-based access control

A production-ready RAG architecture on Elastic Cloud typically includes:

An embedding model (self-hosted or API-based)
An Elastic Cloud deployment with vector-enabled indices
A backend service that performs retrieval and prompt construction
An LLM provider for generation
Monitoring via Elastic Stack (metrics, logs, performance tracking)

As embeddings scale into millions of vectors, cluster sizing becomes critical. Elastic Cloud allows vertical and horizontal scaling by adjusting node size and instance count without downtime. This is essential when handling increasing search traffic or expanding document collections.

Security is also a major factor. Elastic Cloud provides TLS encryption, API keys, and access controls out of the box. In AI applications dealing with private documents or enterprise data, this becomes non-negotiable.

In real-world systems, RAG is not only about retrieval and generation quality. It is about cluster stability, index performance, scaling strategy, and operational visibility. Elastic Cloud provides the infrastructure layer that makes large-scale RAG systems stable, secure, and production-ready.

CONCLUSION

Engineers can over-engineer things. The true value of RAG lies in strengthening LLM responses with real context from scalable systems like Elasticsearch. RAG makes LLMs less prone to hallucination and vastly improves relevance and accuracy.

If neither step 1 (retrieval) nor step 2 (generation) gives high-quality results, then consider improving both parts of a RAG pipeline and the retrieval components.

Project Repository:

Github on RAG

Note: The content of this blog is fully organic. AI was utilized solely for grammatical error correction and Structural alignment.